Previously, I was using an ObservedObject to hold layout data. This allowed precise control of when the view graph was updated but the data wasn't persisting across updates. State wrapped values do persist but you loose some precision and flexibility.
The diffing that goes on behind the scenes is a bit magical and I've learned to trust it a bit more. Regardless, now that layout data isn't being reset between view updates, the scrolling and inline adjustments to layout are smooth and imperceptible.