I was helping someone recently who had an issue where their Largest Contentful Paint oscillated between very different results. Most of the time, it was ~6 seconds, but every once in a while (maybe 2 out of 9 tests or so), it would be 11 seconds or more.
Their initial reaction was to get frustrated at the measurement process and tool in question—why was it messing up so frequently?
Turns out, they had a race condition
The tests weren’t flawed—they were exposing a very real issue with their website.
Variability in performance data is a frequent industry complaint, and as a result, we see a lot of tools going out of their way to iron out that variability—to ensure “consistent” results. And sometimes, yeah, the tool is at fault.
But just as often, going out of our way to smooth over variability in data is counterproductive. Variability is natural and glossing over it hides very real issues.
Variability itself isn’t the issue. Variability without providing a clear way to explore that variability and understand why it occurs? That’s the real problem.