- The cost of downloading the file on the network
- The cost of parsing and compiling the uncompressed file once downloaded
- The memory cost
The combination is very expensive.
I see a lot of very heavy sites using them, but then, my perspective is very biased as the companies that I work with work with me precisely because they are facing performance challenges. I was curious just how common the situation is and just how much of a penalty we’re paying when we make these frameworks the default starting point.
Thanks to HTTP Archive, we can figure that out.
In total, HTTP Archive tracks 4,308,655 desktop URLs, and 5,484,239 mobile URLs. Among the many data points HTTP Archive reports for those URLs is a list of the detected technologies for a given site. That means we can pick out the thousands of sites that use various frameworks and see how much code they’re shipping, and what that costs the CPU.
I ran all the queries against March of 2020, the most recent run at the time.
I decided to compare the aggregate HTTP Archive data for all sites recorded against sites with React, Vue.js, and Angular detected1.
|Framework||Mobile URLs||Desktop URLs|
Hopes and dreams
Before we dig in, here’s what I would hope.
In an ideal world, I believe a framework should go beyond developer experience value and provide concrete value for the people using our sites. Performance is just one part of that—accessibility and security both come to mind as well—but it’s an essential part.
So in an ideal world, a framework makes it easier to perform well by either providing a better starting point or providing constraints and characteristics that make it hard to build something that doesn’t perform well.
The best of frameworks would do both: provide a better starting point and help to restrict how out of hands things can get.
Looking at the median for our data isn’t going to tell us that, and in fact leaves a ton of information out. Instead, for each stat, I pulled the following percentiles: the 10th, 25th, 50th (the median), 75th, and 90th.
The 10th and 90th percentiles are particularly interesting to me. The 10th percentile represents the best of class (or at least, reasonably close to the best of class) for a given framework. In other words, only 10% of all sites using a given framework reach that mark or better. The 90th percentile, on the other hand, is the opposite of the spectrum—it shows us how bad things can get. The 90th percentile represents the long-tail—that last 10% of sites with the highest number of bytes or largest amount of main thread time.
|Sites with jQuery||110.3kb||219.8kb||430.4kb||748.6kb||1,162.3kb|
|Sites with Vue.js||244.7kb||409.3kb||692.1kb||1,065.5kb||1,570.7kb|
|Sites with Angular||445.1kb||675.6kb||1,066.4kb||1,761.5kb||2,893.2kb|
|Sites with React||345.8kb||441.6kb||690.3kb||1,238.5kb||1,893.6kb|
|Sites with jQuery||121.7kb||242.2kb||458.3kb||803.4kb||1,235.3kb|
|Sites with Vue.js||248.0kb||420.1kb||718.0kb||1,122.5kb||1,643.1kb|
|Sites with Angular||468.8kb||716.9kb||1,144.2kb||1,930.0kb||3,283.1kb|
|Sites with React||308.6kb||469.0kb||841.9kb||1,472.2kb||2,197.8kb|
I mentioned earlier that even if the starting point is a little off, I would hope that a framework could still provide value by limiting the upper bound in some way.
The same can’t be said of the other frameworks.
Just as with the 10th percentile, Angular and React driven sites tend to distance themselves from others at the 90th percentile, and not in a very flattering way.
It’s clear from the data that sites with these frameworks in place tend to pay a large penalty in terms of bytes. But of course, that’s just one part of the equation.
|Sites with jQuery||575.3ms||1,147.4ms||2,555.9ms||5,511.0ms||10,349.4ms|
|Sites with Vue.js||1,130.0ms||2,087.9ms||4,100.4ms||7,676.1ms||12,849.4ms|
|Sites with Angular||1,471.3ms||2,380.1ms||4,118.6ms||7,450.8ms||13,296.4ms|
|Sites with React||2,700.1ms||5,090.3ms||9,287.6ms||14,509.6ms||20,813.3ms|
|Sites with jQuery||199.6ms||399.2ms||877.5ms||1,779.9ms||3,215.5ms|
|Sites with Vue.js||350.4ms||650.8ms||1,280.7ms||2,388.5ms||4,010.8ms|
|Sites with Angular||482.2ms||777.9ms||1,365.5ms||2,400.6ms||4,171.8ms|
|Sites with React||508.0ms||1,045.6ms||2,121.1ms||4,235.1ms||7,444.3ms|
There are some very familiar themes here.
Those percentages look much better than at the 10th percentile, but keep in mind that the bulk numbers are pretty scary: that’s 20.8s of main thread work for sites built with React at the 90th percentile on mobile devices. (What, exactly, is happening during that time is a topic for a follow-up post, I think.)
There’s one potential gotcha (thanks Jeremy for making sure I double-checked the stats from this angle)—many sites will pull in multiple libraries. In particular, I see a lot of sites pulling jQuery in alongside React or Vue.js as they’re migrating to that architecture. So, I re-ran the queries, only this time I only included URLs that included only React, jQuery, Angular or Vue.js not some combination of them.
|Sites with only jQuery||542.9ms||1,062.2ms||2,297.4ms||4,769.7ms||8,718.2ms|
|Sites with only Vue.js||944.0ms||1,716.3ms||3,194.7ms||5,959.6ms||9,843.8ms|
|Sites with Angular||1,328.9ms||2,151.9ms||3,695.3ms||6,629.3ms||11,607.7ms|
|Sites with React||2,443.2ms||4,620.5ms||10,061.4ms||17,074.3ms||24,956.3ms|
First, the unsurprising bit: when only one framework is used, performance improves far more often than not. The numbers for every framework look better at the 10th and 25th percentile. That makes sense. A site that is built well with one framework should perform better than a site that is built well with two or more.
In fact, the numbers for every framework look better at each percentile with one curious exception. What surprised me, and the reason I ended up including this data, is that at the 50th percentile and beyond, sites using React perform worse when React is the only framework in use.
It’s a bit odd, but here’s my best guess.
If you have React and jQuery running alongside each other, you’re more likely to be in the midst of a migration to React, or a mixed codebase. Since we have already seen that sites with jQuery spend less time on the main thread than sites with React, it makes sense that having some functionality still driven by jQuery would bring the numbers down a bit.
As you move away from jQuery and focus more on React exclusively, though, that changes. If the site is built really well and you’re using React sparingly, you’re fine. But for the average site, more work inside of React means the main thread receives an increasing amount of strain.
The mobile/desktop gap
Another angle that’s worth looking at is just how large the gap is between that mobile experience and the desktop experience. Looking at it from a bytes perspective, nothing too scary jumps out. Sure, I’d love to see fewer bytes passed along, but neither mobile nor desktop devices receive significantly more bytes than the other.
But once you look at the processing time, the gap is significant.
|Sites with jQuery||188.2%||187.4%||191.3%||209.6%||221.9%|
|Sites with Vue.js||222.5%||220.8%||220.2%||221.4%||220.4%|
|Sites with Angular||205.1%||206.0%||201.6%||210.4%||218.7%|
|Sites with React||431.5%||386.8%||337.9%||242.6%||179.6%|
While some variance is expected between a phone and a laptop, seeing numbers this high tells me that the current crop of frameworks isn’t doing enough to prioritize less powerful devices and help to close that gap. Even at the 10th percentile, React sites spend 431.5% more time on the main thread on mobile devices as they do on desktop devices. jQuery has the lowest gap of all frameworks, but even there, it equates to 188.2% more time on mobile devices. When we make the CPU work harder—and increasingly we are—folks with less powerful devices end up holding the bill.
The big picture
Good frameworks should provide a better starting point on the essentials (security, accessibility, performance) or have built-in constraints that make it harder to ship something that violates those.
That doesn’t appear to be happening with performance (nor with accessibility, apparently).
It’s worth noting that because sites with React or Angular spend more time on the CPU than others, that isn’t necessarily the same as saying React is more expensive on the CPU than Vue.js. In fact, it says very little about the performance of the core frameworks in play and much more about the approach to development these frameworks may encourage (whether intentionally or not) through documentation, ecosystem, and general coding practices.
What is clear: right now, if you’re using a framework to build your site, you’re making a trade-off in terms of initial performance—even in the best of scenarios.
Some trade-off may be acceptable in the right situations, but it’s important that we make that exchange consciously.
There’s reason for optimism. I’m encouraged by how closely the folks at Chrome have been working with some of these frameworks to help improve their performance.
But I’m also pragmatic. New architectures tend to create performance problems just as often as they solve them, and it takes time to right the ship. Just as we shouldn’t expect new networks to solve all our performance woes, we shouldn’t expect that the next version of our favorite framework is going to solve them all either.
If you are going to use one of these frameworks, then you have to take extra steps to make sure you don’t negatively impact performance in the meantime. Here are a few great starting considerations:
- Is there a lighter alternative (Preact, Svelte, etc.) that gets you 90% of the way there?
- If you’re going with a framework, does anything exist that provides better, more opinionated defaults (ex: Nuxt.js instead of Vue.js, Next.js instead of React, etc.)?2
- If you’re using a framework for the developer ergonomics, do you need to ship it down to the client, or can you handle that all on the server?
These are generally good things to consider regardless of your technology choice, but they’re particularly important if you’re starting with a performance deficit from the beginning.
1. I considered using other baselines.
With either baseline, the data would have looked less favorable and more dramatic. Ultimately, I ended up using the aggregate data as the baseline because:
- It's what is used broadly in the community whenever this stuff is discussed.
- It avoids derailing the conversation with debates about whether or not sites without frameworks are complex enough to be an accurate comparison. You can build large, complex sites without using a framework—I've worked with companies who have—but it's an argument that would distract from the otherwise pretty clear conclusions from the data.
Still, as a few folks pointed out to me when I was showing them the data, it's interesting to see these alternate baselines as they do make it very clear how much these tools are messing with the averages.
2. I suspect folks could start with just the framework and build something that outperforms what Next.js or Nuxt.js gives you. The long-tail of data here, though, shows that likely isn't the case for many sites. It takes time to build out all the plumbing they provide, and build it well. Still, it is worth considering how many layers of abstraction we're willing to pull in before making the decision.