TUESDAY 11TH OF FEBRUARY 2020
Here are some common questions I’m asked when I talk with people about performance:
Today, I’m very excited to announce the release of a new project that helps answer those questions – and more!
Page Speed Benchmarks is an interactive dashboard that lets you explore and compare web performance data for leading websites across several industries – from retail to media.
With Page Speed Benchmarks, you can do things like:
If you already like tools like the HTTP Archive, I think you'll love how you can use Page Speed Benchmarks to complement the insights you're already getting. Keep reading to find out how we set up these benchmarks, and how you can mine our test data – even if you're not a SpeedCurve user – for your own performance research.
Over the past ten years or so, I’ve put a lot of time into researching industry benchmarks for web performance. What makes me especially excited about Page Speed Benchmarks is that all our data is interactive, ongoing, and publicly available.
This means you have access to the same data that we do. You don’t have to wait for us to release monthly reports. And you don’t need to have a SpeedCurve account or any sort of login to start exploring our test results and mining for meaningful data you can use in your own projects. How awesome is that?!
You can use the dropdowns at the top of the Page Speed Benchmarks dashboard to explore some of the metrics we're tracking. If you want to get more insight, click the "Explore this site" link below each chart to see the test results for any of the sites. There you’ll see a waterfall chart alongside a filmstrip view that shows how the page renders, like this one:
We focus on these metrics in the main Benchmarks dashboard, and from there you can drill down into the individual test results to see many more:
It’s inspiring to see how fast a page can be under the right conditions. And it’s eye-opening to see how that same page can suffer on low-CPU devices and slow connections. (Tim Kadlec gives a strong argument for why you should test on slower connections in this excellent post.)
Drill down into the waterfalls for each page to see exactly what separates the fast sites from the slow ones.
This is one of my favourite aspects of this research. More on this to come!
Don’t think of this as "competitive benchmarking" as much as it is “aspirational benchmarking”. Looking at the historical data for each site will give you an idea as to which sites you can benchmark in your own testing. #perfgoals
These tests are snapshots, based on synthetic tests, with all the caveats that this entails.
Page Speed Benchmarks isn’t a contest or a set of "top ten" lists. These aren’t the fastest or slowest sites on the web. They’re simply a set of sites we’ve chosen to track over time. We’re more interested in digging into the test results to see what we can learn about what makes fast sites fast and slow sites slow.
Every site has bad hair days. For example, highly dynamic sites, such as media sites, serve different third parties throughout the day. A slow start render time could be caused by a single script that our synthetic test happened to capture. Or it could be caused by a temporary backend issue. You get the idea.
Let’s start by looking at the results for US media sites on a fast desktop connection. This is an interesting group to start with because these sites face some aggressive performance challenges:
All of the above traits are even more pronounced in the US than in the EU, which is what makes this industry compelling to track if you care about performance.
In previous posts (here and here, for example), we’ve strongly advocated validating your metrics by looking at filmstrip views of your pages rendering. It may challenge some of your assumptions about the relevance of some metrics as proxies for user experience.
With Page Speed Benchmarks, we can sort sites by how they rank across popular metrics. The highlighted frames in the filmstrips show when each metric is measured. So let’s do that now, looking at a trio of metrics that often are conflated with user-perceived performance: Start Render, Speed Index, and Visually Complete. (Here's a helpful glossary of common web performance metrics.)
This sorting stacks up pretty neatly from fastest to slowest. This isn’t a huge surprise, which is why it’s a good idea for you to consider including start render as a metric to track.
This sorting isn't quite as consistent as start render, but with the exception of USA Today at the bottom, seems mostly valid.
For this set of sites, visually complete is all over the map. For half the sites, it doesn't fire until after 5.5 seconds, which is why we're not seeing it in this screenshot.
The purpose of doing this investigation wasn’t to criticize any of these metrics, but rather to make it clear that metrics that are relevant for some pages won’t necessarily be relevant for others. At the risk of sounding like a broken record, you really need to look at your own data and your own filmstrips to make sure you’re picking the right metrics to focus on for your site.
Now let’s look a bit closer at this same set of US media sites and see what we can learn by drilling down into the test results...
At the time of writing this post, all ten sites started rendering in less than 3 seconds, and six sites had a start render of 1 second or less. No surprises here.
Now let’s take a look at how the same ten sites rendered on a slow mobile connection. (Remember: This is an emulated Nexus 5X mobile on a slow 3G connection.) The fastest site didn’t start to render till around 7 seconds, and the slowest start render was at 36 seconds.
Now that we've gotten the obvious observations out of the way, let's ask some meatier questions...
While a couple of pages fall below the HTTP Archive median size of 1900KB, the rest are larger – with two pages topping out in the 6-8MB range. (As someone who’s also obsessed with data usage – I spend a lot of time reading articles while I'm out roaming the world – this is a reminder that I need to be extra mindful of my news consumption when I’m out and about.)
If you’re anything like me, you’re extremely curious about those spikes, so let’s take a closer look. To do that, all you need to do is click on any test point in a time series, and then click the "View Test" link:
This takes us to a detailed test result for The New York Times, where we can see how popular rendering metrics align with the filmstrip. What’s great to see here is that, despite the total size of this page, the filmstrip view shows that it actually renders pretty quickly – yay!
So let’s look into the mobile test results for The New York Times to see what they were serving mobile devices at the same date and time:
As you can see above, the page doesn’t start rendering until more than 5 seconds have passed. It’s not ideal, but it’s not terrible – again, keeping in mind this is on a slow 3G connection.
Next, let’s open up the waterfall to see what’s happening before the page starts to render:
In these charts, the blue bars represent HTML, so there’s roughly 4.5 seconds of downloading and parsing there alone. The green hashed bars represent blocking CSS. These assets have download/parse durations ranging from roughly 500 milliseconds to 2.5 seconds. So there you go. No big mystery here – just a reminder to optimize/reduce your CSS whenever you can.
We came here to look at page weight, so let’s do that. On the plus side, The New York Times is serving a much lighter version of its home page to mobile, partly thanks to the fact that it’s not serving video content. This is great to see:
These are Long Tasks – browser events that block the CPU for more than 50ms. Long Tasks make pages feel janky. The more Long Tasks there are on a page, the greater the likelihood that the page will hurt user experience.
Again, keep in mind that these results are for a slow mobile connection – but users on slow mobile connections matter! Remember that people’s devices and connections vary throughout the day. Someone who visits your site on a fast desktop connection when they’re at work could also very well come back when they’re riding the train home from work, using spotty 3G on their phone. Inconsistent performance is frustrating!
We’re veering toward the topic of third-party performance, which is HUGE and merits a separate post of its own, so I’m going to stop here for today. I’m already planning another dive into this data, where I’ll focus exclusively on third parties. And of course, you’re welcome to go ahead and get started without me. :)
What we’ve covered today is truly just the tip of the iceberg. I strongly encourage you to head to Page Speed Benchmarks and do your own investigation. If you find anything interesting, let me know! And if you have any suggestions or feedback about the benchmarks, I'd love to hear that, too.comments powered by Disqus