SpeedCurve Blog

Introducing Page Speed Benchmarks – a new resource for the performance community

Tue, 11 Feb 2020 00:00:00 +1300

Here are some common questions I’m asked when I talk with people about performance:

Which metrics should I care about?
What types of devices and connections should I test on?
Which third parties should I be most concerned about?
How fast should I be?
What are some good sites I can use for benchmarking?

Today, I’m very excited to announce the release of a new project that helps answer those questions – and more!

Page Speed Benchmarks is an interactive dashboard that lets you explore and compare web performance data for leading websites across several industries – from retail to media.

With Page Speed Benchmarks, you can do things like:

See what the different metrics actually mean in terms of user-perceived performance
Compare how the same page renders on fast vs slow devices and connections
Understand what makes fast sites fast (and slow sites slow)
Get insights into how third parties can perform on different sites
Identify sites you can use for your own competitive benchmarking

If you already like tools like the HTTP Archive, I think you'll love how you can use Page Speed Benchmarks to complement the insights you're already getting. Keep reading to find out how we set up these benchmarks, and how you can mine our test data – even if you're not a SpeedCurve user – for your own performance research.

Interactive, ongoing, and public

Over the past ten years or so, I’ve put a lot of time into researching industry benchmarks for web performance. What makes me especially excited about Page Speed Benchmarks is that all our data is interactive, ongoing, and publicly available.

This means you have access to the same data that we do. You don’t have to wait for us to release monthly reports. And you don’t need to have a SpeedCurve account or any sort of login to start exploring our test results and mining for meaningful data you can use in your own projects. How awesome is that?!

How we set up our testing

Home pages of 10 leading sites in the US and the EU, in each of the following industries: Auto, Finance, Media, Retail, Tech, and Travel. (We’ll add more later, but we felt like this was a solid start.)
Tested on our private agents in Frankfurt (for the EU sites) and US East Coast (for the US sites).
Tested once per day on a Chrome desktop browser with a fast connection (25Mbps/10Mbps 10ms RTT).
Tested once per day on an emulated Nexus 5X mobile at 3G Slow (400Kbps/400Kbps 400ms RTT). Find out more about why it's so important to test at slow speeds.
Three tests per test time, with the medians used in the charts.

What you can do with these benchmarks:

1. Learn what the different metrics mean in terms of user-perceived performance.

You can use the dropdowns at the top of the Page Speed Benchmarks dashboard to explore some of the metrics we're tracking. If you want to get more insight, click the "Explore this site" link below each chart to see the test results for any of the sites. There you’ll see a waterfall chart alongside a filmstrip view that shows how the page renders, like this one:

We focus on these metrics in the main Benchmarks dashboard, and from there you can drill down into the individual test results to see many more:

Backend – The first byte of the base HTML is received by the browser (AKA Time to First Byte)
Start Render – The first non-white content is painted in the browser
SpeedIndex – The average time at which visible parts of the page are displayed
Largest Contentful Paint – The largest element – usually image or video – in the viewport is rendered
First CPU Idle – The page is minimally interactive (most UI elements are interactive, page responds to most user input reasonably quickly)
Time to Interactive – The page becomes interactive (displays useful content, UI elements are interactive, page responds to user interactions within 50ms)
Visually Complete – All content in the viewport has rendered, and nothing in the viewport changes as the rest of the page loads
Fully Loaded – The page has completely loaded with no network activity for 2 seconds

2. Appreciate the difference between fast vs slow.

It’s inspiring to see how fast a page can be under the right conditions. And it’s eye-opening to see how that same page can suffer on low-CPU devices and slow connections. (Tim Kadlec gives a strong argument for why you should test on slower connections in this excellent post.)

3. Understand what makes fast sites fast (and slow sites slow).

Drill down into the waterfalls for each page to see exactly what separates the fast sites from the slow ones.

4. Get some insights into how third parties can perform on different sites.

This is one of my favourite aspects of this research. More on this to come!

5. Find sites you can include with your own benchmarking

Don’t think of this as "competitive benchmarking" as much as it is “aspirational benchmarking”. Looking at the historical data for each site will give you an idea as to which sites you can benchmark in your own testing. #perfgoals

What we DON'T want you to do with these benchmarks:

1. Don’t confuse these findings with real user data and experiences.

These tests are snapshots, based on synthetic tests, with all the caveats that this entails.

2. Don’t use these results as leaderboards.

Page Speed Benchmarks isn’t a contest or a set of "top ten" lists. These aren’t the fastest or slowest sites on the web. They’re simply a set of sites we’ve chosen to track over time. We’re more interested in digging into the test results to see what we can learn about what makes fast sites fast and slow sites slow.

3. Don’t name and shame.

Every site has bad hair days. For example, highly dynamic sites, such as media sites, serve different third parties throughout the day. A slow start render time could be caused by a single script that our synthetic test happened to capture. Or it could be caused by a temporary backend issue. You get the idea.

Now let’s look around…

Let’s start by looking at the results for US media sites on a fast desktop connection. This is an interesting group to start with because these sites face some aggressive performance challenges:

Pages tend to have a lot of content, including heavy images and video.
Content is consumed on mobile devices, often over spotty connections.
Media sites are under tremendous pressure to generate ad revenue, so we see a lot of third-party ad services.
Teams at media organizations are tasked to collect a huge amount of user data, so we see a lot of tracking and analytics tags.

All of the above traits are even more pronounced in the US than in the EU, which is what makes this industry compelling to track if you care about performance.

Which metrics correlate with user-perceived performance?

In previous posts (here and here, for example), we’ve strongly advocated validating your metrics by looking at filmstrip views of your pages rendering. It may challenge some of your assumptions about the relevance of some metrics as proxies for user experience.

With Page Speed Benchmarks, we can sort sites by how they rank across popular metrics. The highlighted frames in the filmstrips show when each metric is measured. So let’s do that now, looking at a trio of metrics that often are conflated with user-perceived performance: Start Render, Speed Index, and Visually Complete. (Here's a helpful glossary of common web performance metrics.)

Start Render (first non-white content is painted in the browser)

This sorting stacks up pretty neatly from fastest to slowest. This isn’t a huge surprise, which is why it’s a good idea for you to consider including start render as a metric to track.

Speed Index (average time at which visible parts of the page are displayed)

This sorting isn't quite as consistent as start render, but with the exception of USA Today at the bottom, seems mostly valid.

Visually Complete (all content in the viewport has rendered, and nothing in the viewport changes)

For this set of sites, visually complete is all over the map. For half the sites, it doesn't fire until after 5.5 seconds, which is why we're not seeing it in this screenshot.

Takeaway: Test, validate, then test some more

The purpose of doing this investigation wasn’t to criticize any of these metrics, but rather to make it clear that metrics that are relevant for some pages won’t necessarily be relevant for others. At the risk of sounding like a broken record, you really need to look at your own data and your own filmstrips to make sure you’re picking the right metrics to focus on for your site.

Now let’s look a bit closer at this same set of US media sites and see what we can learn by drilling down into the test results...

Not surprisingly, all sites are fast on a fast desktop connection.

At the time of writing this post, all ten sites started rendering in less than 3 seconds, and six sites had a start render of 1 second or less. No surprises here.

And no surprise, all sites are slow on a slow mobile connection.

Now let’s take a look at how the same ten sites rendered on a slow mobile connection. (Remember: This is an emulated Nexus 5X mobile on a slow 3G connection.) The fastest site didn’t start to render till around 7 seconds, and the slowest start render was at 36 seconds.

Now that we've gotten the obvious observations out of the way, let's ask some meatier questions...

Page bloat is real. But does it always matter?

As someone who’s somewhat obsessed (okay, very obsessed) with page bloat, I find it interesting to click through to look at the detailed benchmarks, so let’s do that for the fast desktop group.

While a couple of pages fall below the HTTP Archive median size of 1900KB, the rest are larger – with two pages topping out in the 6-8MB range. (As someone who’s also obsessed with data usage – I spend a lot of time reading articles while I'm out roaming the world – this is a reminder that I need to be extra mindful of my news consumption when I’m out and about.)

If you’re anything like me, you’re extremely curious about those spikes, so let’s take a closer look. To do that, all you need to do is click on any test point in a time series, and then click the "View Test" link:

This takes us to a detailed test result for The New York Times, where we can see how popular rendering metrics align with the filmstrip. What’s great to see here is that, despite the total size of this page, the filmstrip view shows that it actually renders pretty quickly – yay!

When we scroll down the test results to see the breakdown of resources on the page, you can see that a lot of the weight can be attributed to JavaScript and images – and, notably, 8MB worth of video content.

So let’s look into the mobile test results for The New York Times to see what they were serving mobile devices at the same date and time:

As you can see above, the page doesn’t start rendering until more than 5 seconds have passed. It’s not ideal, but it’s not terrible – again, keeping in mind this is on a slow 3G connection.

Next, let’s open up the waterfall to see what’s happening before the page starts to render:

In these charts, the blue bars represent HTML, so there’s roughly 4.5 seconds of downloading and parsing there alone. The green hashed bars represent blocking CSS. These assets have download/parse durations ranging from roughly 500 milliseconds to 2.5 seconds. So there you go. No big mystery here – just a reminder to optimize/reduce your CSS whenever you can.

We came here to look at page weight, so let’s do that. On the plus side, The New York Times is serving a much lighter version of its home page to mobile, partly thanks to the fact that it’s not serving video content. This is great to see:

The not-so-good news is that the page is still over 4MB in size, with around 3MB of that coming from JavaScript alone. I’m pretty surprised to see that, as I really expected the bulk of the weight to come from images. Image weight comes in at 163KB, which is good by any standard.

JavaScript is clearly an issue here, so for a bit more fun, we can head over to the JavaScript dashboard (also available via the left-hand navbar). Below you can see a waterfall that focuses solely on JS. (Note that this chart was generated at the time of writing this post. We always shoe you the latest waterfall on the JS dashboard, so what you see when you visit the dashboard will probably look different than what you see here.)

Clearly there’s a lot of JavaScript on this page, but really what we care about is the impact of all that JS on user experience. To help do that, we show you all the scripts that take more than 50ms to execute:

These are Long Tasks – browser events that block the CPU for more than 50ms. Long Tasks make pages feel janky. The more Long Tasks there are on a page, the greater the likelihood that the page will hurt user experience.

Again, keep in mind that these results are for a slow mobile connection – but users on slow mobile connections matter! Remember that people’s devices and connections vary throughout the day. Someone who visits your site on a fast desktop connection when they’re at work could also very well come back when they’re riding the train home from work, using spotty 3G on their phone. Inconsistent performance is frustrating!

We’re veering toward the topic of third-party performance, which is HUGE and merits a separate post of its own, so I’m going to stop here for today. I’m already planning another dive into this data, where I’ll focus exclusively on third parties. And of course, you’re welcome to go ahead and get started without me. :)

Start exploring!

What we’ve covered today is truly just the tip of the iceberg. I strongly encourage you to head to Page Speed Benchmarks and do your own investigation. If you find anything interesting, let me know! And if you have any suggestions or feedback about the benchmarks, I'd love to hear that, too.

Six web performance resolutions for the new year

Wed, 08 Jan 2020 00:00:00 +1300

For the past two years, the performance.now() conference has been the most valuable performance event I've attended. So valuable, in fact, that I've made some of the talks the cornerstone of this list of performance resolutions for 2020. I'd love to know how many – if any – of these are on your list. As always, I'd love people's feedback!

1. Connect the impact of web performance to your business

Harry Roberts did an eye-opening talk called From Milliseconds to Millions, in which he shared practical wisdom gleaned from his experience as a performance consultant for some of the biggest sites in the world. There's a ton of useful material and inspiring case studies in here, including some really great examples of using correlation charts to see the relationship between performance and conversions.

Harry also shared how to use real user monitoring as part of an A/B test to measure the impact of a single third-party script on bounce rate:

To run your own A/B tests and create your own correlation charts in LUX, our RUM tool, here are some resources you'll find helpful:

2. Help other people in your organization care about performance

In my role here at SpeedCurve, I've talked with hundreds of our customers about how they do performance. One thing the most successful teams agree on: having a strong culture of web performance is a huge success factor.

In my talk, The 7 Habits of Highly Effective Performance Teams, I shared some tips and best practices that demonstrate how a strong culture of performance can help you:

Prevent regression
Reduce gatekeeping
Increase investment from the business

MORE: Performance culture tips and best practices

3. Find out how happy your users are

How do we know if users are happy? As members of the web performance community, we’ve been thinking about the best ways to answer that question for years. Now the observability community is asking the same questions, but coming at them from the opposite side of the stack.

In her talk Observability is for User Happiness, Emily Nakashima talked about how approaching web performance through the lens of observability has changed the way her team thinks about performance instrumentation and optimization.

Among other things, Emily talked about how performance budgets and SLOs can help you stay accountable. Here's how you can use SpeedCurve to set performance budgets and alerts.

4. Get to the bottom of third-party performance

In his talk Deep Dive into Third-Party Performance, Simon Hearne covered which third-party tags have the greatest impact on user experience, how ad blockers affect site speed, and what to look out for when evaluating a new third-party service. He also shared valuable techniques to manage third-party performance and security without creating friction with your marketing and analytics teams.

Your SpeedCurve third-party dashboard gives you the ability to monitor individual third parties over time, and create performance budgets for them.

5. Reduce the JavaScript on your pages... and keep it off

JavaScript is, byte-for-byte, the most expensive resource on the web and we’re using more of it than ever in our sites. In his talk When JavaScript Bytes, Tim Kadlec shared practical ways to reduce the amount of JS you're sending your users, as well as tools and approaches to make sure those bytes stay off.

Here's how SpeedCurve can help you manage your JavaScript:

6. Understand the latest performance metrics

We're obsessed with finding metrics that best represent the user experience. Largest Contentful Paint is one of the newest UX-oriented metrics on the landscape, and it looks promising. LCP measures when the largest element (usually an image or video) renders in the viewport.

Largest Contentful Paint is currently only available in Chrome, so it was fascinating to learn directly from Annie Sullivan, a member of the Chrome team, how they develop and test new metrics like LCP. Annie's talk, Lessons Learned from Performance Monitoring in Chrome, is a must-watch if you're interesting in learning about the life cycle of metrics and the rigour with which new metrics are developed and tested.

Here's how you can track Largest Contentful Paint using SpeedCurve:

Create performance budgets for LCP in both LUX and Synthetic
SpeedCurve's User Happiness metric (available in LUX) is an aggregate metric that includes Largest Contentful Paint

And here's our glossary of commonly used performance metrics.

What are your performance resolutions?

Do you have questions about anything I've mentioned here? Are you focusing on anything I haven't talked about? Let me know in the comments, or reach me on Twitter!

Sampling RUM: When and why it's a good idea

Mon, 16 Dec 2019 00:00:00 +1300

I confess, I’m not a statistician. While I pride myself on the 'A' I received in my college statistics class, admittedly it was on a pretty steep curve. That said, I’ve been looking at performance data for many years and have found myself on both sides of the debate about whether or not the practice of sampling performance data is inherently a good or bad idea.

When it comes to real user monitoring (RUM), I’m convinced that the marginal cost of collection, computation, storage, etc. is not always great enough to warrant a practice of collecting ALL THE THINGS by default.

Like any experiment, how you sample RUM data – as well as how much data to sample – depends on the answers you seek. While certainly not an exhaustive list, here are some questions you might ask when looking at implementing a sampled approach to real user monitoring...

Are sessions important to you?

At SpeedCurve, we’re big fans of not only looking at individual page performance, but also looking at performance in the context of a user’s entire visit to the site. This helps us do things like:

correlate various performance metrics with user behavior,
understand the user journey and popular pages so you can better model your synthetic testing, and
identify bottlenecks impacting a site’s business goals.

However, some teams we've worked with want the option to look at individual pages in isolation if they are part of a product team just focused on one area. That’s fine, too!

A lot of performance users really love that they can track business metrics specific to a session alongside their performance data. While real user monitoring tools aren’t typically looked at as a replacement for marketing analytics tools, having session metrics available with your performance data is pretty great for anyone who cares about performance. This way you can easily base your performance budgets on a business outcome, such as a cart conversion or a click on a “call to action” link.

In the case where sessions are important to you, finding a RUM solution (or creating one) that does session-based sampling is key. This does mean that the size of your sample for page views will likely not be exact. At SpeedCurve, we’re good with that and prefer to maintain the integrity of the session vs. sample too aggressively and break them up to stay within a strict page view budget.

Where should you do the sampling?

Another big thing to think about when applying sampling is where you want to sample. Having the ability to sample at the collection point as well as in the client/browser itself provides you with a lot of flexibility. We offer both options at SpeedCurve, but sometimes people aren’t sure which to go with.

In most cases, sampling from the collection endpoint (either through a vendor, your CDN or your origin) gets the job done and is a fine approach. However, for some more sophisticated users who want to be very sensitive to how they are spending their “beacon budget”, you may elect to sample at the client. This allows you to do things like increase the sample rate for an A/B test group for comparison with a control group when the test group is extremely small. Potentially you want to sample based on the location of the user agent, the type of user agent, or even the type of user you are serving content to.

One thing to be careful of: If you do manipulate your sampling to collect more data for specific segments, make sure you are able to exclude these segments appropriately or keep them completely separate when drawing conclusions about the entire population of your users. One great way to do this is to create a separate flag for these groups that you can use to isolate from your greater population. SpeedCurve allows for adding Customer Data using our LUX.addData API.

Having this flexibility is really great, and helps you make sure you are getting the most out of your monitoring budget.

Considerations for determining your sample size

How distributed is your traffic?

We’ve heard great accounts of customers who assume their traffic is 100% regional, only to find they have an entire contingent of users in another country! (Watch this great talk from Harry Roberts at performance.now() to hear such a tale. It’s at minute 10:45, but the entire talk is fantastic, so make sure to watch the whole thing!)

For most sites, they want to understand where their users are. If sampled too heavily, the dataset for these important cohorts of users is often too small to get relevant insight. That said, when you have to make decisions about where to spend your monitoring budget, drawing insights from your primary geographic region in isolation may be warranted.

Are you running experiments in production?

This is a rapidly growing practice that is becoming more and more important for product teams. Whether you’re pushing canary builds or running A/B test variants, you want to ensure that you have enough data in those buckets to factor performance into the decision criteria. We’ve seen sites make business decisions based on performance data that was so inconclusive that it would have been better if it had been ignored.

Are you only interested in a specific type of traffic?

Most organizations want to understand how their site is performing across multiple form factors (desktop/tablet/mobile) as well as browser types. Oftentimes an important subset of users using a technology may represent a significantly smaller user base. Given the massive fragmentation of mobile devices –namely Android – understanding performance is crucial to creating a more inclusive web. Ensuring a proper sample size for these cohorts is crucial and often hard to do if you aren’t paying attention. That said, for some teams the focus may be more specifically targeted toward a certain technology or user agent, so a smaller sample for a more popular user agent or device may work just fine.

Do you have a relatively small population of REALLY important users?

Whether you're a retail site, financial services, media or other, you likely have a smaller set of users that you are VERY interested in. Getting an adequate dataset for a retailer when the rate of conversion is really low (say 1-2%) may be really hard to do if your sample size is too small. Similarly, media sites care a lot about registered users on the other side of a metered paywall, but most likely they make up a MUCH smaller segment of the population. These examples exist across all of our sites, and it’s important that we have the ability to represent them in our datasets.

The important thing is to get started

Whether you are considering getting started with RUM or changing the approach you use today, don’t let the decision of how much to sample send you into paralysis. We always advise customers who are unsure to start small, see what you can learn and continue to expand as necessary.

I hope this is helpful and encourage you to check out some of these support articles below or reach out if you have questions. Otherwise, I highly encourage you to get started with a free trial of LUX today!

A long time coming...

Mon, 04 Nov 2019 00:00:00 +1300

I’ve joined SpeedCurve! I’m thrilled to share this news and have never been more excited about a career change than I am today. I’ve known this cast of characters for a while and am humbled that they have brought me onto the team. As Tammy put it when she joined, if this crew invited you to work with them, “what would you say?”

Tammy, Steve and Cliff at Velocity Conference circa 2015

Who am I?

As a veteran in the performance industry, I’ve spent a large part of my career helping to build performance culture. I’ve been in countless rooms and discussions defending the case for performance and helping to educate cross-functional teams about the impact of performance on the user experience and ultimately the health of the business.

My journey has taken me to both sides – as a product leader focused on building tools and solutions for customers, and as a practitioner focused on creating a culture of performance for one the world’s largest brands.

What am I most excited about?

As Chief Product Officer for SpeedCurve, I’m thrilled to continue this journey. I’ve been a longtime fan of the product and can’t wait to start contributing to its growth and adding more value for our customers!

Here are some of the areas you’ll see me getting most excited about:

RUM + Synthetic

I LOVE how the team has focused on showing real user data (LUX) with synthetic (WebPageTest). Mark and I started exploring this years ago, and I feel like we're just now scratching the surface of what’s possible. I’ll be looking at ways we can continue to combine and compare these datasets for new insights while continuing to educate practitioners on how best to leverage the unique perspective each technology brings.

Building a more inclusive web

Traditionally, we’ve spent a lot of time focusing on entitled groups as it relates to performance – fast connection speeds, devices with high-end CPUs, low-latency geographies. While it’s important to understand the characteristics of these users, I would argue that the more important segment to focus on exists in the longtail. I’ve been inspired by Tim Kadlec’s talk on this and am thrilled to start looking at ways to highlight this in our products.

Building performance culture across the organization

While you may say to yourself “nothing new here”, every day I see more and more opportunity to bring performance into the mainstream. Whether we adopt performance as a principle in our design systems, set and maintain performance budgets for product owners, integrate performance tests into our build systems, or simply create a conversation with non-technical users around perceived performance, there is always more to do in our organizations. Making performance data consumable by the masses is perhaps the thing I love most about SpeedCurve, and the thing I’m most excited to be contributing to and talking to you about...

...with one exception...

I’m most excited about keeping the focus on you!

Building products and features “outside-in” is my top priority and guiding principle. SpeedCurve is known for this today and I plan to do even more on behalf of you as we prioritize and build our roadmap. Understanding your pain, your users’ pain, and shaping products and solutions to make your job easier is what I care about most.

What can you expect from me?

While I’m diving into product, vision, and strategy, I’ll also be working closely with Tammy in her outreach to customers and prospective users to get insight as we build out our roadmap for SpeedCurve over the next year.

Keep an eye out for me on Twitter, and at events and meetups in your area. If you’d like to chat about the direction of SpeedCurve's products – or if you’d just like to catch up and learn more about us – I’d love to hear from you!

New! User Happiness metric, CI plugin, and an inspiring third-party success story

Wed, 30 Oct 2019 00:00:00 +1300

Here at SpeedCurve, the past few months have found us obsessing over how to define and measure user happiness. We've also been scrutinizing JS performance, particularly as it applies to third parties. And as always, we're constantly working to find ways to improve your experience with using our tools. See below for exciting updates on all these fronts.

As always, we love hearing from you, so please send your feedback and suggestions our way!

New metric! User Happiness Score

The goal of making websites faster is to make users happier. Since user happiness can't be measured directly, we've developed the User Happiness metric. It's a combination of multiple performance metrics that are closely tied to delivering a joyous user experience. Combining these into an aggregate metric makes User Happiness more applicable across a wider variety of websites and conditions.

Another new metric! Largest Contentful Paint

On some pages, the most important content is visual. As its name suggests, Largest Contentful Paint is a new metric that can help you know when the largest element (image or video) in the viewport is rendered. It's provided by browsers via the Largest Contentful Paint API, and it's now available in both SpeedCurve Synthetic and LUX. Here’s a good example of LCP on The New York Times home page:

More: Glossary of popular SpeedCurve metrics

Case study: Improving third-party performance at The Telegraph

Here's a great success story from our friends at The Telegraph: "By using custom performance marks in the ad code, deferring JavaScript and reducing bundle sizes, the First Ad Loaded metric improved by an average of 4 seconds." (See more great case studies.)

Custom colors in your charts

Now you can standardize your site's colors across all your SpeedCurve charts. Just go to your Settings page, click on the site, then use the color picker to select your custom color.

Copy your Favorites dashboards

Sometimes you may find yourself needing to create a new Favorite dashboard that's very similar to an existing one. Now you don't have to build it from scratch. You can copy the existing dashboard and then use the chart filters to change whatever variable you need to change. (Learn more about creating Favorite dashboards and charts.)

Get SpeedCurve status updates

If you're ever wondering about SpeedCurve's uptime or if we're experiencing technical issues, visit our status page. From there you can also subscribe to receive updates.

Is your monitoring fully set up?

We want to make sure you're getting the most possible value from your SpeedCurve account. We recently added a setup guide (visible in the left-hand navbar) to help you identify which steps you still need to complete.

Netlify build plugin for SpeedCurve

If you use Netlify and SpeedCurve, then you might be interested in a plugin that Tim Kadlec recently created to trigger a round of tests in SpeedCurve for each deploy.

It's about your JavaScript...

In October, I had the privilege of speaking at JAMstack SF. I talked about JS performance gotchas, the best metrics to explore, and how to use performance budgets to deliver and maintain the better experience that the JAMstack promises. (Video here.)

"You’re not going to have a company if you don’t have happy users."

I'm so excited about the launch of the Planet Performance podcast. In the first episode, Stoyan Stefanov interviews our very own Steve Souders. Among other things, Stoyan and Steve talk about the early days of performance and why you're not going to have a company if you don't have happy users. Bonus: You get to hear Steve play the ukulele! :)

No such thing as too many webperf podcasts

Speaking of podcasts, Tim Kadlec has already put out two episodes of Chasing Waterfalls – conversations with the people working to make the web faster for everyone. The first episode was with Malek Hakim, about creating a performance culture at Priceline. Episode 2 was with Reefath Rajali, who talks about PayPal's performance journey.

Find me at Beyond Tellerrand and PerfNow in November!

In November, I'll be attending Beyond Tellerrand in Berlin (November 13-16) and then performance.now() in Amsterdam (November 21-22), which I'm co-chairing alongside Tim Kadlec. Each of these events has a fantastic speaker lineup, and SpeedCurve is proud to be sponsoring both events. If you're there, be sure to say hi!

Getting started with web performance? Here's what you need to focus on.

Tue, 25 Jun 2019 00:00:00 +1200

A while back, our friends at Shopify published this great case study, showing how they optimized one of their newer themes from the ground up – and how they worked to keep it fast. Inspired by that post, I wanted to dig a bit deeper into a few of the best practices they mentioned, which fall loosely into these three buckets:

Analyze your pages – understand the critical rendering path and page composition.
Create performance budgets and fight regression.
Build a performance culture that embraces collaboration between design and dev.

Keep reading to learn how you can apply these best practices to your own site and give your pages a speed boost.

What's hurting the user experience?

When we think about things that can hurt the user experience, it's important to know that issues fall into two camps:

Blocking resources – Resources like CSS, JS (including third parties), custom fonts, etc., which can block the critical rendering path.
Non-blocking resources – Resources that are so numerous or so poorly optimized that they hamper the CPU and cause jankiness and a poor user experience.

Blocking resources, as their name suggests, block the critical rendering path. The critical rendering path is the set of steps browsers must take to convert HTML, CSS and JavaScript into living, breathing websites. To optimize for performance, you need to understand what happens in the steps between receiving the HTML, CSS, and JavaScript bytes and the processing that's required to turn them into rendered pixels.

It's not enough to focus on when the page first starts displaying pixels – you need to make sure you're showing content your users care about. That's why it's a good idea to not just focus on metrics like Start Render – you should also investigate Hero Rendering metrics, which track when the most meaningful content on your pages (e.g., hero images, headlines) renders.

It's also not enough to focus on blocking resources. It's possible for pages to contain zero blocking resources and still have less-than-optimal performance because of how your JavaScript is rendered. That's why it's so important to understand CPU usage on your pages – because JavaScript consumes more CPU than all other browser activities combined. While JS blocks the CPU, the browser can't respond to user input. This creates what’s commonly called “jank” – that annoying feeling of jittery, unstable page rendering.

To help focus attention on CPU usage, we’ve been evangelizing a couple of new performance metrics:

First CPU Idle measures when the page is no longer janky. Specifically, it's the first span of 5 seconds where the browser main thread is never blocked for more than 50ms after First Contentful Paint. A value of 2-4 seconds is typical.
Total Long Task CPU Time is the sum of all long tasks that occur in the page. A "long task" is a browser event that blocks the main thread for more than 50ms.

If you still need to be convinced, this correlation chart shows a distinct correlation between increased CPU Time and higher bounce rates:

(If you're a SpeedCurve LUX user, this chart is at the top of your JavaScript dashboard. If you're not a LUX user, here's how you can try it for free.)

After you understand First CPU Idle and CPU Time, there are a couple more metrics that will help you understand the relationship between the rendering path and how people interact with your pages:

First Interaction Time is the first time a user interacts with a page – by either scrolling, clicking, or using the keyboard. (At SpeedCurve we call these metrics "IX Time" and they're available on your LUX Performance dashboard.) First Interaction Time can vary widely depending on the type of site and page. For example, a search results page might have a low First Interaction Time because users scroll and click quickly. A media site, on the other hand, might have a higher First Interaction Time because users start reading content before they interact with the page.
First Input Delay measures the gap between when a user interacts (e.g., clicks or scrolls) with the page and when the browser acts on that input. A good target for FID is 10ms, though it's common to see results as high as 25ms is common. (FID is also available on your LUX Performance dashboard.)

A few things to keep in mind when you're looking at all these metrics in relation to one another:

Know how many blocking resources are on your pages. Wherever possible, load them asynchronously or even defer them.
Observe how often First Interaction Time is after First CPU Idle.
Understand the amount of CPU time that's consumed by Long Tasks.
Monitor your longest Long Tasks.
Track First Input Delay relative to interaction and rendering times.

Read more: Measuring Jank and UX

Understand page bloat

According to HTTP Archive data for the top million websites, the median page served to desktop is close to 1900 KB. The median page served to mobile isn’t much smaller, at close to 1700 KB. It’s important to remember that these numbers are medians. I routinely see pages that are between 5 and 10 MB in size. Images account for roughly half the size. JavaScript, fonts, and video carry the bulk of the rest of the weight.

It's important to note that you can have large, robust pages that still feel fast. Take Amazon, for example. It's widely considered to be a performance leader, yet it has relatively heavy pages. (I'm defining "heavy" as 3MB or more.)

But you should still keep an eye on page and resource sizes for a couple of reasons:

Consider the cost and performance impact of serving bloated pages to mobile users who are dealing with bandwidth constraints or data limits.
Sudden increases in page size can also be an indicator that you’ve got rogue unoptimized images (or other assets) that could be wreaking havoc on user experience. For example, I see this particularly with hero images on campaign landing pages, which are sometimes built and released quickly without proper due diligence paid to ensuring they’re performant.

Create performance budgets

Performance budgets are a vital tool in your web performance toolbox. They take the hassle out of monitoring your page performance by notifying you whenever your metrics cross a certain threshold.

If you've been following along so far, you may already be guessing that it's a good idea to consider creating performance budgets for the CPU- and size-related metrics we've covered. You should also consider creating performance budgets for:

Start render (synthetic and RUM): 2 seconds recommended
Speed Index (synthetic only): 4 seconds recommended
Last Painted Hero (synthetic): 5 seconds recommended

IMPORTANT: These defaults are intended to be a jumping-off point for your own analysis. You may find that some of these metrics and numbers are relevant for your site, while others may be less relevant. If you're tracking custom metrics, you probably want to create performance budgets for them.

Since Google research suggests that you should aim to be at least 20% faster than your fastest competitor, it might also be a good idea to monitor your competitors' sites so you can create budgets that will help you stay faster than the competition.

To motivate you, here are some real-world performance budget wins:

Casper used perf budgets to track the performance impact of self-hosting a third-party script.
Zillow set budgets for their quantity (e.g. image size) and milestone timing metrics in mobile search.

Read: How to create performance budgets and set alerts

Fight performance regression

Ask anyone who’s been in the performance space for a while, and they’ll tell you that making your pages faster isn’t the hardest part – it’s keeping your pages fast. Over time, so many insidious factors can cause pages to slow down:

New features
More third-party scripts
Changes to your code base
Team changes
Your company de-prioritizes performance

One of the most effective ways to fight performance regression is to integrate performance testing into your continuous delivery process. This is a natural extension of using performance budgets in an ongoing, meaningful way. In an ideal setup, the build is blocked until performance testing is complete, and it can potentially fail if the performance budgets for any tests are violated. In other words, performance regressions are caught before they make it to production. Treating performance as a first-class acceptance criteria creates a complete feedback loop for teams to manage the performance of their work.

Read: Performance testing in CI: Let's break the build!

Build a culture of web performance throughout your company

Shopify emphasizes that collaboration between designers and developers was a critical part of the success of their project:

“Collaborate with other disciplines, and contribute to levelling up their work by considering performance.”

At SpeedCurve, we talk a lot about “performance culture” and what that looks like. In simple terms, creating a strong performance culture means creating a feedback loop in your company or team that gets people to care, shows them what they can do to help, and then gives them positive reinforcement when you get results. It's surprisingly easy to skip these steps and then wonder why all your performance efforts feel like such a painful uphill slog – and why regression is inevitable.

Some performance culture best practices include:

Identify people’s goals and the performance metrics that correlate to those goals.
Connect the dots. For example, correlation charts, like the one below, show people the relationship between performance metrics and conversion rate.
Benchmark your site against your competitors.
Collaborate on performance budgets. I can’t emphasize this enough: performance budgets shouldn’t be a top-down process.
Score some quick, inspiring wins by fixing the low-hanging fruit. The first places to look are images and blocking stylesheets and scripts (especially third parties).
Share and celebrate. This can be in the form of company- or department-wide emails, posts on your in-house dev blog, and internal meetups.

Read: Performance culture tips and best practices

You can't fix what you don't measure

The first step is to understand how your pages perform and identify roadblocks. If you're already a SpeedCurve user, you can start trying out these tips right away. If you're not a SpeedCurve user, give us a try for free!

Performance testing in CI: Let's break the build!

Wed, 19 Jun 2019 00:00:00 +1200

Raise your hand if you've ever poured countless hours into making a fast website, only to have it slowly degrade over time. New features, tweaks, and Super Important Tracking Snippets all pile up and slow things down. At some point you'll be given permission to "focus on performance" and after many more hours, the website will be fast again. But a few months later, things start to slow again. The cycle repeats.

What if there was a way that you could prevent performance from degrading in the first place? Some sort of performance gateway that only allows changes to production code if they meet performance requirements? I think it's time we talked about having performance regressions break the build.

The secret to a faster website? Break the build!

If "breaking the build" is a foreign concept to you, then allow me to explain. Continuous integration ("CI") is a stepping stone on the journey to streamlined software delivery. With sufficient automated testing, you can have a high level of confidence that an isolated change can be built and integrated without breaking anything else. If any of the automated tests fail, then the build is broken and the change cannot be integrated.

This concept can be extended even further by automatically deploying changes to testing or staging environments (continuous delivery), running more automated tests, and potentially even deploying the change to a production environment with zero human input (continuous deployment).

Performance as acceptance criteria

When automated testing is talked about in the context of CI, it almost exclusively refers to functional testing - testing whether a feature functions as expected. Performance typically doesn't impact the functionality of a feature, so it is often absent from the list of acceptance criteria. However, if our goal is to prevent performance regressions then performance must become a non-functional requirement in our acceptance criteria. We want to be able to break the build and block a change from being shipped if its performance is not up to standard.

Where performance fits in a CI pipeline

So where in a CI/CD pipeline do we put performance testing? The easy answer is: right after a change is deployed to an integration or testing environment. Typically, performance testing should happen at the same time as integration and acceptance testing. The slightly more complex answer is: it depends. For performance testing to be accurate, the integration environment should replicate the production environment as closely as possible. Here's a short (and definitely not exhaustive) list of things that you may want to consider when running performance tests on an integration environment:

Caches may need to be warmed up beforehand. Production environments typically have high cache hit ratios, and this should be replicated during performance testing.
If system resources are a constraint, it may be better to run performance testing in isolation. Running it at the same time as integration testing or load testing could cause resource contention that negatively affects performance.
Network latency will affect performance testing. If your production environment is accessed via a CDN, try to run your integration environment through the same CDN.

Performance in CI without breaking the build

I know, I know. The title of this post says that to be fast you have to break the build! The reality is that many software delivery teams are not in a position to do this. Most stakeholders that I've worked with would find it unacceptable to block an important feature or bug fix due to it failing performance tests.

Thankfully, there are some other options. Let's go through them, starting with the holy grail: staying fast by breaking the build.

Break the build on performance regressions

The build is blocked until performance testing completes, and can potentially fail if any performance tests fail. This option only works in teams that have 100% buy-in from all levels, since they will need to fix performance issues before they can continue with any other work.

Pros

Treats performance as a first-class acceptance criteria.
Creates a complete feedback loop for teams to manage the performance of their work.
Performance regressions are caught before they make it to production.

Cons

Increases build time.
Requires buy-in from the whole team.

Report performance regressions but do not break the build

The build can be blocked until performance testing completes, but this is not mandatory. Any performance test failures are reported but do not fail the build. This option works well in teams with a strong performance culture who take performance regressions seriously. Performance issues won't block the build and delay important work, but they will be treated as a high priority and addressed in a timely manner.

Pros

Provides actionable feedback loop for teams to fix performance regressions.
Performance regressions can be caught before production.
Doesn't block things from being pushed to production.

Cons

Requires strong discipline to prevent performance regressions.

Trigger performance tests without any reporting

Performance tests are triggered as part of the build but do not block anything. This option is good for teams who are just starting to monitor performance, or who prefer to have regular "performance sprints" rather than fix regressions as they occur.

Pros

Can be used to retrospectively identify the build that caused a performance regression.
Doesn't require any buy-in from team members.
Can be easily transformed into other options later.

Cons

Requires strong discipline to prevent performance regressions.
Makes it easy to consider performance as a "nice to have" feature or forget about it entirely.

Using the SpeedCurve CLI to monitor performance

SpeedCurve recently released an official CLI and Node.js API that simplifies running performance tests as part of a CI/CD pipeline. I'll put some example commands below to give you an idea of how it can be used.

Option 1: Break the build on performance regressions.
Runs performance tests and waits until they finish. Upon completion, checks the
status of any performance budgets and exits with a non-zero exit code if any
budgets were broken.
⇒ speedcurve deploy --check-budgets

Option 2: Report performance regressions but do not break the build.
Runs performance tests and waits until they finish. Then uses the "budgets"
command to output the status of all performance budgets as JSON.
⇒ speedcurve deploy --wait
⇒ speedcurve budgets --json

Option 3: Trigger performance tests without any reporting.
Runs performance tests but does not wait for them to finish. The --note flag
annotates the deploy in SpeedCurve so that this build can be identified when
reviewing performance regressions at a later date.
⇒ speedcurve deploy --note v2.48.0-b28bc40

If you'd like to try out the SpeedCurve CLI for yourself, head on over to the GitHub project for installation instructions and usage examples. If you don't already have a SpeedCurve account, you can start a free trial and begin testing your pages within minutes.

Third party blame game

Wed, 15 May 2019 00:00:00 +1200

Our third party metrics and dashboard have had an exciting revamp. With new metrics like blocking CPU, you can now see exactly who is really to blame for a crappy user experience. We've also given you the ability to monitor individual third parties over time and create performance budgets for them.

It's not you, it's me

Or is it really you, and not me? We now automatically group all the requests in our third party waterfall chart, letting you easily identify all the third party services used on your website.

For each third party, you get the number of requests and size for each content type. There's also a first party comparison you can toggle on/off to see what proportion of your requests come from first party vs third party.

Blocking JS, CSS & CPU

People love to hate on third parties and blame all their user experience issues on them.

Look at all those requests, it must be them!

We don't believe that just looking at the number of requests or the size of requests a third party makes is a good enough indicator that they are a bad actor. A well-built web page that manages its requests can easily negate any issues by delaying those request till after the page is rendered and interactive.

What really matters is how a third party might be blocking the rendering or the interactivity of the page.

So for each third party, we show you if it includes any blocking JS or CSS requests. This tells you if the rendering of the page is being delayed by a blocking request from a third party. We also look for any blocking CPU caused by a third party. Blocking CPU is any JS function that takes longer than 50ms (also known as a long task), which often leads to page jank. Blocking CPU stops your users from smoothly interacting with your page and delays other metrics like First CPU Idle and Time To Interactive. These third party blocking metrics are a far better indicator of who might be causing real issues on your page.

In this example, Vidible has 594ms of blocking CPU, causing jank on the page, while all the other third parties don't trigger any signifiant CPU activity at all. This helps you quickly focus in on which third parties are causing real user experience problems.

Collect the evidence!

For any third party in your waterfall, you can turn on "Track History" and we'll collect detailed third party metrics every time we run a test. You can then monitor a third party over time and set performance budgets and alerts. This is a great way to keep an eye on any changes a third party might be making and alerting you to impacts on your page. You could even agree to an SLA with a third party and then use SpeedCurve to monitor and enforce it, keeping everyone honest and on the same page.

Track any request

Even though it's called third party, you can actually build your own custom filters and track any group of requests you like. You can even track the history and set performance budget and get alerts for a single request. If your task is to trim down a specific JS request and make sure its size or CPU usage is within budget, then you can setup a specific filter and monitor your improvements.

Contribute

We used to have our own in-house library for identifying third parties but we've now switched to third-party-web by Patrick Hulce, which has a quickly growing list of third parties. Many people in the perf community, such as Simon Hearne of requestmap fame, have been identifying and contributing third parties to third-party-web – and we've recently added a bunch, too! If you see any "Unrecognized Third Party" requests in your waterfall charts, then you can contribute them to the project.

Getting started

If you already use SpeedCurve, then you can start tracking your third parties straight away in your Settings or on the Synthetic > Third Party dashboard. Otherwise, kick off a free trial to see if it's a third party that's eating up your bandwidth and CPU, or your own requests.

Measuring Jank and UX

Wed, 01 May 2019 00:00:00 +1200

Ten years ago the network was the biggest problem when it came to making websites fast. Today, CPU is the main concern. This happened because networks got faster while JavaScript moved in the other direction growing 3x in size in the last six years. This growth is important because JavaScript consumes more CPU than all other browser activities combined. While JavaScript and other activities block the CPU, the browser can't respond to user input creating the sensation of a slow, jittery, or broken page, AKA "jank".

To help focus our attention on CPU, several new performance metrics have been defined and evangelized over the last year or three. In this post I'm going to focus on these:

First CPU Idle measures when the page is no longer janky. Specifically, it is the first span of 5 seconds where the browser main thread is never blocked for more than 50ms after First Contentful Paint. A value of 2-4 seconds is typical.
First Input Delay measures the gap between when a user interacts with the page (e.g, clicks or scrolls) and when the browser is able to act on that input. First Input Delay values are much lower - a good target is 10ms, but 25ms is common.
First Interaction Time is when the first user input takes place. This varies widely depending on the type of site and page. A good search results page might have a low First Interaction Time because users scroll and click quickly. A media site might have a high First Interaction Time because users start reading content (headlines, stories) before interacting with the page. At SpeedCurve we call this "IX Time".
Total Long Task CPU Time is the sum of all long tasks that occur in the page. A "long task" is a browser event that blocks the main thread for more than 50ms.

Here's a figure to help visualize these metrics.

The figure above is a (slightly modified) CPU timeline from WebPageTest. Long Tasks (where the CPU is blocked for more than 50ms) are shown in red. In the timeline above there are two long tasks. Green indicates where there are no Long Tasks. First CPU Idle happens at 3.0 seconds - this is the first point in the timeline where there are 5 seconds without any Long Tasks (from 3s to 8s). In this timeline the user first interacted with the page around 5.5s - this is the First Interaction Time (or IX Time). Since there were no Long Tasks, the browser was able to respond to the user's input in only 8ms, which gives us the First Input Delay (shown in blue).

As illustrated by this example, looking at the relationship between these metrics is the key to determining whether jankiness is affecting your users, and why.

In the timeline above, First Interaction Time is after First CPU Idle, so it makes sense that First Input Delay is very fast - there's nothing blocking the CPU. Seems obvious, but typically when I talk with performance folks about First Input Delay the discussion focuses on how First Input Delay is thwarted by Long Tasks, but if First Interaction Time is greater than First CPU Idle that's just not the case. So for site owners to appreciate what First Input Delay is telling them, it's important to know:

How often is the first interaction after First CPU Idle?

For our RUM customers, First Interaction Time is after First CPU Idle about 66% of the time. That's important to understand and keep in mind. It means that most of the time First Input Delay is not affected by Long Task CPU activity.

About a third of the time the first interaction takes place before First CPU Idle. This scenario is illustrated in the figure below. In this example, the first interaction is unlucky enough to happen during a Long Task, which means the user has to wait until that Long Task is completed before their interaction can be handled. This is when we see First Input Delay values jump to hundreds of milliseconds.

It's not always the case that when first interaction happens before First CPU Idle it will overlap with a Long Task. In the timeline above it's much more likely that the user input will not collide with a Long Task, which would result in a fast First Input Delay value. In the example timeline below, however, there are many (short) Long Tasks, so it's likely that First Input Delay values will be higher leading to more jankiness. It's important to understand the amount of CPU time being consumed by Long Tasks.

The timeline above has many Long Tasks resulting in higher First Input Delay values. However, the Long Tasks are fairly short, so First Input Delay values never exceed a few hundred milliseconds. The timeline below is a different story. It only has two Long Tasks, but one of them is almost 1.5 seconds long. (I've seen this in the real world!) If the first user input overlaps with this Long Task, we start seeing First Input Delay values over 1000ms. It's important to understand how long your Long Tasks are.

One last thing to keep in mind about First Input Delay is that it doesn't take rendering into consideration. This is a big blind spot. Looking at the timeline at the very top, perhaps the reason the user waited until 5.5 seconds to interact with the page is because the critical content took a long time to render. In which case, if rendering times were improved, First Input Delay would get worse even though the number of Long Tasks and CPU consumed remained the same. That's because improving rendering would get the user interacting sooner causing them to overlap with the Long Tasks that previously were out of harm's way. It's important to track First Input Delay relative to interaction and rendering times.

This last point is why being vigilant about web performance is difficult. It's not possible to focus on one metric. Focusing on the user experience is the key. And when it comes to user experience, CPU and rendering are the most important metrics to watch. Fortunately, (here comes the pitch) SpeedCurve's RUM product, LUX, includes First CPU Idle, First Interaction Time, First Input Delay, Total CPU time, number of Long Tasks, median Long Task, and longest Long Task, as well as many rendering metrics. If you're not already using LUX, signup for free trial.

And keep these important takeaways in mind:

Know how often First Interaction Time is after First CPU Idle.
Understand the amount of CPU time being consumed by Long Tasks.
Monitor your longest Long Tasks.
Track First Input Delay relative to interaction and rendering times.

New LUX JavaScript Dashboards

Wed, 03 Apr 2019 00:00:00 +1300

As organizations work to improve performance for users around the world on slower networks and devices, the focus on JavaScript continues to grow. LUX's new JavaScript dashboards help to identify the problems and solutions for creating a fast, joyous user experience.

LUX is SpeedCurve's real user monitoring product. We launched it two years ago with four dashboards: Live, Users, Performance, and Design. Today we've added two more LUX dashboards: JavaScript and JS Errors. These new dashboards let you see the impact JavaScript has on your site and on your users with new metrics, including First CPU Idle and First Input Delay, and new features, such as correlation charts that show you how CPU time correlates with bounce rate.

LUX JavaScript Dashboard

Ten years ago, the network was the primary bottleneck slowing down web pages. Today it's the browser CPU, especially on mobile devices, and JavaScript consumes more CPU than all other browser activities combined. That's why LUX focuses on the impact JavaScript has on the CPU and on your users.

LUX measures CPU using the Long Tasks API. A "long task" is defined as a browser event that takes more than 50ms to execute. LUX shows the total of all long tasks, the longest long task, and number of long tasks in the page. In this example, an improvement made on March 22nd shaved off more than one second of CPU time:

Long tasks make pages feel janky. Two metrics that capture jankiness are shown in the LUX chart below:

First Input Delay (FID) measures the gap between when a user interacts with the page (e.g, clicks or scrolls) and when the browser is able to act on that interaction.
First CPU Idle measures how long it takes before the page is no longer janky.

The LUX JavaScript dashboard uses heatmaps to help teams identify what they should work on next. The heatmap below shows how well different page types are performing against a set of JavaScript metrics. Green is good, yellow is okay, and red is poor. The first three page types have mixed results, but the fourth page is doing poorly across the board. Since it accounts for a large amount of traffic (5%), focusing on this page has the greatest chance for improvements that impact a large number of users.

In addition to showing heatmaps for page types, the LUX JavaScript dashboard also shows them for device types and browsers. The device type heatmap below indicates that tablets need the most attention.

LUX JS Errors Dashboard

With the increased focus on and activity around improving JavaScript, it's important to track errors to make sure JavaScript enhancements don't break functionality. The LUX JS Errors dashboards helps you monitor and isolate errors.

In addition to showing the total number of errors, the chart below shows the error rate. This is helpful if your traffic has regular peaks and valleys (e.g., weekends) where the number of errors might fluctuate but error rate should remain fairly flat. In the example, a JavaScript error on March 27th caused both metrics to spike!

The list of Top Errors helps you identify the errors that are most affecting customers. The LUX Errors dashboard also shows the distribution of errors across top browsers, regions, and pages. You can click on any error message to drill down on it.

Note that "Script error" is typically from cross origin scripts where error details are not available. This can happen for third party scripts but also for scripts you own that are served from a different domain, such as a CDN. To see the error details you must include the crossorigin SCRIPT attribute and the Access-Control-Allow-Origin response header.

The details for individual errors makes it easy to find and fix problems. The name of the offending script is given, or "HTML document" if the error occurred in an inline script. The line and column number help track down exactly where the error occurs.

Get started

If you're already a LUX user, these JS dashboards are available in your LUX menu. If you're not using LUX, give it a try for free.

Chart sizes and TV Mode

Wed, 06 Mar 2019 00:00:00 +1300

SpeedCurve now has different chart sizes and a special TV Mode to help you build a performance culture in your organisation.

From its inception, SpeedCurve has always been designed to look awesome on the big screen. We see SpeedCurve as not just a tool for debugging web performance, but as a communication tool to rally your organisation around the importance of web performance. SpeedCurve helps bring together the development, design, and management teams, and gets everyone focused on turning your product into a fast and joyous experience for your users.

Chart sizes

In the dropdown at the top right of each dashboard you can now switch between three different chart sizes. The amount of information shown is tuned based on the chart size. Small charts providing a quick overview of trends in a dashboard, while larger sizes give you a lot more detail and metrics.

TV Mode

In the same dropdown you can also toggle TV Mode, which turns off the SpeedCurve navigation and lets you focus on just the charts themselves. This is ideal for putting a set of your favourite charts on a monitor in your office. (Pro tip: Many of our enterprise customers do this.) This is a great strategy that can help make your performance budgets visible and build a performance culture in your organisation.

Questions? Feedback?

As always, we welcome your input. We'd also love to learn how your organisation shares performance dashboards and the impact this had had on your business and culture. Let us know in the comments or at support@speedcurve.com.

And if you're not using SpeedCurve yet, give us a try for free!

JavaScript dominates browser CPU

Tue, 05 Feb 2019 00:00:00 +1300

Loading scripts asynchronously is critical for getting pages to render more quickly. We care about rendering because that's what users see; if rendering is slow users have a negative experience. But it's not just about what users see - how the site feels is also important. That's why we focus so much on CPU time. If the CPU is blocked, then browsers are delayed responding to user interactions like scrolling and clicking on links. In other words, the page feels janky. And what consumes the most CPU in browsers? You guessed it: JavaScript!

The pie chart above comes from a study based on data from the HTTP Archive. It shows browser CPU time broken down into buckets. I use colors to show the high-level categories: orange for JavaScript, purple for Layout, green for Paint, and blue for Loading. These are the median values across 1.3 million sites on desktop. Here's a similar chart for mobile:

The absolute values are higher for mobile, but the key is that in both charts JavaScript consumes the most browser CPU. In addition to median, I looked at the 95th and 99th percentiles for both desktop and mobile, and resized the pie charts based on total time. Here they are overlaid on top of each other:

In addition to being beautiful, this graphic shows an important trend: as browser CPU time grows the percentage used by JavaScript also increases. Ten years ago the network was the main bottleneck. Today, the main bottleneck is JavaScript. The amount of JavaScript on pages is growing rapidly (nearly 5x in the last 7 years). In order to keep pages rendering and feeling fast, we need to focus on JavaScript CPU time to reduce blocking the browser main thread. If you're not tracking your site's JavaScript CPU usage, start doing that now. Even better, setup a performance budget to alert you if (when?) it becomes a problem.

New LUX metrics

Wed, 23 Jan 2019 00:00:00 +1300

Over the winter holiday we added a bunch of new metrics to LUX:

First Contentful Paint
First CPU Idle
Longest Long Task
Number of Long Tasks
Connection type
HTML transfer size
Total # of image requests

We think it's important to focus on the user experience which translates into when content is rendered to the page. We added First Contentful Paint to provide more visibility into rendering. First Contentful Paint and Start Render (which was already available in LUX) are provided by browsers as part of the Paint Timing spec. It's the time at which users can start consuming page content. Specifically, it is "the time when the browser first rendered any text, image (including background images), non-white canvas or SVG." This can be found in the LUX Performance dashboard.

When it comes to creating a fast, joyous user experience, it's important to understand when the browser is blocked from rendering and accepting user input because the CPU is pegged. The Long Tasks API allows us to measure when JavaScript blocks the browser main thread for more than 50 milliseconds. In addition to viewing the total time of all Long Tasks, you can now see the longest Long Task and the median Long Task. You can see all these Long Task and CPU metrics in the LUX Performance dashboard.

The Chrome team aggregates the Long Task data into a metric called First CPU Idle. It's the first span of 5 seconds where the browser main thread is never blocked for more than 50ms after First Contentful Paint. When your site reaches First CPU Idle it's an indication that rendering isn't blocked and the user can interact without experiencing jank, so it's important to keep this metric as low as possible. You can also find this metric in the LUX Performance dashboard.

Segmenting real users by connection type lets you see what type of bandwidth your users have and how that correlates to performance. LUX gets this data from the Network Information API. LUX typically reports the effective connection type, which is calculated based on roundtrip network times and recently observed download times. A breakdown of your users based on connection type is available in the LUX Users dashboard, alongside existing regional, browser, and page breakdowns.

Finally, we had a customer request the transfer size of the main HTML document as a new metric, so we added that to the Favorites chart editor. We already reported the number of images above-the-fold, but added the total number of images. Viewing these metrics in the LUX Design dashboard allows you to see if lazy loading images below-the fold would help improve performance and reduce data costs.

If you're already a LUX customer, take a look at these new metrics. If you're not a LUX customer, signup now to start a free trial.

Preload scripts

Tue, 22 Jan 2019 00:00:00 +1300

In my previous post I talked about how loading scripts asynchronously reduces the impact of JavaScript resulting in a (much) faster user experience. But even when scripts are loaded async, the browser may still twiddle its thumbs for a second or more waiting for the first script to arrive. This delay can be decreased by using link rel=preload like this:

<link rel="preload" href="main.js" as="script">

Link rel=preload is useful for downloading any important resource more quickly, such as stylesheets that contain critical CSS, fonts that are used in important design elements, and hero images. It's especially important for scripts because they block page content from rendering and consume the most CPU during page load. (We'll explore this topic in the next post in this series.)

Including the as attribute is critical. The browser uses this value when determining the download priority. Possible values included script, style, font, image, video, audio, and more. Only use preload for truly critical content, otherwise less important content will consume bandwidth and CPU that would have been better allocated to what matters most. In the case of scripts, link rel=preload is best for the first few synchronous scripts in the page.

Link rel=preload has been around for awhile, but is lesser known in the dev community. Nevertheless, Chrome reports that 25% of web pages visited using Chrome have this feature.

While true, this might overstate the adoption of link rel=preload where it matters most. Instead of including all uses of link rel=preload, I did an analysis based on HTTP Archive data to see how many sites use link rel=preload to download scripts for their page. That number turned out to be closer to 1% (12,469 pages out of 1,255,904). This visualization helps make my point:

The browser has a built-in Preloader which helps get scripts downloading sooner. Using link rel=preload in the markup might accelerate that even more, depending on the structure of your HTML document. It's likely to have a bigger impact, however, if you serve it as an HTTP response header, like this:

Link: <main.js>; rel="preload"; as="script"

This HTTP response header is received by the browser before the HTML, so could initiate the script request even sooner. It's best to do both the HTTP response header and the markup as some browsers don't yet support link rel=preload response headers.

We know that synchronous scripts block rendering, which makes the user experience feel slow. And we know that most scripts today are downloaded synchronously (rather than async). And yet only 1% of sites are using link rel=preload to download their scripts. If your site has any synchronous scripts, do an A/B test adding link rel=preload for them. It's likely this will be a win and help you create a more joyous experience for your users.

Load scripts async

Thu, 20 Dec 2018 00:00:00 +1300

This blog post has a simple conclusion: Load script asynchronously! Simple, and yet the reality is that most scripts are still loaded synchronously. Understanding the importance of loading scripts asynchronously might help increase adoption of this critical performance improvement, so we're going to walk through the evolution of async script loading starting way back in 2007. Here's what loading 14 scripts looked like in Internet Explorer 7:

I've added a red arrow at the beginning of each script request. What we see is that the HTML parser stops when it encounters a SCRIPT tag until that script is downloaded, parsed, and executed. Because the HTML parser stopped, all network traffic also stopped (because no other HTML tags were parsed to initiate HTTP requests). It wasn't just IE7 that did this - all browsers behaved this way back then.

Enter the Preloader

While it's necessary that scripts get executed in the order specified, there's no reason they can't be downloaded in parallel so that the execution daisy chain moves more quickly. To achieve this, browsers created a lightweight parser called the preloader. The preloader (AKA, speculative or lookahead parser) skims over the HTML looking for tags that might initiate HTTP requests, like SCRIPT, LINK, and IMG. It prioritizes and initiates those requests so that the responses are more likely to be ready when the HTML parser actually creates the corresponding DOM elements.

Here's a site with 15 synchronous scripts (also primarily from turner.com). Instead of taking 6 seconds like it did in IE7 with one-at-a-time script downloading, the preloader helps get these scripts downloaded in ~1.2 seconds. (Notes about this waterfall chart: The rendering filmstrip is shown at the top. The browser CPU timeline is shown below the waterfall. Scripts are shown in orange. Synchronous scripts are shown with a red hash pattern. The red and green bars to the right of each script are when the script uses the CPU. Total CPU time for each script is shown in parentheses after the URL.)

The preloader is the most important performance improvement browsers ever made. IE8 was the first browser to have a preloader in 2009. Most of the other major browsers followed suit within the year.

While the preloader dramatically improved web performance, it wasn't a panacea. Looking at the previous waterfall, we see that:

Rendering is blocked until all the synchronous scripts are done downloading, parsing, and executing. If they were downloaded asynchronously, the browser could start rendering sooner.
The second script, jquery.js, takes a long time to download. Because it's loaded synchronously, all the subsequent scripts have to wait to be executed, even though they finish downloading before jquery.js. This results in the browser CPU being idle for the first 1.2 seconds and then being pegged when all the JS and layout happens. If the scripts were downloaded asynchronously, they could be parsed and executing immediately when they arrive spreading out the CPU activity which would avoid blocking the CPU.
The preloader gives synchronous scripts a higher download priority than images. Since the synchronous scripts take so long to download, the image requests are blocked and the initial rendering only includes text. If the scripts were loaded asynchronously, images would get downloaded sooner providing a better user experience.

Finally: Async

Looking at the previous waterfall, we see that the preloader dramatically improved performance and rendering. However, it didn't solve all the problems. That page has 15 synchronous scripts, which block rendering, peg the CPU, and delay images. The solution to these problems is to load scripts asynchronously.

The best way to load scripts asynchronously is to use the async attribute. Browsers added support for this attribute starting with Firefox and Chrome in 2010, followed by Safari and Internet Explorer in 2012. Before these were widely supported, developers could load scripts asynchronously using a variety of techniques (i.e., hacks). These techniques are still used today for special situations, but it's simpler to use async in markup. It's also better for performance, as the preloader can spot these scripts and get the HTTP requests started sooner than if they are loaded programmatically.

The benefits of loading scripts asynchronously are highlighted in the following waterfall chart for nytimes.com. Notice that none of the scripts have the red hash pattern that marks a synchronous script, which means all of these 20+ scripts are loaded asynchronously. As a result, the page renders quickly (0.7 seconds) and the script parsing and execution are not blocked - each script is handled as soon as it arrives.

Async Adoption

It's great that the async attribute is available and well known, but synchronous script loading is still the default. In other words, website owners need to make an extra effort to get their scripts to load asynchronously. It might not be a lot of work (just add the async attribute) but even small changes can take a long time to reach critical mass on the Web.

The chart below shows the adoption of asynchronous script loading. These are median stats from the world's top ~500K websites based on an analysis from the HTTP Archive. In late 2015, the median was 10 synchronous scripts and 2 asynchronous scripts. The number of sync dropped and async rose in June 2016. This is likely when some dominant third party snippets (Google Analytics, Facebook, etc.) switched from sync to async. The total number of scripts has continued to climb since then, but all the new scripts use async, which is a great sign. Nevertheless, the most recent data shows that synchronous scripts outnumber asynchronous scripts 7 to 6.

This growth in the number of scripts loaded asynchronously is impressive. Three years is a long time, but we're talking about half-a-million websites. While the change to make synchronous scripts be asynchronous is trivial (just add the async attribute), making sure that change doesn't produce errors can be challenging. If JavaScript is parsed and executed out-of-order, undefined symbol errors are likely. Untangling the interdependencies to ensure safe async script loading may take time, but the performance benefits are worth it, for you and your users.

Trend metrics and compare time periods

Tue, 18 Dec 2018 00:00:00 +1300

The Internet really is a complicated series of tubes. As a result, any time-based metrics we capture can have variations as those tubes wobble a bit as we shove data down them. To help reduce that variation, when we do synthetic tests, we always load a page at least three times and take the median result. But even then you'll find that, over time, your charts will still show plenty of variation.

All that variation can make it difficult to see if your metrics are getting better or worse over time. We recently released a couple of new features in your Synthetic and LUX charts that make it easier for you to visualize trends and compare discrete time periods within your historical data.

Highlight trends in your metrics

To make it easier for you to see which direction your metrics are heading, we've added an option to all your charts to show a trend line which helps you visualize how a particular metric is changing over the timespan of the chart. You can hover over the legend to highlight a trend line or hover over any point on the trend to see the estimated value at that point.

Compare time periods

There's really only one question that always needs answering when discussing web performance:

Are we getting faster or slower?

Our new 'compare' mode helps answer that question and shows just how much things have improved or regressed over the most recent time period compared to the previous one. Things to try:

Pick a day of real user traffic before and after the release of a new version of your site.
Track your month-to-month progress on a project, such as refactoring your Javascript.
Keep a watchful eye on third-party metrics. Review and compare them each month to make sure you're not regressing.

I like to think of this option as boss mode. Take these charts to show off to your boss all the progress you're making on that new performance project. Or scare the jeepers out of your boss by showing him or her performance regressions... and unlock more budget for new performance optimization projects.

To turn on compare mode, go to your dashboards and select the length of time you want to compare (7 days, 1 month, etc.). You can also select a custom start/end date. Then turn on "Compare Previous Period".

Get started

If you're already a SpeedCurve user, trend lines and compare mode are available in your charts. We'd love to hear how you've used them, and how they've been helpful!

If you're not a SpeedCurve user, why not give us a try for free?

JavaScript growth and third parties

Tue, 11 Dec 2018 00:00:00 +1300

JavaScript is the main cause for making websites slow. Ten years ago it was network bottlenecks, but the growth of JavaScript has outpaced network and CPU improvements on today's devices. In the chart below, based on an analysis from the HTTP Archive, we see the number of requests has increased for both first and third party JavaScript since 2011.

The following chart shows the growth in the total size of JavaScript from 2011.

Certainly the amount of JavaScript has increased. Total number of requests went from 9 to 18, and total size increased from 85 KB to 364 KB. (These are medians from the world's top ~1.3 million sites tested on desktop.) It's interesting to see that the size of scripts is growing more rapidly than the number of scripts - we're packing twice as much code into each script compared to 7 years ago.

If we look a little further there are some concerning trends when it comes to third parties.

In terms of the number of JavaScript requests, first party has gone up 50% from 4 to 6 requests, whereas third party has increased 140% from 5 to 12 requests.
Third party growth in terms of JavaScript size is more alarming. First party JavaScript doubled from 53 KB to 106 KB. Third party JavaScript octupled(!) from 32 KB to 258 KB.
Just looking at the amount of JavaScript being served today, third parties are responsible for twice as many requests (12 vs 6) and about two-and-a-half times as many kilobytes (258 KB vs 106 KB).

In the early days back in 2010, the amount of first and third party JavaScript was about the same, whereas now third party JavaScript is more than twice that of first party. A possible explanation for this is the greater availability of and reliance on third party JavaScript across the Web.

Third party JavaScript is a significant part of today’s websites and should be watched closely. If you don’t have performance budgets setup to watch your site’s use of third party JavaScript, you should do that now.

Metrics from 1M sites

Tue, 04 Dec 2018 00:00:00 +1300

The number of performance metrics is large and increases every year. It's important to understand what the different metrics represent and pick metrics that are important for your site. Our Evaluating rendering metrics post was a popular (and fun) way to compare and choose rendering metrics. Recently I created this timeline of performance metric medians from the HTTP Archive for the world's top ~1.3 million sites:

Here's the same chart for mobile:

People who have worked on web performance won't be surprised by how much slower mobile is. As much as we hear about improvements to mobile networks and CPUs, they're not on par with desktop. And yet, many sites deliver the same amount of content (especially JavaScript) to mobile devices.

What catches my eye are the gaps between TTFB and the paint metrics, and between the paint metrics and First CPU Idle. These gaps are caused by JavaScript dominating the browser main thread. This happens after TTFB when all the blocking scripts are executed – these have to finish before any rendering can happen. The gap between the paint metrics and First CPU Idle is caused by subsequent scripts being executed and JavaScript frameworks building the DOM dynamically.

If you're looking for guidelines on where your metrics should be, these timelines provide a useful guide. Stay tuned for more posts that explore how to make JavaScript faster so it's no longer the longest pole in the tent (or at least shorten the pole).

Related posts:

Deeper performance analysis with histograms and correlations

Wed, 28 Nov 2018 00:00:00 +1300

This week we've made some pretty exciting new changes to your Favorites dashboards. Aside from a brand-new chart editor interface, you'll also notice that we've introduced two new chart types: histograms and correlations.

In this post, I'm going to talk through some of the features in our new chart editor. I'll also explain in detail explain why I think histograms are such an important tool in your performance toolkit, and how you can get some fascinating insights by correlating other metrics on top of a histogram.

Histograms are useful for painting a more detailed picture of your performance, because they show you the distribution of a metric's value.

But what exactly is a distribution? In the chart above, the numbers along the X-axis show the start render time (in seconds), and the height of the bars shows how many page views had that start render time. This shows us that mobile users (blue) pretty consistently have a start render time of between 0.8 and 1.4 seconds.

Desktop users (yellow) on the other hand have an extremely varied experience, with start render times anywhere from 0.4 to 2.0 seconds and above.

This illustrates one of the ways histograms are so helpful: medians by themselves simply don't convey this depth of information.

To make your charts even more insightful, we also give you the option to correlate one or more metrics over top of a histogram.

In the chart below we plot the bounce rate for a website's home page against a distribution of the page load time. What it shows is that as the page load increases, so does the bounce rate. Faster pages mean fewer bounces!

Correlations are also useful for narrowing down the cause of slow page load times.

For example, in the next chart we can see that a higher page load time correlates to more JavaScript and CSS being loaded in the page. This might indicate that reducing the amount of JavaScript and CSS would have an impact on page load times.

These are just a few examples of the kinds of things you can learn from creating histograms and correlation charts. We're really looking forward to seeing what sort of insights you can gain from these new charts!

New chart editor

As well as the new chart types, we've also developed a brand-new chart editor. What we like the most about this editor is the live preview, which makes it super easy to visualize changes to your charts as you make them.

If you're a LUX customer, you'll also notice that you can now add multiple customer data filters. This allows you to dive much deeper into your real user data.

Getting started

These improvements are already available for all SpeedCurve customers. Open up one of your Favorites dashboards and create or edit a chart to see the new features in action.

Need a refresher? No problem! We have a short tutorial that walks you through the process of creating custom dashboards and charts.

If you’re not a SpeedCurve user: Sign up for your free trial and see what sort of performance insights you can gain!

An analysis of Chromium's paint timing metrics

Wed, 26 Sep 2018 00:00:00 +1200

Here at SpeedCurve, we are continually gathering detailed performance data from tens of thousands of web pages. This gives us a relatively unique opportunity to analyse and aggregate performance metrics to gain some interesting insights. In this post, I'm going to analyse some browser-based paint timing metrics: First Paint & First Contentful Paint (defined in the Paint Timing spec and implemented in Chromium). I'm also going to analyse First Meaningful Paint (defined in a draft spec and implemented as a Chromium trace metric).

What are paint timing metrics?

The aim of almost any performance optimisation on the web is to improve the user experience. The folk at Google have been pushing this sentiment with a focus on user-centric performance metrics, which aim to answer four questions about users’ experiences:

Is it happening? Is the page that I want to see actually loading?
Is it useful? Has enough content rendered that I can engage with it?
Is it usable? Can I scroll and interact with the page, or is it still loading?
Is it delightful? Are my interactions with the page smooth?

First Paint (FP) measures the point at which pixels are first rendered to the screen after navigating to a new page. First Contentful Paint (FCP) is slightly more specific, in that it measures the point at which text or graphics are first rendered to the screen. Both of these metrics are available in Chromium browsers (Chrome, Opera, Samsung Internet, etc) via the Performance API: performance.getEntriesByType('paint').

The paint timing metrics are important because they aim to answer the first question: is it happening? My analysis will look at performance data from some popular websites in an attempt to figure out whether the paint timing metrics really do answer that question.

What makes a metric useful?

If paint timing metrics accurately reflect users’ experiences, then we can use that information in real user monitoring (RUM) to gain insights into how web pages perform in the real world. Since these metrics are supposed to answer the question is it happening? then doesn’t that make them useful by default? Well, maybe not. There’s an important caveat that is mentioned in the Paint Timing spec:

The rendering pipeline is very complex, and the timestamp should be the latest timestamp the browser is able to note in this pipeline (best effort). Typically the time at which the frame is submitted to the OS for display is recommended for this API.

Even once the operating system gets hold of a frame, there can be a noticeable delay before that frame is actually rendered to the screen. So when we use a browser’s paint metrics, we have to deal with the possibility that they do not accurately represent when a user actually sees anything on their screen.

It just so happens that we can test the accuracy of the paint timing metrics by comparing them to the Start Render metric in WebPageTest. WebPageTest loads a web page in a real web browser and captures a video of the screen as the page loads. It calculates the Start Render time by performing a frame-by-frame comparison to find the point at which the first pixels are rendered to the screen. This means that Start Render represents when a user would actually see the pixels, accurate to within 16.6667ms (the frame duration for a video captured at 60 fps).

Before I continue, it's important to note that while First Paint is analogous to Start Render, First Contentful Paint by definition is not. However, I’ve chosen to include FCP in this analysis anyway, because I think the data is quite interesting.

Doing the analysis

Under the hood, SpeedCurve uses WebPageTest to run performance tests, which means we have access to plenty of data that will enable us to compare the browser's paint metrics to WebPageTest's Start Render. I collated data from one of our test accounts that routinely runs performance tests on 40 of the Alexa top sites, and performed some basic statistical analysis on it. This data, along with the code that I used to perform the analysis are both available on GitHub.

The main question I have about the paint metrics is how much do they differ from Start Render? To answer this, I calculated how different First Paint and First Contentful Paint are from Start Render in every test result (FP − SR and FCP − SR) Then I plotted this data as a histogram. Let’s take a look.

In case you're not familiar with histograms, here's how you can interpret this chart:

The X-Axis is how many milliseconds different each metric is to Start Render. A negative value means that the metric occurred before Start Render. Each bar on the X-Axis is a 20ms bucket.
The Y-Axis is the number of tests whose metric falls into each bucket. For example there are 372 tests for which First Contentful Paint happened between -40ms and -21ms before Start Render.

The first thing I notice about this histogram is how it shows that more often than not, the paint timing metrics occur before Start Render. To put this another way: more often than not, FP and FCP are recorded before anything is rendered to the screen. That's an interesting discovery, although it's not entirely unexpected given the "complex rendering pipeline" caveat.

We can extract some human-friendly conclusions from this data by calculating some percentiles:

95% of FP and 85% of FCP events occur before Start Render.
50% of FP and FCP events occur within 90ms of Start Render.
80% of FP and FCP events occur within 200ms of Start Render.

Since this data comes from pages with varying load times, it might be more useful to view the deltas as a percentage of Start Render time:

The shape of the histogram hasn't changed too much, but this time we have a slightly more useful scale and we can see that in most cases the paint timing metrics occur within 10% of Start Render. Some human-friendly conclusions from this data:

50% of FP and FCP events occur within 4% of Start Render.
80% of FP and FCP events occur within 10% of Start Render.

Put in this context, the difference between the paint timing metrics and Start Render seems much more reasonable.

What to make of these numbers?

If we can accept that 10% is a reasonable margin of error, then this data tells us that 80% of FP and FCP times are a reasonably accurate representation of when the first pixels are rendered to the user's screen.

What about the remaining 20% of page loads?

The FP values that fall outside of our 10% margin of error are all before the Start Render time, which means that they represent a point in time when the screen is still blank. The outlying FCP values, on the other hand, are mostly after the Start Render time. I would guess this is where the contentful part of First Contentful Paint is having a positive effect. While these outliers don’t represent the first render, they at least represent a point in time where something has been rendered. It feels like this difference makes FCP a slightly more useful metric than FP, at least for representing when users see the first render on their screens.

What about First Meaningful Paint?

I said that I would also analyse First Meaningful Paint. Just like First Contentful Paint, FMP is intended to be a fundamentally different metric to Start Render. The "meaningful" part of First Meaningful Paint comes from a set of heuristics like the number of layout objects, page height, and web fonts. You would therefore expect FMP to occur much later than Start Render. Let's take a look at a histogram of the difference between Start Render and FP, FCP, and FMP.

Somewhat surprisingly, FMP suffers from the same problem as the other two metrics in that it regularly occurs before Start Render. Thanks to its more advanced heuristics, FMP is much more likely to represent a point in time when there are actually pixels on the screen. There is one interesting human-readable conclusion that comes out of this data:

50% of FMP events occur before Start Render.

Key findings

There are a few key findings that stand out from this analysis:

Paint timing metrics are too optimistic. In 85-95% of cases, they are recorded before any pixels are actually rendered to the screen.
First Meaningful Paint, while being an improvement over the other metrics, still suffers from the "complex rendering pipeline" issue, and half of all FMP events occur before any pixels have actually been rendered to the screen.
If you want a metric that truly represents what users are seeing on their screens, you are limited to synthetic testing tools like WebPageTest that capture a video of the screen as the page is being loaded. For real user monitoring (RUM), you are probably still better off using your own custom metrics to measure the most meaningful parts of your page load.

As a final disclaimer: it’s important to keep in mind that I analysed a small amount of data from a small sample of web pages. Modern web pages are extremely complex and very rarely do two pages render in the same way. You should always use data from your own web pages to decide which metrics best represent the experience of users of your pages.

Performance improvement checklist for your whole site

Thu, 09 Aug 2018 00:00:00 +1200

One of the longest-standing items on my performance monitoring tool wishlist is an automatically-generated performance improvement checklist, with the improvements ordered by the impact that they will have on the website. In a previous life, I spent countless hours writing performance reports and conducting A/B performance tests to figure out which change would have the biggest impact on a website's performance.

So I'm understandably very excited that today we're announcing the new Improve dashboard – a prioritised performance improvement checklist that aggregates Lighthouse and Google PageSpeed results across all the URLs in your site to identify the most impactful performance changes you can make.

Why we created the Improve dashboard

Since we added the ability to view Lighthouse scores in your test results, we realised that there needs to be an easier way to view these scores as an aggregate across all of your tests. While you could already plot the average scores in your Favorites charts, there wasn't a way to aggregate the individual audits that make up these scores – until now.

The Improve dashboard combines the Lighthouse and PageSpeed audits for all of your latest test results, and shows you how each audit affects your different URLs.

How to use it

The Improve dashboard is available in the global left-hand navbar. The UI should look relatively familiar – it's the same UI that we use to display Lighthouse and PageSpeed scores on the test results page. Along the top you'll see the average scores across all of the URLs in your site.

Below the score gauges, you'll see the individual audits for the selected score. To help you identify which audits are the most important, we give you a few ways to order them:

Order audits by their impact score

Audits can be ordered by their impact score, which is the average impact score across all of the URLs that are affected by the audit. This is the default audit order, because it helps you pick out which changes are deemed most important by Lighthouse or PageSpeed.

Order audits by how many URLs they affect

You can also order audits by their Synthetic Impact, which is calculated as the percentage of your SpeedCurve URLs that are affected by each audit. This enables you to quickly pick out the performance improvements that will affect the greatest number of your pages.

Order audits by RUM page views

LUX is SpeedCurve's Real User Monitoring (RUM) product. If you're a LUX customer, you can order audits by their LUX Impact. This is calculated as the percentage of your LUX page views that the audited URLs account for. We're especially excited about this feature, because it enables you to pick out the audits that have the biggest impact on your users in the real world.

Don't forget about performance budgets!

Now is a great time to remind you that you can also set performance budgets on your aggregate Lighthouse scores. This is an ideal way to keep track of your scores and be notified when one of your pages brings the average score for your entire site down.

If you want a hand with setting up some performance budgets, check out our support article.

Getting started

We're really looking forward to seeing how you use the Improve dashboard to drive your day-to-day performance work. If you're a SpeedCurve user, you can dive in and start using this dashboard straight away. If you're not already a SpeedCurve user, we'd love to have you give us a try for free!

Lighthouse scores now available in your test results

Wed, 25 Jul 2018 00:00:00 +1200

In the year since Google rolled out Lighthouse, it's safe to say that "Will you be adding Lighthouse scoring?" is one the most common questions we've fielded here at SpeedCurve HQ. And since Google cranked up the pressure on sites to deliver better mobile performance (or suffer the SEO consequences) earlier this month, we've been getting that question even more often.

We take a rigorous approach to adding new metrics. We think the best solution is always to give you the right data, not just more data. So we're very happy to announce that after much analysis and consideration, we've added Lighthouse scores to SpeedCurve. Here's why – as well as how you can see your scores if you're already a SpeedCurve user.

Why we added Lighthouse to SpeedCurve

Lighthouse is an open-source automated tool for auditing the quality of web pages. You can perform Lighthouse tests on any web page, and get a series of scores for performance, accessibility, SEO, and more.

You can run Lighthouse within Chrome DevTools, from the command line, and as a Node module. But there's no need to do any of those if you're already using SpeedCurve Synthetic monitoring. Every time you run a synthetic test, your Lighthouse scores appear at the top of your test results page by default.

There were a number of compelling reasons why we made the choice to add Lighthouse to SpeedCurve:

Track metrics that correlate to UX

The best performance metrics are those that help you understand how people experience your site. One of the things we like about Lighthouse is that – like SpeedCurve – it tracks user-oriented metrics like Speed Index, First Meaningful Paint, and Time to Interactive.

See all your improvement recommendations in one place

SpeedCurve already gives you performance recommendations on your test results pages. Now you can get all your recommendations – including those for accessibility and SEO – on the same page. Because we also include your PageSpeed score, SpeedCurve is the only place where you can get your Lighthouse audits and your PageSpeed Insights under one roof.

Improve SEO, especially for mobile

Several of our customers have told us that their SEO teams are very interested in using Lighthouse to help them feel their way forward now that Google has announced that page speed is a ranking factor for mobile search.

Monitor unicorn metric for CEOs and executives

When Google talks, executives listen. Many of our customers have told us that their CEO or other C-level folks don't really care about individual metrics. They want a single aggregated score – a unicorn metric – that's easy to digest and to track over time.

Get alerts when your Lighthouse scores "fail"

One of the great things about running your Lighthouse tests within SpeedCurve is that you can use your Favorites dashboard to create custom charts that let you track each Lighthouse metric. You can also create performance budgets for the metrics you (or your executive team) care about most and get alerts when that budget goes out of bounds.

For example, the custom chart below tracks three Lighthouse scores – performance, accessibility, and SEO – for the Amazon.com home page. I've also created a (very modest) performance budget of 50 out of 100 for the Lighthouse Performance score. That budget is tracked in the same chart, and you can see that the budget has gone out of bounds a couple of times in the week since we activated Lighthouse. You can also see, interestingly, that this score has much more variability than the other scores. This would merit some deeper digging.

Get started

If you're already a SpeedCurve user, you can drill down into your individual test results and find your Lighthouse scores at the top of the page. We activated Lighthouse about a week ago, so you'll find a week's worth of test data to explore.

If you're not a SpeedCurve user, you can sign up for a free trial and check out your Lighthouse scores and the dozens of other performance and UX metrics we track for you.

As always, we love your feedback!

I'd especially love to hear what your experience has been with Lighthouse. Are you using it to help with something not covered in this post? Have you learned something with Lighthouse that might otherwise have eluded you? Let me know in the comments!

Ouch, your JavaScript hurts!

Mon, 25 Jun 2018 00:00:00 +1200

When looking to improve the performance and user experience of our sites we often start by looking at the network:

What's the time to first byte?
How many requests are we making and how long are they taking?
What's blocking the browser from rendering my precious pixels?

While these are entirely valid questions, over the last few years we've seen a growing number of web performance issues that are caused, not by the network, but by the browser's main thread getting clogged up by excessive CPU usage.

When creating great user experiences, managing CPU usage is now just as important as fast networks.

How often have you been on your mobile and loaded what looks like a complete page only to discover than when you tap or scroll nothing happens or the page lags far behind your input? We need a fresh set of questions that reflect how congested the CPU on our devices might be and how that affects users:

How long is the browser's main thread trashing at 100% CPU?
When can users start interacting smoothly without interruption?
Which scripts are hurting my users?

Through the development of new web performance metrics and visualisations, we're now at a point where we can see the true cost of JavaScript, as well as monitor and improve its impact on users.

First Interactive & First Input Delay

These new JavaScript performance metrics capture for the first time just how long users are having to wait before they can smoothly interact with the page:

First Interactive is a Synthetic metric that marks the point in time when the browser's main thread is not blocked for more than 50ms by any single task so it will be able to respond to user input quickly.

First Input Delay is a RUM metric that we recently added to LUX. It measures the time from when a user first interacts with your site (i.e. when they click a link or tap on a button) to the time when the browser is actually able to respond to that interaction.

JavaScript Waterfall

One of the challenges with monitoring JavaScript performance is the ability to attribute which scripts are causing issues. To help we're excited to introduce the concept of a JavaScript Waterfall chart that blends the network and execution time of your scripts into a filterable view that lets you quickly focus on the scripts that are causing delays.

The Javascript waterfall lets you correlate the user experience (filmstrip up top) with the browser main thread (CPU usage along the bottom) and identify which scripts (middle) are delaying important browser metrics (vertical lines).

In the view below, we've filtered on JavaScript activity longer than 50ms (in red). You can see how a bunch of third-party ads have pushed the First Interactive metric right out to 12.74s. In fact, the hero content in the viewport was visible at 3s but because of those third-party ads the page was not considered usable for another 10s! This really highlights the hidden cost of JavaScript and how easily it can lead to a poor user experience.

We also group JavaScript by domain to help bubble up the most problematic consumers of CPU and also trend the CPU time used through to important browser events like start render. You can also set performance budgets on metrics like CPU scripting time to Page Load and receive Slack alerts when your page's JavaScript performance deteriorates.

The cost of JavaScript

At the recent O'Reilly Fluent Conference, Addy Osmani gave an insightful look into The Cost of JavaScript. He really challenged the complacency we often have as developers who work and test on fast devices and fast networks. It's vital we jump on old devices that reflect what average users experience and throttle the CPU of our tests. We need to remember that, byte for byte, JavaScript consumes a heck of a lot more CPU than CSS or images and can block our users from interacting with the page.

JS frameworks, friend or foe?

I love the ease of development and the functionality that JavaScript frameworks like React and Vue.js have enabled over recent years, but often we don't talk about their cost. You don't get anything for free, and there's little discussion when choosing these frameworks about the cost to users. When we push all our app logic and rendering from the server to the client, we become reliant on the device to perform. Often those devices are just not as fast as we expect them to be.

Addy highlighted an example of this tradeoff when Netflix moved parts of their website signup and video player from vanilla JavaScript to React. In the initial release they saw a degradation in client side performance and user experience. They ended up removing React from some pages and putting considerable effort into optimising performance of the player to get it back on par with the original.

What struck me is that the user experience ended up being the same. So was it worth the effort? And if they hadn't put the extra effort in, would the default React version actually have been worse for users. Are you monitoring metrics like First Interactive to make sure your users don't also suffer?

Don't let the development joy of a JavaScript framework blind you to the pain it might cause your users

When moving to these JavaScript frameworks, I believe it's critical to keep the user experience in mind and keep asking yourself if this will improve the speed of your website or hinder it. You can use tools like SpeedCurve to A/B test a prototype and quantify the cost of a JavaScript framework before jumping in boots and all.

We've set ourselves a new goal at SpeedCurve: to be the best tool for monitoring the performance of JavaScript. Jump in and explore your own JS waterfall and let us know what else you'd like to see as you dive deeper into the performance of your JavaScript stack.

First Input Delay shows how quickly your site responds to user interaction

Thu, 14 Jun 2018 00:00:00 +1200

We're excited to announce the availability of the First Input Delay metric as part of LUX, SpeedCurve's RUM product.

First Input Delay (FID) was developed by Google to capture how quickly websites respond to user interaction. It's fairly simple to implement: We add event handlers for click, mousedown, keydown, pointerdown, and touchstart. When the user first interacts with the page in one of those ways, we measure the time between when the event happened and when the event handler was actually called. That delta is FID.

For many websites, FID is typically quick. In the FID chart for WebPageTest (shown above), we see the median FID value is only 3 milliseconds. But the 95th percentile value is 30ms. (Click on the chart to interact with the live dashboard.) That's coming close to the 50ms interval that defines a Long Task when responsiveness starts feeling sluggish. To help correlate these issues we show the Long Tasks chart right below the FID chart. Along with FID we show our interaction metrics which reveal how long it takes before the user clicks, keys, or scrolls.

FID, Long Tasks, and interaction metrics combine to provide insight on how JavaScript on your page hogs the CPU and impacts the user experience. FID is only available in RUM (since synthetic tests don't have users that interact). If you're not already using RUM, start a free trial of LUX to see these metrics from your website.

Monitor performance budgets at a glance with your Status dashboard

Mon, 11 Jun 2018 00:00:00 +1200

This may sound counter-intuitive, but we don't want you to spend countless hours using SpeedCurve. In fact, our goal is to make your web performance data so accessible, understandable, and actionable that you can get everything you need from us in just a few minutes.

That's why we're so excited to announce the brand-new Status dashboard – a visualization that lets you see at a glance all your web performance budgets, as well as which budgets have been violated.

Keep reading to find out how to start using your Status dashboard to diagnose and fix your performance pains. But first, let's talk about why we built this feature.

Why we created the Status dashboard

TLDR: To make your life easier.

We're constantly debating the most important metrics to focus on and the most helpful ways to visualize your data for you. We have decades of collective web performance experience, which translates to a lot of strong opinions. You can see those opinions represented in the default charts in a few of your dashboards. These curated charts are a good way to get started. But they're just the beginning.

You know your site and your performance goals better than anyone. That's why last year we launched Favorites dashboards – where you can create your own custom charts and organize them in separate dashboards geared toward different teams and projects. And it's why we recently made it easy for you to create your own performance budgets within your Favorites dashboards. (If you're not familiar with web performance budgets and why they're essential, I encourage you to read this post.)

Some of our customers have dozens of Favorites dashboards, with dozens of charts on each. Many of our customers have signed up to get alerts when your budget thresholds are crossed, as well as weekly email reports that summarize the activity of your key metrics. These are all important features, but we wanted to do more. We wanted to make it even easier for you to see at a glance the status of all your performance budgets so that you can quickly home in on the metrics that are hurting the most.

How to use it

The Status dashboard is available as in your global left-hand navbar. Click on any budget to drill down into each metric.

You're probably wondering about that red alert, am I right? Of course you are. Below you can see that while the target backend time is 0.25 seconds, the actual backend time is 0.52 seconds. When you expand this metric, you can also see that not only is backend time at least 27% over budget across all segments in this budget, it's been that way for more than 30 days.

To drill down into the charts for each budget, just click on the link for that metric. That will take you to the detailed time series chart in your Favorites dashboard. You can then click on any point in the series to drill down into the results page for that test:

Your test results page contains a lot of great information, including suggested performance fixes for that page. Here are the recommendations for the page we've been tracking... and yes, I'm aware that it doesn't relate to back-end time. ;)

Getting started

We're super excited about the Status dashboard and its potential to simplify your daily workflow. If you're a SpeedCurve user, here's how to start taking advantage of it right away:

If you've already created custom charts and assigned performance budgets for them, then your Status dashboard is live right now. Check it out and give us your feedback!
If you haven't already done so, create some custom charts within your Favorites dashboard.
Within those charts, create a performance budget for each chart.
Go to your Status dashboard to see at a glance which of your metrics are over and under budget.

If you're not already a SpeedCurve user, we'd love to have you give us a try for free!

Weekly email reports from your SpeedCurve dashboards

Fri, 18 May 2018 00:00:00 +1200

Part of building a strong performance culture in an organisation is lowering the barrier to getting people excited about performance. One of the most effective ways I've found to do this is to send around a performance report every week that can, at a glance, answer an important question: did performance get better or worse?

That was the motivation behind our new Weekly Report feature. Now you can configure any of your Favorites dashboards to be summarized in a weekly email, like this one:

Your weekly report emails are sent out at 9am every Monday (in your time zone) and contain a week-over-week comparison of all of the metrics in the dashboard. You can see at a glance whether things are improving or getting worse.

How to enable your weekly email reports

You can enable weekly emails when you are creating or editing a Favorites dashboard by ticking the Send this dashboard as a weekly email report checkbox. You can specify recipients on a per-dashboard basis. This enables you to create dashboards – and weekly reports – with different audiences in mind.

Questions? Feedback?

As always, we love hearing your feedback on new features! Leave a comment below or send an email to support@speedcurve.com to tell us how you're using the weekly emails, or let us know if you have any suggestions for how to improve them. And if you're not already using SpeedCurve, give us a try for free.

Using RUM to track CPU time

Thu, 12 Apr 2018 00:00:00 +1200

It's exciting working at SpeedCurve and pushing the envelope on performance monitoring to better measure the user's experience. We believe when it comes to web performance it's important to measure what the user sees and experiences when they interact with your site. A big part of our focus on metrics has been around rendering including comparing TTI to FMP, Hero Rendering, and critical blocking resources.

The main bottleneck when it comes to rendering is the browser main thread getting blocked. This is why we launched CPU charts for synthetic testing over a year ago. Back then it wasn't possible to gather CPU information using real user monitoring (RUM), but the Long Tasks API changes that. Starting today, you can track how CPU impacts your users with SpeedCurve's RUM product, LUX.

We've been tracking and studying Long Tasks data for several months before launching the dashboards today, so if you're a LUX customer, you'll actually have historical data going back to January.

For example, the chart above shows CPU timing for scripts on webpagetest.org. (WebPageTest uses LUX and has generously agreed to share its data.) In this month view, we see four spikes for CPU time. At first we might think the cause is an increase in script size or number of scripts in the page, but the real answer is simply the cycle of weekday traffic from work versus weekend traffic from home.

What is a "long task"?

I summarized my early experiences with the Long Tasks API in my PerfPlanet article Tracking CPU with Long Tasks API. A "long task" is defined as any process that consumes more than 50ms on the browser main thread.

In LUX, we track the sum of the Long Tasks time for each page, and show aggregate metrics in the LUX dashboards. You can filter the data by page, region, and browser. As the Long Tasks API evolves, we'll get attribution information so you can identify which scripts are the real culprits and more information about layout, painting, and other CPU-intensive activities.

Getting started

If you're already using LUX: Check out the new CPU charts in your LUX Performance dashboard.

If you'd like to try LUX: Sign up for a free trial or contact us and we'll hook you up.

Introducing Last Painted Hero

Wed, 04 Apr 2018 00:00:00 +1200

We're excited to announce that we've launched Last Painted Hero as an official metric. Last Painted Hero is a synthetic metric that shows you when the last piece of critical content is painted. Keep reading to learn how Last Painted Hero works, why (and how) we created it, and how it can help you understand how your users perceive the speed of your pages.

The case for smarter heuristics

When choosing the right performance metric, my soapbox for the last few years has been "not every pixel has the same value". In other words, rather than chase dozens of different performance metrics, focus on the metrics that measure what's critical in your page.

Here at SpeedCurve, we think it's good to focus on rendering metrics, because they're a closer approximation to what the user experiences. There are some good rendering metrics out there, like start render and Speed Index, but the downside to these metrics is that they give every pixel the same value. For example, if the background renders and some ads render, that could improve your start render time and Speed Index score, but it might not have a big impact on the user's experience. Instead, it's better to measure the parts of the page that matter the most to users. We call those parts of the page the "hero elements".

Measuring hero elements sounds logical, but deploying metrics that do this is difficult. It's especially difficult for browsers and performance monitoring tools that don't know the content in the page. How can browsers and tools be expected to know which parts of the page are most important? The people who are best equipped to know this are website owners, which is why we encourage adding custom metrics using User Timing and Element Timing.

There are two drawbacks to custom metrics and element timing: every website has to do work to implement them and you can't compare your metrics to your competitors. To solve these problems, we came up with Hero Rendering Times, which use heuristics to determine the critical parts of the page. After analyzing thousands of websites we settled on three hero elements that are critical across almost all websites:

largest image – Many websites have a "hero image" - an eye-catching image that dominates the viewport.
largest background image – Instead of using an IMG image, sites like Airbnb and Apple use a background image. (In fact, Apple's background image has "heroes" in the URL.)
largest text – Often the critical part of the page is text, especially on media sites that contain headlines for each story. Since these pages often use custom fonts, measuring largest text (within the first h1 tag on the page) is a good way to track how fonts affect rendering.

We launched Hero Rendering Times last year as part of SpeedCurve synthetic testing. Below is an example from The New York Times. With Hero Rendering Times, site owners can track when critical parts of the page actually render and get a much better idea of what the user experiences.

Other rendering metrics have come along recently, including First Contentful Paint and First Meaningful Paint. These metrics use different heuristics to identify when critical content has been rendered, but the results vary. In my Rendering Metrics blog post – which compares how these metrics work and how well they approximate user-perceived performance – I shared a picker where you can test your own perception of critical content rendering and see how your perceptions align with a variety of popular metrics. For the purposes of that blog post, I had to come up with a composite metric for the Hero Rendering Times:

max(h1, (biggest_img || bg_img))

The composite metric is computed by taking the maximum of the largest text time ("h1") and the biggest IMG time (or biggest background image if biggest IMG doesn't exist).

Introducing Last Painted Hero

While at #PerfMatters Conference last week I bumped into Harald Kirschner (Mozilla) and Paul Irish (Chrome) who were talking about Hero Rendering Times as a standard metric. Paul asked if the composite metric had a name. I said it didn't – and in fact we hadn't yet rolled out the composite metric at SpeedCurve. Paul got on stage the next day and proposed the name Last Painted Hero. I like it!

After a busy few days, today we formally launched Last Painted Hero. If you're already a SpeedCurve user, Last Painted Hero is available in your Favorites dashboards, where you can chart it alongside your other metrics – or alongside your competitors. (If you're not a SpeedCurve user yet, try us for free!)

We also provide First Painted Hero, so you can track your best case and your worst case for hero elements in your pages.

This is just the beginning for First and Last Painted Hero

We're excited to get your input and make improvements. To that end, we're working on a pull request to add these metrics to public WebPageTest. Regardless of whether you use SpeedCurve or WebPageTest, we hope you'll take a look at First and Last Painted Hero and give us your feedback.

SpeedCurve Consulting Services with Tim Kadlec

Mon, 26 Mar 2018 00:00:00 +1300

We're excited to announce that SpeedCurve is partnering with Tim Kadlec to provide consulting services to our customers!

Tim is a recognized expert when it comes to web performance. He has spoken at numerous conferences including Velocity, Fluent, QCon, SmashingConf, Beyond Tellerand, and WebStock. He wrote High Performance Images and Implementing Responsive Design, as well as contributing to other books. Mark, Tammy and I have each collaborated with Tim on side projects. We're full of gleeful anticipation as we look forward to this opportunity to work together again.

In the first sentence I mentioned that this is a partnership. Here's what that means: Tim will continue to do consulting outside of SpeedCurve, and if you're not a SpeedCurve customer we encourage you to contact him directly. Tim will also be running SpeedCurve's consulting services. This partnership brings special advantages to SpeedCurve customers:

This partnership lets Tim get intimately familiar with SpeedCurve. This allows him to help our clients set up and focus their SpeedCurve testing, and get the most out of the results.
As Tim works with SpeedCurve clients, he will learn more about what they need from SpeedCurve. He'll feed that back into our product development, resulting in an even better product and more successful customers.
This partnership provides a single billing entity for both performance monitoring and consulting.
A big area of focus for SpeedCurve going forward is our Improve section, where we provide specific recommendations on how to make your website faster. This is a fun challenge where we codify the knowledge inside the heads of performance experts. Since Tim will be identifying these key performance recommendations as part of his consulting engagements, he'll be well positioned to help make the Improve offering better by translating those suggestions into code so that all SpeedCurve customers can benefit.

Take a look at our Pricing page to read more about SpeedCurve's consulting services. Send us an email if you'd like more information or want to enquire about engaging Tim. And watch the Improve content as we try to convert everything that's in Tim's head to code (just the performance stuff).

More RUM metrics in your Favorites dashboards

Wed, 14 Mar 2018 00:00:00 +1300

SpeedCurve comes with a great set of dashboards for synthetic and RUM. But we know that one size does not fit all when it comes to data charts, which is why we've invested so much work into the Favorites dashboards. For customers who use LUX, it provides a place to create custom charts that combine metrics from synthetic and RUM.

We just added some new RUM metrics from LUX in Favorites to allow for even more customized monitoring:

Page Views – The number of page views, including Single-Page-App page transitions
Sessions – The number of unique sessions
Session Length – The number of page views per session
Bounced Sessions – The number of sessions that only have one page view
Bounce Rate – The percentage of bounced sessions out of the total number of sessions

Bounce rate is a powerful metric for correlating performance metrics with business metrics. In the chart below, we see that as frontend time gets faster, bounce rate drops from 49% to 42%, and then rises again as frontend time regresses. This is a great way of showing why staying vigilant about the speed of your site is important for keeping your users happy.

How to get started with these new metrics

If you're already a LUX user: These metrics are waiting in your Favorites dashboard. Give them a try and send us your feedback at support@speedcurve.com.

If you're not a LUX user yet: We'd love to have you try us out for free. Sign up for your 30-day trial.

Faster test agents and more regions

Tue, 06 Feb 2018 00:00:00 +1300

We’re kicking off 2018 with a bang and a donut. We’ve been busy preparing all-new faster test agents and they’re now available for you to switch to – at no extra cost!

Go to your Settings to switch to faster test agents.

Faster agents

All our agents now run on much faster AWS C4.Large instances, which better approximate current desktop browsers. We’ve also switched to the latest WebPageTest agent running on Linux.

How much faster? 20-30%

When you switch to the new agents, some of your metrics will get 20-30% faster, so pick a time to switch that suits your team. In between projects or the start of the month are good times to set a new baseline.

More regions

We’ve added four new regions you can run your tests from. Hello… India, Canada, England, and Korea!

CPU throttling

SpeedCurve now supports CPU throttling, so you can be faster and slower at the same time! Prepare for the next billion people coming online with sloooow emulated mobile devices.

See ya Safari!

Safari is awesome – but not the version on Windows that's been dead in the water for a few years now. We’re dropping support for Safari on Windows. When you switch to the new agents, it will go poof and disappear from your settings.

Tweak your perf budgets

With your metrics changing 20-30% now’s the perfect time to review your performance budgets and alerts. To help, we’ve added performance budgets to all Favorites charts. This means you can see the data you’re setting a budget for as you create the budget!

Celebrate your perf culture

Time to gather the team and share what’s new in SpeedCurve. While you’re together, Tammy has some awesome tips on strengthening your performance culture.

How to create a healthy, happy performance culture

Thu, 01 Feb 2018 00:00:00 +1300

One of the best parts of my job is getting to talk to so many people from so many different companies about web performance. Every company is different, and I learn a ton from talking to each one. But one question that almost every person asks me – regardless of what industry they're in or the size of their organization – is this:

How can I create a stronger culture of web performance at my company?

Creating a performance culture means creating a feedback loop in your company or team that looks like this:

In other words: Get people to care, show them what they can do to help, and then give them positive reinforcement when you get results. It's basic human psychology, and it might seem obvious when you see it in a super-simple graphic. But it's surprisingly easy to miss these steps and instead skip ahead to the part where you invest in awesome performance tools, and then wonder why all your performance efforts feel like such a painful uphill slog.

In this post, I'm going to share some proven tips and best practices to help you create a healthy, happy, celebratory performance culture.

Side note: You'll notice that nowhere in that feedback loop does it say "Invest in whizbang performance tool [INSERT PRODUCT NAME HERE]." That's because these tips are about people, not tools. It goes without saying that you'll want to make sure you're using the right tools to monitor, measure, and optimize your site. But all the tools in the world won't make a difference if your people aren't invested in making a difference.

Identify what people care about

If you ask different people in your organization how much they care about performance today, you might just get a bunch of blank stares. But if you gave them a checklist and asked them to tick off any of the following boxes of things they care about, you'd probably get a more enthusiastic response:

bounce rate
cart size
conversions
revenue
time on site
page views
search traffic
user satisfaction
user retention

Over the years, I've learned that performance can be mapped to all of these metrics – and almost any other business metric you can think of. To hook different people on performance, you need to understand which metric motivates them.

For example, an executive might want to know the impact of performance improvements and slowdowns on conversions and overall revenue, while folks in your marketing team might focus on the impact of performance on everything from SEO to user engagement.

Once you know what people care about, connect the dots for them. Case studies are an awesome way to do this, which is why I've been helping curate WPOstats, a vendor-neutral directory of performance case studies. For example, if someone on your executive team cares about conversions and revenue, you can direct them to a set of studies that focus on the impact of performance on revenue and conversion rate. People on your marketing team probably care about traffic and engagement. And so on.

Case studies are a great way to convert performance skeptics. They're also a great way to get people fired up about making your site faster. When you learn, say, that Staples reduced its median load time by 1 second and saw a 10% increase in conversion rate, that's pretty compelling.

Case studies from other companies are compelling, but using your own data is even more compelling. If you're a SpeedCurve LUX user, the user engagement charts that are at the top of your Performance dashboard are an easy way to illustrate the correlation between performance and bounce rate using your own data. (One thing we've found is that almost everyone can relate to bounce rate as a meaningful metric.)

Fire up people's competitive spirit

One of the fastest routes to getting people to care about performance is to show them how slow their site is compared to their competitors. One of the great things about synthetic monitoring via tools like WebPageTest (which SpeedCurve synthetic is built on top of) is that you can test any page on the web, not just your own. This lets you do great things like generate side-by-side filmstrips – and even better: videos – of your site alongside your competitors' sites (or any other aspirational sites).

Because SpeedCurve synthetic monitoring is built on top of WebPageTest, you can take advantage of WPT's filmstrip and video capabilities. Here's how to set up ongoing competitive benchmarking and generate comparison videos .

Make things visible (but not overwhelming)

If you want to see non-technical stakeholders' eyes glaze over, show them an endless series of dashboards and charts. Less is more should be your mantra. In the same way that you need to understand what motivates different people in your organization, you also need to tailor your reporting. You can start by creating separate dashboards for different stakeholders. Each dashboard should contain only the graphs that are relevant to that stakeholder group. For some people, your performance report might be just a single graph or a very simple dashboard that shows them the data points they care about. (It stands to reason that you should always be ready to go deeper upon request.)

One excellent practice that's used effectively by companies like Lonely Planet and Ticketmaster is to have monitors mounted in open areas of their offices, displaying key performance stats and comparison videos. Lara Hogan (previously director of engineering at Etsy) wrote a great blog post demonstrating how Etsy took advantage of the power of showing versus telling.

Collaborate on performance budgets

Performance budgets are an important tool for ensuring your site is delivering a great user experience. The practice of using budgets to track performance took off with this post by Tim Kadlec. The idea is to identify your performance goals, then track the metrics that help you achieve your goals. Setting performance budgets lets you focus on the metrics you care about most, create goals, set budgets based on those goals, and get alerts when your numbers go out of bounds.

As mentioned near the top of this post, there's no one-size-fits-all unicorn metric that will get everyone in your company excited about performance. This means there's no one-size-fits-all metric you should focus on for your performance budgets. Instead, your performance budgets can run a gamut that includes:

Someone in ops needs to know Time to First Byte so that they can investigate back-end issues.
One of your developers might care about start render or Speed Index, because they want to get a sense of how well the page is built – for instance, are there any blocking scripts or stylesheets.
Someone in your marketing team wants to know how quickly the page delivers the most important content from a user's perspective, so they might want to track the rendering of hero images.

To make people accountable, give them ownership over their own performance budgets and make sure they receive alerts when their budgets are exceeded.

Super important: No sandbagging allowed. You wouldn't say you're going on a diet, then give yourself a limit of 3000 calories a day. In the same way, you want to keep your budgets realistic without making them overly forgiving. The point of a performance budget is to blend the aspirational with the achievable.

Case study: Here's how Zillow creates and enforces performance budgets

Score some easy wins

If you're new to performance – or if you're tackling a site that's new to you – there's a good chance there's some low-hanging fruit you can optimize. The first places to look are images and blocking stylesheets and scripts (especially third parties).

For example, the team at Fanatics.com did a one-month performance sprint where the most impactful changes they made were simple image optimizations: they improved image quality and compression, and they fixed an image sprite that was blocking pages from rendering. As a result, they saw a 2-second improvement in median load times and almost doubled their mobile conversions – a really big deal, since more than half their revenue comes from mobile. This relatively easy win was a fantastic way to get performance buy-in throughout their company.

With synthetic monitoring, you can drill down to find the most critical defects that are hurting specific pages. Pair that with real user monitoring, which lets you map performance to biz metrics like conversions.

Celebrate!

We live in a culture that doesn't celebrate success as much as it should. (Disclosure: I'm totally guilty of this.) So consider this a reminder: Every time you move the needle on performance – and that moves the needle on user engagement or revenue or whatever your company cares about – shout it from the rooftop. Or if that's not allowed at your office, email is fine, too.

Here at SpeedCurve, we have a #ring_the_bell Slack channel where we share wins and milestones, both big and small. And in the spirit of Lara Hogan's donut manifesto, we send each other fun gift baskets to celebrate victories.

Share what you've learned

Shouting from the rooftops (and eating donuts) is a great start. Next, share what you did and what you learned from it. This can be in the form of company- or department-wide emails, posts on your in-house dev blog, and internal meetups. I know some performance teams that do regular monthly technical meetups, and once or twice a year they do a company-wide event where they share all their greatest webperf stats and victories.

Cultivating a healthy in-house performance culture also means embracing the fact that you're a member of a culture that's much bigger than your company. There are so many ways to get involved and to learn from other webperf enthusiasts:

Share your case studies and successes (if your company allows it) on a public-facing tech blog. Good examples: Etsy's Code as Craft, Cars.com's Tech Blog, and the Financial Times' Engine Room.
Follow the #webperf hashtag on Twitter.
Participate in one of the many Web Performance Meetups around the world.
Attend performance-oriented conferences like #PerfMatters, DeltaV, and O'Reilly Fluent.

Additional resources

Cars.com explains how they use SpeedCurve to create a performance culture.
This RWD podcast episode is with a couple of members of Vox Media's performance team. Among other things, they do a great job of talking about perf culture.
Lara Hogan has written an excellent book called Designing for Performance, and she's kindly made the text available online. The entire book is great, and chapter 8 is dedicated to performance culture.

Evaluating rendering metrics

Mon, 11 Dec 2017 00:00:00 +1300

At SpeedCurve, we're fond of the phrase "a joyous user experience". Creating this joy requires delivering what users want as quickly as possible. It's important that the critical content is downloaded and rendered before users get frustrated.

Network metrics have been around for decades, but rendering metrics are newer. Speed Index. Start Render. Time to First Interactive. First Meaningful Paint. These are a few of the rendering metrics that currently exist. What do they mean? How do they compare? Which are best for you? Let's take a look.

A brief history of performance metrics

Metrics quantify behavior. In the case of performance metrics, we're trying to capture the behavior of a website in terms of speed and construction. By "construction" I mean statistics about how the site is built such as number of HTTP requests, size of stylesheets, and DOM depth. Speed is tracked by time-based metrics that capture when things happen during the user's experience with the website: start render, DOM interactive, page load, etc. Often, construction metrics are useful for diagnosing the cause of changes in speed metrics.

Why did start render get slower?

Ah-ha, the number of blocking scripts increased!

While construction metrics are fairly well established, time-based performance metrics are newer and still evolving. In the old days, the main time-based performance metric was window.onload. (Actually, before window.onload the main performance metric was the response time of the HTTP request for the HTML document!)

The W3C Web Performance Working Group changed everything with the Navigation Timing, Resource Timing, and User Timing specifications. Navigation Timing and Resource Timing let us measure things like DNS lookups, TCP connections, and SSL negotiations for the main page as well as all the resources in the page. Most importantly, Navigation Timing gives us a spec'ed milestone for when the page begins to load (performance.timing.navigationStart), as well as other milestones like DOM Interactive and DOM Complete. User Timing doesn't come with any predefined metrics, but instead provides an API for capturing website-specific milestones by simply calling performance.mark and performance.measure.

Gaps in today's performance metrics

While the specifications mentioned previously have greatly improved our ability to measure performance, they actually don't help us capture the most important aspects of web performance. There are two big gaps:

Gap 1: Browser Main Thread

The gating factor in web performance ten years ago was network. Today, the main bottleneck is CPU. The cause of this shift is the increase in CSS and JavaScript, as well as more mobile users. Over the last seven years, the number of scripts jumped from 12 to 36 and the compressed size increased from 110K to 617K for the world's top 1000 websites according to the HTTP Archive.

Construction metrics provide information about the number and size of scripts and stylesheets. But this isn't a direct measure of how JavaScript is impacting the page. Suppose a website has just one script, but that script takes 3 seconds to execute. Another website might have four scripts, but each of them takes less than 100 milliseconds to execute. The number of scripts is a good estimate for whether JavaScript is the bottleneck, but it's based on assumptions about typical execution times.

It would be better if there was a way to measure the time each script, especially the blocking scripts, consume on the browser's main thread. Chrome Dev Tools has this information, and a year or so ago Pat Meenan added this to WebPageTest's waterfall charts using pink bars:

This example illustrates how construction metrics can be misleading. The first script in the waterfall above is the smallest so many people might assume it would execute the fastest, but it actually takes the longest to execute. This insight into browser main thread activity isn't limited to JavaScript execution; WebPageTest also provides information about layout, paint, and loading. In order to help teams focus on CPU, SpeedCurve extracts this information from WebPageTest and shows it in our default dashboards.

However, this insight into browser main thread bottlenecks is limited to synthetic testing. It would be great if there was a CPU Timing specification so these metrics were also available in Real User Monitoring (RUM). The Long Tasks API is new and could help address this gap.

Gap 2: Rendering

There are a number of rendering metrics available today, but there's a "gap" because of the lack of standardization and accuracy. Let's take a quick survey.

WebPageTest is the pioneer in rendering metrics (and many many other parts of performance monitoring). Here's what it offers when it comes to rendering:

Speed Index is "the average time at which visible parts of the page are displayed. It is expressed in milliseconds". Speed Index is not a specific point-in-time but instead is an aggregate stat so is a bit apples-to-oranges when compared to these other time-based metrics.
Start Render is "the time from the start of the initial navigation until the first non-white content is painted".
Time to First Interactive (aka, Time to CPU Idle) is "when the page is first expected to be usable and will respond to input quickly." Specifically, it is the first span of 5 seconds where the browser main thread is never blocked for more than 50ms after First Contentful Paint. Note that this is more than just a rendering metric as it also captures interactivity.
Time to Consistently Interactive (aka, Time to Interactive) is "when the page is first expected to be usable and will respond to input quickly." Specifically, it is the first span of 5 seconds where the browser main thread is never blocked for more than 50ms after First Contentful Paint with no more than 2 in-flight requests. Note that this is more than just a rendering metric as it also captures interactivity.
Visually Complete is the "first time when the visual progress reaches 100%".

The new Paint Timing specification defines two new metrics. You can extract these values using performance.getEntriesByType("paint"):

First Paint is "the time when the browser first rendered after navigation. This excludes the default background paint, but includes non-default background paint."
First Contentful Paint is "when the browser first rendered any text, image (including background images), non-white canvas or SVG."

To compensate for shortcomings in First Contentful Paint, Chrome's Lighthouse project attempts to capture when primary content is rendered to the user. This metric can be found in Chrome traces.

First Meaningful Paint is "the paint after which the biggest above-the-fold layout change has happened, and web fonts have loaded." (More detail is available here.)

SpeedCurve also created a new metric that tries to capture when critical content is rendered to the page:

Hero Rendering Times is a combination of when the largest IMG, H1, and background image in the viewport are rendered. The composite metric is computed by taking the maximum of the H1 time and the biggest IMG time (or biggest background image if biggest IMG doesn't exist): max(h1, (biggest_img || bg_img)).

Evaluating rendering metrics

In order to evaluate the different rendering metrics, we recorded WebPageTest filmstrips for the world's top ~100 websites. To make the evaluation more interactive (and fun), we created the rendering metrics Picker page. It shows the screenshot that corresponds to each metric, but the metric names are hidden until you choose the one that corresponds to what you consider to be a "good UX". It keeps track of which metrics you're most aligned with. (If you want to see all the metrics without clicking on each filmstrip, just scroll to the bottom and click "Show All".)

I wish I could say "X is the best rendering metric", but it really depends on what you're after. Hopefully you went through the exercise of picking which rendering metric best corresponds to what you feel is a "good UX". For your site, you might be most interested in getting any content to the page as quickly as possible (Start Render). Or maybe you're interested in getting a significant amount of content to the page (First Meaningful Paint, Speed Index). Perhaps you care most about when critical content appears in the page (Hero Rendering Times).

Regardless of the question of which rendering matters the most to you, there are a few criteria to keep in mind when evaluating these metrics.

Does every pixel have the same value?

It's important to focus on rendering when it comes to delivering a good user experience. At SpeedCurve, we feel it's more than just rendering pixels - it's really about rendering the most important pixels. Not every part of the page has the same value. There are certain design elements - the call-to-action, the hero image, etc. - that are more important. A good metric should be able to measure the most important pixels when it comes to rendering.

Automatic or requires work?

One of the reasons why window.onload is still the most-used performance metric is that it exists in all browsers by default - the website owner doesn't have to do any work to generate this metric. On the other hand, User Timing lets teams track metrics that matter the most to them, but it requires writing, pushing, and maintaining code to generate these metrics - which is probably why only ~5% of the world's top 500K websites use it. All other things being equal, there's a preference for metrics that work automatically.

Synthetic or RUM?

Performance metrics are gathered either synthetically or via Real User Monitoring. Ideally, the metric of choice supports both. (Note: If a technique can be gathered by RUM it's considered available to synthetic as well, since synthetic monitoring solutions generally have access to JavaScript.)

The following table categorizes each metric into the appropriate quadrant:

	Requires Work	Automatic
Measures important pixels	User Timing (both) Hero Element Timing (both)	Hero Rendering Times (syn)
All pixels the same	User Timing (both)	Speed Index (syn) Start Render (syn) Time to First Interactive (syn) Time to Consistently Interactive (syn) Visually Complete (syn) First Paint (both) First Contentful Paint (both) First Meaningful Paint (syn)

We're pretty excited about Hero Rendering Times. It uses heuristics to estimate which content is critical (H1, hero image, and largest background image). It's available without the website owner having to do any extra work, which means it can also track competitor websites. And it scores well in the subjective results from "which metric best approximates a good UX". However, it's only available in synthetic monitoring.

There isn't a good rendering metric available in RUM. There are only three choices right now: First Paint, First Contentful Paint, and User Timing. The results in our examples for First Paint and First Contentful Paint don't map well to actual screen content. This is probably due to the fact that Chrome doesn't know when the pixels actually get painted to the screen. For synthetic results we could round up to the next painted thumbnail, but it's not possible to make this adjustment for RUM. Perhaps website owners could always add an offset, but in our results the delta ranged from 100ms to 3 seconds.

User Timing is the third alternative for RUM. The good thing about User Timing is it can be used to measure the rendering time for the critical content in the page, but this requires using special techniques that are tricky at best. In addition, it's possible that the results suffer from a similar delta delay as we see with First Paint and First Contentful Paint. (This is an area for further research.)

Hero Element Timing is a fourth candidate for RUM rendering metrics, but it's in the early proposal stages and not currently available in any browser. And again, it's likely that this metric would suffer from the same delta delay as the others discussed previously.

What's next

There's work to do in rendering metrics for RUM. First Paint and First Contentful Paint don't necessarily correspond to a "good UX" (i.e., critical content). Hero Element Timing will be extremely useful for tracking critical content in RUM once browsers support it.

The question of measuring "meaningful" content is an area for more research. In our experience with the Picker, First Meaningful Paint often doesn't match the point where critical content is rendered. Hero Rendering Times does better at identifying when critical content hits the screen, but isn't without flaws. Its calculations are based on analyzing the filmstrip screenshot images. Rotating content and pop-ups are a challenge, as is transparent H1 elements that overlay hero images and background images. Identifying critical content without requiring website owners to modify their code is an exciting goal for future work.

Further down the road it'd be great to get more CPU information for RUM possibly from the Long Tasks API. This could help developers diagnose why their rendering metrics have changed, especially if timing information can be associated with individual scripts and stylesheets.

In this blog post I focused on rendering metrics and haven't spent much time on the interactive metrics (Time to First Interactive and Time to Consistently Interactive). It's important that these interactivity metrics incorporate a rendering metric as a baseline. Currently, they rely on First Contentful Paint but other rendering metrics should also be considered, especially Hero Rendering Times since it's performed well in our studies.

By popular demand: Multiple Favorites dashboards!

Thu, 02 Nov 2017 00:00:00 +1300

Rolling out new features is always a blast, and it's extra rewarding when the new feature is a response to a customer request. We've had many conversations with SpeedCurve users who've told us that multiple Favorite dashboards would be a huge benefit for their teams.

Today, we're very excited to announce that multiple Favorites dashboards are now available. Here's why you need them and how to create them.

Who needs multiple Favorites dashboards?

The ability to create multiple Favorites dashboards is a huge boon for organizations that have many different teams and projects on the go at any given time. They're also helpful if you need to curate charts for different audiences.

For example, if you need to do regular performance presentations at monthly management meetings, you can curate high-level charts in an "Executive Summary" dashboard. Each dashboard also allows you to generate a "share" URL – which can be sent to non-users as well – so you can also share individual dashboards via email or on a display monitor.

Within your performance team, you can create specific dashboards for separate projects. For example, if you're using custom timers to track ads on your pages, you can create a dashboard called "Third Party Ads". And so on.

Why you need custom charts

We think our curated charts, which are available by default with your SpeedCurve plan, are pretty great. We've put a lot of thought into creating visualizations that are as meaningful as possible.

But you know your own business better than we do. By building custom charts, you can cherry pick the metrics and data that are most important to your business and track them over time.

Custom charts let you do things like:

Combine multiple metrics
Combine synthetic and LUX (real user monitoring) data in one place
Choose average, median, or 95th percentile
Select multiple values for a filter (e.g., browser = Chrome or Firefox, country = UK or US)
Compare A/B tests in a single chart

Getting started

If you’re already a SpeedCurve user: Creating multiple Favorites dashboards is easy! You can get started in minutes. Check out our short tutorial that walks you through the process for creating custom dashboards and charts.

If you’re not a SpeedCurve user: Sign up for your free trial and start creating your own custom charts and dashboards.

How have custom performance charts helped you?

We'd love to know what kind of custom charts you've created, and how they've helped your performance team and your company. Let us know in the comments, or email us at support@speedcurve.com.

Engagement charts: See correlations between performance and user engagement

Thu, 26 Oct 2017 00:00:00 +1300

One of the best – and worst – things about real user monitoring is that it gives you unparalleled access to massive amounts of user data. The problem is when all this data leads to data indigestion. How do you know where to begin? And how do you know what to leave out in order to present a clear case for performance?

At SpeedCurve, we care about more than just showing you all your data. We want to show you the most important data. And we want to make it easy for you to share that data with people throughout your organization. That’s why we’re excited about the newest addition to our family of visualizations: engagement charts.

What are engagement charts?

Engagement charts – which are generated by the RUM data gathered by SpeedCurve LUX – are the first member in our new family of correlation charts. (We'll be introducing new members of the family down the road, but for today, let's focus on engagement.)

Engagement charts give you a histogram view of all your user traffic, broken out into cohorts based on start render and load time. You also get an overlay that shows you the bounce rate that correlates to each of these cohorts. This lets you see at a glance the relationship between page speed and user engagement. (Spoiler alert: The dominant trend is that as pages get slower, bounce rate gets worse.)

>> Demo: Interact with engagement charts in our live LUX demo account

Why you need them

Engagement charts are a great way to communicate about performance to a business audience. These visualizations let even the most non-technical stakeholder easily see the correlation between performance and user engagement. In my experience with talking about performance to a wide variety of audiences, graphs like this (which I’ve manually created the old-fashioned way in the past) can be extremely effective in winning performance buy-in.

But wait… there’s more!

Spot performance-blocking trends

We’ve created overlays that let you see the correlation between blocking JavaScript and CSS with your other metrics.

This lets you spot trends on your pages – for example, in the sample graph above, you can see that while there are fewer blocking resources on the faster pages, this number takes a sharp upturn starting with the cohort of pages that have a start render time of 1.1 seconds. If your goal is to deliver faster start render times to more users (and 1.1 seconds is a pretty good goal to shoot for), then this might trigger you to do an audit of your pages to analyze how your scripts and stylesheets are being executed.

Get started

If you’re already a SpeedCurve LUX user: Engagement charts are available by default at the top of your LUX Performance Dashboard. Use the arrow icon in the top right corner of the dashboard to easily share them with your team members and other folks in your organization.

If you're a SpeedCurve Synthetic user, but haven't tried LUX yet: Activating your LUX trial is easy: all you have to do is grab the LUX ID for your team (available in the Admin>Teams panel), install the lux.js snippet on your site, and let us know when it’s live so that we can turn on your trial.

If you’re not a SpeedCurve user: Explore engagement charts in our live LUX demo account, then sign up for your free trial and get these graphs for your own site.

How would you use engagement and correlation charts?

As always, we’d love to hear your feedback and suggestions for iterating on these charts. Which metrics would you love to see by default, and why? Let us know in the comments, reply to our Twitter announcement, or send us a note at support@speedcurve.com.

Keeping track of performance changes

Wed, 11 Oct 2017 00:00:00 +1300

Being able to track which changes have an impact – either positive or negative – on your site’s performance is an important part of performance monitoring that can provide valuable feedback to your team. We wanted to make it easier to see at a glance all the changes to your site, without the cognitive overhead of interpreting charts. That’s why we created the new Changes dashboard, which gives you a newsfeed-style overview of recent activity in your SpeedCurve account.

Your Changes dashboard shows your performance budget alerts, deploys, site notes, and SpeedCurve product updates:

The Changes dashboard is designed to be quickly scanned for changes that are relevant to you. Everything is ordered chronologically to make it easy to correlate changes, and each change can be expanded to reveal more detail.

To save you the trouble of checking back for updates, we show a small notification in the navigation bar whenever there are any new changes for your sites:

Get started

The Changes dashboard is a great place to keep track of performance changes across your sites. If you're already a SpeedCurve user, you can start exploring this feature right away. If you're not yet a SpeedCurve user, sign up for your free trial and check it out.

A retailer's guide to web performance

Thu, 28 Sep 2017 00:00:00 +1300

I’m at Shop.org this week, having really interesting conversations with online retailers. What I love about talking with this crowd is that – like me – they're super focused on user-perceived performance. Not surprisingly, we have a lot to talk about.

Making customers happy is the not-so-secret secret to retail success. Delivering a fast, consistent online experience has been proven to measurably increase every metric retailers care about – from conversions and revenue to retention and brand perception. (In fact, there's so much research in this area, I dedicated an entire chapter in my book to it. You can also find a number of great studies on WPO Stats.)

Delivering great, fast online experiences starts with asking two questions:

What’s making my pages seem slower for my users?
What can I do about it?

The good news is that most of the issues that are making pages slow for your shoppers are right on your pages, which means you have control over them. Here's an overview of the most common performance issues on retail sites, and how you can track them down and fix them.

What's making my pages seem slower?

I’ve analyzed hundreds (and possibly thousands) of retail web pages over the past several years. In my experience, the vast majority of performance issues are caused by four things:

Third parties, such as trackers and analytics
Stylesheets
Custom fonts
Page size, particularly image size

Third parties

A typical retail web page these days can contain upwards of 75 third-party scripts, such as trackers and analytics beacons. Third parties add tons of value by increasing conversions (via targeting beacons) and helping you understand your users in unprecedented ways (via analytics tags), so they’re not going away any time soon. But as pretty much everyone knows by now, they can really affect how your page renders.

What you can do: Monitor the speed of your third parties

Start by taking a proactive approach to understanding any potential performance damage your third parties can cause. You can't fix what you're not measuring, which is why we built a third-party waterfall chart into SpeedCurve, so you can see at a glance how your third parties are performing.

Read > UX focus for third parties

Case study > Marks & Spencer revolutionized performance by focusing on third party content

Live demo account > SpeedCurve Third Party Dashboard

Stylesheets

Stylesheets are an incredible boon for modern webpages. They solve a myriad of design problems, from browser compatibility to design maintenance and updating. Without stylesheets, we wouldn’t have great things like responsive design. However, poorly executed stylesheets can create a host of performance problems. These range from CSS that takes too long to download and parse, to improperly placed stylesheets that blocks the rest of the page from rendering.

What you can do: Know which of your stylesheets can block your pages from rendering

Again, you can't fix what you don't measure. We've made it easy for you to find out if your pages have critical blocking resources, such as CSS and JavaScript. This chart is available on your Site dashboard and lets you quickly spot any issues.

Read > Measuring blocking resources

Live demo account > SpeedCurve Site Dashboard

Custom fonts

Custom fonts allow designers unprecedented control over the typefaces used in their designs. This desire for control accounts for the surge in popularity for custom fonts. In 2010, only 1% of the top 1,000 websites used custom fonts. Today, that number has grown to 65%.

This popularity comes with a performance price tag. Some fonts require huge amounts of CSS code, while others have heavy JavaScript or are hosted externally – all of which can dramatically slow down page rendering.

What you can do: Track how quickly your fonts render

SpeedCurve’s new Hero Rendering Times are a set of metrics that measure when a page's most important content finishes rendering in the browser. H1 Render is one of these metrics. (I'll be talking about the other metrics later in this post.) It measures when your first H1 element finishes rendering. If a site uses custom fonts, it's quite likely that those fonts will be applied to H1 copy, which makes this metric a great way to measure how quickly your custom fonts are rendering.

Read > Hero Rendering Times: New metrics for measuring UX

Case study > NerdWallet used the H1 Render metric to help improve font loading by up to 30%

Live demo account > SpeedCurve Site Dashboard

Images and page size

I’ve written about page bloat a lot over the years (most recently in this post). While page size doesn’t always correlate to a slower user experience, it often does. Huge unoptimized images are a frequent culprit. I’ve seen retail pages that contained hero images that were up to 5MB in size. That’s a lot of bytes, folks.

What you can do: Create performance budgets

Performance budgets are a great way to manage page bloat. A performance budget starts with your team (everybody: marketing, designers and developers) agreeing on the principles around the user experience and how fast your website should feel.

You define a budget for each metric, such as the total size of your JavaScript, how many images should be on the page, and how quickly the page should start rendering. Once you define your budget, SpeedCurve starts monitoring and alerts you if the target budget is exceeded.

Read > Performance budgets in action

Case study > How Zillow became bigger, faster, and more engaging while on a budget

Live demo account > SpeedCurve Site Dashboard

What you can do (part deux): Track the speed of your hero images

We’ve also made it easy for you to see at a glance how quickly your hero images render. For example, the Largest Image Render metric identifies when the largest image in the viewport finishes rendering. This metric is especially relevant to retail sites, where hero images on home, product, and campaign landing pages are critical elements.

Read > Hero Rendering Times: New metrics for measuring UX

Live demo account > SpeedCurve Site Dashboard

Get started

Delivering fast, joyous experiences to online shoppers has never been as important as it is now. If you're already a SpeedCurve user, you can take advantage of the many ways to make your site feel faster for your users. If you're not already a SpeedCurve user, explore our live demo account, sign up for your free trial so you can explore these features and get to know your site better – the way your users see it.

Woohoo! I'm helping build SpeedCurve

Tue, 12 Sep 2017 00:00:00 +1200

I’m super excited to be able to say that I’ve joined Mark, Steve, and Tammy at SpeedCurve!

I’ve watched how Mark has shown over the last couple of years that performance monitoring doesn’t have to be dry and data-heavy; it can be insightful, interactive, and actionable. I’ve also been a follower of Steve’s work for many years. In fact, I should probably thank Steve for providing me with the knowledge that got me interested in web performance in the first place! Tammy’s work has been really interesting to follow, too - her focus on real people and how web performance impacts the way they use our websites is something that resonates strongly with me.

Joseph making BBC News way faster for all users.

For the last 3 years I worked at BBC News as a Principal Software Engineer. The BBC News website is accessed by 10 million people every day - people from all around the world using every conceivable device on all sorts of network connections. I always felt that it was important to understand whether or not all of those people were having a good experience when they visited our website.

I got a glimpse of what sort of experiences people were having every time I went travelling. In rural Scotland, BBC News pages would take up to a minute to load. When I visited Morocco the connection would time out before I saw any content. In New Zealand the pages would initially load quickly but then the content would jump around while adverts were being loaded. But this was all anecdotal and I had no way to back it up empirically.

We trialled several performance monitoring tools at the BBC, but I was always frustrated that they were unable to give a simple answer to questions like: How fast are our pages? Are they getting faster or slower? What should we work on next to improve performance? When I was introduced to SpeedCurve, this all changed. The important data is presented in a way that’s easy to understand and -more importantly- easy to communicate to non-technical stakeholders.

I’ve always believed that performance is not a technical problem - that the hardest part of making a website fast is educating other people about the impact that performance has on users and on the business. That’s why I’m really excited to help build what I think is the best tool available for doing that.

Waterfall with browser events

Fri, 08 Sep 2017 00:00:00 +1200

We've improved our already fantastic interactive waterfall chart with a new collapsed mode that highlights all the key browser events. This lets you quickly scan all the events that happen as the page loads and if you scrub your mouse across the waterfall you can easily correlate each event to what the user could see at that moment.

Along with all the browser metrics you also get to see our new hero rendering times in context. Click on any event to see a large version of that moment in the filmstrip.

If you want to dig into the detail of the network requests and how they relate to the browser events, just click "Expand" or on the background of the waterfall chart.

We've now made it even easier to correlate network requests, browser events and browser main thread with what the user is actually seeing as the page loads. All your web performance debugging goodness in one powerful interactive tool!

If you're already a SpeedCurve user, you can start exploring this feature right away. If you're not yet a SpeedCurve user, sign up for your free trial and check it out.

Hero Rendering Times: New metrics for measuring UX

Tue, 22 Aug 2017 00:00:00 +1200

The key to a good user experience is quickly delivering the content your visitors care about the most. This is easy to say, but tricky to do. Every site has unique content and user engagement goals, which is why measuring how fast critical content renders has historically been a challenging task.

That's why we're very excited to introduce Hero Rendering Times, a set of new metrics for measuring the user experience. Hero Times measure when a page's most important content finishes rendering in the browser. These metrics are available right now to SpeedCurve users.

More on how Hero Rendering Times work further down in this post. But first, I want to give a bit of back story that explains how we got to here.

A brief history of UX metrics

In the beginning, there was load time. But then we realized that there was too much stuff on pages (e.g., third parties, outside-the-viewport content) that resulted in long load times, and these long load times didn't actually reflect how fast pages felt to users.

Start render is better than load time, because it measures when content starts to render – and when users can ostensibly begin to interact with the page. The other advantage is that it can be used in synthetic, so it's a good-enough way to compare the UX of your site to your competitors. But start render isn't ideal, because it only measures the beginning of content rendering, not when users can actually engage meaningfully with the page.

Then Speed Index came along. This metric, which is measured in WebPageTest, identifies when the visible parts of a page are displayed. This was a huge evolutionary leap in that it represented the first time any tool attempted to tackle the issue of measuring what users actually see. Speed Index is based on measuring when visual content (e.g., images and video) render within the viewport.

We like Speed Index (which is why we've included it in SpeedCurve), but because it's dependent on how the page is built, it's not a one-size-fits-all metric. For example, I recently spoke with a customer who was getting confusing Speed Index results for mobile. It turned out that the reason why their Speed Index was fast – despite the fact that their viewport was almost empty – was because most of their "above the fold" content was not image-based.

Recently, metrics like Time to Interact (TTI) and Time to First Meaningful Paint (TTFMP) have gotten a lot of attention. But TTI doesn't consistently and accurately measure user experience. And while TTFMP – which is available through the new Paint Timing API – does illustrate when content becomes visible in the browser, its weakness is that, like Start Render and Speed Index, it gives equal value to every pixel. In other words, it doesn't discriminate between useful and non-useful content. (This is a hugely fascinating topic for discussion, which is why I'll be talking more about these metrics in an upcoming post.)

Custom metrics are a great solution (and they remain the best solution for real user monitoring), but they require additional dev time and expertise, which might be why only 14% of sites use them.

We need better metrics for measuring user experience

Given the constraints and limitations of existing metrics, we realized we need a solution that:

Correlates to what users actually see in the browser.
Recognizes that not all pixels and page elements are equal, from a user's perspective.
Allows us to customize what we measure on specific pages.
Is easy to use and accessible right out of the box.

Introducing Hero Rendering Times

Here's the great news: You don't have to do anything at all to get started! If you're a SpeedCurve user, these core Hero Rendering Times are already sitting in your Sites dashboard. If you're not a SpeedCurve user, click on each of the images below to check out these metrics in our live demo dashboards.

Largest Image Render

Just as its name suggests, Largest Image Render identifies when the largest image in the viewport finishes rendering. This metric is especially relevant to retail sites, where images on home, product, and campaign landing pages are critical elements.

H1 Render

H1 Render measures when your first H1 element finishes rendering. This metric is especially useful to media and informational sites. Because the H1 tag is usually wrapped around header copy, there's a reasonable assumption that this is copy you want your users to see quickly.

>> Case study: How NerdWallet used H1 Render Time to help improve font loading by up to 30%

Largest Background Image Render

For some pages, the background image is just as – or more – important than the largest image. We created this metric to ensure that you're not missing out.

These core hero metrics are a great start, and they're now available by default in SpeedCurve. We've also created one more Hero Rendering Time that requires a bit more work on your end, but which will allow you to fine-tune your monitoring...

Hero Element Timing

Default metrics are important to have – not least because they're based on universal page elements, which means you can use them to compare your site to your competitors. But chances are you want to get a more nuanced look at hero elements that are unique to your site, your pages, and your users. This is where Hero Element Timing comes in.

Hero Element Timing – which is based on the Hero Element Timing API – lets you select and annotate specific hero elements on your pages, such as search boxes, image carousels, and blocks of text.

Right now, if you're a SpeedCurve user, you can follow the instructions in the API spec to annotate your pages, and see the results in your SpeedCurve results. (As a bonus, when browsers inevitably catch up and adopt the spec, you'll be ahead of the game.)

Get started

That's it! Login to your SpeedCurve account to check out your core Hero Rendering Times and start exploring Hero Element Timing. If you're not already using SpeedCurve, sign up for a free trial.

If you have any questions or feedback about Hero Rendering Times, we'd love to hear them.

The average web page is 3MB. How much should we care?

Wed, 09 Aug 2017 00:00:00 +1200

A couple of month ago, someone asked if I'd written a page bloat update recently. The answer was no. I've written a lot of posts about page bloat, starting way back in 2012, when the average page hit 1MB. To my mind, the topic had been well covered. We know that the general trend is that pages are getting bigger at a fairly consistent rate of growth. It didn't feel like there was much new territory to cover.

Also: it felt like Ilya Grigorik dropped the mic on the page bloat conversation with this awesome post, where he illustrated why the "average page" is a myth. Among the many things Ilya observed after analyzing HTTP Archive data for desktop sites, when you have outliers that weigh in at 30MB+ and more than 90% of your pages are under 5MB, an "average page size" of 2227KB (back in 2016) doesn't mean much.

The mic dropped. We all stared at it on the floor for a while, then wandered away. And now I want to propose we wander back. Why? Because the average page is now 3MB in size, and this seems like a good time to pause, check our assumptions, and ask ourselves:

Is there any reason to care about page size as a performance metric? And if we don't consider page size a meaningful metric, then what should we care about?

Before we wade into this topic (again), some big important caveats

The averages we're about to look at, which are taken from the HTTP Archive, are just that – averages of large datasets. They don't represent the "typical" website, because there's no such thing as a typical website.
These numbers are mainly relevant when viewed in a historical context. They represent trends – that's all.
These numbers should not in any way even remotely be taken as a benchmark for your own site. You haven't necessarily achieved anything great if your pages are smaller than this, nor have you failed if your pages are bigger.
Not all pages are getting bigger. Many have gotten smaller over the years. Maybe yours is one of them!

Graphs or it didn't happen, right?

Here you can see page growth from 2011 to now, broken out by content type:

The first thing that jumps out is the large amount of page real estate being taken up by video. Not a huge surprise, given the popularity of hero videos and such, but still interesting to note that this seems to account for much of the recent growth.

Custom font use continues to increase: 69% of the top 500,000 websites use them. Interesting to note the dip that took place, in terms of total KB, in 2016.

It's interesting to graph some of the current HTTP Archive data (below) and see which metrics remain flat relative to page size and which ones don't.

You can see above that start render is fairly consistent regardless of page size. This is quite interesting, because it suggests that bigger pages don't necessarily correlate to when users start being able to see content.

Also, you can also see how easy it is to be misled by onload as a performance metric, because it correlates so strongly with page size. In this graph, the steep rise of onload slightly hides the fact that Speed Index trends upward quite significantly – from 2393 (~2.4 seconds) for pages in the 500KB cohort to 10266 (~10.3 seconds) for pages in the 20MB cohort. This serves as a good reminder that Speed Index is generally a solid synthetic metric for user experience.

Prediction: 4MB pages by 2019?

I'm putting this out there as an interesting talking point, not as a cause for panic. Assuming a roughly 16% year-over-year increase in page size, the average page could exceed 4MB in just over two years.

But again, going back to Ilya's point, this is just an average. 4MB pages are already here. According to the HTTP Archive, almost 16% of pages today – in other words, about 1 out of 6 pages – are 4 MB or greater in size. I routinely see pages (and I'm sure you do, too) that are 10MB or larger. When I was talking about this issue with Mark and Steve, Mark referred to the fact that he's built pages that were 30MB that were still highly performant.

If you care about user experience, page size isn't the right metric to track

Metrics like page size and load time typically are not good indicators of user-perceived performance.

Take Amazon, for example. It's widely considered to be a performance leader, yet it has relatively heavy pages (I'm defining "heavy" as 3MB or more) and slow load times (I'm defining "slow" as 5 seconds or more). But for Amazon, page size and load time are the wrong metrics to look at.

For example, looking at this recent test result page for the Amazon home page (which weighs in at just over 5MB), you can see a start render time of 1.4 seconds and a well-populated viewport at 2.5 seconds – despite the fact that the page doesn't fully load till 18.8 seconds.

(If you're not already a SpeedCurve user and you want to noodle around with the SpeedCurve synthetic monitoring dashboard, check out our demo account, which lets you explore data for a handful of media sites, including The Guardian, Huffington Post, and The New York Times. Even better, sign up for a free trial and take SpeedCurve out for a spin.)

Takeaways

1. Page size matters, but perhaps not in the way you think

You can have large, robust pages that still feel fast. But you should care about page bloat in terms of how it affects mobile users, especially mobile-only users who are dealing with bandwidth constraints or data limits. At Fluent this past June, Tim Kadlec delivered a passionate keynote that addressed this issue. You should also check out Tim's nifty online calculator, which calculates the cost, in dollars, of your pages in countries around the world. It's an eye-opener.

What you can do: If you're not actively using performance budgets to set thresholds for metrics like page size, start render, and Speed Index, you should start. I love this short explainer video that explains how performance budgets work.

2. Worry about images, but not too much

Yes, images make up the bulk of the average page, and you should definitely make sure you're not serving huge unoptimized images to your users. But this is one of those low-hanging fruit that's relatively easily addressable.

What you can do: Find and fix problem images on your pages.

3. Worry more about CSS and JavaScript

If you're serving asynchronous versions of your stylesheets and scripts, you need to know that these have the potential to block your pages altogether, because they're major CPU hogs.

What you can do: Asynchronous scripts are better than synchronous, but there's an argument for deferring scripts (if you can wrangle it). And if you're not already measuring CPU usage, you should consider starting now.

4. If you care about measuring user experience, then use custom metrics

Correlating page size with user experience is like presenting someone with an entire buffet dinner and assuming that it represents what they actually ate. To properly measure user experience, we need to focus on the content – such as the navbar or hero product image – that users actually want to consume. The best performance metric for measuring user experience is one that measures how long the user waits before seeing this critical content.

What you can do: This is where custom timers, via the W3C User Timing spec, come in. To implement custom timers, you need to identify the critical content on your pages, then add marks and measures to track when they render. Steve wrote a great blog post that goes into custom timers in more detail, as well as providing a handful of sample metrics to get you started. If you care about measuring UX, I strongly recommend checking it out.

To sum up...

At SpeedCurve, we don't think you need more performance data. We think you need the right performance data. That's why we're always working to develop metrics that give you meaningful insight into how your users experience your site. And it's why we believe setting performance budgets and alerts for those metrics is crucial. (If you're not already using SpeedCurve to monitor your site's performance, set up your free trial here.)

The key to a good user experience is quickly delivering the critical content. This is easy to say, but historically has been tricky to do. Custom metrics are a huge evolutionary step forward. If you're using custom metrics, I'd love to hear how they're working for you and what you're learning from them. And if you're not using custom metrics, I'm curious to learn what the barriers are for you.

Find and fix what matters most

Thu, 22 Jun 2017 00:00:00 +1200

Being able to monitor and measure the performance of your pages is crucial. You know that already. You also know that the next step is to quickly find out what’s hurting your pages so you can stop the pain.

You want to know:

Which performance rules is my page breaking?
How do I prioritize my optimization efforts?
How can I communicate this quickly and clearly to my team?

We’re super excited to announce that you can now use SpeedCurve to answer these questions.

Inspired by the great work done by Google’s PageSpeed Insights team, we developed our own curated list of performance rules. We took the rules that are most relevant to developers in the real world, then augmented them with data that’s unique to SpeedCurve – such as critical blocking JavaScript and CSS. And we packaged it with an intuitive UI that makes it easier for you to share your test results with your team, as well as other non-technical teams throughout your organization.

Get started

To get to your page-level diagnostics, go to your test timeline and open up any test results page. Scroll down, and voila. At a glance, you can see our curated list of performance rules, stacked and color-coded in their order of importance for this particular page. (We define “importance” as “the total amount of positive impact that optimizing for this rule would have on performance”.)

You can then expand any of these rules and see a list of specific assets that are causing trouble – again, listed in order of importance.

And here’s a feature I really love: When you look at your list of images that need optimization, you actually see thumbnails of each one, as well as a calculation of how much further each image can be optimized. This is a fantastic, fast, effective way to communicate to other teams in your organization, such as marketing and sales.

At SpeedCurve, we know you don’t have infinite amounts of developer hours to optimize your pages. We want to help you focus on what matters. This new feature is just the beginning.

I've joined SpeedCurve! Here's why

Mon, 15 May 2017 00:00:00 +1200

TL;DR

If Mark and Steve invited you to work with them, what would you say? Exactly.

Long version

Okay, I have to elaborate a bit more about why I’m ridiculously excited about working with Mark and Steve. My first foray into the performance space was at the Velocity Conference in 2009. If you had told me then that someday I’d be working with that tall guy rocking the main stage, I would’ve thanked you for the kind words… while secretly thinking you were nuts. But here I am!

Tammy at International Women's Day Tech Talks in Toronto

And the first time I saw Mark's awesome Perfmap plugin, I knew that it was created by someone who thinks about performance and usability the same way I do... and someone I needed to get to know.

Saying yes to working with Mark and Steve was a no-brainer, of course. Here's why I'm excited not just about working with these guys, but also about working at SpeedCurve.

I spent the past two decades studying how people use the web. As already mentioned, for the past eight years I focused on how web performance intersects with user experience and business metrics. I’ve done everything from studying massive data sets of billions of user experiences to far-out neuroscientific lab studies involving EEG headsets. I even wrote a book about it.

I tend to think of my work as answering three different questions:

How do people use the web?
How do site owners want to understand how people use the web?
How can we build better and better tools that let site owners understand their visitors in ways that are most meaningful to them and their business?

As Chief Experience Officer at SpeedCurve, I get to answer all three of those questions in new and exciting ways. I’ll be marrying two aspects of experience that go together like PB&J:

User experience, as measured and monitored by our synthetic and RUM tools. I’ve said this countless times: Every company is different. Every site is different. Every visitor is different. Every visit is different. I never stop being curious about these differences, and there are always new metrics to create and new things to learn. I love SpeedCurve’s tools and I’m thrilled to have the opportunity to use them every day.

Customer experience, in terms of how our customers use our tools. I’ll be working closely with our customers to understand the specific ways they use SpeedCurve to understand their visitors. I’ll use that understanding to help make our tools – and our own customers’ experience – even better. Full disclosure: While I’ve been involved in a lot of conversations with customers in the past, it’s never been part of my official job description. I’m super excited about this aspect of my new role!

The best tools are driven by listening to what people want. I’m a good listener. Hit me up.

A/B Testing JSONP and the Preloader with RUM

Tue, 02 May 2017 00:00:00 +1200

SpeedCurve is a SPA (Single Page App) so we construct the charts dynamically using JSONP. It works great, but we're always looking for ways to make the dashboards faster. One downside to making requests dynamically is that the browser preloader isn't used. This isn't a factor for later SPA requests, but on the first page view the preloader might still bring some benefits. Or maybe not. We weren't sure, so we ran an A/B test. Long story short, doing the first JSONP request via markup caused charts to render 300 milliseconds faster.

It's easy to run this A/B test using our RUM product, LUX. The first step is to add a User Timing metric that marks when the JSONP response is received. We call this metric OnDataReceived. Some popular browsers don't support User Timing, so we use LUX's shim for mark() and measure() since it works in all browsers:

    function OnDataReceived(jsondata) {
        LUX.measure("OnDataReceived");
        ...

The code above measures the time from NavigationStart to when the JSONP response is processed. All User Timing marks and measures are automatically captured by LUX and displayed in customers' dashboards.

The next step is to split users across two buckets. In the control bucket we continue to fetch the JSONP dynamically. In the test bucket we use markup to make the JSONP request. We use LUX's addData API to segment the RUM data across these buckets. The control bucket has JsonpMarkup=no, while the test bucket has JsonpMarkup=yes. Here's the backend PHP snippet:

    if ( rand(0,1) ) {
        echo "<script>onGo();</script>\n"; // request JSONP dynamically
        echo "<script>LUX.addData('JsonpMarkup', 'no');</script>\n";
    }
    else {
        echo "<script src='/site-data?jsonp=OnDataReceived'></script>\n";
        echo "<script>LUX.addData('JsonpMarkup', 'yes');</script>\n";
    }

Within a few minutes the data shows that OnDataReceived is ~300 milliseconds faster for the JsonpMarkup=yes bucket. We let the A/B test run a few days to confirm that using markup for the initial JSONP request is faster thanks to the browser preloader.

If you make requests dynamically using JavaScript, you might be missing a performance opportunity by letting the preloader do its work. This isn't just true for JSONP requests, it also applies to images. If you're curious about this optimization, run an A/B test to see if making initial requests using markup makes your site faster and your users happier. We'd be happy to set you up with a LUX free trial to make it easy peasy (lemon squeezy).

Improved Favorites

Tue, 18 Apr 2017 00:00:00 +1200

We've improved the "Favorites" dashboard which now lets you build your own charts which:

Combine synthetic tests and LUX (real user monitoring) in one chart.
Choose average, median, or 95th percentile.
Create charts that have multiple metrics.
Select multiple values for a filter, eg, browser = Chrome or Firefox, country = UK or US.
Compare A/B tests in a single chart.

Here's a walkthrough showing you some of the new features:

We've also improved the performance and user experience of the Favorites charts. There are a lot more filter options and each chart loads individually getting charts to you faster.

We'd love your feedback. Let us know what else you'd like to see added to the Favorites dashboard. Here's a few of the things we're considering adding like filmstrips and histograms.

Measuring Blocking Resources

Thu, 02 Feb 2017 00:00:00 +1300

SpeedCurve reports the number of critical blocking resources in the page. These are the resources that block rendering. Since it's important that users see your content as quickly as possible, it's important to know what might be causing your page to render slowly. We recently enhanced the way we measure blocking resources and wanted to share those improvements with our customers as well as the performance community at large.

The main culprits that block rendering are scripts and stylesheets that are loaded synchronously. A great way to avoid this blocking problem is to load your scripts and stylesheets asynchronously. You can do that for scripts by using the async and defer attributes, plus other programmatic techniques. Loading stylesheets asynchronously is less popular but is still possible using techniques like loadCSS.

We recently improved how blocking stylesheets are measured. The first change is to ignore stylesheets loaded via loadCSS; these are asynchronous and thus don't block rendering. Similarly, we ignore stylesheets that are initially loaded with link-rel-preload and later added to the DOM dynamically. Finally, we ignore stylesheets with a "print" media type. If your site uses any of these techniques, then your "Blocking CSS" SpeedCurve metric will decrease.

We also improved how blocking scripts are measured. But before diving into that, I need to explain a subtle difference in how scripts and stylesheets block rendering. Synchronous stylesheets block the entire DOM from rendering. Synchronous scripts, however, only block the elements that follow the SCRIPT tag in the DOM tree. This is illustrated in the figure below where scripts that block rendering are shown in red. Async and defer scripts are always green (non-blocking) no matter where they occur in the DOM. Synchronous scripts, however, block rendering if they occur in the HEAD or in the BODY above any DOM elements that are in the viewport. If synchronous scripts occur below the visible DOM, then they don't block rendering.

This leads to an important optimization in how to measure blocking JS. If a page has a synchronous script at the bottom of the page (below all the visible DOM elements), then it does not block rendering (because all the visible DOM elements precede the SCRIPT tag).

In order to accurately measure blocking scripts, SpeedCurve does more than just track whether scripts are sync or async. We also determine if the script blocks rendering. We do this by determining if the synchronous script occurs before any of the DOM elements in the viewport. If a script is loaded synchronously and occurs before any DOM elements in the viewport, then we count it as "blocking JS". If it occurs after all the viewport's DOM elements, then it is not counted as blocking (even if it's loaded synchronously).

This change may decrease your "Blocking JS" SpeedCurve metric if you have synchronous scripts in the body of the page. It did for us! In the SpeedCurve.com dashboards, all the charts are loaded via synchronous scripts, but all those synchronous scripts are below the visible DOM. So our Blocking JS metric dropped from 4 to 0! This helps us focus on the resources that are most important for improving rendering.

Measuring blocking resources is difficult to do, but it's one of the most important metrics to track if you want to optimize your Start Render time. We're glad we have both of these metrics. We're also glad this code is behind us. Phew!

SpeedCurve RUM: LUX

Tue, 01 Nov 2016 00:00:00 +1300

We're excited to announce SpeedCurve's RUM product, LUX.

The name LUX is a play on "Live User eXperience" and reflects how we've taken a different approach compared to other Real User Monitoring products. SpeedCurve's mission is to help designers and developers create joyous, fast user experiences. To do that, we focus on metrics that do a better job of revealing what the user's experience is really like.

In addition to standard RUM metrics like page load time and total size, LUX includes innovative new metrics that have more to do with the user experience like start render time, number of critical blocking resources, images above the fold, and viewport size. LUX's RUM metrics help you figure out which design and development improvements will make your users happier and your business more successful.

Now that SpeedCurve has both synthetic monitoring and RUM, we can mash those up to give better insights. For example, we compare your synthetic metrics to RUM so you understand how your synthetic performance will translate when staged code is pushed to production. Also, we give you feedback on which URLs, browsers, and geo locations are popular from RUM to help you improve your synthetic test settings to better approximate the real user experience.

You can see SpeedCurve RUM in action by viewing the LUX demo account. The data comes from webpagetest.org where Pat Meenan agreed to inject the LUX snippet. There are four LUX dashboards: Live, Users, Performance, and Design.

The LUX Live dashboard constantly updates to show the users currently visiting your site.

You can hover over a row to highlight all visits for that user's session. Clicking on a row shows the details for that user's visit including Navigation Timing, User Timing, user interaction, and page design metrics.

The LUX Users dashboard shows information about your users including page views broken out by type of user interaction. It also shows the top pages, browsers, viewports, cities, and countries across all your users.

LUX Performance shows the core performance metrics like backend vs frontend time (where everyone should start), User Timing custom metrics (if you have any), and time to first user interaction (often the best reflection of the user's experience). It also contains a great example of mashing up RUM and synthetic for start render and page load time.

The LUX Design dashboard has most of our innovative views of how performance and design impact the user experience. If your start render time is slow, it's good to see how many critical blocking stylesheets and scripts are being served to real users, especially if you have third party tags. Comparing the number of images above-the-fold with the total number of images indicates whether you should be lazy loading images outside of the initial viewport. Similarly, comparing viewport size to document size indicates if you have other content that you should lazy load. LUX also shows the most popular DOM elements that users interact with.

We also make it easy to slice-and-dice the data. Expanding the Dropdown reveals various dimensions that can be used to segment the data. You can choose from the top values that are shown, or enter a value. Wildcards are supported. You can also click on any of the lists of values in the dashboards to filter on that value.

Take LUX for a test drive by going to the LUX demo account. More information, including an FAQ and API, can be found on the LUX features page.

If you'd like to signup for LUX, head over to our pricing page and fill out the signup form. We look forward to helping you create fast, joyous experiences for your users.

Build your own charts

Mon, 10 Oct 2016 00:00:00 +1300

We put a lot of thought into curating a thematic set of dashboards that help you understand the performance of your front-end, but sometimes you just want to play with the data yourself and slice 'n' dice the data in all sorts of different ways. We've added a new "Favorites" dashboard that lets you do just that. You can explore the data and build your own charts, then rearrange them and share them with the team to help demonstrate the performance issues you're focused on right now.

Here's a walkthrough showing you how to slice the available data in different ways:

We'd love your feedback. Let us know what else you'd like to see added to the Favorites dashboard. Here's a few of things we're considering adding:

Select multiple metrics on the same chart
Choose median rather than average along with support for percentiles
Custom metrics
Click through to an individual test for each data point
Filmstrips
Histograms

Track down front-end CPU hogs

Thu, 22 Sep 2016 00:00:00 +1200

Often when monitoring and debugging site performance we focus on network activity and individual resources, but what about the CPU? As more and more sites switch to using large Javascript frameworks and manipulating the page using Javascript, the execution time this code takes and the available CPU can instead become the performance bottleneck.

CPU usage for all Chrome tests

For any of SpeedCurve's Chrome-based tests, including emulated devices, we capture the Chrome Dev Tools timeline. From the browser main thread usage in the timeline we extract how busy the CPU is and what it's spending time on. Is it busy executing Javascript functions or busy laying out elements and painting pixels?

We also measure the CPU usage to different key events in the rendering of the page. SpeedCurve's focus is on the user experience and getting content in front of people as fast as possible, so we show you what the CPU is doing up till the page starts to render. This reflects CPU usage during the browser critical rendering path and can highlight various issues. If there's lots of CPU idle time then you're not delivering your resources efficiently. You want to get the CPU busy nice and early rendering the page, rather than sitting idle waiting for slow resources.

In the test below we see in the first pie chart that the CPU is spending a lot of time on layout up to the start render event, which is quite a different picture from the Fully Loaded CPU usage.

CPU trends and budgets

We've also added trending charts for the CPU time to both the Start Render and Fully Loaded browser events. This allows you to spot any significant changes over time and can highlight both CSS and Javascript changes which are causing CPU issues. In the example below we see a dramatic change in the Layout CPU time for this URL. The browser was being tied up by dodgy Javascript which was causing a huge amount of layout thrashing and reflows. Once fixed the Layout time drops dramatically.

We also support performance budgets for all the CPU metrics allowing you to get email and Slack alerts whenever a threshold is crossed. It's going to be interesting to see what sort of budgets teams put on script execution and layout. Not only can you monitor the size and number of Javascript or CSS scripts but you can ensure it's well written code that's not thrashing the CPU.

Chrome Dev Tools timeline goodness

It's amazing how much performance data is packed into the Chrome Dev Tools timeline and for every single Chrome based test on SpeedCurve we capture the timeline. You can click through to an online timeline viewer at the bottom of the SpeedCurve waterfall chart.

Here's the timeline for the URL above which had a layout thrashing issue, highlighted by the dark red bar between 2-4s. It's amazing to have this level of debugging for every test!

Test agent CPU

It's worth noting that the test agents we currently run at SpeedCurve are fairly low spec and so don't have oodles of available CPU. We do this to help emulate lower spec mobile devices. Over the last year we've seen CPU usage grow and some URLs with a lot of heavy Javascript and CSS are now maxing out the CPUs on our test agents. Although it's fascinating to see well constructed sites (I'm looking at you Smashing) not have this issue at all, with plenty of CPU idle time available as the page renders.

We all need to add CPU usage to our front-end performance toolkits!

We're thinking about introducing new agents with higher CPU specs so that you can test on both a lower spec CPU that represents an emulated device and a high CPU agent that better represents a desktop browser. Sing out in the comments if this is something you're interested in or you've discovered something interesting in your CPU usage charts.

If you're not already using SpeedCurve to monitor your site's performance and track your own CPU hogs, set up your free trial here.

SpeedCurve Waterfall

Tue, 20 Sep 2016 00:00:00 +1200

If you're a performance engineer, then you're familiar with waterfall charts. They are found in browser dev tools as well as other performance services. I use multiple waterfall tools every day, but the waterfall chart I love the most is the one we've built at SpeedCurve:

We've added a number of great features to our waterfall chart. In the screenshot above we see the vertical lines we've added to show Backend, Start Render, and Onload. My favorite feature is the thumbnail "scrubber" at the bottom of the waterfall. As you move your mouse horizontally through the timeline, the scrubber shows the thumbnail that corresponds to that point in time. This makes it easier to correlate the rendering experience with what's happening in the network.

For example, in the waterfall above we see that Start Render occurs at 5208 milliseconds. It's a good thing that we can see the thumbnail at that point in time because the only thing being rendered is a spinner! To find out when the actual page content is displayed, we move the mouse forward in time while watching the scrubber. Eventually we see that the actual content on the page appears at 9139 milliseconds, as shown below. By correlating this thumbnail with the network requests we can track down the cause for this four second delay, most likely the late arrival of a JSON payload containing the search results.

By default we color the network requests to reflect the Content-Type: scripts are orange, images are purple, etc. We also provide options that allow you to use colors that reflect the connection state: DNS lookup (aqua), TCP connection (orange), SSL negotiation (purple), Time-to-First-Byte (green), and content download (blue). You can also choose to have the height of each request reflect the content size. Both of these options are shown here:

I've added a screencast below to give a taste of what these features are like. But the best way to experience these awesome features is to do a free trial. Signup today to scrub your thumbnails and give your users a fast, joyous experience.

PWA Performance

Mon, 12 Sep 2016 00:00:00 +1200

Progressive Web Apps (PWAs) combine the best and newest features of the Web to deliver an experience that rivals native applications on mobile. Even better, they work on desktop, too. In fact, they work everywhere that the Web works! "Ah", you say, "that's not true! They require features that don't exist in all browsers." Because PWAs are "progressive", they can adapt to older browsers to deliver the best experience possible given the features that are available.

Given these winning attributes, it's no surprize that PWAs are the most popular movement in web development today. And while there are already numerous conferences, videos, and evangelists for PWAs, there's less focus on testing and tracking the performance of PWAs. Mark and I discussed this gap in PWA performance. In response, we added some new features to SpeedCurve that, coupled with some existing features, provide a great toolkit for evaluating the performance of PWAs.

PWA Offline Script

To answer the question "What's the key to measuring PWA performance?" let's look at some of the key features of PWAs that have the biggest impact on performance:

responsive - PWAs work across different devices and viewport sizes
progressive - they adapt to different browsers, regardless of missing features
offline - they work offline (after an initial lightweight installation)

The trickiest part is testing the offline experience. SpeedCurve is built on top of WebPageTest, so we use its scripting language to load the PWA while online, and then load it again without any network access. Here's the script using Pokedex as our example PWA:

logData 0
navigate https://www.pokedex.org/
execAndWait document.body.innerHTML=""
logData 1
blockDomainsExcept unused.zz
navigate https://www.pokedex.org/

For the initial load while we're online, we turn off data logging and navigate to the URL. For cleaner filmstrips we clear the body's content between loads. Then we turn logging on and navigate to the URL again, but this time we're offline. The offline experience is created by blocking all domains. This forces the browser to fallback to its offline capabilities.

Responsive, Progressive, & Offline

Let's look at how SpeedCurve measures these key PWA features. Using the PWA offline script shown above, we can verify that Pokedex loads without a network. Looking at the Pokedex test results for Samsung Galaxy S III, we notice something really peculiar. There's a filmstrip, but there's no waterfall chart! What also stands out is that the Start Render time is 0.1 seconds! Pokedex loads instantly because it falls back to its offline experience. This is a web developer's (and user's) dream come true: a fast experience even while offline.

SpeedCurve's Responsive dashboard for Pokedex sheds light on both the PWA's responsive and progressive behavior. Even while offline, the app loads on all browsers except IE 11. The site also behaves in a responsive manner as well. Notice how the left nav only appears on larger viewports. (Note: Since SpeedCurve uses emulation for mobile devices, it's not worth testing iOS since it doesn't support offline.)

Script Templates

Mark added a cool new "script templates" feature in SpeedCurve to make it easier to create this PWA offline script, as well as other important scripts.

In addition to the PWA offline script, we have these other script templates:

Repeat View - See what performance is like when the cache is full.
Block Third Party - This script blocks all third party requests. This is helpful for teams who want to focus on the parts of the page that they control and eliminate variability from external resources.
Don't add "PTST" to user agent - By default, WebPageTest includes the string "PTST" in the User-Agent request header, but that may cause some web services (especially ad networks) to send back a different response. Using this script ensures third parties are behaving the way they do in the real world, but it also removes the ability to filter out the testing in your own analytics.

NOTE: You need a SpeedCurve Enterprise Plan to see the scripting UI. If you don't have an Enterprise Plan you can try the PWA offline script in WebPageTest.

Happy PWA

Thanks to the PWA offline script, you can verify that your PWA loads without any network. SpeedCurve also helps you see that it's progressive across all browsers, and responsive across all viewport sizes. And don't forget about the initial load experience! That's easy to test with SpeedCurve (just enter the URL without any script). It's important to make sure that your PWA fully loads as quickly as possible so that it's ready to work if the user goes offline.

Making sure your PWA is responsive, progressive, and works offline is key. SpeedCurve is a great way to keep track of how your PWA performs and to deliver a fast, enjoyable experience for your users across all devices.

Measuring the User Experience

Tue, 03 May 2016 00:00:00 +1200

SpeedCurve’s sweet spot is the intersection of design and performance - where the user experience lives. Other monitoring services focus on network behavior and the mechanics of the browser. Yet users rarely complain that “the DNS lookups are too slow” or “the load event fired late”. Instead, users get frustrated when they have to wait for the content they care about to appear on the screen.

The key to a good user experience is quickly delivering the critical content.

SpeedCurve helps you do this by focusing on metrics that reflect what the user actually experiences. For web pages, this comes down to when the content gets rendered. In last month’s redesign, we added a new rendering chart. Here’s an example of Smashing’s rendering chart:

To emphasize the importance of rendering, we made this the first chart in the dashboard. The rendering chart includes metrics for Start Render, Speed Index, and Visually Complete. Start Render is the lower bound indicating render time for the first pixel. Visually complete bookends that with a metric for rendering the last pixel. Speed Index sits in-between showing the average render time across all pixels. (Many people were surprised to find out that Speed Index is “expressed in milliseconds”.)

Rendering is important, but not all pixels have equal importance. Instead, most pages have critical design elements (e.g, the call-to-action or hero image). Knowing when this critical content is available is even more important than overall rendering metrics. Measuring critical content is done with Custom Metrics and the User Timing spec. We jazzed up the Custom Metrics chart as part of the recent redesign:

If rendering and critical content get slower, the first place to investigate is the number of critical blocking resources. Since it’s hard for developers to track changes in the number of blocking scripts and stylesheets across releases, we added that as a new metric in SpeedCurve. Here’s the chart of critical blocking resources for NY Times:

We’re excited to find such fertile ground for accelerating how our customers deliver their critical content to users. If you’d like to see these metrics for the user experience on your website, start a free trial today and give SpeedCurve a try.

More emulated devices and improved flexibility

Thu, 24 Mar 2016 00:00:00 +1300

SpeedCurve users may have noticed some changes recently. At the beginning of March we released a major redesign of our Settings UI that gives users more flexibility to get the exact test results they want. The two biggest changes were to the way we emulate devices and to allow different testing configurations for different sites.

Emulated Phones & Tablets

Previously, emulated devices were hidden in the Responsive testing option. Now you can choose the specific phones and tablets you want to emulate. We use the device emulation built into Chrome DevTools so your tests run in the same environment as your local development. SpeedCurve provides nine emulated phones & tablets or you can build your own custom browsers and control settings like viewport size, device pixel ratio, connection speed and even packet rate loss.

Flexible Testing

Previously, geographic regions used average network speeds, but the new version of SpeedCurve allows you to test specific network configurations for different browsers. For example, you can now emulate the experience of a phone on a 2G network in Brazil or a desktop PC on fibre in Germany.

The new SpeedCurve architecture allows you to access all the granularity of WebPageTest via our user interface.

You can also distribute checks more flexibly across your sites and the competitors you are benchmarking. For example, you might currently check your website several times a day but with the new version you can test production and pre-prod hourly to see the impact of each change, and drop your competitor checks to once a day.

We are constantly working on the SpeedCurve codebase and this is the third complete re-write. Our changes are driven by customer feedback. So if you've provided feedback via Github - thank you - we are listening and please keep posting suggestions. We love hearing how we can make SpeedCurve better.

Critical Blocking Resources

Mon, 23 Nov 2015 00:00:00 +1300

At SpeedCurve, we focus on metrics that capture the user experience. A big part of the user experience is when content actually appears in front of the user. Since stylesheets and synchronous scripts are the culprits when it comes to blocking rendering, we've rolled out some new metrics that focus on these critical blocking resources.

The most helpful innovation we made is to highlight the critical blocking stylesheets and synchronous scripts in our waterfall charts. In the following example waterfall chart for ESPN, notice how the critical stylesheets (green) and synchronous scripts (orange) have a red hash pattern. Not surprisingly, the Start Render metric is delayed until after the last of these critical blocking resources is done loading. The "scrubber" at the bottom of the waterfall (showing the screenshot at that point in time) confirms that rendering has been blocked up to this point. Explore an example of a interactive waterfall chart.

Not all scripts and stylesheets necessarily have a big impact on performance, so it's important to be able to tell the blocking requests when trying to improve the user experience. In the waterfall above we see that there's an early script (request #6, "lazysizes.js") that is not marked as a critical resource. That's because this script is loaded asynchronously and thus doesn't block rendering of the page. Since we only want to highlight resources that block rendering, asynchronous scripts are not highlighted.

Similarly, in some cases it's possible to have stylesheets that don't block rendering of the main page. This typically happens with third party widgets that are loaded in an iframe, like the Twitter widget. Since stylesheets in iframes do not block the main page from rendering, they're not marked as critical resources.

In addition to highlighting critical resources in the waterfall, the number of blocking stylesheets and synchronous scripts is shown under Other Metrics.

Once we have a month of historical data for critical requests, we'll add charts to track this metric over time. This new visibility into the number of blocking requests, and which ones they are, will help teams create fast, enjoyable experiences for their users.

The critical rendering path

For more on what the critical rendering path is, why it matters so much and how to optimize asset delivery to ensure you don't block it IIya Grigorik gave a great overview of the topic at Velocity Conf 2014.

User Timing and custom metrics

Thu, 12 Nov 2015 00:00:00 +1300

If you want to improve performance, you must start by measuring performance. But what should you measure?

Across the performance industry, the metric that's used the most is "page load time" (i.e, "window.onload" or "document complete"). Page load time was pretty good at approximating the user experience in the days of Web 1.0 when pages were simpler and each user action loaded a new web page (multi-page websites). In the days of Web 2.0 and single-page apps, page load time no longer correlates well with what the user sees. A great illustration is found by comparing Gmail to Amazon.

In the last few years some better alternatives to page load time have gained popularity, such as start render time and Speed Index. But these metrics suffer from the same major drawback as page load time: they are ignorant of what content the user is most interested in on the page.

Any performance metric that values all the content the same is not a good metric.

Users don't give equal value to everything in the page. Instead, users typically focus on one or more critical design elements in the page, such as a product image or navbar. In searching for a good performance metric, ideally we would find one that measures how long the user waits before seeing this critical content. Since browsers don't know which content is the most important, it's necessary for website owners to put these performance metrics in place. The way to do this is to create custom metrics with User Timing.

User Timing Spec

The W3C User Timing spec provides an API for developers to add custom metrics to their web apps. This is done via two main functions: performance.mark and performance.measure.

performance.mark records the time (in milliseconds) since navigationStart
performance.measure records the delta between two marks

There are other User Timing APIs, but mark and measure are the main functions. Once your page is chock full of marks and measures, the question arises as to how to actually collect these metrics so you can track them. Thankfully, because User Timing is an official W3C specification, many metrics services automatically extract and report these custom metrics in their dashboards. This is true for SOASTA's mPulse, WebPageTest, and SpeedCurve.

Sample Custom Metrics

While the User Timing API is simple, it can sometimes be difficult to know where and when to capture these marks and measures. This is due to the complexity of the browser's inner workings, primarily the way that stylesheets and synchronous scripts block parsing and rendering. Let's look at a few examples of likely custom metrics to gain a better understanding.

A quick note about browser compatibility: User Timing is available in most browsers except Safari. In the examples below we use the native API, but in your live code you'll need to check for compatibility and either use a shim or wrapper. A good polyfill is available from Pat Meenan.

example 1: stylesheets done blocking

Loading a stylesheet blocks the entire page from rendering. This is true in all browsers as well as for stylesheets that are loaded dynamically. Therefore, knowing when stylesheets are done blocking is a good metric for understanding why rendering might start later than desired. Here's sample code for capturing this custom metric:

<link rel="stylesheet" href="/sheet1.css">
<link rel="stylesheet" href="/sheet4.css">
<script>
performance.mark("stylesheets done blocking");
</script>

The key to this snippet is making sure that it's placed below all the stylesheet LINK tags in the page. That's a reasonable requirement since most stylesheets are listed at the top of the page. While simple in appearance, some complex performance know-how is behind this snippet:

Inline scripts are not executed until all previous stylesheets are downloaded, parsed, and applied to the page. Therefore, placing the inline script after the LINK tags ensures that the mark is set after all the blocking CSS has been processed.
It's possible that a stylesheet might include @import which would pull in other stylesheets, for example, sheet1.css might cause sheet2.css and sheet3.css to be downloaded, parsed, and applied. Therefore, simply tracking a stylesheet's load time (with Resource Timing for example) isn't enough. Doing so would produce a measurement that was too short and underestimated the impact of stylesheet blocking.

In order to produce a good (fast) user experience, it's important to understand what might be blocking your web app's rendering. The likely culprits are stylesheets and synchronous scripts, but often it's hard to disambiguate which is responsible for delayed rendering. This "stylesheets done blocking" custom metric is handy because if rendering starts well after the "stylesheets done blocking" metric, then the investigation should focus on synchronous scripts, which leads to our next example.

example 2: scripts done blocking

Synchronous scripts block rendering for all following DOM elements, so knowing when script blocking is finished is also important. Here's a snippet for capturing this measurement:

<script src="a.js"></script>
<script src="b.js" async></script>
<script>
performance.mark("scripts done blocking");
</script>

The inline script (which calls performance.mark) is guaranteed to execute after script a.js has been downloaded, parsed, and executed. On the other hand, the mark will be recorded before b.js is done loading because b.js has the async attribute. This behavior matches our goal of measuring script blocking because a.js blocks rendering, but b.js doesn't.

Capturing an accurate "scripts done blocking" metric is a bit more challenging than its stylesheet counterpart. The reason is that, similar to stylesheets, the User Timing mark must be placed below all synchronous scripts. This can be difficult since websites frequently sprinkle scripts throughout the page. If that's the case, the mark measurement may occur after earlier DOM elements have rendered, and thus it doesn't accurately measure when rendering starts.

Therefore, this example is only useful for pages that have their synchronous scripts in the HEAD. Even with this limitation, this custom metric can be very useful for pages that have large scripts where the blocking time is much longer than the download time due to parsing and execution. When looking at waterfalls for these types of pages, all the scripts finish downloading but there's still a long gap before rendering starts. With a "scripts done blocking" mark, it would be possible to identify long parse and execute times as the culprit for render blocking.

example 3: fonts loaded

If your site uses custom fonts, it's important to know that some browsers won't render the text that use those fonts until after the font file is downloaded. In other browsers the text may be rendered in a system default font and then re-rendered ("flash"ed) when the custom font is loaded. Therefore, a "fonts loaded" custom metric would be valuable to indicate when text elements were rendered.

Currently, there is not an "onload" event for fonts. Nevertheless, there are techniques for observing fonts and determining when they load, for example, Font Loading Revisited with Font Events by Scott Jehl from the Filament Group as well as Web Font Loader used by Typekit and Google Fonts. You could use these to track when your fonts are done loading, and include a call to performance.mark to set the "fonts loaded" custom metric at that time.

example 4: hero images

A "hero image" is often the most important element in the page when it comes to user experience. The Hero Image Custom Metrics article describes an innovative technique for measuring when images are rendered. The snippet looks like this:

<img src="hero.jpg" onload="performance.clearMarks('img displayed'); performance.mark('img displayed');">
<script>
performance.clearMarks("img displayed");
performance.mark("img displayed");
</script>

The innovation is to take the greater of the image onload mark and the inline script mark. This accurately captures when the image is actually rendered. The image onload mark reflects the render time when the image download takes longer than any blocking resources. The inline script mark reflects the render time when the image downloads quickly but blocking resources prevent it from being rendered. In many cases, the most critical content in a page involves images, so this snippet is useful for a wide variety of custom metrics.

example 5: paragraph of text

Surprisingly, one of the most challenging custom metrics is determining when text is displayed. In addition to being blocked by stylesheets and synchronous scripts, text elements (P, SPAN, LI, DIV, etc.) can also be blocked from rendering if they use a custom font that takes a long time to download.

If the text element does not use a custom font, then the time at which it renders is captured by simply putting an inline script after the text element:

<p>This is the call to action text element.</p>
<script>
performance.mark("text displayed");
</script>

If the text element DOES use a custom font, then the render time is the maximum of the inline script's mark and the font loading mark (see previous example). The key here is to use a pattern similar to the hero image where we record the "text displayed" mark twice: once in the font-loading snippet and again in the inline script after the text element. Additionally, we clear any previous marks of the same name before setting the new mark, thus ensuring that we get the maximum of the two values which properly reflects when the text is displayed.

example 6: Single Page Apps

One of the drawbacks of relying on page load time as a metric is that it doesn't measure user actions inside a single page app (SPA) where window.onload only fires once but the user might perform multiple actions, each of which should be measured. Custom metrics solve this problem, but it requires taking on the added responsibility of creating an appropriate start time.

In order to show some sample code, lets assume we have a SPA that has an "Update" button. When clicked, an XHR or JSON request is launched to fetch new data. That returned data is passed to the updateData() function which changes the DOM. Here's sample code that measures this SPA user action:

<input type=button value="Update" onclick="fetchData()">

<script>
function fetchData() {
    performance.clearMarks("start update");
    performance.mark("start update");

    // Do an XHR or JSON request that calls updateData() with the new data.
}

function updateData(data) {
    // Update the DOM with the new data.

    performance.clearMarks("finish update");
    performance.mark("finish update");
    perforance.measure("update data", "start update", "finish update");
}
</script>

Notice that the "start update" mark is the first JavaScript in the SPA action, and "finish update" is the last JavaScript executed in the SPA action. This ensures that the entire SPA action is measured, including downloading the XHR or JSON and updating the DOM. Also notice that we used performance.measure for this first time. That's because we want to get the delta relative to a start time other than navigationStart.

Custom Metrics in SpeedCurve

A big benefit of building custom metrics with User Timing is that many performance metric services automatically extract the User Timing marks and measures to show in your performance dashboards. WebPageTest does this, and since SpeedCurve is built on top of WebPageTest we do it as well.

You can see your custom metrics in SpeedCurve by giving them a name in your Settings. In the speedcurve.com website, we use a custom metric called "heromark" to measure when the hero image (typically the chart at the top of the page) is displayed. We can give it a prettier name, "Hero Mark", using the form found under Settings:

You can add multiple custom metrics to SpeedCurve Settings. Since some pages contain dozens of marks and measures, including ones set by third party scripts, this step ensures that only the most important custom metrics are shown.

Custom metrics are visible on various SpeedCurve dashboards. For example, the Site dashboard shows a trendline of our "Hero Mark" custom metric over time.

It's extremely valuable to see custom metrics on a waterfall chart. This allows you to "debug" if your custom metric is measuring the right thing. For example, in the waterfall below for speedcurve.com, we can see that the "Hero Mark" custom metric is blocked until all the preceding scripts (orange) and stylesheets (green) are done downloading. Using the thumbnails at the bottom confirms that this is when the hero image (the chart) actually gets displayed.

No one knows your website as well as you do. Therefore, it's important that you add custom metrics that measure the content that you and your users care about most. This allows you to track the speed of what your user's are experiencing in both synthetic and RUM.

If you're not already using SpeedCurve to create custom metrics and monitor your site's performance, set up your free trial here.

Performance budgets in action

Tue, 20 Oct 2015 00:00:00 +1300

Performance budgets are an important tool for ensuring your site is delivering a great user experience. Steve first experienced performance budgets while Head Performance Engineer at Google. The practice of using budgets to track performance took off with Tim Kadlec's blog post Setting a Performance Budget. The idea is to identify your performance goals and track the metrics that help you achieve your goals.

At SpeedCurve, we give performance budgets first-class status by tracking them in the Site dashboard. Here's an example of tracking a budget for image size.

Before setting your performance budgets, you first have to be monitoring your user experience. Only then can you set budgets and thresholds for improving your baseline user experience. This also allows you to quantify the improvements you're making and share success stories across the organization like "We just improved start render by 32% by reducing image requests to half the budgeted amount".

What should my budget be?

There are various ways to approach setting a performance budget depending on the nature of your team, and what's going to make the most sense to them and motivate change:

Be faster than your competition. Benchmark a variety of metrics and choose a target of 20% faster or less resources.
Be faster than what’s in production right now. This is great when doing a responsive redesign. Fork your code, rip out the old cruft, set new aggressive budgets and watch for alerts as you add new components and content back in.
Adopt Human Computer Interface (HCI) guidelines such as those described in Response Times: The 3 Important Limits. Aim for rendering the first meaningful content in under 1000ms.
Base your budget on user research and design principles, such as Better performance through better design. What are your users actually trying to achieve and how long will they wait?

SpeedCurve's Benchmark dashboard is a great way to see how you compare to your competitors for setting budgets. In the chart below, we can see why Smashing Magazine is so much faster - they're simply delivering a lot less JavaScript. If this was your site you could pick a metric like JS requests, set a lower budget and see if you can beat the competition!

Not all metrics are equal

It’s vitally important you evaluate which metrics you’ll budget against. For a long time now we’ve known that metrics like the page onload event are terrible indicators of user experience and yet many performance tools and organisations still use it as a key indicator of performance.

At SpeedCurve we often audit sites and it’s surprising how even very large sites can be misled into delivering a very poor user experience by focusing on optimising for the wrong metric or taking a performance guideline too literally.

The only metrics you should care about and champion across your organization are the ones that represent what a user is actually seeing as the page renders. Other metrics can be useful secondary measures to track the changing nature of your site but don't necessarily correlate to what makes a good user experience. Your users don't care how your website is constructed. What they really care about is how long it takes before they can engage with your awesome content. You need to focus on letting them see that content as quickly as possible!

Herein lies the problem, the majority of metrics are not visual metrics based on what a user is actually seeing but rather events that reflect the processes running in the browser. These two sets of metrics can paint very different pictures and it’s the reason SpeedCurve is built on WebPageTest the leading open source performance tool.

WebPageTest records video and creates filmstrips of how a page loads in a real web browser leading to unique metrics based on visual analysis from a users perspective rather than what a browser thinks it’s doing. It’s an independent measure and provides two unique metrics we can budget against, start render and speed index. These two metrics are a far better representation of the user experience than any of the built-in browser events.

Better yet define your own custom metrics

While start render and speed index are great, ultimately the best metric is the one you define yourself. Like Twitter’s time to first tweet you should be adding a custom metric that represents when the user can actually view and interact with your content. Custom metrics are easy to add with user timing marks and SpeedCurve automatically picks up any marks added to your site without additional configuration. There can be challenges around instrumenting assets like images though and Steve recently discussed these in hero image custom metrics.

Recommended metrics for performance budgets

In order of preference, here are the metrics we recommend you benchmark your user experience against:

A single UX based custom metric for each template. E.g., the hero image on a product page.
Speed Index. Paul Irish recommends aiming for under 1000.
Start render under 1000ms.

Then for each of these metrics pick a budget, set alerts and monitor it as your codebase evolves and (hopefully) improves. You can also add secondary budgets like the size and number of requests to help correlate the construction of your code base with the user experience but the three above are the ones you should really focus on.

Performance budgets in action

See how a SpeedCurve budget alert led Zillow to discover a bug that increased its image size on mobile by 4x in Bigger, Faster, and More Engaging while on a Budget. Interestingly the increase in images didn’t have a noticeable effect on Speed Index and the user experience. Does this mean Zillow could be delivering even more content than it does now as long as the initial content in the viewport renders just as quick? Breaking your performance budgets can lead to really interesting insights on how users interact with your content.

Vox Media, makers of The Verge and SB Nation, benchmarked themselves against the competition and then declared performance bankruptcy. This is a great way to engage with your community and be transparent about the challenges of delivering a rich and fast user experience.

It's oftentimes hard to give a direct answer when someone asks you, "how fast is your site?" so we're educating our team on how we quantify performance at Vox by explaining what each metric means and how they interplay with one another.
Dan Chilton - Front-End Engineering Director, Vox Media

Etsy have also done a great job of publicly reporting on performance and meeting the budgets they set for themselves.

Performance budgets are a great way to ensure you deliver an enjoyable, fast user experience. But they only work if you pick the right metrics and set the right thresholds. It's best to focus on metrics that more accurately measure the user experience, such as custom metrics, Speed Index, and start render. All of these metrics are available in SpeedCurve. We focus on measuring what users actually see, and tracking that so you can deliver it even faster.

Visual diffs on every deploy

Mon, 12 Oct 2015 00:00:00 +1300

SpeedCurve now provides a visual diff of every deploy. A full resolution PNG is captured for each URL and each pixel is diffed with the previous deploy allowing you to easily spot any visual changes you may or may not have expected.

The key to practising safe continuous deployment is to have a robust set of tools that give you immediate feedback on how your code has changed between deploys and its effect on the user experience. It's often very hard to spot all the visual changes in each deploy, especially in fast moving teams where a lot of the focus is on unit tests and other automated pass/fail systems. Visual diffs bring an increased level of tracking and confidence when you're able to compare any two deploys and see exactly what has visually changed.

We do a visual diff every time you click "Test Now" on the Deploy dashboard or you use the SpeedCurve API to trigger a round of deploy testing. Integrating with the Deploy API is super easy and provides a robust set of metrics and before/after screenshots, visual diffs, waterfall charts, filmstrips and videos for each deploy. You can then compare any two deploys to see exactly what's changed over time.

Spot the difference

Sometimes when just scanning before/after screenshots themselves without a visual diff it can be very hard to spot subtle changes. In the Tommy John example above just one extra item has been added to the main navigation. Do you think you would have spotted it without the visual diff giving you a clue as to what has changed? The visual diff really highlights the pixel level changes and also shows the effect on other nav items when a new item is added.

Quickly spot changes in grid and CSS alignment. Did you mean to move it or change a column width? With pixel by pixel matching it's easy to see exactly what's changed in your layout.

Sometimes you want to see what hasn't changed as well as what has. Here all the foreground elements in white remain in exactly the same place while the background which has been swapped out becomes bright red, really drawing your attention immediately to what's changed.

Surprisingly you can also spot the really small stuff when highlighted in red. Visual diffs are sensitive enough to pick up even single character changes helping you focus in on every single pixel that has changed from one deploy to another.

We also provide side by side filmstrips and video comparisons which are fantastic not only for reviewing the visual differences but also putting in presentations so everyone across the organization can see exactly how long it takes to render the page from a user's perspective.

Make continuous deployment safe

Google uses visual diffs to spot release issues and make continuous deployment a safer practice. Brett Slatkin gave a great presentation at Velocity on the sorts of unexpected bugs you can discover with visual diffs. Check out the video of Brett's talk and learn more about how visual diffs help you track changes to what you're delivering to users.

Multiple teams and multiple users

Wed, 30 Sep 2015 00:00:00 +1300

This week we added support for organizations with multiple teams and multiple users.

One of the toughest challenges was simply working out what to call the different layers within the SpeedCurve app. Developers can spend years buried inside a data model but at the end of the day the UI has to be intuitive and easy to use! I hope we’ve done that and if not we’d love your feedback.

SpeedCurve now has a new “organization” and “team” structure that provides a great deal of flexibility when setting up a new account.

Larger organisations or agencies can now add multiple teams to their Enterprise plan. Billing and budgets are held at the organization level making it easy to manage costs while giving new teams access to SpeedCurve monitoring for their projects.

If you're an agency you can swap the idea of internal teams for clients. This lets you have an isolated set of dashboards for each client while easily swapping between them using the new menu if you belong to the organization. Each client only has access to their dashboards, while you have the ability to browse across dashboards.

Multiple users can be invited to join a team and you control their view or edit permissions for any of the SpeedCurve dashboards. Users can belong to multiple teams and even multiple organizations.

We also recently added new dashboard sharing links so you can either send someone a read-only private url (without them having to login) or invite a user to join your team.

Here’s a walk-through of the new account structure and features. If you're on an Enterprise plan you have access to these features now, otherwise get in touch to upgrade your existing account.

Up next we’re doing a complete redesign of the Settings UI which will provide a great deal more granularity for each Site being monitored.

UX Focus for Waterfalls and Third Parties

Tue, 23 Jun 2015 00:00:00 +1200

At SpeedCurve, we want to help designers and developers have better insight into the user experience they're delivering. For websites, this means understanding when the critical parts of the page render and what might be blocking rendering.

We've redesigned our waterfall chart to really highlight the relationship between the assets on the page and their affect on the user experience. Now as you move you mouse over the waterfall chart we show you exactly what a user is seeing at that millsecond while the page loads. This makes it much easier to identify any Javascript or CSS that might be blocking the page from rendering. I recently used this new combined waterfall and filmstrip view to identify a common issue with hero images being delayed.

If rendering is blocked, it's important to figure out the cause. This isn't always easy, especially with the amount of third party content on websites today. Often the scripts and stylesheets that come with ads, analytics, and social widgets aren't delivered in a high performant way and can block the main content on the page from ever being seen by users. To help identify this possibility, you can use the Start Render line in the Third Party waterfall chart.

The Third Party waterfall chart bundles HTTP requests into groups representing first and third parties. The grouping logic is based on technology developed at SpeedCurve. We draw a vertical line representing the Start Render point on top of the waterfall. Any third parties to the left on the line have the potential of blocking rendering. You can click on each third party to see more detail and go to the full waterfall chart to investigate the filmstrip thumbnails.

Waterfall charts are great. Adding the ability to see when rendering occurs and how it's affected by third parties makes them even an even better tool for designers and developers to create great experiences for their users.

The language and metrics of UX evolve at Velocity 2015

Thu, 11 Jun 2015 00:00:00 +1200

Originally published on the O'Reilly Radar Blog

I’ve attended four O’Reilly Velocity conferences over the last year, and I was struck by a notable shift in the conversations at Velocity in Santa Clara, Calif. Many speakers and attendees have started to change their language and describe the experience of their websites and apps from the user’s perspective.

The balance has shifted from just talking about how fast or reliable a particular system is to the overall experience a user has when they interact with and experience a product. Many people are now looking at themselves from the outside in and developing more empathy for their users. The words “user” and “user experience” were mentioned again and again by speakers.

Here are recent talks from Velocity and other events that highlight this shift to UX concerns.

The next billion users

At Velocity 2015 in Santa Clara, Bruce Lawson described the dramatic growth of connected users in Asia and Africa, while highlighting the challenge faced when trying to deliver great experiences to a broad range of devices. Companies looking to these regions for their next billion users need to keep in mind that these users are on underpowered phones over slow networks with high data costs.

Reaching everyone

The techniques required to reach such a broad audience were covered by Tim Kadlec at Velocity 2015 in Santa Clara, who took us through the process of creating a high-performant responsive site that can effectively reach a diverse global audience. Here’s Kadlec’s related talk from Velocity 2014 in New York:

The path to performance

Successful performance begins far earlier than development. So, how do you get your entire team excited by it? At Velocity 2015 in Santa Clara, Katie Kovalcin discussed everyone’s role in performance and how to get all teammates caring.

Design + performance

Steve Souders really captured the growing crossover of design and performance cultures within organizations and the desperate need to move from purely technical instrumentation to a focus on custom metrics that capture what the user actually sees as the page loads. Below you’ll find Souders’ related talk from Fluent 2015:

Visualizing performance data in engaging ways

In the talk I gave, I focused on how performance data can be visualized in ways that engage the wider team, ensuring that everyone across the organization understands the impact of their work on the user experience:

Coming from a design background, I’m happy discussions at Velocity are focusing on the importance of viewing performance from the user’s perspective. The key is ensuring that the design and development process keeps the user and their experience of a product as a constant focus.

Operations has users, too

Running and maintaining high-performance, highly-available sites and apps necessitates a near obsession with monitoring and metrics. With operational metrics come a host of tools, and the current state of these tools is unfortunately well behind the end-user customer experience. Two talks focused on the need for a UX-focused approach for operations tools, one from the perspective of UX designer Tim Sheiner, the other from Rob Treat, who uses ops tools on a daily basis. Both stressed the importance of understanding the goals, tasks, and mindset of operations teams when building tools, notably for handling stressful situations like site outages.

Velocity was one of the early places where DevOps originated, bringing developers and ops folks closer together. Last week, we witnessed an additional convergence of developers, ops, and designers. It’s exciting that Velocity is helping these siloed groups find ways to share with and learn from each other. Much of this thinking comes from Lara Callender Hogan’s book Designing for Performance. If you want to produce amazing, fast, enjoyable user experiences, I recommend you read Hogan’s book and review the sessions highlighted above. And keep your focus on the user.

Responsive Design Dashboard

Wed, 29 Apr 2015 00:00:00 +1200

The dramatic growth of mobile traffic has created an urgency for websites to adapt to different screen sizes. Responsive design is the preferred technique for making web content adapt to the device on which it's viewed. This approach ensures that the appropriate content is rendered in an appropriate way on phones, tables, laptops, and any other screen being used.

A corollary to responsive design is the need for performance. Delivering content that's customized for a particular device isn't enough if its delivery is slow and frustrating. Indeed, last week Google announced that websites need to deliver their design appropriately and quickly on mobile devices or they will be demoted in Google's search results.

How can you know if your website matches these standards for adaptive content and speed? SpeedCurve's Responsive Design Dashboard answers both questions in one view.

The Responsive Design Dashbard shows the URL's filmstrip across different viewport sizes: 320x568, 568x320, 768x1024, 1024x768, 1280x768, and 1366x768. Viewing these thumbnails over time reveals how quickly the content is rendered. Similarly, you can see how the site's design adapts to different screen sizes.

The Responsive Design Dashboard is created using Chrome's mobile browser emulation feature. In addition to specifying the viewport size, the User-Agent string is also adapted to correspond to the emulated device's default characteristics. (Soon we'll adapt the network settings, as well.)

Asset Breakpoints

The filmstrips show how the website adapts to different visual breakpoints. It's also important to think about asset breakpoints - how the size and number of requests adapts based on the screen size. For example, we know we don't want to send huge desktop images to smaller screens.

The Responsive Design Dashboard shows asset breakpoints by revealing information about HTTP requests at each viewport size. The screenshot below, for example, paints a picture that we like to see where smaller screens receive fewer bytes, while larger screens get a heavier experience.

The Responsive Design Dashboard perfectly captures what SpeedCurve is about: the intersection of design and performance. Understanding how a design adapts across viewports to tracking kilobytes by Content-Type. We're having fun bringing you the information you need to create great user experiences.

Mark+Steve, Performance+Design

Tue, 24 Mar 2015 00:00:00 +1300

I'm excited to announce that I've joined SpeedCurve!

When SpeedCurve was just a twinkle in Mark's eye, he contacted me about the concept and I encouraged him that a commercial version of WebPageTest was needed. When I saw the early versions of SpeedCurve, I was blown away. Mark presents traditional performance data in a way that is more compelling, revealing his strong design background.

Mark has pioneered this new territory where performance and design overlap. It's exciting to say "overlap". Many times there's little interaction between designers and performance engineers. When there is interaction, it can feel adversarial with no one wanting to give any ground. And yet, designers and performance engineers are after the same thing: creating a great user experience!

Design and performance are connected, like the yin and yang. They aren't opposing forces, but instead complement each other. Users want an experience that is rich and fast. The trick is figuring out how to do that. That's where SpeedCurve comes in.

SpeedCurve focuses on measuring the interplay between design and performance. Rather than providing typical performance metrics that focus on resource downloads, SpeedCurve focuses on metrics and visualizations that correspond more closely to the user experience, such as:

the responsive design dashboard that shows pages rendering in different viewports, orientations, and connection speeds
custom metrics that measure what you decide is critical to your users
the impact of third party content on your site
whether you're sticking to your performance budget

I've been working on performance for more than a decade and have witnessed (and instigated) a lot of changes. I'm excited about this movement to bring design & performance closer together. In the end it'll produce what matters the most: a great web experience for users.

Mark and I are a good team to work on this - with his background heavy in design and mine in performance. Today we just re-launched the site with a major design overhaul and many new features. (Plus a new pay-as-you-go pricing plan that actually lowers costs for most of our existing customers!) We hope you'll give SpeedCurve a try, and that you'll find it useful for understanding what your users are experiencing and how to make that experience richer and faster.

Steve and Mark at the SpeedCurve re-launch in New Zealand.

Velocity: Better performance through better design

Mon, 17 Nov 2014 00:00:00 +1300

Improve web performance by improving your design process… it needs to be iterative, mindful, principled and visual.

At my third Velocity conference for the year (this time in beautiful Barcelona) my keynote presentation explored the ways in which a thoughtfully developed design process can lead to higher functioning teams and better web performance.

I start out by suggesting that before a project is launched the team must sit down and design the process that they are going to use to reach a solution. This process needs to be guided by a project specific set of principles, higher level than requirements, identifying the core ‘values’ and priorities of the project. A performance budget should be represented in one of the principles and will guide the team as they navigate through each iteration.

Small multi disciplinary teams are key to executing this better designed approach. An agile style of working results, one which no longer excludes designers but has the entire team - designer, developer, researcher and the client, revolving around a prototype. Iteration after iteration is refined through the team effort.

Once again I talk about the importance of effective visualization for engaging the entire team and beyond, to the wider organization.

What tactics do you use in your organization to improve your design process?

How is performance incorporated into discussions within your project teams?

Are you working in Agile and are designers part of that process?

What metric are you choosing to share with the wider organization around your project - and how?

Velocity: A better waterfall chart

Tue, 16 Sep 2014 00:00:00 +1200

The way we visualize performance data can have an impact on how we interpret and communicate performance issues within our teams.

In this talk from Velocity New York I explored the importance of data visualization and presented some of my own explorations into re-imagining the classic waterfall chart which is the mainstay of front-end performance analysis.

Skip to 15:30 if you just want to see the data visualization experiments, and you can play with them directly on lab.speedcurve.com

One of these experiments was also turned into a performance heatmap bookmarklet.

To date waterfall charts have been a mainstay in visualizing web performance, giving us details on each asset and how it cumulatively effects the load time of a web page. But there are many ways of visualizing performance: histograms, flame charts, heatmaps, frequency tails, and colony graphs to name just a few.

I explored the strengths and weaknesses of various types of charts and how well suited they are at answering a range of performance questions from both the perspective of a manager looking for quick high level answers and a developer needed to dig into the details. It’s important to start with the questions about performance first and then find the tools, metrics, and visualizations to best answer a question rather than inferring problems and answers from an existing metric.

Often charts are rendered along just two dimensions, which can limit the amount of information displayed and hide important detail. I presented several new visualizations that introduce depth, animation, and interaction to reveal performance bottlenecks and insights at both a holistic and detailed level.

These new visualizations make the most of modern browser techniques allowing greater interaction and demonstrating just how powerful the modern browser has become at visualizing it’s own performance.

Let me know which visualizations you enjoyed the most and I'll add them to SpeedCurve!

Velocity: Responsive in the wild

Wed, 25 Jun 2014 00:00:00 +1200

I was lucky enough to give a Lighting Demo at Velocity Conference in Santa Clara. The focus was on research I conducted into 250 responsive websites and how well optimized for performance they were.

Methodology

250 responsive websites were loaded at 17 different viewport widths between 320px and 1920px using WebPagetest. The number of requests and size of assets were captured for each width and used to build a visulization that revealed the cross-section of a site's performance breakpoints rather than it's visual breakpoints.

Results

Averages can hide the patterns in data so after visually sorting and grouping the cross sections of the sites we see that there are distinct patterns and approaches used by sites to tune their assets for performance at different widths.

Faster EC2 testing agents

Sun, 01 Jun 2014 00:00:00 +1200

As of the 1st of June SpeedCurve has switched to using faster testing agents at Amazon EC2 data centers. As web pages become more Javascript and resource heavy I've noticed more and more pages max out the CPU while performance testing.

SpeedCurve now uses EC2 m1.medium instances which have 3.75GB of RAM with faster CPU and network performance. For CPU heavy pages you should see a drop in overall load time. The connection speed for testing agents is still throttled to an average US connection speed so simple pages may not improve as much.

Overall this should provide more consistent results that are more on par with what an average user of your website might experience.

Do note that SpeedCurve's strength is competitive benchmarking and detailed build analysis. If you're after actual on the ground performance numbers then we suggest combining SpeedCurve with Real User Monitoring.

Speed Index now available

Wed, 28 May 2014 00:00:00 +1200

Speed Index is now available on SpeedCurve. Choose "SpeedIndex" from the top right of the main graphs.

Speed Index is an important measure of a user's experience as the page loads. It's based on video analysis of how the page loads over time. Here's a great video from Paul Irish introducing Speed Index at the recent Front End Ops Conference.

IE11 and average speed update

Tue, 01 Apr 2014 00:00:00 +1300

To keep inline with browser trends and average connection speeds SpeedCurve will begin updating the test setting every quarter.

For Q2 2014 we're switching to IE11 which is now the most popular version of IE according to Akamai and StatsCounter and the average connection speed has been updated to 9.8Mbps download, 2.5Mbps upload with a 10ms latency. The 9.8Mbps download speed is based on the latest State of the Internet report from Akamai.

SpeedCurve wins Webstock BNZ Start-Up Alley

Thu, 23 Jan 2014 00:00:00 +1300

I'm stoked that SpeedCurve has been chosen as one of two winners in the 2014 Webstock Start-Up Alley. I'll be using the 10k prize money and two flights to the US to attend Velocity Conf in San Fran in June and New York in September.

Performance tips for building responsive sites

Thu, 26 Dec 2013 00:00:00 +1300

The following article was originally published in the 2013 Performance Calendar. There's 31 great articles to explore in the calendar including Steve Souders's browser wishlist and Tim Kadlec's take on what it takes to create a performance culture.

----

Responsive Web Design (RWD) is now a well established technique yet it’s adoption is still surprisingly low. Guy Podjarny’s recent research shows that only one in eight websites is responsive. Often when responsive sites are designed the approach is primarily from a visual design perspective and teams struggle with the complexity of changing design and navigation patterns let alone any performance concerns, which become secondary. This can lead to serving the same sized assets to all browser widths resulting in a responsive site that is visually scaled but not performance optimised. On average responsive sites at 320 pixels wide are only 8% small than at 1600 pixels wide. Unsurprisingly these performance concerns and a lack of established techniques have made some hesitate when considering RWD adoption.

Responsive page sizes across different screen resolutions. Source: Guy Podjarny

However, we do now have a full set of techniques to effectively deliver highly performative sites that not only visually scale across devices but also deliver code and assets tuned to the width of a device. Sites need to be built utilising these techniques from a mobile first (and performance first) perspective.

Here are three key techniques that you can use to optimise performance using mobile as your starting point. Let’s tackle these in order of the page load…

Inline the initial experience (within 14k if possible)

First impressions do count. While it’s easy to throw jQuery, a framework like Bootstrap/Foundation and web fonts on a page you trade ease of use for performance. It’s increasingly a mobile world and your main aim should be to engage users as quickly as possible.

Ilya Grigorik’s recent Velocity conference presentation explored the browser's critical rendering path and highlighted exactly what it takes to deliver a mobile page within one second. Surprisingly you’ve got just one HTTP request and only 14k of HTML to play with to get content in front of users. One of the only ways to achieve this is by inlining any “above the fold” CSS and JS required to render the first screen of content. Recently Grigorik has been championing this approach and the Google Pagespeed Insight rules have been updated to reflect this best practice with recommendations on how to reduce the size of "above the fold" content.

Control and branch the loading of CSS and JS

There are plenty of automated front-end optimisation tools available like modpagespeed and CloudFlare but being a bit old-school and craft-orientated I prefer the ability to control the load order and optimise not only the individual assets but their sequencing as well.

Often responsive sites combine the desktop and mobile CSS and JS into one set of files but this results in delivering unnecessary code to the width you’re viewing. You can optimise what’s delivered by using javascript to detect the width of the page and then request specific styles and javascript appropriate to that width. There might even be whole features or different navigation patterns introduced or removed at different widths to optimize the size of the user experience.

Github write mobile specific CSS and JS resulting in much smaller and faster pages. Source: Github Blog

Here’s a few libraries you can use to enable the decision making and branching of assets as the page loads: eCSSential, Interchange, and good old Modernizer.

Shrink and art direct your images

Images often make up the largest set of assets on a page. There are two key considerations here - serving the right image to the right width and being able to art direct the image for smaller screen sizes, i.e. crop into the focal point of an image for smaller widths.

If possible eliminate an image altogether by using an icon font for icons and small graphics. Alternatively use SVG images for logos and other simple graphics. Both these techniques use vector data rather than pixels so they are small in file size and scale efficiently across all sizes including high resolution retina screens.

However when it comes to photos you need to stick with pixel based formats and this requires switching between different sized images to save bandwidth. Picturefill is an interim polyfill for responsive images allowing you to curate which image gets served to which width.

Often image generation can be a real chore and rather than manually outputting images in different formats and sizes your backend or CMS should be doing the hard work for you. CDN Connect is a promising new service that allows you to transform a single high quality image into many different formats and has an impressive range of methods to crop, filter and manipulate images. All CMS's should be offering this level of image manipulation and caching.

While you’re outputting your images to different formats consider adding WebP into the mix. It’s a new image format developed by Google that’s 30-40% smaller than a jpeg. Currently WebP is only supported in Chrome and Firefox are considering adding support. David Newton has created a fork of Picturefill that adds WebP and SVG detection so you can not only deliver the right size but the smallest format to different browsers.

Case study: the new responsive Guardian site

Comparison of page size and assets types across different responsive widths. Source: SpeedCurve

A great example using the above techniques is the new Guardian newspaper website, currently in alpha. On average it’s start render time is 3 seconds faster than the current Guardian site and the mobile width inlines CSS and is 42% smaller overall, leading to a 1 second saving over the desktop width.

Comparison of full loaded time for new Guardian responsive site (Guardian NGW) vs current Guardian site and the New York Times. Source: SpeedCurve

The Guardian have released their responsive code base on Github so you can easily dig through their development setup and see these techniques in action.

Establish a performance baseline and then go performance first

Performance optimisations like these are often invisible and it can be hard to demonstrate a return on investment without clear metrics in place. Some of these techniques can require a significant amount of work and it’s critical you establish a performance baseline first and then experiment with the techniques above to clearly show the improvements in user experience. There are great tools available to monitor the actual in browser speed and benchmark your site against others.

Putting all these techniques together you can dramatically improve the performance of your responsive site. There’s really no excuse for serving the same large sized assets across all browser widths. Make your responsive website respond not only to changing design patterns but to the browser environment it’s being served into. Go mobile first and performance first when designing and coding your next responsive website!

Thanks to Guypo for his ongoing responsive research and Patrick Hamann from The Guardian for letting me share their SpeedCurve dashboard.

New weekly emails

Wed, 06 Nov 2013 00:00:00 +1300

I've redesigned the weekly email reports to provide more trending information so you can compare week on week performance improvements. They're nice simple visualizations for forwarding around your whole team.

Let me know what you'd like to see added to these emails.

Share dashboards, basic auth and export HAR's

Tue, 15 Oct 2013 00:00:00 +1300

New features:

Share your dashboard via a private url. Find the link on the "account" page.
During site setup you can now add a user/pass for developments sites using basic authentication.
On the individual test pages below the waterfall are links to the underlying WebPagetest waterfall and ability to export a HAR file.

SpeedCurve dashboard walkthrough

Tue, 08 Oct 2013 00:00:00 +1300

See how powerful the SpeedCurve dashboard is for digging into your website's front-end performance.

Let me know which parts of the dashboard you'd like to learn more about.

Live!

Tue, 08 Oct 2013 00:00:00 +1300

SpeedCurve is now live! To mark the occasion I took the day off work and Helena & Bruno helped send out the launch emails.

All done while stuffing our faces with cake at Vevo cafe in Titirangi, Auckland. We watched the stats explode and 3 people signed up for subscriptions before we finished our hot chocolates. Start how you intend to continue I say!

SpeedCurve elevator pitch

Thu, 26 Sep 2013 00:00:00 +1200

I've been having a blast over the last couple of weeks putting an intro video together for SpeedCurve.

Big thanks to Matt from Assembly for casting an eye over my dodgy 3D AfterEffects camera work.

This is just the elevator pitch and I'll be working on some product walk through videos shortly. Let me know what web performance techniques, strategies or theory you'd like to see explained in video format. I've been inspired by the charming team at Wistia to create a video learning center for SpeedCurve.

Speed Matters talk at Auckland Web Dev Nights

Thu, 05 Sep 2013 00:00:00 +1200

Here's the slides from my presentation at the Auckland Web Dev Nights meetup. It's an overview of why front-end performance matter, how to monitor it and the challenges faced when building for an increasingly mobile world.

Content

Why speed matters, examples of the impact saving a few seconds of load time has had on revenue and engagement. The network constraints and what makes the web slow? Bandwidth, latency and it's fundamental impact on the speed of the web. An overview of tools for measuring performance, uptime monitoring, real user monitoring and performance benchmarking. How to make your website faster. Optimization tools and techniques. Muti-device challenges. Responsive vs Adaptive and delivering to mobile within a second. Drop that donut superman!

Performance Tools

Diagnotic Tools
WebPagetest and how to read a browser waterfall
Google Pagespeed Insights
YSlow

Uptime Monitoring
Pingdom

Competitive Benchmarking
SpeedCurve

Real User Monitoring (RUM)
Pingdom
New Relic (also backend, database and server health monitoring)
Google Analytics
mPulse

Performance Checklists

14 Rules for Faster-Loading Web Sites
25 tips on how to lose weight in the browser

References

Impact of performance improvements
Optimizing the Critical Rendering Path for Instant Mobile Websites (Video)
What are responsive websites made of?
Responsive web design is solid gold
Etsy - Building Resilient User Experiences
Velocity Conference Videos
Browser Diet References

Credits

These slides are heavily derived and mangled from awesome presentations by Andy Davies and Ilya Grigorik. Illustrations by Scott Johnson via Browser Diet. Feel free to steal it, adapt it and spread the word on front-end web performance.

Sort browser waterfalls

Wed, 14 Aug 2013 00:00:00 +1200

Sort the items in a browser waterfall by Time, Savings and Slowest. Time is the default showing how assets loaded in the browser. Savings places assets at the top that have the greatest optimization potential and Slowest shows you which are the slowest overall requests.

Annotate your graphs

Sun, 21 Jul 2013 00:00:00 +1200

You can now annotate the main graph on SpeedCurve with notes about deployments or specific performance optimizations you may have put live. Just click on the "Annotate" link at the bottom right of the main graph to start adding them and comparing the before and after performance of your website. A vertical line will be placed on the graph where ever a note has been added.

Easily add WebP, SVG and Retina responsive images to your site

Sun, 07 Jul 2013 00:00:00 +1200

A lot of sites are now using responsive design techniques to create great user experiences across mobile, tablet and desktop. Unfortunately images are often just set to 100% width with the same large image being delivered to all browsers and platforms. Picturefill was an important step in delivering responsive images and David Newton's fork now extends that approach to include WebP and SVG images in the mix you send down the pipe.

Picturefill

Picturefill is a fantastic browser polyfill by Scott Jehl that mimics the proposed picture element allowing you to deliver responsive images to users. It gives you control over what image is loaded and displayed at each browser width using conditional loading based on media queries. You can then curate each user's experience optimizing for screen proportions and assumed bandwidth. You could send smaller images with different aspect ratios to mobile users while desktop users with HiDPI Retina screens might get images twice as large as normal.

Now with file types

David Newton has forked Picturefill adding detection for WebP and SVG, allowing you to combine the switching of sizes and formats into one easy to implement HTML structure.

30% smaller images with WebP

With WebP detection now in place it's easy to send this new image format to browsers that support it and save 24%-34% in file size. WebP is a new image format from Google that provides lossless and lossy compression for images. The Chrome Web Store recently switched to WebP images and is saving terabytes of bandwidth a day.

If you're looking to experiment with WebP, XnConvert is an easy to use cross platform batch processing tool that supports WebP or grab the latest package from Google.

I've added WebP to the SpeedCurve homepage and now people using Chrome and Opera actually get images 56% smaller.

For more on detail on the WebP format, development status and various implementation options check out the video below and Ilya Grigorik's great overview of WebP.

How SpeedCurve fits into your web performance toolkit

Wed, 03 Jul 2013 00:00:00 +1200

Over the last few years the web performance monitoring toolset has expanded dramatically with the introduction of many new services and products. There are two main types of web performance monitoring, uptime monitoring and real user monitoring. SpeedCurve focuses on a third which I like to call web performance benchmarking.

Uptime Monitoring

Uptime services like Pingdom and Uptime Robot ping your website with an HTTP request every couple of minutes to check that it's online and available. These services often check from various geographic locations to keep an eye on network routes to your server and will send you alerts via email and txt if your website is down. Reporting is often simplistic with just the load time of the initial HTML page reported but it's perfect for knowing if your server is up and running. It's often called synthetic testing as tests are run from servers in a data centre and don't accurately represent what speeds an actual user might get.

Real User Monitoring

Real user monitoring (RUM) sends performance data directly from a user's browser to a cloud service like New Relic or Google Analytics that aggregates and reports on millions of combined measurements. RUM provides the most realistic picture of what users are actually experiencing in their browsers across different platforms and countries. Google Analytics recently updated their Site Speed report, allowing you to measure up to 10,000 page loads a day with a simple change to the sample rate in your tracking code. RUM code must be added to the pages you want to track so you can only get detailed performance insights for your own website.

Real user monitoring dashboard on New Relic.

Web Performance Benchmarking

SpeedCurve is neither an uptime or real user monitoring service. We focus on bringing two important benchmarking techniques together in a way that no other product does.

Firstly we provide detailed analysis of the build of your website over time. We dive right down into the detail of each request a page makes and demonstrate how your front-end code is effecting the performance of your site. Our analysis is based on the industry leading open source project Webpagetest. Importantly we do this analysis every 24 hours which allows you to monitor changes to your front-end code base over time, catching any regression issues and monitoring the ongoing effects of any performance optimization changes.

Secondly we benchmark your website against two of your competitors or category leaders. We do this by monitoring 3 pages on each site so you get a robust across-site comparison rather than just individual pages. It's not just your homepage that needs to be fast. This benchmarking provides you with the relative performance of each website and allows you to see what a competitor might be doing differently to make their site faster than yours. Are they making less requests than you? Are their images more optimised and grouped into sprite sheets making them a full 2 seconds faster to load? SpeedCurve will tell you.

Web performance benchmarking helps you focus on performance problems with your website build and compares you directly with your competitors to ensure you're keeping your users happy with a smokingly fast website.

Recommended Web Performance Toolset

It's vital you combine a real user monitoring tool with the build detail and competitive benchmarking that SpeedCurve provides. This way you get the best of both worlds with robust high level performance metrics from actual users and detailed build analysis and benchmarking from SpeedCurve.

Our favourite RUM tool is New Relic which gives you great insight across your stack from server health, uptime and backend right through to real user monitoring. It does require an agent to be installed on each server, so won't work for shared hosting. In that case use Google Analytics or Pingdom.

Combine New Relic with SpeedCurve and you've got a very detailed picture of performance across your entire backend stack, the front-end of your site and your competitors.

Filter graphs by browser events

Sat, 22 Jun 2013 00:00:00 +1200

It's important to not obsess over the full load time of your website. The point at which a user can see content and interact with your site is arguably more important. To help keep a focus on this you can now filter the main graph by the different browser events like "start render" and "DOM complete".

SpeedCurve launches at Velocity Conf

Wed, 19 Jun 2013 00:00:00 +1200

SpeedCurve is now in beta. Sign up, check it out and then vote for the features you'd like to see.

A huge thanks Steve Souders for encouraging me to build SpeedCurve and introducing it at Velocity Conference 2013.