Measuring Jank and UX

Ten years ago the network was the biggest problem when it came to making websites fast. Today, CPU is the main concern. This happened because networks got faster while JavaScript moved in the other direction growing 3x in size in the last six years. This growth is important because JavaScript consumes more CPU than all other browser activities combined. While JavaScript and other activities block the CPU, the browser can't respond to user input creating the sensation of a slow, jittery, or broken page, AKA "jank".

To help focus our attention on CPU, several new performance metrics have been defined and evangelized over the last year or three. In this post I'm going to focus on these:

• First CPU Idle measures when the page is no longer janky. Specifically, it is the first span of 5 seconds where the browser main thread is never blocked for more than 50ms after First Contentful Paint. A value of 2-4 seconds is typical.
• First Input Delay measures the gap between when a user interacts with the page (e.g, clicks or scrolls) and when the browser is able to act on that input. First Input Delay values are much lower - a good target is 10ms, but 25ms is common.
• First Interaction Time is when the first user input takes place. This varies widely depending on the type of site and page. A good search results page might have a low First Interaction Time because users scroll and click quickly. A media site might have a high First Interaction Time because users start reading content (headlines, stories) before interacting with the page. At SpeedCurve we call this "IX Time".
• Total Long Task CPU Time is the sum of all long tasks that occur in the page. A "long task" is a browser event that blocks the main thread for more than 50ms.

Here's a figure to help visualize these metrics.

The figure above is a (slightly modified) CPU timeline from WebPageTest. Long Tasks (where the CPU is blocked for more than 50ms) are shown in red. In the timeline above there are two long tasks. Green indicates where there are no Long Tasks. First CPU Idle happens at 3.0 seconds - this is the first point in the timeline where there are 5 seconds without any Long Tasks (from 3s to 8s). In this timeline the user first interacted with the page around 5.5s - this is the First Interaction Time (or IX Time). Since there were no Long Tasks, the browser was able to respond to the user's input in only 8ms, which gives us the First Input Delay (shown in blue).

As illustrated by this example, looking at the relationship between these metrics is the key to determining whether jankiness is affecting your users, and why.

In the timeline above, First Interaction Time is after First CPU Idle, so it makes sense that First Input Delay is very fast - there's nothing blocking the CPU. Seems obvious, but typically when I talk with performance folks about First Input Delay the discussion focuses on how First Input Delay is thwarted by Long Tasks, but if First Interaction Time is greater than First CPU Idle that's just not the case. So for site owners to appreciate what First Input Delay is telling them, it's important to know:

          How often is the first interaction after First CPU Idle?

For our RUM customers, First Interaction Time is after First CPU Idle about 66% of the time. That's important to understand and keep in mind. It means that most of the time First Input Delay is not affected by Long Task CPU activity.

About a third of the time the first interaction takes place before First CPU Idle. This scenario is illustrated in the figure below. In this example, the first interaction is unlucky enough to happen during a Long Task, which means the user has to wait until that Long Task is completed before their interaction can be handled. This is when we see First Input Delay values jump to hundreds of milliseconds. 

It's not always the case that when first interaction happens before First CPU Idle it will overlap with a Long Task. In the timeline above it's much more likely that the user input will not collide with a Long Task, which would result in a fast First Input Delay value. In the example timeline below, however, there are many (short) Long Tasks, so it's likely that First Input Delay values will be higher leading to more jankiness. It's important to understand the amount of CPU time being consumed by Long Tasks.

The timeline above has many Long Tasks resulting in higher First Input Delay values. However, the Long Tasks are fairly short, so First Input Delay values never exceed a few hundred milliseconds. The timeline below is a different story. It only has two Long Tasks, but one of them is almost 1.5 seconds long. (I've seen this in the real world!) If the first user input overlaps with this Long Task, we start seeing First Input Delay values over 1000ms. It's important to understand how long your Long Tasks are.

One last thing to keep in mind about First Input Delay is that it doesn't take rendering into consideration. This is a big blind spot. Looking at the timeline at the very top, perhaps the reason the user waited until 5.5 seconds to interact with the page is because the critical content took a long time to render. In which case, if rendering times were improved, First Input Delay would get worse even though the number of Long Tasks and CPU consumed remained the same. That's because improving rendering would get the user interacting sooner causing them to overlap with the Long Tasks that previously were out of harm's way. It's important to track First Input Delay relative to interaction and rendering times.

This last point is why being vigilant about web performance is difficult. It's not possible to focus on one metric. Focusing on the user experience is the key. And when it comes to user experience, CPU and rendering are the most important metrics to watch. Fortunately, (here comes the pitch) SpeedCurve's RUM product, LUX, includes First CPU Idle, First Interaction Time, First Input Delay, Total CPU time, number of Long Tasks, median Long Task, and longest Long Task, as well as many rendering metrics. If you're not already using LUX, signup for free trial.

And keep these important takeaways in mind:

• Know how often First Interaction Time is after First CPU Idle.
• Understand the amount of CPU time being consumed by Long Tasks.
• Monitor your longest Long Tasks.
• Track First Input Delay relative to interaction and rendering times.