How to Use Activity Impact Scores

March 21, 2016 · by Tammy Everts ·

I’ve spent years researching and writing about the intersection of web performance and business metrics, and I can tell you with a huge degree of certainty that while there’s a massive amount of research available about the correlation between performance and revenue/conversions, there’s a dearth of similar data around performance and engagement.


Why the murkiness around performance and user engagement?

True user engagement — beyond just bounce rate and page views — is notoriously tricky to measure. And because retail sites have, historically, been at the forefront of performance optimization and measurement, it’s not a huge surprise that our industry has focused on the correlation between page speed and retail metrics like cart size, conversions, and revenue.

But I’ve noticed a shift in recent years. Media sites are now aggressively pursuing performance:

It’s been heartening to see that performance is no longer the baileywick of retail sites. As more and more media is consumed via mobile devices — and as media sites realize they’re competing not just with each other, but also with Facebook, Instagram, and a trillion speedy apps — media site owners have prioritized performance.

That’s why, at SOASTA, we’ve been working hard to develop a new metric that will help media sites understand the impact of performance changes on their pages.


Introducing the Activity Impact Score

You may already be familiar with the Conversion Impact Score, which answers the question “How much impact does the performance of this page have on conversions?”

The Activity Impact Score is a kissing cousin to the Conversion Impact Score. Using your own real user data, it answers the question “How much impact does the performance of this page have on session length?”


Why measure session length?

A session is a group of interactions that a user takes within a given time frame on your website. If you run a media site, a lengthy session is a good indicator that people are staying on your site and consuming your content. The more time people spend on your site, the more ads they can be served.

Longer session lengths benefit both media sites and advertisers:

  • If you run a media site and rely on ads for revenue, longer sessions translate to more ad views.
  • If you’re an advertiser, more ad views translate to more conversions. (A massive online advertising study by comScore, which measured 263 million ad impressions over nine months, found that ad viewability and hover time are more strongly correlated with conversions — defined as purchases and requests for information — than clicks or total impressions. In other words, being seen matters even more than being clicked.)
  • There’s also a soft (i.e. not data-driven) argument that longer session length equals greater reader satisfaction, which equals higher retention/return rate.

So how do you get your Activity Impact Score and use it in your day-to-day operation?


Use case: How to use your Activity Impact Score to prioritize performance optimization

Let’s walk through how to use the Activity Impact Score to make decisions about optimizing your pages.

Assuming that…

  • you’re using some kind of real user monitoring solution (which I hope you are);
  • you have access to 100% of your data, including all your historical data;
  • you have a talented data scientist on your team (shout-out to our own talented data scientist, Ben Polovick!);
  • you’ve tagged your pages into meaningful groups, e.g. articles, departments, etc.; and
  • your RUM solution is correlating the performance of your page groups to metrics such as bounce rate, page views, session length, etc.

…then you can use your RUM data to calculate Activity Impact Scores for your pages.

(Note: We use mPulse to generate Conversion Impact Scores for our customers’ pages, but any real-user monitoring solution should allow you to do this.)

In the graph below (which represents anonymized data from one of our mPulse customers), you can see the Activity Impact Scores and load times for a set of page groups on the site. The blue bars represent the Activity Impact Score for each page group, and the green line represents the median page load time for each page group.

The page groups are ranked from those with the highest Activity Impact Scores (such as the Home page and the US and Politics page groups) to page groups with the lowest scores (such as Search and Fashion pages).

Some quick observations:

  • Three of the fastest page groups — Videos, Americas, and Search — all have relatively low Activity Impact Scores, despite being relatively fast. This means that page speed isn’t a significant factor in how well these pages engage visitors.
  • The Home page has the highest Activity Impact Score, but its median load time is on the slow side: between 7 and 8 seconds.
  • The slowest page groups are the Asia and Fashion groups.


Without Activity Impact Scoring, here are five mistakes you might make

Mistake 1: Waste massive optimization resources on the wrong pages

If you take a “fix the slowest pages first” triage approach, then you’ll use a lot of resources to optimize the Asia and Fashion page groups. Activity Impact Scoring lets you know that making these pages faster won’t have a significant impact on user engagement for these pages. (There’s an argument to be made for the fact that, in an ideal world, it would be great to deliver speedy online experiences to all your users, but that’s another discussion. Today we’re talking about “fast enough” from a business metrics perspective.)

Mistake 2: Continue to optimize pages that are already fast enough

In this example, the Americas and Search page groups have okay load times (in the 5-6 seconds range). Sure, they could be improved, but their lower Activity Impact Scores indicate that optimizing them won’t significantly improve user engagement.

Mistake 3: Ignore pages that have relatively good performance

In the performance spectrum for this set of page groups, the Home page falls more or less in the middle, making it somewhat forgettable from a numbers perspective (though in the real world, most organizations tend to care about their homepage performance, regardless of how fast or slow that page already may be). Activity Impact Scoring makes it clear that, for this site, optimization efforts should focus on the page groups in this order: Home, US, Politics, Videos, Entertainment, and Health.

Mistake 4: Forget that every site is different

This is huge. If you run a media site, what I don’t want you to take away from this post is the idea that you should immediately start fixing your home page, just because that’s what we looked at in this example. Every site is different, which means every site’s Activity Impact Score is different.

To illustrate, here’s a chart showing the Activity Impact Scores for another media site:


In this case, the Home page is in distant fourth place, and the page group with the highest score is the Articles group. So unlike the first example we looked at, optimizing the Home page is less critical than addressing performance on the individual article pages.


Mistake 5: Assume that Activity Impact Scores are unchanging

Activity Impact Scores can change over time, as your site changes and as your users and their behavior changes. When it comes to the web, nothing is static.



As I mentioned earlier in this post, in an ideal world it would be lovely if we could serve 2-second page loads to all of our visitors. Unfortunately, this is the real world, where optimization resources are limited and we need to be smart about how we use them. Activity Impact Scoring is an excellent tool for media site owners — or anyone else who cares about session length as a KPI — to learn how fast is fast enough for their visitors and, armed with this data, tackle performance issues where they matter most.