Digital Performance Management: Combining Real, Synthetic, and Virtual Users

February 25, 2016 · by James Urquhart ·

To understand how real, synthetic, and virtual user agents complement each other, it’s critical to understand what each represents. Sorting out why one might need both synthetic users and virtual users, for example — or even what the difference between the two might be — can be quite confusing.

Here’s a useful guide to the three user types.


Real users

Monitoring the actual experiences of users s absolutely essential to creating optimal user experiences each and every time. Monitoring real users allows you to test your site or application running on or in all of the different types of networks, platforms, device types, and geographical environments.

Real users allow you to measure what the end user actually experiences through their front-end stack (browser or app, OS, etc), but at the cost of control. Being able to capture anomalies quickly, especially for things like key transactions or usage under exceptional loads requires the ability to collect data with more known initial conditions or baselines.


Synthetic users

The easiest way to run a known transaction end to end, from the user device to backend services with known conditions is to use an artificial “user” to drive that transaction. Synthetic user monitoring simplifies asking questions like “is the backend behaving the way it should at this moment for this transaction” or “are users on the east coast of the US experiencing worse performance than users in Europe?”

Unfortunately, because these systems don’t scale well, and these synthetic users can themselves affect performance if they generate too much load, these users don’t scale well at all. So, how do you create a baseline of expected behavior on the backend systems that have to scale to meet customer demand when stressed?


Virtual users

Virtual users aren’t meant to be used as day-to-day real-time monitoring of current conditions but are intended more to give a solid baseline measurement of backend performance under varying amounts of load. Those baselines can be regularly verified in production, and operators can be warned if virtual user performance slips out of expected ranges.

Virtual users can also be run at incredible scales during scheduled load and stress tests. Their initial conditions can be closely controlled, though the combination of scale and control of conditions comes at the cost of driving load primarily against the backend, without driving actual browsers or mobile applications.


How virtual users complement real and synthetic users

We have a pretty good idea of how synthetic and real user monitoring complement each other, let’s explore what virtual users bring to digital performance management.

Cloud testing introduced the concept of load testing actual production systems, instead of focusing only on “staging” environments before release to production. This is a powerful concept for today’s rapidly evolving digital applications, but it’s only possible if the load testing system is capable of dynamically adjusting load from an extremely light baseline to a massive peak load without rebuilding the load tests.

The way we achieve this is by utilizing CloudTest to drive virtual users through the system at a load that can be adjusted throughout a test run as needed. So you can choose to either run baseline tests as separate tests from time to time or continuously run a load test that can be throttled up and down as needed over time.

Low levels of testing give you a baseline of behavior for these test users, which can be compared to behavior when the throttle is turned up. (Low-level tests are also great for finding code changes that negatively affect performance or break the load test scripts before a critical load test finds them first.)

This comes at the cost of not using actual browser and device profiles, of course. You won’t know from a virtual user if the experience is better in Chrome or Firefox, for instance, from a virtual user in a load test. However, you will be able to confirm that your backend servers can handle any anticipated (and all but the most unimaginable unanticipated) loads when they come.


How real and synthetic users complement virtual users

Equally important to the information that virtual users bring to understanding digital performance is the power of utilizing real and synthetic user measurements when designing virtual user load tests. Beyond the simple comparisons of all three types of data — which alone can help greatly with troubleshooting performance and availability problems — there is the power of using real and synthetic data to shape the tests you run.


Example 1: Real user monitoring data will tell you a lot about the situations that cost you the most in terms of whatever your conversion metric might be

Sometimes the situations that cost you the most aren’t obvious. We’ve seen situations in which specific product pages had serious conversion issues, though not all such pages did. Real user data can make that clear, and then load tests can be created to test that condition, at least until you’ve confirmed the problem no longer exists. (Even then, you can run the test occasionally as a form of regression testing.)


Example 2: Using the virtual user baselines noted above as additional baselines for real and synthetic user measurements

Does your backend perform differently under varying loads? When is a real or synthetic measurement truly an anomaly vs expected (though not necessarily “acceptable”) behavior? Having a strong, controlled profile of your system performance gives you greater confidence in tagging events as unexpected versus expected.



The most complete protection against performance-related failure needs a three-user team with:

  • Synthetic users giving you a sense of user experience in well regulated “experiments” across browsers and devices
  • Real user monitoring telling you about the actual experiences discovered by actual users
  • Virtual users enabling you to build a feedback loop to test, measure, and verify performance variability, and resiliency

With this combination, you have the makings of a complete picture of where, how, and why performance affects user outcomes.

And understanding user outcomes enables understanding of both business and technical outcomes.