Performance A/B Testing (Beta)

Let’s say you’re not sure if a new app or script is slowing down your website. You could add it to your website and check to see how much it impacts your website’s performance with Render Better’s RUM (Real User Monitoring) analytics dashboard. However, this approach is tedious and it’s hard to isolate whether it was the new thing you added or something else that might be affecting the website’s performance.

Render Better supports the ability to A/B test new code and measure its performance with real users so you can truly determine the exact impact that code makes.

How it works

When a new experiment is configured in Render Better, it will have multiple variants that Render Better will randomly choose from and display to the visitor. Each variant is code you want to test in the website’s HTML.

Typically you have two variants: the control (old code) and the treatment (new code). Render Better supports as many variants as you want, and different weighting for how often you want one variant to be shown versus another (e.g. if you want the control to be shown 25% of the time and the treatment to be shown 75% of the time).

Once a visitor lands on your website, they will be added to a specific variant group and be shown the same variant during the entirety of their session on the website.

As the visitor navigates your website, Render Better collects performance data on how each page loads during the visitor’s session and aggregates it for you in its RUM analytics dashboard. The results of how the website performed for each variant can then be compared, giving you a true performance measurement of each variant, so you can choose the best performing code possible!

Configure a new experiment

Before an experiment can be run, there’s a few steps that must be completed: experiment configuration and adding the A/B test code to the HTML (see the next section).

Experiment set up specifies which variants are in the experiment (typically an A and B variant) and how often each variant should be seen.

This beta feature requires the help of your accounts team to set up. Reach out to success@renderbetter.com to configure a new experiment.

Add the A/B test HTML code

Once an experiment is configured, the A/B test code can be added to your website’s HTML. The test code allows you to specify what code will run depending on which variant the visitor should see (e.g. control group vs. treatment group).

Let’s say we have two variants in our experiment: “A” and “B”. Let’s also say we want the A variant to represent a “control” group, which means it’s the code we had before this experiment. And the B variant represents a “treatment” group, which is the new code being tested. With the A variant, we want to load a different script and CSS file in the head tag than in the B variant.

Adding the A variant code

To set this up in the HTML, the existing HTML elements should have the rb-ab-variant attribute added to it, like so:

...
<head>
...

<!-- Add the rb-ab-variant attribute to variant A existing element code -->

<!-- Variant A code... -->
<script rb-ab-variant="A" src="..."></script>
<link rb-ab-variant="A" rel="stylesheet" href="...">

...

This tells Render Better these elements should be removed from the HTML when the visitor is in the treatment group, variant B.

Adding the B variant code

...
<head>
...

<!-- Add the rb-ab-variant attribute to variant A existing element code -->

<!-- Variant A code... -->
<script rb-ab-variant="A" src="..."></script>
<link rb-ab-variant="A" rel="stylesheet" href="...">


<!-- Wrap B treatment code in a template tag, and add the rb-ab-variant
     attribute to the template tag -->

<template rb-ab-variant="B">
<!-- Variant B code... -->
<script src="..."></script>
<link rel="stylesheet" href="...">
</template>

...

When the visitor is in the control group, variant A, Render Better will remove the template tag and all of the content inside it. But when the visitor is in the treatment group, variant B, Render Better will unwrap the code inside the template tag and replace the template tag with its inner code, as well as remove the variant A code.

Why shouldn’t variant A code be wrapped in a template tag?

You may be wondering why the variant A code is not wrapped in a template tag where as the variant B code is. In case Render Better is not running in front of the website, or if it has been disabled, or if in certain cases it doesn’t run, you will want to safely fallback to code still running as expected. In this case, if Render Better isn’t running, we want the variant A code to always load, where as the variant B code should not. By wrapping the variant B code in a template tag, the browser will skip processing this code even if Render Better doesn’t manage to remove it.

Run the experiment

Once the experiment has been configured and the experiment code has been added, you’re ready to start testing.

Reach out to success@renderbetter.com to enable the new experiment.

Analyze the results

After the experiment has been enabled, Render Better will collect performance analytics data so that it can be displayed as a dimension to the RUM analytics charts in the dashboard.