A/B testing with Sanity and LaunchDarkly

Seif Ghezala

Updated 2023-11-13 · 3 min

Table of contents

We use Sanity to build fast websites that rank on the Google search engine and attract users.

We know that A/B testing is a key great way to find out what content fits better with your target users, so integrating it on Sanity is a great way to test your content with real users before going live with them.

This article will be a demo where we show how our clients usually create A/B tests from Sanity and control them from an A/B testing platform called LaunchDarkly.

You can also jump straight to the Youtube video if you prefer:

Goal

We have a homepage in a light theme, but we think that users might prefer a dark variation of the page.

To test that, we’ll create an A/B test with the 2 variations, serve each variation to 50% of users, and then decide which variation to keep.

A/B Testing with Sanity and LaunchDarkly

Step 1: Creating the A/B test on Sanity

On the "A/B Tests" tab, we create a new test and call it "Homepage theme".

Since we want to A/B test the homepage, we assign it the slug "/".

In the "Variations" field we set the existing light-themed homepage as the "Control".

We add a "Variant" variation and assign it a page that is identical to the existing light-themed homepage, but we set the theme to "Dark".

Step 2: Setting up the A/B test on LaunchDarkly

On LaunchDarkly, we create a new feature flag with the key "homepage", as indicated on the Sanity A/B test page.

We give it 2 string variations: Control and Variant.

Once the flag is created, we configure it to serve the Control homepage to 50% of users and the Variant homepage to the other half.

Once we publish both the LaunchDarkly and Sanity changes, we can test if the A/B test is working as configured by opening 2 incognito windows.

We'll notice that one of them is served the dark theme while the other is served the light one.

Concluding the experiment

Let's say we ran the A/B test for a while and realized that most users prefer the light theme (Control).

We can now delete the A/B test and the dark-themed version (Variant) from Sanity.

Once this change is published, 100% of users will be served the light-themed variation.

Frequently asked questions (FAQ)

Why am I served the same variation if I keep refreshing the page?

When you run an A/B test, you want to do so against unique users. If you reload the page during the same session, you're identified as the same user and thus served the same variation during the A/B test.

How to identify unique users during an A/B test?

If the website has authentication in place, we use users' ids to identify them. For anonymous users or for websites that don't have authentication, we generate a unique UUID and store it in the browser's cookies.

How to know which A/B test variation was successful?

You first have to ask yourself: what user action makes my page successful? For example, it could be pressing a Call-To-Action button or subscribing to your service.

Once you know what action makes a page variation successful, you need to have proper analytics in place to track that action.

You can read in more details about that in the Analytics with Sanity article or Zego case study.