A/B Split Testing with Nginx & Craft CMS
Tags: nginx, devops, splittesting
Published: 2020-02-21 01:36:20

---
title: A/B Split Testing with Nginx & Craft CMS
published: true
date: 2017-05-10 18:52:00 UTC
tags: nginx,devops,split-testing
canonical_url: https://nystudio107.com/blog/a-b-split-testing-with-nginx-craft-cms
---

# A/B Split Testing with Nginx & Craft CMS

### Doing A/B split test­ing can be very use­ful in mea­sur­ing the effec­tive­ness of your pages; here’s how to do it with Nginx & Craft CMS

Andrew Welch / [nystudio107](https://nystudio107.com)

![Ab Split Testing First Image](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_1200x675_crop_center-center_82_line/ab-split-testing-first-image.jpg)

Doing A/B Split Test­ing is all the rage these days. It allows you to present a sin­gle page to the user, but change the con­tent such that group <kbd>A</kbd> sees some­thing dif­fer­ent from group <kbd>B</kbd>.

Then you can mea­sure which vari­ant has high­er con­ver­sions or goal achieve­ments, and pro­ceed from there.

So this is a very use­ful thing to do, and an entire indus­try has sprung up around it, with var­i­ous ser­vices such as [Opti­mize­ly](https://www.optimizely.com/) (and there are dozens of oth­ers) that pur­port to allow you to do this A/B Test­ing more easily.

And they do what they claim; the only prob­lem is that they typ­i­cal­ly work by hav­ing you inject some JavaScript into your <kbd>&lt;head&gt;</kbd> which then rewrites the con­tent that users see based on the A/B Split test.

<aside>
                                Why is this a prob­lem? Per­for­mance and <span>UX</span>.
                            </aside>

It should be pret­ty obvi­ous that the JavaScript can’t do its thing and deliv­er the A/B con­tent until the web­page has loaded, and the JavaScript has exe­cut­ed. This can result in non-per­for­mant pages, and UX glitch­es as the page is loaded, and then the A/B con­tent is swapped in.

Plus, they cost money.

We’ve gone through great pains to have a per­for­mant web­site as per the [A Pret­ty Web­site Isn’t Enough](https://dev.to/gaijinity/a-pretty-website-isn-8217-t-enough-1nbj-temp-slug-3932740) arti­cle, and we’re not gonna screw it up now.

Addi­tion­al­ly, just like with Google Tag Man­ag­er ​“Tags”, we don’t want the tech­ni­cal imple­men­ta­tion of our test affect­ing the results! If our page loads slow­er than nor­mal, or has strange load­ing glitch­es, this could poten­tial­ly affect how peo­ple per­ceive the page, and the message.

Check out the [Tags Gone Wild! Man­ag­ing Tag Man­agers](https://dev.to/gaijinity/tags-gone-wild-managing-tag-managers-1np1-temp-slug-2370979) arti­cle for more on how the act of obser­va­tion can affect the result.

So, then, how do we do this in a per­for­mant, non-intru­sive way?

## First, a Confession

I have a con­fes­sion to make. Depend­ing on who you are, you’re see­ing dif­fer­ent con­tent when you load this very blog page.

![Ab Split Test Confession](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_1200x675_crop_center-center_82_line/ab-split-test-confession.jpg)

I’ve sub­tly changed the con­tent of this page so that not every­one is see­ing the same thing.

Some of you will see this image at the top of the page:

![Ab Split Testing First Image](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_1200x675_crop_center-center_82_line/ab-split-testing-first-image.jpg)

…and some of you will see this image at the top of the page:

![Ab Split Testing Second Image Deux](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_992x558_crop_center-center_82_line/ab-split-testing-second-image-deux.jpg)

What you see will depend on a com­bi­na­tion of your IP address, the brows­er you’re using, and the time of day that you vis­it the blog page. I then store this infor­ma­tion in an <kbd>abtest</kbd> cook­ie that last 24 hours, so you’ll see the same image no mat­ter how many times you load the page.

<aside>
                                I know, you were just start­ing to trust me. It’s in the name of sci­ence, though.
                            </aside>

I then send the data of which A/B Split test you saw along to Google as a [Cus­tom Dimen­sion](https://support.google.com/analytics/answer/2709829?hl=en) so that lat­er on I can look at the ana­lyt­ics, and see which image result­ed in bet­ter conversions.

Don’t believe me? Open up your Devel­op­er Tools in Chrome, and have a look at the cook­ies set on this page:

![Ab Split Test Cookie](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_576x313_crop_center-center_100_line/ab-split-test-cookie.png)

You can delete this cook­ie, and reload the page, and you’ll get a dif­fer­ent image (it may take a cou­ple of tries, since it’s a 50/50 chance either way).

![Side By Side Split Test Deux](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_1200x553_crop_center-center_100_line/side-by-side-split-test-deux.jpg)

Browsers can some­times be fun­ny about the way they cache cook­ies, so the eas­i­est way to test it is via a new Private/​Incognito brows­er window.

So this is pret­ty cool. How do we do it?

## Nginx & Craft CMS to the Rescue!

As it turns out, it’s rel­a­tive­ly sim­ple to get this work­ing, but you will have to get your hands dirty edit­ing the <kbd>nginx.conf</kbd> file. If you’re using Apache, there are sim­i­lar arti­cles on the sub­ject you can find via Google.

The first thing we need to do is edit the base <kbd>nginx.conf</kbd> file (this is usu­al­ly at <kbd>/etc/nginx/nginx.conf</kbd>) to add this inside of the <kbd>http</kbd> block:

```

        ##
        # A/B Split Testing
        ##

        split_clients "${remote_addr}${http_user_agent}${date_gmt}" $abtest_split {
            50% "blue";
            50% "green";
        }

        map $cookie_abtest $abtest {
            default $cookie_abtest;
            "" $abtest_split;
        }

```

What this does is it uses the [split\_​clients](http://nginx.org/en/docs/http/ngx_http_split_clients_module.html) direc­tive to make a hash out of the string we’re pass­ing in (which is a com­bi­na­tion of the <kbd>remote_addr</kbd>, <kbd>http_user_agent</kbd>, and <kbd>date_gmt</kbd>) and set the vari­able <kbd>$abtest_split</kbd> such that 50% of the time it’s <kbd>green</kbd> and 50% of the time, it’s <kbd>blue</kbd>.

But we don’t want to have this change every time the page is reloaded (which it nor­mal­ly would, because of the <kbd>date_gmt</kbd> being in the mix), so we use the [map](http://nginx.org/en/docs/http/ngx_http_map_module.html) direc­tive to set the vari­able <kbd>$abtest</kbd> to this com­put­ed <kbd>$abtest_split</kbd> only if the cook­ie <kbd>abtest</kbd> is not set.

Oth­er­wise, we just use the cook­ie <kbd>abtest</kbd> value.

Then in our <kbd>virtualhost.conf</kbd> we just need to add a line in the serv­er block to set the cook­ie based on this <kbd>$abtest</kbd> value:

```

    # A/B Split Testing cookie
    add_header Set-Cookie "abtest=$abtest;Path=/;Max-Age=86400";

```

Here’s what that looks like in [Forge](https://forge.laravel.com):

![Ab Split Test Forge](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_576x347_crop_center-center_100_line/ab-split-test-forge.png)

All this does is add a cook­ie called <kbd>abtest</kbd> with the val­ue of the com­put­ed <kbd>$abtest</kbd> vari­able, with the path <kbd>/</kbd> that lasts for 24 hours (that’s 86,400 seconds).

Sweet.

## Cook­ies & Craft CMS

So now we have this cook­ie <kbd>abtest</kbd> that will be either set to <kbd>green</kbd> or <kbd>blue</kbd>. We can use [Craft CMS](https://craftcms.com/) and the [Cook­ies plu­g­in](https://github.com/nystudio107/cookies) to dis­play dif­fer­ent con­tent depend­ing on how this is set!

If you’re using a CMS or back­end oth­er than Craft CMS, that’s fine. The same prin­ci­ples apply, we change what con­tent we dis­play based on the <kbd>abtest</kbd> cookie.

![Yummy Cookies](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_1200x675_crop_center-center_82_line/yummy-cookies.jpg)

We just add a line like this to top top of our main <kbd>layout.twig</kbd> template:

```

{% set abtestVariant = getCookie('abtest') |default('blue') %}

```

This just sets the Twig vari­able <kbd>abtestVariant</kbd> to what­ev­er the <kbd>abtest</kbd> cook­ie is set to, while default­ing to <kbd>blue</kbd> in case there’s no cook­ie present for what­ev­er reason.

Then for this exam­ple, we just change our image dis­play­ing code so that it’ll pick a dif­fer­ent asset if the <kbd>abtest</kbd> cook­ie is set to <kbd>green</kbd>:

```

{% set image = block.image[0] %}
{% if abtestVariant == 'green' %}
    {% set image = block.image[1] |default(block.image[0]) %}
{% endif %}

```

We fall back on the first image, in the event that some­one for­got to add more than one image to the assets field.

And then because we’re doing full-page caching on our site as per the [The Craft `{% cache %}` Tag In-Depth](https://dev.to/gaijinity/the-craft-cache-tag-in-depth-207b-temp-slug-8529389) arti­cle, we also want to cache each A/B page con­tent dis­crete­ly, so we just do:

```

{% cache globally using key (craft.request.path ~ abtestVariant) unless craft.retour.getHttpStatus == 404 %}

```

We also need­n’t do a sim­ple 50%/50% traf­fic split, we could just as eas­i­ly have four vari­ants, each get­ting 25% of the traf­fic. Or we could weight it how­ev­er we choose to.

This opens up a world of pos­si­bil­i­ties for mar­keters and ana­lyt­ics-types to _go wild…_ because we can entire­ly change what we dis­play to vis­i­tors, and mea­sure the results.

## A/B Split Test­ing Entries

Bear in mind that we could do some­thing much more inter­est­ing than just chang­ing the head­line image, such as con­di­tion­al­ly fetch an entire­ly dif­fer­ent entry depend­ing on the val­ue of the <kbd>abtest</kbd> cookie.

For instance, we could have a ​“con­tent builder” sec­tion built as per the [Cre­at­ing a Con­tent Builder in Craft CMS](https://dev.to/gaijinity/creating-a-content-builder-in-craft-cms-29ap-temp-slug-2753628) arti­cle, which allows mar­ket­ing the free­dom to cre­ate what­ev­er lay­outs they like.

Then we can add a Craft [Entries](https://craftcms.com/docs/entries-fields) field to it that lets them pick the A/B test for that page:

![A B Split Test Entry](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_546x346_crop_center-center_line/a-b-split-test-entry.png)

This could be right at the top of the ​“con­tent builder” entry, or it could be tucked away in the ​“SEO” tab or what have you. Then we add a lit­tle Twig code to our template:

```

  {% set abtestVariant = getCookie('abtest') %}
  {% do craft.cookies.set('abtest', '' ) %}
  {% if abtestVariant is defined and abtestVariant is not empty %}
      {% if entry is defined and entry |length %}
          {% if entry.aBSplitTestVariant is defined and entry.aBSplitTestVariant |length %}
              {% set cookieValue = '' %}
              {% if abtestVariant != "blue" %}
                  {% set entry = entry.aBSplitTestVariant.first() %}
                  {% set cookieValue = entry.slug %}
              {% endif %}
              {% if cookieValue is empty %}
                  {% set cookieValue = abtestVariant %}
              {% endif %}
              {% do craft.cookies.set('abtest', cookieValue, now | date_modify("+24 hours").timestamp ) %}
          {% endif %}
      {% endif %}
  {% endif %}

```

So if the <kbd>abtest</kbd> cook­ie is not <kbd>blue</kbd> (the default), we swap in an entire­ly new entry, and set the <kbd>abtest</kbd> cook­ie to be that entry’s slug.

On the back­end, that allows the client to build as many a/​b test vari­ants as they like using our con­tent builder. They turn **Enabled** off for entries that are split test vari­ants, so they won’t ever appear on the live site.

This keeps the URL the same, but the con­tent dif­fer­ent, and offers a very pow­er­ful way to do split testing.

## The Google Connection

The last step in all of this is send­ing data to Google so that we can link this A/B test to our con­ver­sions or goals. After all, it’d be pret­ty use­less if all we did was dis­play dif­fer­ent con­tent to peo­ple, we need to mea­sure the results!

This is where [Cus­tom Dimen­sions](https://support.google.com/analytics/answer/2709829?hl=en) come into play. Log into your Google Ana­lyt­ics account, go to **Admin** → **Prop­er­ty** → **Cus­tom Def­i­n­i­tions** and click on **New Cus­tom Dimen­sion** to name your dimension:

![Google Custom Dimension](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_576x152_crop_center-center_100_line/google-custom-dimension.png)

Then we just need to add a bit of JavaScript before our <kbd>ga('send', 'pageview')</kbd> to send along our cus­tom dimension:

```

ga('set', 'dimension1', '{{ abtestVariant }}');

```

If you’re using the [Instant Ana­lyt­ics plu­g­in](https://github.com/nystudio107/instantanalytics) to send serv­er-side ana­lyt­ics via the [Google Ana­lyt­ics Mea­sure­ment Pro­to­col](https://developers.google.com/analytics/devguides/collection/protocol/v1/), you’ll just need to do this before the <kbd>`{% hook 'iaSendPageView' %}`</kbd>:

```

{% do instantAnalytics.setCustomDimension(abtestVariant, 1) %}

```

And then in our Google Ana­lyt­ics, we can add this cus­tom dimen­sion to our view:

![Google Custom Dimension Secondary Dimension](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_394x338_crop_center-center_line/google-custom-dimension-secondary-dimension.png)

And then you can eval­u­ate the results of your A/B Split Test in your famil­iar Google Ana­lyt­ics reporting.

This is all done serv­er-side, so it’s is per­for­mant, and results in a clean user expe­ri­ence. And since this is all cook­ie-based, we have great flex­i­bil­i­ty in what we present to the user.

Pret­ty neat.

## Further Reading

_If you want to be notified about new articles, follow [nystudio107](https://twitter.com/nystudio107) on Twitter._

<small>Copyright ©2020 nystudio107. Designed by nystudio107</small>