A/B testing: introduction
Tags: abtesting, experimentation, product, business
Published: 2018-01-05 20:47:29

---
title: "A/B testing: introduction"
published: true
description: I’ve been running A/B tests for 2 years now, and during this time I have run more than 300 experiments.
I will share with you my knowledge: I will explain what A/B testing is, when do you need it (and when you don’t), and how to utilize it to the greatest extent.
tags: a-b-testing, experimentation, product, business
cover_image: https://thepracticaldev.s3.amazonaws.com/i/0wdybm9zwrm9kfr0o72g.jpg
---

<em>This article has been first published on my website <a href="http://smartpuffin.com/ab-testing-basics/">smartpuffin.com</a>.</em>
<em>It is based on my (first ever!) conference talk about A/B testing - check out <a href="http://smartpuffin.com/ab-testing-of-geodata-in-booking-com/">the slides</a>.</em>
<h1>What is A/B testing, anyway?</h1>
<span style="font-weight: 400;">Let’s imagine you own a webpage for selling flowers, and you have a red <b><span style="color: #ff0000;">"buy"</span></b> button in there.</span>

<span style="font-weight: 400;">Now let’s imagine your new designer tells you: hey, <b><span style="color: #008000;">green</span></b><b> </b>buttons are a new trend in the flower selling industry. People like green more than red, red is aggressive, green is soft and persuading. Green button will be more visible on the page. Let's make it green.

<img src="https://thepracticaldev.s3.amazonaws.com/i/ql5p4vvknwrzmtbqkolt.jpg" alt="Two mockups of a website, one with a red button and one with a green button"/>

<span style="font-weight: 400;">But you <i>like</i> red buttons. You think red is the best color in the world. It’s very bright, straight to the point, it pops up and calls you to buy some flowers for your significant other.</span>

<span style="font-weight: 400;">The question is: how do you two agree?</span>

<span style="font-weight: 400;">Do you trust your designer enough to let them do this risky change? What if you are right, and the red button is better? What if they are right, and you are missing on an opportunity?</span>
<h2>Let the users decide!</h2>
With the A/B testing approach, you don't have to argue which button is <em>the best</em>. You simply let the users decide!

<span style="font-weight: 400;">In a case like this you can set up an <b>A/B test</b>, also called an <b>experiment</b>.</span>

<span style="font-weight: 400;">Think of it as of a <em>sports game</em>. You let the buttons compete for your users. Which button is <em>the fairest of them all</em>? For which button do the users implicitly "vote"?</span>

<span style="font-weight: 400;">You let 50% of visitors see your favorite red button, <i>bright like a rose</i> 🌹.</span>

<span style="font-weight: 400;">And the other 50% of them will see the green button, <i>gentle like a leaf</i> 🍃.</span>

<img src="https://thepracticaldev.s3.amazonaws.com/i/wtezgpve7wpz5669r4iq.jpg" alt="An image of an a/b test flow. Users are divided in two halves; one half sees the red button, the other - the green button. The red ones have 10% conversion, the green ones have 20% conversion, and they win." /> <i>A/B test flow</i>

<span style="font-weight: 400;">You set this up and you watch for a while. You log how many people visit your page, and how many buy your flowers, so that you could compare.</span>

<span style="font-weight: 400;">And after a while you will see the result. Let's say that, for example, <b>10%</b> of people who saw the red button, bought a bouquet. And of these who saw the green one, <b>20%</b> bought.</span>

<span style="font-weight: 400;">Clearly, the green one is a winner here! Yay! Go green!</span>
<h2>What does "the best" mean?</h2>
<span style="font-weight: 400;">Let’s see step by step what you did there.</span>

<span style="font-weight: 400;">First, you said "<em>which button is the best".</em> What does "the best" mean?</span>

<span style="font-weight: 400;">In the case of selling flowers, you want to see how many people came to your website. Some people may go away after taking a look, but the others will buy a bouquet.</span>

<span style="font-weight: 400;">The ratio of everyone vs buyers is called "<b>conversion".</b> And it’s a <b>metric</b> you, as a business owner, probably would like to use to understand if the new button color is better or worse.</span>
<h3>What metrics make sense to use?</h3>
Conversion is one example, but is A/B testing useful for something else?

Of course it is.

<span style="font-weight: 400;">It all depends on <em>what your business wants your customers to do</em>. If you want to collect emails to reach your customers later, then the number of emails submitted would be the best metric. If you need people to log in, then the number of log-ins. If you want them to post a photo, use that. Anything you need to do, and you can measure, you can use an A/B test for.</span>

Here are some <strong>business metrics</strong> you can consider:
<ul>
 	<li>Conversion</li>
 	<li>Click-through rate</li>
 	<li>Log-ins</li>
 	<li>Sign-ups</li>
 	<li>Bounce rate</li>
 	<li>Submitted content (photos, articles, reviews of your product).</li>
</ul>
<h3>What additional metrics can I use?</h3>
<span style="font-weight: 400;">To better understand your customers' behavior on your website or in your app, you can use a number of additional metrics .</span>

<span style="font-weight: 400;">Log and measure everything you can!</span>

<span style="font-weight: 400;">Then you can analyze this data and see some interesting stuff. For example, if you notice that people who <em>scroll down</em> the page more than the others, end up buying a flower more often. Great! Next time, when changing something on the page, you’ll want to watch how it affects this behavior. If the change is going badly, and people don’t scroll down a lot, but close the page right away - you know why. Now you can start thinking: how does my change prevent people from looking at my beautiful flower list and scroll down.</span>

Here are some metrics you can use to understand the <strong>behavior</strong> better:
<ul>
 	<li>User clicked/hovered/interacted with your element</li>
 	<li>User clicked on a <em>different</em> element - to check if your new element draws user's attention too much</li>
 	<li>Time the user spent on the page</li>
 	<li>Scrolled down</li>
 	<li>Looked through the list</li>
 	<li>Filled a field</li>
 	<li>Selected an item; or selected several items.</li>
</ul>
These metrics are not <em>business</em> metrics - meaning that in the end your business doesn't care so much about where exactly users click. You want to look at these metrics to understand your customers better, with the ultimate purpose to drive your <em>business</em> metric.
<h2>Building a hypothesis</h2>
<span style="font-weight: 400;">But why did you change the button in the first place? Most likely, you had a good reason in mind, something like last trends in the industry, making the button aligned with the rest of the design, making it more visible or less aggressive? This is called a <b>hypothesis</b>.</span>

A hypothesis can be<span style="font-weight: 400;"> just one sentence, or it can be much longer. In any case, the general structure is similar to this:</span>
<blockquote><p>If I implement X,</p>
<p>the users will do Y,</p>
<p>and it’s good because Z.</p></blockquote>
So, our hypothesis for the button in the flower shop may look like this:
<blockquote><p>If I paint the button green, instead of red,</p>

<p>the users will find the button more easily and click on it more,</p>

<p>and it’s good for my business because they will buy more flowers.</p>
</blockquote>
I'm going to try another hypothesis for this article:
<blockquote><p>If I add a cat to my article,</p>
<p>the readers will say AAAaaw,</p>
<p>and it’s good because they will remember my article.</p></blockquote>

<img src="https://thepracticaldev.s3.amazonaws.com/i/rkj2ozew1xsfc9vjusgo.jpg" alt="Two photos of cats for the readers to look at and say &quot;aawwww&quot;"/> <i>Cat on the left of cat on the right? Please say AAaww.</i>

<h2>In case of success</h2>
<span style="font-weight: 400;">Let’s call it "<b>success"</b> when the new color is <em>better </em>than the other. (Note that I’m saying "<em>better</em>" here - remember that it means "better on your business metrics".)</span>

Right, so you’ve changed your button color, and saw the new color winning over the other. What’s next?
<ol>
 	<li><span style="font-weight: 400;">First of all, you <b>celebrate</b> your achievement! Bring chocolates or a pizza to your team, congratulate them, they've done a great job and deserve some praise.</span></li>
 	<li><b><span style="font-weight: 400;">Where else can I apply</span></b><strong> the same approach</strong><b><span style="font-weight: 400;">?</span></b>
You want to start thinking what <i>other </i>red buttons you want to paint green. Now you have a strong case for doing so, right?</li>
 	<li><strong>Continue</strong> on topic.
You might start thinking: since my customers react so strongly to this softer button color, this must be a very important button. Maybe there is another color I could try <b>at the same place</b>. How about yellow? And maybe a little flower icon on it?</li>
<img src="https://thepracticaldev.s3.amazonaws.com/i/0h2mvd584l0qia1o4y8i.jpg" alt="The website mockup with the yellow button with a little flower icon." width="300" height="177"/>

</ol>
<h2>In case of failure</h2>
<span style="font-weight: 400;">Now, let’s talk about the failure. Let’s call it "<strong>failure</strong>" when the new color is performing the same or <em>worse</em> on your business metric.</span>

<span style="font-weight: 400;">Don’t worry! It didn’t all go to waste. You learned a lot about your customers. Analyze the results you have and learn from them. Let’s say you tried blue color - and the customers hated it, the flower sales went down. Well, that’s unfortunate, but let’s stay optimistic and use this opportunity to learn. Next time you won’t use it anymore, when adding a new button somewhere on the same page.</span>

And don't give up! You just learned something about your customers: since they react so strongly to this button color, this must be a very important button, and we can have a great effect changing it. So how about we try something else with it? If they like the color as it is... how about making it bigger?

And if they don't react at all, well, you just learned not to waste your time here anymore. Go on and experiment on something else.
<h2>Learning is the most important thing</h2>
Just to remind you how important it is: <em>learning is the most important thing</em>.

Why is that?

You are never sure if your experiment will be successful. If you knew for sure, you wouldn't bother with running an experiment. You would just implement your feature without any ado. (There is nothing wrong with this, many businesses do this, and it is perfectly fine depending on your domain area and you customer requirements.)

But since you decide to run an experiment, you don't know the result.

And whatever the result is, you learned something about your customers. They don't like red buttons? Perfect, next time you won't paint another button red. You just learned this important thing for the price of some lost purchases, but now you are informed and won't lose any more purchases in the future.

Long-term you will value learnings about your customers a lot.
<h2>Repeat</h2>
In both cases, whether it is a success or not, you learn and repeat. Learn the most you can from all metrics you’ve been collecting and apply your new knowledge: repeat your experiment in the best way possible.

Try several times. If you have several unsuccessful tries, you will be pretty sure there is nothing to catch, and move on to another idea. But it's too early to give up after just one try.

So, I am going to <em>repeat</em> my cat experiment again.

<img src="https://thepracticaldev.s3.amazonaws.com/i/fmokcmn25oyymhts9w7e.jpg" alt="Another cat photo" />

<h1>5 Steps to a good experiment</h1>
<span style="font-weight: 400;">So, let’s go through the whole process again, to recap.</span>
<ol>
 	<li><span style="font-weight: 400;">First, you build a <strong>hypothesis</strong>. "If I paint the button green, my customers will buy my product more often because green button is more visible".</span></li>
 	<li><span style="font-weight: 400;">Then, you choose <strong>metrics</strong> you’re going to use. Conversion, number of log-ins, sign-ups, etc.</span></li>
 	<li><span style="font-weight: 400;">After that, you <strong>implement</strong> the experiment and <strong>run</strong> it.</span></li>
 	<li><span style="font-weight: 400;">Then, you <strong>analyze</strong> the results. Success? Great, I will apply this elsewhere. Failure? Great, I just learned.</span></li>
 	<li><span style="font-weight: 400;">And <strong>repeat</strong>.</span></li>
</ol>

<h1>And when not to run A/B tests?</h1>
Check <a href="https://dev.to/ice_lenor/when-not-to-ab-test-4cc7">this write-up</a> to know what features are best to be implemented without A/B tests.