An Introduction to A/B Testing

An introductory guide to optimising your website to make the most of your traffic.

What is A/B Testing?

In web design and development, A/B testing is the process of comparing two versions of the same web page against each other to see which performs the best. Half of your traffic gets the first page (let’s call it page A), and the other half gets the second page (known now as page B). The version that convinces your visitors to perform your desired goal the most is the winner.

But what’s my desired goal?

Every website has a purpose. E-commerce websites want to convince their visitors to buy products. Marketing websites want their visitors to get in touch with them, usually so that they can sell their services. SaaS (Sales as a Service) websites want their visitors to sign up for their product (usually a monthly recurring subscription, like Netflix or Spotify).

How do I know which version is doing better?

In order to track the effectiveness of the test, we calculate a metric known as the Conversion Rate. A conversion is simply a visitor that performs your site’s desired goal (such as placing a successful order on an e-commerce site). If we take the number of successful conversions and divide them by the total number of visitors, we get the conversion rate! The version with the highest conversion rate is the better performing version.

Wouldn’t it be easier just to get more traffic instead?

It is much more cost-effective to convert a higher percentage of your existing traffic than to acquire new traffic. Even if you were to increase your traffic you won’t necessarily get more conversions with an un-optimized website. Small changes, tested on your existing user base, can lead to dramatically increased sales or leads for little additional cost.

divider line

Ok, you’ve convinced me… How do I set up these tests?

There are a few steps to a successful A/B test. Following these will provide the highest chance of discovering problems within your current website.

1. Look at your website’s existing data

If you have a form of analytics implemented on your website (Google Analytics is the most common), use the data from this to search for pages with high bounce or exit rates. 

2. Study how your visitors use your website

Look at what your visitors aren’t doing on your site. For example, they may not be filling in your enquiry form or clicking your call to action button on your homepage.

3. Use this information to form a theory

Using the information gathered in the first two steps of this list, form a theory about why the visitor is not performing the behaviour you wish them to. For example, 80% of people leave your home page straight away without taking any further action on your website. The issue may be that you are not giving them clear information. We now have a theory that we need declutter and add a call to action button.

There are many other areas that you may decide to test, such as headlines, images, adding banners/awards or adding links/changing link text. More complicated tests may include changing prices, free trial lengths or delivery costs.

4. Test the theory with an A/B test

Create the new version of your page with the change you theorised above. In this case, we may add a prominent call to action button to the homepage, labelled “Start your Free Trial Now”. We’ll keep the other version of the page the same, as it will act as our control group.

5. Check the results of the test

After the test has been running for a while, check the results. We had 80% of our visitors leave the site straight away; has this number decreased for visitors that were shown page B? Are we getting more new trial members than before? If there is no clear result, it’s worth going back to the third step and coming up with a new theory.
If there is conclusive evidence that the new variation has succeeded, you now have the evidence you need to make the change permanently on your site.

divider line

A/B Testing best practices

That was a simple example of how making a small change can potentially make a big difference to the usability of your website and increase the likelihood that your visitors will perform your desired goal. There are some small things to keep in mind when you’re carrying out your A/B tests.

  • Only make 1 change at a time.
    Making multiple changes on the same page during a single test may produce inaccurate results. Making a single change and analysing the data from that will ensure that the differences noted are related to that change and not a combination of other changes.
  • Run the test for a long enough period of time.
    The sample size of your data is very important when it comes to deciding if the change you theorised was effective. The larger the data sample, the more likely that it is to be accurate. To increase the amount of data within your sample size, run the test for a longer period of time.
  • Returning visitors should see the same version they were originally shown.
    To further improve the consistency of your data, you should ensure that a returning visitor is shown the same version of the page that they were originally shown the first time that they visited the website. Visitors being shown a new version of a web page that they only visited yesterday could potentially confuse them, especially if the element being changed is something like the navigation structure or controls.

At Mercurytide we believe in designing a truly fantastic web experience for customers that delivers results. That's why we have developed a range of web design services and e-commerce software to rapidly deliver everything you'd expect, but with a fully customised solution for your business.

If you have any questions about the points I've raised, or want a more in-depth chat about all things web design, please get in touch.