6 tips for significant A/B tests

If this is your first time carrying out A/B testing, you should start with relatively simple tests, changing only small elements like graphics or buttons. A simple A/B test with only one variable is implemented quickly and delivers accurate and reliable results.

Multivariate tests are often too complex to start with. This is because they have a higher error rate, and they might deliver falsified results. So the key is to start small, as even little changes can offer big results later on. Run multiple tests in a row, and if each causes a small improvement in the conversion rate, these positive effects will add up in the end.

Time also factors into A/B testing. It makes sense to run a test for at least a week, so that you can include any weekday fluctuations. In addition, any abrupt positive or negative impacts can be evaluated realistically over the course of a week. A typical error when it comes to A/B tests is impatience. If you come to conclusions too quickly, you may miss some interesting results. For this reason you should evaluate tests above a statistical significance. In order to check whether results are significant, you can use the free A/B Split Test Significance Calculator.

Hypotheses are the basis of each A/B test. At the same time, it’s challenging to formulate these and record them. Efficient A/B testing is based on clear hypotheses, where you can work out exactly which influences have had changes on user behaviour. Whether your hypotheses are confirmed or disproved in the test, you gain more valuable insights into what you are testing if they are well thought out.

A/B testing not only helps to increase profit and conversions, but it is also a survey of potential customers: it helps you get to know your audience better. Any knowledge gained about the target group is helpful, as it can be used in other steps of the targeting process. As a result, it brings you one step closer towards winning over and retaining customers.

It’s possible for A/B tests to run parallel on different subpages at the same time, but this can lead to erroneous results. It’s recommended to test one change after the other, since this allows you to optimise the website one step at a time. Multipage experiments can be useful if you want to change numerous elements at once. Visitors will either be shown the new page or the old page. Even if you’re just making really small changes on pages, it’s still important that a similar number of visitors are shown each page. Yet it tends to be more beneficial to your results to modify one element after the other.

A possible scenario: after testing a landing page, you find out that variant B scored better in terms of click and conversion rate than variant A. The conclusion would be to keep variant B and get rid of variant A, but why not go a step further? If you use additional web analysis tools (such as Google Analytics) alongside A/B testing, you can segment the test group and differentiate, for example, between new and existing customers. The result could be that regular customers react more positively to version B, and that new customers prefer version A, so taking out version A would be a mistake. With added web analysis, you get a better view of the test results and, as a result, can draw more specific conclusions.

Resources are sometimes limited with A/B testing – on one hand, the time you’re investing is limited, and on the other hand, so is the number of visitors accessing the site or individual subpages. In order not to waste these resources unnecessarily, you should prioritise what’s most important. It makes sense to decide on concrete goals before performing each test. Marketers often rely on the SMART principle. With A/B testing, your goals should be specific, measurable, attainable, realistic, and timely. You can read more about this principle here.

As long as you have realistic targets, the appropriate hypotheses and the proper testing tools, there are many benefits to successful A/B testing. When implemented properly, split tests are a great opportunity for optimising any online business. Even if it means you have to invest a lot of time and effort at the beginning, it’s definitely worth it: the result is an optimised and improved site that matches your target group’s needs.

More about A/B testing such as the pros and cons can be found in our guide.