AB Testing, What is it?
For the past couple of months, I have been part of a team at my company whose mission is to improve the conversion rate on our eCommerce site through ab testing. The first thing to know is what are AB tests, why companies use them, and the best testing processes.
What is A/B testing?
A/B testing is an experimentation process where two or more variants are compared to determine which variable is more effective.
For improving a website we need to compare how new features stack up to the original. We do this by running ab tests on the website.
There are 4 Different Testing Types
- A/B is a test that splits traffic equally between a control and a variant.
- A/B/n is with more than two variations. This type of test is great for implementing more variations of an idea but a downside is this requires more traffic because the traffic is split among more variants.
- Multivariate is like the A/B/n test, in that they test the original against variations, but each variation contains different design elements. This is best used for testing layouts and if elements interact well together. But note this needs lots of traffic for this type of test to consider using it.
- Bandit Algorithms is an A/B/n test that uses an algorithm to update in real-time based on the performance of the variants 🤯! This prevents you from losing conversion while testing a possible bad-performing variant. This is best used for short-term campaigns and headlines.
Most companies prioritize A/B testing due to testing significant changes and they are the easiest to run.
AB Test Process
Research
The first step is to research what can be tested on your website. Below are straightforward ways to come up with ab test ideas 💡
- Technical analysis — Navigation through your site and take note of any bugs or performance improvements. Bugs and slow pages are conversion killers. Make sure your site works with all browsers and devices. These are quick wins and prove highly profitable returns.
- Web analytics analysis — Check all analytic tracking is working and take note of what pages and elements have been interacted with the most. This will let you see where you need to improve and also what changes can provide the highest returns.
- Mouse-tracking analysis — Observing how users interact with your website is another great way to understand your user and come up with experiment ideas. Watch user sessions, heat maps, and click maps. My company uses Clarity which is free!
- Qualitative research — Talk to your customers! This way there is no guessing what they are thinking when they navigate through your site. Set up customer surveys and interviews.
Prioritization
After the above, you should have a ton of ideas! This can get overwhelming not knowing what test to prioritize. Believe me, I have been there.
Ask yourself the following questions to insure your test addresses at least one of the following…
- Is it addressing an issue discovered through user testing?
- Is it addressing an issue discovered through surveys or interviews?
- Is the theory supported by mouse tracking, heat maps, and/or user session replay?
- Is it addressing insights found through digital analytics?
Experimentation
Once you have a test running it is very important to avoid misreading or producing false results/conclusions.
The most common mistake according to CXL’s Testing Guide is don’t stop a test early just because it reaches statistical significance. Initial results usually vary wildly in the first few days of testing. To get accurate results you need to have enough data to read.
To avoid false results predetermine a test size through this CXL Tool and run the test for the full determined weeks, usually at least two business cycles. My team also uses this significant calculator to check if we have gathered enough data to end an experiment and appoint a winning variant.
It is also important to pick the right technical tools for your company to run tests. We are currently using Optimize which is a client-side ab testing service. Optimize is sadly being sunset this fall but that is another blog post 😆.
One major categorization in AB testing services is server-side or client-side. Server-side tools render code on the server level. They send a randomized version of the page to the viewer with no modification on the visitor’s browser. Client-side tools send the same page, but JavaScript on the client’s browser manipulates the appearance of the original and the variation.
There are many AB testing tools to choose from and the most important thing is figuring out which one is the best for you!
Analyzing
When analyzing your data do not just look at your technical tools like Optimize for example. It is important to compare the results to other data-tracking resources. We use Segment and Sigma to track and analyze our data.
For example, your experiment has a control and a variant. Your variant might lose to the control of the overall results. However, the variant might beat the control in different data segments.
There are tons of data segments you could consider but the three most important are
- Desktop vs. Tablet/Mobile
- New vs. Returning Customers
- Traffic that lands on the page vs. traffic from internal links
It will benefit you to have your AB tests organized and archive the test results when a test is done. This way you can refer back to old tests and learn from them and make better/more valuable experiments going forward.
I hope this gave you a better understanding of AB testing and important things to consider during the whole cycle of an AB test.
I hope this helped you learn more about ab testing. Below are the articles and resources I mentioned throughout this post. Hope you enjoy and happy testing 🧪
