When optimizing a website’s conversion rate, A/B testing usually comes up first. Create two versions and see which performs better. Sometimes testing more alternatives proves necessary, for which multivariate testing performs well as a method.
So which to choose, multivariate testing or A/B testing? How do they differ? When does one work better? The following case studies will describe the test methods and compare them to answer these questions.
A/B testing serves well when it comes to major design decisions or in the end of the design process when you need to refine the layout of a given page.
The aim of A/B testing (aka split testing) is to compare two different versions of the design to see which one performs better. Each user sees just one design (A or B), even if they update the interface. This way the same number of people will view each version and you can measure which version worked better.
You can test many things with a split test: different images, call-to-actions, layouts or copy.
It’s important to make sure there is just one design element that differs in the two variants, so you will know what caused the better results. To make it more scientific: we have only one variable here, compared to multivariate tests, where we can change many things at once.
Running an A/B test might look easy, but you have to carefully plan your test to get the right results you can trust. So we collected the 5 things you should know when you prepare your tests.
Certain apps allow you to execute A/B testing where the application randomly shows the two versions to visitors. You can use Optimizely, Unbounce, Visual Website Optimizer, or Google Analytics Experiments. Some of them have powerful visual editor features, so you don’t even need to code to run experiments.
If you don’t know how big your sample size should be, use Optimizely’s sample size calculator.
The Smartwool webshop tested their category page. They had two design versions. One of them was more visual, with a nice picture at the top of the page, presenting individual products playfully, in different sizes. The other followed traditional e-commerce conventions, presenting the products in a rectangular grid, in similar size, and the pages started with the product immediately.
During the test, they wanted to know which induced more shopping (in dollars) in visitors. The traditional and slightly boring version won. The average income increased by 17.1% per visitor, not a bad result considering only the category page changed.
Note they have measured the end-result of the whole shopping process. Perhaps the large photos on the category page led to more clicks through for individual products, but a harder-to-browse page let fewer people find what they wanted, and therefore they purchased fewer products.
At least this is what we would assume, but as we only see the numbers, we don’t know for sure why this design won. Besides the quantitative A/B tests it is useful to run some qualitative research as well (like user tests) to understand why certain designs work better.
It’s also worth noting that this experiment was not perfect, as two different things were changed simultaneously. So we don’t know whether the header image or the product list layout caused the difference in the results.
Read more about how these UX research methods fit into the product design process in our e-book: Product Managers’ Guide to UX Design.
Multivariate testing requires a bit more statistical knowledge from the research team but is a valuable tool when you are in the early stage of experimenting with many different ideas to try.
As per Optimizely’s definition, “multivariate testing is a technique for testing a hypothesis in which multiple variables are modified.”
Although data collection takes more time to be statistically significant, once you have the results, it’s easier to implement multiple changes simultaneously based on the data.
Multivariate testing looks at different combinations of at least two variables, so you can measure more design elements in all possible variations. These can be, for example, smaller changes on a website (e.g. an image, button, or text change).
This testing method can also grant insights into how the different variables interact with each other. You can examine which copy and design combinations helps more in reaching the website’s goals. You can read more about UX copytesting here.
In this case, the control group plays an even bigger role when analyzing the results, so don’t forget to define it!
You can test it the same way as in the case of A/B tests – the webpage will show the different versions to the same amount of visitors.
Our Product Design book covers many case studies as well as the following example. In this multivariate test, we had to change things on the search result page. This included the call-to-action button copy with three different versions and the travel agencies’ logos.
At times we displayed the colorful logo and at others, we just put the company name in plain text. We created all the possible combinations to illustrate the impact of the two changes again. One change had three versions and the other two, making a total of six.
Consequently, both versions with the logo and without had three versions each of the call-to-action texts. See them below:
Displaying them all at the same time immediately shows the problem with the multivariate tests. Testing six versions simultaneously and quickly requires quite a few visitors.
Low-traffic sites do not work well with multivariate tests. We can only test two versions or wait a lot for the results. Either way takes too long.
The following diagram shows the test’s result:
The winning version generated five times more leads, quite a good number. This version has the same call-to-action text as the original, so the copy (or at least the new versions) did not have a huge impact on the result. In contrast, taking away the logo resulted in an increase of at least three times higher.
Later we suspected this might have happened because people will not visit the travel agency’s own website and connect with them there if their logo does not appear. This remained only a hypothesis since the A/B test reveals the truth but does not provide answers to ‘why?’ questions. Moderated in-person user tests can.
Here at our UX agency, we use a lot of both A/B and multivariate testing. We collected the most important differences and present them in the table below.
A/B and multivariate tests really shine in showing which solution works best in a live environment, during real use. Most product development teams run A/B tests continuously. These can tell us exactly which solution works best, but it doesn’t reveal why.
For this reason, neither A/B tests nor multivariate on their own ever suffice, always conduct qualitative research, e.g. in-person, moderated user tests.
This UX research case study tried both A/B testing and in-person testing. See how they performed.
Have you used any of these methods? What do you think about multivariate testing vs A/B testing? We would love to hear about your experiences, please share them with us in the comment section below.
UX studio has successfully handled 250+ collaborations with clients worldwide.
Is there anything we can do for you at this moment? Get in touch with us and let’s discuss your current challenges.
Our experts would be happy to assist with the UX strategy, product and user research, UX/UI design.
It's hard to find any product designers nowadays who don't use or at least don't…
As we step into 2024, the significance of investing in user experience (UX) and user…
Are you interested in how eye-tracking in UX research can be leveraged to gather reliable…
In this episode, we have a conversation with Sándor Zelenka and Benedek Göbölös as they…
Current generative AI tools can come up with videos and images within minutes. We still…
Even if you are far from being a football fan, I’m sure you’ve heard of…