Choosing Between A/B and Multivariate Testing

So you're ready to start doing conversion optimization. Some people will tell you that you should be doing A/B testing. Others will tell you that A/B testing isn't enough and you should do multivariate testing instead. In this article, I'll explain the differences between A/B and multivariate testing. And help you decide what's right for you.

1. What is A/B Testing?

Let's start with the Wikipedia definition of A/B testing: “Two versions (A and B) are compared, which are identical except for one variation that might impact a user's behavior.” In other words, with A/B testing we can test two identical versions of a web page, except for one variation e.g. color of a button, to understand user behavior. ab-testing-overview

In the above example, the A/B test is designed to test which button drives more clicks. Everything else on the two pages is identical. 

Pros of A/B Testing

  • Simple to design and setup experiments
  • Does not require a lot of traffic to get reliable results
  • Experiments can be completed fairly quickly

Cons of A/B Testing

  • Limited number of changes can be tested at a time
  • No insights on how different variables on page impact each other

2. What is Multivariate Testing?

Let's start with the Wikipedia definition of multivariate testing: “Multivariate testing is a process by which more than one component of a website may be tested in a live environment.” In simpler terms, with multivariate testing we can perform multiple A/B tests at the same time. multivariate-testing-overview In the above example, the multivariate test is designed to test two components on the page (headline and button text). With just 2 headlines and 2 buttons, we now have 4 variations of pages that we need to test.

Pros of Multivariate Testing

  • More powerful way to test page changes or re-designs
  • Provides insights on most effective combination of variables

Cons of Multivariate Testing

  • More complicated to design and setup experiments
  • Requires a lot more traffic to get reliable results
  • Experiments take longer time to complete

3. Choosing Between A/B and Multivariate Testing

So the question is, how do you choose between A/B and multivariate testing for your needs? Our opinion is that you should keep things as simple as possible. Develop a solid A/B testing plan (with well reasoned hypotheses) and focus on a number of experiments quickly and frequently.

Once you feel that you've exhausted your A/B testing, then you should consider multivariate testing provided that you have enough traffic to be able run such experiments.  

Image: BaneStudios

Leave a Reply

Your email address will not be published. Required fields are marked *