Choosing Between A/B and Multivariate Testing

So you're ready to start doing conversion optimization. Some people will tell you that you should be doing A/B testing. Others will tell you that A/B testing isn't enough and you should do multivariate testing instead. In this article, I'll explain the differences between A/B and multivariate testing. And help you decide what's right for you.

1. What is A/B Testing?

Two versions (A and B) are compared, which are identical except for one variation that might impact a user's behavior. In other words, with A/B testing we can test two identical versions of a web page, except for one variation e.g. color of a button, to understand user behavior.

The A/B test is designed to test which button drives more clicks. Everything else on the two pages is identical.

Pros of A/B Testing:

Simple to design and setup experiments
Does not require a lot of traffic to get reliable results
Experiments can be completed fairly quickly

Cons of A/B Testing:

Limited number of changes can be tested at a time
No insights on how different variables on page impact each other

2. What is Multivariate Testing?

Multivariate testing is a process by which more than one component of a website may be tested in a live environment. In simpler terms, with multivariate testing we can perform multiple A/B tests at the same time.

With just 2 headlines and 2 buttons, we now have 4 variations of pages that we need to test.

Pros of Multivariate Testing:

More powerful way to test page changes or re-designs
Provides insights on most effective combination of variables

Cons of Multivariate Testing:

More complicated to design and setup experiments
Requires a lot more traffic to get reliable results
Experiments take longer time to complete

3. Choosing Between A/B and Multivariate Testing

So the question is, how do you choose between A/B and multivariate testing for your needs? Our opinion is that you should keep things as simple as possible. Develop a solid A/B testing plan (with well reasoned hypotheses) and focus on a number of experiments quickly and frequently.

Once you feel that you've exhausted your A/B testing, then you should consider multivariate testing provided that you have enough traffic to be able run such experiments.