So you're ready to start doing conversion optimization. Some people will tell you that you should be doing A/B testing. Others will tell you that A/B testing isn't enough and you should do multivariate testing instead. In this article, I'll explain the differences between A/B and multivariate testing. And help you decide what's right for you.
1. What is A/B Testing?
Let's start with the Wikipedia definition of A/B testing: “Two versions (A and B) are compared, which are identical except for one variation that might impact a user's behavior.” In other words, with A/B testing we can test two identical versions of a web page, except for one variation e.g. color of a button, to understand user behavior. 
In the above example, the A/B test is designed to test which button drives more clicks. Everything else on the two pages is identical.
Pros of A/B Testing
- Simple to design and setup experiments
- Does not require a lot of traffic to get reliable results
- Experiments can be completed fairly quickly
Cons of A/B Testing
- Limited number of changes can be tested at a time
- No insights on how different variables on page impact each other
2. What is Multivariate Testing?
Let's start with the Wikipedia definition of multivariate testing: “Multivariate testing is a process by which more than one component of a website may be tested in a live environment.” In simpler terms, with multivariate testing we can perform multiple A/B tests at the same time. 
In the above example, the multivariate test is designed to test two components on the page (headline and button text). With just 2 headlines and 2 buttons, we now have 4 variations of pages that we need to test.
Pros of Multivariate Testing
- More powerful way to test page changes or re-designs
- Provides insights on most effective combination of variables
Cons of Multivariate Testing
- More complicated to design and setup experiments
- Requires a lot more traffic to get reliable results
- Experiments take longer time to complete
3. Choosing Between A/B and Multivariate Testing
So the question is, how do you choose between A/B and multivariate testing for your needs? Our opinion is that you should keep things as simple as possible. Develop a solid A/B testing plan (with well reasoned hypotheses) and focus on a number of experiments quickly and frequently.
Once you feel that you've exhausted your A/B testing, then you should consider multivariate testing provided that you have enough traffic to be able run such experiments.
Image: BaneStudios
