AB Testing Vs. Multivariate Testing

With many marketers taking a data-driven approach, A/B testing is becoming more commonplace and more widely understood.

However, when speaking with clients, we find that that less people understand the benefits of multivariate testing and the differences of doing a simple A/B test vs a standard multivariate test. When would you do one or the other?

In this article, I’ll review standard AB testing vs multivariate testing, as well as the cases in which you would use one or the other.

A/B Testing

A/B testing is simple – it lets you easily measure the impact of making simple changes on the site.

I’m sure most of you reading this are already familiar with an A/B test, but as an example:

In this example, the marketer is testing to see what the effect of the headline is going to be on conversion rate.

headline AB testing on smartphone

Nothing new here and this is pretty standard. You’ll run a test over a certain amount of days, determine that one site is the “winner” at statistical significance, and then push the winning design live.

Multivariate Testing

Multivariate testing is simply a sequence of AB tests.

It’s beneficial to run multivariate testing as opposed to traditional AB tests because you are able to test the impact of many variables rather than have to test every single variable at once.

Here is a very basic example:

Multivariate Testing Example

In the simple test above, there are a few things being tested:

  • Copy
  • Button

Because there are two variables, you need four different tests:

  1. Copy A, Button A
  2. Copy B, Button A
  3. Copy A, Button B
  4. Copy B, Button B

Do you see how simple this is? The word multivariate testing sounds complicated, but all you are doing is breaking down several variables into an individual A/B test.

More Complicated Example

Let’s look at Clarks, a real-life ecommerce site, and see the different variables that we could test:

Complicated Multivariate Test

First start by outlining, how many different variables could you test here?

  • Picture on the homepage: Size, placement, etc.
  • Free Shipping Offer: Should this be larger or smaller? Should you include details with free shipping or not include any additional information?
  • Navigation Links: What would be the effect of adding or removing these?
  • Number of Products on Homepage: How many products could you test?

How do you determine what variables to test?

The key to multivariate testing is not just laying out your tests in a logical sequence testing only variables that are likely to have an impact. This prevents redundancy, improves efficiency, and saves time (which is the same as making more money).

For example, in this case, probably the most important areas to test are shipping, homepage offers, and the number of products or what products there are on the homepage. You could run a multivariate test as follows:

  1. Larger Free Shipping Offer
  2. Fewer Products on Homepage
  3. Larger Free Shipping Offer + Fewer Products

Just like before, this is a multivariate test (sounds complicated) but really is just three A/B tests compared to the original.

In our experience, altering the # of products on the homepage as well as correctly positioning the free shipping offer are the two variables that most dramatically affect conversion rate – they are the “low hanging fruit” in conversion rate optimization for e-commerce companies and it is your best bet to target these first.

Practical Guide: When Do You Use Multivariate Testing?

There are a few practical limitations to running a multivariate test:

1. Do you have enough traffic?

Having enough traffic to justify multivariate testing is by far the biggest blocker to running these types of tests.

Each additional variable that you have will grow the number of tests exponentially – for example, with 2 variables you have 4 tests, but with 3 variables you have 9.

You then will need to divide the total amount of traffic that you normally need to reach statistical significance, and in most cases, this is what will block the test. So what should you do?

In practice, multivariate testing is as much an art as it is a science. The biggest question you need to ask yourself is, do you need to test everything?

While “test everything” is a standard mantra across CRO experts, a more practical term would be “test almost everything”. For example, we recently needed to redesign the footer section for a client we worked with and because we were under a tight deadline, we didn’t test the impact as we were confident these changes would not dramatically impact conversion.

2. How long will it take?

Before running any test, including multivariate testing, it’s extremely important to calculate the amount of time you’ll need to reach statistical significance.

Here’s an example calculation using AB Tester’s calculator:

Split and A/B Test

Let’s say that your website is getting 4,000 unique visitors per month. This calculation shows that you could comfortably test 4 different variations of your site and have significant results (with the exception of treatment 2) within 1 month.

Note that this calculation is not exact because the amount of time depends on how significant your results are. If you get a killer test result, then you’ll need less time to show that the test is performing better at statistical significance; if you don’t get a great test result, then you’re going to need more time.

As a rough rule of thumb we recommend that you overestimate the amount of time that you need for testing and give yourself at least 1 month to get results. Even when we are running multivariate tests on large sites, we prefer to run the tests longer than the time required to reach statistical significance just to be sure that the results “last” and are not going to dip when traffic sources change (more on this later).


Multivariate testing allows you to logically test the impact of multiple variables on a site. While sounding complex, multivariate testing simply breaks down several variables into their individual components for more simplified A/B tests.

Always remember that, before running any A/B tests or multivariate tests, you need to determine how much time you’ll need to get statistical significance. This will greatly help you in planning your tests.

Author Bio

Henry Black – Multivariate testing and A/B testing are my main interests and passions. When I am not blogging on the web I can be found testing landing pages for optimal conversion and helping mid size brands maximize conversion rate.