How YouTube Does A/B Testing And How YOU Should Do It

YouTube recently finished a huge mutivariate experiment (1024 variations) on their home page. They used Google Website Optimizer and found a variation that performed some 15% better than their previous homepage.

These are some of their key takeaways:

While we could have hypothesized which elements result in greater conversions (for example, the color red is more eye-catching), multivariate testing reveals and proves the combinatorial impact of different configurations. Running tests like this also help guide our design process: instead of relying on our own ideas and intuition, you have a big part in steering us in the right direction. In fact, we plan on incorporating many of these elements in future evolutions of our homepage.

What I wonder is, if you’re not YouTube and have billions of pageviews, if your site or app is just starting out, what and how do you do A/B testing?

Here are some suggestions:

• Setup some goals you might have and prioritize them. You could start with the sign-up conversion and optimize around that. Once you’ve improved it massively, you can start optimizing other events on your site or product
• Test for maximized learning. If you have to choose between a test that teaches you nothing about your users, although it improves something, and a test that teaches you something new about your users, chose the latter
• Do A/B instead of multivariate. A/B requires less traffic exposed to the test and is easier to setup, as you don’t have to figure out which variables are independent.

I’m sure there are lots more and that you also have your own suggestions. Shoot in the comments and let’s discuss.