On August 1, we officially said goodbye to website optimizer and hello to Google Analytics Experiments. One of the most notable differences with Analytics Content Experiments is that A/B test pages are no longer always served 50/50. Google determines the percent of daily visits allocated to each page based on that pages statistical likelihood of success, a method whose principles are adopted from the idea of the “Multi-Armed Bandit”.

What is a multi-armed bandit? (And should I be afraid)?

There is no need to be scared, the multi-armed bandit is our friend! Traditionally used in machine learning, the idea behind the multi-armed bandit is to maximize return while conducting a statistically valid experiment. As the experiment progresses, the machine learns the actions (in our case, which page to serve) that will yield the highest return. It then allocates a higher percentage of page views to the page with the highest probability of delivering a payoff.

Example time! Let’s say I run an online lamp store and currently have a charming chimpanzee delivering my call to action. But I have this nagging hunch that a wise professor kitten would speak to my target audience better.

vs

My average conversion rate is currently a strong 20% and I don’t want to lose potential conversions, especially this close to the end of the quarter. But the more conversions the better, right?

Using the multi-armed bandit, I can test my hunch without sacrificing a large number of conversions. Google will look at performance metrics each day to decide the fraction of traffic that will be greeted by the chimpanzee and the fraction greeted by our wise kitty friend.

This means I will come out of the experiment with:

• Lower opportunity cost (lost conversions) due to experimentation
• Higher profit
• An answer as to which page is the winner of the experiment

If you are interested in technical details about the statistical validity of multi-armed bandit experiments, Google has written a great explanation that I dare not butcher by attempting to summarize.

Have you had any experience with Analytics Experiments yet? Who do you think would win the experiment, charming chimpanzee or professor kitten? Leave your comments below or find me on Twitter @CaseyDavenport.

Comments

This entry was posted on Wednesday, August 29th, 2012 at 9:22 am and is filed under Business. You can skip to the end and leave a response. Pinging is currently not allowed.

About the Author: Casey joined Search Mojo as a Junior Account Manager in May 2012. She graduated from James Madison University with degrees in Computer Information Systems and Marketing. Prior to joining Search Mojo, she worked on the Portal Development team at MSN Arabia and the Product Management team at Time Warner Cable. She enjoys running, reading, and eating tacos.