Glossary

 Email This Page    |      >  SHARE ON TWITTER

Term: A/B Testing

Anne’s Definition: Also known as split testing. This is a proven, scientific testing method that’s been used successfully by direct response marketers since the 1930s. Now it’s used extensively on the Web as well as email, telemarketing, TV, postal mail, and other channels.

Online, the “A/B” refers to the way traffic is split so that every other person coming to a page sees a different version of the page being tested. You don’t have to split traffic 50/50 always, just split enough visitors to get statistically valid (aka conclusive) results for each test creative. So, if you had 100,000 visitors to a page, perhaps only 20,000 might be served a particular test version of that page. What’s critical is that all visitors come from the same general traffic source at about the same time. You don’t want to serve 80,000 people one version and then switch over and serve the following 20,000 the second version because time of day (among other external factors) can affect results.

To reach conclusive results about your changes, in A/B testing either you can test changing a single element on the page (while everything else remains the same) or you can test an entirely different page. The former gives you nice clean data about how much a particular element may affect response rates. That element might be price, button size, headline copy, form length, etc. The latter (changing multiple items on the page) will tell you which page did better overall; however, you will have NO IDEA which of the elements on the page were responsible for what part of the lift. (You need to expand to multivariate testing for that data if it’s worth the extra effort and cost to you.)

Sometimes you’ll see an “N” as in A/B/N testing. The “N” refers to “Nth name,” meaning you can run additional test panels at the same time – either each one completely different or each one varying in just one element versus the control — if you have enough traffic to support them. For example, you might want to test two new prices against your control price. You’d have three pages being tested simultaneously, each to an “Nth” percent of incoming visitors equaling a valid sample size.