Toolbar Paid Search Landing Page Test
| VERSION B | |
How site visitors voted:
VERSION A (34%)
VERSION B (66%)
|
Actual Test Results:
Version B was the winner with 73.15% more toolbar downloads. But Version A was also a test winner, keep reading… Webtrends, an enterprise customer intelligence company, conducted a series of two tests for ALOT using Webtrends Optimize testing platform. The first was an A/B test to determine the best page layout. It resulted in a 21.13% lift in toolbar downloads. Version A was the winner of that test. Next, Webtrends ran a multivariate test using A/B test winner (Version A) as their control. The multivariate test produced 16 versions and measured the impact of page element tweaks including headline, button, security icons, and images. Version B pictured here was the final winner with a 73.15% conversion lift compared to the control. This test shows that optimization shouldn’t invariably end after A/B testing is done. If ALOT had stopped there, they would have missed out on an additional 73.15% conversion lift! Have you ever done a multivariate test on a page that was already optimized with A/B testing? If so, please contribute a comment below. |
Interesting methodology of first testing to find the optimal layout then testing to find the right message, image, etc. I assume the message in the A/B test was the same the only variable was the layout?
Hmmm – bit of a misleading twitter post. You aren’t comparing MVT vs. A/B test here so I was disappointed to read the article is really nothing about the merits (and downsides) of each. I personally think A/B without MVT is prone to all sorts of misleading interpretation, especially when tests are rerun with different traffic mixes (A wins one week, B wins next week). I think badly designed versions of both can be misleading for marketers so it is less about the method and more about what (and how) you test.
Optimize is an amazing toolset. The ability to actually see the dependencies on elements on each other is invaluable for page optimization. Great test!
I’m guessing that de-emphasis of a ‘toolbar’ also helped, especially for those of us who were toolbar-addicted back in the early days of the web.
@Bobby Hewitt
They tested layout, yes, and a few page elements in the original A/B test.
Natalie Myers
Senior Reporter
WhichTestWon.com
Re: Craig Sullivan, well, the samples we showed and measured results from were the winner of an A/B test vs the winner of a multivariate test, so my Twitter was on target although I can see how it might be thought misleading.
If you’re interested in a more in-depth look into multivariate vs A.B testing (and how to use the two together sequentially as well as traffic source issues you mentioned above), check out our free 20-page PDF which features info on this starting on page 8 at http://whichtestwon.com/wp-content/uploads/2009/12/WhichTestingService.pdf
Thanks,
We do use both and particularly A/B for lower traffic sites (we have 32 of them) to at least get some initial direction in terms of further areas to test. A/B in our case helps to validate larger site multi-variate tests being ‘exported’ in terms of learnings to the other web properties. I also use A/B when I want to break out of over polishing multi-variate test patterns and find a new direction or bolder/more radical test designs. Just out of interest, we should talk soon about two tests we’re doing – how does site performance affect global conversion rates (across these 32 sites) and also one singular test on body language (i.e. we only test variants of a persons posture, uniform, eye gaze).
Simple and personal. Not surprised.
Vortex, turbulence, emotions, urgency…. It’s all in Version B.
Abe