Ecommerce 404 Page Test
| VERSION B | |
How site visitors voted:
VERSION A (28%)
VERSION B (72%)
|
Actual Test Results:
Initial conversion data can be deceiving. The big-image 404 page version got only a 3.56% lift compared to the text-link-heavy page (which was the site’s original 404). But, visitors stayed on the site after seeing that image, spent an average of 73.62% more money! This test won a Gold Ribbon at our 2009 Testing Awards in the Ecommerce Site Test category. Amadesa, a conversion optimization and SaaS firm, conducted the A/B split test using the Amadesa Customer Experience Suite on client Mountain Equipment Co-op’s 404 page. This is the page visitors reached if the hotlink they clicked (perhaps an old link they’d bookmarked or some other error) didn’t reach an active page anymore. In the past, 63% of 404 page traffic bounced off the site. Amadesa’s goal of reducing bounces was a lot harder than expected. Nevertheless, their billboard image-style test added thousands of dollars to the bottom line immediately. That’s because, although both 404 pages ‘converted’ roughly the same amount of continued site shopping, the shoppers who’d seen the big image wound up spending about 73.62% more money than those who hadn’t. So, a warm-fuzzy branding image in place of a plain-vanilla 404 message actually propelled more sales. Anyone who doubts the power of branding on the Web can doubt no further. Time to test it! WhichTestWon.com is sponsored by: WiderFunnel, the A/B and multivariate testing firm that guarantees Conversion Rate Lift. Click here for info on improving your site’s results – risk free – today! |
Wow!
This test made me realize how images are important.
Thank you.
I’d argue that the “hero” image was not the key driver in the optimization, but rather the content and messaging is simply much more digestible than the site map link approach. Even more, the call to action is user defined as opposed to company-centric list of links. Think of it as the Yahoo! vs. Google homepage in the early 2000s. I have run a similar tests with 404, error pages, and an empty cart and seen similar improvements.
Great test (as always).
I think you’re missing the word “who” in first paragraph:
But, visitors [who] stayed on the site after seeing that image, spent an average of 73.62% more money!
Great article – expected the result but it was very unexpected to find the higher spend of 73%.
The article is a very heavy endorsement for everything you do. ie if you do not test you might just be a little bit slow.
Very cool test. And I did get it wrong, so I learned something
How can one conclude that the image is what improved the sales when there are so many other variables. Perhaps the amount of links was overwhelming and pushed people away. Perhaps the search box on the page with the image was more useful than a myriad of links.
What other testing was done to prove the “branding” is what improved sales? Simply saying that the page with the image created higher sales is misleading. If you wanted to validate that, you’d have to put the image on both pages and/or remove it from both pages and see what conversion does. Otherwise you have no “control” in this study.
Scott — You’re right, no one can ever know what precisely changed behavior from a test, only that behavior did change. In this case, I’m guessing the branding did because the difference in “conversion” responses didn’t happen on the page itself or immediately in reaction to the page, but rather, much further down the sales pipeline. (People had to pick another page to go to, then pick thing(s) to buy, and then get all the way through the cart.) That tells me the overall impact of that big image must have been profound to have stuck with them for so long. And that image was a branding image if I ever saw one!
Hi Anne,
Very true, it is one large branding image!
Don’t get me wrong, I do believe that branding impacts customer experience, but even in your explanation you mention “people had to pick another page to go to, then thing(s) to buy, and then get all the way through the cart.” From this, it seems more reasonable to assume that if the customer recovers (gets to the page they “want”) quicker, then they can likely buy the product quicker (pending the checkout isn’t a black hole.) So while the image may have stuck with them, if the next page(s) in their path are more “dead ends”, wouldn’t it be fair to think they’d be more likely to leave? So, even if the image made them feel good about the brand, the rest of the experience seems like it would have more impact on their behavior.
I’d love to see this study taken another step or two further in the testing to help us better understand the WHY of the results.
Thanks for sharing and discussing further with us Anne!
Wow – who would have thought there was a branding opportunity in the much-unloved 404 pages, which of “course” none of us ever strive to have!
My choice between version a and b wasn’t based on the image itself, so I don’t if branding is what helped sales. I chose version b because it did a better job of explaining the situation to the customer (3 reasons as opposed to 1) and it very clearly directed the customer how to solve the problem in a variety of ways. With the site map, I would have been frustrated, feeling that I need to hunt down a page that I might not even be able to find in the site map’s contents. The nice thing about the image in my mind was it put the focus on the text directions, thereby making the solution seem simpler and with less work on my part. Even though it is a beautiful shot, I would be interested if the company could get the same drastic improvements with a different photo or even a generic image.
I wonder why the old 404-page is still on line? Or are they still testing?
http://www.mec.ca/Sitemap/sitemap.jsp?type=404
@Robin, they’re probably using programming logic to manipulate their sitemap, I see the page just fine… but it looks like they are still testing. I saw a different variation: http://www.mec.ca/Sitemap/404_page.jsp
We should have an actual comparison as to what it was on those pages that caused the added conversion, or if it caused it at all.
But, if I were a consumer of the product, I would have loved page B simply because it reminded me of what I wanted. I know I want buy mountain equipment, but picture B placed that image in my mind and evoked the excitement in doing what I loved. It showed me what I was going to gain out of visiting this website, not just a bunch of words.
Or maybe I’m a simpleton. Pictures always do it for me
Was the conversion better than just fixing the 404s and sending the visitors to an appropraite landing page with exactly what was initially promised by whichever link they clicked?
Plus those are free links thus potentially more long-term traffic
Hi Anne,
This is a great little test.
I believe here it is a question of emotion too when considering the wording used on the different test pages. The site does appear to have taken that into account when using “!Oops, wrong track” (test: “!Oops, product not found”) and “We’re Sorry…” on the two types of error pages as discussed above.
Analysing the pages using attention heatmaps also shows how an image draws focus to the main message. In a recent blog post you can see what I mean:
http://www.stott.nl/2010/03/404-error-and-spend-more.html
Thanks for the continued insights your site gives.
I think the link page just confuses the person. Whereas the picture page makes them think they are still on the site and not a cheap looking link page for click through s.
Make a simple 404 and they will come! Or stay……
Very creative method for increasing conversions.
If I saw this landing page on one of my favorite sites I would feel like they cared about me by showing me a pleasing image instead of making me hunt through hundreds of links.
I really like that the tracking on this test made it all the way through to the measuring orders. Often, measuring a lift in opens or clicks is the end of it.