Customer Survey Email Test
| VERSION B | |
How site visitors voted:
VERSION A (26%)
VERSION B (74%)
|
Actual Test Results:
(Click back to see Versions A & B) Version B got a whopping 51% more clicks than Version A. Campaign Monitor, an email software provider, conducted this A/B test in-house using its own technology. The test was sent to all active customers, which makes this response disparity even more remarkable because, after all, your customers have a vested interest in giving their feedback to you. This was a clean test — both versions were sent at the same time and used the same subject line, “from” name, and body copy. The only difference was the wording of the largest text link. Version A read, “Give us your best Campaign Monitor ideas!” Version B read, “Tell us what we can do better.” The more direct wording won. Kudos to the team for testing their call to action — which, aside from list and subject line, is often the most important element in any email. Have you tested your email calls to action lately? (Yes, even a “read more” link in an email newsletter is a call to action that can be tested and optimized for better response rates.) Let us know about your email tests! |
This makes a lot of sense to me — in my experience, direct language (especially in CTAs) works best. This is an important lesson for marketers — especially when you look at differences of 51% or more!
“Guys, size means nothing compared to response rate.”
What a silly thing to say. Would you rather have a list of 20 with a 80% response rate, or a list of 2 million with 10%? Both are very important to a good marketer.
Even my son aged 12 got this one right! Well done Thomas.
Goes to show how much words really do count!
hi, Anne, i love the site & have fun guessing on each test.
It would be great to see some indication of statistical significance on these. & on this one in particular I’d love to know whether open rates were roughly equal for the 2 emails (validates that the send was random), and to see the number as the difference in “open:click” rate, rather than just click.
Thanks again – look forward to the next.
dan
Interesting. I chose option A, but solely because I thought the recipients of this list would be more receptive to that phrasing. I can see a less email-interested group definitely choosing option B however.
I find these tests really interesting. I found the call to action A difficult to understand but didn’t have to think about B. The only thing I am curious about with these tests is the rigorousness of the statistics. For instance with this test how many responses were collected for each version and what were the response rates.
Perhaps the difference is not (only) just in the number of words or the directness, but in the difficulty of the proposed ‘tasks’. It’s so much harder (and carries a bigger risk of failure) to “give your best ideas” than to “tell what we can do better”.
Kevin – Let me put it another way. Unless your list is too tiny to test, it’s a heck of a lot easier/cheaper/quicker to bump your responses by 50% via testing than it is by growing your list size by 50%. Plus, as a very (perhaps too) generalized statement, larger lists tend to often get lower response rates than smaller ones as a whole. That’s because every list has a natural “sweet spot” of truly interested opt-ins, and no matter how many names you add beyond those, those other people will never care as much about your topic as the fans in your true sweet spot. So why do media buyers love big, big lists? Often it’s mainly for efficiency’s sake, not response rate.
I think a factor is the amount of work implied. Coming up with “ideas” vs. “tell us what’s irritating” = hard vs. easy. Shorter is better, but easier is also better.
This one makes alot of sense, and more of us got it right than usual. I think the combination of clearer/shorter text and a broader request made the difference. While I think predicting the better of the two was easy, I’m surprised at the level of increased obtained by B. Excellent test.
Dang it – I’ll admit I got this wrong. My reasoning was that the use of ‘Campaign Monitor’ in the CTA would make it more appealing to the audience, i.e., users of this service – and in that sense was more direct. Working for a large company, I tend to steer clear of statements like ‘tell us what we can do better’ because it is too vague and open-ended. You’ll get all sorts of responses that have nothing to do with your intent. So — I can’t argue with the results. I was wrong. But I’m guessing that in different contexts, the result would be different – just one more argument for test, test, test.
I second the request to have stats. I realize there are privacy issues, but perhaps at a minimum you could include total emails sent. I love this website!
Ideas? Hard. Opinions? Easy.
I second Roberta’s view. Also: “Give us your best Campaign Monitor ideas” says to me, give me something of yours free, which I’ll then proceed to go and make some money from. On the other hand “Tell us what we can do better” sounds like I might actually get better service as a result of contributing. On these grounds it’s not surprising that self-interest works better. Also impacting, I think, is the request in B better matches the headline while the A request better matches the body copy, and since fewer people read the body (I didn’t, initially) we have another reason for B to win.
Great demonstration. One thing to keep in mind is how subtle difference may also play into these scenarios and that we may have arrived at the right answer for an alternative or wrong reason (which may cause false positives in future tests). Specifically, I suspect a significant percentage of positive responses were generated not only due to positive, active language but… Many people are more apt to “tell” you something than they are to “give” you something. Giving may leave the reader with a feeling that he is relinquishing something where-as allowing them to tell something allows the receiver the opportunity to be heard. Again, the difference here is even more subtle than active vs. passive voice. I suspect I’m not the only one that would respond more positively to being asked to tell someone else how to improve rather than being asked to give away my best ideas. Both may have the same result but there is a difference physiologically between telling and giving.
My 2 cents.
If you really wanted ideas for Campaign Monitoring, then “A” would still be the better choice. A lot of general complaints would do you no good at all if your goal is to improve campaign monitoring, would it?
The overall point is valid – testing is beneficial, but I don’t think this test was necessarily well done. “B” is not necessarily a good option to “A”.
buy web traffic…
Thank you for the information! It will be of great value for my online endeavors and my marketing efforts. I will save it for future reference….