How To Address Internal Statements About Tests That Get Null Results

When two or more versions of a test page perform the same as or close to the original page and the test does NOT get an 80% or more confidence level, how do you communicate internally that these tests are still valuable?

That’s the question a site testing manager for a major ecommerce retailer recently asked us.

The site manager’s current internal statement regarding these kinds of tests was:

“There was no significant difference in customer behavior to meet a confidence level of 80% that one version outperformed another.”

Our response: “Often the biggest challenge for testing is managing the office politics around it. In this case, your challenge is a copywriting one. You don’t want to make it seem like testing isn’t a worthwhile thing to do. Instead, you can make it look like testing is risk management — a very safe and sane thing to do especially in this economy.

So perhaps revamping your internal statement to start out saying something like:

Good News: Testing shows [company].com is currently safe using either of these options. Neither one should perform significantly worse than the other.

We suggest you run with the option that meets any or several of these guidelines:
- Falls in line better with our branding
- Easier for in-house staff to manage the creative
- Strongly preferred by HIPPO

Notes: this test was measured to XX% conclusiveness. However market and brand conditions will change over time. If warranted, we recommend re-running this test in XX months. Any questions contact the testing team at…”