Will Your Test Reach Conclusive Results? Try Microsoft’s Free Power Calculator for A/B Tests

Most marketers don’t know if the test they want to run will get conclusive results. To be honest, I didn’t even know there was a way to find that out BEFORE running a test until I spoke with Thomas Crook, a member of the Experimentation Platform Team at Microsoft.

He recently emailed me the latest version of a Power Calculator for A/B Tests. (The tool was mentioned in a ‘Top 7 Testing Pitfalls’ webinar, and since been improved.)

Here’s how it works: (Basically just enter the requested data into the fields.)

Field 1. Conversion you want to measure (they call it ‘Metric’)

Three choices:
event count = clicks, page views, etc.
averages = average dollar spend, etc.
conversion rate = % of visitors who take a desired action

Field 2. Unique monthly visitors

Field 3. The expected number of events, expected averages, or expected conversion rate

Field 4. % lift that would make the test worthwhile for your company (could be as low as 2% if you’re a large company with lots of traffic)

Note: Field 7. ‘Power to Detect’ is the rate of conclusiveness you want. (95% is ideal)

What it tells you is:

1. The amount of traffic you need to get conclusive results (in other words if you don’t have this much traffic, you won’t get the desired outcome)

2. The number of days it will take to get the conclusive results you want (Note: If it tells you it will take longer than 6 weeks, you might need more traffic or to tweak something else such as Field 4. Rule of thumb is a test that needs to runs longer than 6 weeks really isn’t worth doing.)

“People think they can run a test for 3 months, but then external factors can have an effect,” Crook said. “The world changes over time.”