Key Takeaway
An email A/B test only pays off when the winning numbers survive a second look. Sample size and the metric you choose decide whether that lift is real. Get either wrong and you roll out a subject line that never did anything.
The Report That Started the Argument
Priya runs email for a 10,000-subscriber list. Last month she split a subject line test across 20 percent of the audience. That gave her 2,000 people, 1,000 per version.
Version A landed a 21 percent open rate while version B hit 24 percent and the report flagged B as the winner. It told her to ship it.
She almost did. Then she asked one question: could a 3-point gap show up by pure chance? The answer changed her whole read of the report.
Sample Size Comes Before Cleverness
Every test has a smallest lift it can actually detect. Experts call it the minimum detectable effect. If your real lift is smaller than that number the test stays blind to it.
For a 22 percent open rate and 1,000 people per version, the smallest lift you can trust is about 5 percentage points. Priya cared about 3. Her test was built too small to answer her question.
That is the part most reports hide. A dashboard will happily show a green arrow on a gap the math cannot support.
Quick rule: to catch a 3-point lift from a 22 percent baseline you need roughly 1,500 subscribers per version. Priya tested 1,000 and still called it.
The Significance Formula in Plain Terms
Significance is arithmetic. For two equal groups the test statistic looks like this:
z = (p1 – p2) / sqrt( p(1-p) * (2/n) )
Here p is the pooled rate and n is the size of each group. Priya’s pooled rate is 22.5 percent. Plug in her numbers and z lands near 1.61.
You need z above 1.96 for 95 percent confidence. Her test fell short and the 3-point lift sat inside the noise.
Opens Are a Fuzzy Referee
Open rate has a tracking problem. Some mail services now load tracking pixels on the user’s behalf. On many lists that inflates opens before a human ever looks.
So a subject line test judged only on opens measures a mix of real interest and automated pings. It gets shakier when the lift is tiny.
Priya saw her 3 points and felt clever. The honest read was that automated pixels moved while the readers stayed put.
Remember this: an open is a guess, but a click needs a finger, so judge subject lines on the harder signal.
Let Clicks and Revenue Settle It
Track click-to-open rate to see if the subject line attracted the right people. Then track revenue per subscriber to see if the whole send made money.
In Priya’s test version B pulled 240 opens against A’s 210, while clicks came in at 18 for A and 19 for B. Strip out the open inflation and the two versions look the same.
No real winner and no rollout. She just saved herself from rewriting every subject line around a coin flip.
Why a Bold Test Beats a Tiny Tweak
Small lists cannot prove small lifts, so stop chasing them and test a completely different angle instead.
Swap a plain text subject for a short question. Try a promo hook against a curiosity hook. Big swings produce big gaps and big gaps clear the bar even on a modest list.
If your list only holds 2,000 people, plan for tests that can move 10 points.
Checks Before You Crown a Winner
Run these before you ship.
- Did each version get at least a few hundred conversions, not just sends?
- Does the lift clear 95 percent confidence or is it inside the margin?
- Was the split random and sent at the same hour?
- Does the winning metric connect to revenue, not just vanity?
What to Do Next
Before your next test write down the metric and the minimum lift you care about. Then size the groups so the math can actually find it.
If you cannot fill those groups, skip the test and pick the subject line you believe in. One clean decision beats ten shaky ones.
A readout is a filter. Use it to stop shipping losers with confidence.
Read the full guide: Email Marketing Reports: How to Turn Numbers Into Next Steps