Most small business email tests are not experiments. They are two versions sent to a few hundred people each, a 4% difference in an outcome that varies by more than that on its own, and a confident conclusion. The discipline that separates a test from a coin flip is not complicated, but it does require doing arithmetic before you send rather than after.
Work out the sample size first
The standard rule of thumb for comparing two proportions at 95% confidence and 80% power is roughly 16 × p̄(1−p̄) ÷ Δ², per variant, where p̄ is the baseline rate and Δ is the absolute difference you want to be able to detect. Plugging in real numbers is sobering.
| Baseline | Lift you want to detect | Recipients per variant |
|---|---|---|
| 3% click rate | +20% relative (3.0% to 3.6%) | ~13,000 |
| 3% click rate | +50% relative (3.0% to 4.5%) | ~2,600 |
| 1% conversion rate | +50% relative (1.0% to 1.5%) | ~8,000 |
| 30% open rate | +10% relative (30% to 33%) | ~3,800 |
Calculated from the standard two-proportion sample size approximation, 16·p̄(1−p̄)/Δ². Verify with a calculator such as Evan Miller’s before running a test.
Read the first row again. To reliably detect a 20% relative improvement in clicks, you need around 13,000 recipients per arm, meaning 26,000 for the test. Most small business lists cannot do that on a single send, ever. This is not a reason to stop testing. It is a reason to be honest that small, plausible improvements are undetectable on your volume, and to test only changes large enough to show up.
Do not test subject lines on open rate
This is the single most common email test and it no longer works. Apple Mail Privacy Protection, live since September 2021, fetches your tracking pixel whether or not the message was opened, and Litmus put Apple Mail clients at roughly 46% of tracked opens in September 2025. Roughly half your measured "opens" are therefore identical in both arms regardless of what the subject line said, which mechanically compresses any real difference toward zero.
The consequence is not just that the test is noisy. It is biased toward declaring no difference, so you will conclude your subject lines do not matter when they do. Judge subject line tests on clicks instead, accepting that the click rate is affected by the body copy too, or segment your reporting to exclude Apple Mail opens if your platform allows it. Neither is perfect. Both beat measuring Apple’s proxy.
One variable, one metric, decided in advance
Change one thing so a result is interpretable, and write down before you send which metric decides the winner. The failure mode here is not laziness, it is the very human habit of finding the metric that makes the result interesting: the subject line lost on clicks but won on opens, so we report opens. Deciding first removes the temptation.
Pair the metric to the change. A CTA wording or placement change is judged on clicks. An offer change is judged on conversions and revenue per recipient, because a discount that lifts conversion while destroying margin is a loss. A send-time change is judged on clicks within a fixed window after send, not lifetime clicks, or you are measuring how long the test ran.
Stop when you said you would stop
Peeking at a running test and stopping it the moment it looks significant is the fastest way to generate false winners. The significance calculation assumes you looked once, at a predetermined sample size; checking repeatedly and stopping on the first favourable reading roughly doubles or triples your false positive rate. Set the sample size, set the end date, and look once.
For campaigns, most engagement arrives within the first day or two, so a 24 to 48 hour window is usually sufficient if the volume is there. For flows, the calendar is irrelevant: run until each arm has hit the number you calculated.
A test you cannot power is not a test. It is a decision you have already made, wearing a chart.
Keep a log, or you will run the same test twice
One row per test: date, hypothesis, the single variable, the pre-declared metric, the sample size per arm, the result, and whether you rolled it into templates. Two years of that log is the most valuable marketing asset a small business can build, because it is the only record of what is true about your specific audience rather than what is true in someone else’s benchmark report.
Key takeaways
- Calculate sample size before sending. Detecting a 20% relative lift on a 3% click rate needs roughly 13,000 recipients per variant.
- Subject line tests judged on open rate are broken by Apple Mail Privacy Protection and are biased toward showing no difference.
- Triggered flows accumulate the volume that one-off campaigns cannot, so that is where a small list can run real experiments.
- Declare the deciding metric before you send, and pair it to what you changed.
- Peeking and stopping early inflates false positives. Set the end condition and look once.
Related reading
Sources
- Sample Size Calculator for A/B tests, Evan Miller
- Apple advances its privacy leadership with iOS 15, Apple Newsroom (2021)
- Email Client Market Share, Litmus
- Analytics
- Email Marketing
- Optimization
Written by
Valter Brandt
Chief Marketing Officer
Valter Brandt is the Chief Marketing Officer of ThisCom, working with clients across the United States and internationally. He has led marketing strategy through the major shifts in social advertising, mobile, content marketing, programmatic media, and marketing automation.
All articles by Valter Brandt →
