A/B test sample size, significance and when to stop

An A/B test compares two versions of the same page or flow under the same conditions, and it measures one difference that you named before you started. It answers whether the gap you saw is larger than the noise your traffic produces. It doesn’t answer why the gap exists, and it doesn’t settle a small effect on a small sample.

What an A/B test answers and what it doesn’t

A test answers one question: given this traffic, is version B better than version A on the metric I chose. Everything else is a second question wearing the same data.

What a test answers well:

A directional call on a single metric, with a stated error rate.
Whether a change worth deploying shows up in behavior at all.
Which of two versions wins when the gap is large enough to see.

What a test doesn’t answer:

Why the change worked. The test reports the gap, not the cause.
Whether the change works in a different audience, season or channel. The sample came from the traffic you had.
What happens over months. Most tests run for days, and the novelty of a new layout fades.
Whether a small effect matters. That is a business question and not a statistical one.
Anything about a metric you didn’t track before the test started.

Somit Gupta, Pavel Dmitriev, Garnet Vaz and Dong Woo Kim, of Microsoft, published A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments at KDD 2017. The paper draws on thousands of experiments run across the company and lists twelve ways teams misread a metric movement. The authors state that a wrong interpretation can hurt the business by millions of dollars. The test is rarely the weak link. The reading is.

How to size the sample from the effect you want to detect

Four inputs decide the sample, and all four are declarations you make before the test runs.

The baseline rate. The conversion rate your current version already produces.
The minimum detectable effect. The smallest change that would still make you act. Evan Miller’s calculator defines it as “the smallest effect that will be detected (1-β)% of the time”.
The power, written as 1−β. The same page defines it as “the percent of the time the minimum effect size will be detected, assuming it exists”.
The significance level, written as α. Miller defines it as “the percent of the time a difference will be detected, assuming one does NOT exist”.

Miller’s rule of thumb for the count is n = 16σ²/δ², where δ is the minimum effect you wish to detect and σ² is the sample variance you expect. His sample size calculator applies the same logic to a conversion rate, with the baseline, the effect, the power and the alpha as inputs.

The cost of a smaller effect is traffic, and the cost isn’t linear. The formula carries it: n moves with the inverse square of the effect, so halving the effect you want to detect quadruples the sample. Miller’s calculator makes the scale visible. Its default example, a 10.2% baseline with a relative effect between 10.2% and 13.2%, returns 2,545 subjects per variation (Evan Miller).

The calculator also prints a gray band around the baseline, and that band is where most programs fail. Rates inside it won’t be distinguishable from the baseline. Optimizely’s glossary on statistical significance lists three reasons a test never reaches significance. The change was too small, the baseline rate was too low, or the team tracked too many goals at once. Each cause is a sizing problem and not a statistics problem.

What significance means and what it doesn’t guarantee

Significance is a statement about the noise, not about the size of the win. Optimizely’s glossary puts it as the measure of how unusual your results would be if there were no real difference and the gap came from random chance alone. VWO’s glossary states the same rule from the frequentist side: the result is significant when the p-value falls below the significance level chosen in advance, usually 0.05.

A significant result says this. If the two versions were identical, a gap this large would appear less than 5% of the time. It doesn’t say the gap is big, permanent or caused by your change (Optimizely).

That definition has three limits.

The threshold is a convention and not a law. VWO’s glossary calls 0.05 the suggested industry standard rather than a value derived from the data.
Significance is one metric wide. Optimizely lists too many goals as a common cause of tests that never reach the threshold. Check ten metrics at 5% each and a false winner is guaranteed by construction.
The number depends on the sample being fixed in advance. Evan Miller’s 2010 note states that assumption plainly and adds that it is the one most teams violate without noticing.

The two statistical schools also report different numbers for the same test, which is worth knowing before you compare tools. VWO’s glossary carries the frequentist reading below the 0.05 line, and Optimizely’s glossary carries the same threshold from the tooling side. VWO’s platform works in the Bayesian framework and reports the Probability to be Best with a 95% threshold, plus a Potential Loss figure. A frequentist p-value and a Bayesian probability aren’t the same quantity, and a tool swap can move a result across the line.

Why peeking and stopping early breaks the reading

This is the most common error, and the math explains it better than any lecture on discipline.

Miller’s note “How Not To Run an A/B Test”, published in April 2010, walks through it. The significance calculation assumes the sample size was fixed before the test started. If you instead say “we will run it until we see a significant difference”, every reported significance level becomes meaningless.

His example makes the size of the error concrete. Take a 50% conversion rate, and stop as soon as the test hits 5% significance, or call it off after 150 observations. If the change does nothing at all, the test still reports a significant result 26.1% of the time. That is more than five times the 5% error rate you thought you had bought (Evan Miller).

Every peek makes it worse. Miller’s table gives the reported threshold you would need in order to keep a real 5% error rate (Evan Miller):

Peek once: 2.9% reported significance.
Peek twice: 2.2%.
Peek three times: 1.8%.
Peek five times: 1.4%.
Peek ten times: 1.0%.

Repeated significance testing moves the error in one direction only. It creates false positives. It doesn’t hide real effects.

The first fix is a rule fixed before the test. Miller’s advice is to decide the sample in advance and to wait until the experiment is over before believing the dashboard. Watching the data is fine. Stopping on it isn’t.

A second fix exists for teams that need to look every day. Ramesh Johari, Pete Koomen, Leonid Pekelis and David Walsh wrote Always Valid Inference: Bringing Sequential Analysis to A/B Testing, the work behind today’s sequential methods. Evan Miller documents a simpler version in his note on sequential testing. The authors state that frequentist p-values and confidence intervals are wholly unreliable when users choose the sample size by continuously monitoring the test. They define always-valid p-values and intervals that stay correct at every look, and the method ran in a commercial platform across hundreds of thousands of experiments. If your tool reports a sequential or always-valid interval, looking every day is allowed. If it reports a plain p-value, it isn’t.

Duration, the calendar and what actually limits a test

Sample size isn’t a deadline. Two weeks of traffic and two days of traffic can hold the same number of visitors, and they aren’t the same test.

Traffic mix moves with the calendar. A weekday morning and a Sunday night bring different people with different intent. A test that runs from Monday to Wednesday measures a Monday-to-Wednesday audience, and the result belongs to that window and not to the month.

Google’s own tooling is gone, and that removed a default many teams relied on. Google Optimize and Optimize 360 stopped being available on September 30, 2023, according to the Analytics help page on the sunset. Google now points to integrations with Optimizely, VWO and AB Tasty. Firebase A/B Testing, which used the Optimize infrastructure for inference, moved to frequentist inference for experiments started on or after October 23, 2023.

Firebase’s own documentation states a position worth knowing, because it contradicts the fixed-sample rule. Its inference doesn’t require a minimum sample size before the experiment starts, and it recommends picking the largest exposure level you are comfortable with. The same page adds that larger samples raise the chance of a significant result, especially when the variants differ only a little. It points to a sample size calculator for the recommended count.

One more duration constraint comes from the tool rather than from the statistics. Firebase notes that a new Google Analytics audience can take 24 to 48 hours to accumulate users, and that new users are enrolled 24 to 48 hours after they qualify. A test that starts on a fresh audience spends its first day measuring a group that is still filling up.

That last figure is the real minimum. A tool that enrolls a new audience over two days can’t be read before the audience exists, and the count of subjects is what the sample size calculation was always about.

Statistical significance against practical relevance

A real effect can be too small to matter, and a significance test won’t tell you so.

Significance answers whether the gap is distinguishable from noise. It says nothing about whether the gap is worth the engineering time, the migration cost and the support load. A lift too small to pay for the build can still be significant once the traffic is large enough.

Nielsen Norman Group draws the line in Practical vs. Statistical Significance, and the numbers on that page make the problem hard to forget.

A 1.00% to 1.10% lift at 6,000 users carries p = 0.03. The result is significant by the usual threshold and irrelevant to the business (NN/g).
A 20.0% to 85.0% lift at 8 users carries p = 0.37. The result isn’t significant and the effect is enormous (NN/g).

The reading that works is a second calculation on top of the test. Take the observed effect, convert it to the unit your business already uses, and compare it with the cost of shipping. If the gain doesn’t clear the cost, the result is real and the change isn’t worth making.

Two habits keep the reading clean.

Declare the smallest effect that would justify shipping before the test, which is the same number you used for the sample size.
Report the confidence interval along with the winner. A wide interval around a positive point estimate is a request for more traffic, not a green light.

How to record the result so you don’t repeat the wrong test

A test that ends without a written record gets run again by someone else next quarter. The record is what lasts.

Write the hypothesis, the metric and the declared sample before you run anything. The declaration is what makes the later p-value meaningful.
Store the decision rule, including the stopping point, with the test. A rule that lives in someone’s memory isn’t a rule.
Log the result whatever it is. A flat result is a finding and it closes a question.
Note the window, the audience and the traffic source. A result without its window can’t be compared with the next one.
Write the practical verdict next to the statistical one: ship, don’t ship, or needs more traffic. That third verdict is the one Establishing a Baseline describes, where a team keeps measuring until the baseline is stable enough to compare against.
Note the confounders that were present, such as a campaign, a holiday, a release or a tracking bug. Nielsen Norman Group collects those under confounding variables in quantitative UX research.
Keep the list of what was tested where the next person will read it. The value of a null result is that it stops the repeat.

The method is old and the failure mode is old. Fix the sample before you start, read significance as a statement about noise, and write down what you found. The test is cheap. Repeating the same wrong test for a year isn’t.

Two more readings cover the size of the group. How Many Participants for Quantitative Usability Studies gives the count a quantitative study needs, and A/B Testing states where the method is weak.

Where Nielsen Norman Group doesn’t tell one story

The Nielsen Norman Group articles don’t agree about large samples, and it is worth showing where they differ.

The 2005 article on quantitative studies treats a large sample as an advantage. It says that with a very large number of users a team can detect a 1% difference with high significance. In that reading, sample size is what buys precision.

The 2024 article on A/B testing uses the same traffic volume as a limitation. It states that the method requires thousands of users, and that this is why A/B testing fits poorly on a low traffic page. The same volume that reads as an advantage in one article is the constraint in the other.

Practical vs. Statistical Significance shows the other side, and it is the one to keep. A result can be significant and worthless at the same time, and the page’s own example is a lift from 85.0% to 85.2% with p = 0.03. The gap is real by the usual threshold and it is too small for any business to act on. The article also shows the reverse: a gap from 20.0% to 85.0% with 8 users and p = 0.37 is large and still not significant.

The position in this post is the narrower one. A large sample buys precision and nothing else, and precision is only worth having once the effect is worth detecting. Putting A/B Testing in Its Place reaches a compatible conclusion from the method side.

Sources

Two related notes sit in the usability test post, which covers the qualitative floor, and in the checkout post, which covers deciding before building.