Skip to content
tecuy
Marketing & Advertising

A/B Test Significance & Sample Size Calculator

Check whether an A/B test result is statistically significant, see the confidence interval for the difference, and plan how many visitors a test needs.

Variant A (control)
Variant B

Calculated entirely in your browser — nothing you enter here is sent to a server.

How the significance test works

The calculator compares two conversion rates with a two-proportion z-test, the standard test for A/B experiments on email subject lines, ads and landing pages. It asks how likely a difference at least this large would be if both variants really converted at the same rate.

  • Conversion rate = conversions ÷ visitors for each variant
  • Pooled rate p = (conversions A + conversions B) ÷ (visitors A + visitors B)
  • z = (rate B − rate A) ÷ √(p × (1 − p) × (1/visitors A + 1/visitors B))
  • The p-value comes from the normal distribution; the result is significant when it is below 1 − confidence (0.05 at 95%).
  • The interval for B − A uses the unpooled standard error: √(rA(1 − rA)/nA + rB(1 − rB)/nB).

These are the same methods as R's prop.test without continuity correction and the statsmodels two-proportion z-test, so you can reproduce every figure.

Worked example

An email subject-line test sends variant A to 1,000 people and 100 click (10%); variant B goes to 1,000 people and 130 click (13%). The difference is +3 percentage points, a relative change of +30%. The test gives z = 2.103 and a two-sided p-value of 0.0355, so the result is significant at 95% confidence but not at 99%. The 95% interval for the difference runs from +0.21 to +5.79 percentage points: the data rule out no effect, yet the true lift could be much smaller than the 3 points observed.

Planning the next test, with a 10% baseline and a 20% relative change (10% → 12%) at 95% confidence and 80% power, each variant needs 3,841 recipients, 7,682 in total.

Reading the result honestly

  • Significant is not the same as important. With very large samples, tiny differences become significant. Check the interval and decide whether the smallest plausible lift is worth acting on.
  • Not significant does not mean “no difference”. It means this sample can't tell the variants apart. The plan mode shows how many visitors a reliable answer needs.
  • Don't stop early. Checking repeatedly and stopping the first time p dips below 0.05 inflates false positives well beyond 5%. Fix the sample size first, then evaluate once.
  • Choose one-sided only if you decided before the test that you would act on an improvement and ignore a decline. It needs less data but can never flag that B is worse.

Limitations

  • Two variants and one conversion metric. Testing several variants at once raises the chance of a false positive unless you adjust the confidence level.
  • The test assumes each visitor is counted once and acts independently. Counting repeat sessions or page views as visitors overstates the sample.
  • It uses a large-sample approximation; the calculator warns when any variant has fewer than 10 conversions or 10 non-conversions.
  • It compares conversion rates only, not revenue per visitor or other continuous metrics.

Work out each variant's rate with the conversion rate calculator, compare email results with email marketing benchmarks, or estimate campaign outcomes in the email marketing calculator.

How to use the A/B Test Significance & Sample Size Calculator

  1. Choose Analyse test results, then enter the visitors and conversions for variant A (your control) and variant B.
  2. Pick a confidence level (95% is the common default) and whether you are testing for any difference (two-sided) or only for an improvement (one-sided).
  3. Read the conversion rates, the relative change, the p-value and the confidence interval for B − A, then the plain-language verdict underneath.
  4. Before your next test, switch to Plan sample size, enter your current conversion rate and the smallest relative change worth detecting, and note the visitors needed per variant.
  5. Use Copy summary or Copy plan to paste the figures into a report or ticket.

Example: An email subject-line test

Variant A: 1,000 recipients, 100 clicks (10%). Variant B: 1,000 recipients, 130 clicks (13%). The calculator shows +3 percentage points (+30% relative), z = 2.103 and p = 0.0355, which is significant at 95% but not at 99%. The 95% interval for the difference is +0.21 to +5.79 points, so the true lift may be much smaller than the 3 points observed.

Questions about the A/B Test Significance & Sample Size Calculator

What does the p-value mean here?
It is the probability of seeing a difference at least this large if both variants truly converted at the same rate. A p-value of 0.03 does not mean there is a 97% chance that B is better; it means a gap this big would be unusual if there were no real difference.
My result is not significant. Does that mean the variants perform the same?
No. It means this sample cannot distinguish them. The confidence interval shows the range of differences the data are consistent with; if it is wide, run the test longer or use Plan sample size to see how many visitors a clearer answer needs.
Can I check the result every day and stop when it becomes significant?
Not with this test. Repeatedly checking and stopping at the first significant result raises the real false-positive rate well above the confidence level you chose. Decide the sample size up front and evaluate once it is reached.
When should I use a one-sided test?
Only when you decided before the test that you would act on an improvement and ignore a decline. It needs fewer visitors, but it can never tell you that B is worse than A.
Which statistical method does the calculator use?
A two-proportion z-test with a pooled standard error for the p-value, an unpooled (Wald) interval for the difference, and the normal-approximation sample-size formula used by R's power.prop.test. These are standard large-sample methods; the calculator warns when a variant has fewer than 10 conversions or non-conversions.
Can I test more than two variants?
This calculator compares two at a time. Comparing several variants against each other increases the chance of a false positive, so with three or more variants use a stricter confidence level or a method designed for multiple comparisons.