Home›Marketing

A/B test: real result or luck?

Compare two versions of a page, ad, email or price and find out whether the difference in conversion is statistically significant or could be chance. See the estimated lift, the confidence interval and how many visitors the test needs to detect an improvement.

Educational simulation. Not investment advice. Results are estimates based on the data you enter and do not replace advice from a qualified professional.

Reviewed on Oct 31, 2026

Example values. Visitors who saw each version and how many converted (bought, signed up, clicked).

Version A (control)
Version B (variant)
Test settings

Your inputs stay only on this browser and are not sent anywhere. Untick to delete them.

—Conversion lift of version B
—

Where each version’s true conversion rate lies

Likely range of each version’s real conversion rate. The less the curves overlap, the more reliable the difference.

Tap the chart to see the values.

Visitors needed by the lift you want to detect

Visitors per version to detect each relative lift, at the chosen confidence and power. Small lifts require much larger tests.

Tap the chart to see the values.

Statistics

The summary lists your inputs, results, formulas, assumptions and sources. The full report also includes the charts and interpretations, in color. In the print dialog, choose Save as PDF.

Method, assumptions and sources

Formulas

z = (pB − pA) ÷ √(p × (1 − p) × (1/nA + 1/nB)), with p = total conversions ÷ total visitors
Two-sided p-value = 2 × (1 − Φ(|z|))
Difference interval = (pB − pA) ± critical z × √(pA(1 − pA)/nA + pB(1 − pB)/nB)
Sample per version = (zα/2 × √(2p̄(1 − p̄)) + zβ × √(p1(1 − p1) + p2(1 − p2)))² ÷ (p2 − p1)²

Assumptions and limits

  • Educational calculation using the two-proportion z-test. Difference interval with unpooled standard error; sample size to detect the entered relative lift over version A’s conversion rate.
  • Good practice: set the test size before starting, do not stop as soon as it turns significant, run full weeks and test one change at a time.
  • The test assumes independent samples and a single metric. Stopping before the planned size increases the chance of a false positive.

Official sources and references

Frequently asked questions

What is an A/B test?

An experiment that shows two versions of a page, ad, email or price to similar groups at the same time and compares a single metric, usually the conversion rate. The calculator tells you whether the difference between them is likely real or could be chance.

What does the p-value mean?

It is the probability of seeing a difference at least as large as the one observed if the two versions were actually the same. With 95% confidence, a p-value below 5% is considered significant. It is not the probability that B is better.

How do I read the default example?

Version A converted 3.00% (156 out of 5,200) and B 3.71% (191 out of 5,150): a 23.6% lift. The p-value is 4.5%, below the 5% accepted at 95% confidence, so the difference is significant, though only just. The true lift is likely between 0.5% and 46.8%. To detect 10% lifts, the test would need about 53,000 visitors per version.

How long should a test run?

Long enough to reach the planned sample size and at least one or two full weeks, to include weekdays and weekends. At the example’s 700 visitors a day, detecting 10% lifts would take more than 150 days: with little traffic, test bolder changes.

What if the result is not significant?

It means the test did not find enough evidence, not that the versions are the same. If the planned size has been reached, the real lift is probably smaller than the one you were looking for: keep the simpler or cheaper version and test a bolder change.

What are power and minimum detectable effect?

Power is the chance of detecting an improvement that really exists (80% is standard). The minimum detectable effect is the smallest relative lift the test is designed to catch. Smaller effects and higher power both require more visitors.

What mistakes should I avoid?

Stopping the test as soon as it looks significant, running partial weeks, changing many things at once and checking many metrics until one looks good. Each of these raises the chance of a false positive.

Related calculators

See all in Marketing ›

A/B testing: how to tell if a result is significant

An A/B test shows two versions of a page, ad, email or price to similar groups of people at the same time and compares the results. The question is always the same: is the difference real, or could it be chance?

Significance and p-value

The p-value is the probability of seeing a difference at least as large as the one observed if both versions were actually the same. With 95% confidence, the result is considered significant when the p-value is below 5%. This calculator uses the z-test for two proportions, the standard method for conversion rates.

Confidence interval

Besides saying whether there is a difference, the test estimates its size. A lift of +22% with an interval from +2% to +42% means the improvement is real but could be small. For business decisions, look at the lower bound.

Sample size before you start

The smaller the improvement you want to detect, the more visitors you need. Detecting a 5% lift takes about four times as many visitors as detecting a 10% lift. Decide the size and duration before starting.

Common mistakes

  • Stopping as soon as it turns significant: checking results every day and stopping at the first good day greatly increases false positives.
  • Partial weeks: weekday and weekend behavior differ; run full weeks.
  • Many changes at once: you will not know which one made the difference.
  • Many metrics: with 20 metrics, one will look significant by chance; choose the main metric in advance.