A/B Test Calculator
Enter the number of visitors and conversions for variant A and variant B, choose a confidence level, and the calculator tells you whether the difference is statistically significant. It shows the two conversion rates, the lift of B over A, the p-value, the z score and a confidence interval for the difference. A second section tells you how many visitors each variant needs before you start. Everything runs in your browser.
How many visitors do you need?
The test is a two-proportion z-test: the standard error uses the combined conversion rate, the p-value is the chance of a gap this large if A and B really convert equally, and the interval for the difference uses each variant's own rate. Decide the sample size before you start and do not stop the test the moment the result looks significant, because repeated peeking raises the chance of a false win. The normal approximation needs at least about 10 conversions and 10 non-conversions in each variant. Significant does not mean important, so look at the size of the lift too. Your numbers stay in your browser.
Related Tools
[5 suggestions]t-Test Calculator: One Sample, Two Sample and Paired
Run a one-sample, Welch, pooled or paired t-test on your data and get the t value, degrees of freedom, p-value, confidence interval and effect size.
Student ToolsChi-Square Calculator: Independence and Goodness of Fit
Run a chi-square test of independence on a table of counts or a goodness of fit test, and get the statistic, p-value, expected counts and Cramer's V.
Student ToolsSample Size Calculator
Find how many survey responses you need for a confidence level and margin of error, or the margin of error a sample gives.
Math & MoneyConfidence Interval Calculator
Find a confidence interval for a mean (from summary numbers or raw data) or a proportion, with the t value and margin of error.
Math & MoneyZ-Score Calculator
Turn a value into a z-score, a z-score into probabilities and percentiles, or a percentile back into a z-score, using the standard normal distribution.
Math & MoneyAbout the A/B Test Calculator
An A/B test compares two versions of something, such as a page or an email, by the share of visitors who convert. The calculator uses the two-proportion z-test. It combines both variants to estimate the conversion rate under the assumption that nothing differs, works out the standard error of the gap, and divides the observed gap by it to get a z score. The two-sided p-value is the chance of a gap at least that large if the variants really convert equally. With 10,000 visitors each, 500 conversions for A (5.0%) and 600 for B (6.0%), the lift is 20%, z is 3.1 and the p-value is about 0.002, so the result is significant at 95%. With 560 conversions for B the same test gives a p-value of 0.058, which is not. The confidence interval for the difference uses each rate's own variance. To plan a test, the sample size section uses the standard formula for a two-sided test with 80% power: detecting a change from 10% to 12% (a 20% lift) at 95% confidence needs 3,841 visitors in each variant, and detecting 5% to 5.5% needs 31,234. Decide the sample size in advance and avoid stopping the moment a result turns significant, because repeated peeking raises the chance of a false positive. The normal approximation works best with at least 10 conversions and 10 non-conversions in each variant.
How do I know if an A/B test result is significant?▾
Compare the p-value with your threshold. At 95% confidence the p-value must be below 0.05. A p-value of 0.058 is not significant, and 0.002 is.
What does the p-value mean?▾
It is the chance of seeing a gap at least this large if the two variants truly convert at the same rate. It is not the chance that B is better.
How many visitors do I need for an A/B test?▾
It depends on your current rate and the smallest change you care about. To detect a lift from 10% to 12% at 95% confidence and 80% power you need about 3,841 visitors per variant.
Can I stop the test as soon as it is significant?▾
Not safely. Checking repeatedly and stopping at the first significant result inflates false positives. Set the sample size first and read the result at the end.
Is my data uploaded?▾
No. The calculation runs in your browser and nothing you enter is sent anywhere.