A/B Tests Statistical Calculator
Pre-test calculator
Use this before you run the experiment. It’ll show you the sample size needed for the experiment, how long you need to run it for, and what the minimum detectable effect is.
Post-test calculator
Use this during or after the experiment. It’ll show you whether you have a statistically significant result, and will help you analyse performance.
Significance level
Significance level (known also as confidence level) tells you how sure you can be your results were not affected by type I error (false positive – rejecting the null hypothesis that is actually true).
Sample size
The number of users required per variant in order to detect a certain Minimum Detectable Effect at the specified significance level. When reaching the required number of users per variant, the test can be concluded and the data analysed.
One-sided / Two-sided experiment
A one-sided experiment will tell you the percentage chance that your variant performs better than control. It will not tell you if the variant performs worse than control. To determine if your experiment performs better OR worse than control, use a two-sided experiment.
Minimum Detectable Effect (MDE)
The difference between control and variant conversion rates (lift or uplift) that you can detect within a given timeframe
Required visitors per variant
The number of users required/variant in order to detect a certain MDE at the specified significance level. When reaching the required number of users/variant, the test can be concluded and the data analysed.