A/B Tests Statistical Calculator

Pre-test calculator

Use this before you run the experiment. It’ll show you the sample size needed for the experiment, how long you need to run it for, and what the minimum detectable effect is.

Post-test calculator

Use this during or after the experiment. It’ll show you whether you have a statistically significant result, and will help you analyse performance.

P-value
Z-score
Changing to two-sided experiment might change the outcome of the test

Test Results

Insights

 
Control
Variant
Visitors
Conversions
Conversion rate
Uplift
–
not significant
Significance level
–
 
Visitors
Conversions
Conversion rate
Uplift
Significance level
Control
–
–
Variant
not significant

Next Steps

Have you reached your sample size?  

We need to know if you reached your sample size to decide on the outcome of the test and recommend next steps.

Don’t know
Outcome

You’ve got a significant result! Your variant performs better than control.

Next steps

Conclude your experiment and analyse the results.

Outcome

If you haven’t reach your sample size, the observed uplift should not be analysed yet, even if it reached the desired confidence level.

Next steps

Continue to run your experiment until you’ve reached your sample size. We estimate this would take another 12 days based on the data you provided.

Next steps

We recommend using our Pre-test calculator to find out what your sample size should be.

We estimate it would take another 12 days to get a conclusive result based on the data you provided.

Significance level

Significance level (known also as confidence level) tells you how sure you can be your results were not affected by type I error (false positive – rejecting the null hypothesis that is actually true).

Sample size

The number of users required per variant in order to detect a certain Minimum Detectable Effect at the specified significance level. When reaching the required number of users per variant, the test can be concluded and the data analysed.

One-sided / Two-sided experiment

A one-sided experiment will tell you the percentage chance that your variant performs better than control. It will not tell you if the variant performs worse than control. To determine if your experiment performs better OR worse than control, use a two-sided experiment.

Minimum Detectable Effect (MDE)

The difference between control and variant conversion rates (lift or uplift) that you can detect within a given timeframe

Required visitors per variant

The number of users required/variant in order to detect a certain MDE at the specified significance level. When reaching the required number of users/variant, the test can be concluded and the data analysed.