Sample Size Calculator for A/B Tests

Contributors: Sri Santhosh Hari, Sooraj Subrahmannian, Devesh Maheshwari

URL of the APP: https://sscalc.herokuapp.com/

Power analysis and sample size calculations are key to drawing valid conclusion from any A/B test. These address the concerns around identifying number of units required in each condition (control and treatment) of the experiment. Theoretically it is easy to consider large number of samples in experiments but in practice it is expensive (time spent, lost revenue, lost customers, etc.). This makes sample size calculations key to the the entire experimentation process.

In this repository, we have developed functions to compute sample sizes for tests involving comparison of means and comparison of proportions. The functions are flexible enough to handle multiple testing conditions like one/two-sided tests, can take in range of power/significance level/effect size. An interactive version of these functions can be viewed as a dash app version. Local version of the app can be launched by executing the following steps

pip install -r requirements.txt

python sample_size_calculator.py

Open http://0.0.0.0:8055/ in a browser.

Formula for computing sample size

Variable Description:

$n_1$ : Number of samples in control group
$n_2$ : Number of samples in treatment group
$\delta$ : Effect Size in terms of standard deviation ( $\sigma$ )
$k:n_1/n_2$
$z_{\alpha/2}$ : Z-Score corresponding to the probability of incorrectly rejecting null hypothesis
$z_{1-\beta}$ : Z-Score corresponding to the propability of correctly rejecting null hypothesis
$\pi_1$ : Proportion of success (ex - click through rate) in control group (already known)
$\pi_2$ : Proportion of success in treatment group (estimated)