The A/B Testing Problem with Gaussian Priors

Abstract: A risk-neutral firm can perform a randomized experiment (A/B test) to learn about the effects of implementing an idea of unknown quality. The firm’s goal is to decide the experiment’s sample size and whether or not the idea should be implemented after observing the experiment’s outcome. We show that when the distribution for idea quality is Gaussian and there are linear costs of experimentation, there are exact formulae for the firm’s optimal implementation decisions, the value of obtaining more data, and optimal experiment sizes. Our formulae—which assume that companies use randomized experiments to help them maximize expected profits—provide a simple alternative to i) the standard rules-of-thumb of power calculations for determining the sample size of an experiment, and also to ii) ad hoc thresholds based on statistical significance to interpret the outcome of an experiment.