A/B Testing with Fat Tails
Large and thus statistically powerful A/B tests are increasingly popular in business and policy to evaluate potential innovations. We study how to optimally use scarce experimental resources to screen innovations. To do so, we propose a new framework for optimal experimentation that we call the A/B testing problem. The key insight of the model is that the optimal experimentation strategy depends on whether most gains accrue from typical innovations, or from rare and unpredictable large successes that can be detected using tests with small samples. We show that, if the tails of the (prior) distribution of true effect sizes is not too fat, the standard approach of trying a few high-powered experiments is optimal. However, when this distribution is very fat tailed, a lean experimentation strategy of trying more but smaller interventions is optimal. We measure this tail parameter using experiments from Microsoft Bing's EXP platform and find extremely fat tails. Our theoretical results and empirical analysis suggest that even simple changes to business practices within Bing could dramatically increase innovation productivity
Year of publication: |
2019
|
---|---|
Authors: | Azevedo, Eduardo M. |
Other Persons: | Alex, Deng (contributor) ; Montiel Olea, Jose (contributor) ; Rao, Justin M. (contributor) ; Weyl, E. Glen (contributor) |
Publisher: |
[2019]: [S.l.] : SSRN |
Saved in:
freely available
Extent: | 1 Online-Ressource (59 p) |
---|---|
Type of publication: | Book / Working Paper |
Language: | English |
Notes: | Nach Informationen von SSRN wurde die ursprüngliche Fassung des Dokuments February 26, 2019 erstellt |
Other identifiers: | 10.2139/ssrn.3171224 [DOI] |
Source: | ECONIS - Online Catalogue of the ZBW |
Persistent link: https://www.econbiz.de/10012899429
Saved in favorites
Similar items by person
-
Walrasian equilibrium in large, quasi-linear markets
Weyl, E. Glen, (2013)
-
Walrasian equilibrium in large, quasi-linear markets
Weyl, E. Glen, (2013)
-
Azevedo, Eduardo M., (2020)
- More ...