Machine learning to improve experimental design

This paper proposes a way of using observational pretest data for the design of experiments. In particular, this paper trains a random forest on the pretest data and stratifies the allocation of treatments to experimental units on the predicted dependent variables. This approach reduces much of the arbitrariness involved in defining strata directly on the basis of covariates. A simulation on 300 random samples drawn from six data sets shows that this algorithm is extremely effective in reducing the variance of the estimation compared to random allocation and to traditional ways of stratification. On average, this stratification approach requires half the sample size to estimate the treatment effect with the same precision as complete randomization. In more than 80% of all samples the estimated variance of the treatment estimator is lower and the estimated statistical power is higher than for standard designs such as complete randomization, conventional stratification or Mahalanobis matching.