A Sample Size Calculation for Training and Certifying Targeting Policies

We propose an approach for determining the sample size required when using an experiment to train and certify a targeting policy. Calculating the rate at which the performance of a targeting model improves with additional training data is a complex problem. We approximate the problem by grouping customers into segments. This allows us to develop computationally efficient algorithms that calculate the required sample size using simulations or closed-form analytical approximations. We consider two problem formulations. The first formulation focuses on training a targeting policy that satisfies a predefined performance threshold. The second formulation involves out-of-sample certification of a trained targeting policy in a statistical test. We illustrate our analysis using data from a luxury fashion retailer