The quantile performance of statistical treatment rules using hypothesis tests to allocate a population to two treatments

This paper modifies the Wald development of statistical decision theory to offer new perspective on the performance of certain statistical treatment rules. We study the quantile performance of test rules, ones that use the outcomes of hypothesis tests to allocate a population to two treatments. Let lambda denote the quantile used to evaluate performance. Define a test rule to be lambda-quantile optimal if it maximizes lambda-quantile welfare in every state of nature. We show that a test rule is lambda-quantile optimal if and only if its error probabilities are less than lambda in all states where the two treatments yield different welfare. We give conditions under which lambda-quantile optimal test rules do and do not exist. A sufficient condition for existence of optimal rules is that the state space be finite and the data enable sufficiently precise estimation of the true state. Optimal rules do not exist when the state space is connected and other regularity conditions hold, but near-optimal rules may exist. These nuanced findings differ sharply from measurement of mean performance, as mean optimal test rules generically do not exist. We present further analysis that holds when the data are real-valued and generated by a sampling distribution which satisfies the monotone-likelihood ratio (MLR) property with respect to the average treatment effect. We use the MLR property to characterize the stochastic-dominance admissibility of STRs when the data have a continuous distribution and then generate findings on the quantile admissibility of test rules.