Specification Choice in Randomized and Natural Experiments : Lessons from the Regulation SHO Experiment

During 2005-2007, the SEC conducted a randomized trial in which it removed short-sale restrictions from one-third of the Russell 3000 firms. Early studies found modest microstructure-related effects of removing the restrictions but no effect on short interest or share prices. More recently, however, many studies have attributed a wide range of substantive indirect outcomes to this experiment. We revisit the principal findings in four recent studies in major journals, using a sample that closely matches the actual experiment and a pre-specified research design, and find no support for any of the reported results. If we instead match their specifications as best we can based on the published descriptions, we still obtain quite different results; only two of 13 outcomes are statistically significant, barely so, and even those results are tenuous. For two papers, we have the authors’ original data and code. We can technically replicate two (different) results, but those results are highly sensitive to specification. Our findings have implications for the robustness of other studies finding indirect effects of the short-sale experiment. More broadly, they have implications for the credibility of “causal” research designs which rely on randomization or on natural experiments. Researchers retain extensive discretion over both the sample and model specification, even for a true randomized experiment. The choices in these studies produced significant results, when other reasonable choices would not.The Online Appendix is available at https://ssrn.com/abstract=3657200.Our pre-specified analysis plan is available at https://ssrn.com/abstract=3415529