EXPLORING THE USEFULNESS OF A NONRANDOM HOLDOUT SAMPLE FOR MODEL VALIDATION: WELFARE EFFECTS ON FEMALE BEHAVIOR

A particularly challenging use of decision-theoretic models in economics is to forecast the impact of large changes in the environment. The problem we explore in this article is how to gain confidence in a model's ability to predict the impact of such large changes. We show that an approach to validation and model selection that includes the choice of a "nonrandom holdout sample," a sample that differs significantly from the estimation sample along the policy dimension that the model is meant to forecast, can be fruitful. Copyright 2007 by the Economics Department Of The University Of Pennsylvania And Osaka University Institute Of Social And Economic Research Association.