Consistent preordering with an estimated criterion function, with an application to the evaluation and comparison of volatility models

When alternatives are compared using an estimated criterion function, this may introduce a discrepancy between the true and the estimated criterion. In this paper, we consider a situation where a preordering (ranking) of stochastic sequences is defined from expected loss/gain, using a parametric criterion function. Evaluation based on estimated parameters induces a second preordering, and using sample averages in place of expectations induces a third (empirical) preordering, and we derive conditions that ensure equivalence of the three preorderings. We apply the framework to the comparison of ARCH-type models. In practice, the conditional variance, σ2t , t = 1, 2, . . . is unobserved, such that evaluation must be based on a proxy for σ2t. We show that some commonly used criteria for evaluation of volatility models, may induce a different preordering than the one intended. This problem is cause by the measurement error of σ2t , which defines (part of) the empirical criterion. An empirical analysis and a simulation study show the practical relevance of this inconsistency problem. The results provide an additional argument for using intra-day data to approximate σ2t , such as realized volatility.