Statistical analysis of performance indicators in UK higher education

Attempts to measure the quality with which institutions such as hospitals and universities carry out their public mandates have gained in frequency and sophistication over the last decade. We examine methods for creating performance indicators in multilevel or hierarchical settings (e.g. students nested within universities) based on a dichotomous outcome variable (e.g. drop-out from the higher education system). The "profiling" methods that we study involve the indirect measurement of quality, by comparing institutional outputs after adjusting for inputs, rather than directly attempting to measure the quality of the processes unfolding inside the institutions. In the context of an extended case-study of the creation of performance indicators for universities in the UK higher education system, we demonstrate the large sample functional equivalence between a method based on indirect standardization and an approach based on fixed effects hierarchical modelling, offer simulation results on the performance of the standardization method in null and non-null settings, examine the sensitivity of this method to the inadvertent omission of relevant adjustment variables, explore random-effects reformulations and characterize settings in which they are preferable to fixed effects hierarchical modelling in this type of quality assessment and discuss extensions to longitudinal quality modelling and the overall pros and cons of institutional profiling. Our results are couched in the language of higher education but apply with equal force to other settings with dichotomous response variables, such as the examination of observed and expected rates of mortality (or other adverse outcomes) in investigations of the quality of health care or the study of retention rates in the workplace. Copyright 2004 Royal Statistical Society.