Achievement Estimates and Deviations from Cardinal Comparability

This paper develops a general method for assessing the robustness of standard empirical methods measuring group differences in achievement to violations in the cardinal comparability of achievement test scores. The paper defines a distance measure over possible test scales and then constructs bounds for the bias that could result from using the observed rather than the true scale, given that the two are no more than a fixed distance from each other. This bounding methodology can be applied to estimators employing mean differences, as well as OLS and IV regression coefficients. Although the paper focuses on achievement, the methodology can be equally well applied to any situation in which there is uncertainty about the scale of the outcome under consideration. Finally, a simple empirical exercise suggests that estimated trends in black/white and high/low-income achievement inequality are extremely sensitive to minor rescalings of test scores, while cross-sectional estimates are quite robust