Refinement : Measuring Informativeness of Ratings in the Absence of a Gold Standard

We propose a new metric for evaluating the informativeness of a set of ratings from a single rater on a given scale. Such evaluations are of interest when raters rate numerous comparable items on the same scale, as occurs in hiring, college admissions, and peer review. Our exposition takes the context of peer review, which involves uni- and multi-variate cardinal ratings. We draw on this context to motivate an information-theoretic measure of the refinement of a set of ratings---Rounding-induced Entropy Decrease---as well as two secondary measures. A mathematical analysis of the three measures reveals that only the first, which directly measures the information content of the ratings, possesses properties appropriate to a refinement metric. Finally, we analyze refinement in real-world grant-review data, finding evidence that overall merit ratings, which are used to determine funding, are more refined than criterion scores, which do not directly determine funding