PISA Country Rankings and the Difficulty of Exams

Although exam scores are reported using numerical values, their measurement scale is ordinal: They depend on the difficulty distribution of the questions in the exam. Hence, population groups or countries should not be ranked according to average exam scores because there may exist an admissible exam with a different distribution of difficulty of the exam questions that would change the mean-based rankings. In this paper, we develop the conditions that indicate whether an alternative exam can reverse a mean-based ranking of the groups. Further, we suggest alternative group ranking procedures that are both informative and robust. An implementation of these procedures using the PISA 2012 math scores of students in OECD countries and a comparison with a mean-based ranking suggests that a mean based-ranking is sensitive to the difficulty distribution of the questions asked in the PISA exams