The Objectivity of National Research Foundation Peer Review Based Ratings in South Africa

This paper examines the strength of association between the outcomes of National Research Foundation (NRF) peer review based rating mechanisms, and a range of objective measures of performance of researchers. The analysis is conducted on 1932 scholars that have received an NRF rating or an NRF research chair. We find that scholars with higher NRF ratings record higher performance on average against the objective measures of absolute output and the impact of their research, than scholars at lower ratings. In addition, the higher the performance of scholars against all objective measures of absolute output and impact, increases the probability of higher rating. However, we also find that the probability of achieving a B-rating remains higher than that of acheiving an A-rating even at the very highest levels of recorded performance for South African scholars. In addition, scholars who have received the highest ratings record objective levels of research output and impact of their research that are no different from the minimum levels of objective performance at much lower NRF ratings. Moreover, we find strong cross-disciplinary differences in terms of the impact that objective levels of performance have on the probability of achieving different NRF ratings. Finally, we report evidence that NRF peer review is less likely to reward multiauthored research output than single-authored output.