Is economics self-correcting? Replications in the American Economic Review

Replication and constructive controversy are essential for scientific progress. This paper reviews the impact of all replications published as comments in the American Economic Review between 2010 and 2020. We investigate the citation rates of comments and whether a comment affects its original paper's citation rates. We find that most comments are barely cited, and they have no impact on the original papers' subsequent citations. This finding holds for original papers for which the comment diagnoses a substantive problem. We conclude from these citation patterns that replications do not update the economics literature. In an online opinion survey, we elicited viewpoints of both comment authors and original authors and find that in most cases, there is no consensus regarding the replication's success and to what extent the original paper's contribution sustains. This resonates with the conventional wisdom that robustness and replicability are hard to define in economics.