  Peer review is widely regarded as essential for advancing scientific
research. However, reviewers may be biased by authors' prestige or other
characteristics. Double-blind peer review, in which the authors' identities are
masked from the reviewers, has been proposed as a way to reduce reviewer bias.
Although intuitive, evidence for the effectiveness of double-blind peer review
in reducing bias is limited and mixed. Here, we examine the effects of
double-blind peer review on prestige bias by analyzing the peer review files of
5027 papers submitted to the International Conference on Learning
Representations (ICLR), a top computer science conference that changed its
reviewing policy from single-blind peer review to double-blind peer review in
2018. We find that after switching to double-blind review, the scores given to
the most prestigious authors significantly decreased. However, because many of
these papers were above the threshold for acceptance, the change did not affect
paper acceptance decisions significantly. Nevertheless, we show that
double-blind peer review may have improved the quality of the selections by
limiting other (non-author-prestige) biases. Specifically, papers rejected in
the single-blind format are cited more than those rejected under the
double-blind format, suggesting that double-blind review better identifies
poorer quality papers. Interestingly, an apparently unrelated change - the
change of rating scale from 10 to 4 points - likely reduced prestige bias
significantly, to an extent that affected papers' acceptance. These results
provide some support for the effectiveness of double-blind review in reducing
prestige bias, while opening new research directions on the impact of peer
review formats.
