What Can We Do to Improve Peer Review in NLP ? (2020)

Abstract: Peer review is our best tool for judging the quality of conference submissions, but it is becoming increasingly spurious. We argue that a part of the problem is that the reviewers and area chairs face a poorly defined task forcing apples-to-oranges comparisons. There are several potential ways forward, but the key difficulty is creating the incentives and mechanisms for their consistent implementation in the NLP community.

## introduction traditionally, peer review is expected to act as a filter for high-quality, impactful work @xcite , but this does not hold in practice: the correlation between conference rejection rates and conference impact in terms of citations is not strong @xcite @xcite , and rejects from one conference sometimes receive awards 1 at another. the problem is that both expectations are unrealistic to begin with. a peer reviewer cannot perform real quality control, because that would mean ensuring that a paper is reproducible. not only is that impossible, only having a few hours to review a paper, but it is a general problem for deep learning (dl)-based nlp @xcite . the reproducibility checklist at emnlp 2020 @xcite is the first step in that direction. 1 @xcite discusses the example of a paper by branavan et al. that received the award at acl 2009 after being rejected from naacl (scored at 2.3/5). more recently, elmo @xcite received low scores from iclr reviewers and was resubmitted to naacl to win the award there. submissions merit acceptance threshold realistic paper merit distribution, adapted from @xcite paper merit distribution, with which peer review could be reliable as for paper impact, it is distinct from its scientific merit @xcite , and strongly depends on completely orthogonal factors: how niche is the topic, how much promotion was done, whether the paper offers room to innovate with a low entry barrier foot_0 @xcite . what we could realistically expect from peer review is to reject the papers with obvious methodology flaws, and turn the spotlight on the ideas which would be beneficial for the field to discuss. however, the current process is not set up to achieve either purpose. instead, it aims to rank all submissions by their merit so as to identify the top 25%. that task, we argue, is fundamentally impossible. ## why is peer review so difficult? peer review would be easy if the paper merit distribution had a clear boundary between good and bad papers (and ideally that boundary would match the conference acceptance threshold). however, that is clearly not the case. based on citation counts, @xcite hypothesize that paper merit is zipf-distributed, as shown in figure 1 . that means that even with the most objective reviewers, the difference between the worst accepted paper and the best rejected paper is less than 1%. to make matters worse, there are no clear criteria that would help to draw the decision boundary. @xcite discusses an experiment at sig-comm 2006, where they first made the easy accept/reject decisions for papers with low review score variance, and then assigned up to 9 additional reviewers to papers with high variance. the reviewers who had to discuss the difficult cases were reportedly "nearly driven insane" by the apples-tooranges comparisons, such as incomplete evaluation in one borderline paper vs narrow applicability of another. no matter how long we agonize over such decisions, they will look random. a case in point: at nips 2014, 10% submissions were reviewed by two different pcs, who disagreed on 57% of papers @xcite . for large *acl conferences, the situation is even worse: we often weigh against each other different types of papers with different strengths and weaknesses (figure 2 ). there can be no 'correct' answer as to which one has more scientific merit. ## how reviewers cope faced with an objectively impossible task, reviewers do what humans generally do to reason under uncertainty: they default to heuristics, which introduces unwanted biases @xcite . there is an extra incentive to do so because applesto-oranges comparisons are a slow, deliberate, cognitively expensive process -and peer review is currently invisible work performed for free. this section lists some of the most problematic reviewer heuristics in nlp. writing style. language errors, non-standard style or rhetorical structure are easy to spot and interpret as sloppiness. this puts almost everybody at a disadvantage against north americans. papers with worse english may even be perceived as worse than those with better content @xcite . results not surpassing sota. an easy heuristic is to check if the paper beats the state of the art (sota). while an engineering contribution should demonstrate a significant improvement over prior methods, it does not have to be an improvement in predictive performance: advances in compute or data efficiency, interpretability, cognitive plausibility etc. are also valuable @xcite . the focus on predictive performance encourages the 'arms race' for pre-training data and compute, and exacerbates methodological issues foot_1 . the requirement for comparisons with the latest sota model also puts us in the hamster wheel, making experiments outdated already by the submission time. narrow topics. it is easier to publish on trendy, 'scientifically sexy' topics @xcite . in the last two years, there has been little talk of anything other than large pretrained transformers, with bert alone becoming the target of over 150 studies proposing analysis and various modifications @xcite . the 'hot trend' forms the prototype for the kind of paper that should be recommended for acceptance. niche topics such as historical text normalization are downvoted (unless, of course, bert could somehow be used for that). work not on english. since prototypical nlp experiments use english as the target language, other languages mark the paper as narrow. this heuristic is indefensible, since approaches tested only on e.g. estonian are as generalizable as those tested only on english. it also strengthens the 'default' status of english @xcite . already-famous work and work from wellknown labs. if reviewers feel that a paper was already accepted by the research community, they do not need to do any more vetting. for example, there was no way for bert to go through fully anonymous peer review @xcite . early preprint citations are arguably a better indicator of paper quality than peer review @xcite ), but they are also influenced by how famous the authors are 4 , and how much they pub-licize their work. well-known labs tend to have large online followings or even pr departments, propagating the 'rich get richer' phenomenon (also known as the 'matthew effect', @xcite ). the proposed solution seems too simple. since a prototypical 'acceptable' paper features a sophisticated dl model, simple solutions may look like the authors did not do enough work. this is obviously flawed, as the goal is to solve the problem, rather than to solve it in a complex way. non-mainstream approaches. since a 'mainstream' *acl paper currently uses dl-based methods, anything else might look like it does not really belong in the main track -even though acl stands for 'association for computational linguistics'. that puts interdisciplinary efforts at a disadvantage, and continues the trend for intellectual segregation of nlp @xcite . e.g., theoretical papers and linguistic resources should not be a priori at a disadvantage just because they do not contain dl experiments. resource papers. surprising as it may seem in a field that relies so much on supervised machine learning, resource papers are routinely rejected simply for being resource papers. linguistically deeper papers may also receive extra penalties for linguistic details at the cost of dl experiments, for non-english resources, and, by analogy with the sota heuristic for engineering papers, for not offering the largest resource @xcite . novel approaches. this sounds almost absurd, but scientific peer review is systematically biased towards unobjectionable (rather than novel) work @xcite @xcite . a reviewer faced with evaluating a completely new idea without prior art has to make a more difficult call than one for a paper with clear predecessors and a leaderboard table, and is more likely to fall back to one of the heuristics. the very process based on majority votes necessarily promotes 'safe', incremental, likely boring work, and puts non-mainstream work at a disadvantage. ## can we just abolish peer review? if the task is fundamentally impossible, should we just give up and look for alternatives to peer review? each round of conference notifications spurs calls on social media to just abolish the whole system, to increase acceptance rate, to let citations be the metric of the paper quality. unfortunately, this is not realistic, even if there were no co-dependence between citation counts and scientific fame or promotion efforts. fundamentally, low acceptance rates are a proxy for paper quality for non-experts, and that metric is expected by almost every hiring and grant committee. we are not aware of serious proposals on how to change that. and any experiments will require a generation of extremely brave students who are willing to graduate with no 'respectable' publication record. emnlp 2020 essentially increased the acceptance rate by creating a second-tier publication named findings of emnlp, which has "no requirement for high perceived impact, and accordingly solid work in untrendy areas and other more niche works will be eligible". 6 it enabled the organizers to accept 15.5% of extra submissions (including this one), in addition to 22.4% in the main track. unfortunately, this approach does not address the fundamental issue (comparing the incomparable), and introduces new problems: until there are systemic changes in how researchers are evaluated, peer review remains 'the least bad system available' @xcite . still, there is clearly room for improvement. first, peer review has to become a valued part of academic cvs, and something that employers budget time for. reviews done by overworked people in their free time will not be top-quality. second, we need to reduce the need for reviewers and acs to reason under high uncertainty. it cannot be fully eliminated, but there are several obvious directions for improvement. better reviewer matching. reviewers are more likely to resort to heuristics when they are not experts in the same narrow area as the paper they are reviewing. a matching should take into account both the tasks and the methods (e.g. a paper on coreference annotation is unlikely to be appreciated by a practitioner who only worked on coreference applications). since it is not always possible to find perfect matches, reviewers with complementary partial expertise (e.g. someone who speaks the language if the paper is not on english, plus an area expert) could be a fall-back strategy. more fine-grained tracks: acs should never have to decide between different types of papers. if surveys, opinion pieces, resource and analysis papers etc. are all welcome, they should have their own acceptance quotas and best paper awards. review forms tailored for different paper types: it does not make much sense to evaluate a reproducibility report for novelty, or a resource paper for sota results. coling 2018 developed review forms taking into account different types of contributions, possibly several per paper. announcing editorial priorities presubmission. what is the primary focus of a particular conference: sota engineering, diversity of approaches, fresh ideas? what counts more towards acceptance? stating this clearly would help authors find an appropriate event for their work, and help reviewers and area chairs be more consistent in their recommendations. not asking the reviewers for overall recommendation scores. this is where similar papers get seemingly random rankings from different reviewers, because they disagree on whether e.g. originality outweighs weaker evaluation. even having a clear policy does not help foot_5 @xcite . the obvious solution is that reviewers should only be asked for specific scores (originality, technical soundness etc.), which would be the basis of the decisions according to the editorial policy. the above solutions focus on reducing apples-tooranges comparisons. a fundamentally different approach is to increase reviewer accountability, e.g. by making reviews public. unfortunately, this does not address the core problem (reasoning under high uncertainty), and would introduce other problems. 8 6 what holds us back? at this point, the reader might join the disappointed anonymous reviewers of this paper and say that we are not proposing anything new. this is precisely why the problem is so difficult: we lack the implementation, not the conceptual innovation -and as researchers, we tend to only value the latter. on the organization side, each *acl conference is organized by a new set of people each year who set their own policies. such diversity by it-self would be fantastic, but often, many things are changed at the same time, no systematic comparisons are drawn, and even the obviously successful innovations might not stay on. consequently, next year we are no wiser about what works and what does not. we are running continuous experiments on ourselves, and never check foot_7 the results. on the community side, we are not aware of any quantitative studies of how peer review is discussed on social media, but as active members of the twitter #nlproc community, our impression is that this topic mostly gets on the feed during author rebuttals and after acceptance notifications at major conferences, as sketched in figure 3 . at that time, there are bitter complaints and reform suggestions, but few practical initiatives (which ensures that the cycle is repeated at the next conference). fundamentally, peer review is an annotation problem, and we can try to tackle it because we know enough about experimental methodology, iterative guideline development, inter-annotator agreement, and biases. here is where we fail: to illustrate the latter point: a quick search in the acl anthology revealed only four conference papers on peer review from a meta-research perspective: a paper-reviewer matching tool @xcite , a corpus of reviews @xcite , and two experimental studies using nlp to explain the observed reviews @xcite . we could not find any acl-published organizers are announced cfp is out papers are in ac/reviewer recruitment time review rebuttals decisions notifications conference community attention to peer review papers on testing different peer review policies or review form design for the nlp community. yet without such work nothing will change, and no other field will do it for us. the work by @xcite offers an actionable insight: acl reviewers appear to be victims of conformity bias, converging to the mean of reviews. one solution would be to let reviewers interact only with the authors during the rebuttal, but not with each other. the paper was published in naaclyet, to the best of our knowledge, there have been no attempts to change any policies accordingly. ## conclusion as a community familiar with annotation, we know that asking people to perform ill-defined tasks is not going to work well. yet this is exactly what we expect of ourselves as reviewers. we can do better. there are many known ways to reduce uncertainty in paper merit estimation, such as improving the review forms and reviewer matching. the problem is that implementing any of it would take a lot of work beside what acl is already doing, sometimes counter to its current practices. big changes in any large organization are difficult (especially in a volunteer-driven one), but this is the only way. the first step towards turning all the frustration on social media into action is to (a) recognize such work as respectable, main-track-worthy metaresearch (so that there are incentives to do it at all), and (b) create new, voted-in acl roles for systematic development, testing and comparison of review policies, as well as community feedback loops for authors and reviewers. a special acl committee is working on a rolling review reform foot_10 to address the increasing volume of reviews, but improving their quality is a different, long-term project.