What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review (2022)

Abstract: Both scientific progress and individual researcher careers depend on the quality of peer review, which in turn depends on paper-reviewer matching. Surprisingly, this problem has been mostly approached as an automated recommendation problem rather than as a matter where different stakeholders (area chairs, reviewers, authors) have accumulated experience worth taking into account. We present the results of the first survey of the NLP community, identifying common issues and perspectives on what factors should be considered by paper-reviewer matching systems. This study contributes actionable recommendations for improving future NLP conferences, and desiderata for interpretable peer review assignments.

## introduction peer review is increasingly coming under criticism for its arbitrariness. two neurips experiments @xcite @xcite have shown that the reviewers are good at identifying papers that are clearly bad, but the agreement on the "good" papers appears to be close to random. among the likely reasons for that are cognitive and social biases of nlp reviewers (see overview by @xcite , fundamental disagreements in such an interdisciplinary field as nlp, and acceptance rates that are kept low 1 irrespective of the ratio of high-quality submissions. such arbitrariness leads to understandable frustration on the part of the authors whose jobs and graduation depend on publications, and it also means lost time and opportunities @xcite for science overall. reviews written by someone who does not have the requisite expertise, or does not even consider the given type of research as a contribution, it is a loss for all parties: the authors do not get the intellectual exchange that could improve their projects and ideas, and reviewers simply lose valuable time without learning something they could use. it is also a loss for the field overall: less popular topics could be systematically disadvantaged, leading to ossification of the field @xcite . this paper contributes a snapshot of this problem in nlp venues, based on a survey of authors, reviewers and area chairs (acs). we collected 180 responses, which is is comparable to the volume of feedback collected for implementing the acl rolling review (arr). the overall distribution of respondents' roles is shown in fig. 1 . we present the commonly reported issues and community preferences for different paper assignment workflows (section 4). we derive actionable recommendations to how peer review in nlp could be improved (section 5), discuss the limitations of survey methodology (section 6.2), and conclude with desiderata for interpretable peer review assignments (section 6.3). among the systems used in recent nlp conferences, the softconf matching algorithm takes into account bidding, quotas, and manual assignments, and randomly assigns the remaining papers as evenly as possible foot_0 . naacl and acl 2021 used softconf, but also provided their acs with affinity scores produced by a "paraphrastic similarity" system based on an lstm encoder, which is trained on semantic scholar abstracts @xcite . affinity scores are scores indicating how well a given submission matches a given reviewer. they are typically computed as the similarity (e.g. cosine similarity) between the embeddings of certain information about the submission and the reviewer's publication history (e.g. abstracts and titles). arr switched to openreview and currently uses foot_1 their specter-mfr system @xcite which is based on specter @xcite and mfr embeddings @xcite for computing affinity scores. the assignments are then made with the minmax matching algorithm 4 . the problem of paper-reviewer assignment is by itself an active area of research (see overview of key issues for cs conferences by shah ( 2022 )). there are many proposals for paper-reviewer assignment systems @xcite @xcite , some of which also consider the problem of "fair" assignments @xcite . such studies tend to be hypothesis-driven: they make an assumption about what criteria should be taken into account, design a system and evaluate it. to the best of our knowledge, ours is the first study in the field to address the opposite question: what criteria should be taken into account, given the diversity of perspectives in an interdisciplinary field? we take that question to the community. ## methodology: survey structure and distribution we developed three separate surveys for the main groups of stakeholders in the peer review process: authors, reviewers and acs. they follow the same basic structure: consent to participation (see impact statement), background information, questions on most recent experiences in the role which the survey pertains to, and how the respondents believe paper-reviewer matching should be performed. most questions are asked to respondents in all three roles, reformulated to match their different perspectives. the responses were collected late 2021 and all respondents are required to confirm that their most recent experience as an ac/reviewer/author is in 2019-2021. the full surveys and response data are publicly available foot_3 . ## results overall we received 38 responses from acs, 87 from reviewers and 81 from authors (206 in total). after removing 20 incomplete responses and 8 responses inconsistent with the "trap" questions, we report the results for 30 responses from acs, 77 from reviewers and 73 from authors (180 in total). ## conclusion we present the results of the first survey on paperreviewer assignment from the perspective of three groups of stakeholders in the nlp community: authors, reviewers, and acs. the results point at a host of issues, some immediately actionable (e.g. providing the acs with better information), some normative (e.g. different kinds of research may need different assignment strategies), and some open (e.g. how do we evaluate the effect of any changes to peer review process?) a big issue for both authors and reviewers is mismatches due to lack of interest, which is in tension with explorative aspects of peer review. we recommend to address this issue with a combination of assignments based on bidding and random matches within area, backed up by reviewer training.