NLP eer: A Unified Resource for the Computational Study of Peer Review (2023)

Abstract: Peer review constitutes a core component of scholarly publishing; yet it demands substantial expertise and training, and is susceptible to errors and biases. Various applications of NLP for peer reviewing assistance aim to support reviewers in this complex process, but the lack of clearly licensed datasets and multi-domain corpora prevent the systematic study of NLP for peer review. To remedy this, we introduce NLPeer– the first ethically sourced multidomain corpus of more than 5k papers and 11k review reports from five different venues. In addition to the new datasets of paper drafts, camera-ready versions and peer reviews from the NLP community, we establish a unified data representation and augment previous peer review datasets to include parsed and structured paper representations, rich metadata and versioning information. We complement our resource with implementations and analysis of three reviewing assistance tasks, including a novel guided skimming task. Our work paves the path towards systematic, multi-faceted, evidence-based study of peer review in NLP and beyond. The data and code are publicly available.

## introduction research publication is the primary unit of scientific communication. to ensure publication quality and to prioritise research outputs, most scientific communities rely on peer review @xcite -a distributed procedure where independent referees determine if a manuscript adheres to the standards of the field. despite its utility and wide application, peer review is an effortful activity that requires expertise and is prone to bias @xcite @xcite ). an active line of research in nlp for peer review strive to address these challenges by supporting the underlying editorial process (e.g. @xcite @xcite , decision making (e.g. @xcite @xcite , review writing @xcite , and by studying review discourse (e.g. @xcite @xcite @xcite . despite the methodological advancements, several factors prevent nlp research for peer review at large. the computational study of peer review lacks a (1) solid data foundation: reviewing data is rarely public and comes with legal and ethical challenges; existing sources of peer reviewing data and the derivative datasets are not licensed, which legally prevents reuse and redistribution @xcite . peer reviewing practices vary across research communities (walker and rocha da silva, 2015; bornmann, 2011) -yet the vast majority of nlp research in peer review so far focused on a few machine learning conferences that make their data available through the openreview.net platform (e.g. @xcite . a (2) multi-domain perspective on peer review is thus missing, and the transferability of findings between different communities and reviewing workflows remains unclear. finally, a (3) unified data model for representing peer reviewing data is lacking: most existing datasets of peer reviews adhere to task-specific data models and formats, making it hard to develop and evaluate approaches for peer review support across datasets and domains. to address these issues, we introduce nlpeer. we apply a state-of-the-art workflow @xcite to gather ethically and legally compliant reviewing data from natural language processing (nlp) and computational linguistics (cl) communities. we complement it with multi-domain reviewing data from the f1000 research 3 platform and historical data from the openly licensed portion of the peerread @xcite corpus. the resulting resource (figure 1 ) is the most comprehensive collection of clearly licensed, open peer reviewing datasets available to nlp to date. nlpeer includes peer reviews, paper drafts and revisions from diverse research fields and reviewing systems, over the time span from as early as 2012 until 2022. this -for the first time -enables systematic computational study of peer review across domains, communities, reviewing systems and time. the paper revisions make nlpeer well-suited for the study of collaborative text work. to facilitate the analysis, we unify the datasets under a common data model that preserves document structure, non-textual elements and is well suited for cross-document analysis. to explore the new possibilities opened by our resource, we conduct cross-domain experiments on review score prediction (to encourage consistent review scores), pragmatic labeling (to encourage balanced reviews) and the novel guided skimming for peer review task (to help guide review focus), along with easy-to-extend implementations. our results indicate substantial variation in performance of nlp assistance between venues and research communities, point at synergies between different approaches to review structure analysis, and pave the path towards exploiting cross-document links between peer reviews and research papers for language model benchmarking. in summary, this work contributes (1) the first unified, openly licensed, multi-domain collection of datasets for the computational study of peer review, including (2) two novel datasets of peer reviews from the nlp and cl communities, and complemented by a (3) descriptive analysis of the resulting data and (4) extensive experiments in three applied nlp tasks for peer reviewing assistance. there exist different implementations of peer review, including blind review (where reviewer and/or author identities are hidden to promote objectivity) and open review (identities are open). moreover, reviewing standards and practices vary by research community, venue and publication type. review forms and templates are one important varying factor; in the following, we differentiate between unstructured (only one main text) and structured (several predefined sections) review forms. we refer to the particular implementation of peer review at a certain venue as reviewing system. along with the natural domain shift based on research field, the reviewing system contributes to the composition of the peer reviewing data it produces. ## statistics the datasets in nlpeer originate from different domains and peer reviewing systems, each with its own policies, reviewing guidelines, and community norms. while a comprehensive comparison of publishing and reviewing practices lies beyond the scope of this work, this section provides a brief overview of the key statistics for nlpeer datasets. ## reviewing assistance with nlpeer nlpeer is a unique resource for the study of computational assistance during peer review authoring. to demonstrate its versatility, we define three tasks -review score prediction, pragmatic labeling and guided skimming -that are anchored in realistic assistance scenarios and targeted towards helping junior reviewers to improve their review. figure 5 illustrates the tasks. for readability, we use s to 11 the paper vocabulary overlap follows a similar trend, see appendix (figure 7 ). denote sentences, par for paragraphs, and sec to denote sections for reviews (e.g. s r ) and papers (e.g. s p ), respectively. we report the main results here, and refer to the appendix b for details and the limitations section for an ethical discussion of risks and opportunities of these tasks. ## further applications nlpeer includes rich representations and metadata for a large number of inter-linked peer review and manuscript texts. this enables new nlp studies for peer reviewing assistance and beyond. in this section we highlight and critically reflect on further applications of nlpeer in general nlp research and in practice. peer review reports are expert-written texts that typically reflect deep understanding of the underlying paper. hereby, they can serve as a valuable resource for distant or direct supervision of generalpurpose machine learning models. for instance, paper summaries in review reports can be used as a basis for a challenging document summarization task; aspect score disagreements between reviews may serve as a weak supervision signal for aligning arguments for and against the paper across reviews. on the other hand, nlpeer and peer reviewing data in general might be exploited to train models with dual uses and practical risks. one such example is automatic review generation, where given a paper the model should generate a review report. while a resulting model may be used benevolently as a paper-writing aid or an analytical device for the study peer review, using it to avoid the reviewing effort or replace the reviewer bears a wide range of risks. as it is implausible that current state-ofthe-art nlp models could produce a non-generic, meaningful review of a novel paper @xcite , such applications could compromise the academic quality assurance process. due to the central role of peer review in research and publishing, we emphasize that future work should carefully reflect on the real-world impact of any models developed based on nlpeer. ## conclusion and intended use we have presented nlpeer-the first clearly licensed large-scale reference corpus for the study of nlp for peer review. nlpeer opens many new opportunities for the empirical study of scholarly communication. for nlp, it allows developing new annotated datasets based on clearly licensed, richly formatted, unified corpora that span multiple research domains, reviewing systems and time periods. it can be used as a testing ground for domain transfer @xcite , and enables the study of cross-document relationships between papers, paper revisions and peer reviews @xcite . from the meta-scientific perspective, it allows comparing peer reviewing practices across research communities, and can provide crucial insights into how researchers review and revise scientific texts. our task implementations and results can guide the development of nlp assistance systems and can be used for systematically comparing pre-trained language models in the context of peer reviewing applications. finally, our resource can serve as a blueprint for future aggregate resources for the study of peer review in nlp. for their support during the implementation of the data collection at arr. we express our gratitude to nuria bel for her feedback during the first iterations of our data collection initiative at coling-2020, and to richard gerber for helping us with early technical challenges in softconf. last but not least, we would like to thank our reviewers for their valuable suggestions, as well as the community members who have engaged in a lively debate on this initiative and provided us with both encouragement and useful feedback. this research work has been funded by the german federal ministry of education and research and the hessian ministry of higher education, research, science and the arts within their joint support of the national research center for applied cybersecurity athene. it is co-funded by the european union (erc, intertext, 101054961). views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the european union or the european research council. neither the european union nor the granting authority can be held responsible for them. finally, parts of this work have been co-funded by the german research foundation (dfg) as part of the peer project (grant gu 798/28-1). this work is part of the intertext initiative 13 .