An Analysis of Tasks and Datasets in Peer Reviewing (2024)

Abstract: Taking note of the current challenges of the peer review system, this paper inventories the research tasks for analysing and possibly automating parts of the reviewing process, like matching submissions with a reviewer’s domain of expertise. For each of these tasks we list their associated datasets, analysing their quality in terms of available documentation of creation and use. Building up on this, we give a set of recommendations to take into account when collecting and releasing data.

## introduction peer reviewing is a vital part of academic integrity since as early as 1665, when henry oldenburg founded the philosophical transactions of the royal society and established the concept of reviewing the work of others @xcite . since then, the concept of peer review has remained largely unchanged, and nearly every journal has adopted it to select the best publications to publish. in recent decades, peer reviewing has become more of a burden for various reasons, like the overwhelming number of article submissions to review, their often less than optimal quality of content, lack of time to keep up with the majority of published research, or the fact that reviewing is voluntary and researchers find they have less time to review other work. technological advances that allow to share research and cooperate worldwide combined with the increasing number of researchers have caused a dramatic increase in the research published daily. the availability of large language models (llms) has in fact simplified some tasks of the scientific writing process, like proof reading or text reformulation. this means that the time from idea to written research is significantly shorter. in the computer science domain, the number of publications over the last 10 years @xcite @xcite @xcite @xcite @xcite has increased dramat-ically, submissions to acl have increased sevenfold, and submissions to neurips eight-fold foot_0 . accounting for these factors, it is harder to find sufficient numbers of reviewers for such conferences. in 2016, about 33% of the neurips reviewers were phd students @xcite , a percentage that is most likely higher today. the outcome is that there are more inexperienced researchers that have to review larger numbers of submissions, eventually leading to reviews of lower quality. in recent years, there are efforts to automate and optimize the different peer review steps to lighten the load of the reviewers, area chairs, and programme committees 2 . researchers interested in this domain have defined research tasks and collected data to address them. in this work, we make an inventory of tasks and datasets associated with peer review automation and optimization, and give recommendations on how to collect and publish such datasets. without claiming that our work is exhaustive, we see it as a necessary contribution to this field. we examine the available datasets, the tasks they have been used or created for, discussing use cases and automation tasks.this work complements the very recently published dagstuhl proceedings @xcite , and the analyses in @xcite or @xcite . the paper is structured as follows: section 2 gives an account of the tasks and their datasets in the peer review analysis domain. section 3 shows the overlaps between various datasets. section 4 discusses some recommendations on the creation of datasets in the peer review domain. section 5 comments on the current advances in this domain, concluding with section 6. ## datasets and tasks in our effort to understand the research challenges in the scientific peer reviewing domain we analyzed 53 datasets that contain peer reviews. for each of them, mostly created in the last 5 years, we looked at the research tasks for which these sets were used. datasets: of these datasets, 37 cover publications that are in the ml and nlp scientific domains, obtained from the openreview portal foot_2 , 6 are in the nlp domain and obtained from acl rolling review foot_3 , 5 datasets cover multiple domains and journals, 3 obtained from f1000 foot_4 and 2 from peerj foot_5 . in tables 1 to 8 we use the following abbreviations to indicate the source of the collected reviews: or (openreview), acl (acl rolling review), co (conll and coling), f (f1000 journal), pj (peerj), rs (shitmyreviewersays), and sg for self gathered data. though most of the datasets can be used for more than one task, in our work we choose to assign them to the task they were used for at their creation time. what we observed is that most of the researchers in this domain opt to create their own datasets from scratch using the openreview api, instead of using one of the many available datasets. this affects the reproducibility and comparability of research results, as not all datasets are publicly available or well documented to foster their reuse. tasks: reviews can be looked at with different research aims in mind. earlier work @xcite , for example, analyzed the helpfulness of student reviews, while more recent work looks at various aspects of peer reviewing, like discourse analysis @xcite , peer review quality @xcite @xcite or meta review generation @xcite . in the following we will discuss the most common tasks and datasets for peer reviewing. ## dataset similarities and discrepancies in the last years, openreview has become the primary data source for review-related tasks, and, therefore, many of the published datasets overlap. we look more closely into the datasets which, among others, contain the iclr conference review data (35 sets, see most of the datasets were extended with annotations to allow the evaluation of the conducted experiments. however, the number of annotations is rather small (in the hundreds), and do not follow a common structure, to allow the integration of multiple datasets and the interchangeability of datasets. while many of the datasets contain several hundred annotated sentences or reviews, such as d'arcy et al. (2023) and @xcite ; others, like @xcite and @xcite annotated larger amounts of data. the asap-review dataset was extended by manually annotating 8,582 publications with sentence level labels on sentiment and aspect categories @xcite . @xcite automatically aligned more than 3.7 million sentence pairs for revision analysis and labeled the most likely intention behind each edit. furthermore, different conferences use different review guidelines and numerical scoring scales, which can lead to biases if these conferences are combined. for example, ictir allows reviewers to score papers on a scale from -3 to 3, while neurips uses a scale from 1 to 10. ## guidelines for dataset publication as is obvious from the previous sections, the number of datasets recently published in the area of scientific peer review analysis is already considerable, although their re-use is impeded by the nontransparent way in which the data was collected. therefore, we advocate for the adherence to data creation and publication guidelines such that their reuse is facilitated. making sure that such datasets are findable, that they are well documented, and additionally, that machine-readable meta-data about them is available are ways to contribute towards their reuse. the current list of available datasets published following the above-mentioned dagstuhl seminar foot_7 is a first step toward achieving this, however incomplete or lacking machine-interpretable data and meta-data. some of the recommendations we make in the following subsections are, in fact, aligning with the fair principles for data sharing 9 . furthermore, whenever possible, we recommend researchers to describe their datasets through appropriate metadata schema foot_9 . ## discussion the recent dagstuhl proceedings, what can natural language processing do for peer review, suggest that it is mostly up to researchers in the nlp domain to solve many tasks until ai-assisted peer reviewing or automated (peer) reviewing becomes possible. while this is true for most of the tasks in section 2, there are challenges that, we believe, will profit from solutions that don't rely solely on large language models or text analysis. in our work on this paper, we note that research in peer review @xcite @xcite @xcite aim to automatically generate or predict peer reviews, leaving out the use of domain knowledge to measure the impact, novelty or compare the results to previous research results that use same datasets. one can argue that the knowledge is already encoded in llms, we comment that its use is not tractable. to our knowledge, re-viewrobot @xcite is the only tool that uses controlled knowledge-driven-generation, to generate reviews using domain knowledge by accessing multiple knowledge graphs.the quality of the results is, however, heavily dependent on the quality of the information extraction processes that provide the content of the knowledge graphs. in the ir community, information extraction is a long-standing research task, the latest efforts for fine-grained entity extraction being visible in the organization of the rufeers task at trec foot_13 , and the state-of-the-art task at clef foot_14 . we hope that research in information extraction may contribute to automated population of knowledge graphs, noting that the most extensive knowledge graph (open research knowledge graph) @xcite , which supports comparing scientific publications by various parameters (including datasets), is currently curated by hand. in conclusion, we believe connecting the research on llms with information extraction as pursued by the ir community or the semantic web community, can help extract the contributions of papers and compare their results to previous ones and effectively support reviewers in their work. ## conclusion in this work, we gave an overview of 53 different datasets for scientific peer reviewing and the tasks these datasets have been created for. we found several issues with these datasets, such as poor documentation, overlapping data, unclear licensing, and a lack of canonical splits. for this reasons we propose several guidelines for publishing new datasets in this domain, guidelines that, in fact, apply to publishing datasets in general. we also see a need for the creation of a taxonomy of tasks in ai-assisted or automated peer reviewing, and the need for a well-documented dataset that consists of peer reviews from various domains, which can be used for different subtasks. currently, the analysis of peer review texts and tool or model development has been done by researchers in the nlp domain. we hope that researchers in other computer science fields will assist with their expertise. the semantic web community or the information retrieval community, are currently only slightly involved in this process. therefore, we believe that it is necessary to get these communities more involved in this research. this can be done via future shared-tasks, and, although we see the irony, new datasets and tasks that are tailored to make use of the expertise in these domains.