Automatic Analysis of Substantiation in Scientific Peer Reviews (2023)

Abstract: With the increasing amount of problematic peer reviews in top AI conferences, the community is urgently in need of automatic quality control measures. In this paper, we restrict our attention to substantiation — one popular quality aspect indicating whether the claims in a review are sufficiently supported by evidence — and provide a solution automatizing this evaluation process. To achieve this goal, we first formulate the problem as claim-evidence pair extraction in scientific peer reviews, and collect SubstanReview, the first annotated dataset for this task. SubstanReview consists of 550 reviews from NLP conferences annotated by domain experts. On the basis of this dataset, we train an argument mining system to automatically analyze the level of substantiation in peer reviews. We also perform data analysis on the SubstanReview dataset to obtain meaningful insights on peer reviewing quality in NLP conferences over recent years. The dataset is available at https://github.com/YanzhuGuo/SubstanReview.

## introduction peer review is an essential practice for gauging the quality and suitability of scientific papers in the academic publication process @xcite . however, in recent years, this step has been widely criticised for its unreliability, especially in top artificial intelligence (ai) conferences @xcite . this is partially due to the surge in numbers of submitted papers and the shortage of domain experts fulfilling the requirements to serve as reviewers @xcite . this leads to increasing workload per reviewer and paper-vetting by less-experienced researchers @xcite , which consequently increases the likelihood of poor-quality reviews. therefore, developments that can make the quality control process associated with peer reviewing more efficient, are likely to be welcomed by the research community @xcite . to this end, our research aims to develop an automatic system to analyze the quality of peer reviews. such a system would be of great value not only to chairs, who could use them to eliminate identified poor-quality reviews from the decision-making process, but also to reviewers, who might be instructed to improve the writing at the time of reviewing. they could be exploited by conference managers as well to analyze overall review quality and reviewer performance, in order to better organize the next review round. clearly, and according to the review guidelines of several top ai conferences, including iclr, icml, neurips, acl and emnlp, reviews should be assessed from multiple perspectives (e.g., domain knowledgeability, factuality, clarity, comprehensiveness, and kindness). in this paper, we decide to focus on one specific quality dimensionsubstantiation, which has appeared in nearly all review guidelines, sometimes under different names such as specificity, objectiveness, or justification. this criterion states that a good review should be based on objective facts and reasoning rather than sentiment and ideology. specifically, each subjective statement (claim) in the review should be backed up by details or justification (evidence). more discussion on substantiation will be provided in section 3.1. to progress towards the goal of automatically evaluating the level of substantiation for any given review, we employ an argument mining approach. scientific peer reviewing can be viewed as an argumentation process @xcite , in which reviewers convince the program committee of a conference to either accept or reject a paper by providing arguments. in our annotation scheme for arguments (see section 4.2), the two basic components are claims and evidence. a substantiated argument is one where the claims are supported by evidence. therefore, we formulate the task of claim-evidence pair extraction for scientific peer reviews. we release substanreview, a dataset of 550 peer reviews with paired claim and evidence spans annotated by domain experts. on the basis of this dataset, we develop an argument mining system to perform the task automatically, achieving satisfactory performance. we also propose substanscore, a metric based on the percentage of supported claims to quantify the level of substantiation of each review. finally, we use the substanreview dataset to analyze the substantiation patterns in recent conferences. results show a concerning decrease in the level of substantiation of peer reviews in major nlp conferences over the last few years. our contributions are threefold: a quality aspect highly relevant to substantiation, is "justification", previously studied by @xcite as an evaluation measure for their automatic review generation model. more specifically, they state that "a good review should provide specific reasons for its assessment, particularly whenever it states that the paper is lacking in some aspect". however, their evaluation protocol for justification relies solely on human annotators. our work is the first one to automatically assess the substantiation level of scientific peer reviews. more importantly, we do not only provide a final quantitative score, but also highly interpretable claim-evidence pairs extracted through an argument mining approach. argument mining. @xcite define the task of argument mining as the automatic identification and extraction of argument components and structures. the state-of-the-art in argument mining was initially based on feature engineering, while more recent methods rely on neural networks @xcite , especially following the introduction of the transformer architecture. previous works applying argument mining to peer reviews typically focus on identifying argumentative content and classifying it. @xcite introduced the ampere dataset containing 400 reviews annotated for proposition segmentation and proposition classification (evaluation, request, fact, reference, quote, or non argument) and trained neural models to perform the two tasks. similarly, @xcite performed annotations on 70 reviews for supporting arguments, attacking arguments and non arguments, and trained a bert model for the tasks of argumentation detection and stance detection. none of these efforts take into account the structure of arguments, making our work the first to examine the link between different argument components in scientific peer reviews. ## task formulation in this section, we first take a closer look at substantiation and its definition. we then formulate its estimation as the claim-evidence pair extraction task. finally, we introduce fundamental concepts in argumentation theory and explain how the claimevidence pair extraction task can be tackled by an argument mining approach. ## substanreview dataset in this section, we introduce substanreview, the first human annotated dataset for the task of claimevidence pair extraction in scientific peer reviews. we first discuss the source of the reviews and then elaborate on the annotation process. finally, we provide a statistic overview of our annotated dataset. ## experiments and results we tackle the claim-evidence pair extraction task formulated in section 3.2 and construct a benchmark for the substanreview dataset. the claim tagging is treated as a token classification task while the evidence linkage is approached as a questionanswering task. we solve both of these tasks by fine-tuning pretrained transformer encoders ## conclusion in this paper, we focus on automatically analyzing the level of substantiation in scientific peer reviews. we first formulate the task of claim-evidence pair extraction, comprised of two subtasks: claim tagging and evidence linkage. we then annotate and release the first dataset for this task: substanreview. we perform data analysis on substanreview and show interesting patterns in the substantiation level of peer reviews over recent years. we also define substanscore, which positively correlates with human judgements for substantiation. finally, we develop an argument mining system to automatically perform claim-evidence pair extraction and obtain a great increase in performance over the baseline approach. we hope that our work inspires further research on the automatic analysis and evaluation of peer review quality, which is an increasingly important but understudied topic.