DISAPERE : A Dataset for Discourse Structure in Peer Review Discussions (2022)

Abstract: At the foundation of scientific evaluation is the labor-intensive process of peer review. This critical task requires participants to consume vast amounts of highly technical text. Prior work has annotated different aspects of review argumentation, but discourse relations between reviews and rebuttals have yet to be examined. We present DISAPERE, a labeled dataset of 20k sentences contained in 506 review-rebuttal pairs in English, annotated by experts. DISAPERE synthesizes label sets from prior work and extends them to include fine-grained annotation of the rebuttal sentences, characterizing their context in the review and the authors’ stance towards review arguments. Further, we annotate every review and rebuttal sentence. We show that discourse cues from rebuttals can shed light on the quality and interpretation of reviews. Further, an understanding of the argumentative strategies employed by the reviewers and authors provides useful signal for area chairs and other decision makers.

## introduction peer review performs the essential role of quality control in the dissemination of scientific knowledge. the recent rapid increase in academic output places an immense burden on decision makers such as area chairs and editors, as their decisions must take into account not only extensive manuscripts, but enormous additional amounts of technical text including reviews, rebuttals, and other discussions. one long term goal of research in peer review is to support decision makers in managing their workload by providing tools to help them efficiently absorb the discussions they must read. while machine learning should not be used to produce condensed accounts of the peer review text due to the risk of amplifying biases @xcite , ml tools could nevertheless help manage information overload by identifying patterns in the data, such as argumentative strategies, goals, and intentions. any such research requires an extensive labeled dataset. while the openreview platform @xcite has made it easy to obtain unlabeled public peer review text, labeling this data for supervised nlp requires highly qualified annotators. correct interpretation of the discourse structure of the text requires an understanding of the technical content, precluding the use of standard crowdsourcing techniques. prior work on discourse in peer review has focused this qualified labor force on labeling arguments extracted from the text, which enables the complete annotation of more examples, at the expense of research on non-argumentative behaviors in peer review. while there has been extensive research and deep analysis of different aspects of peer review, the taxonomies used to describe review argumentation are disparate and not directly compatible. finally, there has been limited research into understanding the discourse relations between rebuttals and reviews @xcite , and none so far into the discourse structure of rebuttals. this paper presents disapere (discourse structure in academic peer review), a dataset focusing on the interaction between reviewer and author foot_0 . we give reviews and rebuttals equal importance, and emphasize the relations between them. to enable the study of behaviors beyond the core arguments, we also annotate every sentence of both the review and rebuttal, and provide fine-grained labels for non-argumentative types. we annotate at the sentence level not only for completeness but also to avoid the propagation of errors from argument detection. we annotate four properties (review-action, fine-review-action, aspect, polarity) of each review sentence, where the set of properties and their values were developed by synthesizing taxonomies from prior work. we also annotate each sentence of a rebuttal with a fine-grained label indicating the author's intentions and commitment, and a link to the set of review sentences that form its context. figure 1 shows the disapere annotation scheme on a minimal, fictional example review-rebuttal pair. disapere is intended as a comprehensive and high-quality test collection, along with training data to fine-tune models. our annotations are carried out by graduate students in computer science who have undergone training and calibration, amounting to over 850 person-hours of annotation work. much of the test data is double-annotated, and we report inter-annotator agreement on all aspects of the annotation. we describe the performance of state of the art models on the tasks of predicting labels and contexts, showing that interesting ambiguities in the data provide the nlp community with research challenges. we also show an example that demonstrates how decision makers could use models like these to understand trends and inform policies for future conferences ( § 5). the contributions of this paper are as follows: (1) a new labeled training dataset of 506 reviewrebuttal pairs (over 20k sentences) of peer review discussion text in english, where review sentences are annotated with four properties, and rebuttal sentences are annotated with context and labels from a novel scheme to describe discourse structure; (2) a taxonomy of discourse labels synthesizing prior work on discourse in peer review and extending it to add useful subcategories; (3) a summary of the performance of baseline models on the dataset ( § 6); ( 4 ) examples of analyses on the dataset that could benefit peer review decision makers ( § § 4 and 5), and (5) extensive annotation guidelines and software to support future labeling efforts. ## related work the design of this dataset draws upon extensive, but disparate prior work on this topic. many works, some addressed below, have taken advantage of the availability of review text hosted on openreview. argument-level review labeling prior work has developed label sets that address different phenomena. @xcite introduced the study of discourse structure in peer review by annotating argumentative propositions in the ampere dataset with a set of labels tailored to the peer review domain (evaluation, request, fact, reference, and quote). similarly, @xcite 's amsr dataset frames the problem as an argumentation process, in which the stance of each argument towards the paper's acceptance or rejection is of paramount importance. both view peer review as argumentation, using argument mining techniques to highlight spans of interest. while its goal is not to examine discourse structure per se, @xcite uses polarity labels to indicate each argument's support or attack of the authors' bid for acceptance. besides polarity, these examples follow @xcite by annotating each argument with the aspect of the paper it comments on. 2 in contrast to @xcite , we do not attempt or recommend generat-ing peer review text, instead focusing on analyzing human-generated text in peer review. ## dataset each example in disapere consists of a pair of texts: a review and a rebuttal. labels for reviews and rebuttal sentences are described below. review sentence labels are summarized in ## baselines two types of machine learning tasks can be defined in disapere. first, a sentence-level classification task for each of the four review labels and the two levels of rebuttal labels. second, an alignment task in which, given a rebuttal sentence, the set of review sentences that form its context are to be predicted. the models described below are not intended to introduce innovations in discourse modeling, rather, we intend to show the off-the-shelf performance of state-of-the-art models, and indicate through error analysis the phenomena that are yet to be captured. ## conclusion as the burden of academic peer reviewing grows, it is important for program chairs and editors to act upon data-driven insights rather than heuristics, to make the best possible use of participants' scarce time. models trained on data like disapere will allow decision makers to glean deep insights on the interactions occurring during peer review. almost all publicly available peer review data is from the domain of artificial intelligence, limiting the scope of disapere and any similar project. while this means that models trained on disapere won't necessarily generalize to all new domains, we hope that with the detailed annotation guidelines and seamless data collection using the software provided with this paper support, users 8 ethics the outcomes of peer review can have outsize effects on the careers of participating scholars. as machine learning models are known to amplify biases, we strongly recommend against using the outputs of any machine learning system to make decisions about individual cases. a dataset like disapere is best used to survey participants' behavior. any interventions based on this information should be subjected to studies in order to ensure that they do not introduce or exacerbate bias.