ARIES : A Corpus of Scientific Paper Edits Made in Response to Peer Reviews (2024)

Abstract: We introduce the task of automatically revising scientific papers based on peer feedback and release ARIES, a dataset of review comments and their corresponding paper edits. The data is drawn from real reviewer-author interactions from computer science, and we provide labels linking each reviewer comment to the specific paper edits made by the author in response. We automatically create a high-precision silver training set, as well as an expert-labeled test set that shows high inter-annotator agreement. In experiments with 10 models covering the state of the art, we find that they struggle even to identify which edits correspond to a comment—especially when the relationship between the edit and the comment is indirect and requires reasoning to uncover. We also extensively analyze GPT-4’s ability to generate edits given a comment and the original paper. We find that it often succeeds on a superficial level, but tends to rigidly follow the wording of the feedback rather than the underlying intent, and lacks technical details compared to human-written edits.

## introduction feedback on paper drafts, whether from co-authors, readers, or reviewers, can be challenging to interpret and address. consider a reviewer who wants authors to use a more realistic dataset in their evaluation. this could be expressed in a variety of ways: it could be stated as a direct request ("apply the method to a realistic dataset"), or more indirectly as a criticism ("the evaluation is only on a synthetic dataset") or question ("is the current dataset truly representative of the real-world?"). similarly, an author editing the manuscript in response has several options. they could comply with the request, or clarify that no realistic datasets are publicly available, or even argue that the reviewer is mistaken and add a justification of their dataset's realism. properly interpreting the request and then selecting a suitable edit in response requires deep reasoning and expertise. while existing nlp systems produce fluent and coherent text, their capabilities remain limited on the most demanding writing tasks that require interpretation and reasoning. research on systems that can interpret complex writing feedback and edit documents in response could enable dramatically improved writing assistants, in scientific writing and elsewhere. however, as we discuss in section 2, datasets for studying such tasks are limited, and scientific writing represents a particularly challenging and understudied domain. in this paper, we introduce and study the task of revising scientific papers based on peer feedback. to facilitate research on the task, we release aries (aligned, review-informed edits of scientific papers), a real-world dataset of computer science paper drafts from openreview, the corresponding reviewer feedback, and the author responses and revisions that address the feedback. 1 to link the review comments to their corresponding edits, we devise an automatic method based on author responses to produce silver data at scale, and obtain labels from experts to form a gold test set. we believe aries is the first dataset to enable training and evaluation of models on contentful edits in a technical domain. using the dataset, we formulate two novel tasks, shown in figure 1 . the first is comment-edit alignment, in which a model determines which review comments made about a paper correspond to each of the edits made after the feedback. the second task is edit generation, in which a model generates edits directly from a given reviewer comment and paper text. the alignment task involves identifying rather than generating appropriate edits; it serves as a stepping stone toward the more challenging generative task and admits automatic evaluation. in comment-edit alignment, a model is given a review comment and set of candidate edits derived from a source paper and a revised target paper, and it must align the comment to the edit(s) that are associated with it. in edit generation, a model is given a review comment and a source paper and must generate an edit that addresses the comment, possibly using placeholders for missing information. we evaluate ten baseline methods and find that the alignment task is challenging for existing models, including even large models such as gpt-4, and that comments and edits with indirect relationships are especially difficult. for the generation task, we find that gpt-4 does produce edits that are coherent and on-topic on a surface level, but fails to model the underlying intent; unlike real authors, it almost never makes edits that suggest the feedback is mistaken, often paraphrases the feedback rather than tightly integrating edits into the paper, and tends to include less technical detail. our contributions are as follows: the best model (gpt-4) achieves only 27.0 micro-f1 compared to human performance of 70.7 (section 5). ## related work to our knowledge, our work is the first to study contentful edits conditioned on complex feedback in a highly technical domain (scientific papers). previous work on edit modeling either focuses on stylistic and grammatical edits @xcite @xcite @xcite or incorporates no feedback or very different kinds of feedback-such as instructions @xcite @xcite @xcite or post-hoc descriptions of edits @xcite @xcite . those settings do not present the same challenging reasoning requirements as our tasks. figure 2 illustrates how the content and linguistic complexity of review comments differs substantially from that of the conditioning information used in past work. argrewrite @xcite is a dataset of student essay revisions with teacher feedback, and contains some contentful comments, but the essays are very short (~500 words) compared to scientific papers (~5000 words) and the comments are not aligned to specific edits. some work does explore scientific-domain edits, but these do not associate edits with reviewer comments and often focus on classification rather than generation. @xcite and @xcite analyze and tag edit intentions on arxiv papers but it would be of interest to provide numerical experiments on more "realistic" data and tasks (instead of the toy model presented in section 7). did you try training post-ln transformers and pre-ln transformer with different # of layers from scratch (i.e., different l)? repeatedly, the claim is made that the model is more than 10x smaller than "current large scale sequence models" but out of the evaluated baselines only "bart" is that much larger. ## task definitions as shown in figure 1 , we consider two versions of the task of determining how a document should be edited to address a given piece of feedback: comment-edit alignment and edit generation. both tasks express the differences between an original (source) document and revised (target) document as a list of edits, where each edit represents a specific change from source text at some location in the paper into new text in the target paper. specifically, an edit consists of a paragraph in the source and its corresponding revised paragraph in the target, where either paragraph (but not both) can be null in the case of deletions or additions. in the comment-edit alignment task, the goal is to identify the edit(s) that correspond to a given review comment. the input is a comment and a list of edits, which include both original and revised text. in our evaluation, we derive the list of input edits by using a paper's gold revisions, but they could consist of any candidate revisions. the output is a set of binary classifications over the list of edits, indicating whether each edit addresses the comment. note that this results in a many-to-many mapping; one comment may result in several edits to the paper, and (less commonly in our data) multiple comments may be addressed by one edit. in the edit generation task, the objective is to generate appropriate edits to a paper based on feedback. the input for this task consists of a comment and the original paper text. the output is the generated edit, which should address the reviewer's feedback and be coherent within the context of the paper. ## dataset construction both the comment-edit alignment and edit generation tasks require a dataset with paper edits aligned to specific feedback comments. in this section, we provide an overview of our approach for collecting and annotating aries, a corpus of cs paper revisions and reviews with both manual and synthetic annotations of comment-edit alignments. on a high-level, the process is as follows: first, we obtain a corpus of paper draft pdfs, their peer reviews, and revised drafts from openreview (section 4.1) and manually identify token spans in reviews that represent actionable feedback (section 4.2). after identifying actionable feedback comments, we manually identify the (paragraphlevel) edits that correspond to each comment to obtain a small (196 comments) but high-quality dataset for evaluating models (section 4.3). to obtain a larger dataset (3.9k comments) suitable for training models, we develop a synthetic labeling approach to automatically extract comments and align them to edits based on the observation that author responses often directly quote review comments and respond with text that is very similar to the corresponding edit text (section 4.4). ## comment-edit alignment in this section, we evaluate models on the commentedit alignment task using our constructed dataset. as described in section 3, the comment-edit alignment task is a binary classification task where the input is a review comment and a list of candidate edits, and the output is binary for each commentedit pair, specifying whether the comment and edit are aligned. in model inputs, edits are textually represented using a "diff" format with additions and deletions enclosed in [+ +] and [--] brackets. for manually-annotated data, for a given comment, we consider all edits for the corresponding paper as candidate edits, labeled as positive if the edit was annotated as addressing the comment and negative otherwise. given the low recall of the synthetic data (discussed in section 4.4), we can only use the synthetic labels to produce positive comment-edit alignment pairs; thus, we pair com-ments with edits sampled from other documents as negative candidates. additional details are provided in appendix e. ## conclusion and future work in this work, we have introduced the novel tasks of comment-edit alignment and edit generation for scientific paper revisions based on high-level draft feedback from reviewers. we have constructed and released aries, a dataset containing computer science paper drafts, reviews, author responses, and it is interesting that models (including gpt-4) do so poorly on the comment-edit alignment task despite gpt being able to generate plausible edits in the generation task. as our analysis shows, the kinds of edits produced by gpt can be very different from the real edits authors make to their papers, and the fact that gpt fails to recognize many of the real comment-edit pairs suggests that it may have gaps in its reasoning that would be interesting to explore further in future work. we hope that the insights from our analyses can help motivate and guide future studies.