P ara R ev : Building a dataset for Scientific Paragraph Revision annotated with revision instruction (2025)

Abstract: Revision is a crucial step in scientific writing, where authors refine their work to improve clarity, structure, and academic quality. Existing approaches to automated writing assistance often focus on sentence-level revisions, which fail to capture the broader context needed for effective modification. In this paper, we explore the impact of shifting from sentence-level to paragraph-level scope for the task of scientific text revision. The paragraph level definition of the task allows for more meaningful changes, and is guided by detailed revision instructions rather than general ones. To support this task, we introduce ParaRev, the first dataset of revised scientific paragraphs with an evaluation subset manually annotated with revision instructions. Our experiments demonstrate that using detailed instructions significantly improves the quality of automated revisions compared to general approaches, no matter the model or the metric considered.

## introduction in the scientific domain, writing assistance is crucial as researchers share their findings through articles published in conferences or journals. however, writing articles is challenging and time-consuming, notably for non-native english speakers or young researchers @xcite . the field of writing assistance has grown rapidly to address these challenges leading to the development of various tools (grammarly, trinka ai 1 , . . . ) and specialized workshops (in2writing, wraicogs 2 ). due to past limitations in processing long texts, prior research has focused on the sentence revision task (see figure 1 ). in this task, a sentence is given to a seq2seq model or a large language model (llm) along with a general revision prompt, which could take the form of a label (e.g., coherence, style) @xcite or a general instruction @xcite . in this definition of the task, labels are assigned to specific modifications within a sentence, targeting particular spans of text to revise. thanks to the recent advances in nlp in the past years, we propose to expand the traditional scope of this sentence-level paradigm to detailed personalised instructions guiding the model on revisions to conduct at the paragraph level, as illustrated in figure 1 . we argue that this new paradigm aligns better with how human writers revise the text and how llms are used today, allowing more comprehensive changes such as merging, splitting, or reorganizing sentences. additionally, personalised instructions enable more nuanced control over the degree of revision, specifying whether minor edits or major restructuring is required. they can also target specific areas within a paragraph, while other sentences provide essential context. to support this task, we introduce pararev, a corpus of paragraphs revised by their authors annotated with human revision intention labels and instructions (e.g. in figure 2 ). our contributions are as follows: existing corpora for scientific text revision provide aligned versions of revised texts, with varying scope. some datasets focus only on the abstract and introduction sections of scientific papers @xcite @xcite , while others include full-length articles @xcite @xcite . most of these resources align revisions at the sentence level, though paragraphlevel reconstruction is possible to capture broader, more substantial revisions. however, not all datasets include revision annotations with explicit intention labels. some, such as those designed for tasks related to peerreview @xcite , focus on tracking changes without offering structured guidance for the revision process. in revision tasks, having an explicit revision intention is crucial for guiding models in performing meaningful modifications. in sentence-level revision datasets, individual modifications (i.e. spans of text) are commonly associated with a label indicating the revision intention. the taxonomies for these labels can vary across corpora @xcite . however, labels associated with short spans of text often lack the contextual information needed for more substantial, long-range revisions. they also do not provide the specificity that detailed instructions could offer to guide more precise edits. recent efforts have attempted to bridge this gap by converting labels into general instructions to better align with how llms are utilized for revision @xcite . our work aims to extend this approach by introducing detailed, personalized paragraph-level instructions that provide richer contextual and precise guidance for revisions. ## dataset construction figure 3 summarizes the overall data pipeline described in this section. ## dataset statistics the dataset contains 48 203 paragraph pairs from 16 664 pairs of revised articles. from this total 48k paragraphs, 641 were manually annotated (470 were double annotated). this subset was chosen to represent the overall corpus based on paper revision extent: 218 paragraphs are from heavily revised papers (where over 19 paragraphs are revised), 213 from moderately revised papers (4-5 revised paragraphs), 210 from low revised papers (1-2 revised paragraphs). figure 4 shows the label distribution across the dataset. for fairness in the analysis, when annotators picked two labels, they were weighted 0.5 each. additionally, paragraphs with only one annotation are counted twice. the figure distinguishes between paragraphs from articles with different degrees of revision. heavily revised papers tend to mainly feature rewriting revisions, suggesting that the entire document was evenly reworked. in contrast, low-revised papers are more likely to involve small content modifications, such as adding or removing forgotten information. finally, we report the instructions' distribution as follows: of the 641 annotated paragraphs, 328 have no instruction, 55 have one, and 258 have two. these 258 paragraphs form our evaluation set in section 5. ## impact of task definition on revision to verify our hypothesis that using detailed instructions better guides the revision process compared to generic instruction labels, we conducted a comparative experiment. for this, we evaluated how different models performed when given either a general prompt mapped from an intention label or a personalised instruction tailored to the specific changes needed (see appendix e). we experimented with multiple models to ensure the results were robust across various architectures: coedit foot_1 , a t5-based model fine-tuned on sentence revision task @xcite , as well as llama3 foot_2 , mistral foot_3 , and gpt-4o, state-of-theart foundation models with strong language understanding and generation capabilities. all models are used in zero-shot, the prompt used is given in appendix e. additionally, as a control baseline, we included a copyinput method, which does not apply any edits to the input paragraph. to assess the quality of revisions, we employed traditional sentence revision metrics, rouge-l (lin, 2004) and sari @xcite , alongside bertscore @xcite to measure similarity between the generated and gold revised paragraphs. the results are summarized in across all models, we observed consistent improvements when using detailed instructions over general prompts. they are even statistically significant for mistral, llama3, and gpt-4o, with p-values below 0.05 (paired student's t-test). the experiment confirms our hypothesis: instructions that provide specific revision guidance allow the models to produce more accurate revisions compared to relying solely on general labels. however, when examining the performances of the models, we observe that the copyinput and coedit achieve the best results. a manual overview of a subset of outputs reveals that co-edit only suggests minor changes, such as grammar corrections, while other models propose more substantial modifications. evaluation remains a significant challenge in the text revision domain, as widely used metrics compare the proposed revision to a single reference version. this approach penalizes revisions that deviate from the gold standard, even if they result in valid improvements. consequently, unless the model's modifications exactly replicate those made by the original author, the score will be lower than proposing no modifications (copyinput). this limitation need to be address in future work to develop more robust and reliable evaluation methods for this task. ## conclusion we proposed a definition of the scientific text revision task at paragraph-level, enabling more contextaware revisions using full-length instruction. additionally, we presented pararev, a dataset of revised paragraphs, with an evaluation split annotated with revision instructions. our experiments demonstrate that providing detailed personalised instructions leads to more effective revisions than general ones, across multiple models. in future work, as manual annotation is costly and time-consuming, we aim to annotate the remaining non-annotated wide split of the dataset automatically. this silver dataset will then be used to fine-tune an open-source model specifically for paragraph-level revision tasks.