ar X iv E dits: Understanding the Human Revision Process in Scientific Writing (2022)

Abstract: Scientific publications are the primary means to communicate research discoveries, where the writing quality is of crucial importance. However, prior work studying the human editing process in this domain mainly focused on the abstract or introduction sections, resulting in an incomplete picture. In this work, we provide a complete computational framework for studying text revision in scientific writing. We first introduce arXivEdits, a new annotated corpus of 751 full papers from arXiv with gold sentence alignment across their multiple versions of revision, as well as fine-grained span-level edits and their underlying intentions for 1,000 sentence pairs. It supports our data-driven analysis to unveil the common strategies practiced by researchers for revising their papers. To scale up the analysis, we also develop automatic methods to extract revision at document-, sentence-, and word-levels. A neural CRF sentence alignment model trained on our corpus achieves 93.8 F1, enabling the reliable matching of sentences between different versions. We formulate the edit extraction task as a span alignment problem, and our proposed method extracts more fine-grained and explainable edits, compared to the commonly used diff algorithm. An intention classifier trained on our dataset achieves 78.9 F1 on the fine-grained intent classification task. Our data and system are released at tiny.one/arxivedits.

## introduction writing is essential for sharing scientific findings. researchers devote a huge amount of effort to revising their papers by improving the writing quality or updating new discoveries. valuable knowledge is encoded in this revision process. up to january 1st, 2022, arxiv ( https://arxiv.org/ ), an open access eprint service, has archived over 1.9 million papers, among which more than 600k papers have multiple versions available. this provides an amazing data source for studying text revision in scientific writing. specifically, revisions between different versions of papers contain valuable information about logical and structural improvements at documentlevel, as well as stylistic and grammatical refinements at sentence-and word-levels. it also can support various natural language processing (nlp) applications, including writing quality assessment and error correction @xcite @xcite ), text simplification and compression @xcite , style transfer @xcite , hedge detection @xcite , and paraphrase generation @xcite . in this paper, we present a complete solution for studying the human revision process in the scientific writing domain, including annotated data, analysis, and system. we first construct arxivedits, which consists of 751 full arxiv papers with gold sentence alignment across their multiple versions of revisions, as shown in figure 1 . our corpus spans 6 research areas, including physics, mathematics, computer science, quantitative biology, quantitative finance, and statistics, published in 23 years @xcite . to the best of our knowledge, this is the first text revision corpus that covers full multi-page research papers. to study sentence-level revision, we manually annotated fine-grained edits and their underlying intentions that reflect why the edits are being made for 1,000 sentence pairs, based on a taxonomy that we developed consisting of 7 categories. our dataset addresses two major limitations in prior work. first, previous researchers mainly focus on the abstract @xcite @xcite and introduction @xcite sections, limiting the generalizability of their conclusions. in addition, a sentence-level revision may consist of multiple fine-grained edits made for different purposes (see ## constructing arxivedits corpus in this section, we present the detailed procedure for constructing the arxivedits corpus. after posting preprints on arxiv, researchers can continu-ally update the submission, and that constitutes the revisions. more specifically, a revision denotes two adjacent versions of the same paper. 1 an article group refers to all versions of a paper on arxiv @xcite . in this work, we refer to the changes applied to tokens or phrases within one sentence as sentence-level revision. the document-level revision refers to the change of an entire or several sentences, and the changes to the paragraphs can be derived from sentences. ## analysis of document-level revisions as a distinct style, scientific writing needs to be clear, succinct, precise, and logical. to understand what common strategies are used by authors to improve the writing of their papers, we present a data-driven study on document-level revisions in the scientific writing domain. this is enabled by our high-quality manually annotated corpus that consists of 1,790 versions of 751 full papers across 6 research areas in 23 years. 0.0 0.2 0.4 0.6 0.8 1.0 (a) update ratio between the first and last versions of each paper 0 20 40 60 80 100 120 140 160 # of papers the distribution of update ratio paper w/ 2 versions paper w/ >2 versions 0.0 0.2 0.4 0.6 0.8 1.0 (b) update ratio between the adjacent versions of each paper 0 20 40 60 80 100 120 140 160 # of papers the distribution of update ratio paper w/ 2 versions paper w/ >2 versions figure 4: the distribution of update ratio. the figure above demonstrates that papers with more versions are more likely to undergo a significant revision in their life cycle. while the two types of papers have a similar distribution of update ratio between adjacent versions, as shown in the figure below. ## automatic edit extraction and intention identification as manual annotation is costly and timeconsuming, we develop a pipeline system to automatically analyze the revisions at scale. our system consists of sentence alignment, edit extraction, and intention classification modules, which are trained and evaluated on our annotated data. the methods and evaluation results of each step are detailed below. example outputs from our system are presented in figure 11 . ## conclusion in this paper, we present a comprehensive study that investigates the human revision process in the scientific writing domain. we first introduce arx-ivedits, a new annotated corpus of 751 full arxiv papers with gold sentence alignment across their multiple versions of revisions, and fine-grained span-level edits together with their underlying intents for 1,000 sentence pairs. based on this highquality annotated corpus, we perform a series of data-driven studies to analyze the common strategies used by the researchers to improve the writing of their papers. in addition, we develop automatic methods to analyze revision at document-, sentence-, and word-levels. our annotated dataset, analysis, and automatic system together provide a complete solution for studying text revision in the scientific writing domain.