Re3: A Holistic Framework and Dataset for Modeling Collaborative Document Revision (2024)

Abstract: Collaborative review and revision of textual documents is the core of knowledge work and a promising target for empirical analysis and NLP assistance. Yet, a holistic framework that would allow modeling complex relationships between document revisions, reviews and author responses is lacking. To address this gap, we introduce Re3, a framework for joint analysis of collaborative document revision. We instantiate this framework in the scholarly domain, and present Re3-Sci, a large corpus of aligned scientific paper revisions manually labeled according to their action and intent, and supplemented with the respective peer reviews and human-written edit summaries. We use the new data to provide first empirical insights into collaborative document revision in the academic domain, and to assess the capabilities of state-of-the-art LLMs at automating edit analysis and facilitating text-based collaboration. We make our annotation environment and protocols, the resulting data and experimental code publicly available.

## introduction textual documents are a key medium of information exchange in the modern world. these documents often result from a collaboration of multiple individuals. the typical process of collaborative text production involves iterations of drafting, getting feedback (reviews), executing revisions, and providing responses that outline the implemented changes, serving as a vital element in facilitating effective communication @xcite . despite the importance of collaborative text revision and its high potential for nlp applications, we are missing a framework that formally describes this review-revisionresponse procedure grounded in real-world data. while prior work in nlp has studied relationships between original and revised documents @xcite 2022; @xcite , reviews and original documents @xcite , reviews and revisions @xcite , and reviews and responses @xcite -no prior frameworks allow jointly modeling all three components of text-based collaboration. yet, such joint modeling is important as it provides deeper insights into the processes involved in text work, and opens new opportunities for nlp applications. important tasks that involve reviews, revisions and responses such as edit summarization thus remain underexplored. comprehensive analysis of document-level revisions poses additional challenges. contrary to sentence-level analysis, hierarchically structured documents @xcite bring distinct levels of granularity into editing. individuals execute revisions at various granularity levels, with a range of actions and a spectrum of intents, reflecting the what, how, and why of the revisions (figure 1 and §3.2). realistic modeling of document revision in text-based collaboration thus requires datasets and annotations that encompass the entire document context, incorporating all edits made across various levels of granularity, and providing qualitative labels for both action and intent. we further term this kind of analysis as full-scope modeling of document revision. prior research in nlp has primarily studied sentence-level edits while neglecting the broader document context @xcite , variations in granularity @xcite , and the underlying intent behind the edits @xcite . there is thus a gap in both methodologies and datasets for creating and analyzing full-scope annotations of document revisions, limiting our grasp of the intricate nature of the editing process. to close this gap and enable a comprehensive study of text-based collaboration in nlp, we introduce re3: the first holistic framework for modeling review, revision and response in collaborative writing ( §3). we instantiate our framework in the scholarly domain and create re3-sci, the first large-scale human-annotated dataset that comprises 11.6k full-scope revision annotations for over 300 revised documents with substantial inter-annotator agreement (iaa), as well as cross-document connections between reviews, revisions and responses ( §4). our framework and dataset, for the first time, enable large-scale empirical investigation of collaborative document revision, including edit localization and clustering within documents, edit mechanisms and motivations inferred through action and intent labels, and the impact of review requests ( §5). manually analyzing the complex relationships between reviews, revisions and responses is costly, and constitutes a promising nlp automation target. facilitated by our data, we present a first exploration of the capability of large language models (llms) to address novel revision assistance tasks, such as review request extraction, revision alignment, edit intent classification and document edit summarization ( §6). our work thus makes four key contributions: ## the reframework the re3 framework builds upon the recently introduced intertextual model by @xcite . in particular, we represent documents as graphs that preserve document structure, allowing us to work on different levels of granularity, and treat cross-document relations as edges between the corresponding document graphs. ## dataset analysis the framework and dataset allow us to answer new questions about human behavior in collaborative document revision in scholarly publishing. rial, with deletions being infrequent and partition changes even less common. it also suggests that the enhancement of fact or evidence is the primary focus of revisions, highlighting its importance in improving scientific quality. moreover, figure 4 illustrates that additions and deletions of sentences typically pertain to improving factual content or claims, but are never intended for superficial language enhancement. on the other hand, grammar and clarity improvements are usually realized by modifying existing sentences. this suggests that, from a modeling view, the edit action and intent labels may influence the prediction of each other. rq2: how are edits distributed in documents? figure 5 indicates that the initial and final parts of papers experience significantly more revisions. in terms of edit actions, the beginning of the document typically sees more modifications, while the end is characterized by a higher frequency of additions and deletions. regarding edit intents, language enhancements for grammar or clarity are more common in the early parts, whereas changes affecting semantic content, such as facts or claims tend to occur more in the later parts. these suggest that the document position may be a valuable predictor for identifying edit actions and intents. rq3: how significant are the differences between document versions? to gauge the magnitude of change, we introduce the edit ratio metric, determined by the ratio of sentence edits to the sentence count in the original document. while the edit ratio reflects the extent of differences, the significance of document revisions is highlighted by the semantic edit ratio, which is calculated by the ratio of semantic edits labeled with fact/evidence or claim. the average document edit ratio stands at 18.45%. figure 6a and figure 10 in §c show that the majority of documents experience moderate revisions with an edit ratio of 5-25%, while a small proportion has an edit ratio exceeding 50%, and only a few documents appear to have been extensively rewritten. the average semantic edit ratio stands at 11.18%, with most documents showing 0-20% of their content undergoing significant change. notably, documents with a high edit ratio often do not correspond to a high semantic edit ratio, suggesting that documents with extensive revisions typically exhibit language quality issues. rq4: how are edits clustered by paragraphs and sections? we use crest factor (cf), a concept borrowed from signal processing @xcite , to assess the concentration of edits. using a vector of sentence edit counts in each paragraph or section, cf quantifies the peak amplitude of this distribution. a cf value of 1 signifies an even distribution. the average paragraph cf is 3.79, indicating a substantial concentration of edits within a limited number of paragraphs. this trend of high edit concentration in a few paragraphs is further illustrated in figure 6b and figure 11 in §c. when examined at the section level, the average cf is 2.54, indicating a moderate tendency towards clustering. rq5: are reviewers' requests acted upon? how are these realized in revision? annotators categorize the relevant review requests into three types: explicit edit suggestions (28%), implicit edit suggestions (32.1%), and general weakness comments (39.8%). the first provides specific document locations and clear revision instructions; the second delivers guidance without locations; the last highlights general issues without specific suggestions. figure 7 shows that more than half of explicit suggestions are implemented, with 18.47% actualized through multiple sentence edits. implicit suggestions and general weakness comments are realized to a lesser extent. this implies that reviewers' explicit suggestions are more likely to be acted upon. ## automation with llms the re3-sci dataset facilitates a variety of nlp tasks, such as (1) edit intent classification, (2) document edit summarization, (3) revision alignment and ( 4 ) review request extraction. tasks ( 1 ) and ## conclusion we have introduced the re3 framework and the re3-sci dataset, for empirical analysis and development of nlp assistance for text-based collaboration. through annotation study and data analysis, we have demonstrated the utility of the framework and revealed novel insights into human behavior in collaborative document revision and peer review, including relationships between specific edit actions and intents, focused localization patterns, clustering tendencies within paragraphs, and the acceptance rates for review requests. our automation experiments have assessed the icl and cot capabilities of state-of-the-art llms on four tasks for collaborative revision assistance. in the classification tasks with llama2-70b, we noted that using default static icl demonstrations with cot rationale produces satisfactory results, demonstrating the efficacy of this straightforward yet effective prompting strategy. in the document edit summarization task, gpt-4 demonstrated the ability to generate coherent summaries but faced challenges related to factuality and comprehensiveness. our work paves the path towards systematic fullscope study of text-based collaboration in nlp and beyond. the framework, taxonomy, annotation methods and tools are applicable to diverse domains. the dataset offers a robust foundation for multifaceted research for collaborative revision assistance. future work may encompass tasks like identifying text segments necessitating revision and generating revisions guided by review requests or specified actions and intents.