A Tale of Two Revisions: Summarizing Changes Across Document Versions (2024)

Abstract: Document revision is a crucial aspect of the writing process, particularly in collaborative environments where multiple authors contribute simultaneously. However, current tools lack an efficient way to provide a comprehensive overview of changes between versions, leading to difficulties in understanding revisions. To address this, we propose a novel task of providing thematic summary of changes between document versions, organizing individual edits based on shared themes. We assess capabilities of LLMs on this task and further introduce three strategies to tackle this task: (i) representing the input of two documents along with edits in the ‘diff’ format (ii) a two-stage task decomposition with individual edit description generation as an intermediate task and (iii) clustering based chunking and subsequent merging techniques for handling longer documents. Our experiments demonstrate the effectiveness of our approach in improving the model’s capacity to handle this complex task. Additionally, we introduce ChangeSumm, a curated dataset comprising human-written thematic summaries for pairs of document versions, to facilitate evaluation and further research in this direction.

## introduction writing, inherently, is a dynamic and iterative process marked by the continual evolution of ideas, the assimilation of feedback and the collaborative efforts that shape the narrative @xcite @xcite . in this journey, text revision emerges as a pivotal aspect-an adaptive process that involves discerning disparities between intended and realized text, determining the necessary edits, and deciding how to implement those changes @xcite @xcite . in the contemporary landscape, with the proliferation of collaborative document authoring systems introduces an additional layer of complexity with simultaneous edits * work done during internship at adobe research by multiple authors turning the revision process into a labyrinth, challenging the comprehension of the document's evolution and resulting in inconsistencies @xcite . while collaborative authorship fosters creativity and diverse perspectives, it presents the challenge of efficiently managing changes @xcite . consider a scenario where multiple authors or a single author make edits to a document using the track-change feature in document authoring and version management tools. eventually, an author or an approver need to review these changes to incorporate them. current tools display individual edit operations like 'insert' or 'delete', necessitating users to skim through each edit one by one to understand the overall changes across versions. this process makes it challenging to discern the cumulative effect of edits made over the whole document. therefore, it becomes essential to reimagine the presentation of change logs, aiming for a design that facilitates swift human consumption. while there has been prior work on understanding revisions, their main focus has been identifying the intent behind each atomic edit @xcite @xcite @xcite @xcite @xcite . moreover, those intents deal with categorizing surface level edits at fine-grained level such as grammaticality, fluency, clarity, readability etc and associate addition of any new information to a broad intent of content level change. though this can be viewed as some grouping of edit operations based on intent, it predominantly focuses on each edit in isolation, without considering the entire context of the document. moreover, they lack an easy navigable way for users to understand various topical theme edits present in these content-level changes. our proposition centers around providing a condensed bird's-eye view of edits by offering a summary of changes between versions of the document organized based on shared common themes. each theme in the summary is detailed with attributions to individual edits, providing a navigable way to review. for example, in figure 1 , we display two versions of a document with edits scattered throughout, but by providing overarching themes such as 'adding details of new dataset, unsupervised methods, and expert-based manual analysis', we alleviate the burden on the user to figure out the themes of the edits. more than a mere tracking tool, this assistant empowers approvers to effortlessly view and assess groups of edits emerging from shared themes with a single click, eliminating the need to review individual changes one by one. such a method could further support the development of a system that can automatically identify inconsistencies across the document, suggest edits to maintain a cohesive narrative, and generate a response elucidating how feedback has been incorporated, thereby enhancing collaboration productivity. to the best of our knowledge, our work represents the first attempt to define this problem and curate an evaluation-only dataset changesumm containing 45 high-quality human-written thematic summaries along with attribution to individual edits. these versions, spanning both pre-review and camera ready revisions, from nlp-related scientific papers of the nlpeer dataset @xcite . such thematic summaries would greatly assist reviewers to comprehend how their feedback is incorporated into the document, fostering a more collaborative reviewing experience. we evaluated the performance of gpt-4 and gpt-3.5 on our proposed task and found that they fail to correctly spot the edits between two versions exhibiting edit-hallucination, generating a span of text as an edit, when in fact, it may not represent an actual modification, and, at times, produce text that is not present in the original document. to tackle this complex task, we explore three strategies: (i) providing input in the 'diff' format, explicitly indicating edits, alleviating the edit-hallucination problem. (ii) two-stage task decomposition approach which involves generating edit description for each edit in the first stage, which are further used to form thematic clusters and summaries. (iii) a simple chunking-merging strategy for the first stage task of edit description generation and clusteringbased chunking based on an external embedding model, to pack related edit descriptions belonging to each cluster into chunks to obtain a thematic summary in second stage, to handle the challenge of longer documents. our experiments validate that incorporating diff formatted input, two-stage task decomposition and clustering-based chunking strategies improve the llm's capability to handle this complex task. in summary, our contributions are: (a) we propose the novel task of generating a thematic summary of changes between two document versions, along with attributions to individual edits. (b) we curate changesumm, an evaluation-only dataset of 45 high-quality human-written thematic summaries for nlp-related scientific paper versions. (c) we explore three strategies to tackle this complex task using llms. our annotation guidelines and dataset are available at https://github. com/natwar-modani/changesumm . advancing on prior works, yang et al. 2017 proposed a semantic taxonomy of edit intentions for wikipedia, including clarification, elaboration, and fact updates etc. subsequently, wikipedia revisions have become a focal point in research on edit intent identification @xcite @xcite . these revisions have also served as valuable corpora for a spectrum of nlp tasks @xcite @xcite , covering areas such as textual entailment (zanzotto and pennacchiotti, 2010), sentence simplification @xcite , and addressing spelling errors and paraphrases @xcite . moreover, diverse corpora have been curated for edit intent identification from domains such as wikihow @xcite , argumentative essays @xcite @xcite @xcite , news articles @xcite , and scientific documents @xcite @xcite . these corpora served as valuable resources for various tasks, including writing quality assessment @xcite @xcite ), text simplification and compression @xcite , style transfer @xcite , and hedge detection @xcite , within their respective domains. beyond edit understanding, some works leverage feedback/comments/reviews to address the alignment task, aiming to identify the mapping between edits and the corresponding triggering comments @xcite @xcite in our work, we carve a distinct niche by targeting the generation of thematic summaries for edits given the document versions, an aspect not explored in prior works. while existing tasks on edit understanding primarily concentrated on each word/sentence-level edit in isolation, our primary focus is on the global context of the whole document, considering related edits comprehensively. document-level long text tasks: recent efforts, exemplified by @xcite , introduced scrolls, a suite of tasks (summarization @xcite @xcite , question answering @xcite @xcite , and natural language inference @xcite ) requiring reasoning over long texts. subsequently with focus on llm evaluation, @xcite proposed zeroscrolls, an evaluation-only zeroshot benchmark derived from scrolls and and two other aggregation tasks @xcite , addressing long text reasoning. such benchmark creation for long context capability evaluation include longbench bai et al. 2023 (both english and chinese), l-eval @xcite and bamboo @xcite . @xcite llms for document-level machine translation. in our work, we evaluate whether llms possess the capabilities to understand and cluster edits between versions, providing a thematic-level summary. we believe that this task can serve as a challenging testbed for evaluating llms in handling long contexts. ## task definition & dataset curation the task of the thematic summary generation is to provide a concise overview of edits between two versions of a document, organized based on common themes. given the original (source) and revised (target) versions of a document, the task involves clustering the individual edits into groups, where edits within the same group share common themes and generating a descriptive summary for each thematic group, providing an overview of the underlying theme and associated changes in that thematic group. however, defining what constitutes a theme depends on individual interpretation. we define a theme as the collective set of edits deemed to be stemming from a specific comment or feedback from a reviewer, grouped under a single concept. while acknowledging the potential subjectivity, our definition provides a pragmatic framework for operationally tackling this task. notably, thematic groups form a many-to-many mapping: one thematic group includes several edits and one edit can be associated with multiple thematic groups. to construct a dataset with source and target paper and corresponding thematic summaries with attribution to edits under each theme, we first need to define the unit of edit. while phrase-based difference offers a fairly precise definition of the scope of a edit, it may be difficult to achieve consistent annotations and would also turn out laborious resulting in too many edits. we define our unit of revision to be at the sentence level. in other words, even if a sentence contains multiple edits, the entire sentence will be annotated as one edit unit (resulting in many-to-many mapping between edit units and themes). to extract the edit unit, we initially use a character-based text comparison algorithm first and then build edit units at sentence-level based on the differences extracted @xcite . edit segmentation: we use the neil fraser's google-diff-match-patch library foot_0 based on myer's diff algorithm @xcite enhanced with prediff speedups and post-diff semantic cleanups to produce the list of differences as a contiguous sequence of deleted, inserted or equal words. the difference is represented by a sequence of edit segments where each edit segment is a pair, where first element represents the operation {deleted, inserted, equal} and and second element in a sub-string which belongs to either or both documents. edit unit construction: to distill the edit segments into sentence-level edit units, we perform a linear traversal on the list of edit segments to identify the start and end indices for each target sentence. simultaneously, we keep track of the substring of the source text that is covered within these index bounds of the edit segments list, creating an alignment pair between both the target sentence and the substring of source text covered in those bounds. if the target-source alignment pair covers all 'equal' edit segments (i.e., the same sentence in both source and target), we mark it as a non-edit unit. otherwise, we mark the pair as an edit unit. during this traversal process, if there is a span of source text deleted between two target sentences, we include it as an edit unit with no corresponding target sentence in the alignment pair. conversely, if no source text appears within the bounds of the target sentence, the corresponding target text is marked as an edit unit with the alignment complement in the source set to null. detailed algorithm is presented in appendix 1. within each source-target alignment pair in the edit-units, we further detect and mark changes at the word level (i.e., deletions, insertions, and modifications). we achieve this by utilizing the difflib library foot_1 , which tokenizes the sentences using the spacy tokenizer foot_2 and processes the input as a list of words. this approach enables us to efficiently highlight the differences within those pairs. for a visual representation of the edit units obtained through this method, refer to left pane in figure 2 where every edit unit is followed by 'assign to group' button. revision corpus: we leverage the nlpeer dataset @xcite , comprising nlprelated scientific papers submitted for review and their corresponding camera-ready versions as source and target versions for our task. these papers originated from submissions to acl-17 and conll-16 conferences collected as part of the @xcite , with newly curated additions from coling-20 and arr-22 submissions. we obtain the parsed pdf documents from nlpeer dataset. subsequently, we tokenize the obtained paragraphs into sentences using the scispacy tokenizer @xcite . utilizing the segmentation and edit unit algorithms described earlier, we extract sentence-level edit units from these document versions. we select document versions based on the total number of edit-units where the total count of edit-units falls within the 30-70 percentile range across the corpus. annotation workflow: figure 2 illustrates the annotation interface developed for creating the changesumm corpus, which involves writing thematic summaries for the revision corpus collected above. the left side of the interface presents the target document interspersed with the source document in the form of edit units. each edit unit is marked and equipped with a button to assign it to thematic groups, which are dynamically populated as the user creates them. the right-hand pane serves as the workspace for annotators to create new thematic groups, displaying edit units already assigned to each group. once all the edits are attributed to thematic groups, annotators can provide a final summary for each group, highlighting the theme and edits associated with it. to alleviate the annotation burden for common edits like spelling corrections and grammar updates, we automatically detect them using heuristics based on edit distance and root word analysis, assigning them to a thematic group of surface edits. however, annotators have the flexibility to make changes as needed. annotator recruitment: due to the timeconsuming nature and the need for high annotation quality given the complexity of the task, we opted not to use amazon mechanical turk. instead, we recruited a small group of annotators from the upwork freelancing platform. we deliberately refrained from imposing rigid guidelines that might stifle the creativity and opted for hands-on training sessions with annotators with discussions and providing examples illustrating effective and ineffective clustering for themes. they are provided a pre-annotation exercise for which they received detailed feedback to align their understanding with the meta-definition of theme. we release all our annotation guidelines and examples to encourage broader collaboration. finally, they were assigned the task only if they successfully passed the pre-annotation exercise. ultimately, we hired 7 upwork freelancers and obtained a total of 95 responses. given the subjective nature of the task, we then reviewed the responses and selected only those responses which were of good quality, and discarded the ones with quality issues (e.g., having clusters which categorized changes as insertion/deletion, etc., instead of theme or topic based grouping). please note that the review of responses was done completely obviliously to the model outputs and only based on the quality of annotations only. upwork workers were compensated with a base rate of $7-10 for each response (irrespective of their response getting selected or not). changesumm dataset inspection on average, each version pair consists of 75 edits, with the number of edits per pair varying between 25 and 172. thematic summaries produced by annotators comprise an average of 14.5 themes per pair, ranging from 5 to 28 themes. the average number of edits attributed to each theme is 5.33, with the range spanning from 1 edit per cluster to 60. around 25% of total edits are associated with multiple (more than 1) themes (13% mapped to 2 different themes and rest more than more than 2) and the rest assigned to a single theme. ## method we propose a two stage task decomposition approach which takes the input of two documents in the diff format and produces description of each edit in first stage which are then passed as input into second stage to produce the clusters of edits based on shared theme and respective thematic description for each cluster. to account for longer inputs exceeding the context size of llms, we follow divide and conquer approach in both stages of the task. in the first stage, we chunk the input to pass into llm and simply concatenate the obtained edit descriptions from each chunk. for longer input (edit descriptions obtained from first stage) in the second stage, we employ clustering based chunking approach wherein we cluster the edit descriptions using an embedding model and pack edit descriptions belonging to each of the clusters into chunks based on context length of llm and and then concatenate the obtained chunk-level thematic summaries to obtain the final thematic summary. detailed prompts for our method are presented in appendix c. input format design: we present the input to the llm in the textual diff format using xml tags. the input consists of the target document interspersed with the source document in the form of edit units. each edit unit is enclosed @xmath0, where x is a specific number identifying the edit used for attributing each edit to thematic groups. furthermore, word-level edits are represented @xmath1. we keep the unit of text as paragraph to provide sufficient context to llm for generating the description for diffs (so if a 5 sentence paragraph has change in one sentence, we still feed the entire paragraph to llm, but identify the edits at the sentence level). two stage task decomposition: we postulate that an effective thematic summary generator should demonstrate the capability to comprehend each edit in isolation and subsequently group them based on shared themes. to achieve this, we introduce a twostage task decomposition, with the intermediate task of edit description generation. in the first stage, we present the model with input in the diff format, prompting the llm to describe each edit in free-form text, capturing the underlying intent behind the modification. this initial stage focuses on the local context of individual edits. in the second stage, we provide the generated edit descriptions and prompt the llm to generate a thematic summary. this involves clustering the edits into groups based on shared themes (referred to as attributing edits to thematic groups) and providing a theme description for each cluster. we instruct the model to produce output with attributions in inline citation format, where each theme description is followed by edit numbers to represent the attribution. in this stage, the emphasis shifts to establishing shared themes across all edits globally within the document. handling longer documents: when the length of the diff input exceeds the context window size of the llm in first stage, we divide it into smaller, non-overlapping chunks. chunking is performed at the paragraph level, ensuring that edit units do not get separated at the boundaries of chunks and we only pass those paragraphs with atleast an edit unit. subsequently, all edit descriptions from each chunk are concatenated directly to obtain the first stage output. for the second stage process, in order to handle concatenated edit descriptions that exceed the maximum context size of llms, we employ a clustering-based chunking. we convert each edit description into embeddings using the sentence-bert model @xcite and then cluster these embeddings using affinity propagation clustering @xcite . we then pack edit descriptions from the same cluster into the same chunk and pack as many clusters as possible into each chunk based on the context length of the llm. each of these chunks is passed into the second stage llm, resulting in thematic summaries for each chunk which are then concatenated into a final summary. as an alternative, we also pack edit descriptions in order of their occurrence in the llm calls and make an llm call to consolidate them into final summary clusters (using prompt shown in appendix c.5.3). ## experiments & discussion we use gpt-4 and gpt-3.5 models (openai, 2023) with 32k and 16k context window respectively, tem-@xmath2.7, top-@xmath3.95 for all our experiments. we employ all-minilm-l6-v2 model to obtain embeddings for clustering. we use pydantic foot_3 and jxnl instructor foot_4 library to enforce structured format of output for easy parsing. ## conclusion we introduced the task of providing thematic summaries of changes between document versions, aimed at offering users a comprehensive overview of edits organized around shared themes. to facilitate evaluation, we curated the changesumm dataset, consisting of human-written thematic summaries for changes between document versions. our experiments with llms revealed the challenging nature of this task and underscored its potential as a benchmark for assessing llm performance on long-context document-level tasks. to address the complexities involved, we proposed three strategies: (1) using the 'diff' format to represent document versions and edits, mitigating edit hallucination. (2) introducing a two-stage task decomposition approach with edit description generation as an intermediate step. (3) employing clustering-based chunking and merging techniques to address the limited context length of llms. overall, we hope our work fuel further research in this direction to foster collaboration productivity in document revision processes.