Learning to Model Editing Processes (2022)

Abstract: Most existing sequence generation models produce outputs in one pass, usually left-to-right. However, this is in contrast with a more natural approach that humans use in generating content; iterative refinement and editing. Recent work has introduced edit-based models for various tasks (such as neural machine translation and text style transfer), but these generally model a single edit step. In this work, we propose modeling editing processes, modeling the whole process of iteratively generating sequences. We form a conceptual framework to describe the likelihood of multi-step edits, and describe neural models that can learn a generative model of sequences based on these multistep edits. We introduce baseline results and metrics on this task, finding that modeling editing processes improves performance on a variety of axes on both our proposed task and related downstream tasks compared to previous single-step models of edits.

## introduction revising and editing are a central part of the the human creative workflow, with most original content (e.g. art, books, articles, source code) being developed not in a single iteration, but in many iterations with each more refined than the last. how can we model these editing processes from inception to completion? in this paper, we attempt to provide a first answer to this question, specifically focusing on generation of sequential data such as natural language documents or source code. most current work on language generation tasks such as machine translation @xcite , language modeling @xcite , or summarization @xcite generates the target sentence or document in a single pass (usually from left to right). there has been a reasonable the animal cute a is dog the a is dog domestic domesticated desccendant wolf the of the animal cute a is dog and is friendly the animal an is dog and is descended from wolf the insert delete replace keep amount of work that can generate edits to existing sequences for the purposes of post-editing, grammatical error correction @xcite ), text style transfer @xcite @xcite , sentence fusion @xcite , or machine translation @xcite . however, these works all 1) model only a single editing step and 2) do not fully define a model of incrementally editing a document from a blank slate to the final text, and thus do not stand in for the one-pass generative models of sequences described above. in this context, we propose the task of modeling editing processes, in which we look to explicitly model the likelihood of the entire process of revising a document to a polished form. in particular, and in contrast to previous works on modeling edits, we hypothesize that in order to edit more accurately, instead of simply learning to predict the next revision given the current revision, we should have context of multiple previous revisions when deciding when and how to edit the document next. given the novelty of framing generation problems in this way, this paper simultaneously 1) proposes both baseline and novel models for the task, 2) creates evaluation datasets that can be used to compare models, and 3) discusses intrinsic and extrinsic evaluation methodology. the proposed multi-step editing model predicts discrete edit operations @xcite to enable progressive refinement as shown in figure 1 , rather than framing sequence editing as a sequence to sequence task @xcite . in the figure, for each step of the editing process discrete operations (insert, replace, delete, keep) are predicted and then actions (such as generating a replaced span) are performed based on this. this has two benefits: 1) it allows the model to scale well with respect to input sequence length, and 2) allows us to make substantial changes with fewer actions @xcite . we use these edit operations to condition a semi-autoregressive model that is able to insert and replace multiple spans at once. combined with an encoder that is able to quickly specify which spans of text need to be changed and how, this allows for considerable changes to be made to the text (including insertion, deletion, re-ordering, and replacement) in a relatively simple and cheap manner. furthermore, this allows us to disentangle how likely the model is to operate (replace, delete, etc.) on a given span, and how likely the model thinks the generated text for a given span is. as we are modeling editing processes, and hypothesize that context from edits applied to the sequence are helpful, we propose a method for edit-aware sequence compression which can compress sequences into their edit operations and use relative edit positional embeddings to specify the position of edits relative to each other. given that the task of modeling natural editing processes in itself is novel, we collect new datasets to study this behavior; wikirevisions and coderevisions. these datasets, in the code and natural language domains respectively, cover over 2.5m and 2.3m natural sequential revisions. we also discuss evaluation methodology, describing a metric of edit perplexity (eppl), the perplexity of generating an edit given the current state of a document, as well as applications to downstream tasks. we train and evaluate our proposed models on these datasets and find that the proposed methodology of modeling the entire editing process, referencing previous edits while generating the next one, significantly improves both intrinsic and extrinsic performance baselines that model edits in isolation. in particular, our method reduces perplexity by up to 22.9% relative over a state-of-the-art editing baseline, and 11.3% relative over a version of our model that does not consider editing history. we also demonstrate the ability of the model to generate qualitatively natural edit sequences, and the utility of the learned representations on downstream tasks of commit message generation @xcite and edit intention classification @xcite . ## problem definition let @xmath0, x 1 , . . . , x n } be a series of n versions of a document, where the ith revised document is denoted by x i . x 0 represents an initial state (generally the null string), and x n represents the current state of the edited document. the probability of this series of document versions occurring can be decomposed as note that standard generative models (specifically language models; lms) calculate the probability of only the final version p(x n ), whereas the proposed formulation calculates the probability of the entire sequence of document edits. it nonetheless could theoretically be used to calculate the final version's likelihood by treating the editing process as latent and marginalizing over it thus, our formulation, in contrast to previous single-step models of edits @xcite @xcite , can also be used to define a generative model over single documents. it is also worth noting that the final document likelihood is lower-bounded by the edit likelihood; i.e. p(x n ) ≥ p(x). ## modeling editing processes in this section, we now describe our approach to actually modeling these sequences of edits through (1) a decomposition of the modeling process into a sequential process of modeling edit operations then actual edits, and (2) neural model of modeling these operations and edits. ## data while some datasets of edits exist @xcite , to our knowledge they only consider a single editing step, i.e. dealing with a document @xmath1, x 1 }, @xmath2. as we propose learning to model multi-step edits, we develop new datasets in both the code and natural language domains. in addition, previous datasets have only concerned themselves with atomic edits @xcite which only occur at a small scale (usually sentence-level), and we instead look to model larger-scale edits as document level changes, which are more representative of the natural editing process. ## conclusions in this work, we proposed the novel task of modeling editing processes, in which we model the likelihood of documents by way of their natural editing processes. we develop new datasets and curate existing datasets for downstream tasks. we find that modeling editing processes is beneficial to this end, in contrast to modeling single-order edits, as has been done in much of previous literature. more broadly, we believe that tackling iterative refinement tasks in broader nlp may be beneficial given its resemblance to the natural generative and creative process. in future work, we look to investigate methods for transferring pre-trained edit models to a wider range of nlp tasks. for instance, there are other widely acknowledged editing tasks in nlp such as grammatical error correction @xcite or style transfer @xcite . the technical challenge in applying the models presented here to these tasks lies in the lack of datasets annotated with multi-step edits, and potential methods to overcome this challenge would be creative use of existing datasets (e.g. identifying relevant edits in wikirevisions), or latent variable learning methods to approximate the marginal in equation 2 such as variational auto-encoders @xcite .