Text Editing by Command (2021)

Abstract: A prevailing paradigm in neural text generation is one-shot generation, where text is produced in a single step. The one-shot setting is inadequate, however, when the constraints the user wishes to impose on the generated text are dynamic, especially when authoring longer documents. We address this limitation with an interactive text generation setting in which the user interacts with the system by issuing commands to edit existing text. To this end, we propose a novel text editing task, and introduce WikiDocEdits, a dataset of single-sentence edits crawled from Wikipedia. We show that our Interactive Editor, a transformer-based model trained on this dataset, outperforms baselines and obtains positive results in both automatic and human evaluations. We present empirical and qualitative analyses of this model’s performance.

## introduction a long-standing goal of natural language processing research has been to generate long-form text @xcite @xcite . recent large generative language models such as gpt-2 @xcite , and gpt-3 @xcite , demonstrate an impressive ability to generate fluent text, but their outputs are difficult to control beyond a prompt, and they manifest a tendency to hallucinate facts @xcite . much recent work has thus focused on making such models more controllable @xcite @xcite , and factually grounded @xcite . most such work only considers a one-shot generation setting. given a set of inputs, which may be a prompt, a control code @xcite , or a table of data @xcite for example, the system generates text in a single step. humans, though, often produce text through an evolutionary process involving multiple draft-edit cycles. this is not simply because they make mistakes when writing, but because they may require multiple iterations to help them shape and even make sense of what they want to express @xcite . for example, consider a user writing an article about barack obama. they might start with a simple sentence such as "barack obama was the 44th president of the united states". next, they may wish to expand on that sentence, adding information, or rephrasing it to integrate it better with the text. replicating this process in software will mean allowing users to adjust their requirements in response to model outputs. even an error-free system that meets all of a user's initial requirements does not obviate the need for iteration, since those constraints are themselves dynamic. while this work focuses on text, we also note that these arguments extend to other settings where a system must generate a complex, structured object for a user, such as image or code generation. the purpose of this paper is to bring into view the task of controllable text editing, as a step beyond one-shot generation towards interactive document generation. a full interactive document generation system will likely comprise multiple components, possibly including one-shot generation to create a first draft. editing is crucial to interactivity because it allows users to change previously generated text to fit their dynamic constraints. this is a stateful operation, where the state is the current version of the document, as opposed to stateless recasting of text from scratch using a one-shot model. while services like grammarly or ms word already offer rewriting suggestions, they mainly focus on syntactic or stylistic edits such as paraphrases @xcite . in this work, we are interested in a broader range of edits, particularly those that add or remove content, or change the meaning of text. figure 1 illustrates this editing setting with an example from our trained model, where a user produces a sentence about barack obama over multiple edits. in sum, we make the following contributions: we introduce a challenging new text editing task, wherein a model must learn to edit text in response to a user command, while drawing on grounding to avoid problems of hallucination @xcite . to accompany this task, we release an open-source dataset of sentence-level edits extracted from wikipedia, including editor comments, which we leverage as natural language commands, together with pre-retrieved grounding documents. we show that a transformer-based editing model trained on our data outperforms "parrot" and gpt-2 baselines, and obtains competitive results compared to gold-standard edits in human evaluations. we then perform an empirical analysis of our model's performance, showing the importance of the command and grounding, and the varying difficulty of edits in our dataset. ## text editing task we now formalize our text editing task. let d be a document, q a user command foot_0 , and g some appropriate form of grounding. moreover, let d be an edited version of d. then our task is, given a dataset of edits @xmath0, q 0 , g 0 , d 0 ), ..., (d n , q n , g n , d n )}, learn to produce document d , given d, q, and g. note that while previous work on text editing usually only considers d as input, we include both a form of control q and grounding g. the command is needed because otherwise the type of edit to be made is undefined, while the grounding provides external knowledge needed to make an edit. in our specific instance of this task, we will only consider sentence-level edits. more formally, we consider edits d -→ d , where d and d differ only on a single sentence @xmath1, respectively @xmath2. while, in general, edits can vary in complexity from document-level to character-level changes, sentences are a natural way to break down text into relatively independent units of meaning, so it makes sense to edit text one sentence at a time. more complex, document-level edits can be seen as a composition of multiple sentence-level edits. additionally, we will consider user commands q written in natural language, e.g., "add years in office". the command could also take other forms, such as a categorical variable, but natural language allows for the greatest flexibility in specifying what the edit should accomplish. moreover, natural language commands are a good fit for our model, which we will initialize with pretrained language model weights. for similar reasons, we will also consider corpora of text snippets as our grounding g. alternatively, the grounding could also consist of structured data such as tables or graphs. in a real user scenario, this grounding might be supplied by the user, or retrieved on the fly. for our dataset, we pre-retrieve groundings by querying a commercial search engine. ## data to accompany our text editing task we present a novel dataset of nearly 12 million sentence-level edits, wikidocedits. these edits were extracted from the revision histories in the february 1, 2020 dump of english wikipedia. 3 for a given wikipedia page, a revision consists of a source and target text, corresponding to the old and new versions of the page. each revision is also accompanied by an editor comment, which we will use as a proxy for the user command. for a given revision, we split the source and target texts into sentences and then attempt to match the sentences between source and target. for efficiency, we only look at a k-sentence neighborhood. unmatched sentences are candidates for edits. a source sentence s and target sentence t form an edit pair s -→ t if f (s, t) > , where f is sentencelevel bleu foot_2 without smoothing @xmath3.1 in our case. if an unmatched source sentence does not form an edit pair with any target sentence, we consider it to be a sentence deletion. this can also be thought of as matching to an empty sentence. we identify sentence insertions in an analogous manner. importantly, we only consider revisions that contain a single sentence-level edit. otherwise, the editor comment that accompanies each revision may only describe one of the possibly many sentence-level edits. see appendix a for a detailed description of our processing pipeline. ## model we formalize our model, which we refer to as interactive editor, as a standard auto-regressive sequence to sequence model. because our data only contains single-sentence edits, we assume that the sentence to be edited in the source document is given as an input to the model. given a source sentence @xmath4, the context around s, which we will refer to as d by abuse of notation, a user command q, a grounding corpus g, and a candidate target sentence s , the model, f , computes we use the same encoder-decoder architecture as t5 @xcite and initialize our model with pretrained language model weights. the encoder-decoder architecture allows us to perform full attention over the inputs s, d, q, and g, while the decoder allows us to auto-regressively generate s . meanwhile, initializing with pretrained weights has been shown to achieve state-of-the-art results on many nlp tasks @xcite . in order to adapt t5 for our task, we represent all our inputs as sequences of tokens. we then concatenate these sequences together using separator tokens, truncating and padding them to fixed lengths. this is straightforward since all our inputs are text. see fig. 2 for reference. we also use the standard cross-entropy loss to train. ## experiments we train our model on a subset of ∼1,020k edits from wikidocedits. we use a training/validation/test split of 1,000k/10k/10k edits, and train for 3 epochs with a fixed learning rate of 0.0001, and a batch size of 128. we use the t5-base implementation from huggingface @xcite , and finetune all weights in the model. we validate every 200 steps and select the model with the lowest validation loss. ## related work grounded generation large language models can generate fluent text @xcite @xcite ), but they have a tendency to hallucinate facts @xcite . thus, several works have explored using various forms of grounding to enable models to generate factually consistent texts @xcite @xcite @xcite . our work uses grounding to ensure that edits are factually correct, although our task differs from previous work because of the user command, which requires specific information to be retrieved from the grounding during generation. controllable generation while grounding can be seen as a way to implicitly control the contents of generated text, other works have explored more explicit forms of control. @xcite and zhang et al. (2020) use lexical constraints, while keskar et al. (2019) and dathathri et al. ( 2019 ) control higher level attributes of text, such as style, tone, or topic. our task instead uses natural language commands, which can flexibly express different types of constraints, ranging from low-level lexical ones, to high-level topical ones. in this sense, we can also draw the parallel to dialog response generation @xcite , task-oriented dialog @xcite , or open domain question answering @xcite , that also involve user responses or queries, although these tasks are not concerned with text generation in the context of document creation. ## conclusion in this work we argued that text generation should be interactive, and, as a means towards that end, we proposed a general text editing task, where a system must edit a document in response to a user command. in our specific instance of the task we considered single-sentence edits, and we crawled a dataset of several million edits from wikipedia that included commands, in the form of editor comments, as well as grounding documents. we then showed that training a transformer-based model on our data, while initializing with pretrained language model weights, yields encouraging results on both automatic and human evaluations. additionally, our ablation studies showed the crucial role played by the user command and grounding. breaking down our results by types of edits, we saw that our model not only performs well on easier fluency edits, but also on much harder content edits. finally, we discussed future research directions for interactive document generation, as well as possible extensions to other domains such as images or code.