S eq2 E dits: Sequence Transduction Using Span-level Edit Operations (2020)

Abstract: We propose Seq2Edits, an open-vocabulary approach to sequence editing for natural language processing (NLP) tasks with a high degree of overlap between input and output texts. In this approach, each sequence-to-sequence transduction is represented as a sequence of edit operations, where each operation either replaces an entire source span with target tokens or keeps it unchanged. We evaluate our method on five NLP tasks (text normalization, sentence fusion, sentence splitting & rephrasing, text simplification, and grammatical error correction) and report competitive results across the board. For grammatical error correction, our method speeds up inference by up to 5.2x compared to full sequence models because inference time depends on the number of edits rather than the number of target tokens. For text normalization, sentence fusion, and grammatical error correction, our approach improves explainability by associating each edit operation with a human-readable tag.

## introduction neural models that generate a target sequence conditioned on a source sequence were initially proposed for machine translation (mt) @xcite @xcite , but are now used widely as a central component of a variety of nlp systems (e.g. @xcite ; @xcite ). @xcite argue that even problems that are traditionally not viewed from a sequence transduction perspective can benefit from massive pre-training when framed as a text-to-text problem. however, for many nlp tasks such as correcting grammatical errors in a sentence, the input and output sequence may overlap significantly. employing a full sequence model in these cases is often wasteful as most tokens are simply copied over from the input to the output. another disadvantage of a full sequence model is that it does not provide an explanation for why it proposes a particular target sequence. in this work, inspired by a recent increased interest in text-editing @xcite @xcite , we propose seq2edits, a sequence editing model which is tailored towards problems that require only small changes to the input. rather than generating the target sentence as a series of tokens, our model predicts a sequence of edit operations that, when applied to the source sentence, yields the target sentence. each edit operates on a span in the source sentence and either copies, deletes, or replaces it with one or more target tokens. edits are generated auto-regressively from left to right using a modified transformer @xcite architecture to facilitate learning of long-range dependencies. we apply our edit operation based model to five nlp tasks: text normalization, sentence fusion, sentence splitting & rephrasing, text simplification, and grammatical error correction (gec). our model is competitive across all of these tasks, and improves the state-of-the-art on text normalization @xcite , sentence splitting & rephrasing @xcite , and the jfleg test set @xcite for gec. our model is often much faster than a full sequence model for these tasks because its runtime depends on the number of edits rather than the target sentence length. for instance, we report speedups @xmath0. if applicable, we also predict a taskspecific edit-type class ("tag") along with each edit which explains why that edit was proposed. for example in gec, the correction of a misspelled word would be labelled with a spell (spelling error) whereas changing a word from say first person to third person would be associated with a tag such as sva (subject-verb agreement error). 2 edit-based sequence transduction ## experiments we evaluate our edit model on five nlp tasks: 3 our models are trained on packed examples (maximum length: 256) with adafactor @xcite using the tensor2tensor @xcite library. we report results both with and without pre-training. our pre-trained models for all tasks are trained for 1m iterations on 170m sentences extracted from english wikipedia revisions and 176m sentences from english wikipedia round-trip translated via german @xcite . for grammatical error correction we use errant @xcite to derive span-level edits from the parallel text. on other tasks we use a minimum edit distance heuristic to find a token-level edit sequence and convert it to span-level edits by merging neighboring edits. the task-specific data is described in tuning refers to optimizing the decoding parameters listed in we next situate our best results (big models with pre-training in rows e and f of ## related work a popular way to tackle nlp problems with overlap between input and output is to equip seq2seq models with a copying mechanism @xcite @xcite @xcite @xcite , usually borrowing ideas from pointer networks @xcite to point to single tokens in the source sequence. in contrast, we use pointer networks to identify entire spans that are to be copied which results in a much more compact representation and faster decoding. the idea of using span-level edits has also been explored for morphological learning in the form of symbolic span-level rules, but not in combination with neural models @xcite . our work is related to neural multi-task learning for nlp @xcite @xcite . unlike multi-task learning which typically solves separate problems (e.g. pos tagging and named entity recognition @xcite or translation into different languages @xcite ) with the same model, our three output features (tag, source span, and replacement) represent the same output sequence (algorithm 1). thus, it resembles the stack-propagation approach of @xcite who use pos tags to improve parsing performance. a more recent line of research frames sequence editing as a labelling problem using labels such as add, keep, and delete @xcite @xcite @xcite , often relying heavily on bert @xcite pre-training. similar operations such as insertions and deletions have also been used for machine translation @xcite @xcite @xcite . we showed in sec. 3 that our model often performs similarly or better than those approaches, with the added advantage of providing explanations for its predictions. ## discussion we have presented a neural model that represents sequence transduction using span-based edit operations. we reported competitive results on five different nlp problems, improving the state of the art on text normalization, sentence splitting, and the jfleg test set for grammatical error correction. we showed that our approach is 2.0-5.2 times faster than a full sequence model for grammatical error correction. our model can predict labels that explain each edit to improve the interpretability for the end-user. however, we do not make any claim that seq2edits can provide insights into the internal mechanics of the neural model. the underlying neural model in seq2edits is as much of a black-box as a regular full sequence model. while our model is advantageous in terms of speed and explainability, it does have some weaknesses. notably, the model uses a tailored architecture (figure 2 ) that would require some engineering effort to implement efficiently. second, the output of the model tends to be less fluent than a regular full sequence model, as can be seen from the examples in even though our approach is open-vocabulary, future work will explore task specific restrictions. for example, in a model for dialog applications, we may want to restrict the set of response tokens to a predefined list. alternatively, it may be useful to explore generation in a non left-to-right order to improve the efficiency of inference. another line of future work is to extend our model to sequence rewriting tasks, such as machine translation post-editing, that do not have existing error-tag dictionaries. this research would require induction of error tag inventories using either linguistic insights or unsupervised methods.