Text Generation with Text-Editing Models (2022)

Abstract: Text-editing models have recently become a prominent alternative to seq2seq models for monolingual text-generation tasks such as grammatical error correction, text simplification, and style transfer. These tasks share a common trait – they exhibit a large amount of textual overlap between the source and target texts. Text-editing models take advantage of this observation and learn to generate the output by predicting edit operations applied to the source sequence. In contrast, seq2seq models generate outputs word-by-word from scratch thus making them slow at inference time. Text-editing models provide several benefits over seq2seq models including faster inference speed, higher sample efficiency, and better control and interpretability of the outputs. This tutorial provides a comprehensive overview of the text-edit based models and current state-of-the-art approaches analyzing their pros and cons. We discuss challenges related to deployment and how these models help to mitigate hallucination and bias, both pressing challenges in the field of text generation.

## introduction after revolutionizing the field of machine translation @xcite @xcite , sequence-to-sequence (seq2seq) methods have quickly become the standard approach for not only multilingual but also for monolingual sequence transduction / text generation tasks, such as text summarization, style transfer, and grammatical error correction. while delivering significant quality gains, these models, however, are prone to hallucinations @xcite . the seq2seq task setup (where targets are generated from scratch word by word) overlooks the fact that in many monolingual tasks the source and target sequences have a text-editing models attempt to address some of the limitations of seq2seq approaches and there has been recently a surge of interest in applying them to a variety of monolingual tasks including text simplification @xcite @xcite , grammatical error correction @xcite @xcite @xcite @xcite , sentence fusion @xcite (see an example in figure 1 ), mt automatic post-editing @xcite @xcite ), text style transfer @xcite , data-to-text generation @xcite , and utterance rewriting @xcite @xcite . text-editing approaches claim to be more accurate or on-par with seq2seq baselines especially in low resource settings, less prone to hallucinations and faster at inference time. these advantages have generated a substantial and continued level of interest in text-editing research. the goal of this tutorial is to provide the first comprehensive overview of the family of text-editing approaches and to offer practical guidelines for applying them to a variety of text-generation tasks. ## tutorial outline the structure of the tutorial with duration estimates for different sections are shown in introduction. we first define the family of textediting methods: text-editing models are sequencetransduction methods that produce the output text by predicting edit operations which are applied to the inputs. in contrast, the traditional seq2seq methods produce the output from scratch, token by token. we summarize the main pros and cons of these two approaches and provide guidelines for choosing which approach is more suitable for a given task. model design. the similarities and differences of a set of popular text-editing methods will be analyzed in terms of the types of edit operations they employ, their tagging architecture, and whether they are auto-regressive or feedforward. we also discuss methods for converting target texts into target edit sequences, a task which often does not have a unique solution.