FELIX : Flexible Text Editing Through Tagging and Insertion (2020)

Abstract: We present FELIX – a flexible text-editing approach for generation, designed to derive maximum benefit from the ideas of decoding with bi-directional contexts and self-supervised pretraining. In contrast to conventional sequenceto-sequence (seq2seq) models, FELIX is efficient in low-resource settings and fast at inference time, while being capable of modeling flexible input-output transformations. We achieve this by decomposing the text-editing task into two sub-tasks: tagging to decide on the subset of input tokens and their order in the output text and insertion to in-fill the missing tokens in the output not present in the input. The tagging model employs a novel Pointer mechanism, while the insertion model is based on a Masked Language Model (MLM). Both of these models are chosen to be non-autoregressive to guarantee faster inference. FELIX performs favourably when compared to recent text-editing methods and strong seq2seq baselines when evaluated on four NLG tasks: Sentence Fusion, Machine Translation Automatic Post-Editing, Summarization, and Text Simplification

## introduction the ideas of text in-filling coupled with selfsupervised pre-training of deep transformer networks on large text corpora have dramatically changed the landscape in natural language understanding. bert @xcite and its successive refinements roberta @xcite , albert @xcite implement this recipe and have significantly pushed the state-of-the-art on multiple nlu benchmarks such as glue @xcite and squad @xcite . more recently, masked or in-filling style objectives for model pretraining have been applied to seq2seq tasks, significantly pushing the state-of-the-art ## model description felix decomposes the conditional probability of generating an output sequence y from an input x as follows: ## experiments we evaluate felix on four distinct text editing tasks: sentence fusion, text simplification, summarization, and automatic post-editing for machine translation. in addition to reporting previously published results for each task foot_7 , we also compare to a recent text-editing approach lasertag-ger @xcite , which combines editing operations with a fixed vocabulary of additional phrases which can be inserted. we follow their setup and set the phrase vocabulary size to 500 and run all experiments using their most accurate autoregressive model. to decode a batch of 32 on a nvidia tesla p100, lasertagger takes 1,300ms, felix takes 300ms and a a similarly sized seq2seq model takes 27,000ms @xcite . for all tasks we run an ablation study, examining the effect of an open vocabulary with no reordering (felixinsert), and a fixed vocabulary foot_8 with reordering model (felixpoint). task analysis. the chosen tasks cover a diverse set of edit operations and a wide range of dataset sizes. ## related work seq2seq models @xcite have been applied to many text generation tasks that can be cast as monolingual translation, but they suffer from well-known drawbacks @xcite : they require large amounts of training data, and their outputs are difficult to control. whenever input and output sequences have a large overlap, it is reasonable to cast the problem as a text editing task, rather than full-fledged sequence-to-sequence generation. @xcite argued that the general problem of string transduction can be re-duced to sequence labeling. their approach applied only to character deletion and insertion and was based on simple patterns. lasertagger @xcite is a general approach that has been shown to perform well on a number of text editing tasks, but it has two limitations: it does not allow for arbitrary reordering of the input tokens; and insertions are restricted to a fixed phrase vocabulary that is derived from the training data. similarly, ed-itnts @xcite and pie @xcite are two other text-editing models developed specifically for simplification and grammatical error correction, respectively. pointer networks have been previously proposed as a way to copy parts of the input in hybrid seq2seq models. @xcite and @xcite trained a pointer network to specifically deal with out-of-vocabulary words or named entities. @xcite proposed a summarization model that first selects salient sentences and then rewrites them abstractively, using a pointer mechanism to directly copy some out-of-vocabulary words. previous approaches have proposed alternatives to autoregressive decoding @xcite @xcite . instead of the left-to-right autoregressive decoding, insertion transformer @xcite and blm @xcite generate the output sequence through insertion operations, whereas levt @xcite additionally incorporates a deletion operation. these methods produce the output iteratively, while felix requires only two steps: tagging and insertion. the differences between the proposed model, felix, its ablated variants, and a selection of related works is summarized in ## conclusions and future work we have introduced felix, a novel approach to text editing, by decomposing the task into tagging and insertion which are trained independently. such separation allows us to take maximal benefit from the already existing pretrained masked-lm models. felix works extremely well in low-resource settings and it is fully non-autoregressive which favors faster inference. our empirical results demonstrate that it delivers highly competitive performance when compared to strong seq2seq baselines and other recent text editing approaches. in the future we plan to investigate the following ideas: (i) how to effectively share representations between the tagging and insertion models using a single shared encoder, (ii) how to perform joint training of insertion and tagging models instead of training them separately, (iii) strategies for unsupervised pre-training of the tagging model. which appears to be the bottleneck in highly low-resource settings, and (iv) distillations recipes. xingxing zhang and mirella lapata. 2017b. sentence simplification with deep reinforcement learning. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 584-594. association for computational linguistics. sanqiang zhao, rui meng, daqing he, andi saptono, and bambang parmanto. 2018. integrating transformer and paraphrase rules for sentence simplification. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 3164-3173. association for computational linguistics. zhemin zhu, delphine bernhard, and iryna gurevych. 2010. a monolingual tree-based translation model for sentence simplification. in proceedings of the 23rd international conference on computational linguistics (coling 2010), pages 1353-1361. association for computational linguistics, coling 2010 organizing committee.