Non-autoregressive Text Editing with Copy-aware Latent Alignments (2023)

Abstract: Recent work has witnessed a paradigm shift from Seq2Seq to Seq2Edit in the field of text editing, with the aim of addressing the slow autoregressive inference problem posed by the former. Despite promising results, Seq2Edit approaches still face several challenges such as inflexibility in generation and difficulty in generalizing to other languages. In this work, we propose a novel non-autoregressive text editing method to circumvent the above issues, by modeling the edit process with latent CTC alignments. We make a crucial extension to CTC by introducing the copy operation into the edit space, thus enabling more efficient management of textual overlap in editing. We conduct extensive experiments on GEC and sentence fusion tasks, showing that our proposed method significantly outperforms existing Seq2Edit models and achieves similar or even better results than Seq2Seq with over 4\times speedup. Moreover, it demonstrates good generalizability on German and Russian. In-depth analyses reveal the strengths of our method in terms of the robustness under various scenarios and generating fluent and flexible outputs.

## introduction in natural language processing, monolingual text generation involves producing a target sequence from the source text with significant textual overlap @xcite . this includes a range of textediting tasks such as grammatical error correction (gec) @xcite and sentence fusion @xcite , as shown in generally, text editing can be addressed under the standard seq2seq framework @xcite . despite their decent performance, seq2seq has been criticized @xcite for its inferior inference speed due to the ˚work was done during the internship at tencent ai lab. yu zhang and yue zhang make equal contributions. ## preliminaries we begin by introducing some notations. the goal for text-editing is to transform the source sentence connectionist temporal classification was first introduced in auto speech recognition (asr) @xcite , aiming to circumvent the problems of no explicit alignments between asr inputs/outputs. specifically, ctc introduces a special blank token ∅ on top of the vocabulary v, and defines a latent alignment path a " a 0 , a 1 , . . . , a n between x and y with a i p v ť t∅u, which is of equal length as x. during training, ctc models the probability of the target sequence by marginalizing the probabilities of all latent alignments, in this way, ctc permits very efficient calculation of eq. 1 in opn ˆm q via forward algorithm. we refer interested readers to the original paper @xcite and tutorials by @xcite for more details. during inference, ctc defines a collapsing function γ ´1p¨q to recover the target sequence y from a by removing all blanks and consecutive repetitive tokens. for example, assuming a possible alignment path a " ta, a, ∅, a, b, bu, then γ ´1paq returns ta, a, bu. non-autoregressive text generation (nat) differs from its autoregressive counterpart in that it generates all target tokens simultaneously rather than one-by-one. nat often runs several times faster than autoregressive seq2seq models as it is highly parallelized. very recently, ctc has been introduced to non-autoregressive nmt @xcite @xcite . in asr, ctc assumes the input length n is larger that the output length m so that we can safely delete blanks from the alignment, resulting in a shorter output sequence. however, this is not the fact in text generation. to remedy this, @xcite propose to make use of an upsampling layer to amplify the input first and then run ctc as usual. this enables the model to learn target lengths very flexibly, which we believe is an important factor that empowers the ctc model. ## methodology in this section, we will introduce our proposed method that adapts the vanilla ctc to text editing tasks. the main idea is to endow ctc with the ability of modeling the edit processes by extending the latent ctc alignments with interpretable edit operations, especially the copy operation. ## experiments following felix and edit5 @xcite , we evaluate our model by conducting experiments on two text editing tasks: grammatical error correction (gec) and sentence fusion, both of which are representative and have sufficient data for training. we plan to conduct examinations on more tasks in future work due to space limitations. ## analysis we demonstrate the superiority of our proposed ctc model by making comparisons from two perspectives: 1) with vanilla ctc; 2) with other textediting systems. ## related works efficient text editing for years, seq2seq models have been hitherto the best approaches for text editing @xcite due to their effectiveness. some recent works obtain notable gains by injecting edit information into the generation process, making it more controllable and interpretable @xcite . syngec @xcite integrates edit templates into syntax trees, which are then encoded with gnns in or-der to provide hints for the decoder. there is also a broad strand of works that combine the autoregressive fashion with edit operations @xcite @xcite . despite promising results, the slow inference speed limits their applications in real-life online systems. to combat this, @xcite make use of very shallow decoders as well as aggressive copy strategy to speed up the decoding. @xcite suggest to decoding spans needed to edit only for acceleration. @xcite present a novel dynamic programming algorithm to allow for span copy during generation. however, this does not break the inherent flaws. in this work, we instead propose a purely non-autoregressive approach for text editing, demonstrating great efficiency benefits over autoregressive counterparts. ## conclusion in this work, we propose a novel ctc-based method for non-autoregressive text editing. the key idea is to subsume the editing process with latent ctc alignments. we further make a crucial extension by introducing the copy operation, enabling very effective edits with three main operations only: add, keep and delete. we conduct experiments on gec and sentence fusion, showing that our method can reach or surpass the current state-of-the-art works while being 4ˆfaster, and generalize well across german and russian. moreover, compared with the best performing seq2edit model, our method also exhibits very good generation flexibility, which could shed light on further studies on non-autoregressive text editing.