Don’t Take It Literally: An Edit-Invariant Sequence Loss for Text Generation (2022)

Abstract: Neural text generation models are typically trained by maximizing log-likelihood with the sequence cross entropy (CE) loss, which encourages an exact token-by-token match between a target sequence with a generated sequence. Such training objective is sub-optimal when the target sequence is not perfect, e.g., when the target sequence is corrupted with noises, or when only weak sequence supervision is available. To address the challenge, we propose a novel Edit-Invariant Sequence Loss (EISL), which computes the matching loss of a target n -gram with all n -grams in the generated sequence. EISL is designed to be robust to various noises and edits in the target sequences. Moreover, the EISL computation is essentially an approximate convolution operation with target n -grams as kernels, which is easy to implement and efficient to compute with existing libraries. To demonstrate the effectiveness of EISL, we conduct experiments on a wide range of tasks, including machine translation with noisy target sequences, unsupervised text style transfer with only weak training signals, and non-autoregressive generation with non-predefined generation order. Experimental results show our method significantly outperforms the common CE loss and other strong baselines on all the tasks. EISL has a simple API that can be used as a drop-in replacement of the CE loss: https://github.com/guangyliu/EISL .

## introduction neural text generation models have ubiquitous applications in natural language processing, including machine translation @xcite @xcite , summarizations @xcite , dialogue systems @xcite , etc. they are typically trained by maximizing the log-likelihood of the output sequence conditioning on the inputs with the cross entropy (ce) loss. the the problem is even exaggerated when the supervision from a target sequence is not perfect @xcite . on one hand, there could be noises in the reference sequence which makes itself not a valid sentence. as in the last example shown in figure 1 , there is a repetition error in the target sequence, which is common in human generated text. with 2055 we use a fixed machine translation model, synthesize different types of edits on target text, and measure the ce and eisl losses, respectively. the edit types include shuffle (changing the word order), repetition (words being selected are repeated), and word blank (words being replaced with a blank token). ce loss tends to increase drastically once a small amount of edits is applied. in contrast, eisl loss increases much more slowly, showing its robustness. the ce loss, the model is forced to copy all tokens including the error, and assign a high loss for the grammatically correct sequence. the exact tokens matching renders the ce loss sensitive to noises in the target, as shown in figure 2 . on the other hand, there are many problems with only weak supervision for target sequences @xcite @xcite . for example, in tasks of unsupervised text style transfer @xcite aiming to rewrite a sentence from one style to another, the original sentence offers weak supervision for the content (rather than the style). yet using a ce loss here is problematic since it encourages the model to copy every original token. prior works have tried to address this problem using reinforcement learning (rl) @xcite @xcite . for example, policy gradient was used to optimize sequence rewards such as bleu metric @xcite . such algorithms assign high rewards to sentences that are close to the target sentence. though it is a valid objective to optimize, policy optimization faces significant challenges in practice. the high variance of gradient estimate makes the training extremely difficult, and almost all previous attempts rely on fine-tuning from models trained with ce loss, often with unclear improvement @xcite . in this paper, we propose an alternative loss to overcome the above weakness of ce loss, but reserve all nice properties such as being end-to-end differentiable, easy to implement, and efficient to compute, which hence can be used as a drop-in replacement or combined with ce. the loss is based on the observation that a viable candidate sequence shares many sub-sequences with the target. our loss, called edit-invariant sequence loss (eisl), models the matching of each reference n-gram across all n-grams in a candidate sequence. the design is motivated by the translation invariance properties of convnets on images (see figure 3 ), and captures the edit invariance properties of text n-grams in calculating the loss. figure 2 shows the invariance property of eisl in comparison with ce. appealingly, we show the conventional ce loss is a special case of eisl-when n equals to the sequence length, eisl calculates the exact sequence matching loss and reduces to ce. moreover, the computations of eisl is essentially a convolution operation of candidate sequence using target n-grams as kernels, which is very easy to implement with existing deep learning libraries. to demonstrate the effectiveness of eisl loss, we conduct experiments on three representative tasks: machine translation with noisy training target, unsupervised text style transfer (only weak references are available), and non-autoregressive generation with flexible generation order. experiments demonstrate eisl loss can be easily incorporated with a series of sequence models and outperforms ce and other popular baselines across the board. ## related work deep neural sequence models such as recurrent neural networks @xcite and transformers @xcite have achieved great progress in many text generation tasks like machine translation @xcite . these models are typically trained with the maximum-likelihood objective, which can lead to sub-optimal performance due to ce's exact sequence matching assumption. there are lots of works trying to overcome this weakness. for examples, some works @xcite @xcite @xcite proposed to use policy gradient or minimum risk training to optimize the expected bleu metric @xcite . due to the high variance and unstableness in rl training, a variety of training tricks are used in practice. @xcite developed a new reward function based on semantic similarity for translation. @xcite introduced soft q-learning for more efficient rl training. on the other hand, zhukov and kretov (2017), @xcite made the initial attempts to develop differentiable bleu objectives, making soft approximations to the count of n-gram matching in the original bleu formulation. @xcite @xcite ) minimized the n-gram difference between the model outputs and targets in nonautoregressive generation. another line of research that is relevant to our work is learning with noisy labels in classification @xcite @xcite . for text generation, @xcite proposed student forcing to substitute teacher forcing, which can alleviate the influence of noise in the target sequence during decoding. @xcite proposed loss truncation, which adaptively removes high-loss examples considered as invalid data. our empirical study shows substantial improvement of our approach over the previous ones. ## edit-invariant sequence loss in this section, we first review the conventional cross-entropy (ce) loss for sequence learning, and point out its weakness, especially when the target sequence is edited. we then introduce the eisl loss which gives a model the flexibility to learn from sub-sequences in a target sequence. we first establish notations for the sequence generation setting. let (x, y * ) be a paired data sample where x is the input and y * = (y * 1 , ..., y * t * ) is the reference target sequence. define @xmath0, ..., y t ) as a candidate sentence. our goal is to build a model p θ (y|x) that scores a candidate sequence y with parameter θ. in the sequel, we omit the condition x and the subscript θ for simplicity. ## experiments in this section, we present the experimental results on three text generation settings to test eisl's effectiveness, including learning from noisy text, learning from weak sequence supervision, and nonautoregressive generation models that require flexibility in generation orders. more details of the experimental setting are provided in appendix a.2. ## conclusions we have developed edit-invariant sequence loss (eisl) for end-to-end training of neural text generation models. the proposed method is insensitive to the shift of n-grams in target sequences, hence suitable for training with noisy data and weak supervisions, where ce loss fails easily. we show ce loss is a special case of eisl and build the connection of eisl with bleu metric and convolution operation, which both have the invariant property. experiments on translation with noisy target, text style transfer, and non-autoregressive neural machine translation demonstrate the superiority of our method. the more general applications and superiority of eisl on other diverse text generation problems as well as fundamental challenges, such as compositional generalization @xcite and causal invariance @xcite in language, remain to be explored further, which we are excited to study in the future. we also perform human evaluations on yelp data to further test the transfer quality. we first randomly select 100 sentences from the test set, use these sentences as input and generate sentences from the base model @xcite and our model. then for each original sentence, we present the outputs of the base model and ours in random order. the three annotators are asked to evaluate which sentence is preferred as the transferred sentence of the original sentence, in terms of content preservation and sentiment transfer. they can choose either output or select the same quality. we measure the percentage of times each model outperforms the other.