Don’t Take It Literally: An Edit-Invariant Sequence Loss for Text Generation
Abstract
Neural text generation models are typically trained by maximizing log-likelihood with the sequence cross entropy (CE) loss, which encourages an exact token-by-token match between a target sequence with a generated sequence.
Such training objective is sub-optimal when the target sequence is not perfect, e.g., when the target sequence is corrupted with noises, or when only weak sequence supervision is available.
To address the challenge, we propose a novel Edit-Invariant Sequence Loss (EISL), which computes the matching loss of a target n -gram with all n -grams in the generated sequence.
EISL is designed to be robust to various noises and edits in the target sequences.
Moreover, the EISL computation is essentially an approximate convolution operation with target n -grams as kernels, which is easy to implement and efficient to compute with existing libraries.
To demonstrate the effectiveness of EISL, we conduct experiments on a wide range of tasks, including machine translation with noisy target sequences, unsupervised text style transfer with only weak training signals, and non-autoregressive generation with non-predefined generation order.
Experimental results show our method significantly outperforms the common CE loss and other strong baselines on all the tasks.
EISL has a simple API that can be used as a drop-in replacement of the CE loss.
Introduction
Neural text generation models have ubiquitous applications in natural language processing, including machine translation (Bahdanau et al., 2015,
Sutskever et al., 2014, Wu et al., 2016, Vaswani et al., 2017), summarizations (Nallapati et al., 2016,
See et al., 2017), dialogue systems (Li et al., 2016), etc.
They are typically trained by maximizing the log-likelihood of the output sequence conditioning on the inputs with the cross entropy (CE) loss.
The a cat is on the red blanket on the red blanket there is a cat
Paraphrase: a cat is on the red blanket
Noisy Target: a cat is is on the red blanket
Image:
CE loss can be easily factorized into individual loss terms and can be optimized efficiently with stochastic gradient descent.
Due to its computational efficiency and ease to implement, the training paradigm has played an important role in building successful large text generation models (Lewis et al., 2020, Radford et al., 2019).
However, the
CE loss minimizes the negative log-likelihood of only the reference output sequence, while all other sequences are equally penalized through normalization.
This is over-restrictive since for a given reference target sentence, many possible paraphrases are semantically close, hence should not completely be treated as negative samples.
For example, as shown in Figure 1, a cat is on the red blanket should be treated equally with on the red blanket there is a cat .
A model trained with CE loss falls short of modeling such type of invariance for text.
The problem is even exaggerated when the supervision from a target sequence is not perfect (Pinnis, 2018).
On one hand, there could be noises in the reference sequence which makes itself not a valid sentence.
As in the last example shown in Figure 1, there is a repetition error in the target sequence, which is common in human generated text.
With
Loss
Amount of edits: shuffle
CE EISL
Loss
Amount of edits: repetition
CE EISL
Loss
Amount of edits: blank
CE EISL the CE loss, the model is forced to copy all tokens including the error, and assign a high loss for the grammatically correct sequence.
The exact tokens matching renders the CE loss sensitive to noises in the target, as shown in Figure 2.
On the other hand, there are many problems with only weak supervision for target sequences (Tan et al., 2020, Wang et al., 2021, Lin et al., 2020).
For example, in tasks of unsupervised text style transfer (Jin et al., 2022) aiming to rewrite a sentence from one style to another, the original sentence offers weak supervision for the content (rather than the style).
Yet using a
CE loss here is problematic since it encourages the model to copy every original token.
Prior works have tried to address this problem using reinforcement learning (RL) (Guo et al., 2021,
O’Neill and Bollegala, 2019, Wieting et al., 2019).
For example, policy gradient was used to optimize sequence rewards such as BLEU metric (Ranzato et al., 2016, Liu et al., 2017).
Such algorithms assign high rewards to sentences that are close to the target sentence.
Though it is a valid objective to optimize, policy optimization faces significant challenges in practice.
The high variance of gradient estimate makes the training extremely difficult, and almost all previous attempts rely on fine-tuning from models trained with CE loss, often with unclear improvement (Wu et al., 2018).
In this paper, we propose an alternative loss to overcome the above weakness of CE loss, but reserve all nice properties such as being end-to-end differentiable, easy to implement, and efficient to compute, which hence can be used as a drop-in replacement or combined with CE.
The loss is based on the observation that a viable candidate sequence shares many sub-sequences with the target.
Our loss, called edit-invariant sequence loss (EISL), models the matching of each reference n -gram across all n -grams in a candidate sequence.
The design is motivated by the translation invariance properties of ConvNets on images (see Figure 3), and captures the edit invariance properties of text n -grams in calculating the loss.
Figure 2 shows the invariance property of EISL in comparison with
CE.
Appealingly, we show the conventional CE loss is a special case of EISL—when n equals to the sequence length, EISL calculates the exact sequence matching loss and reduces to CE.
Moreover, the computations of EISL is essentially a convolution operation of candidate sequence using target n -grams as kernels, which is very easy to implement with existing deep learning libraries.
To demonstrate the effectiveness of EISL loss, we conduct experiments on three representative tasks: machine translation with noisy training target, unsupervised text style transfer (only weak references are available), and non-autoregressive generation with flexible generation order .
Experiments demonstrate EISL loss can be easily incorporated with a series of sequence models and outperforms
CE and other popular baselines across the board.
Related Work
Deep neural sequence models such as recurrent neural networks (Sutskever et al., 2014, Mikolov et al., 2010) and transformers (Vaswani et al., 2017) have achieved great progress in many text generation tasks like machine translation (Bahdanau et al., 2015, Vaswani et al., 2017).
These models are typically trained with the maximum-likelihood objective, which can lead to sub-optimal performance due to CE’s exact sequence matching assumption.
There are lots of works trying to overcome this weakness.
For examples, some works (Ranzato et al., 2016, Rennie et al., 2017, Liu et al., 2017,
Shen et al., 2016, Smith and Eisner, 2006) proposed to use policy gradient or minimum risk training to optimize the expected BLEU metric (Papineni et al., 2002a).
Due to the high variance and unstableness in RL training, a variety of training tricks are used in practice.
Wieting et al.
(2019) developed a new reward function based on semantic similarity for translation.
Guo et al.
(2021) introduced soft Q-learning for more efficient RL training.
On the other hand, Zhukov and Kretov (2017),
Casas et al.
(2018) made the initial attempts to develop differentiable BLEU objectives, making soft approximations to the count of n -gram matching in the original BLEU formulation.
Shao et al.
(2018, 2021, 2020) minimized the n -gram difference between the model outputs and targets in nonautoregressive generation.
Another line of research that is relevant to our work is learning with noisy labels in classification (Zhang and Sabuncu, 2018, Xu et al., 2019, Wang et al., 2019b, Hu et al., 2019).
For text generation,
Nicolai and Silfverberg (2020) proposed student forcing to substitute teacher forcing, which can alleviate the influence of noise in the target sequence during decoding.
Kang and Hashimoto (2020) proposed loss truncation, which adaptively removes high-loss examples considered as invalid data.
Our empirical study shows substantial improvement of our approach over the previous ones.
Edit-Invariant Sequence Loss
In this section, we first review the conventional cross-entropy (CE) loss for sequence learning, and point out its weakness, especially when the target sequence is edited.
We then introduce the EISL loss which gives a model the flexibility to learn from sub-sequences in a target sequence.
We first establish notations for the sequence generation setting.
Let ( x , y ∗ ) be a paired data sample where x is the input and y ∗ =
( y ∗ 1 , ..., y ∗ T ∗ ) is the reference target sequence.
Define y =
( y 1 , ..., y T ) as a candidate sentence.
Our goal is to build a model p θ ( y | x ) that scores a candidate sequence y with parameter θ .
In the sequel, we omit the condition x and the subscript θ for simplicity.
3. The Difficulty of Cross Entropy Loss
The standard approach to learn the sequence model is to minimize the negative log-likelihood (NLL) of the target sequence, i.e., minimizing the CE loss L CE ( θ )
= − log p ( y ∗ ) .
The CE loss assumes exact matching of a candidate sequence y with the target sequence y ∗ .
In other words, it maximizes the probability of only the target sequence y ∗ while penalizing all other possible sequence outputs that might be close but different with y ∗ .
The assumption can be problematic in many practical scenarios: (1) For a given target sentence, there could be many ways of paraphrasing the sentence such as word reordering, synonyms replacement, active to passive rewriting, etc.
Many of the paraphrases are viable candidate sequences, and/or share many sub-sequences with the reference sentence, and thus should not be treated completely as negative samples.
Similar to the translation invariance which is shown to be effective in image modeling, a sequence loss that is robust to the shift and edits of sub-sequences in the reference sequence is preferred in order to model the rich variations of sequences; (2) The edit-invariance property is particularly desirable when the reference target sequence is corrupted with noise or is only weak sequence supervision.
For instance, in Figure 3, the word is is repeated twice, which is one of the common errors in typing.
Using CE loss in the noisy target setting forces the model to learn the data errors as well.
In contrast, a sequence loss robust or invariant to the shift of sub-sequences assigns a high probability to the correct sentence even though it does not match the noisy target exactly.
The loss thus offers flexibility for the model to select right information for learning.
3. EISL: Edit-Invariant Sequence Loss
Motivated by the above discussion, in this section, we draw inspirations from the convolution operation that enables translation invariance in image modeling (Figure 3, left), and propose an editinvariant sequence loss (EISL) as illustrated in Figure 3 (right).
Intuitively, for instance, given a 4gram on the red blanket , because there is no extra knowledge to determine the position of the 4-gram in the noisy target sequence, we compute the losses across all positions in the noisy target sequence and aggregate.
This is essentially a convolution over the target noisy sequence with the given n -gram as a convolution kernel.
We now derive the EISL loss in more details.
Let y a : b =
( y a , ..., y b − 1 ) denote a sub-sequence of y that starts from index a and ends at index b − 1 , which is of length b − a .
Thus y ∗ i : i + n denotes the ith n -gram in the reference y ∗ .
Denote C ( y ∗ i : i + n , y ) as the number of times this n -gram occurs in y :
[Equation]
where 1 ( · ) is the indicator function that takes value
Desired output: a cat is on the red blanket
Noisy target: a cat is is on the red blanket a cat is is on the red blanket a cat is is on the red blanket a cat is is on the red blanket a cat is is on the red blanket
Image:
1 if the n -grams match, and 0 otherwise.
Intuitively, for a text generation model, we would like to maximize the occurrence of an n -gram from the reference in the target sequence.
For a given probabilistic model p θ ( y ) (we omit the parameter θ wherever the meaning is clear), the expected value of C ( y ∗ i : i + n , y ) can be computed as follow:
[Equation]
Thus, for each i -th n -gram in the reference, a straightforward way to define the learning objective is to minimize the negative log value of its expected occurrence, i.e., − log E y ∼ p
( y )
[ C ( y ∗ i : i + n , y )] .
The above loss requires computation of the marginal probability p
( y i ′ :
i ′ + n = y ∗ i : i + n ) of an ngram, which is intractable in practice.
We therefore derive an upper bound of the loss and use it as the surrogate to minimize in training.
We denote the upper bound surrogate as our EISL loss.
Specifically, since for a given i ′ , p ( y i ′ :
i ′ + n = y ∗ i : i + n )
=
(cid:80) y p ( y <i ′ )
p ( y i ′ :
i ′ + n = y ∗ i : i + n | y <
i ′ )
, then:
[Equation]
The detailed derivation is attached in Appendix A.1.
Notice that the EISL loss involves only the conditional distribution p ( y i ′ :
i ′ + n = y ∗ i : i + n | y <i ′ ) which is convenient to compute—we first sample tokens from the model up to the i ′ position, then compute
NLL of the reference n -gram y ∗ i : i + n occurring at position i ′ under the model distribution.
The full ngram EISL loss is then defined by averaging across all n -gram positions in the reference:
[Equation]
In practice, inspired by the standard BLEU metric (more in section 3.3), we could also straightforwardly combine different n -gram losses depending on tasks:
[Equation]
where w n is the weight of the n -gram loss.
The rule of thumb is that a n -gram EISL loss with lower n is more robust to noises, as shown in our experiments.
Following BLEU, we found that simply using equal weights for different n -grams up to n = 4 often produces good performance.
As discussed shortly, it is appealing that the ngram EISL loss is indeed a direct generalization of the CE loss on the n -gram level: we sum the
CE loss of an n -gram over all candidate sequence positions by conditioning on samples from the model.
Besides, the derivation of the upper bound makes no assumption on the probability function p ( y ) , hence holds for both autogressive and nonautoregressive sequence models as demonstrated in our experiments.
Position Selection Minimizing the gram matching loss over all positions can make the model assign equal probabilities at all positions, which causes the training to collapse.
We further adapt the loss to enable the model to automatically learn the positions of reference n -grams.
For notation simplicity, let g ni,i ′
denote the conditional probability p
( y i ′ :
i ′ + n = y ∗ i : i + n | y <
i ′ ) involved above (Eq.3).
We can vectorize the probability to get g ni =
[ g ni, 1 , ..., g ni,T − n +1 ] T , spanning all potential positions in the candidate sequence.
We then normalize the probability vector g ni by Gumbel
∗
[Figure]
log𝑝(𝒚 2:5 =[ is,on,the ]|𝑦 1 ) =
log𝑝(𝒚 2:5 = 𝒚 3:6 ∗ |𝑦 1 )
[Figure]
on the
Kernel (Ground Truth, 𝒚 ∗ ) softmax (Jang et al., 2017), denoted as q ni =
Gumbel_softmax ( g ni ) , which we use as the weight for every n -gram positions.
We multiply the weight with the original log probability to get the new adjusted loss:
[Equation]
The loss can roughly be viewed as the “entropy” of the unnormalized probabilities g ni , which has minimal value if the mass of the probability is assigned to one location only.
Intuitively, if an g ni,i ′ is large, then it is likely i ′ is the correct position for the reference n -gram, hence the weight for this position should also be large.
This is like the greedy exploitation in reinforcement learning (Mnih et al., 2015).
On the other hand, to overcome over-exploitation, the Gumbel softmax introduces randomness in the weight assignment, which helps balance the exploitation-exploration trade-off in position selection for the model.
Efficient Approximate Computation:
EISL as Convolution We show the EISL loss can be computed efficiently using the common convolution operator, with very little additional cost compared with the CE loss.
The computation involves moderate approximation if the generation model is an autoregressive model, and is exact in the case of a non-autoregressive model (e.g., as in section 4.3).
We first discuss the easy case when the model is a non-autoregressive model, where we have g ni,i ′ =
p ( y i ′ :
i ′ + n = y ∗ i : i + n | y <i ′ )
= (cid:81) n j =
[Equation]
p ( y i ′
+ j − 1 =
y ∗ i + j − 1 ) .
Denote V as the vocabulary size.
Let P =
[ p 1 , p 2 , ...
p T ] be the probability output by the model across positions, where p i ′
∈
R V is the probability output after softmax at i ′ -th position, and each p i ′ is independent with each other.
On this basis, we compute the key quantity log g ni in Eq.
6 as the direct output of the convolution operator.
As shown in Figure 4, we can get log g ni by applying convolution on log P , with y
i : i + n as the kernels:
[Equation]
where Onehot ( · ) maps each token to its corresponding one-hot representation and Conv ( · , · ) is the convolution operation with the first argument as input and the second as the kernel.
We transform P into log domain to turn the probability multiplication into log probability summations, where Conv can be directly applied.
As shown in Figure 4, log P is of shape V × T and Onehot ( y ∗ i : i + n ) is of shape V × n , so Conv (log P , Onehot ( y ∗ i : i + n )) is an one-dimensional convolution on the sequence axis.
Formally, the i ′ -th convolutional output is:
[Equation]
After obtaining g ni by convolution, the EISL loss in Eq.
6 can be easily calculated.
We now discuss the case of autoregressive model, where by definition we have g ni,i ′ =
(cid:81) nj =1 p ( y i ′
+ j − 1 =
y ∗ i + j −
| y <i ′ , y ∗ i : i + j − 1 ) .
The dependence on both y <i ′
and y ∗ i : i + j − 1 in each conditional makes exact estimation of log g ni very complicated and costly.
We thus introduce the approximation where we approximate g n i,i ′ as (cid:101) g n i,i ′ =
(cid:81) n j =1 p ( y i ′
+ j − 1 =
y ∗ i + j − 1 | y <
i ′ + j − 1 ) .
That is, instead of conditioning on y ∗ i : i + j − 1 , we use the model-generated tokens y i ′ :
i ′ + j − 1 as the condition.
This simple approximation enables us to define the probability output
[Figure]
score
[Figure] [Figure]
BLEU score
[Figure] [Figure]
[Figure]
P as in the non-autoregressive case, by just performing a forward pass of the model (i e., sampling a token y
′ i for each position i ′ and feeding it to the next step to get p i ′
+1 ).
We can then apply the same convolution operator to approximately obtain log g ni as in Eq. 7.
Besides the great gain of computational efficiency, we note that the approximation is also effective, especially due to the position selection discussed above.
Specifically, for each reference n -gram y ∗ i : i + n , the position selection in effect (softly) picks those large-value g ni,i ′ (while dropping other low-value ones) to evaluate the loss.
A large g ni,i ′ value indicates the candidate
y i ′ :
i ′ + n is highly likely to match the reference y ∗ i : i + n , meaning that using y i ′ :
i ′ + n in replacement of y ∗ i : i + n is a reasonable approximation for evaluating the above conditionals.
We provide empirical analysis of the approximation in Appendix A.8, where we show the efficient approximate EISL loss values are very close to the exact EISL values.
3. Connections with Common Techniques CE is a special case of EISL
A nice property of
EISL is that it subsumes the standard CE loss as a special case.
To see this, set n = T ∗ (the target sequence length), and we have:
[Equation]
The connection shows the generality of EISL.
As a generalization of CE, it enables learning at arbitrary n -gram granularity.
Connections between BLEU and EISL
Both our method and the popular BLEU (Papineni et al., 2002b) metric use n -grams as the basis in formulation.
Here we articulate the connections and difference between the two.
Let us first take a review of the BLEU metric.
Specifically, BLEU is defined as a weighted geometric mean of n -gram precisions:
[Equation]
where BP is a brevity penalty depending on the lengths of y and y ∗ ; N is the maximum n -gram order (typically N = 4 ); { w n } are the weights which usually take 1 /N ; prec n is the n -gram precision, gram n ( y ) is the set of unique n -gram subsequences of y ; and C ( s , y ) is the number of times a gram s occurs in y as defined in Eq.
The conventional formulation above enumerates over unique n -grams in y .
In contrast, we enumerate over token indexes in calculating the n -gram matching loss.
BLEU considers the n -gram precisions and has a penalty term while EISL simply maximizes the log probability of n -gram matchings.
The non-differentiability of BLEU makes it hard to optimize directly, hence most prior attempts resort to reinforcement learning algorithms and use
BLEU as the reward (Ranzato et al., 2016, Liu et al., 2017).
There are also some works trying to introduce differentiable BLEU metric using approximation like (Zhukov and Kretov, 2017).
However, such losses are often too complicated and have not yet demonstrated to perform well in practice.
Experiments
In this section, we present the experimental results on three text generation settings to test EISL’s effectiveness, including learning from noisy text, learning from weak sequence supervision, and nonautoregressive generation models that require flexibility in generation orders.
More details of the experimental setting are provided in Appendix A.2.
4. Learning from Noisy Text
To test the robustness to noise, we evaluate on the task of machine translation with noisy training target, in which we train the models with noisy sequence targets and evaluate with clean test data.
Setup We test EISL loss on Multi30k and
WMT18 raw corpus.
We use German-to-English (de-en) dataset from Multi30k (Elliott et al., 2016), which contains 29k training instances.
As inspired by Shen et al.
(2019), to simulate various noises in the real data, we introduce four types of noises: shuffle, repetition, blank, and the synthetical noise, i.e., the combination of the aforementioned three types of noise.
The noises are only added to the training target sequences.
To verify the validity of EISL on real noisy data, we also use Germanto-English (de-en) dataset from WMT18 raw corpus, which is a very noisy de-en corpus crawled from the web.
We randomly select different number of training samples to test the influence of the data scale.
We use a Transformer-based pretrained model BART-base (Lewis et al., 2020) and adopt greedy decoding in training and beam search (beam size = 5 ) in evaluation.
We compare EISL loss with CE loss, Policy Gradient (PG), and Loss Truncation (LT).
We also conduct ablation experiments to explore the effect of different n -grams in EISL loss.
We use both BLEU (Papineni et al., 2002b) and BLEURT, an advanced model-based metric (Sellam et al., 2020), as the automatic metrics for evaluation.
Due to space limit, we report BLEU results in the main paper, and defer BLEURT results in the appendix, where we can see BLEURT leads to the same conclusion as BLEU.
Results
The results on noisy Multi30k are presented in Figure 5.
The proposed EISL loss provides significantly better performance than CE loss
[Figure] [Figure]
BLEU score
Number of Training Examples
CE
PG
EISL
[Table]
and PG on all the noise types, especially on the high-level noise end.
For synthetical noise as shown in Figure 5(d), it’s interesting to see that
CE and PG completely fail when the noise level is beyond 6 , but model trained with EISL has high
BLEU score, demonstrating EISL can select useful information to learn despite high noise.
This validates that the proposed EISL is much less sensitive to the noise than the traditional CE loss and policy gradient training method.
The results of different n -gram are shown in Figure 5(e).
As the noise increases, the importance of lower grams, e.g., 1 -gram, is more obvious.
The results on real noisy data, WMT18 raw data, are shown in Figure 6.
EISL loss achieves better performance than
CE loss and PG, and the difference is getting larger when the training data scale increases.
This again demonstrates EISL could learn more valid information in rather noisy data, while CE loss which only considers whole-sentence matching could struggle on noisy data.
In Appendix A.3, we provide more results (e.g., comparison with loss truncation (Kang and Hashimoto, 2020)) and case studies.
4. Learning from Weak Supervisions: Style Transfer
We experiment on transferring two types of text styles (Jin et al., 2022), namely sentiment and political slant, to verify EISL can learn from weak sequence supervisions.
Setup We use the Yelp review dataset and political dataset.
Yelp contains almost 250k negative sentences and 380K positive sentences, of which the ratio of training, valid and test is 7 : 1 : 2 .
Li et al.
(2018) annotated 1000 sentences as ground truth for better evaluation.
The political dataset is comprised of top-level comments on Facebook posts from all 412 members of the United States Senate and House who have public Facebook pages (Voigt et al., 2018).
The data set contains 270K democratic sentences and 270K republican sentences.
And there exists no ground truth for evaluation.
The data preprocessing follows Tian et al.
The structured content preserving model (Tian et al., 2018) is adopted as the base model.
Following previous work, we compute automatic evaluation metrics: accuracy, BLEU score, perplexity (PPL) and POS distance.
We also perform human evaluations on Yelp data to further test the transfer quality.
Results As sentiment results are shown in Table 1, the BLEU gets improved from 65.71 to 68. with EISL loss.
On the premise of the correctness of sentiment transfer, EISL loss plays a critical role to guarantee lexical preservation.
In the meanwhile, all of BLEU(human), PPL, and POS distance get improved.
It is not surprising that EISL loss helps generate sentences more fluently and select the more appropriate words conditions on the content information.
As the human evaluation results are shown in Table 1, the model with EISL loss performs better, in accord with the automatic metrics.
After analyzing the generated samples, we found EISL loss could drive the model to adopt the words which fit the scene better and could understand more semantics but not just replace some keywords.
See some examples in the Appendix A.4.1.
We report the results of political data in Appendix A.4.2.
Our method outperforms all models on BLEU, PPL, and POS distance with comparable accuracy.
For a more fair comparison with the base model, our EISL loss improves the base model on all four metrics, including the accuracy.
The results demonstrate the effectiveness of
EISL for weak supervision task, improving both transfer accuracy fluency and content preservation.
4. Learning Non-Autoregressive Generation
Non-autoregressive neural machine translation (NAT, (Gu et al., 2018)) is proposed to predict tokens simultaneously in a single decoding step, which aims at reducing the inference latency.
The non-autoregressive nature makes it extremely hard for models to keep the order of words in the sentences, hence CE often struggles with NAT problems.
In experiments, we show EISL is superior to CE in NAT which requires modeling flexible generation order of the text.
Setup We use English-to-German dataset from
WMT14 (Luong et al., 2015), which contains 4.5M training instances.
We apply our proposed EISL loss on both fully NAT models (Gu et al., 2018, Sun et al., 2019) and iterative NAT models (Lee et al., 2018, Gu et al., 2019, Ghazvininejad et al., 2019), showing its general applicability and superiority, and we also compare with a wide range of recent methods (Shao et al., 2020, Wang et al., 2019a,
Li et al., 2019, Ghazvininejad et al., 2020).
We evaluate with both BLEU and BLEURT metrics.
Results We first summarize the comparison of
BLEU between EISL loss and CE loss in Table(comparison of BLEURT is in Appendix A.5.2).
The proposed EISL improves the model performance on both the KD and original datasets.
More specifically, for fully NAT models (VanillaNAT and NAT-CRF), EISL gives strong improvement.
For iterative NAT models (iNAT, LevT, and
CMLM), EISL also significantly outperforms the baselines when the iteration step is restricted to a small level as suggested by Kasai et al.
(We show in Appendix A.5.1 that, with increasing iteration steps, the difference fades away.
However, as studied in Kasai et al.
(2020), iterative NAT models with many iteration steps do not hold the intrinsic advantage of speed since Transformer baselines with a shallow decoder can achieve comparable speedup and only at the sacrifice of minor performance drop.)
Table 3 provides more comparison of with recent strong baselines.
Specifically, we apply our EISL on the CMLM base model (Ghazvininejad et al., 2019) which shows strong superiority.
We
[Table] [Table]
loss.
Our method follows the same generation configuration as CMLM-with-AXE.
provide qualitative analysis in Appendix A.5.3.
Conclusions
We have developed Edit-Invariant Sequence Loss (EISL) for end-to-end training of neural text generation models.
The proposed method is insensitive to the shift of n -grams in target sequences, hence suitable for training with noisy data and weak supervisions, where CE loss fails easily.
We show
CE loss is a special case of EISL and build the connection of EISL with BLEU metric and convolution operation, which both have the invariant property.
Experiments on translation with noisy target, text style transfer, and non-autoregressive neural machine translation demonstrate the superiority of our method.
The more general applications and superiority of EISL on other diverse text generation problems as well as fundamental challenges, such as compositional generalization (Andreas et al., 2019) and causal invariance (Hu and Li, 2021) in language, remain to be explored further, which we are excited to study in the future.
A Appendix A. Additional Derivation
For a given i ′ ,
[Equation]
then we derive the detail of Eq.
3 in Eq.
9, where the first inequality holds since T − n + 1 ≥ 0 ; and the second inequality holds by Jensen’s inequality.
A. Detailed Experimental Setup A.2. Learning from Noisy Text
We use a Transformer-based pretrained model
BART-base (Lewis et al., 2020), containing 6 layers in the encoder and decoder.
We train the model using the Adam optimizer with learning rate 3 × 10 −with polynomial decay and the maximum number of tokens is 6000 in one step.
The models are trained on one Tesla V100 DGXS with 32GB memory.
We start with CE training using teacher forcing for fast initialization.
We then switch to combined 1 - and 2 -gram EISL with weight 0 .
2 , which we select using the validation set.
We adopt greedy decoding in training and beam search (beam size = 5 ) in evaluation.
We use fairseq 2 (Ott et al., 2019) to conduct the experiments.
We compare
EISL loss with CE loss and Policy Gradient (PG), where PG is used to finetune the best CE model.
Teacher forcing is employed in CE training.
A.2. Learning from Weak Supervisions: Style Transfer
We use the Adam optimizer with learning rate 5 × 10 − 4 , the batch size is 128 and the model is trained on one Tesla V100 DGXS 32GB.
We compare the results between the base model and the model with EISL.
Specifically, on top of the base model, we add the EISL loss (a combination of 2 , 3 and 4 -gram with the same weights 1 / 3 ) to reduce the discrepancy between the transferred sentence generated by language model and the original sentence.
We assign EISL loss with weight 0 .
Following previous work, we compute automatic evaluation metrics: accuracy, BLEU score, perplexity (PPL) and POS distance.
For accuracy, we adopt a CNN-based classifier, trained on the same training data, to evaluate whether the generated sentence possesses the target style.
Then we measure BLEU score and BLEU(human) score of transferred sentences against the original sentences and ground truth, respectively.
PPL metric is evaluated by GPT2 (Radford et al., 2019) base model after finetuning on the corresponding dataset, with the goal to assess the fluency of the generated sentence.
POS distance is used to measure the model’s semantics preserving ability (Tian et al., 2018).
We also perform human evaluations on Yelp data to further test the transfer quality.
We first randomly select 100 sentences from the test set, use these sentences as input and generate sentences from the base model (Tian et al., 2018) and our model.
Then for each original sentence, we present the outputs of the base model and ours in random order.
The three annotators are asked to evaluate which sentence is preferred as the transferred sentence of the original sentence, in terms of content preservation and sentiment transfer.
They can choose either output or select the same quality.
We measure the percentage of times each model outperforms the other.
A.2. Learning Non-Autoregressive Generation
We use the Adam optimizer with learning rate 5 × 10 − 4 with inverse square root scheduler.
We apply sequence-level knowledge distillation to the dataset, which can reduce the complexity of the dataset, making it easier for the model to learn and improving the performance.
The models are first trained by CE loss for fast initialization, then focus on 2 -gram, 3 -gram, and 4 -gram with the same weights.
Fairseq (Ott et al., 2019) is adopted to conduct the experiments.
We average the lastcheckpoints as the final model.
A. Additional Results of Learning from Noisy Text A.3. Results of BLEURT Metric
In this section, we evaluate the results of CE, PG and EISL on BLEURT (Sellam et al., 2020) metric.
We use the recommended BLEURT-20 checkpoint. It gives a score for every sentence pair, and we averaged the scores to get the final score. The results are shown in Figure 7. Both BLEU metric and BLEURT metric show the superiority of our proposed EISL loss.
A.3. Comparison with Loss Truncation
The Loss Truncation (LT (Kang and Hashimoto, 2020)), method adaptively removes high log loss l EISL n,i ( θ )
= − log
T − n +1 (cid:88) i ′ =
p ( y i ′ :
i ′ + n = y ∗ i : i + n ) , (9) =
− log
T − n +
T − n +1 (cid:88) i ′ =
(cid:88)
y p
( y <i ′ )
p ( y i ′ :
i ′ + n = y ∗ i : i + n | y <
i ′ )
− log( T − n + 1) , ≤ − log
T − n +
T − n +1 (cid:88) i ′ =
(cid:88)
y p
( y <i ′ )
p ( y i ′ :
i ′ + n = y ∗ i : i + n | y <
i ′ ) , ≤ −
T − n +
T − n +1 (cid:88) i ′ =
(cid:88)
y p
( y <i ′ ) log p ( y i ′ :
i ′ + n = y ∗ i : i + n | y <
i ′ ) , = −
T − n + 1 E y ∼ p
( y )
T − n +1 (cid:88) i ′ =
log p ( y i ′ :
i ′ + n = y ∗ i : i + n | y <
i ′ ) ,
=
L EISL n,i ( θ ) , examples as a way to optimize for distinguishability.
In this section, We’d like to show the comparisons with Loss Truncation.
We evaluated two variants of LT: (1) LT_Pre which first trains the model with CE loss and then adds LT for further training, and (2) LT which directly trains the model with CE loss and LT together.
Hyperparameters were selected on the validation set.
For simplicity, we remove the PG curves (Figure 5), and the comparison results with LT are shown in Figure 8.
We can see Loss Truncation can sometimes slightly improve over CE, especially when the data is clean or with low/moderate noise.
However, by simply ignoring high-loss data, LT is not good at handling data with high noise (which often leads to high loss).
In comparison, our proposed EISL achieves a substantial improvement in the presence of high noise.
A.3. Reasons of Better Performance with Lower-gram EISL
In this section, we discuss the reason of why the performance of using lower grams is better than higher-gram EISL in Figure 5(e).
Lower-gram EISL is less sensitive to noise.
For example, 1-gram EISL focuses mostly on matching individual tokens without caring much about the order of tokens; while a high-gram EISL (e.g., consider the extreme case of T ∗ -gram where T ∗ is the target length) reduces to CE (as discussed in
Sec 3.3) and is highly sensitive to noise.
Thus, in the presence of high data noise, lower-gram EISL would be more robust and perform better.
Besides, on low-noise data (e.g., noise-level = 0 or 1), lower-gram EISL performs comparably with higher-gram EISL, both close to the CE performance.
This is because we pretrained the model with CE (as mentioned in the experimental setup), and finetuning with EISL (either with lower- or higher-grams) would not change the performance a lot given the low-noise data.
A.3. Cases Study
As shown in Table 8, 9, 10, 11 and 12, we randomly sample some examples from generated sentences of the models trained with different types of noise on Multi30k dataset.
For the sake of convenience, we use abbreviations in the tables, i.e., SC, RR,
BR and NL are short for Shuffle Count, Repetition
Ratio, Blank Ratio and Noise Level (for Synthetical
Noise), respectively.
Shuffle Noise
When there exist a few shuffle noises, e.g., SC = 3 , CE loss may lead word reduplicated (Example 1 and Example 2) and slightly wrong word order (Example 4 and Example 5), and there are some information mistranslated ( beautiful in Example 4) or extra irrelevant information added ( black in Example 5).
As shuffle count increases, the aforementioned problems are increasingly severe, resulting the generated sentences meaningless.
Especially, there are some words untranslated in PG examples ( eingezäunten in Example 1, irgendwo in Example 2, haben in Example 5, ).
But
EISL loss could keep the content consistency and grammatical correctness as far as possible. [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
BLEURT
[Figure] [Figure] Repetition Noise The main problem of the models trained by CE and PG with repetition noises is that the models can’t filter the repetition noise out in training samples, and try to learn the wrong distribution, leading to generate reduplicated words frequently (Example 1-5).
Specifically, the examples of CE and PG in RR = 50% are very representative.
However, it’s amazing that EISL can almost avoid such a problem even the repetition ratio achieves 50% .
Meanwhile, the main semantics is preserved and the grammar is correct.
Blank Noise
When adding blank noise, some tokens in targets will be substituted as unk so the targets will lose some information.
We could measure from two aspects: one is the term frequency of meaningless token unk in generated sentences, and the other is the meaningful contents preserved by the models.
Obviously, EISL loss handles better than CE loss on both aspects.
Especially, when BR = 20% , unlike models with CE, models with PG and EISL barely generate the unk token, and could translate the core content (Example 1-5).
As BR increases, EISL could preserve more key information and produce less unk than CE and PG.
Moreover,
PG performs rather poor when BR is high (like BR = 45% ), and it almost loses all information (Example 1-5) and generates some confusing words ( teil in Example 1, afroamerikanischer and irgendwo in
Example 3, beachaufsichtgebäude in Example 4, and holzstück in Example 5).
Synthetical Noise
We then evaluate the results of models trained by synthetical noise.
Such a [Table] [Table] situation combines aforementioned three types of noises.
One most highlighted advantage of EISL is that the generated sentences are almost grammatically correct and include main content as far as possible.
However, CE can only stiffly joint some words [Table] and can’t guarantee the grammatical correctness (word order, word repetition and so on).
PG performs worst, involving all the problems in CE cases and the meaningless word generation problem (Example 1-5).
A. Additional Results of Text Style Transfer A.4. Examples on Yelp dataset
Some examples of generated sentences are given in Table 4.
The model with EISL can select more appropriate adjective and improve the quality of the sentences.
In the first example, the model should transfer the negative adjectives cold and watery to some positive adjectives that describe food.
Obviously, the delicious is more appropriate than excellent .
In the second example, the base model reverses both not and stop , leading to wrong sentiment and inconsistent content.
While the model with EISL could avoid such a situation and generate more suitable sentence.
A.4. Results on Political dataset
Since the instances from democratic data and republican data are quite different, names of politicians have high correlation with the political slant.
Therefore the BLEU score and POS distance have a big gap with the sentiment results.
The results are shown in Table 5.
[-LevT on WMT14 en-de KD           
-]
A. Additional Results of Non-Autoregressive Generation A.5. Results of Iterative NAT Models
As shown in Figure 9, with the increasing of iteration steps, the difference fades away.
A.5. Results of BLEURT Metric
To show the superiority of our method, We also evaluate on recent text generation metric,
BLEURT (Sellam et al., 2020).
BLEURT is an evaluation metric for Natural Language Generation.
It takes a pair of sentences as input, a reference and a candidate, and it returns a score that indicates to what extent the candidate is fluent and conveys the mearning of the reference.
We use the recommended BLEURT-20 checkpoint.
It gives a score for every sentence pair, and we averaged the scores to get the final score.
The results are shown in
Table 6.
A.5. Qualitative Analysis on NAT Experiments
Given the non-autoregressive nature (i.e., all tokens are generated simultaneously), the one-to-one matching of CE loss can lead to severe mismatching.
We consider the example: the predicted sentence is a cat is on the red blanket and the target sentence is a cat is sitting on the red blanket .
The "on the red blanket" part of the prediction will be corrected to match the target positions, and this may lead to overcorrection (e.g., "on the red red blanket .").
Repetition is often a sign of overcorrection.
However, with EISL, this situation will not happen because the phrase will be matched to appropriate
9 10 Max decoding iteration
LevT on WMT14 en-de            9 10 Max decoding iteration
{+LevT on WMT14 en-de KD           
+}
BLEU score
Decoding iteration iNAT on WMT14 en-de KD CE EISL [Figure]
[Table]
[Figure]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Figure]
target tokens.
Let’s have a look at a real example in Figure 10.
[Figure]
Take the non-autoregressive model
CMLM (Ghazvininejad et al., 2019) for example, we evaluate the translation of CMLM models trained by CE and EISL.
As shown in Figure 11, our proposed EISL can reduce repetition to a large extent.
[Figure]
[Figure]
A. Efficiency Analysis Complexity analysis
Given T ∗ tokens, the time complexity of CE loss is O ( T ∗ ) , while the complexity of n -gram EISL loss is O ( n ( T ∗ − n + 1) 2 )
≈ O
( T ∗ 2 ) , assuming small n is used in practice (e.g., n ∈ { 1 , 2 , 3 , 4 } ).
However, in practice, the computation cost of the loss (either CE or EISL) is negligible compared to the cost of model forward and backward during training.
Thus, the extra cost introduced by the EISL loss is rather minor.
Empirical comparison of time cost
To quantify the computational cost of different methods, we adopt CE and EISL on top of the same model and setting, and evaluate the consumed time fortraining epoch.
For comparison on both small and large dataset, we evaluate on Multi30k (29k training data, 1k test data) and 1M scale WMT-18 raw corpus (1M training data, 3k test data).
The models are tested on one Tesla V100 DGXS with
GB memory, the batch size is 128, max number of tokens is 6000 and update frequency is 4.
For each method, we test 6 times and average the results as final time.
The results are shown in Figure 12.
Empirical total time cost of EISL training
As discussed in the experiments in the paper, we first pretrain the model with the CE loss until convergence, and then finetune with the EISL loss.
Here we report the total time cost of each stage, based on the WMT-18 translation setting as described in
Section 4.1.
The results are shown in Table 7.
As
[Figure] Time [Figure] Multi30k-Training
[Figure]
Time
[Figure]
Multi30k-
[Figure]
Time (
[Figure]
[Figure]
the data size increases, the convergence time of both pretraining and finetuning grows.
The time cost of the finetuning stage is less than half of that of the pretraining stage.
A. Hyperparameters
Regarding which n -grams to use and their weights w n in the EISL loss, we found in our experiments that the default values largely following the standard BLEU metric (i.e., maximum n = 4 with equal weights) work well.
Specifically, we use n ∈ { 2 , 3 , 4 } and equal weights w n = 1 / 3 as our default values.
Most of our experiments adopt the default values which achieve consistent substantial improvement over CE and other rich baselines as shown in our experiments.
(except for the synthetic experiment where we show the effect of different ngrams including those selected using the validation set).
Besides, in our experiments, we first pretrain the model with the CE loss (i.e., EISL with n =
T ∗ and teacher forcing, see Section 3.3) and then finetune with the EISL loss.
We simply do the CE pretraining until convergence before switching to the EISL finetuning.
Therefore, there is no need of tuning for the training iterations of pretraining.
A. Analysis of Efficient Implementation
In order to validate the efficiency and accuracy of our approximation (for autoregressive models) discussed in Section 3.2, we conduct the analysis experiments, showing that the approximate (and efficient) EISL loss values are very close to exact (but expensive) EISL value.
We use the same setting as section 4.1, and finetune the model with our efficient approximate EISL loss on Multi30k.
Throughout the course of training, we record the loss values of both the exact implementation and our approximate implementation.
As shown in Figure 13(a) and (b), the tendency of two losses is very close to each other.
We also plot the absolute difference of the two losses as shown in Figure 13(c).
We can see the difference decreases as training proceeds.
The observations validate the effectiveness of our approximate implementation.
We note that training the model with the exact loss is costly, which necessitates our approximation.
Specifically, for n -gram loss, we need to run the forward pass of the decoder ( T − n ) 2 times, and keep the whole computation graph for backpropagation, which will consume much more time and memory.
Even for only loss evaluation (without the backward pass), we found the runtime of the exact loss is about 15 times longer than that of the efficient approximate implementation based on convolution operator.
[Table]
[Figure]
[Table]
[Table] [Table]
[Table] [Table]