Non-Monotonic Latent Alignments for CTC-Based Non-Autoregressive Machine Translation
Abstract
Non-autoregressive translation (NAT) models are typically trained with the crossentropy loss, which forces the model outputs to be aligned verbatim with the target sentence and will highly penalize small shifts in word positions.
Latent alignment models relax the explicit alignment by marginalizing out all monotonic latent alignments with the CTC loss.
However, they cannot handle non-monotonic alignments, which is non-negligible as there is typically global word reordering in machine translation.
In this work, we explore non-monotonic latent alignments for NAT.
We extend the alignment space to non-monotonic alignments to allow for the global word reordering and further consider all alignments that overlap with the target sentence.
We non-monotonically match the alignments to the target sentence and train the latent alignment model to maximize the F1 score of nonmonotonic matching.
Extensive experiments on major WMT benchmarks show that our method substantially improves the translation performance of CTC-based models. Our best model achieves 30.06 BLEU on WMT14 En-De with only oneiteration decoding, closing the gap between non-autoregressive and autoregressive models.
Introduction
Non-autoregressive translation (NAT) models achieve signiﬁcant decoding speedup in neural machine translation [NMT, 1, 47] by generating target words simultaneously [17].
This advantage usually comes at the cost of translation quality due to the mismatch of training objectives.
NAT models are typically trained with the cross-entropy loss, which forces the model outputs to be aligned verbatim with the target sentence and will highly penalize small shifts in word positions.
The explicit alignment required by the cross-entropy loss cannot be guaranteed due to the multi-modality problem that there exist many possible translations for the same sentence [17], making the cross-entropy loss intrinsically mismatched with NAT.
As the cross-entropy loss can not evaluate NAT outputs properly, many efforts have been devoted to designing better training objectives for NAT [29, 40–42, 38, 13, 36, 9, 43].
Among them, latent alignment models [29, 36] relax the alignment restriction by marginalizing out all monotonic latent alignments with the connectionist temporal classiﬁcation loss [CTC, 15], which receive much attention for the ability to generate variable length translation.
Latent alignment models generally make a strong monotonic assumption on the mapping between model output and the target sentence.
As illustrated in Figure 1a, the monotonic assumption holds in [Figure] classic application scenarios of CTC like automatic speech recognition (ASR) as there is a natural monotonic mapping between the speech input and ASR target.
However, non-monotonic alignments are non-negligible in machine translation as there is typically global word reordering, which is a common source of the multi-modality problem.
As Figure 1b shows, when the target sentence is “I ate pizza this afternoon” but the model produces a translation with a different but correct word ordering “this afternoon I ate pizza”, the CTC loss cannot handle this non-monotonic alignment between output and target and wrongly penalizes the model.
In this paper, we propose to model non-monotonic latent alignments for non-autoregressive machine translation.
We ﬁrst extend the alignment space from monotonic alignments to non-monotonic alignments to allow for the global word reordering in machine translation.
Without the monotonic structure, we have to optimize the best alignment found by the Hungarian algorithm [46, 28, 7, 9] since it becomes difﬁcult to marginalize out all non-monotonic alignments with dynamic programming.
This difﬁculty can be overcome by not requiring an exact match between alignments and the target sentence.
In practice, it is not necessary to have the translation include exact words as contained in the target sentence, but it would be favorable to have a large overlap between them.
Therefore, we further extend the alignment space by considering all alignments that overlap with the target sentence.
Speciﬁcally, we are interested in the overlap of n-grams, which is the core of some evaluation metrics (e.g., BLEU).
We accumulate n-grams from all alignments regardless of their positions and non-monotonically match them to target n-grams.
The latent alignment model is trained to maximize the F1 score of n-gram matching, which reﬂects the translation quality to a certain extent [32].
We conduct experiments on major WMT benchmarks for NAT (WMT14 En ↔ De, WMT16 En ↔ Ro), which shows that our method substantially improves the translation performance and achieves comparable performance to autoregressive Transformer with only one-iteration parallel decoding.
Background 2.1 Non-Autoregressive Machine Translation
[17] proposes non-autoregressive neural machine translation, which achieves signiﬁcant decoding speedup by generating target words simultaneously.
NAT breaks the dependency among target tokens and factorizes the joint probability of target words in the following form:
[Equation]
where x is the source sentence belonging to the input space X , y = { y 1 , ..., y T } is the target sentence belonging to the output space Y , and p t ( y t | x, θ ) indicates the translation probability in position t .
The vanilla-NAT has a length predictor that takes encoder states as input to predict the target length.
During the training, the target length is set to the golden length, and the vanilla-NAT is trained with the cross-entropy loss, which explicitly aligns model outputs to target words:
[Equation]
CTC: y =
# −1 (') ' = ( ε , A,
[Equation] [Equation] [Equation] [Equation]
[Equation]
[Equation]
(A, A,
[Equation] [Equation]
[Equation] [Equation]
[Equation]
) (A, A, B, C, D)
SCTC:
[Figure]
= #
[Figure]
The collapsing function of SCTC is β − 1 s , which only removeS blanks in the alignment a , where the collapsing function of CTC removes repetitions ﬁrst and then removes blanks. 2.2 NAT with Latent Alignments
The vanilla-NAT suffers from two major limitations.
The ﬁrst limitation is the explicit alignment required by the cross-entropy loss, which cannot be guaranteed due to the multi-modality problem and therefore leads to the inaccuracy of the loss.
The second limitation is the requirement of target length prediction.
The predicted length may not be optimal and cannot be changed dynamically, so it is often required to use multiple length candidates and re-score them to produce the ﬁnal translation.
The two limitations can be addressed by using CTC-based latent alignment models [15, 29, 36], which extend the output space Y with a blank token (cid:15) that means ‘output nothing’.
We deﬁne the extended output space as Y ∗ .
Following prior works [14, 36], we refer to the elements a ∈ Y ∗ as alignments, since the location of the blank tokens determines an alignment between the extended output and target sentence.
Assume the generated alignments have length T a , we deﬁne a function β ( y ) which returns a subset of Y ∗ including all possible alignments of length T a for y , where β − 1 : Y ∗ (cid:55)→ Y is the collapsing function that ﬁrst collapses all consecutive repeated words in a and then removes all blank tokens to obtain the target sentence.
As illustrated in Figure 2, the alignment a = ( (cid:15), A, A, (cid:15), A, B, B, C, (cid:15), D ) is collapsed to the target sentence y =
( A, A, B, C, D ) with a monotonic mapping from target positions to alignment positions.
During the training, latent alignment models marginalize all alignments with the CTC loss [15]:
[Equation]
where the alignment probability p ( a | x, θ ) is modeled by a non-autoregressive Transformer:
[Equation]
NAT models with latent alignments achieve superior performance by overcoming the two major limitations of NAT [29, 36, 16, 25, 33, 49].
The weakness in handling non-monotonic latent alignments is noticed in previous work [36, 16] but remains unsolved.
Approach
In this section, we explore non-monotonic latent alignments for NAT.
We introduce a simpliﬁed CTC loss in section 3.1, which has a simpler structure that helps further analysis and derivation under the regular CTC loss.
We explore non-monotonic alignments under the simpliﬁed CTC loss in section 3.2, and utilize the results to derive non-monotonic alignments under the regular CTC loss in section 3.3.
Finally, we introduce the training strategy to combine monotonic and non-monotonic latent alignments in section 3.4.
3.1 Simpliﬁed Connectionist Temporal Classiﬁcation
In CTC, the collapsing function β − 1 maps an alignment to a target sentence in two steps: (1) collapses all consecutive repeated words, (2) removes all blank tokens.
The two operations make the alignment space β ( y ) complex.
Therefore, we ﬁrst introduce the simpliﬁed connectionist temporal classiﬁcation loss (SCTC), which has a simpler structure that helps the derivation of non-monotonic alignments, and the results on SCTC are helpful for further analysis under the regular CTC loss.
To simplify the alignment structure, we consider only using one operation in the collapsing function.
One option is only collapsing all consecutive repeated words in the alignment.
The concern is that it will limit the expressive power of the model.
For example, the target sentence y
=
( A, A, B, C, D ) has probability 0 since it has repeated words that cannot be mapped from any alignment.
Therefore, we favor another option that only removes all blank tokens.
In this way, the expressive power stays the same and the alignment space becomes much simpler, where alignments of y simply contain T target words and T a − T blank tokens.
We denote this simpliﬁed loss as SCTC, and illustrate the difference between CTC and SCTC in
Figure 2.
The alignment space for the target sentence y is deﬁned as β s ( y ) , and the collapsing function is deﬁned as β − 1 s .
We can still obtain the translation probability p ( y | x, θ ) with dynamic programming.
We deﬁne the forward variable α t ( s ) to be the total probability of y 1: s considering all possible alignments a 1: t , which can be calculated recursively from α t − 1 ( s ) and α t − 1 ( s − 1) :
[Equation]
Finally, the total translation probability p ( y | x, θ ) is given by the forward variable α T a ( T ) , so we can train the model with the cross-entropy loss L sctc ( θ )
= − log α T a ( T ) .
3.2 Non-Monotonic Alignments under SCTC 3.2.1 Bipartite Matching
In this section, we explore non-monotonic alignments under the SCTC loss, which are helpful for further analysis under the regular CTC loss.
We ﬁrst extend the alignment space from monotonic alignments β s ( y ) to non-monotonic alignments γ s ( y ) to allow for the global word reordering in machine translation, where γ s ( y ) is deﬁned as:
[Equation]
In the above deﬁnition, P ( y ) represents all permutations of y .
For example, P (( A, B ))
= { ( A, B ) , ( B, A ) } .
By enumerating P ( y ) , we consider all possible word reorderings of the target sentence.
Ideally, we want to traverse all alignments in γ s ( y ) to calculate the log-likelihood loss:
[Equation]
However, without the monotonic structure, it becomes difﬁcult to marginalize out all latent alignments with dynamic programming.
Alternatively, we can minimize the loss of the best alignment, which is an upper bound of Equation 7:
[Equation]
Following prior work [46, 7, 9], we formulize ﬁnding the best alignment as a maximum bipartite matching problem and solve it with the Hungarian algorithm [28].
Speciﬁcally, we observe that the alignment space γ s ( y ) is simply permutations of T target words and T a − T blank tokens.
Therefore, ﬁnding the best alignment is equivalent to ﬁnding the best bipartite matching between the T a model predictions and the T target words plus T a − T blank tokens, where the two sets of nodes are connected by edges with the prediction log-probability as weights.
3.2.2 N-Gram Matching
Without the monotonic structure desired for dynamic programming, calculating Equation 7 becomes difﬁcult.
This difﬁculty is also caused by the strict requirement of exact match for alignments, which makes it intractable to simplify the summation of probabilities.
In practice, it is not necessary to force the exact match between alignments and the target sentence since a large overlap is also favorable.
Therefore, we further extend the alignment space by considering all alignments that overlap with the target sentence.
Speciﬁcally, we are interested in the overlap of n-grams, which is the core of some evaluation metrics (e.g., BLEU).
We propose a non-monotonic and non-exclusive n-gram matching objective based on SCTC to encourage the overlap of n-grams.
Following the underlying idea of probabilistic matching [39], we introduce the probabilistic variant of the n-gram count to make the objective differentiable.
Speciﬁcally, we use C g ( y ) to denote the occurrence count of n-gram g =
( g 1 , ..., g n ) in the target sentence y , and use C sg ( θ ) to denote the probabilistic count of n-gram g for the model with input x and parameter θ , where the superscript s indicates SCTC.
For simplicity, we omit the source sentence x in C sg ( θ ) and the following notations.
The probabilistic n-gram count is obtained by accumulating n-grams from all possible target sentences:
[Equation]
The calculation of C sg ( θ ) is the core of this method and will be described in detail later.
We use
M sg ( θ ) to denote the match count of n-gram g between the latent alignment model and the target sentence y
(we omit y in M sg ( θ ) for simplicity), which is deﬁned as follows:
[Equation]
Our objective is to maximize the n-gram overlap between the target sentence and model output, so we can train the model to maximize the precision or recall of n-gram matching:
[Equation]
where we use G n to denote the set of all n-grams.
However, since latent alignment models have the ability to control the translation length, both precision and recall cannot accurately reﬂect the translation quality.
The precision will encourage short translations to reduce the denominator, while the recall prefers long translations.
Therefore, we consider both of them and maximize the F1 score:
[Equation]
where we use G n to denote the set of all n-grams.
In the numerator of Equation 12, M sg ( θ ) is nonzero only if C g ( y ) is non-zero, so we only need to calculate probabilistic n-gram counts C sg ( θ ) for n-grams g in the target sentence.
In the denominator, (cid:80) g ∈ G n C g
( y ) equals to the constant T − n + 1 .
Therefore, there are only two tasks left: calculating C sg ( θ ) and the summation (cid:80) g ∈ G n C sg ( θ ) .
For the ﬁrst task of calculating C sg ( θ ) , we ﬁrst consider the simple case n = 1 .
The main difference between 1-gram matching and bipartite matching is that 1-gram matching is non-exclusive, which allows us to marginalize out all alignments using the property of non-autoregressive generation.
Due to the space limit, we refer readers to Appendix.
A for the derivation and directly give the probabilistic 1-gram count as follows:
[Equation]
where G 1 denotes the set of all 1-grams g = ( g 1 ) .
Both 1-gram matching and bipartite matching are completely non-monotonic, which aim at generating correct target words regardless of the word order.
When the target sentence is “I ate pizza this afternoon”, they will think that “pizza ate I this afternoon” is also a good translation.
We consider 2-gram matching to overcome this limitation, which is also non-monotonic but takes target dependency into consideration.
We ﬁrst introduce a transition matrix A of size T a × T a , where A i,j is the probability that all positions between i and j are blank tokens:
[Equation]
Then we give the probabilistic 2-gram count for g
= ( g 1 , g 2 ) (see Appendix.
A for derivation):
[Equation]
For a larger n [Equation] the probabilistic n-gram count is:
[Equation]
Equation 15 can be efﬁciently calculated in matrix form with O ( T 2 a ) time complexity.
For Equation 16, we need O ( nT na ) operations to calculate it by brute force, which is impractical when n is large.
We refer readers to Appendix.
B for the O ( nT 2 a ) algorithm, which is similar to calculating the n-step transition probability of a Markov chain.
For the second task of efﬁciently calculating the summation of probabilistic n-gram counts (cid:80) g ∈ G n C sg ( θ ) , we also consider the simple case n = 1 ﬁrst, where the summation can be calculated by directly summing all probabilities:
[Equation]
For n > 1 , the search space is too big for the direct summation.
Fortunately, the deﬁnition of C sg ( θ ) in Equation 9 allows us to safely extend the result of n = 1 to a larger n since n-grams count in a sentence is always n − 1 less than 1-grams count.
[Equation]
In summary, with the above results (Equation 13-18), we are able to efﬁciently calculate the loss deﬁned in Equation 12 to maximize the F1 score of non-monotonic n-gram matching, which reﬂects the translation quality to a certain extent.
3.3 Non-Monotonic Alignments under CTC
In this section, we explore non-monotonic latent alignments under the regular CTC loss.
Under
SCTC, we can use both the bipartite matching and n-gram matching to train the latent alignment model.
However, bipartite matching is infeasible under the regular CTC loss.
Recall that the search space γ s ( y ) is simply permutations of target words and T a − T blank tokens, which allows us to ﬁnd the best alignment with the Hungarian algorithm.
However, as consecutive repeated words will be collapsed in the regular CTC loss, alignments in γ ( y ) may contain different words, so the best alignment cannot be found by the Hungarian algorithm.
Fortunately, for n-gram matching, we can utilize previous results of SCTC to help the analysis and derivation under the regular CTC loss.
As the output of the collapsing function β − 1 s ( a ) simply contains more repeated words than β − 1 ( a ) , we can ﬁrst identify how much inﬂuence these repeated words have, and then remove this inﬂuence from the results of SCTC.
We ﬁrst formally deﬁne the probabilistic n-gram count under the regular CTC:
[Equation]
We deﬁne R g ( θ ) to be the gap bewteen C g ( θ ) and C sg ( θ ) when n = 1 , which represents the number of repeated words g 1 removed by the collapsing function.
R g ( θ ) can be efﬁciently calculated as follows (see Appendix.
C for derivation):
[Equation]
The above equation enables us to efﬁciently calculate C g ( θ ) and (cid:80) g C g
( θ ) when g ∈ G 1 .
For 2-grams g = ( g 1 , g 2 ) , we divide it into two cases g 1 = g 2 and g 1 (cid:54) =
g 2 .
The collapsing of consecutive repeated words will only remove 2-grams of the ﬁrst case, where each removed word corresponds to a removed 2-gram.
Taking Figure 2 as an example, the output of SCTC β − 1 s ( a ) have two more 2-grams { ( A, A ) , ( B, B ) } of the g 1 = g 2 case than the output of CTC β − 1 ( a ) , and they contain exactly the same 2-grams of the g 1 (cid:54) =
g 2 case.
Therefore, we can directly reuse previous results of
SCTC and the number of repeated words R g ( θ ) to calculate the probabilistic 2-gram count:
[Equation]
Unfortunately, for n > 2 , there is no such a clear relationship between C g ( θ ) and C sg ( θ ) , where the
CTC output can contain n-grams (for example, ( A, B, C ) in Figure 2) that the SCTC output does not have.
If we directly formulize C g ( θ ) like in Equation 16, it will be more complex and the O ( nT 2 a ) algorithm is no longer applicable.
Therefore, we do not use n-gram matching with n > 2 under the regular CTC loss.
Finally, we deﬁne the match count M g ( θ ) similarly as in Equation 10 and train the
CTC-based latent alignment model to maximize the F1 score:
[Equation]
3.4 Training
In the above, we propose non-monotonic training objectives for the latent alignment model.
They need to be combined with monotonic training objectives, otherwise, they may generate translations like “pizza ate I this afternoon”.
Therefore, we ﬁrst pretrain the latent alignment model with monotonic training objectives like SCTC or CTC.
The model pretrained by SCTC can be ﬁnetuned with the bipartite matching objective (Equation 8) or the n-gram matching objective (Equation 12).
The model pretrained by CTC can be ﬁnetuned with the n-gram matching objective (Equation 22), and the choices of n are limited to { 1 , 2 } .
Experiments 4.1 Experimental Setup
Datasets We evaluate our methods on the most widely used public benchmarks in previous NAT studies: WMT14 English ↔ German (En ↔ De, 4.5M sentence pairs) [5] and WMT16 English ↔ Romanian (En ↔ Ro, 0.6M sentence pairs) [6].
For WMT14 En ↔ De, the validation set is newstest2013 and the test set is newstest2014 .
For WMT16 En ↔ Ro, the validation set is newsdev-2016 and the test set is newstest-2016 .
Following prior works, we use tokenized BLEU [32] to evaluate the translation quality.
Considering that BLEU might be biased as our method optimizes the n-gram overlap, we also report METEOR [2], which is not directly based on n-gram overlap.
We learn a joint BPE model [37] with 32K operations to process the data and share the vocabulary for source and target languages.
Knowledge Distillation Following prior works [17], we apply sequence-level knowledge distillation [26] to reduce the complexity of training data.
We use Transformer-base setting for the teacher model and train NAT on the distilled data.
Implementation Details We adopt Transformer-base [47] as our autoregressive baseline as well as the teacher model.
We uniformly copy encoder outputs to construct decoder inputs.
For CTC-based
NAT models, the length for decoder inputs is 3 × as long as the source length.
On WMT14 En ↔ De, we use a dropout rate of 0.2 to train NAT models and use a dropout rate of 0.1 for ﬁnetuning.
On
WMT16 En ↔ Ro, the dropout rate is 0.3 for both the pretraining and ﬁnetuning.
We use the batch size 64K and train NAT models for 300K steps on WMT14 En ↔ De and 150K steps on [-WMT
-]WMT
En ↔ Ro.
During the ﬁnetuning, we train NAT models for 6K steps with the batch size 256K. All models are optimized with Adam [27] with β = (0 . 9 , 0 . 98) and (cid:15) =
10 − 8 .
The learning rate warms up to 5 · 10 − 4 within 10K steps in the pretraining and warms up to e · 10 − 4 within 500 steps in the ﬁnetuning.
We use the GeForce RTX 3090 GPU to train models and measure the translation latency.
We implement our models based on the open-source framework of fairseq [MIT License, 31].
Decoding For autoregressive models, we use beam search with beam size 5 for the decoding.
For
Vanilla-NAT, we use argmax decoding to generate the translation.
For CTC-based models, we can also use argmax decoding to generate the alignment and then collapse it to obtain the translation.
Besides, following [29, 16], we can use beam search decoding combined with a 4-gram language model [20] to ﬁnd the translation that maximizes:
[Equation]
(23) where α and β are hyperparameters for the language model score and length bonus.
We use a ﬁxed beam size 20 with grid-search α , β .
The beam search does not contain any neural network computations and can be implemented efﬁciently in C++ 3 . [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
4.2 Main Results
We ﬁrst compare the performance of different non-monotonic training objectives for latent alignment models.
Baseline models include the Transformer, vanilla-NAT, SCTC, and CTC, and we ﬁnetune latent alignment models with bipartite matching and n-gram matching objectives.
In Table 1, we report BLEU and METEOR scores along with the speedup on WMT14 En-De test set.
From Table 1, we have the following observations: (1) Non-monotonic training objectives can effectively improve the performance of latent alignment models, which improves SCTC by [-1.
-]1.
BLEU and 1.48 METEOR and improves CTC by 1.23 BLEU and 1.35 METEOR, illustrating the importance of non-monotonic alignments.
(2) Despite having the same expressive power, SCTC underperforms CTC in the task of non-autoregressive translation.
We speculate that it is because of the difference in the distribution of the alignment space.
Most target sentences do not contain repeated words, in which case the alignment space of SCTC β − 1 s ( y ) is a subset of β − 1 ( y ) , making the ability of SCTC relatively weak.
(3) SCTC achieves the best performance with 2-gram ﬁnetuning, and then the performance decreases with the increase of n due to the sparsity of higher rank n-grams.
It suggests that we do not need a large n and relieves the concern on CTC that we cannot calculate n-gram matching for n > 2 .
The correlation between the reported BLEU and METEOR scores is very strong, suggesting that BLEU is not biased when evaluating our methods.
Therefore, we can safely use BLEU to evaluate the translation quality in the following experiments.
We call the 2-gram matching objective NMLA since it represents our best non-monotonic training method for latent alignment models.
Besides CTC, we also apply NMLA ﬁnetuning on a stronger baseline DDRS [43] 4 , which trains the CTC model with multiple references.
To extend our method to multiple references, we calculate the n-gram counts for each reference and use their average as the target n-gram count.
We compare their performance with existing approaches in Table 2.
Notably, CTC with NMLA achieves comparable performance to autoregressive Transformer on all benchmarks, demonstrating the effectiveness of our method.
On the strong baseline DDRS, NMLA ﬁnetuning also brings remarkable improvements and outperforms autoregressive baselines.
With the help of beam search and 4-gram language model, NMLA even outperforms other iterative NAT models and the autoregressive Transformer with only one-iteration parallel decoding.
Compared to our implementations of prior works that explore non-monotonic matching for NAT [41, 9], we can see that they underperform the CTC baseline even when rescoring 5 candidates, and their speedup after rescoring also lays behind CTC.
It demonstrates the importance of exploring non-monotonic matching on latent alignment models, which is more desirable for its ﬂexibility of generating predictions with variable lengths.
As NMLA ﬁnetuning brings remarkable improvements over the strong baseline DDRS, we further conduct experiments to verify whether the NMLA is orthogonal to other methods.
In Table 3, we reimplement GLAT+CTC (Fully-NAT)
[16, 33] based on their open-source code 5 and ﬁnetune it with the NMLA objective.
Our implementation replaces the SoftCopy mechanism with UniformCopy, which surprisingly improves 0.7 BLEU and achieves 27.07 BLEU on WMT14 En-De test set. Finetuning with NMLA can further improve its performance to 27.75 BLEU, demonstrating the extensibility of NMLA. In Table 4, we apply NMLA on the big version of DDRS to explore its capability on large models. The results show that the performance of DDRS-big can be greatly boosted by NMLA ﬁnetuning.
Combined with beam search, it achieves 30.06 BLEU on WMT
En-De, which narrows the performance gap between non-autoregressive and autoregressive models to a new level.
[Table]
[Table]
[Table]
[Table]
4.3 Training Cost
In this section, we will analyze the training cost and show that our method is cost-effective.
The additional training cost of our method only comes from the NMLA ﬁnetuning.
In Table 5, we compare the training cost of CTC and NMLA on WMT14 En-De.
We measure the training speed by Words Per
Second (WPS) and ﬁnd that the training speed of NMLA is slower than CTC, which can be attributed to the large calculation cost of the NMLA objective.
However, since we only ﬁnetune the model for 6K steps, the overall cost of the NMLA ﬁnetuning is much smaller than the CTC pretraining.
On the whole, the training cost of CTC with NMLA is less than 1 . 2 × compared to the CTC baseline.
[Table]
Related Work
[17] ﬁrst proposes non-autoregressive machine translation to accelerate the decoding process of NMT.
The cross-entropy loss is applied to train the Vanilla-NAT, which requires the explicit alignment between the model output and target sentence.
The explicit alignment cannot be guaranteed due to the multi-modality problem, and many efforts have been devoted to mitigating this issue.
One research direction is to enable explicit alignment by leaking part of the target-side information during the training.
[17] uses fertilities as a latent variable to specify the number of output words corresponding to each input word.
[23, 35, 4] apply vector quantization to train NAT with discrete latent variables, and [30, 44, 16] apply variational inference to train NAT with continuous latent variables.
[34, 3, 45] also introduce latent variables that carry the reordering information.
Besides using latent variables, [12] directly feeds the partially masked target sentence as the decoder input, and [33] applies a glancing sampling strategy to control the mask ratio.
Another research direction is to propose better training objectives for NAT.
[48] introduces two auxiliary regularization terms to reduce errors of repeated translations and incomplete translations. [40] directly optimizes sequence-level objectives with reinforcement learning. [21] trains NAT with deep supervision and feeds additional layer-wise predictions.
[43] creates a dataset with multiple references and dynamically selects an appropriate reference for the training.
Prior to this work, [41, 13, 9] have explored ways to establish an alignment between the model output and target sentence, which can promote the training of NAT.
Their limitation is that they are basically built on the framework of Vanilla-NAT, which requires a target length predictor and cannot dynamically adjust the output sequence length.
CTC-based latent alignment models [29, 36, 16] are more desirable due to their superior performance and the ﬂexibility of generating predictions with variable lengths.
However, latent alignment models have a weakness in handling non-monotonic alignments, which is noticed in previous work [36, 16] and solved in this work.
Conclusion
Latent alignment models cannot handle non-monotonic alignments during the training, which is nonnegligible in machine translation.
In this paper, we explore non-monotonic latent alignments for NAT and propose two matching objectives named bipartite matching and n-gram matching.
Experiments show that our method achieves strong improvements on latent alignment models.  Acknowledgement
We thank the anonymous reviewers for their insightful comments.
This work was supported by
National Key R&D Program of China (NO.2018AAA0102502).
A Proof of Equation 13, Equation 15, and Equation
Equation 13 and and Equation 15 are the special cases of Equation 16 when n is 1 or 2, so we directly provide the proof of Equation 16:
[Equation]
We begin with the deﬁnition of C sg ( θ ) , which accumulates the n-gram g from all alignments:
[Equation]
where C g ( β − 1 s ( a )) is the occurrence count of n-gram g in the collapsed sentence β − 1 s ( a ) .
We use i 1: n = { i 1 , ..., i n } to denote the possible position of g = { g 1 , ..., g n } ∈ G n .
There is an n-gram g ∈ G n in the position i 1: n if and only if a i k = g k for all k ∈ { 1 , ..., n } and all other words between them are blank tokens.
By traversing all possible positions, we can obtain the n-gram count as follows:
[Equation]
where 1( · ) is the indicator function that takes value 1 when the inside condition holds, otherwise it takes value 0 .
Recall that A ij is the probability that all positions between i and j are blank tokens, which satisﬁes the following equation:
[Equation]
Using the above equations [Equation] we can rewrite C sg ( θ ) as:
[Equation]
B The O ( nT 2 a ) algorithm for Equation
Here we give the O ( nT 2 a ) algorithm to calculate C sg ( θ ) according to Equation 16.
Notice that
Equation 16 has a similar form with the n-step transition probability of a Markov chain, so we can try a similar method as calculating the transition probability for a Markov chain.
Speciﬁcally, we introduce a set of state vectors s k , where k ∈ { 1 , 2 , ..., T a } and the dimension of each vector s k is
T a .
We deﬁne the state vectors as follows:
[Equation]
From the above equation, the time complexity to calculate s k given s k − 1 is O
( T 2 a ) and can be efﬁciently calculated in matrix form.
Therefore, the time complexity to sequentially calculate s 1: n is
O ( nT 2 a ) .
Finally, we can easily obtain C sg ( θ ) by summing the last vector s n :
C sg ( θ ) =
T a (cid:88) i 1 =
T a (cid:88) i 2 = . . .
T a (cid:88) i n = p i 1 ( g 1 | x, θ ) × n (cid:89) k
=
A
i k − 1 ,i k p i k ( g k | x, θ ) =
T a (cid:88) i n = . . .
T a (cid:88) i 2 =
T a (cid:88) i 1 =
s 1 ( i 1 ) n (cid:89) k
=
A
i k − 1 ,i k p i k ( g k | x, θ ) =
T a (cid:88) i n = . . .
T a (cid:88) i 3 =
T a (cid:88) i 2 =
s 2 ( i 2 ) n (cid:89)
k
=
A
i k − 1 ,i k p i k ( g k | x, θ ) = ......
=
T a (cid:88) i n =
T a (cid:88) i n − 1 =
s n − 1 ( i n − 1 )
× A i n − 1 ,i n p i n ( g n | x, θ ) =
T a (cid:88) i n = s n ( i n ) , g ∈ G n .
(cid:3)
C Proof of Equation
Here we provide the proof of Equation 20:
[Equation]
which is equivalent to proving the following equation:
[Equation]
G 1 .
We begin with the deﬁnition of C g ( θ ) , which accumulates the n-gram g from all alignments:
[Equation]
C g ( β − 1 ( a )) is the occurrence count of n-gram g in the collapsed sentence β − 1 ( a ) .
Besides blank tokens, repeated words will also be removed by β − 1 .
Therefore, we call a word in the alignment a valid word when it does not repeat the word before it.
We can calculate the 1-gram count by collecting all valid words, which will not be removed by the collapsing function:
[Equation]
Using the above equation [Equation] we can rewrite C g ( θ ) as:
[Equation]
D Analysis
To better understand our method, we present a qualitative analysis to provide some insights into the improvements of NMLA.
We analyze the generated outputs of the WMT14 En-De test set in the following experiments.
NMLA Handles Long Sequences Better
We ﬁrst investigate the performance of AT and NAT models for different sequence lengths.
We split the test set into different buckets based on target sequence length and calculate BLEU for each bucket.
We report the results in Table 6.
The ﬁrst observation is that the performance of Vanilla-NAT drops drastically as the sequence length increases.
It is a well-known property of NAT and can be explained as the increased probability of observing misalignment between model output and target as the target sequence becomes longer.
The misalignment in long sequences causes the inaccuracy of loss and therefore harms the translation quality.
We have a similar observation when comparing CTC with NMLA.
The performance of CTC is close to NMLA on short sequences, but the gap becomes larger as the sequence length increases.
Our explanation is that the longer the sequence, the more likely the monotonic assumption fails due to the global word reordering, which causes the inaccuracy of loss and therefore harms the translation quality.
NMLA models non-monotonic alignments and alleviates the inaccuracy of loss on long sequences, so it achieves strong improvements on long sequences and even outperforms AT by more than 2 BLEU, which suffers from the error accumulation on long sequences.
[Table]
NMLA Improves Model Conﬁdence NAT suffers from the multi-modality problem, which makes the model consider many possible translations at the same time and become less conﬁdent in generating outputs.
We investigate whether our method alleviates the multi-modality problem by measuring how conﬁdent the model is during the decoding.
We use the information entropy to measure the conﬁdence, which is calculated as H ( X ) =
− (cid:80) x ∈X p ( x ) log p ( x ) , and lower entropy indicates higher conﬁdence.
We calculate the entropy in every position when decoding the [-WMT
-]WMT
En-De test set and use the average entropy to represent the model conﬁdence.
We do not report the entropy of AT since it varies with the decoding algorithm.
The average entropy scores of NAT models are reported in Table 7.
We can see that the Vanilla-NAT has low prediction conﬁdence, which is substantially improved by CTC.
NMLA further reduces the average entropy to 0 . 061 , showing that the model conﬁdence can be improved by modeling non-monotonic latent alignments.
[Table]
NMLA Improves Generation Fluency Since NAT generates target words independently, generation ﬂuency is a main weakness of NAT.
The multi-modality problem makes NAT consider many possible translations at the same time and generate an inconsistent output, which further reduces the generation ﬂuency.
We investigate whether our method improves the generation ﬂuency with an n-gram language model [20] trained on the target side of WMT14 En-De training data.
We use the language model to calculate the perplexity scores of model outputs and report the results in Table 8.
We can see that Vanilla-NAT and CTC have high perplexity which indicates low ﬂuency.
NMLA substantially reduces the perplexity score, which is only 42.6 higher than the gold reference.
For AT, its perplexity score is even lower than the gold reference, which is consistent with prior works that autoregressive NMT generally improves ﬂuency at the cost of adequacy [8, 10].
[Table]
E Batch Speedup
For CTC-based models including NMLA, a common concern is that they require a larger number of calculations than the autoregressive model due to the longer decoder length.
Though there is a signiﬁcant speedup when decoding sentence by sentence, this advantage may disappear when we use a larger decoding batch.
In response to this concern, we measure the decoding speedup on WMT En-De test set under different batch sizes and report both BLEU and speedup in Figure 3.
Under the memory limit, the maximum batch size is 400 . From Figure 3, NMLA still maintains 2 .
4 × speedup under the maximum batch size while achieving comparable performance to the autoregressive model.
For NMLA+beam&lm, it simultaneously achieves superior translation quality and speed to the autoregressive model, which addresses the concern on batch speedup for CTC-based models.
F Combination of N-grams
In Table 1, we only ﬁnetune the baseline with different n-gram matching objectives respectively.
Here we combine different n-gram matching objectives to see whether the combination can improve the model performance.
First, we linearly combine 1-gram and 2-gram matching with a hyperparameter α :
[Equation]
We train CTC and SCTC baselines with different α and report the results in Table 9, which shows that the combination of 1-gram and 2-gram underperforms simply using 2-gram matching. [Table]
As n > 2 is allowed in SCTC, we further try to combine higher rank n-grams.
We consider n ∈ { 1 , 2 , 3 , 4 } and combine them with arithmetic average and geometric average respectively.
The
BLEU scores of arithmetic average and geometric average are respectively 24 .
94 BLEU and 24 .[-
-]
BLEU on WMT14 En-De validation set, which also underperforms simply using 2-gram matching.
Therefore, we conclude that simply combining different n-gram matching objectives cannot improve the model performance.