Exploring Non-Monotonic Latent Alignments for Non-Autoregressive Machine Translation
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Non-autoregressive translation (NAT) models are typically trained with the crossentropy loss, which forces the model outputs to be aligned verbatim with thetarget sentence and will highly penalize small shifts in word positions.
Latentalignment models relax the explicit alignment by marginalizing out all monotoniclatent alignments with the CTC loss.
However, they cannot handle non-monotonicalignments, which is non-negligible as there is typically global word reorderingin machine translation.
In this work, we explore non-monotonic latent alignmentsfor NAT.
We extend the alignment space to non-monotonic alignments to allowfor the global word reordering and further consider all alignments that overlapwith the target sentence.
We non-monotonically match the alignments to the targetsentence and train the latent alignment model to maximize the F1-score of nonmonotonic matching.
Extensive experiments on major WMT benchmarks showthat our method substantially improves the translation performance and achievescomparable performance to the autoregressive Transformer with only one-iterationparallel decoding.
Introduction
Non-autoregressive translation (NAT) models achieve signiﬁcant decoding speedup in neural machinetranslation [NMT, 1, 43] by generating target words simultaneously [17].
This advantage usuallycomes at the cost of translation quality due to the mismatch of training objectives.
NAT models aretypically trained with the cross-entropy loss which forces the model outputs to be aligned verbatimwith the target sentence and will highly penalize small shifts in word positions.
The explicit alignmentrequired by the cross-entropy loss cannot be guaranteed due to the multi-modality problem thatthere exist many possible translations for the same sentence [17] making the cross-entropy lossintrinsically mismatched with NAT.
As the cross-entropy loss can not evaluate NAT outputs properly, many efforts have been devoted todesigning better training objectives for NAT [28, 38, 39, 13, 35, 9].
Among them, latent alignmentmodels [28, 35] relax the alignment restriction by marginalizing out all monotonic latent alignmentswith the connectionist temporal classiﬁcation loss [CTC, 15], which receive much attention for theability to generate variable length translation.
Latent alignment models generally make a strong monotonic assumption on the mapping betweenmodel output and the target sentence.
As illustrated in Figure 1a, the monotonic assumption holds inclassic application scenarios of CTC like automatic speech recognition (ASR) as there is a naturalmonotonic mapping between the speech input and ASR target.
However, non-monotonic alignmentsare non-negligible in machine translation as there is typically global word reordering, which is acommon source of the multi-modality problem.
As Figure 1b shows, when the target sentence is“I ate pizza this afternoon” but the model produces a translation with a different but correct word
[Figure]
ordering “this afternoon I ate pizza”, the CTC loss cannot handle this non-monotonic alignmentbetween output and target and wrongly penalizes the model.
In this paper, we propose to model non-monotonic latent alignments for non-autoregressive machinetranslation.
We ﬁrst extend the alignment space from monotonic alignments to non-monotonicalignments to allow for the global word reordering in machine translation.
Without the monotonicstructure, we have to optimize the best alignment found by the Hungarian algorithm [42, 27, 7, 9] sinceit becomes difﬁcult to marginalize out all non-monotonic alignments with dynamic programming.
This difﬁculty can be overcome by not requiring an exact match between alignments and the targetsentence.
In practice, it is not necessary to have the translation include exact words as contained inthe target sentence, but it would be favorable to have a large overlap between them.
Therefore, wefurther extend the alignment space by considering all alignments that overlap with the target sentence.
Speciﬁcally, we are interested in the overlap of n-grams, which is the core of some evaluationmetrics (e.g., BLEU).
We accumulate n-grams from all alignments regardless of their positions andnon-monotonically match them to target n-grams.
The latent alignment model is trained to maximizethe F1-score of n-gram matching, which reﬂects the translation quality to a certain extent [31].
We conduct experiments on major WMT benchmarks for NAT (WMT14 En ↔ De, WMT16 En ↔ Ro),which shows that our method substantially improves the translation performance and achievescomparable performance to autoregressive Transformer with only one-iteration parallel decoding.
Background2.1 Non-Autoregressive Machine Translation
[17] proposes non-autoregressive neural machine translation, which achieves signiﬁcant decodingspeedup by generating target words simultaneously.
NAT breaks the dependency among target tokensand factorizes the joint probability of target words in the following form:
[Equation]
where x is the source sentence belonging to the input space X , y = { y 1 , ..., y T } is the target sentencebelonging to the output space Y , and p t ( y t | x, θ ) indicates the translation probability in position t .
The vanilla-NAT has a length predictor that takes encoder states as input to predict the target length.
During the training, the target length is set to the golden length, and the vanilla-NAT is trained withthe cross-entropy loss, which explicitly aligns model outputs to target words:
[Equation]
2.2 NAT with Latent Alignments
The vanilla-NAT suffers from two major limitations.
The ﬁrst limitation is the explicit alignmentrequired by the cross-entropy loss, which cannot be guaranteed due to the multi-modality problem
CTC: y =
# −1 (') ' =
[Equation] [Equation]
(A,
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
) (A, A, B, C, D)
SCTC:
[Figure]
= #
[Figure]
The collapsing function of SCTC is β − 1 s , which only remove blanks in the alignment a , where the collapsing function of CTC and and therefore leads to the inaccuracy of the loss.
The second limitation is the requirement of targetlength prediction.
The predicted length may not be optimal and cannot be changed dynamically, so itis often required to use multiple length candidates and re-score them to produce the ﬁnal translation.
The two limitations can be addressed by using CTC-based latent alignment models [15, 28, 35],which extend the output space Y with a blank token (cid:15) that means ‘output nothing’.
We deﬁne theextended output space as Y ∗ .
Following prior works [14, 35], we refer to the elements a ∈ Y ∗ asalignments, since the location of the blank tokens determines an alignment between the extendedoutput and target sentence.
Assume the generated alignments have length T a , we deﬁne a functionβ ( y ) which returns a subset of Y ∗ including all possible alignments of length T a for y , whereβ − 1 : Y ∗ (cid:55)→ Y is the collapsing function that ﬁrst collapses all consecutive repeated words ina and then removes all blank tokens to obtain the target sentence.
As illustrated in Figure 2, thealignment a =( (cid:15), A, A, (cid:15), A, B, B, C, (cid:15), D ) is collapsed to the target sentence y
=
( A, A, B, C, D )with a monotonic mapping from target positions to alignment positions.
During the training, latentalignment models marginalize out all alignments with the CTC loss [15]:
[Equation]
where the alignment probability p ( a | x, θ ) is modeled by a non-autoregressive Transformer:
[Equation]
NAT models with latent alignments achieve superior performance by overcoming the two major limitations of NAT [28, 35, 16, 24, 32, 45].
The weakness in handling non-monotonic latent alignmentsis noticed in previous work [35, 16] but remains unsolved.
Approach
In this section, we explore non-monotonic latent alignments for NAT.
We introduce a simpliﬁed CTCloss in section 3.1, which has a simpler structure that helps further analysis and derivation under theregular CTC loss.
We explore non-monotonic alignments under the simpliﬁed CTC loss in section3.2, and utilize the results to derive non-monotonic alignments under the regular CTC loss in section3.3.
Finally, we introduce the training strategy to combine monotonic and non-monotonic latentalignments in section 3.4.
3.1 Simpliﬁed Connectionist Temporal Classiﬁcation
In CTC, the collapsing function β − 1 maps an alignment to a target sentence in two steps: (1) collapsesall consecutive repeated words, (2) removes all blank tokens.
The two operations make the alignmentspace β ( y ) complex.
Therefore, we ﬁrst introduce the simpliﬁed connectionist temporal classiﬁcationloss (SCTC), which has a simpler structure that helps the derivation of non-monotonic alignments,and the results on SCTC are helpful for further analysis under the regular CTC loss.
To simplify the alignment structure, we consider only using one operation in the collapsing function.
One option is only collapsing all consecutive repeated words in the alignment.
The concern is that itwill limit the expressive power of the model.
For example, the target sentence y
=
( A, A, has probability 0 since it has repeated words that cannot be mapped from any alignment.
Therefore,we favor another option which only removes all blank tokens.
In this way, the expressive power staysthe same and the alignment space becomes much simpler, where alignments of y simply contain Ttarget words and T a − T blank tokens.
We denote this simpliﬁed loss as SCTC, and illustrate the difference between CTC and SCTC in
Figure 2.
The alignment space for the target sentence y is deﬁned as β s ( y ) , and the collapsingfunction is deﬁned as β − 1 s .
We can still obtain the translation probability p ( y | x, θ ) with dynamicprogramming.
We deﬁne the forward variable α t ( s ) to be the total probability of y 1: s considering allpossible alignments a 1: t , which can be calculated recursively from α t − 1 ( s ) and α t − 1 ( s − 1) :
[Equation]
Finally, the total translation probability p ( y | x, θ ) is given by the forward variable α T a ( T ) , so we cantrain the model with the cross-entropy loss L sctc ( θ ) =
− log α T a ( T ) .
3.2 Non-Monotonic Alignments under SCTC
3.2.1 Bipartite Matching
In this section, we explore non-monotonic alignments under the SCTC loss, which are helpful forfurther analysis under the regular CTC loss.
We ﬁrst extend the alignment space from monotonicalignments β s ( y ) to non-monotonic alignments γ s ( y ) to allow for the global word reordering inmachine translation, where γ s ( y ) is deﬁned as:
[Equation]
In the above deﬁnition, P ( y ) represents all permutations of y .
For example, P (( A, B ))
={ ( A, B ) , ( B, A ) } .
By enumerating P ( y ) , we consider all possible word reorderings of the target sentence.
Ideally, we want to traverse all alignments in γ s ( y ) to calculate the log-likelihood loss:
[Equation]
However, without the monotonic structure, it becomes difﬁcult to marginalize out all latent alignmentswith dynamic programming.
Alternatively, we can minimize the loss of the best alignment, which isan upperbound of Equation 7:
[Equation]
Following prior work [42, 7, 9], we formulize ﬁnding the best alignment as a maximum bipartitematching problem and solve it with the Hungarian algorithm [27].
Speciﬁcally, we observe that thealignment space γ s ( y ) is simply permutations of T target words and T a − T blank tokens.
Therefore,ﬁnding the best alignment is equivalent to ﬁnding the best bipartite matching between the T a modelpredictions and the T target words plus T a − T blank tokens, where the two sets of nodes are connectedby edges with the prediction log-probability as weights.
3.2.2 N-Gram Matching
Without the monotonic structure desired for dynamic programming, calculating Equation 7 becomesdifﬁcult.
This difﬁculty is also caused by the strict requirement of exact match for alignments, whichmakes it intractable to simplify the summation of probabilities.
In practice, it is not necessary to forcethe exact match between alignments and the target sentence since a large overlap is also favorable.
Therefore, we further extend the alignment space by considering all alignments that overlap with thetarget sentence.
Speciﬁcally, we are interested in the overlap of n-grams, which is the core of someevaluation metrics (e.g., BLEU).
We propose a non-monotonic and non-exclusive n-gram matching objective based on SCTC toencourage the overlap of n-grams.
Following the underlying idea of probabilistic matching [37], weintroduce the probabilistic variant of the n-gram count to make the objective differentiable.
Speciﬁcally, we use C g ( y ) to denote the occurrence count of n-gram g =
( g 1 , ..., g n ) in the targetsentence y , and use C sg ( θ ) to denote the probabilistic count of n-gram g for the model with input x and parameter θ , where the superscript s indicates SCTC.
For simplicity, we omit the source sentencex in C sg ( θ ) and the following notations.
The probabilistic n-gram count is obtained by accumulatingn-grams from all possible target sentences:
[Equation]
The calculation of C sg ( θ ) is the core of this method and will be described in detail later.
We use
M sg ( θ ) to denote the match count of n-gram g between the latent alignment model and the targetsentence y
(we omit y in M sg ( θ ) for simplicity), which is deﬁned as follows:
[Equation]
Our objective is to maximize the n-gram overlap between the target sentence and model output, sowe can train the model to maximize the precision or recall of n-gram matching:
[Equation]
where we use G n to denote the set of all n-grams.
However, since the latent alignment model havethe ability to control the translation length, both precision and recall cannot accurately reﬂect thetranslation quality.
The precision will encourage short translations to reduce the denominator, whilethe recall prefers long translations.
Therefore, we consider both of them and maximize the F1 score:
[Equation]
where we use G n to denote the set of all n-grams.
In the numerator of Equation 12, M sg ( θ ) is non-zeroonly if C g ( y ) is non-zero, so we only need to calculate probabilistic n-gram counts C sg ( θ ) for n-gramsg in the target sentence.
In the denominator, (cid:80) g ∈ G n C g
( y ) equals to the constant T − n +1 .
Therefore,there are only two tasks left: calculating C sg ( θ ) and the summation (cid:80) g ∈ G n C sg ( θ ) .
For the ﬁrst task of calculating C sg ( θ ) , we ﬁrst consider the simple case n =1 .
The main differencebetween 1-gram matching and bipartite matching is that 1-gram matching is non-exclusive, whichallows us to marginalize out all alignments using the property of non-autoregressive generation.
Dueto the space limit, we refer readers to Appendix.
A for the derivation and directly give the probabilistic1-gram count as follows:
[Equation]
where G 1 denotes the set of all 1-grams g = ( g 1 ) .
Both 1-gram matching and bipartite matchingare completely non-monotonic, which aim at generating correct target words regardless of the wordordering.
When the target sentence is “I ate pizza this afternoon”, they will think that “pizza ate Ithis afternoon” is also a good translation.
We consider 2-gram matching to overcome this limitation,which is also non-monotonic but takes target dependency into consideration.
We ﬁrst introduce atransition matrix A of size T a × T a , where A i,j is the probability that all positions between i and jare blank tokens:
[Equation] [Equation]
Then we give the probabilistic 2-gram count for g
= ( g 1 , g 2 ) (see Appendix.
A for derivation):
[Equation]
For a larger n , the probabilistic n-gram count is:
[Equation]
Equation 15 can be efﬁciently calculated in matrix form with O ( T 2 a ) time complexity.
For Equation16, we need O ( nT na ) operations to calculate it by brute force, which is impractical when n is large.
We refer readers to Appendix.
B for the O ( nT 2 a ) algorithm, which is similar to calculating the n-steptransition probability of a Markov chain.
For the second task of efﬁciently calculating the summation of probabilistic n-gram counts 179 (cid:80) g ∈ G n C sg ( θ ) , we also consider the simple case n = 1 ﬁrst, where the summation can be calculated by directly summing all probabilities:
[Equation]
For n > 1 , the search space is too big for the direct summation.
Fortunately, the deﬁnition of C sg ( θ )in Equation 9 allows us to safely extend the result of n = 1 to a larger n since n-grams count in asentence is always n − 1 less than 1-grams count.
[Equation]
In summary, with the above results (Equation 13-18), we are able to efﬁciently calculate the lossdeﬁned in Equation 12 to maximize the F1 score non-monotonic n-gram matching, which reﬂects thetranslation quality to a certain extent.
3.3 Non-Monotonic Alignments under CTC
In this section, we explore non-monotonic latent alignments under the regular CTC loss.
Under
SCTC, we can use both the bipartite matching and n-gram matching to train the latent alignmentmodel.
However, bipartite matching is infeasible under the regular CTC loss.
Recall that the searchspace γ s ( y ) is simply permutations of target words and T a − T blank tokens, which allows us toﬁnd the best alignment with the Hungarian algorithm.
However, as consecutive repeated words willbe collapsed in the regular CTC loss, alignments in γ ( y ) may contain different words, so the bestalignment cannot be found by the Hungarian algorithm.
Fortunately, for n-gram matching, we can utilize previous results of SCTC to help the analysis andderivation under the regular CTC loss.
As the output of the collapsing function β − 1 s ( a ) simplycontains more repeated words than β − 1 ( a ) , we can ﬁrst identify how much inﬂuence these repeatedwords have, and then remove this inﬂuence from the results of SCTC.
We ﬁrst formally deﬁne theprobabilistic n-gram count under the regular CTC:
[Equation]
We deﬁne R g ( θ ) to be the gap bewteen C g ( θ ) and C sg ( θ ) when n = 1 , which represents the numberof repeated words g 1 removed by the collapsing function.
R g ( θ ) can be efﬁciently calculated asfollows (see Appendix.
C for derivation):
[Equation] [Equation]
The above equation enables us to efﬁciently calculate C g ( θ ) and (cid:80) g C g
( θ ) when g ∈ G 1 .
For2-grams g = ( g 1 , g 2 ) , we divide it into two cases g 1 = g 2 and g 1 (cid:54) =
g 2 .
The collapsing of consecutiverepeated words will only remove 2-grams of the ﬁrst case, where each removed word correspondsto a removed 2-gram.
Taking Figure 2 as an example, the output of SCTC β − 1 s ( a ) have two more2-grams { ( A, A ) , ( B, B ) } of the g 1 = g 2 case than the output of CTC β − 1 ( a ) , and they containexactly the same 2-grams of the g 1 (cid:54) =
g 2 case.
Therefore, we can directly reuse previous results of
SCTC and the number of repeated words R g ( θ ) to calculate the probabilistic 2-gram count:
[Equation]
Unfortunately, for n > 2 , there is no such a clear relationship between C g ( θ ) and C sg ( θ ) , where the
CTC output can contain n-grams (for example, ( A, B, C ) in Figure 2) that the SCTC output does not have.
If we directly formulize C g ( θ ) like in Equation 16, it will be more complex and the O ( nT 2 a )algorithm is no longer applicable.
Therefore, we do not use n-gram matching with n > 2 under theregular CTC loss.
Finally, we deﬁne the match count M g ( θ ) similarly as in Equation 10 and train the
CTC-based latent alignment model to maximize the F1 score:
[Equation]
3.4 Training
In the above, we propose non-monotonic training objectives for the latent alignment model.
They needto be combined with monotonic training objectives, otherwise, they may generate translations like“pizza ate I this afternoon”.
Therefore, we ﬁrst pretrain the latent alignment model with monotonictraining objectives like SCTC or CTC.
The model pretrained by SCTC can be ﬁnetuned with thebipartite matching objective (Equation 8) or the n-gram matching objective (Equation 12).
The modelpretrained by CTC can be ﬁnetuned with the n-gram matching objective (Equation 22), and thechoices of n are limited to { 1 , 2 } .
Experiments
4.1 Experimental Setup
Datasets We evaluate our methods on the most widely used public benchmarks in previous NAT studies: WMT14 English ↔ German (En ↔ De, 4.5M sentence pairs) [5] and WMT16 English ↔ Romanian(En ↔ Ro, 0.6M sentence pairs) [6].
For WMT14 En ↔ De, the validation set is newstest2013 and thetest set is newstest2014 .
For WMT16 En ↔ Ro, the validation set is newsdev-2016 and the test setis newstest-2016 .
Following prior works, we use tokenized BLEU [31] to evaluate the translationquality.
Considering that BLEU might be biased as our method optimizes the n-gram overlap, we alsoreport METEOR [2], which is not directly based on n-gram overlap.
We learn a joint BPE model [36]with 32K operations to process the data and share the vocabulary for source and target languages.
Knowledge Distillation Following prior works [17], we apply sequence-level knowledge distillation[25] to reduce the complexity of training data.
We use Transformer-base setting for the teacher modeland train NAT on the distilled data.
Implementation Details We adopt Transformer-base [43] as our autoregressive baseline as well asthe teacher model.
We uniformly copy encoder outputs to construct decoder inputs.
For CTC-based
NAT models, the length for decoder inputs is 3 × as long as the source length.
On WMT14 En ↔ De,we use a dropout rate of 0.2 to train NAT models and use a dropout rate of 0.1 for ﬁnetuning.
On
WMT16 En ↔ Ro, the dropout rate is 0.3 for both the pretraining and ﬁnetuning.
We use the batchsize 64K and train NAT models for 300K steps on WMT14 En ↔ De and 150K steps on {+WMT
+} En ↔ Ro.
During the ﬁnetuning, we train NAT models for 2K steps with the batch size 256K. Allmodels are optimized with Adam [26] with β = (0 . 9 , 0 . 98) and (cid:15) =
10 − 8 .
The learning rate warmsup to 5 · 10 − 4 within 10K steps in the pretraining and warms up to 2 · 10 − 4 within 500 steps in theﬁnetuning.
We use the GeForce RTX 3090 GPU to train models and measure the translation latency.
We implement our models based on the open-source framework of fairseq [MIT License, 30].
Decoding For autoregressive models, we use beam search with beam size 5 for the decoding.
For
Vanilla-NAT, we use argmax decoding to generate the translation.
For CTC-based models, we canalso use argmax decoding to generate the alignment and then collapse it to obtain the translation.
Besides, following [28, 16], we can use beam search decoding combined with a 4-gram languagemodel [20] to ﬁnd the translation that maximizes:
[Equation]
23) where α and β are hyperparameters for language model scores and length bonus.
We use a ﬁxedbeam size 20 with grid-search α , β based on the performance on the validation set.
The beam searchdoes not contain any neural network computations and can be implemented efﬁciently in C++ 1 .
4.2 Main Results
We ﬁrst compare the performance of different non-monotonic training objectives for latent alignmentmodels.
Baseline models include the Transformer, vanilla-NAT, SCTC, and CTC, and we ﬁnetunelatent alignment models with bipartite matching and n-gram matching objectives.
In Table 1, wereport BLEU and METEOR scores along with the speedup on WMT14 En-De test set.
[Table] [Table]
NAT-FT [17] 15.6 × 17.
29.06 AXE [13] – 23.
31.54 GLAT [32] 15.3 × 25.
32.04 Non-CTC
AligNART [41] 13.4 × 26.
33.10 NAT
CMLM [12] – 27.
33.31 LevT [18] 2.
4.0 × 27.
– – 33.26 JM-NAT [19] – 27.
33.72 RewriteNAT [11] 2.
CTC [28] – 16.
24.67 Imputer [35] – 25.
31.70 CTC-based
REDER [45] 15.5 × 26.
33.23 NAT + beam&reranking 5.5 × 27.
34.03 Fully-NAT [16] 16.8 × 27.
Imputer [35] – 28.
Bag-of-ngrams [38] 15.5 × 25.
31.51 + rescore 5 candidates 9.5 × 25.
Our
OAXE [9] 15.3 × 24.
31.37 Implementation + rescore 5 candidates 14.
[Table]
From Table 1, we have the following observations: (1) Non-monotonic training objectives caneffectively improve the performance of latent alignment models, which improves SCTC by 1.
BLEU and 1.48 METEOR and improves CTC by 1.23 BLEU and 1.35 METEOR, illustrating theimportance of non-monotonic alignments.
(2) Despite having the same expressive power, SCTCunderperforms CTC in the task of non-autoregressive translation.
We speculate that it is becauseof the difference in the distribution of the alignment space.
Most target sentences do not containrepeated words, in which case the alignment space of SCTC β − 1 s ( y ) is a subset of β − 1 ( y ) , makingthe ability of SCTC relatively weak.
(3) SCTC achieves the best performance with 2-gram ﬁnetuning,and then the performance decreases with the increase of n due to the sparsity of higher rank n-grams.
It suggests that we do not need a large n and relieves the concern on CTC that we cannot calculaten-gram matching for n > 2 .
The correlation between the reported BLEU and METEOR scores isvery strong, suggesting that BLEU is not biased when evaluating our methods.
Therefore, we cansafely use BLEU to evaluate the translation quality in the following experiments.
We call the CTC+2-gram method NMLA since it represents our best non-monotonic training methodfor latent alignment models.
We report the performance of NMLA comparing with existing approaches in Table 2.
Notably, NMLA achieves comparable performance to autoregressive Transformeron all benchmarks, demonstrating the effectiveness of our method.
With the help of beam search and4-gram language model, NMLA even outperforms other iterative NAT models and the autoregressive
Transformer with only one-iteration parallel decoding.
Compared to our implementations of priorworks that explore non-monotonic matching for NAT [38, 9], we can see that they underperform the
CTC baseline even when rescoring 5 candidates, and their speedup after rescoring also lays behind
CTC.
It demonstrates the importance of exploring non-monotonic matching on latent alignmentmodels, which is more desirable for its ﬂexibility of generating predictions with variable lengths.
Related Work
[17] ﬁrst proposes non-autoregressive machine translation to accelerate the decoding process of NMT.
The cross-entropy loss is applied to train the Vanilla-NAT, which requires the explicit alignmentbetween the model output and target sentence.
The explicit alignment cannot be guaranteed due tothe multi-modality problem, and many efforts have been devoted to mitigating this issue.
One research direction is to enable explicit alignment by leaking part of the target-side informationduring the training.
[17] uses fertilities as a latent variable to specify the number of output wordscorresponding to each input word.
[22, 34, 4] apply vector quantization to train NAT with discretelatent variables, and [29, 40, 16] apply variational inference to train NAT with continuous latentvariables.
[33, 3, 41] also introduce latent variables that carry the reordering information.
Besidesusing latent variables, [12] directly feeds the partially masked target sentence as the decoder input,and [32] applies a glancing sampling strategy to control the mask ratio.
Another research direction is to propose better training objectives for NAT.
[44] introduces twoauxiliary regularization terms to reduce errors of repeated translations and incomplete translations.[21] trains NAT with deep supervision and feed additional layer-wise predictions.
Prior to thiswork, [38, 13, 9] have explored ways to establish an alignment between the model output and targetsentence, which can promote the training of NAT.
Their limitation is that they are basically built onthe framework of Vanilla-NAT, which requires a target length predictor and cannot dynamically adjustthe output sequence length.
CTC-based latent alignment models [28, 35, 16] are more desirabledue to their superior performance and the ﬂexibility of generating predictions with variable lengths.
However, latent alignment models have a weakness in handling non-monotonic alignments, which isnoticed in previous work [35, 16] and solved in this work.
Conclusion
Latent alignment models cannot handle non-monotonic alignments during the training which is nonnegligible in machine translation.
In this paper we explore non-monotonic latent alignments for NAT and propose two matching objectives named bipartite matching and n-gram matching.
Experiments show that our method achieves strong improvements on latent alignment models
A Proof of Equation 13, Equation 15, and Equation 16
Equation 13 and and Equation 15 are the special cases of Equation 16 when n is 1 or 2, so we directlyprovide the proof of Equation 16:
[Equation]
We begin with the deﬁnition of C sg ( θ ) , which accumulates the n-gram g from all alignments:
[Equation]
where C g ( β − 1 s ( a )) is the occurrence count of n-gram g in the collapsed sentence β − 1 s ( a ) .
We usei 1: n = { i 1 , ..., i n } to denote the possible position of g = { g 1 , ..., g n } ∈ G n .
There is an n-gramg ∈ G n in the position i 1: n if and only if a i k = g k for all k ∈ { 1 , ..., n } and all other words betweenthem are blank tokens.
By traversing all possible positions, we can obtain the n-gram count asfollows:
[Equation]
where 1( · ) is the indicator function that takes value 1 when the inside condition holds, otherwise ittakes value 0 .
Recall that A ij is the probability that all positions between i and j are blank tokens,which satisﬁes the following equation:
[Equation]
Using the above equations, we can rewrite C sg ( θ ) as:
[Equation]
B The O ( nT 2 a ) algorithm for Equation
Here we give the O ( nT 2 a ) algorithm to calculate C sg ( θ ) according to Equation 16.
Notice that
Equation 16 has a similar form with the n-step transition probability of a Markov chain, so we can trya similar method as the calculating the transition probability for a Markov chain.
Speciﬁcally, weintroduce a set of state vectors s k , where k ∈ { 1 , 2 , ..., T a } and the dimension of each vector s k is
T a .
We deﬁne the state vectors as follows:
[Equation]
From the above equation, the time complexity to calculate s k given s k − 1 is O
( T 2 a ) and can beefﬁciently calculated in matrix form.
Therefore, the time complexity to sequentially calculate s 1: n is
O ( nT 2 a ) .
Finally, we can easily obtain C sg ( θ ) by summing the last vector s n :
C sg ( θ ) =
T a (cid:88) i 1 =
T a (cid:88) i 2 = . . .
T a (cid:88) i n = p i 1 ( g 1 | x, θ ) × n (cid:89) k
=
A
i k − 1 ,i k p i k ( g k | x, θ ) =
T a (cid:88) i n = . . .
T a (cid:88) i 2 =
T a (cid:88) i 1 =
s 1 ( i 1 ) n (cid:89) k
=
A
i k − 1 ,i k p i k ( g k | x, θ ) =
T a (cid:88) i n = . . .
T a (cid:88) i 3 =
T a (cid:88) i 2 =
s 2 ( i 2 ) n (cid:89)
k
=
A
i k − 1 ,i k p i k ( g k | x, θ ) = ......
=
T a (cid:88) i n =
T a (cid:88) i n − 1 =
s n − 1 ( i n − 1 )
× A i n − 1 ,i n p i n ( g n | x, θ ) =
T a (cid:88) i n = s n ( i n ) , g ∈ G n .
(cid:3)
C Proof of Equation 20
Here we provide the proof of Equation 20:
[Equation]
which is equivalent to proving the following equation:
[Equation]
θ )) , g ∈ G 1 .
We begin with the deﬁnition of C g ( θ ) , which accumulates the n-gram g from all alignments:
[Equation]
C g ( β − 1 ( a )) is the occurrence count of n-gram g in the collapsed sentence β − 1 ( a ) .
Besides blanktokens, repeated words will also be removed by β − 1 .
Therefore, we call a word in the alignment avalid word when it does not repeat the word before it.
We can calculate the 1-gram count by collectingall valid words, which will not be removed by the collapsing function:
[Equation]
Using the above equation, we can rewrite C g ( θ ) as:
[Equation]
D Analysis
To better understand our method, we present a qualitative analysis to provide some insights onwhere modeling non-monotonic latent alignments improves over the CTC baseline.
We analyze thegenerated outputs of the WMT14 En-De test set.
NMLA Handles Long Sequences Better
We ﬁrst investigate the performance of AT and NATmodels for different sequence lengths.
We split the test set into different buckets based on targetsequence length and calculate BLEU for each bucket.
We report the results in Table 3.
Theﬁrst observation is that the performance of Vanilla-NAT drops drastically as the sequence lengthincreases.
It is a well known property of NAT and can be explained as the increasing probability ofobserving misalignment between model output and target as the target sequence becomes longer.
Themisalignment in long sequences causes the inaccuracy of loss and therefore harms the translationquality.
We have a similar observation when comparing CTC with NMLA.
The performance of CTCis close to NMLA on short sequences, but the gap becomes larger as the sequence length increases.
Our explanation is that the longer the sequence, the more likely the monotonic assumption fails dueto the global word reordering, which causes the inaccuracy of loss and therefore harms the translationquality.
NMLA models non-monotonic alignments and alleviates the inaccuracy of loss on longsequences, so it achieves strong improvements on long sequences and even outperforms AT by morethan 2 BLEU, which suffers from the error accumulation on long sequences.
[Table]
NMLA Improves Model Conﬁdence NAT suffers from the multi-modality problem, which makesthe model consider many possible translations at the same time and become less conﬁdent ingenerating outputs.
We investigate whether our method alleviates the multi-modality problem bymeasuring how conﬁdent the model is during the decoding.
We use the information entropy tomeasure the conﬁdence, which is calculated as H ( X ) =
− (cid:80) x ∈X p ( x ) log p ( x ) , and lower entropyindicates higher conﬁdence.
We calculate the entropy in every position when decoding the WMT
En-De test set and use the average entropy to represent the model conﬁdence.
We do not report theentropy of AT since it varies with the decoding algorithm.
The average entropy scores of NAT modelsare reported in Table 4.
We can see that the Vanilla-NAT has low prediction conﬁdence, which issubstantially improved by CTC.
NMLA further reduces the average entropy to 0 . 061 , showing thatthe model conﬁdence can be improved by modeling non-monotonic latent alignments.
[Table]
NMLA Improves Generation Fluency Since NAT generates target words independently, generationﬂuency is a main weakness of NAT.
The multi-modality problem makes NAT consider many possibletranslations at the same time and generate an inconsistent output, which further reduces the generationﬂuency.
We investigate whether our method improves the generation ﬂuency with an n-gram languagemodel [20] trained on the target-side of WMT14 En-De training data.
We use the language model tocalculate the perplexity scores of model outputs and report the results in Table 5.
We can see that
Vanilla-NAT and CTC have high perplexity that indicates low ﬂuency.
NMLA substantially reducesthe perplexity score, which is only 42.6 higher than the gold reference.
For AT, its perplexity score iseven lower than the gold reference, which is consistent with prior works that autoregressive NMTgenerally improves ﬂuency at the cost of adequacy [8, 10].
[Table]
E Training Cost
Our method requires to pretrain the CTC model ﬁrst and then ﬁnetune it with non-monotonic trainingobjectives, so more training time is required.
We may also use a larger batch size for the trainingor ﬁnetuning to improve model performance at the cost of more training time.
In Table 6, we usedifferent batch sizes and report the BLEU scores along with the training time on WMT14 En-De.
Wecan see that NMLA improves the CTC baseline by 1 . 23 BLEU with only 1 .
× training cost, whichcannot be achieved by simply increasing the batch size for the training or ﬁnetuning.
[Table]
F Finetuning Steps
To illustrate the impact of the ﬁnetuning steps, we use the bigram matching objective to ﬁnetune
SCTC and CTC baselines, and draw their BLEU curves on WMT14 En-De validation set in Figure 3.
For both SCTC and CTC, their BLEU scores rise rapidly in the ﬁrst 500 steps, and then the curvesbecome more steady.
After 2000 steps, continuing ﬁnetuning can barely bring a performance gain, soit is cost-effective to ﬁnetune the baseline for only 2000 steps.
G Combination of N-grams
In Table 1, we only ﬁnetune the baseline with different n-gram matching objectives respectively.
Herewe combine diffent n-gram matching objectives to see whether the combination can improve themodel performance.
First, we linearly combine 1-gram and 2-gram matching with a hyperparameterα :
[Equation]
We train CTC and SCTC baselines with different α and report the results in Table 7, which showsthat the combination of 1-gram and 2-gram underperforms simply using 2-gram matching.
As n > 2 is allowed in SCTC, we further try to combine higher rank n-grams.
We considern ∈ { 1 , 2 , 3 , 4 } and combine them with arithmetic average and geometric average respectively.
The
[Table]
BLEU scores of arithmetic average and geometric average are respectively 24 .
94 BLEU and 24 .
BLEU on WMT14 En-De validation set, which also underperforms simply using 2-gram matching.
Therefore, we conclude that simply combining different n-gram matching objectives cannot improvethe model performance.
H Performance on Stronger Baseline
In this section, we ﬁnetune Fully-NAT [16] with the NMLA objective to show whether our method isstill effective on strong baselines.
We reimplement Fully-NAT following their settings and ﬁnetuneit with the NMLA objective.
As Table 8 shows, our method achieves consider improvement on
Fully-NAT, demonstrating the effectiveness of our method on strong baselines.
[Table]