F UZZY A LIGNMENTS IN D IRECTED A CYCLIC G RAPH FOR N ON -A UTOREGRESSIVE M ACHINE T RANSLATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Non-autoregressive translation (NAT) reduces the decoding latency but suffers from performance degradation due to the multi-modality problem.
Recently, the structure of Directed Acyclic Graph has achieved great success in NAT, which tackles the multi-modality problem by introducing dependency between vertices.
However, training it with Negative Log Likelihood loss implicitly requires a strict alignment between reference tokens and vertices, weakening its ability to handle multiple translation modalities.
In this paper, we hold the view that all paths in the graph are fuzzily aligned with the reference sentence.
We do not require the exact alignment but train the model to maximize a fuzzy alignment score between the graph and reference, which takes captured translations in all modalities into account.
Extensive experiments on major WMT benchmarks show that our method substantially improves translation performance and increases prediction confidence, setting a new state of the art for NAT on the raw training data.
I NTRODUCTION
Non-autoregressive translation (NAT) (Gu et al., 2018) reduces the decoding latency by generating all target tokens in parallel.
Compared with the autoregressive (AT) counterpart (Vaswani et al., 2017), NAT often suffers from performance degradation due to the severe multi-modality problem (Gu et al., 2018), which refers to the fact that one source sentence may have multiple translations in the target language.
NAT models are usually trained with the cross entropy loss, which strictly aligns model predictions with target tokens.
The strict alignment does not allow multi-modality such as position shifts and word reorderings, so proper translations are likely to be wrongly penalized.
The inaccurate training signal makes NAT tend to generate a mixture of different translations rather than a consistent translation, which typically contains many repeated tokens in generated results.
Many efforts have been devoted to address the above problem (Libovick ´ y & Helcl, 2018;
Shao et al., 2020; Ghazvininejad et al., 2020a; Du et al., 2021; Huang et al., 2022c).
Among them, Directed Acyclic Transformer (DA-Transformer) (Huang et al., 2022c) introduces a Directed Acyclic
Graph (DAG) on top of the NAT decoder, where decoder hidden states are organized as a graph rather than a sequence.
By modeling the dependency between vertices, DAG is able to capture multiple translation modalities simultaneously by assigning tokens in different translations to distinct vertices.
In this way, DA-Transformer does not heavily rely on knowledge distillation (KD) (Kim &
Rush, 2016; Zhou et al., 2020) to reduce data modalities and can achieve superior performance on raw data.
Despite the success of DA-Transformer, training it with Negative Log Likelihood (NLL) loss, which marginalizes all the joint probability of DAG paths and references, is sub-optimal in the scenario of
NAT.
It implicitly introduces a strict monotonic alignment between reference tokens and vertices on all paths.
Although DAG enables the model to capture different translations in different transition paths, only paths that are aligned verbatim with reference with a large probability will be well calibrated by NLL (See Section 2.3 for detailed analysis).
It weakens DAG’s ability to handle data multi-modality, making the model less confident in generating outputs and requiring a large graph size to achieve satisfying performance.
In this paper, we extend the verbatim alignment between reference and DAG path to a fuzzy alignment, aiming to better handle the multi-modality problem.
Specifically, we do not require the exact alignment but hold the view that all paths in DAG are fuzzily aligned with the reference sentence.
To indicate the quality of alignment, an alignment score is assigned to each DAG path based on the expectation of n -gram overlapping.
We further define an alignment score between the whole DAG and reference as the expected alignment score of all its paths.
The model is trained to maximize the alignment score, which takes captured translations in all modalities into account.
Experiments on major WMT benchmarks show that our method substantially improves the translation quality of DA-Transformer.
It achieves comparable performance to the autoregressive Transformer without the help of knowledge distillation and beam search decoding, setting a new state of the art for NAT on the raw training data.
B ACKGROUND 2. N ON - AUTOREGRESSIVE M ACHINE T RANSLATION
Non-autoregressive translation (Gu et al., 2018) is proposed to reduce the decoding latency.
It abandons the assumption of autoregressive dependency between output tokens and generates all tokens simultaneously.
Given a source sentence x = { x 1 , ..., x N } , NAT factorizes the joint probability of target tokens y =
{ y 1 , ..., y M } as,
[Equation]
where θ is the model parameter and P θ ( y i | x ) denotes the translation probability of y i at position i .
In vanilla NAT, the decoder length is set to reference length during the training and determined by a trainable length predictor during the inference.
Standard NAT model is trained with the cross entropy loss, which strictly requires the generation of word y i at position i :
[Equation] [Equation]
D
[Equation]
CYCLIC T RANSFORMER
Vanilla NAT model suffers two major drawbacks, including inflexible length prediction and disability to handle multi-modality.
DA-Transformer (Huang et al., 2022c) addresses these problems by stacking a Directed Acyclic Graph (DAG) on the top of NAT decoder, where hidden states and transitions between states represent vertices and edges in DAG.
Formally, given a bilingual pair x = { x 1 , ..., x N } and y = { y 1 , ..., y M } , DA-Transformer sets the decoder length L = λ · N and models the translation probability by marginalizing over all valid paths in DAG:
[Equation]
where a = { a 1 , ..., a M } is a path represented by a sequence of vertex indexes with the bound 1 = a 1 < ...
< a M = L and Γ y contains all paths with the same length as the target sentence y .
P θ ( a | x ) and P θ ( y | a , x ) indicate the probability of path a
and the probability of target sentence y conditioned on path a respectively.
DAG factorizes the path probability
P θ ( a | x ) based on the
Markov hypothesis:
[Equation]
where E ∈ R L × L is the row-normalized transition matrix.
Due to the unidirectional inherence of
DAG, the lower triangular part of E is masked to zeros.
Once path a is determined, token y i can be generated conditioned on the decoder hidden state with index a i :
[Equation]
DA-Transformer is trained to minimize the Negative Log Likelihood loss via dynamic programming:
[Equation]
The structure of DAG helps NAT model token dependency through transition probability between vertices while almost zero overhead of sequential operation is paid in decoding.
2. M ULTI - MODALITY IN DAG
DA-Transformer alleviates the multi-modality problem by arranging tokens in different translations to distinct vertices, and the learned vertex transition probability prevents them from co-occurring in the same DAG path.
Despite DAG’s ability to capture different translations simultaneously, only paths that are aligned verbatim with reference with a large probability will be well calibrated by the
NLL loss.
It can be demonstrated by inspecting the gradients:
[Equation]
where L a = − log P θ ( y , a | x ) .
Equation 7 shows that NLL loss will assign paths with small posterior probability smaller gradient weights, making paths corresponding to translations in other modalities poorly calibrated.
Considering that only one reference is provided in major translation benchmarks, training DAG with the NLL loss inevitably hurts its potential to handle translation multi-modality.
M ETHODOLOGY
In this section, we extend the verbatim alignment between reference and DAG path to an n -gram based fuzzy alignment.
In Section 3.1 and 3.2, we explore a metric to calibrate DAG under fuzzy alignment.
We develop an efficient algorithm to calculate the alignment score in Section 3.3, and introduce a training strategy with fuzzy alignment in Section 3.4.
3. F UZZY A LIGNMENT
We do not require the exact alignment but hold the view that all paths in DAG are fuzzily aligned with the reference sentence to some extent.
In order to allow multi-modality such as position shifts and word reorderings, we measure the quality of fuzzy alignment by the order-agnostic n -gram overlapping, which is a widely used factor in machine translation evaluation (Papineni et al., 2002).
Given a target sentence y
=
{ y 1 , ..., y M } and an integer n ≥ 1 , we denote the set of all its nonrepeating n -grams as G n ( y ) , the elements of which are mutually exclusive.
For any n -gram g ∈
G n ( y ) , the number of its appearances in y is denoted as C g ( y ) .
We measure the alignment quality by clipped n -gram precision of model output y ′ against reference y :
[Equation]
Considering that the DAG path is composed of consecutive vertices, and each vertex represents a word distribution rather than a concrete word, we assign an alignment score to each DAG path with the expected n -gram precision: [Equation]
[Equation]
3. E STIMATING F UZZY A LIGNMENT OF DAG
Directed Acyclic Graph simultaneously retains multiple translations in different paths, requiring us to measure the alignment between generated graph and reference sentence by the averaged alignment score of all its possible transition paths:
[Equation]
However, the search spaces for translations and paths are both exponentially large, making Equation 10 intractable.
Alternatively, we turn to calculate the ratio of the clipped expected count of n -gram matching to expected number of n -grams, which can be considered as an approximation of p n ( θ, y ) :
[Equation]
Though p ′ n ( θ, y ) is much simplified, the direct calculation will still suffer from exponential time complexity due to the intractable expectation terms in it.
In the following section, we will focus on developing an efficient algorithm to calculate E y ′
[ C g ( y ′ )] and E y ′
[ (cid:80) g
[Equation]
y
[Equation]
y
[Equation]
[Equation]
3. E FFICIENT C ALCULATION OF F UZZY A LIGNMENT
Fortunately, the Markov property of DAG makes it possible to simplify the calculation of Equation 11.
We first consider an n -vertex subpath v 1 , ..., v n , which is bounded by 1 ≤ v 1 , ..., v n ≤ L 3 .
An indicator function 1 ( v 1 , ..., v n ∈ a ) is introduced to indicate whether v 1 , ..., v n is a part of path a .
Then we can define the passing probability of a subpath by enumerating all possible paths a :
[Equation]
With the definition of passing probability, we can transform the summation operation from enumerating all translations to enumerating all the n -vertex subpaths, which largely narrows the search space.
We directly give the following theorem and refer readers to Appendix B for detailed derivation:
[Equation]
where P θ ( g i | v i ) denotes the probability that vertex with index v i generates the i -th token in n -gram g .
By narrowing the search space, we can find that those expected counts are actually the sum and weighted sum of passing probabilities of n -vertex subpaths.
It is worth noting that the n -vertex subpath v 1 , ..., v n forms a Markov chain, leading us to factorize the passing probability into the product of transitions.
We reuse the transition matrix E introduced in Section 2.2 for notation, where E v i ,v j is the transition probability from v i to v j .
The passing probability of an n -vertex subpath can be formulated as:
[Equation]
where P ( 1 ( v ) | x ) denotes the passing probability of vertex with index v .
For any 1 ≤ v ≤ L ,
P ( 1 ( v ) | x ) can be calculated efficiently via dynamic programming:
[Equation]
with the boundary condition that the passing probability of the first vertex in the graph is equal to 1.
For convenience, we introduce a passing probability vector p ∈ R L , where p v = P ( 1 ( v ) | x ) .
Equation 15 reminds us that the sum of n -vertex passing probability is actually the sum of ( n − 1) - hop transition probability in a Markov chain with initial probability p and transition matrix E .
On the basis of Equation 13, we have:
[Equation]
By rearranging the order of products, we can handle the weighted sum of passing probabilities in
Equation 14 in a similar way.
We introduce G ∈ R n × L , with the i th row of G representing the token probability of g i at each vertex, i.e., G i,v = P
θ ( g i | v ) .
Then we can obtain:
[Equation]
where ⊙ denotes position-wise product, and G i, : should be broadcast when i > 1 .
We refer readers to Appendix C for detailed derivation.
Based on the discussion above, we finally avoid the exponential time complexity in calculating p ′ n ( θ, y ) .
Instead, it can be implemented with O ( n + L ) parallel operations, making the proposed fuzzy alignment objective practical for the training.
We summarize the calculation process in Algorithm 1.
3. T RAINING S TRATEGY
We have found a way to train DA-Transformer to maximize the fuzzy alignment score p ′ n ( θ, y ) efficiently.
However, it may bias the model to favor shorter translations.
To this end, we adopt the idea of brief penalty (Papineni et al., 2002) for regularization.
Specifically, we define the brief penalty of DAG based on the ratio of reference length to the expected translation length:
[Equation]
where T y denotes the length of y .
As the expected translation length is equal to the expected number of 1 -grams in outputs, it is convenient to calculate E y ′
[ T y ′ ] efficiently with the help of Equation 17.
Finally, we train the model with the loss:
[Equation]
We first pretrain DA-Transformer with the NLL loss to provide a good initialization, and then finetune the model with our fuzzy alignment objective.
E XPERIMENTS 4. E XPERIMENTAL S ETUP
Datasets We conduct experiments on two major benchmarks that are widely used in previous studies: WMT14 English ↔ German (EN ↔ DE, 4M) and WMT17 Chinese ↔ English (ZH ↔ EN, 20M).
For WMT EN ↔ DE, we use newstest2013 as the validation set and newstest2014 as the test set.
For WMT ZH ↔ EN, we use devtest2017 as the validation set and newstest2017 as the test set.
We apply BPE (Sennrich et al., 2016) to learn a joint subword vocabulary for EN ↔ DE and separate vocabularies for ZH ↔ EN on the tokenized data.
For fair comparison, we evaluate our method with
SacreBLEU (Post, 2018) 4 for WMT ZH-EN task and tokenized BLEU
(Papineni et al., 2002) for other benchmarks.
Considering BLEU may be biased, we also measure the models with BERTScore (Zhang et al., 2020).
The decoding speedup is measured with a batch size of 1.
Implementation Details We adopt Transformer- base (Vaswani et al., 2017) as our autoregressive baseline.
For the architecture of our model, we strictly follow the settings of DA-Transformer, where we set the decoder length to 8 times the source length ( λ = 8 ) and use graph positional embeddings as decoder inputs unless otherwise specified.
During both pretraining and finetuning, we set dropout rate to 0.1, weight decay to 0.01 and no label smoothing is applied.
On pretraining, all models are trained for 300k updates with the batch size of 64k tokens.
The learning rate warms up to 5 · 10 −within 10k steps.
In finetuning, we use the batch size of 192k tokens to stabilize the gradients and train models for 5k updates.
The learning rate warms up to 2 · 10 − 4 on WMT EN ↔ DE withinsteps and to 3 · 10 − 4 on WMT ZH ↔ EN within 700 steps.
We evaluate BLEU scores on the validation set and average the best 5 checkpoints for the final model.
We implement our models with opensource toolkit fairseq (Ott et al., 2019).
All the experiments are conducted on GeForce RTX 3090 GPUs.
Glancing Glancing (Qian et al., 2021a) is a promising strategy to alleviate data multimodality while training.
We apply the same glancing strategy as Huang et al.
(2022c), which assigns the decoder vertex inputs with the most probable path: (cid:101) a = arg max a P θ ( a | x , y ) .
In the pretraining, we linearly anneal the unmasking ratio τ from 0 .
5 to 0 .
In the finetuning, we fix τ to 0.1.
Decoding We apply beam search with beam size of 5 for autoregressive baselines and argmax decoding for vanilla NAT to generate translations.
Following Huang et al.
(2022c)
, we find the translation of DAG with Greedy and Lookahead decoding algorithm.
The former only searches paths greedily and then uses argmax decoding on vertices on the path:
[Equation] [Equation]
And the latter jointly searches the transitions and tokens in a greedy way:
[Equation]
Parallel argmax operations can be performed to obtain tokens and transitions for all vertices.
The only needed sequential operation is collecting tokens along the chosen path.
Both decoding methods can realize a comparable speedup with fully NAT models.
4. P RELIMINARY E XPERIMENT : E FFECTS OF n
We first study the effects of n on our fuzzy alignment objective.
In this experiment, we set the decoder length to 4 times the source length ( λ = 4 ) and apply Lookahead decoding to generate outputs.
We compare different settings of n and report BLEU scores and BERTScores on the raw
WMT14 EN-DE dataset in Table 1.
As shown in Table 1, the performance of our fuzzy alignment objective differs as n varies.
We find that using a large n is helpful for DAG to generate better translations.
Specifically, it improves the
[Table] [Table]
DA-Transformer baseline by 0.82 BLEU and 1.11 BERTScore when n = 2 and by 0.65 BLEU and 0.99 BERTScore when n = 3 .
We speculate the reason for performance degradation with n =is that the 1-gram alignment objective thoroughly breaks the order dependency among tokens in a sentence, which will encourage the model to output bag-of-words instead of a fluent sentence.
It is also noteworthy that 2-gram alignment performs better than 3-gram.
We attribute the reason to the
Markov property of DAG, which only models the dependency between adjacent tokens, thus making the 2-gram alignment more appropriate.
In the following experiments, we will apply n = 2 as the default setting and name it as F uzzy A ligned D irected A cyclic T ransformer (FA-DAT).
4. M AIN R ESULTS
We compare our FA-DAT with the AT baseline and previous NAT approaches in Table 2 and 3.
As shown in Table 2, FA-DAT substantially improves the translation quality of the DA-Transformer baseline by a large margin under both Greedy and Lookahead decoding.
Notably, it achieves comparable performance with the autoregressive baseline without knowledge distillation and beam search decoding.
The average gap between AT and NAT model on raw data is further reduced to 0.
BLEU on WMT EN ↔ DE and 0.80 BLEU on WMT ZH ↔ EN, while maintaining 14.0 times decoding speedup.
We observe similar trending in Table 3.
Compared with DA-Transformer, FA-DAT achieves the improvement of 1.09 BERTScore on average under Lookahead decoding, which further demonstrates its effectiveness.
[Table]
[Figure]
4. A NALYSIS AND D ISCUSSION
8 Size Facto λ 25.
B L E U 25.
Effects of λ Ou s DA-T ansfo me
Generation Confidence We note that the performance of Greedy decoding is on par with that of Lookahead decoding in FA-DAT while left far behind in
DA-Transformer in Table 2.
To conduct the analysis, we collect the probabilities of path ( P θ ( a | x ) ) and translation given path ( P θ ( y | a , x ) ) generated by both
Greedy and Lookahead decoding and further calculate the probability of generated translation ( P θ ( y | x ) ) via dynamic programming.
Statistics are shown in Table 4.
We observe that − log P θ ( y | a , x ) is extremely low in FA-DAT, suggesting every vertex in DAG is assigned with a particular token with high confidence, making Greedy and Lookahead decoding comparable.
Additionally, paths searched in FA-DAT has a larger probability and the model is more confident about its output translation.
We owe it to FA-DAT’s ability to well calibrate vertices in different modalities.
It can be verified by checking the distribution of vertices’ passing probabilities and max token probabilities in Figure 1.
Vertices of FA-DAT have passing probabilities either close to 0 or to 1 and token probabilities all close to 1.
It shows fuzzy alignment objective reduces model perplexity and improves generation confidence.
Effects of Graph Size We further study how the size of the graph affects our method.
We vary the graph size λ from 2 to 8 and measure the performance of the proposed FA-DAT and DA-Transformer baseline with BLEU on WMT14 EN-DE.
As shown in Figure 2, FA-DAT consistently outperforms
DA-Transformer by 0.96 BLEU on average across all sizes.
Notably, FA-DAT does not necessarily
[Table]
require a larger graph size to perform competitively.
It makes use of vertices in DAG more efficiently, realizing comparable performance with λ = 3 to DA-Transformer with λ = 8 (26.47 vs 26.55).
We attribute the success to that fuzzy alignment considers all translation modalities during training.
Vertices corresponding to tokens in all modalities are incorporated in the backward flow with non-negligible gradients.
Oppositely, NLL loss only trains paths that are strictly aligned with the reference.
As shown in Figure 1, most vertices’ token probabilities are close to 1 in FA-DAT while are scattered in DA-Transformer.
It suggests all the vertices in DAG are well calibrated with the fuzzy alignment objective.
Since model complexity is quadratic to graph size, massive overhead will be introduced with a larger graph size.
FA-DAT achieves a better trade-off between graph size and translation quality, making the structure of DAG more appealing in practical application.
R ELATED W ORK
Gu et al.
(2018) first proposes non-autoregressive translation to reduce the decoding latency.
Compared with the autoregressive counterpart, NAT suffers from the performance degradation due to the multi-modality problem and heavily relies on knowledge distilled from AT (Kim & Rush, 2016;
Zhou et al., 2020; Tu et al., 2020; Ding et al., 2021; Shao et al., 2022).
The multi-modality problem in training data has been analyzed comprehensively in the view of lexical choice (Ding et al., 2021), syntactic structure (Zhang et al., 2022) and information theory (Huang et al., 2022b).
To mitigate the performance gap, some researchers introduce semi-autoregressive decoding (Wang et al., 2018; Ran et al., 2020;
Ghazvininejad et al., 2020b; Wang et al., 2021) or iterative decoding mechanisms to refine NAT outputs (Lee et al., 2018; Ghazvininejad et al., 2019;
Gu et al., 2019; Kasai et al., 2020; Saharia et al., 2020; Huang et al., 2022d).
However, these approaches inevitably weakens the advantage of fast inference (Kasai et al., 2021).
To this end, some researchers focus on improving the training procedure of fully NAT models.
Qian et al.
(2021a) introduces glancing training to NAT, which helps the model to calibrate outputs by feeding partial targets.
Huang et al.
(2022a) further applies a similar idea to middle layers with deep supervision.
In another direction, researchers are looking for flexible training objectives that alleviate strict position-wise alignment required by the naive NLL loss.
Libovick ´ y & Helcl (2018) proposes latent alignment model with CTC loss (Graves et al., 2006).
Wang et al.
(2019) introduces two auxiliary regularization terms to improve the quality of decoder hidden representations.
Shao et al.
(2020; 2021) introduce sequence-level training objectives with reinforcement learning and bag-of-ngrams difference.
Ghazvininejad et al.
(2020a) trains NAT model with the best monotonic alignment found by dynamic programming and Du et al.
(2021) further extends it to order-agnostic cross entropy loss.
Recently, Huang et al. (2022c) introduces DA-Transformer, which alleviates multi-modality by modeling dependency among vertices in DAG.
Despite the success of DA-Transformer, it still implicitly requires monotonic one-to-one alignment between target tokens and vertices in the training, which weakens its ability to handle multi-modality.
C ONCLUSION
In this paper, we introduce a fuzzy alignment objective between the directed acyclic graph and reference based on n -gram matching.
Our proposed objective can better handle predictions with position shift, word reordering or length variation, which are critical sources of translation multi-modality.
Experiments demonstrate that our method facilitates training of DA-Transformer, achieving comparable performance to autoregressive baseline with fully parallel decoding and sets new state of the art for NAT on the raw training data. A D ERIVATION OF E QUATION We can inspect the NLL gradients from different translation paths in the graph by applying the chain rule: [Equation]
Note that − log P θ ( y , a | x ) is the loss of one specific translation path.
NLL implicitly assigns each path its posterior probability P θ ( a | y , x ) as the weight during training, making paths corresponding to translations in other modalities poorly calibrated.
B P ROOF OF E QUATION 13 AND E QUATION
We denote the length of arbitrary translation y ′
and that of path a as T y ′ and T a respectively.
Note that the averaged length of generated translations should be the same as the averaged length of DAG paths:
[Equation]
Then we have:
[Equation]
We can handle E y ′
[ C g ( y ′ )] in a similar way.
When a particular path a is given, it is convenient to calculate the averaged appearance of n -gram g through sliding a window of size n on token distributions along the path (Shao et al., 2020):
[Equation]
Then we apply the indicator function 1 ( v 1 , ..., v n ∈ a ) to transform the enumerating space from the space of n adjacent vertices in path a to the space of arbitrary n vertices in graph.
[Equation]
We can find E y ′
[ C g ( y ′ )] is actually a weighted sum of passing probabilities of n -vertex subpaths.
C P ROOF OF E QUATION
On the basis of Equation 14 and Equation 15, we first rewrite it as:
[Equation]
We can construct a similar form to Equation 17 by commutating and associating terms in the product:
[Equation]
Equation 29 indicates that the weighted sum of passing probability shares the same form with the sum of ( n − 1) -hop transition probabilities of a Markov chain with unnormalized initial probability and step-variant transition matrix.
The initial probability and the i th step transition probability are given as:
[Equation]
With passing probability vector p and prediction matrix G introduced in Section 3.3, above equations be expressed in matrix form:
[Equation]
31) where G i +1 , : should be broadcast to the size of E .
Then we can obtain the expression of Equationin matrix form based on the sum of ( n − 1) -hop transition probabilities in Markov chain:
[Equation]
Considering G i +1 , : is broadcast across rows, removing parentheses in Equation 32 will not change the results:
[Equation]
D A NALYSIS OF H ANDLING M ULTI - MODALITY
Generation Fluency We argue that FA-DAT improves NAT performance by better capturing multimodality with the fuzzy alignment objective.
It helps the model avoid generating a mixture of different translations.
To demonstrate that, we measure the fluency of outputs from different models.
The perplexity (PPL) score is calculated by a pretrained language model (Ng et al., 2019) 5 with context window size 128.
Lookahead decoding is applied to DA-Transformer and FA-DAT.
[Table]
As shown in Table 5, AT achieves comparable fluency with gold reference while vanilla-NAT suffers a significant drop due to the disability to handle multi-modality.
DA-Transformer improves NAT fluency by a large margin and proposed FA-DAT further reduces the fluency gap between AT and
NAT.
Sequence Length As longer sentences tend to have more complex grammatical structures, multimodality usually occurs in their translation distribution.
Motivated by this, we also investigate model performance for different sequence lengths.
We split the test set of WMT14 EN-DE into different buckets based on reference length and report the translation quality of each bucket in Table 6.
[Table]
We find that FA-DAT improves the translation quality of all length buckets, especially for L ≥ 60 , where the gain is 2.73 BLEU. We argue that both word reordering and position shift issues are more common in long sentences due to the syntactic multi-modalities, where the model trained by NLL loss will get confused. As shown in Table 6, NLL-trained DA-Transformer suffers a steep performance drop by around 4 BLEU when translating the bucket of longest sentences.
In contrast,
FA-DAT shows a much better ability to deal with long sentences, which demonstrates its effectiveness in handling multi-modality.
Though FA-DAT is proposed to deal with the multi-modality problem which is severe on raw training data, we also conduct experiments to evaluate the performance of FA-DAT on distilled data, where the data distribution is much simplified.
Following Gu et al.
(2018), we use Transformer- base as the teacher model to generate the distilled dataset.
We apply Lookahead decoding to both DATransformer and FA-DAT and report the results in Table 7.
We find that FA-DAT consistently outperforms DA-Transformer on both settings of training data.
However, the improvement on the distilled dataset is relatively marginal.
We attribute it to that multi-modality in distilled data is already alleviated due to the simplification by the autoregressive teacher.
Interestingly, we note that FA-DAT performs better on raw training data by a margin of 0.36 BLEU, which shows a distinct feature in contrast to NAT models in the literature.
We argue that traditional sequence-level KD limits the performance of NAT model by imposing an upper [Table] bound (AT’s performance).
It demonstrates that NAT model endowed with the ability to handle data multi-modality can benefit more from training on authentic data, showing the potential of further developing stronger NAT models. F M EASURE T RANSLATION Q UALITY WITH BLEURT
Considering the proposed fuzzy alignment objective is a natural fit with n -gram matching-based evaluation metrics, we also measure the translation quality with other metrics based on pretrained models.
BLEURT (Sellam et al., 2020) is a BERT-based evaluation metric learned on rating data, which has been demonstrated to correlate well with human judgments.
We use the currently recommended checkpoint BLEURT-20 6 (Pu et al., 2021) to generate scores.
The results are reported in
Table 8.
[Table]
As shown in Table
[Table]
FA-DAT improves BLEURT scores on all four translation tasks, which is consistent with the results of BLEU and BERTScore.
Interestingly, we find BLEURT tends to prefer translations generated by Greedy decoding, especially for the generation of DA-Transformer.
We leave the discussion of this finding in future work.
G Q UALITY - LATENCY T RADEOFF UNDER B ATCH D ECODING
Gu & Kong (2021) has pointed out that the speed advantage of non-autoregressive model shrinks when the size of the decoding batch gets large.
As both DA-Transformer and proposed FA-DAT obtain the best translation performance at the cost of a high upsampling ratio ( λ = 8 ), the extra overhead of computation and memory can make the speedup degradation worse.
To have a better understanding of the problem, we compare the speed-quality tradeoffs under different settings of upsampling ratio and decoding batch size.
The results are plotted in Figure 3.
It is consistent with previous findings that the speedup drops under a large batch setting.
Moreover, the degradation becomes even worse with a larger upsampling ratio.
As shown in Figure 3, the relative speedup ratio is reduced to around 2 under decoding batch size 64 with 8 times upsampling model.
We attribute this problem to the intrinsic property of the directed acyclic graph, which requires a large number of redundant vertices to model the translation multimodality.
However, we find proposed FA-DAT is beneficial to relieve the problem.
4× upsampled FA-DAT achieves a better quality than 8× upsampled DA-Transformer by around 0.5 BLEU with significantly shorter decoding latency.
We argue that fuzzy alignment is capable of well-calibrating all the vertices in the graph (as analyzed in Section 4.4), which uses upsampled vertices more efficiently and reduces redundancy.
[Figure]
Speed-quality Tradeoff
[Figure]
H E FFECTS OF P RETRAINING
As discussed in Section 3.4, we initialize FA-DAT with NLL-trained DA-Transformer.
In this section, we further investigate the effects of NLL pretraining.
We train FA-DAT with different settings of the pretraining step.
BLEU scores of FA-DAT and NLL-pretrained models under different settings are plotted in Figure 4.
[Figure]
Effects of NLL Pretraining
[Figure]
We find that fuzzy alignment training can boost the performance of NLL-pretrained model under all circumstances.
Moreover, initializing FA-DAT with a well-NLL-pretrained model can further improve the translation quality.
We argue that n -gram-based fuzzy alignment objective models word reordering at the cost of ignoring higher (i.e., > n ) order dependency.
However, such ignored information is modeled when training with NLL loss due to its verbatim alignment property, which makes NLL pretraining complementary and essential to fuzzy alignment training.