U NDERSTANDING AND I MPROVING L EXICAL C HOICE IN N ON -A UTOREGRESSIVE T RANSLATION
A BSTRACT
Knowledge distillation (KD) is essential for training non-autoregressive translation (NAT) models by reducing the complexity of the raw data with an autoregressive teacher model
In this study we empirically show that as a side effect of this training, the lexical choice errors on low-frequency words are propagated to the
NAT model from the teacher model.
To alleviate this problem , we propose to expose the raw data to NAT models to restore the useful information of low-frequency words, which are missed in the distilled data.
To this end, we introduce an extra
Kullback-Leibler divergence term derived by comparing the lexical choice of NAT model and that embedded in the raw data.
Experimental results across language pairs and model architectures demonstrate the effectiveness and universality of the proposed approach.
Extensive analyses conﬁrm our claim that our approach improves performance by reducing the lexical choice errors on low-frequency words
Encouragingly, our approach pushes the SOTA NAT performance on the
WMT14 English German and WMT16 Romanian English datasets up to
8 and 8 BLEU points respectively.
I NTRODUCTION
When translating a word, translation models need to spend a substantial amount of its capacity in disambiguating its sense in the source language and choose a lexeme in the target language which adequately express its meaning (Choi et al., 2017; Tamchyna, 2017).
However, neural machine translation (NMT) has a severe problem on lexical choice, since it usually has mistranslation errors on low-frequency words (Koehn & Knowles, 2017; Nguyen & Chiang, 2018;
Gu et al., 2020).
[Table]
The going at Newmarket is soft ...
[Table]
Today, Newmargot ’s runway is soft ...
[Table]
The Newmarket stakes is always ...
[Table]
The Newmarquette races always ...
[Table]
, 我 ...
[Table]
... in the 3.45 at Newmarket .
[Table]
3:45 a.m. in Newmarquite .
In recent years, there has been a growing interest in non-autoregressive translation (NAT, Gu et al., 2018), which improves decoding efﬁciency by predicting all tokens independently and simultaneously.
Well-performed NAT models are generally trained on synthetic data distilled by autoregressive translation (AT) teachers instead of the raw training data (Figure 1(a))
(Stern et al., 2019; Lee et al., 2018;
Ghazvininejad et al., 2019;
Gu et al., 2019; Hao et al., 2021).
Recent studies have revealed that knowledge distillation (KD) reduces the modes (i.e. multiple lexical choices for a source word) in the raw data by re-weighting the training examples (Furlanello et al., 2018;
Tang et al., 2020), which lowers the intrinsic uncertainty (Ott et al., 2018) and learning difﬁculty for NAT (Zhou et al., 2020;
Ren et al., 2020).
However, the side effect of KD has not been fully studied.
In this work, we investigate this problem from the perspective of lexical choice, which is at the core of machine translation.
We argue that the lexical choice errors of AT teacher can be propagated to the NAT model via the distilled training data.
To verify this hypothesis, we qualitatively compare raw and distilled training corpora.
Table 1 lists all samples whose source sentences contain the place name “ 纽 马 基 特 ”.
In the raw corpus (“RAW-TGT”), this low-frequency word totally occurs three times and corresponds to correct translation “Newmarket”.
However, in the KD corpus (“KD-TGT”), the word is incorrectly translated into a person name “Newmargot” (Margot Robbie is an Australian actress) or organization name “Newmarquette” (Marquette is an university in Wisconsin) or even invalid one “Newmarquite”.
Motivated by this ﬁnding, we explore NAT from the lexical choice perspective .
We ﬁrst validate our hypothesis by analyzing the lexical choice behaviors of NAT models (§3).
Concretely, we propose a new metric AoLC ( accuracy of lexical choice ) to evaluate the lexical translation accuracy of a given NAT model.
Experimental results across different language pairs show that NAT models trained on distilled data have higher accuracy of global lexical translation (AoLC ↑ ), which results in better sequence generation.
However, ﬁne-grained analyses revealed that although KD improves the accuracy on high-frequency tokens, it meanwhile harms performance on low-frequency ones ( Low freq.
AoLC ↓ ).
And with the improvement of teacher models, this issue becomes more severe.
We conclude that the lexical choice of the low-frequency tokens is a typical kind of lost information when using knowledge distillation from AT model.
In order to rejuvenate this lost information in raw data, we propose to expose the raw data to the training of NAT models, which augments NAT models the ability to learn the lost knowledge by themselves.
Speciﬁcally, we propose two bi-lingual lexical-level data-dependent priors ( Word
Alignment Distribution and Self-Distilled Distribution ) extracted from raw data, which is integrated into NAT training via Kullback-Leibler divergence.
Both approaches expose the lexical knowledge in the raw data to NAT, which makes it learn to restore the useful information of low-frequency words to accomplish the translation.
We validated our approach on several datasets that widely used in previous studies (i.e. WMT
En-De, WMT16 Ro-En, WMT17 Zh-En, and WAT17 Ja-En) and model architectures (i.e. MaskPredict (Ghazvininejad et al., 2019) and Levenshtein Transformer (Gu et al., 2019)).
Experimental results show that the proposed method consistently improve translation performance over the standard NAT models across languages and advanced NAT architectures.
The improvements come from the better lexical translation accuracy (low-frequency tokens in particular) of NAT models (AoLC ↑ ), which leads to less mis-translations and low-frequency words prediction errors.
The main contributions of this work are:
• Our study reveals the side effect of NAT models’ knowledge distillation on low-frequency lexicons, which makes the standard NAT training on the distilled data sub-optimal.
• We demonstrate the necessity of letting NAT models learn to distill lexical choices from the raw data by themselves.
• We propose an simple yet effective approach to accomplish this goal 1 , which are robustly applicable to several model architectures and language pairs.
P RELIMINARIES 2. N ON -A UTOREGRESSIVE T RANSLATION
The idea of NAT has been pioneered by Gu et al.
(2018), which enables the inference process goes in parallel.
Different from AT models that generate each target word conditioned on previously generated ones, NAT models break the autoregressive factorization and produce target words in parallel.
Given a source sentence x , the probability of generating its target sentence y with length T is calculated as:
[Equation]
where p L ( · ) is a separate conditional distribution to predict the length of target sequence.
During training, the negative loglikelihood loss function of NAT is accordingly L NAT ( θ )
=
− log p ( y | x ) .
To bridge the performance gap between NAT and AT models, a variety approaches have been proposed, such as multi-turn reﬁnement mechanism (Lee et al., 2018;
Ghazvininejad et al., 2019; Gu et al., 2019; Kasai et al., 2020), rescoring with AT models (Wei et al., 2019; Ma et al., 2019; Sun et al., 2019), adding auxiliary signals to improve model capacity (Wang et al., 2019; Ran et al., 2019; Guo et al., 2019; Ding et al., 2020), and advanced training objective (Wei et al., 2019; Shao et al., 2019; Ma et al., 2020).
Our work is complementary to theirs: while they focus on improving NAT models trained on the distilled data, we reﬁne the NAT models by exploiting the knowledge in the raw data.
Sentence-Level Knowledge Distillation NAT models suffer from the multimodality problem , in which the conditional independence assumption prevents a model from properly capturing the highly multimodal distribution of target translations.
For example, one English source sentence “Thank you.”
can be accurately translated into German as any one of “Danke.”
, “Danke sch¨on.” or “Vielen Dank.”, all of which occur in the training data.
[Figure]
To alleviate this problem, Gu et al.
(2018) applied sequence-level KD (Kim & Rush, 2016) to construct a synthetic corpus, whose target sentences are generated by an AT model trained on the raw data, as shown in
Figure 1(a).
The NAT model is only trained on distilled data with lower modes, which makes it easily acquire more deterministic knowledge (e.g. one lexical choice for each source word).
While separating KD and model training makes the pipeline simple and efﬁcient, it has one potential threat: the re-weighted samples distilled with AT model may have lost some important information.
Lee et al.
(2020) show that distillation beneﬁts the sequence generation but harms the density estimation.
In this study, we exploit to bridge this gap by exposing the raw data to the training of NAT models, as shown in Figure 1(b).
2. E XPERIMENTAL S ETUP
Datasets Experiments were conducted on four widely-used translation datasets: WMT14 EnglishGerman (En-De, Vaswani et al. 2017), WMT16 Romanian-English (Ro-En, Gu et al. 2018), WMT
Chinese-English (Zh-En, Hassan et al. 2018), and WAT17 Japanese-English (Ja-En, Morishita et al.
2017), which consist of 4.5M, 0.6M, 20M, and 2M sentence pairs, respectively.
We use the same validation and test datasets with previous works for fair comparison.
To avoid unknown words, we preprocessed data via BPE (Sennrich et al., 2016) with 32K merge operations.
The GIZA++ (Och & Ney, 2003) was employed to build word alignments for the training datasets.
We evaluated the translation quality with BLEU (Papineni et al., 2002).
NAT Models We validated our research hypotheses on two SOTA NAT models:
• MaskPredict (MaskT, Ghazvininejad et al. 2019) that uses the conditional mask LM (Devlin et al., 2019) to iteratively generate the target sequence from the masked input.
We followed its optimal settings to keep the iteration number be 10 and length beam be 5, respectively.
• Levenshtein Transformer (LevT, Gu et al. 2019) that introduces three steps: deletion, placeholder prediction and token prediction.
The decoding iterations in LevT adaptively depends on certain conditions.
For regularization, we tune the dropout rate from [0.1, 0.2, 0.3] based on validation performance in each direction, and apply weight decay with 0.01 and label smoothing with (cid:15) = 0.1.
We train batches of approximately 128K tokens using Adam (Kingma & Ba, 2015).
The learning rate warms up to 5 × 10 − 4 in the ﬁrst 10K steps, and then decays with the inverse square-root schedule.
We followed the common practices (Ghazvininejad et al., 2019; Kasai et al., 2020) to evaluate the translation performance on an ensemble of top 5 checkpoints to avoid stochasticity. [Table]
AT Teachers We closely followed previous works on NAT to apply sequence-level knowledge distillation (Kim & Rush, 2016) to reduce the modes of the training data.
More precisely, to assess the effectiveness of our method under different of AT teachers, we trained three kinds of
Transformer (Vaswani et al., 2017) models, including Transformer-B ASE , Transformer-B IG and
Transformer-S TRONG .
The main results employ L ARGE for all directions except Ro-En, which is distilled by B ASE .
The architectures of Transformer-B IG and Transformer-S TRONG are unchanged, but S TRONG utilizes a large batch (458K tokens) training strategy.
U NDERSTANDING L EXICAL C HOICE IN NAT M ODELS 3. E VALUATING L EXICAL C HOICE OF NAT M ODELS
Recently, Zhou et al.
(2020) argue that knowledge distillation is necessary for the uncertain nature of the machine translation task.
Accordingly, they propose a metric to estimate the complexity of the data ( CoD ), which is driven from an external word alignment model.
They reveal that the distilled data is indeed less complex, which facilitates easier training for the NAT model.
Inspired by this, we propose a metric to measure the lexical level accuracy of model predictions.
Accuracy of Lexical Choice (AoLC) evaluates the accuracy of target lexicon chosen by a trained
NAT model M for each source word.
Speciﬁcally, the model M takes a source word f as the input, and produce a hypothesis candidate list with their corresponding word conﬁdence:
[Equation]
where V trg is the target side vocabularies over whole corpus.
The AoLC score is calculated by averaging the probability of the gold target word e f of each source word f :
[Equation]
where V testsrc is the set of source side tokens in test set.
Each gold word e f is chosen with the help of the word alignment model P
Af .
The chosen procedure is as follows:
Step 1) collecting the references of the source sentences that contains source word f , and generating the target side word bag B f with these references.
Step 2) Descending P Af in terms of alignment probabilities and looking up the word that ﬁrst appears in B f as the gold word until the B f is traversed.
Step 3)
If the gold word is still not found, let the word with the highest alignment probability in P
Af as the gold word.
Generally, higher accuracy of lexical translation represents more conﬁdent of the predictions.
We discuss the reliability of word alignment-based AoLC in Appendix A.1.
G LOBAL E FFECT OF K NOWLEDGE D ISTILLATION ON L EXICAL C HOICE
In this section, we analyze the lexical choice behaviors of NAT models with our proposed AoLC.
In particular, We evaluated three MaskT models, which are respectively trained on the raw data,
AT-B ASE and AT-B IG distilled data.
We compared the AoLC with other two metrics (i.e. BLEU and
CoD) on three different datasets (i.e. En-De, Zh-En and Ja-En).
As shown in Table 2, KD is able to improve translation quality of NAT models (BLEU: KD(B IG ) > KD(B ASE ) >
Raw) by increasing the lexical choice accuracy of data (AoLC: KD(B IG ) > KD(B ASE ) > Raw).
As expected, NAT models trained on more deterministic data (CoD ↓ ) have lower lexical choice errors (AoLC ↑ ) globally , resulting in better model generation performance (BLEU ↑ ).
3. D ISCREPANCY BETWEEN H IGH - AND L OW -F REQUENCY W ORDS ON L EXICAL C HOICE
To better understand more detailed lexical change within data caused by distillation, we break down the lexicons to three categories in terms of frequency.
And we revisit it from two angles: training data and translated data.
We ﬁrst visualize the changing of training data when adopting KD in terms of words frequency density.
[Figure]
As shown in Figure 2, we ﬁnd that the kurtosis of KD data distribution is higher than that of raw, which becomes more signiﬁcant when adopting stronger teacher.
The side effect is obvious, that is, the original high- / low-frequency words become more / fewer, making the distribution of training data more imbalance and skewed, which is problematic in data mining ﬁeld (Chawla et al., 2004).
This discrepancy may erode the translation performance of low-frequency words and generalization performance on other domains .
Here we focus on lowfrequency words, and generalization performance degradation will be exploited in future work.
In order to understand the detailed change during inference, we then analyze the lexical accuracy with different frequencies in the test set.
We make the comprehensive comparison cross languages based on our proposed AoLC.
As shown in Figure 3, as the teacher model becomes better, i.e. KD(base) → KD(big), the lexical choice of high-frequency words becomes signiﬁcantly more accurate (AoLC ↑ ) while that of low-frequency words becomes worse (AoLC ↓ ).
Through ﬁne-grained analysis, we uncover this interesting discrepancy between high- and low- frequency words.
The same phenomena (lexical choice errors on low-frequency words propagated from teacher model) also can be found in general cases, e.g. distillation when training smaller AT models.
Details can be found in Appendix A.2.
To keep the accuracy of high-frequency words and compensate for the imbalanced low-frequency words caused by KD, we present a simple yet effective approach below.
[Figure]
cc u r ac y ( % )
[Figure]
A cc u r ac y ( % )
[Figure]
I MPROVING L EXICAL C HOICE IN NAT M ODELS 4. M ETHODOLOGY
Our goal is to augment NAT models to learn needed lexical choices from the raw data to achieve better performance.
To this end, we introduce an extra bilingual data-dependent prior objective to augment the current NAT models to distill the required lexical choices from the raw data.
Speciﬁcally, we use
Kullback-Leibler divergence to guide the probability distribution of model predictions
P M ( e | f ) to match the prior probability distributions Q ( · ) :
[Equation]
where f is the source sentence, and e is the target sentence.
The bilingual prior distribution Q ( · ) is derived from the raw data, which is independent of the model M and will be described later.
The ﬁnal objective for training the NAT model becomes:
[Equation]
in which the imitation rate λ follows the logarithmic decay function:
[Equation]
where i is the current step, I is the total training step for distilled data.
Accordingly, the NAT model is merely fed with the priori knowledge derived from the raw data at beginning.
Along with training, the supervision signal of the prior information is getting weaker while that of the distilled data gradually prevails in the training objective.
We run all models for 300K steps to ensure adequate training, thus the bilingual prior distributions will be exposed at the ﬁrst 150K steps.
Choices of Prior Distribution Q ( · ) The goal of the prior objective is to guide the NAT models to learn to distill the lexical choices itself from the raw data.
For each target word e , we use the external word alignment to select the source word f with the maximum alignment probability, and Q ( · ) is rewritten as:
[Equation]
Speciﬁcally, we use two types of bilingual prior distributions:
• Word Alignment Distribution (WAD) is the distribution derived from the external word alignment P Df =
{ P D ( e 1 | f ) , . . .
, P D ( e N | f ) } where { e 1 , . . .
, e N } are the set of target words aligned to the source word in the training data.
We follow Hinton et al.
(2015) to use the softmax temperature mechanism to map P Df over the whole target vocabulary:
[Equation]
We tune the temperature from [ 0.5, 1, 2, 5] on WMT14 En-De dataset and use
τ
= as the default setting for incorporating word alignment distribution in all datasets. • Self-Distilled Distribution (SDD) is the probability distribution for the source word f , which is produced by a same NAT model pre-trained on raw data.
Speciﬁcally, the model M takes a source word f as input and produces a probability distribution over whole words in target vocabulary:
[Equation]
This prior distribution signal can be characterized as self-distilled lexicon level “born-again networks” (Furlanello et al., 2018) or self-knowledge distillation (Liu et al., 2020)
where the teacher and student have the same neural architecture and model size
and yet surprisingly the student is able to surpass the teacher’s accuracy.
4. E XPERIMENTAL R ESULTS
Ablation Study on Raw Data Prior Table 3 shows the results of our proposed two bilingual data dependent prior distributions across language pairs.
The word alignment distribution (WAD) and self-distilled distribution (SDD) variants consistently improves performance over the vanilla two-step training scheme NAT model (“NAT+KD”) when used individually (averagely +0.5 BLEU point), and combining them (“+Both”) by simply averaging the two distributions can achieve a further improvement (averagely +0.9 BLEU point).
The improvements on translation performance are due to a increase of AoLC, especially for low-frequency tokens (averagely +3.2), which reconﬁrms our claim.
Notably, averaging the two prior distributions could rectify each other, thus leading to a further increase.
We explore the complementarity of two prior schemes in Section 4.3.
In the following experiments, we use the combination of WAD and prior. [Table] [Table] al
[Table]
[Table]
Iterative
[Table]
(Lee et al
[Table]
[Table]
DisCo (Kasai et al., 2020)
[Table] Our NAT Models [Table] Comparison with Previous Work Table 4 lists the results of previously competitive studies (Gu et al., 2018; Lee et al., 2018;
Kasai et al., 2020; Ghazvininejad et al., 2019;
Gu et al., 2019) on the widely-used WMT14 En-De and WMT16 Ro-En datasets.
Clearly, our bilingual data-dependent prior signiﬁcantly improves translation (BLEU ↑ ) by substantially increasing the lexical choice accuracy (AoLC ↑ ).
It is worth noting that our approaches merely modify the training process, thus does not increase any latency (“Speed”), maintaining the intrinsic advantages of non-autoregressive generation.
[Table]
KD model
and Ours is our proposed approach.
Comparison with Data Manipulation Strategies
Instead of using the proposed priors, we also investigate two effective data manipulation strategies, i.e.
Data Mixing and Curriculum Learning , to force the
NAT model learns from both the raw and distilled data.
For data mixing, we design two settings: a) Mix: simply combine the raw and distilled data, and then shufﬂe the mixed dataset.
b) Tagged Mix: Inspired by successes of tagged back-translation (Caswell et al., 2019;
Marie et al., 2020), we add tags to distinguish between
KD and Raw sentences in the mixed dataset.
For decay curriculum schedule, the NAT models learn more from raw data at the beginning and then learn more from KD as the training goes on.
The details of curriculum can be found in Appendix A.3.
As seen in Table 5, data mixing and decay curriculum schedule improve performance on both AoLC and BLEU, which conﬁrm the necessity of exposing raw data to NAT models during training.
Besides, our approach still outperforms those effective strategies, demonstrating the superiority of our learning scheme.
[Table]
[Table]
4. E XPERIMENTAL A NALYSIS
[Table]
In this section, we conducted extensive analyses on the lexical choice to better understand our approach.
Unless otherwise stated, results are reported on the
MaskPredict models in Table [Table]
Our approach improves translation performance by reducing mis-translation errors.
The lexical choice ability of NAT models correlates to mistranslation errors, in which wrong lexicons are chosen to translate source words.
To better understand whether our method alleviates the mis-translation problem, we assessed system output by human judgments.
In particular, we randomly selectedsentences from the Zh-En testset
[Table]
and manually labelled the words with lexical choice error.
We deﬁned the lexical choice error rate as E/N , where E is the number of lexical choice errors and N is the number of content words in source sentences, since such errors mainly occur in translating content words.
As seen in Table 6, our approache consistently improves BLEU scores by reducing the lexical choice errors, which conﬁrm our claim.
Additionally, AoLC metric correlates well with both the automatic BLEU score and the subjective evaluation, demonstrating its reasonableness.
Our approach signiﬁcantly improves the accuracy of lexical choice for low-frequency source words.
As aforementioned discrepancy between high- & low-frquency words in Section 3.3, we focus on revealing the ﬁne-grained lexical choice accuracy w.r.t our proposed AoLC.
In Table 7, the majority of improvements is from the low-frequency words, conﬁrming our hypothesis.
Our approach generates translations that contain more low-frequency words.
Besides improving the lexical choice of low-frequency words, our method results in more low-frequency words being recalled in the translation.
In Table 8, although KD improves the translation, it biases the NAT model towards generating high-frequency tokens ( Low freq.
↓ ) while our method can not only correct this bias (averagely +32% relative change), but also enhance translation (BLEU ↑ in Table 3). [Table]
[Table]
Our proposed two priors complement each other by facilitating different tokens.
As aforementioned in Table 3, combining two individual schemes can further increase the NAT performance.
To explain how they complement each other, especially for low-frequency tokens, we classify low-frequency tokens into two categories according to their linguistic roles: content words (e.g. noun, verb, and adjective) and function words (e.g. preposition, determiner, and punctuation).
The results are listed in Table 9.
We show that WAD facilitates more on the understanding and generation of content tokens, while SDD brings more gains for function (i.e. content-free) tokens.
We leave a more thorough exploration of this aspect for future work.
Effect of Word Alignment Quality on Model Performance.
Both the proposed AoLC and priors depend heavily on the quality of word alignment, we therefore design two weaker alignment scenarios to verify the robustness of our method.
First, We adopt fast-align (Dyer et al., 2013), which is slightly weaker than GIZA++.
Using fastalign, our methods can still achieve +0.6 and +0.7 improvements in terms of BLEU on En-De and
Zh-En datasets, which are marginally lower than that using GIZA++ (i.e. +0.8 and +1.0 BLEU).
Encouragingly, we ﬁnd that the improvements in translation accuracy on low-frequency words still hold (+5.5% and +5.3% vs. +5.9% and +5.8%), which demonstrates the robustness of our approach.
In addition, we insert noises into the alignment distributions to deliberately reduce the alignment quality (Noise injection details can be found in Appendix A.4.
The performances still signiﬁcantly outperform the baseline, indicating that our method can tolerate alignment errors and maintain model performance to some extent.
[Table]
Effect of AT Teacher To further dissect the different effects when applying different AT teachers, we employ three teachers.
Tableshows our method can enhance NAT models under variety of teacher-student scenarios, including base, big and strong teacher-guided models.
Our approach obtains averagely +0.7 BLEU points, potentially complementary to the majority of existing work on improving knowledge distillation for NAT models.
R ELATED W ORK
Understanding Knowledge Distillation for NAT Knowledge distillation is a crucial early step in the training of most NAT models.
Ren et al.
(2020) reveal that the difﬁculty of NAT heavily depends on the strongness of dependency among target tokens, and knowledge distillation reduces the token dependency in target sequence and thus improves the accuracy of NAT models.
In the pioneering work of NAT, Gu et al.
(2018) claim that NAT suffers from the multi-modality problem (i.e. multiple lexical translations for a source word), and knowledge distillation can simplify the dataset, which is empirically validated by Zhou et al.
We conﬁrm and extend these results, showing that
the
AT-distilled dataset indeed leads to more deterministic predictions but propagates the low-frequency lexical choices errors.
To this end, we enhance the NAT lexical predictions by making them learn to distill knowledge from the raw data.
Lexical Choice Problem in NMT Models Beneﬁting from continuous representations abstracted from the training data, NMT models have advanced the state of the art in the machine translation community.
However, recent studies have revealed that NMT models suffer from inadequate translation (Tu et al., 2016), in which mis-translation error caused by the lexical choice problem is one main reason.
For AT models, Arthur et al.
(2016) alleviate this issue by integrating a count-based lexicon, and Nguyen & Chiang (2018) propose an additional lexical model, which is jointly trained with the AT model.
The lexical choice problem is more serious for NAT models, since 1) the lexical choice errors (low-resource words in particular) of AT distillation will propagate to NAT models; and 2) NAT lacks target-side dependencies thus misses necessary target-side context.
In this work, we alleviate this problem by solving the ﬁrst challenge.
C ONCLUSION
In this study, we investigated effects of KD on lexical choice in NAT.
We proposed a new metric to evaluate lexical translation accuracy of NAT models, and found that 1) KD improves global lexical predictions; and 2) KD beneﬁts the accuracy of high-frequency words but harms the low-frequency ones.
There exists a discrepancy between high- and low-frequency words after adopting KD.
To bridge this discrepancy, we exposed the useful information in raw data to the training of NAT models.
Experiments show that our approach consistently and signiﬁcantly improves translation performance across language pairs and model architectures.
Extensive analyses reveal that our method reduces mistranslation errors, improves the accuracy of lexical choices for low-frequency source words, recalling more low-frequency words in the translations as well, which conﬁrms our claim.
A CKNOWLEDGMENTS
This work was supported by Australian Research Council Projects under grants FL-170100117,
DP- and IC-190100031.
Xuebo and Derek were supported in part by the Science and Technology Development Fund, Macau SAR (Grant No. 0101/2019/A2) and the Multi-year
Research Grant from the University of Macau (Grant No. MYRG2020-00054-FST).
We also thank the anonymous reviewers for their insightful comments A A PPENDIX A. D ISCUSSION ON THE R ELIABILITY OF W ORD A LIGNMENT -B ASED A O LC
We randomly select 20 sentence pairs from the Zh-En test set, which contains 576 source tokens.
We use the trained word alignment model to produce alignments for the 20 sentence pairs, and then perform the gold word chosen procedure as described in Section 3.1.
We manually evaluate these bilingual lexicons, and ﬁnd that 551 out of 576 source words are aligned to reasonable equivalences (i.e. 96% accuracy).
This demonstrates that it is reliable to calculate AoLC based on automatic word alignments.
A. G ENERAL C ASES OF THE S IDE -E FFECT OF K NOWLEDGE D ISTILLATION
To verify the universality of our ﬁndings that lexical choice error will propagate from teacher model, we conduct the following experiments.
In particular, we experiment AT-Base and AT-Small models on the En-De data, which are distilled by the AT-Strong model.
Note that the AT-Small model consists of 256 model dimensions, 4 heads, 3 encoder and 3 decoder layers.
As shown in Table 11, the same phenomena can be found in AT models when distillation is used.
We leave a thorough exploration of this aspect for future work.
[Table]
A. D ECAY C URRICULUM S ETUP
Speciﬁcally, the training process is divided into 5 phases, which differ at the constituent of training data.
At Phase 1, all training examples are from the raw data; and at Phase 2, 75% of the training examples are from the raw data and the other 25% are from the distilled data (note that the two kinds of training examples should cover all source sentences).
Similarly, the constituent ratios at the later phases are (50%, 50%), (25%, 75%), and (0%, 100%).
A. N OISE I NJECTION S ETUP
We swap the maximal probability tokens with other random tokens under the change ratio of N%.
With 2% and 5% noises, our method respectively decreased by -0.1 and -0.2 BLEU scores on En-De.
The improvements in translation accuracy on low-frequency words are +5.7%
and +5.3%, which is comparable to non-noisy one (i.e. +5.9%).