P SEUDO -L ABEL T RAINING AND M ODEL I NERTIA IN N EURAL M ACHINE T RANSLATION
A BSTRACT
Like many other machine learning applications neural machine translation (NMT) beneﬁts from over-parameterized deep neural models.
However, these models have been observed to be brittle: NMT model predictions are sensi - tive to small input changes and can show signiﬁcant variation across re-training or incremental model updates.
This work studies a frequently used method in
NMT, pseudo-label training (PLT), which is common to the related techniques of forward-translation (or self-training) and sequence-level knowledge distillation
While the effect of PLT on quality is well- documented, we highlight a lesserknown effect: PLT can enhance a model’s stability to model updates and input perturbations, a set of properties we call model inertia .
We study inertia effects under different training settings and we identify distribution simpliﬁcation as a mechanism behind the observed results
I NTRODUCTION
Self-training (Fralick, 1967; Amini et al., 2022) is a popular semi-supervised technique used to boost the performance of neural machine translation (NMT) models.
In self-training for NMT, also known as forward-translation, an initial model is used to translate monolingual data; this data is then concatenated with the original training data in a subsequent training step (Zhang & Zong, 2016;
Marie et al., 2020;
Edunov et al., 2020; Wang et al., 2021).
Self-training is believed to be effective through inducing input smoothness and leading to better learning of decision boundaries from the addition of unlabeled data (Chapelle et al., 2006;
He et al., 2020; Wei et al., 2021).
It has also been observed to effectively diversify the training distribution (Wang et al., 2021; Nguyen et al., 2020).
A closely related technique is that of knowledge distillation (Hinton et al., 2015; Gou et al., 2021), particularly sequence-level knowledge distillation (SKD), which uses hard targets in training and reduces to pseudo-labeled data augmentation (Kim & Rush, 2016).
In NMT, knowledge distillation is effective through knowledge transfer from ensembles or larger-capacity models and as a data augmentation method (Freitag et al., 2017; Gordon & Duh, 2019; Tan et al., 2019; Currey et al., 2020).
In non-autoregressive translation, Zhou et al.
(2020) explored the effect of SKD on training data complexity and showed that simpler training data from distillation is crucial for the performance of non-autoregressive MT models.
This paper examines the component that is common to these techniques, the introduction of pseudolabeled training (PLT) data.
We focus on the more common autoregressive NMT formulation and show that in addition to the known quality gains, PLT has a large impact on model brittleness in that it increases smoothness as well as stability across model re-training.
Our main contributions are:
• We focus on a set of stability properties in NMT models, which we unify under the umbrella term inertia , and show that PLT increases model inertia.
We further show that both the quality gains and the improved inertia are not properties of any one speciﬁc technique such as self-training or knowledge distillation, but are common to the use of pseudo-labeled data in training.
• We investigate the hypothesis that the observed properties correlate with a training data simpliﬁcation mechanism, similarly to the observations made in Zhou et al.
We compare with other popular semi-supervised techniques to investigate if the model quality and inertia properties hold when distribution simpliﬁcation effects are not present.
• Based on our ﬁndings, we recommend incorporating PLT into NMT training whenever inertia (e.g., stability to input perturbations and across incremental model updates) is important, as it increases inertia without sacriﬁcing quality.
R ELATED WORK
Neural network models are known to be sensitive to input variations, i.e., lacking in smoothness . This can make them brittle or open to adversarial attacks, a property observed across many application domains (Goodfellow et al., 2014; Szegedy et al., 2014; Jia & Liang, 2017).
Neural machine translation models are similarly prone to robustness issues and can be affected by both synthetic and natural noise, leading to lower translation quality (Belinkov & Bisk, 2018;
Li et al., 2019; Niu et al., 2020; Fadaee & Monz, 2020).
In MT, earlier works have found noisy data augmentation (Belinkov & Bisk, 2018) and subword regularization (Kudo, 2018;
Provilkov et al., 2020) to be among the most simple yet effective methods for addressing instability to input perturbations.
In addition to smoothness, neural models are known be sensitive to the various sources of randomness in training, such as initialization or dropout (Bengio, 2012; Reimers & Gurevych, 2017;
Madhyastha & Jain, 2019).
This instability negatively impacts end-users in the form of spurious differences in outputs between model updates, or more acutely, as quality regressions on speciﬁc data points, also known as negative ﬂips (Shen et al., 2020; Xie et al., 2021; Yan et al., 2021).
In
NLP, Cai et al.
(2022) focus on a set of structured prediction tasks and show that when random initialization changes, up to 30% of all errors can be regression errors, and that improved accuracy does not always mean reduced regressions.
While negative ﬂips are more difﬁcult to measure in MT as multiple translations can be valid, the lack of consistency across re-training is a known problem: in our experiments ∼ 80% of the translations change due to different model random initialization alone.
Despite this, to the best of our knowledge, minimizing regressions or improving stability across incremental model updates or re-trainings has not yet been addressed in MT.
This paper examines pseudo-label training in NMT and its effect on stability to both input variations and incremental model updates, which we group under the term inertia .
Earlier work on pseudolabel training in MT focused on measuring quality alone and did not shed light on stability-related properties (Wang et al., 2021;
He et al., 2020; Wei et al., 2021; Yuan et al., 2020).
In terms of stability to input variations, or smoothness, our ﬁndings are related to the work of Papernot et al.
(2015), where authors introduce defensive distillation and show that (self-)distillation increased smoothness when tested on digit and object recognition tasks.
They show that the effect is one of reducing the amplitude of the network gradients.
Unlike our work, they do not test pseudo-label training, but soft-target distillation, where a student is trained using the prediction probabilities of a teacher.
Finally, we hypothesize that PLT techniques are able to increase model inertia based on their distribution simpliﬁcation properties.
Earlier works have explored the distribution simpliﬁcation property of PLT methods in terms of model performance.
In non-autoregressive NMT, Zhou et al.
(2020) and
Xu et al.
(2021) explored the effect of SKD on training data complexity and its correlation with model performance.
As in previous work, they hypothesized that SKD alleviates the multiple modes problem, i.e., the existence of multiple alternative translations (Gu et al., 2018). Similarly to Zhou et al.
, we measure training data complexity when adding pseudo-labeled data and use the entropy of a conditional word-level alignment as a complexity metric.
T RAINING WITH PSEUDO - LABELS IN NMT
Neural machine translation (NMT)
We use the autoregressive formulation of NMT, where given parallel data containing source and target sequences, a model θ is learned using the following objective:
[Equation]
where x = [ x 1 , ..., x I ] and y =
[ y 1 , ..., y J ] are the source/target sequences respectively, I and J are the source/target length, and | V | is the size of the vocabulary.
Unless otherwise stated, we use beam search with a ﬁxed number of hypotheses in order to generate a translation from this model.
Pseudo-label training (PLT)
In this paper, we introduce the term pseudo-label training (PLT) to refer to the general technique of adding pseudo-labeled data during training, where the labels are obtained using a previously trained NMT model.
Speciﬁcally, we consider two-step PLT.
In a ﬁrst stage we estimate a teacher model θ ∗ trained with a supervised loss on samples drawn from p , the empirical distribution of the original training data:
[Equation]
In a second step we estimate the ﬁnal student model θ , combining the supervised loss with a PL (pseudo-label) loss L + L PL , where:
[Equation]
In this case the targets y (cid:48) are given by the teacher distribution p θ ∗ and the samples are drawn from a second distribution, p PL , which varies in the experiments below.
Related techniques As discussed earlier, PLT is a common feature of several widely used techniques in NMT such as self-training (a.k.a. forward-translation) and sequence-level knowledge distillation.
This paper opts for the term pseudo-label training (PLT) in order to avoid confusion with additional assumptions made by these techniques.
Speciﬁcally:
• PLT does not necessarily imply semi-supervision, as self-training does.
• PLT is more speciﬁc than KD in that it is restricted to hard labels (as opposed to training on soft targets as in Hinton et al. 2015), but more generic as it does not assume model compression.
Another technique for introducing synthetic data is the use of back-translation (BT), where target segments are translated into source segments (Sennrich et al., 2016a; Hoang et al., 2018;
Edunov et al., 2020).
PLT does not include BT since the latter does not introduce synthetic targets or labels.
Lastly, note that self-training is closely related to entropy minimization (Grandvalet & Bengio, 2004), a semi-supervised technique that encourages high-conﬁdence predictions on unlabeled data.
When reducing this objective to its mode, it becomes identical to L PL above, also observed in He et al.
M ODEL INERTIA
This section introduces a set of desired stability-related MT properties that we group under the term inertia .
All our metrics are closed-box (based on user- observed model behaviour alone) and we investigate two types of model inertia: (1) robustness to input perturbations (or smoothness) and (2) stability across incremental model updates.
4. I NPUT SMOOTHNESS
Robustness to input variations is important in MT models, which have been shown to be negatively affected by misspellings and other small variations in input (Belinkov & Bisk, 2018).
Niu et al.
(2020) introduced metrics that contrast translations of noisy input with those of their clean counterparts in order to disentangle robustness from generic quality changes.
We evaluate model robustness and consistency to input changes following the deﬁnitions introduced in Niu et al.
Robustness measures degradation in translation quality when small variations are present in the input, while Consistency is a reference-free metric for changes in translation output alone.
Speciﬁcally:
[Equation]
where Y ref stands for reference translations, Y, Y (cid:48) are translations of a clean/noisy versions of the test set (e.g., one with introduced misspellings) and H ( · , · ) stands for the harmonic mean.
In this paper, we expand these deﬁnitions to consider robustness not only to synthetic misspellings, but also to natural grammatical errors.
4. S TABILITY TO MODEL UPDATES
Unlike smoothness metrics, stability metrics are functions of two models: an original one (e.g., one that is deployed and available to users) and an update of this model which implements an incremental change.
We denote a model update as a pair ( θ , D , A ) i , ( θ , D , A ) i +1 , where θ are the model parameters obtained when training using data D and algorithm A .
While many incremental updates are possible, in this work we keep the model size and architectures intact and vary the random parameter initialization in training, following Xie et al.
(2021) and Cai et al.
We deﬁne stability as a measure of similarity between model outputs, irrespective of quality changes, while regressions (negative ﬂips) measure output changes that result in lower quality on a given input segment (Cai et al., 2022; Xie et al., 2021;
Shen et al., 2020; Yan et al., 2021).
S TABILITY Stability is measured as string similarity between the different outputs
Y i , Y
i +1 .
We use a symmetric BLEU-based metric, the harmonic mean between BLEU( Y i , Y i +1 ) and
BLEU( Y i +1 , Y i ), where Y i and Y i +1 are translations obtained with models θ i and θ i +1 , respectively.
NFR Similarly to earlier works (Cai et al., 2022; Xie et al., 2021; Yan et al., 2021), we measure regressions as N egative F lip R ate, the number of sentences for which the translation degrades between model updates over the total number of segments.
We consider degradations in terms of both overall quality and a targeted translation error category.
Unlike other tasks, NMT lacks a reliable automatic segment-level quality metric (Kocmi et al., 2021; Mathur et al., 2020); we use human evaluations for this reason.
Having an additional targeted error category allows us to measure segment-level regression automatically.
In this work, we adopt gender translation accuracy as the targeted error category.
NFI Following Cai et al.
, we also measure regressions in terms of N egative F lip I mpact.
NFI is deﬁned as the proportion of negative ﬂips to the total number of errors made by the new model.
Note that in NMT, error is less well-deﬁned for quality since it is not a categorical concept.
This is not the case with targeted translation error categories.
E XPERIMENTS
We perform experiments across 6 language pairs (LPs): English (en) ↔ German (de), Russian (ru), and Japanese (ja).
We adapt the Transformer-base architecture (Vaswani et al., 2017) to 20 encoder layers and 2 decoder layers (denoted 20:2 ) as recommended by Domhan et al.
and
SSRU decoder layers for faster decoding (Kim et al., 2019).
The deep-encoder-shallow-decoder conﬁguration is widely used (Miceli Barone et al., 2017; Kim et al., 2019; Kasai et al., 2021), and the 20: model was found by Domhan et al.
(2020) to yield comparable quality to the 6:6 and 10:10 models while signiﬁcantly decreasing latency.
Unless otherwise noted, we use beam decoding with a beam size of 5 (further details in Appendix A).
Experiments are carried out with the WMT21 dataset (Akhbardeh et al., 2021).
For en ↔ de we use 286M parallel segments, for en ↔ ja
we use 17.2M parallel segments, and for en ↔ ru we use 34M parallel segments.
For development, we use WMT newstest datasets from earlier years (see
Appendix B for more details on datasets used).
We evaluate quality using BLEU 1 (Papineni et al., 2002) and BLEURT (Sellam et al., 2020) on the WMT21 newstest sets (Akhbardeh et al., 2021).
We use only source-original test sets in order to avoid misestimating model performance due to translationese input (Marie et al., 2020).
We train PLT-augmented models using a mix of the original training data and pseudo-labeled data in a joint training setting following Zhang & Zong (2016); Gordon & Duh (2019).
Based on recommendations by He et al.
(2020), we use dropout for all the models, set to 0.1.
We do not tune the trade-off between the two losses L and L PL (we use an equal amount of original and PLT data) or the number of incremental applications of the PLT augmentation.
[Table]
Thousands of people aree guven a drug and thousands of others are given a placebo ..
B ASELINE
Tausende von Menschen erhalten Guven ein Medikament und Tausende von anderen erhalten ein Placebo.
PLT(T RAIN )
Tausende von Menschen erhalten eine Droge und Tausende von anderen erhalten ein
Placebo.
SRC
Can yo put cites on those?
[Table]
K¨onnen
[Table]
[Table]
Kannst
[Table]
5. Q UALITY AND I NERTIA USING PSEUDO - LABELED DATA
This section evaluates PLT for both generic model quality and for inertia.
Unless otherwise noted, student models share the same architecture as the teacher and are trained using the same parallel data with the addition of pseudo-labeled data.
PLT can be implemented by sampling and labeling data from different source distributions p PL : the original training data (as in KD) or unseen monolingual data (i.e. semi-supervised).
This section tests both: to that end, teacher models are trained on half of the available parallel data, while the other half is reserved as a source of unlabeled monolingual data.
Speciﬁcally, we compare:
• B ASELINE :
Model trained on half the available data without any data augmentation.
• PLT(T RAIN ): Data used in PLT augmentation is sampled from the training data.
• PLT(UL): Data used in PLT augmentation is sampled from unused parallel data.
• A LL D ATA :
Finally , to account for the differences in training data size, we also compare against a model trained on all available parallel data without any PLT.
5.1. I NPUT S MOOTHNESS
For each of these models we compute newstest quality (BLEU score) as well as model smoothness (robustness and consistency).
We measure robustness and consistency as deﬁned in Section 4 with the following sources of input variations:
• Synthetic misspellings: We introduce misspellings as proposed by Niu et al.
(2020) into the newstest set.
Each word is misspelled with probability of 0.1, and the strategy is randomly chosen from single-character deletion, insertion, and substitution (Karpukhin et al., 2019).
• GMEG: The GMEG corpus (Napoles et al., 2019) contains data with natural errors made by English language learners (grammatical misuse, misspellings, etc.).
We compute consistency using the noisy input and a reference correction made by a professional annotator.
We report the average consistency over the four provided reference corrections.
Example translations and results are show in Tables 1 and 2, respectively.
Across all LPs, translation quality improves when pseudo-labeled data is used in training, irrespective of the source of the data added.
However, sampling from unseen data does not bring additional improvements over using seen data for PLT.
Similarly, using all parallel data vs. only half is not beneﬁcial across the board, suggesting limitations of the training data w.r.t.
the test domain.
PLT shows signiﬁcantly higher model consistency on both synthetic misspellings and the GMEG test sets.
3 Unlike Niu et al.
(2020), however, we ﬁnd that robustness scores (translation quality changes relative to input changes) are not as well correlated with consistency scores, suggesting that while translations are more stable under noisy conditions they may not necessarily be better.
In the context translation
Table 2. [Table]
[Table] [Table]
identical across the two models.
For Distillation ( Distil. ), the second model is trained to mimic the ﬁrst model.
of semi-supervised learning, it has been hypothesized that self-training has the effect of making models smoother through the addition of new data (He et al., 2020; Wei et al., 2021).
Our results suggest that this is not necessarily the case, as smoothness results are similar irrespective of the use of new unlabeled (monolingual) data (i.e., PLT(T RAIN ) and PLT(UL) have similar smoothness).
5.1. S TABILITY TO M ODEL U PDATES
Next we investigate stability properties with respect to model updates when PLT is used in training.
We ﬁx the source of the pseudo-labeled data to be the training data (i.e., we consider only PLT(T RAIN )) and compare translation changes when re-training a model.
Recall, a model update consists of a pair ( θ , D , A ) 1 , ( θ , D , A ) 2 , where θ are the model parameters obtained when training using data D and algorithm A .
In these experiments, we keep the network architecture identical and hold A 1 =
A 2 , modulo the random seed used in initialization.
We contrast several settings:
• B ASELINE :
Models are trained and re-trained with half the original data ( D 1 = D 2 ), and no pseudo-labeled data is used.
As above
, we also evaluate the case where all of the original data is used (A LL D ATA ).
We vary the random seed, leading to θ 1 (cid:54)
=
θ 2 .
• PLT-
δ
(S TUDENT ):
This tests the hypothesis that using PLT leads to more stable models that behave similarly when varying minor training conditions.
We consider an identical setup as the baseline ( D 1 = D 2 ), except that the data is augmented to contain PLT data • PLT- δ (T EACHER ):
In this setting, the two models θ 1 and θ 2 use PLT data in training; however, two different teachers are used to create it (the teachers are trained with different random seeds).
This simulates a realistic setting where the teachers
used to create pseudo-labels are not likely to
[Table]
stay constant.
Note however that this is not a direct comparison to the baseline and PLT methods: the models do not vary in random seed alone, but also the contents of the training data ( D 1 (cid:54) =
D 2 ).
• D ISTILLATION :
In this setting D 2 is obtained from D 1 using pseudo-labeled data obtained with model θ 1 .
The training data D 1 is re-translated and merged with the original D 1 to create D 2 .
This setting is a standard distillation approach for minimizing regressions, where θ 2 is trained to explicitly mimic θ 1 ’s predictions (Yan et al., 2021;
Cai et al.,
Stability and regression metrics are averaged over ( θ 1 , θ 2 ) and ( θ 2 , θ 1 ) scores since random initialization changes are not directional model updates.
Tables 3 and 4 show stability and regression metrics respectively (regression results are discussed in the next section).
First, we observe that a striking number of translations change when changing random initialization: only 15% of outputs remain identical for en → de, and 8% and 2% remain identical for the lowerresource en → ru and en → ja pairs respectively.
Doubling the amount of training data (A LL D ATA ) improves stability, but not by a large margin.
Across all LPs tested, PLT improves stability relative to the baseline models and nearly doubles the percentage of segments translated identically.
Interestingly, PLT also improves stability relative to the system trained on all available parallel data, once again indicating that inertia effects do not simply stem from more data.
This result is particularly surprising for the PLT- δ (T EACHER ) setting: unlike the baseline, the two models compared are trained on different data on the target side, yet their outputs are more similar to each other than the baseline outputs are to each other.
This suggests that the high translation variability of the original data (a.k.a.
multiple modes in Gu et al., 2018) is an issue with auto-regressive MT as well, and that pseudo-labeled data alleviates it even when created with different models.
Finally, we also ﬁnd that distillation, where a new model is explicitly trained to mimic the previous model, increases stability between teacher and student, conﬁrming earlier observations on text classiﬁcation Cai et al.
However, this improvement is modest in our experiments.
5.1. N EGATIVE F LIPS
Next, we assess PLT in terms of negative ﬂips (NFs) as described in Section 4.
We evaluate regressions in terms of overall quality (human evaluations on the WMT21 newstest set) and on a targeted error category (gender translation accuracy).
For human evaluations, we used two professional annotators who assigned scores on a scale of 1 to 6 with 0.2 increments, where 6 indicates a perfect translation.
A NF is deﬁned as both annotators agreeing that there is a degradation.
Since quality is evaluated on a scale, and not as a binary score, the concept of NFI is ambiguous.
We therefore compute negative ﬂip rate (NFR) alone.
We evaluate on en → de,ja,ru due to availability of annotators.
For gender translation accuracy, which aggregates categorical measurements, we evaluate both NFR and NFI.
We use the WinoMT benchmark (Stanovsky et al., 2019), a gender accuracy benchmark with a reliable segment-level metric suitable for automatic measurements of negative ﬂips.
The dataset consists of English source segments containing a profession whose gender is ambiguous at the lexical level but disambiguated in the sentential context, along with an automatic morphologybased classiﬁer that evaluates the gender accuracy when translated.
We evaluate on the two of our language pairs that are covered by WinoMT, en → de and en → ru.
Results are shown in Table 4.
First, we observe that, like other NLP tasks, regressions are also an issue for NMT: on WMT21, 15%-20% of the test set samples are judged as having degraded in quality according to both human annotators.
NFR is lower for the gender translation accuracy task; however, NFs still amount to 10%-15% of the total number of errors, as measured by NFI. Mirroring
[Table]
stability results, both have signiﬁcantly fewer
In contrast, the effect of distillation is not consistent across the language pairs or the two test sets.
5. T EACHER QUALITY
In previous sections, we found that quality and model inertia improved when using PLT regardless of the source of the data.
In this section, we examine another dimension which distinguishes different ﬂavors of PLT, namely teacher quality.
Stronger teachers (teachers with larger capacity than the student) are more common in KD applications whereas identical teacher and student models are the norm in self-training/forward translation.
Speciﬁcally, we vary the base 20:2 teacher architecture by decreasing the number of decoder layers to 1 (weaker teacher) and increasing it to 4 (stronger teacher).
We keep the student architecture identical at 20:2 layers and ﬁx the source of pseudolabeled data to the training set (referred to as PLT(T RAIN ) in earlier sections).
Interestingly, we ﬁnd that teacher quality does not play a large role in model stability (Table 5).
There are small improvements in stability and robustness when stronger teachers are used, but gains are in range for all teacher models considered, even for weak teachers.
Stronger teachers, however, are responsible for better performing student models.
Most surprisingly, we found quality improvements over the baseline even when the teacher is of worse quality than the baseline model.
This corroborates other work suggesting that the mechanism behind PLT is not simply that of compressing better performing (sets of) models (Furlanello et al., 2018; Hahn & Choi, 2019; Yuan et al., 2020).
D ISTRIBUTION S IMPLIFICATION
The previous section showed that PLT increases both quality and model inertia under different monolingual data and teacher quality settings.
We hypothesize that the increased inertia observed is correlated with a distribution simpliﬁcation mechanism: PLT leads to simpler training data, resulting in models that are less brittle.
We test this by comparing PLT with other techniques used to improve quality and smoothness, but that may not have a distribution simpliﬁcation effect.
Below, we ﬁx the source of pseudo-labeled data to the training data and test:
• BT: Back-translation, a commonly used semi-supervised method that adds parallel data obtained through translation of target data with a reverse-direction MT model.
• BPE- DROPOUT :
A regularization method that has been shown to improve robustness to noise (Provilkov et al., 2020).
We used a dropout rate of 0.1 as recommended by the authors.
• PLT(S AMPLE ):
A variant of PLT where we vary the decoding strategy and perform sampling decoding which leads to more complex data and weaker student models (Zhou et al., 2020).
Specifically, we sampled the top-8 hypotheses.
In previous work, Zhou et al.
(2020) proposed a conditional entropy measure of training data complexity and showed that non-autoregressive translation performance is dependent on simpler training data distributions, such as those obtained with SKD.
Here, we use the same entropy-based measure.
[Table]
(3) compute the entropy of the aligned data, leading to: C ( d ) = − 1 | V x | (cid:80) x ∈ V x
E y | x align log ( y | x ) , where y is the sequence of training data tokens and x align the sequence of source-side tokens that y tokens are aligned to.
Lower entropy indicates that the data is explained by a simpler word-to-word translation model that uses similar word translations irrespective of context.
Results are shown in Table 6.
First, we observe that the complexity scores conﬁrm the results reported by Zhou et al.
(2020), with smaller-scale differences due to the fact that we mix both original data and pseudo-labeled data.
BPE- DROPOUT performs best on smoothness w.r.t.
synthetic noise: it outperforms all methods by a large margin on robustness, and by a smaller margin on consistency.
This is not the case on data with natural noise (GMEG), where the increased consistency effect is smaller w.r.t.
the B ASELINE model.
On other metrics, BPE- DROPOUT has no effect on quality (BLEURT) and a minor negative effect on stability across re-training.
BPE- DROPOUT is not only the only method that lowers stability, but also the only method that increases the complexity of the data compared to the baseline.
BT shows a data simpliﬁcation effect, mirrored by increased stability when re-training.
However,
BT has a detrimental effect on robustness and consistency metrics.
These results indicate that while back-translation and forward translation are typically seen as very similar methods, they have different properties.
PLT(S AMPLE ) performs very similarly to PLT(T RAIN ): when compared with
PLT(T RAIN )
, it leads to slightly more complex data, and slightly worse quality and inertia scores.
PLT(T RAIN ) shows the lowest complexity scores and the highest stability.
While stability and complexity correlate, not all methods that simplify the data improve smoothness; conversely, smoothness to synthetic noise can be improved signiﬁcantly with complementary methods such as BPE- DROPOUT .
We corroborate Niu et al.
and ﬁnd that synthetic and natural noise are different in nature and not all methods are equally effective on both types of noise.
C ONCLUSION
This paper investigates pseudo-label training, a technique common to a number of methods for boosting NMT performance.
We show that in addition to well-studied gains in generic translation quality, pseudo-label training induces several desirable stability-related properties, which we group under the term inertia .
Empirically, these improvements are not tied to the use of unlabeled data (as in self-training) or the use of stronger teacher models (as in knowledge distillation) but are a consequence of the use of pseudo-labeled data itself.
When compared with other methods designed to improve robustness in NMT, we observed that the effect on stability over re-training occurs only for those methods that simplify the training data.
Based on these ﬁndings, we recommend using PLT with unlabeled data ( ` a la self-training) when developing NMT models where inertia is important due to its beneﬁts to model inertia and its use in addressing potential language coverage bias (Wang et al., 2021).
In future work, we plan to investigate the interplay between PLT and different formulations of NMT (auto- vs. non-autoregressive MT) as well as potential negative side effects such as bias ampliﬁcation (Renduchintala et al., 2021).
Finally, developing automatic metrics to detect negative ﬂips in NMT is an important task that has yet to be examined extensively and can help guide PLT techniques. A CKNOWLEDGEMENTS
We thank Yi Zhang and Elman Mansimov for discussions on negative ﬂips and Miguel Ballesteros,
Surafel Lakew, Cuong Hoang, and anonymous reviewers for their comments and suggestions.
A T RAINING P ARAMETERS
All models used in our experiments utilized the following set of hyperparameters.
Training and development data was tokenized using the Sacremoses tokenizer.
4 Words were segmented using
BPE (Sennrich et al., 2016b) with 32K operations.
Source and target subwords shared the same vocabulary.
Training segments longer than 95 tokens were removed.
The source embeddings, target embeddings, and the output layer’s weight matrix are tied (Press &
Wolf, 2017).
Training is done on 8 GPUs with Sockeye 3’s large batch training.
It has an effective batch size of 327,680 tokens, a learning rate of 0.00113 with 2000 warmup steps and a reduce rate of 0.9, a checkpoint interval of 125 steps, and learning rate reduction after 8 checkpoints without improvement.
After an extended plateau of 60 checkpoints, the 8 checkpoints with the lowest validation perplexity are averaged to produce the ﬁnal model parameters.
Parameters for standard training: 'learning_rate_scheduler_type': 'inv-sqrt-decay', 'keep_last_params': 10, 'update_interval': 16, 'transformer_model_size': (512, 512), 'transformer_postprocess': ('dr', 'dr'), 'learning_rate_warmup': 2000, 'transformer_dropout_act': (0.1, 0.1), 'transformer_feed_forward_num_hidden': (2048, 2048), 'max_num_checkpoint_not_improved': 60, 'weight_init_xavier_factor_type': 'avg', 'optimized_metric': 'perplexity', 'cache_strategy': 'best', 'num_layers': (20, 2), 'use_cpu': False, 'checkpoint_improvement_threshold': 0.001, 'device_ids': [-1], 'learning_rate_reduce_num_not_improved': 8, 'initial_learning_rate': 0.06325, 'seed': 1, 'cache_metric': 'perplexity', 'gradient_clipping_type': 'abs', 'cache_last_best_params': 8, 'weight_init_scale': 3.0, 'dtype': 'float32', 'decode_and_evaluate': 500, 'max_seconds': 1036800, 'amp': True, 'keep_initializations': True, 'transformer_dropout_prepost': (0.1, 0.1), 'transformer_attention_heads': (8, 8), 'weight_tying_type': 'src_trg_softmax', 'learning_rate_reduce_factor': 0.9, 'loss': 'cross-entropy', 'horovod': True, 'num_embed': (512, 512), 'embed_dropout': (0.0, 0.0), 'transformer_preprocess': ('n', 'n'), 'encoder': 'transformer', 'loglevel_secondary_workers': 'ERROR', 'label_smoothing': 0.1, 'batch_size': 2500, 'learning_rate_t_scale': 1.0, 'batch_type': 'max-word', 'optimizer': 'adam', '
[Table]
_dropout_attention': (0.1, 0.1), ' [Table] ': 'ssru [Table]
[Table] num [Table] [Table]
interval': 500, '
[Table]
_positional
[Table]
type':
[Table]
fixed',
[Table]
dir': '/
[Table]
[Table]
_clipping_threshold
[Table]
[Table]
[Table]
[Table]
init': 'xavier',
[Table]
sentences
[Table]
multiple
[Table] [Table] B D ATASET We trained enen ↔ de models on Paracrawl v9 (Ba ˜ n ´ on et al., 2020), WikiMatrix (Schwenk et al., 2021), WikiTitles (Bojar et al., 2018), and news commentary datasets (Barrault et al., 2019).
For en ↔ ru we additionally added the UN v1.0 dataset (Ziemski et al., 2016).
For en ↔ ja we used
JParaCrawl (Morishita et al., 2020) instead of ParaCrawl v9 and additionally added the JapaneseEnglish subtitles dataset (Pryzant et al., 2018).
LP
Datasets # parallel
[Table]
WikiMatrix
[Table]
WikiTitles, news commentary
[Table]
34 M
[Table]
C T RAINING C URVES
Here, we compare pseudo-label training with back-translation (BT).
We ﬁnd that pseudo-label training regularizes the models by controlling for over ﬁtting.
BT also regularizes the model, but it does not simplify the distribution to the extent PLT does, implying that controlling over-ﬁtting is not a main factor for stability.
Comparisons with other methods (i.e. BPE-dropout and PLT(sample)) show similar trends.
D BERTS CORE
We provide quality scores using BERTScore (Zhang et al., 2020).
In terms of generic quality, PLT provides improvements in quality consistent with earlier results using BLEU and BLEURT metrics (see Tables 2, 5, and 5).
We also computed robustness metrics BERTScore:
[Equation]
where Y ref stands for reference translations and Y, Y (cid:48) are translations of a clean/noisy versions of the test set (e.g., one with introduced misspellings).
[Table]
[Table]
[Table]
LPs in each direction).
Stability to model updates is computed w.r.t.
to random seed variation in student models.
X → en averages for robustness and consistency to misspellings involve de,ru → en.