Original: R EVISITING S ELF -T RAINING FOR N EURAL S EQUENCE G ENERATION
Revised: R EVISITING S ELF -T RAINING FOR N EURAL S EQUENCE G ENERATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Self-training is one of the earliest and simplest semi-supervised methods.
Revised: Self-training is one of the earliest and simplest semi-supervised methods.

Original: The key idea is to augment the original labeled dataset with unlabeled data paired with the model’s prediction (i.e. the pseudo-parallel data).
Revised: The key idea is to augment the original labeled dataset with unlabeled data paired with the model’s prediction (i.e. the pseudo-parallel data).

Original: While self-training has been extensively studied on classiﬁcation problems, in complex sequence generation tasks (e.g. machine translation) it is still unclear how self-training works due to the compositionality of the target space.
Revised: While self-training has been extensively studied on classiﬁcation problems, in complex sequence generation tasks (e.g. machine translation) it is still unclear how self-training works due to the compositionality of the target space.

Original: In this work, we ﬁrst empirically show that selftraining is able to decently improve the supervised baseline on neural sequence generation tasks.
Revised: In this work, we ﬁrst empirically show that selftraining is able to decently improve the supervised baseline on neural sequence generation tasks.

Original: Through careful examination of the performance gains, we ﬁnd that the perturbation on the hidden states (i.e. dropout) is critical for self-training to beneﬁt from the pseudo-parallel data, which acts as a regularizer and forces the model to yield close predictions for similar unlabeled inputs.
Revised: Through careful examination of the performance gains, we ﬁnd that the perturbation on the hidden states (i.e. dropout) is critical for self-training to beneﬁt from the pseudo-parallel data, which acts as a regularizer and forces the model to yield close predictions for similar unlabeled inputs.

Original: Such effect helps the model correct some incorrect predictions on unlabeled data.
Revised: Such effect helps the model correct some incorrect predictions on unlabeled data.

Original: To further encourage this mechanism, we propose to inject noise to the input space, resulting in a “noisy” version of self-training.
Revised: To further encourage this mechanism, we propose to inject noise to the input space resulting in a “noisy” version of self-training.

Original: Empirical study on standard machine translation and text summarization benchmarks shows that noisy self-training is able to effectively utilize unlabeled data and improve the performance of the supervised baseline by a large margin.
Revised: Empirical study on standard machine trans lation and text summarization benchmarks shows that noisy self-training is able to effectively utilize unlabeled data and improve the performance of the supervised baseline by a large margin.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks often require large amounts of labeled data to achieve good performance.
Revised: Deep neural networks often require large amounts of labeled data to achieve good performance.

Original: However, acquiring labels is a costly process, which motivates research on methods that can effectively utilize unlabeled data to improve performance.
Revised: However, acquiring labels is a costly process, which motivates research on methods that can effectively utilize unlabeled data to improve performance.

Original: Towards this goal, semi-supervised learning (Chapelle et al., 2009) methods that take advantage of both labeled and unlabeled data are a natural starting point.
Revised: Towards this goal, semi-supervised learning (Chapelle et al., 2009) methods that take advantage of both labeled and unlabeled data are a natural starting point.

Original: In the context of sequence generation problems, semi-supervised approaches have been shown to work well in some cases.
Revised: In the context of sequence generation problems, semi-supervised approaches have been shown to work well in some cases.

Original: For example, back-translation (Sennrich et al., 2015) makes use of the monolingual data on the target side to improve machine translation systems, latent variable models (Kingma et al., 2014) are employed to incorporate unlabeled source data to facilitate sentence compression (Miao & Blunsom, 2016) or
Revised: For example, back-translation (Sennrich et al., 2015) makes use of the monolingual data on the target side to improve machine translation systems, latent variable models (Kingma et al., 2014) are employed to incorporate unlabeled source data to facilitate sentence compression (Miao & Blunsom, 2016) or

Original: code generation (Yin et al., 2018).
Revised: code generation (Yin et al., 2018).

Original: In this work, we revisit a much older and simpler semi-supervised method, self-training (ST, Scudder (1965)), where a base model trained with labeled data acts as a “teacher” to label the unannotated data, which is then used to augment the original small training set.
Revised: In this work, we revisit a much older and simpler semi-supervised method, self-training (ST, Scudder (1965)), where a base model trained with labeled data acts as a “teacher” to label the unannotated data, which is then used to augment the original small training set.

Original: Then, a “student” model is trained with this new training set to yield the ﬁnal model.
Revised: Then, a “student” model is trained with this new training set to yield the ﬁnal model.

Original: Originally designed for classiﬁcation problems, common wisdom suggests that this method may be effective only when a good fraction of the predictions on unlabeled samples are correct, otherwise mistakes are going to be reinforced (Zhu & Goldberg, 2009).
Revised: Originally designed for classiﬁcation problems, common wisdom suggests that this method may be effective only when a good fraction of the predictions on unlabeled samples are correct, otherwise mistakes are going to be reinforced (Zhu & Goldberg, 2009).

Original: In the ﬁeld of natural language processing, some early work have successfully applied self-training to word sense disambiguation (Yarowsky, 1995) and parsing (McClosky et al., 2006; Reichart & Rappoport, 2007; Huang & Harper, 2009).
Revised: In the ﬁeld of natural language processing, some early work have successfully applied self-training to word sense disambiguation (Yarowsky, 1995) and parsing (McClosky et al., 2006; Reichart & Rappoport, 2007; Huang & Harper, 2009).

Original: However, self-training has not been studied extensively when the target output is natural language.
Revised: However, self-training has not been studied extensively when the target output is natural language.

Original: This is partially because in language generation applications (e.g. machine translation) hypotheses are often very far from the ground-truth target, especially in low-resource settings.
Revised: This is partially because in language generation applications (e.g. machine translation) hypotheses are often very far from the ground-truth target, especially in low-resource settings.

Original: It is natural to ask whether self-training can be useful at all in this case.
Revised: It is natural to ask whether self-training can be useful at all in this case.

Original: While Uefﬁng (2006) and Zhang & Zong (2016) explored self-training in statistical and neural machine translation, only relatively limited gains were reported and, to the best of our knowledge, it is still unclear what makes self-training work.
Revised: While Uefﬁng (2006) and Zhang & Zong (2016) explored self-training in statistical and neural machine translation, only relatively limited gains were reported and, to the best of our knowledge, it is still unclear what makes self-training work.

Original: Moreover, Zhang & Zong (2016) did not update the decoder parameters when using pseudo parallel data noting that “ synthetic target parts may negatively inﬂuence the decoder model of NMT ”.
Revised: Moreover, Zhang & Zong (2016) did not update the decoder parameters when using pseudo parallel data noting that “ synthetic target parts may negatively inﬂuence the decoder model of NMT ”.

Original: In this paper, we aim to answer two questions: (1) How does self-training perform in sequence generation tasks like machine translation and text summarization?
Revised: In this paper, we aim to answer two questions: (1) How does self-training perform in sequence generation tasks like machine translation and text summarization?

Original: Are “bad” pseudo targets indeed catastrophic for self-training?
Revised: Are “bad” pseudo targets indeed catastrophic for self-training?

Original: (2) If self-training helps improving the baseline, what contributes to its success?
Revised: (2) If self-training helps improving the baseline, what contributes to its success?

Original: What are the important ingredients to make it work?
Revised: What are the important ingredients to make it work?

Original: Towards this end, we ﬁrst evaluate self-training on a small-scale machine translation task and empirically observe signiﬁcant performance gains over the supervised baseline ( § 3.2), then we perform a comprehensive ablation analysis to understand the key factors that contribute to its success ( § 3.3).
Revised: Towards this end, we ﬁrst evaluate self-training on a small-scale machine translation task and empirically observe signiﬁcant performance gains over the supervised baseline ( § 3.2), then we perform a comprehensive ablation analysis to understand the key factors that contribute to its success ( § 3.3).

Original: We ﬁnd that the decoding method to generate pseudo targets accounts for part of the improvement, but more importantly, the perturbation of hidden states – dropout (Hinton et al., 2012) – turns out to be a crucial ingredient to prevent self-training from falling into the same local optimum as the base model, and this is responsible for most of the gains.
Revised: We ﬁnd that the decoding method to generate pseudo targets accounts for part of the improvement, but more importantly, the perturbation of hidden states – dropout (Hinton et al., 2012) – turns out to be a crucial ingredient to prevent self-training from falling into the same local optimum as the base model, and this is responsible for most of the gains.

Original: To understand the role of such noise in self-training, we use a toy experiment to analyze how noise effectively propagates labels to nearby inputs, sometimes helping correct incorrect predictions ( § 4.1).
Revised: To understand the role of such noise in self-training, we use a toy experiment to analyze how noise effectively propagates labels to nearby inputs, sometimes helping correct incorrect predictions ( § 4.1).

Original: Motivated by this analysis, we propose to inject additional noise by perturbing also the input.
Revised: Motivated by this analysis, we propose to inject additional noise by perturbing also the input.

Original: Comprehensive experiments on machine translation and text summarization tasks demonstrate the effectiveness of noisy self-training.
Revised: Comprehensive experiments on machine translation and text summarization tasks demonstrate the effectiveness of noisy self-training.

Original:  S ELF - TRAINING
Revised:  S ELF - TRAINING

Original: Formally, in conditional sequence generation tasks like machine translation, we have a parallel dataset L = { x i , y i } li =1 and a large unlabeled dataset U = { x j } l + u j = l +1 , where | U | > | L | in most cases.
Revised: Formally, in conditional sequence generation tasks like machine translation, we have a parallel dataset L = { x i , y i } li =1 and a large unlabeled dataset U = { x j } l + u j = l +1 , where | U | > | L | in most cases.

Original: As shown in Algorithm 1, classic self-training starts from a base model trained with parallel data L , and iteratively applies the current model to obtain predictions on unlabeled instances
Revised: As shown in Algorithm 1, classic self-training starts from a base model trained with parallel data L , and iteratively applies the current model to obtain predictions on unlabeled instances

Original: U , then it incorporates a subset of the pseudo parallel data S to update the current model.
Revised: U , then it incorporates a subset of the pseudo parallel data S to update the current model.

Original: There are two key factors: (1) Selection of the subset S .
Revised: There are two key factors: (1) Selection of the subset S .

Original: S is usually selected based on some conﬁdence scores (e.g. log probability)
Revised: S is usually selected based on some conﬁdence scores (e.g. log probability)

Original: (Yarowsky, 1995) but it is also possible for S to be the whole pseudo parallel data (Zhu & Goldberg, 2009).
Revised: (Yarowsky, 1995) but it is also possible for S to be the whole pseudo parallel data (Zhu & Goldberg, 2009).

Original: (2) Combination of real and pseudo parallel data.
Revised: (2) Combination of real and pseudo parallel data.

Original: A new model is often trained on the two datasets jointly as in back-translation, but this introduces an additional hyper-parameter to weigh the importance of the parallel data relative to the pseudo data (Edunov et al., 2018).
Revised: A new model is often trained on the two datasets jointly as in back-translation, but this introduces an additional hyper-parameter to weigh the importance of the parallel data relative to the pseudo data (Edunov et al., 2018).

Original: Another way is to treat them separately – ﬁrst we train the model on pseudo parallel data S , and then ﬁne-tune it on real data L .
Revised: Another way is to treat them separately – ﬁrst we train the model on pseudo parallel data S , and then ﬁne-tune it on real data L .

Original: In our preliminary experiments, we ﬁnd that the separate training strategy with the whole pseudo parallel dataset (i.e.
Revised: In our preliminary experiments, we ﬁnd that the separate training strategy with the whole pseudo parallel dataset (i.e.

Original: S =
Revised: S =

Original: { ( x , f θ ( x )) | x ∈ U } ) produces better or equal performance for neural sequence generation while being simpler.
Revised: { ( x , f θ ( x )) | x ∈ U } ) produces better or equal performance for neural sequence generation while being simpler.

Original: Therefore, in the remainder of this paper we use this simpler setting.
Revised: Therefore, in the remainder of this paper we use this simpler setting.

Original: We include quantitative comparison regarding joint training, separate training, and pseudo-parallel data ﬁltering in Appendix B, where separate training is able to match (or surpass) the performance of joint training.
Revised: We include quantitative comparison regarding joint training, separate training, and pseudo-parallel data ﬁltering in Appendix B, where separate training is able to match (or surpass) the performance of joint training.

Original: In self-training, the unsupervised loss L U from unlabeled instances is deﬁned as:
Revised: In self-training, the unsupervised loss L U from unlabeled instances is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: ) where p ( x ) is the empirical data distribution approximated with samples from S , p θ ( y | x ) is the conditional distribution deﬁned by the model.
Revised: where p ( x ) is the empirical data distribution approximated with samples from S , p θ ( y | x ) is the conditional distribution deﬁned by the model.

Original: θ ∗ is the parameter from the last iteration (initially itis set as the parameter of the supervised baseline), and ﬁxed within the current iteration. Eq.
Revised: θ ∗ is the parameter from the last iteration (initially it

Original: 1 reveals the connection between self-training and entropy regularization (Grandvalet & Bengio, 2005).

Original: [Figure]
Revised: [Figure]

Original: Methods
Revised: Methods

Original: PT
Revised: PT

Original: FT baseline – 15.
Revised: FT baseline – 15.

Original: ST (scratch) 16.
Revised: ST (scratch) 16.

Original: ST (baseline) 16. 
Revised: ST (baseline) 16. is set as the parameter of the supervised baseline), and the iteration. 1 re veals the connection between self-training and entropy regularization (Grandvalet & Bengio, 2005).

Original: In the context of classiﬁcation self-training can be understood from the view of entropy regularization (Lee, 2013) which favors a low-density separation between classes, a commonly assumed prior for semi-supervised learning (Chapelle & Zien, 2005)
Revised: In the context of classiﬁcation, self-training can be understood from the view of entropy regularization (Lee, 2013), which favors a low-density separation between classes, a commonly assumed prior for semi-supervised learning (Chapelle & Zien, 2005).

Original:  A C ASE S TUDY ON M ACHINE T RANSLATION
Revised:  A C ASE S TUDY ON M ACHINE T RANSLATION

Original: To examine the effectiveness of self-training on neural sequence generation we start by analyzing a machine translation task.
Revised: To examine the effectiveness of self-training on neural sequence generation, we start by analyzing a machine translation task.

Original: We then perform ablation analysis to understand the contributing factors of the performance gains.
Revised: We then perform ablation analysis to understand the contributing factors of the performance gains.

Original: 3. S ETUP
Revised: 3. S ETUP

Original: We work with the standard WMT 2014 English-German dataset consisting of about 3.9 million training sentence pairs after ﬁltering long and imbalanced pairs.
Revised: We work with the standard WMT 2014 English-German dataset consisting of about 3.9 million training sentence pairs after ﬁltering long and imbalanced pairs.

Original: Sentences are encoded using 40K byte-pair codes (Sennrich et al., 2016).
Revised: Sentences are encoded using 40K byte-pair codes (Sennrich et al., 2016).

Original: As a preliminary experiment, we randomly sample 100K sentences from the training set to train the model and use the remaining English sentences as the unlabeled monolingual data.
Revised: As a preliminary experiment, we randomly sample 100K sentences from the training set to train the model and use the remaining English sentences as the unlabeled monolingual data.

Original: For convenience, we refer to this dataset as WMT100K. Such synthetic setting allows us to have high-quality unlabeled data to verify the performance of self-training.
Revised: For convenience, we refer to this dataset as WMT100K. Such synthetic setting allows us to have high-quality unlabeled data to verify the performance of self-training.

Original: We train with the Base Transformer architecture (Vaswani et al., 2017) and dropout rate at 0.3.
Revised: We train with the Base Transformer architecture (Vaswani et al., 2017) and dropout rate at 0.3.

Original: Full training and optimization parameters can be found in Appendix A.1.
Revised: Full training and optimization parameters can be found in Appendix A.1.

Original: All experiments throughout this paper including the transformer implementation are based on the fairseq toolkit (Ott et al., 2019), and all results are in terms of case-sensitive tokenized BLEU (Papineni et al., 2002).
Revised: All experiments throughout this paper including the transformer implementation are based on the fairseq toolkit (Ott et al., 2019), and all results are in terms of case-sensitive tokenized BLEU (Papineni et al., 2002).

Original: We use beam search decoding (beam size 5) to create the pseudo targets and to report BLEU on test set.
Revised: We use beam search decoding (beam size 5) to create the pseudo targets and to report BLEU on test set.

Original: 3. O BSERVATIONS
Revised: 3. O BSERVATIONS

Original: In Figure 1, we use green bars to show the result of applying self-training for three iterations.
Revised: In Figure 1, we use green bars to show the result of applying self-training for three iterations.

Original: We include both (1) pseudo-training (PT) : the ﬁrst step of self-training where we train a new model (from scratch) using only the pseudo parallel data generated by the current model, and (2) ﬁnetuning (FT) : the ﬁne-tuned system using real parallel data based on the pretrained model from the
Revised: We include both (1) pseudo-training (PT) : the ﬁrst step of self-training where we train a new model (from scratch) using only the pseudo parallel data generated by the current model, and (2) ﬁnetuning (FT) : the ﬁne-tuned system using real parallel data based on the pretrained model from the

Original: PT step.
Revised: PT step.

Original: Note that in the ﬁne-tuning step the system is re-initialized from scratch.
Revised: Note that in the ﬁne-tuning step the system is re-initialized from scratch.

Original: Surprisingly, we ﬁnd that the pseudo-training step at the ﬁrst iteration is able to improve BLEU even if the model is only trained on its own predictions, and ﬁne-tuning further boosts the performance.
Revised: Surprisingly, we ﬁnd that the pseudo-training step at the ﬁrst iteration is able to improve BLEU even if the model is only trained on its own predictions, and ﬁne-tuning further boosts the performance.

Original: The test BLEU keeps improving over the ﬁrst three iterations, until convergence to outperform the initial baseline by 3 BLEU points.
Revised: The test BLEU keeps improving over the ﬁrst three iterations, until convergence to outperform the initial baseline by 3 BLEU points.

Original: This behaviour is unexpected because no new information seems to be injected during this iterative process – target sentences of the monolingual data are from the base model’s predictions, thus translation errors are likely to remain, if not magniﬁed.

Original: This is different from back-translation where

Original: [Table]
Revised: [Table]

Revised: This behaviour is unexpected because no new information seems to be injected during this iterative process – target sentences of the monolingual data are from the base model’s predictions, thus translation errors are likely to remain, if not magniﬁed.

Original:  new knowledge may originate from an additional backward translation model and real monolingual targets may help the decoder generate more ﬂuent sentences.
Revised: This is different from back-translation where new knowledge may originate from an additional backward translation model and real monolingual targets may help the decoder generate more ﬂuent sentences.

Original: One straightforward hypothesis is that the added pseudo-parallel data might implicitly change the training trajectory towards a (somehow) better local optimum, given that we train a new model from scratch at each iteration.
Revised: One straightforward hypothesis is that the added pseudo-parallel data might implicitly change the training trajectory towards a (somehow) better local optimum, given that we train a new model from scratch at each iteration.

Original: To rule out this hypothesis, we perform an ablation experiment and initialize θ from the last iteration (i.e. θ ∗ ).
Revised: To rule out this hypothesis, we perform an ablation experiment and initialize θ from the last iteration (i.e. θ ∗ ).

Original: Formally, based on Eq.
Revised: Formally, based on Eq.

Original: 1 we have:
Revised: 1 we have:

Original: [Equation]
Revised: [Equation]

Original: because the conditional log likelihood is maximized when p θ ( y | x ) matches the underlying data distribution p θ ∗ ( y | x ) .
Revised: because the conditional log likelihood is maximized when p θ ( y | x ) matches the underlying data distribution [Equation] y | x ) .

Original: Therefore, the parameter θ should not (at least not signiﬁcantly) change if we initialize it with θ ∗ from the last iteration.
Revised: Therefore, the parameter θ should not (at least not signiﬁcantly) change if we initialize it with θ ∗ from the last iteration.

Original: Table 1 shows the comparison results of these two initialization schemes at the ﬁrst iteration.
Revised: Table 1 shows the comparison results of these two initialization schemes at the ﬁrst iteration.

Original: Surprisingly, continuing training from the baseline model also yields an improvement of 1.9 BLEU points, comparable to initializing from random.
Revised: Surprisingly, continuing training from the baseline model also yields an improvement of 1.9 BLEU points, comparable to initializing from random.

Original: While stochastic optimization introduces randomness in the training process, it is startling that continuing training gives such a non-trivial improvement.
Revised: While stochastic optimization introduces randomness in the training process, it is startling that continuing training gives such a non-trivial improvement.

Original: Next, we investigate the underlying reasons for this.
Revised: Next, we investigate the underlying reasons for this.

Original: 3. T HE S ECRET B EHIND S ELF -T RAINING
Revised: 3. T HE S ECRET B EHIND S ELF -T RAINING

Original: To understand why continuing training contradicts Eq. 2 and improves translation performance, we examine possible discrepancies between our assumptions and the actual implementation, and formulate two new hypotheses:
Revised: To understand why continuing training contradicts Eq. 2 and improves translation performance, we examine possible discrepancies between our assumptions and the actual implementation, and formulate two new hypotheses:

Original: H1.
Revised: H1.

Original: Decoding Strategy.
Revised: Decoding Strategy.

Original: According to this hypothesis, the gains come from the use of beam search for decoding unlabeled data.
Revised: According to this hypothesis, the gains come from the use of beam search for decoding unlabeled data.

Original: Since our focus is a sequence generation task, we decode y with beam search to approximate the expectation in E y ∼ p θ ∗ ( y | x ) log p θ ( y | x ) , yielding a biased estimate, while sampling decoding would result in an unbiased Monte Carlo estimator.
Revised: Since our focus is a sequence generation task, we decode y with beam search to approximate the expectation in E y ∼ p θ ∗ ( y | x ) log p θ ( y | x ) , yielding a biased estimate, while sampling decoding would result in an unbiased Monte Carlo estimator.

Original: The results in
Revised: The results in

Original: Table 2 demonstrate that the performance drops by 0.5 BLEU when we change the decoding strategy to sampling, which implies that beam search does contribute a bit to the performance gains.
Revised: Table 2 demonstrate that the performance drops by 0.5 BLEU when we change the decoding strategy to sampling, which implies that beam search does contribute a bit to the performance gains.

Original: This phenomenon makes sense intuitively since beam search tends to generate higher-quality pseudo targets than sampling, and the subsequent cross-entropy training might beneﬁt from implicitly learning the decoding process.
Revised: This phenomenon makes sense intuitively since beam search tends to generate higher-quality pseudo targets than sampling, and the subsequent cross-entropy training might beneﬁt from implicitly learning the decoding process.

Original: However, the decoding strategy hypothesis does not fully explain it, as we still observe a gain of 1.4 BLEU points over the baseline from sampling decoding with dropout.
Revised: However, the decoding strategy hypothesis does not fully explain it, as we still observe a gain of 1.4 BLEU points over the baseline from sampling decoding with dropout.

Original: H2.
Revised: H2.

Original: Dropout (Hinton et al., 2012).
Revised: Dropout (Hinton et al., 2012).

Original: Eq.
Revised: Eq.

Original: 1 and Eq.
Revised: 1 and Eq.

Original: 2 implicitly ignore a (seemingly) small difference between the model used to produce the pseudo targets and the model used for training: at test/decoding time the model does not use dropout while at training time dropout noise is injected in the model hidden states.
Revised: 2 implicitly ignore a (seemingly) small difference between the model used to produce the pseudo targets and the model used for training: at test/decoding time the model does not use dropout while at training time dropout noise is injected in the model hidden states.

Original: At training time, the model is forced to produce the same (pseudo) targets given the same set of inputs and the same parameter set but various noisy versions of the hidden states.
Revised: At training time, the model is forced to produce the same (pseudo) targets given the same set of inputs and the same parameter set but various noisy versions of the hidden states.

Original: The conjecture is that the additional expectation over dropout noise renders Eq.false.
Revised: The conjecture is that the additional expectation over dropout noise renders Eq.false.

Original: To verify this, we remove dropout in the pseudo training step 2 .
Revised: To verify this, we remove dropout in the pseudo training step 2 .

Original: The results in Table 2 indicate that without dropout the performance of beam search decoding drops by 1.2 BLEU, just 0.7 BLEU higher than the baseline.
Revised: The results in Table 2 indicate that without dropout the performance of beam search decoding drops by 1.2 BLEU, just 0.7 BLEU higher than the baseline.

Original: Moreover, the pseudo-training performance of sampling without dropout is almost the same as the baseline, which ﬁnally agrees with our intuitions from Eq. 2.
Revised: Moreover, the pseudo-training performance of sampling without dropout is almost the same as the baseline, which ﬁnally agrees with our intuitions from Eq. 2.

Original: In summary, Table 2 suggests that beam-search decoding contributes only partially to the performance gains, while the implicit perturbation – dropout – accounts for most of it.
Revised: In summary, Table 2 suggests that beam-search decoding contributes only partially to the performance gains, while the implicit perturbation – dropout – accounts for most of it.

Original: However, it is still mysterious why such perturbation results in such large performance gains.
Revised: However, it is still mysterious why such perturbation results in such large performance gains.

Original: If dropout is meant to avoid overﬁtting and ﬁt the target distribution better in the pseudo-training step, why does it bring advantages over the baseline given that the target distribution is from the baseline model itself ?
Revised: If dropout is meant to avoid overﬁtting and ﬁt the target distribution better in the pseudo-training step, why does it bring advantages over the baseline given that the target distribution is from the baseline model itself ?

Original: This is the subject of the investigation in the next section.
Revised: This is the subject of the investigation in the next section.

Original:  N OISE IN S ELF -T RAINING 4. T HE R OLE OF N OISE
Revised:  N OISE IN S ELF -T RAINING 4. T HE R OLE OF N OISE

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table] [Table] [Table] One hypothesis as to why noise (perturbation) is beneﬁcial for self-training, is that it enforces local smoothness for this task, that is, semantically similar inputs are mapped to the same or similar targets.
Revised: One hypothesis as to why noise (perturbation) is beneﬁcial for self-training, is that it enforces local smoothness for this task, that is, semantically similar inputs are mapped to the same or similar targets.

Original: Since the assumption that similar input should ideally produce similar target largely holds for most tasks in practice, this smoothing effect of pseudo-training step may provide a favorable regularization for the subsequent ﬁnetuning step.
Revised: Since the assumption that similar input should ideally produce similar target largely holds for most tasks in practice, this smoothing effect of pseudo-training step may provide a favorable regularization for the subsequent ﬁnetuning step.

Original: Unlike standard regularization in supervised training which is local to the real parallel data, self-training smooths the data space covered by the additional and much larger monolingual data.
Revised: Unlike standard regularization in supervised training which is local to the real parallel data, self-training smooths the data space covered by the additional and much larger monolingual data.

Original: To verify this hypothesis more easily, we work with the toy task of summing two integers in the range 0 to 99.
Revised: To verify this hypothesis more easily, we work with the toy task of summing two integers in the range 0 to 99.

Original: We concatenate the two integers and view them as a sequence of digits, the sum is also predicted at the digit level, thus this is still a sequence to sequence task.
Revised: We concatenate the two integers and view them as a sequence of digits, the sum is also predicted at the digit level, thus this is still a sequence to sequence task.

Original: There arepossible data points in the entire space, and we randomly sample 250 instances for training, 3 100 for validation, 5000 for test, and 4000 as the unlabeled data.
Revised: There arepossible data points in the entire space, and we randomly sample 250 instances for training, 3 100 for validation, 5000 for test, and 4000 as the unlabeled data.

Original: Test errors are computed as the absolute difference between the predicted integer and the ground-truth integer.
Revised: Test errors are computed as the absolute difference between the predicted integer and the ground-truth integer.

Original: We use an LSTM model to tackle this task.
Revised: We use an LSTM model to tackle this task.

Original: We perform self-training for one iteration on this toy sum dataset and initialize the model with the base model to rule out differences due to the initialization.
Revised: We perform self-training for one iteration on this toy sum dataset and initialize the model with the base model to rule out differences due to the initialization.

Original: Setup details are in
Revised: Setup details are in

Original: Appendix A.1.
Revised: Appendix A.1.

Original: For any integer pair ( x 1 , x 2 ) , we measure local smoothness as the standard deviation of the predictions in a 3 × 3 neighborhood of ( x 1 , x 2 ) .
Revised: For any integer pair ( x 1 , x 2 ) , we measure local smoothness as the standard deviation of the predictions in a 3 × 3 neighborhood of ( x 1 , x 2 ) .

Original: These values are averaged over all the 10000 points to obtain the overall smoothness.
Revised: These values are averaged over all the 10000 points to obtain the overall smoothness.

Original: We compare smoothness between baseline and ST pseudo-training in Table 3.
Revised: We compare smoothness between baseline and ST pseudo-training in Table 3.

Original: To demonstrate the effect of smoothing on the ﬁne-tuning step, we also report test errors after ﬁne-tuning.
Revised: To demonstrate the effect of smoothing on the ﬁne-tuning step, we also report test errors after ﬁne-tuning.

Original: We observe that ST pseudo-training attains better smoothness, which helps reducing test errors in the subsequent ﬁne-tuning step.
Revised: We observe that ST pseudo-training attains better smoothness, which helps reducing test errors in the subsequent ﬁne-tuning step.

Original: One natural question is whether we could further improve performance by encouraging even lower smoothness value, although there is a clear trade-off, as a totally smooth model that outputs a constant value is also a bad predictor.
Revised: One natural question is whether we could further improve performance by encouraging even lower smoothness value, although there is a clear trade-off, as a totally smooth model that outputs a constant value is also a bad predictor.

Original: One way to decrease smoothness is by increasing the dropout probability in the pseudo-training step, but a large dropout (like 0.5) makes the model too unstable and slow at converging.
Revised: One way to decrease smoothness is by increasing the dropout probability in the pseudo-training step, but a large dropout (like 0.5) makes the model too unstable and slow at converging.

Original: Therefore, we consider a simple model-agnostic perturbation process – perturbing the input, which we refer to as noisy self-training (noisy ST).
Revised: Therefore, we consider a simple model-agnostic perturbation process – perturbing the input, which we refer to as noisy self-training (noisy ST).

Original: 4. N OISY S ELF -T RAINING
Revised: [Figure] [Figure]

Original: If we perturb the input during the pseudo-training step, then Eq.
Revised: If we perturb the input during the pseudo-training step, then Eq.

Original: 1 would be modiﬁed to:
Revised: 1 would be modiﬁed to:

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: where g ( x ) is a perturbation function.
Revised: where g ( x ) is a perturbation function.

Original: Note that we apply both input perturbation and dropout in the pseudo-training step for noisy ST throughout the paper, but include ablation analysis in § 4.3.
Revised: Note that we apply both input perturbation and dropout in the pseudo-training step for noisy ST throughout the paper, but include ablation analysis in § 4.3.

Original: We ﬁrst validate noisy ST in the toy sum task.
Revised: We ﬁrst validate noisy ST in the toy sum task.

Original: We shufﬂe the two integers in the input as the perturbation function.
Revised: We shufﬂe the two integers in the input as the perturbation function.

Original: Such perturbation is suitable for this task since it would help the model learn the commutative law as well.
Revised: Such perturbation is suitable for this task since it would help the model learn the commutative law as well.

Original: To check that, we also measure the symmetry of the output space.
Revised: To check that, we also measure the symmetry of the output space.

Original: Speciﬁcally, for any point ( x 1 , x 2 ) , we compute | f ( x 1 , x 2 ) − f ( x 2 , x 1 ) | and average it over all the points.
Revised: Speciﬁcally, for any point ( x 1 , x 2 ) , we compute | f ( x 1 , x 2 ) − f ( x 2 , x 1 ) | and average it over all the points.

Original: Both smoothness and symmetry values are reported in Table 3.
Revised: Both smoothness and symmetry values are reported in Table 3.

Original: While we do not explicitly perturb the input at nearby integers, the shufﬂing perturbation greatly improves the smoothness metric as well.
Revised: While we do not explicitly perturb the input at nearby integers, the shufﬂing perturbation greatly improves the smoothness metric as well.

Original: Furthermore, predictions are more symmetric and test errors are reduced.
Revised: Furthermore, predictions are more symmetric and test errors are reduced.

Original: In order to illustrate the effect of smoothness, in Figure 2 we show two examples of error heat map.
Revised: In order to illustrate the effect of smoothness, in Figure 2 we show two examples of error heat map.

Original: When a point with large error is surrounded by points with small errors, the labels might propagate due to smoothing and its error is likely to become smaller, resulting in a “self-correcting” behaviour, as demonstrated in the left example of Figure 2.
Revised: When a point with large error is surrounded by points with small errors, the labels might propagate due to smoothing and its error is likely to become smaller, resulting in a “self-correcting” behaviour, as demonstrated in the left example of Figure 2.

Original: However, the prediction of some points might become worse due to the opposite phenomenon too, as shown in the right example of Figure 2.
Revised: However, the prediction of some points might become worse due to the opposite phenomenon too, as shown in the right example of Figure 2.

Original: Therefore, the smoothing effect by itself does not guarantee a performance gain in the pseudotraining step, but ﬁne-tuning beneﬁts from it and seems to consistently improve the baseline in all datasets we experiment with.
Revised: Therefore, the smoothing effect by itself does not guarantee a performance gain in the pseudotraining step, but ﬁne-tuning beneﬁts from it and seems to consistently improve the baseline in all datasets we experiment with.

Original: 4. O BSERVATIONS ON M ACHINE T RANSLATION
Revised: 4. O BSERVATIONS ON M ACHINE T RANSLATION

Original: Next, we apply noisy self-training to the more realistic WMT100 translation task.
Revised: Next, we apply noisy self-training to the more realistic WMT100 translation task.

Original: We try two different perturbation functions: (1) Synthetic noise as used in unsupervised MT (Lample et al., 2018), where the input tokens are randomly dropped, masked, and shufﬂed.
Revised: We try two different perturbation functions: (1) Synthetic noise as used in unsupervised MT (Lample et al., 2018), where the input tokens are randomly dropped, masked, and shufﬂed.

Original: We use the default noising parameters as in unsupervised MT but study the inﬂuence of noise level in § 5.4.
Revised: We use the default noising parameters as in unsupervised MT but study the inﬂuence of noise level in § 5.4.

Original: (2) Paraphrase.
Revised: (2) Paraphrase.

Original: We translate the source English sentences to German and translate it back to obtain a paraphrase as the perturbation.
Revised: We translate the source English sentences to German and translate it back to obtain a paraphrase as the perturbation.

Original: Figure 1 shows the results over three iterations.
Revised: Figure 1 shows the results over three iterations.

Original: Noisy ST (NST) greatly outperforms the supervised baseline by over 6 BLEU points and normal ST by 3 BLEU points, while synthetic noise does not exhibit much difference from paraphrasing.
Revised: Noisy ST (NST) greatly outperforms the supervised baseline by over 6 BLEU points and normal ST by 3 BLEU points, while synthetic noise does not exhibit much difference from paraphrasing.

Original: Since synthetic noise is much simpler and more general, in the remaining experiments we use synthetic noise unless otherwise speciﬁed.
Revised: Since synthetic noise is much simpler and more general, in the remaining experiments we use synthetic noise unless otherwise speciﬁed.

Original: Next, we report an ablation analysis of noisy ST when removing dropout at the pseudo-training step in Table 2.
Revised: Next, we report an ablation analysis of noisy ST when removing dropout at the pseudo-training step in Table 2.

Original: Noisy ST without dropout improves the baseline by 2.3 BLEU points and is comparable to normal ST with dropout.
Revised: Noisy ST without dropout improves the baseline by 2.3 BLEU points and is comparable to normal ST with dropout.

Original: When combined together, noisy ST with dropout produces another 1.
Revised: When combined together, noisy ST with dropout produces another 1.

Original: BLEU improvement, indicating that the two perturbations are complementary.
Revised: BLEU improvement, indicating that the two perturbations are complementary.

Revised: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our experiments below are designed to examine whether the noisy self-training is generally useful across different sequence generation tasks and resource settings.
Revised: Our experiments below are designed to examine whether the noisy self-training is generally useful across different sequence generation tasks and resource settings.

Original: To this end, we conduct experiments on two machine translation datasets and one text summarization dataset to test the effectiveness under both high-resource and low-resource settings.[Table]
Revised: To this end, we conduct experiments on two machine translation datasets and one text summarization dataset to test the effectiveness under both high-resource and low-resource settings.

Original: 5. G ENERAL S ETUP
Revised: 5. G ENERAL S ETUP

Original: We run noisy self-training for three iterations or until performance converges.
Revised: We run noisy self-training for three iterations or until performance converges.

Original: The model is trained from scratch in the pseudo-training step at each iteration since we found this strategy to work slightly better empirically.
Revised: The model is trained from scratch in the pseudo-training step at each iteration since we found this strategy to work slightly better empirically.

Original: Full model and training details for all the experiments can be found in Appendix A.1.
Revised: Full model and training details for all the experiments can be found in Appendix A.1.

Original: In some settings, we also include back-translation (BT, Sennrich et al., 2015) as a reference point, since this is probably the most successful semi-supervised learning method for machine translation.
Revised: In some settings, we also include back-translation (BT, Sennrich et al., 2015) as a reference point, since this is probably the most successful semi-supervised learning method for machine translation.

Original: However, we want to emphasize that BT is not directly comparable to ST since they use different resources (ST utilizes the unlabeled data on the source side while BT leverages target monolingual data) and use cases.
Revised: However, we want to emphasize that BT is not directly comparable to ST since they use different resources (ST utilizes the unlabeled data on the source side while BT leverages target monolingual data) and use cases.

Original: For example, BT is not very effective when we translate
Revised: For example, BT is not very effective when we translate

Original: English to extremely low-resource languages where there is almost no in-domain target monolingual data available.
Revised: English to extremely low-resource languages where there is almost no in-domain target monolingual data available.

Original: We follow the practice in (Edunov et al., 2018) to implement BT where we use unrestricted sampling to translate the target data back to the source.
Revised: We follow the practice in (Edunov et al., 2018) to implement BT where we use unrestricted sampling to translate the target data back to the source.

Original: Then, we train the real and pseudo parallel data jointly and tune the upsampling ratio of real parallel data.
Revised: Then, we train the real and pseudo parallel data jointly and tune the upsampling ratio of real parallel data.

Original: 5. M ACHINE T RANSLATION
Revised: 5. M ACHINE T RANSLATION

Original: We test the proposed noisy self-training on a high-resource translation benchmark: WMT14 EnglishGerman and a low-resource translation benchmark: FloRes English-Nepali.
Revised: We test the proposed noisy self-training on a high-resource translation benchmark: WMT14 EnglishGerman and a low-resource translation benchmark: FloRes English-Nepali.

Original: • WMT14 English-German:
Revised: • WMT14 English-German:

Original: In addition to WMT100K, we also report results with all 3.9M training examples.
Revised: In addition to WMT100K, we also report results with all 3.9M training examples.

Original: For WMT100K we use the Base Transformer architecture, and the remaining parallel data as the monolingual data.
Revised: For WMT100K we use the Base Transformer architecture, and the remaining parallel data as the monolingual data.

Original: For the full setting, we use the Big Transformer architecture (Vaswani et al., 2017) and randomly sample 20M English sentences from the News Crawl corpus for noisy ST.
Revised: For the full setting, we use the Big Transformer architecture (Vaswani et al., 2017) and randomly sample 20M English sentences from the News Crawl corpus for noisy ST.

Original: • FloRes English-Nepali: We evaluate noisy self-training on a low-resource machine translation dataset FloRes (Guzm ´ an et al., 2019) from English (en) to Nepali (ne), where we have 560K training pairs and a very weak supervised system that attains BLEU smaller than 5 points.
Revised: • FloRes English-Nepali: We evaluate noisy self-training on a low-resource machine translation dataset FloRes (Guzm´an et al., 2019) from English (en) to Nepali (ne), where we have 560K training pairs and a very weak supervised system that attains BLEU smaller than 5 points.

Original: For this dataset we have 3.6M Nepali monolingual instances in total (for BT) but 68M English Wikipedia sentences.
Revised: For this dataset we have 3.6M Nepali monolingual instances in total (for BT) but 68M English Wikipedia sentences.

Original: 5 We randomly sample 5M English sentences for noisy ST.
Revised: 5 We randomly sample 5M English sentences for noisy ST.

Original: We use the same transformer architecture as in (Guzm ´ an et al., 2019).
Revised: We use the same transformer architecture as in (Guzm´an et al., 2019).

Original: The overall results are shown in Table 4.
Revised: The overall results are shown in Table 4.

Original: For almost all cases in both datasets, the noisy ST outperforms the baselines by a large margin ( 1 ∼ 5 BLEU scores), and we see that noisy ST still improves the baseline even when this is very weak.
Revised: For almost all cases in both datasets, the noisy ST outperforms the baselines by a large margin ( 1 ∼ 5 BLEU scores), and we see that noisy ST still improves the baseline even when this is very weak.

Original: Effect of Domain Mismatch.
Revised: Effect of Domain Mismatch.

Original: Test sets of the FloRes benchmark were built with mixed originaltranslationese – some sentences are from English sources and some are from Nepali sources.
Revised: Test sets of the FloRes benchmark were built with mixed originaltranslationese – some sentences are from English sources and some are from Nepali sources.

Original: Intuitively, English monolingual data should be more in-domain with English-origin sentences and
Revised: Intuitively, English monolingual data should be more in-domain with English-origin sentences and

Revised: [Table]

Revised: [Figure]

Revised: -training on WMT English-German dataset, demonstrating the effect of parallel data size, monolingual data size, and noise level.

Original: Nepali monolingual data should help more for Nepali-origin sentences.
Revised: Nepali monolingual data should help more for Nepali-origin sentences.

Original: To demonstrate this possible domain-mismatch effect, in Table 4 we report BLEU on the two different test sets separately.
Revised: To demonstrate this possible domain-mismatch effect, in Table 4 we report BLEU on the two different test sets separately.

Original: As expected, ST is very effective when the source sentences originate from English.
Revised: As expected, ST is very effective when the source sentences originate from English.

Original: Comparison to Back-Translation.
Revised: Comparison to Back-Translation.

Original: Table 4 shows that noisy ST is able to beat BT on WMT100K and on the en-origin test set of FloRes.
Revised: Table 4 shows that noisy ST is able to beat BT on WMT100K and on the en-origin test set of FloRes.

Original: In contrast, BT is more effective on the ne-origin test set according to BLEU, which is not surprising as the ne-origin test is likely to beneﬁt more from Nepali than English monolingual data.[Table] [Figure]
Revised: In contrast, BT is more effective on the ne-origin test set according to BLEU, which is not surprising as the ne-origin test is likely to beneﬁt more from Nepali than English monolingual data.

Original: 5. T EXT S UMMARIZATION
Revised: 5. T EXT S UMMARIZATION

Original: We further evaluate noisy self-training on the Gigaword summarization dataset (Rush et al., 2015) that has 3.8M training sentences.
Revised: We further evaluate noisy self-training on the Gigaword summarization dataset (Rush et al., 2015) that has 3.8M training sentences.

Original: We encode the data with 30K byte-pair codes and use the Base
Revised: We encode the data with 30K byte-pair codes and use the Base

Original: Transformer architecture.
Revised: Transformer architecture.

Original: Similar to the setting of WMT100K, for Gigaword we create two settings where we sample 100K or 640K training examples and use the remaining as unlabeled data to compare with BT.
Revised: Similar to the setting of WMT100K, for Gigaword we create two settings where we sample 100K or 640K training examples and use the remaining as unlabeled data to compare with BT.

Original: We also consider the setting where all the 3.8M parallel samples are used and we mine in-domain monolingual data by revisiting the original preprocessing procedure 7 and using the ∼ 4M samples that Rush et al.
Revised: We also consider the setting where all the 3.8M parallel samples are used and we mine in-domain monolingual data by revisiting the original preprocessing procedure 7 and using the ∼ 4M samples that Rush et al.

Original: (2015) disregarded because they had low-quality targets.
Revised: (2015) disregarded because they had low-quality targets.

Original: We report ROUGE scores (Lin, 2004) in Table 5.
Revised: We report ROUGE scores (Lin, 2004) in Table 5.

Original: Noisy ST consistently outperforms the baseline in all settings, sometimes by a large margin (100K and 640K).
Revised: Noisy ST consistently outperforms the baseline in all settings, sometimes by a large margin (100K and 640K).

Original: It outperforms BT with 100K parallel data but underperforms with 640K parallel data.
Revised: It outperforms BT with 100K parallel data but underperforms with 640K parallel data.

Original: We conjecture that BT is still effective in this case because the task is still somewhat symmetric as Gigaword mostly contains short sentences and their compressed summaries.
Revised: We conjecture that BT is still effective in this case because the task is still somewhat symmetric as Gigaword mostly contains short sentences and their compressed summaries.

Original: Notably, noisy ST in the full setting approaches the performance of state-of-the-art systems which use much larger datasets for pretraining (Song et al., 2019).
Revised: Notably, noisy ST in the full setting approaches the performance of state-of-the-art systems which use much larger datasets for pretraining (Song et al., 2019).

Original: 5. A NALYSIS
Revised: 5. A NALYSIS

Original: In this section, we focus on the WMT English-German dataset to examine the effect of three factors on noisy self-training: the size of the parallel dataset, the size of the monolingual dataset, and the noise level.
Revised: In this section, we focus on the WMT English-German dataset to examine the effect of three factors on noisy self-training: the size of the parallel dataset, the size of the monolingual dataset, and the noise level.

Original: All the noisy ST results are after the ﬁne-tuning step.
Revised: All the noisy ST results are after the ﬁne-tuning step.

Original: Parallel data size.
Revised: Parallel data size.

Original: We ﬁx the monolingual data size as 20M from News Crawl dataset, and vary the parallel data size as shown in Figure 3(a).
Revised: We ﬁx the monolingual data size as 20M from News Crawl dataset, and vary the parallel data size as shown in Figure 3(a).

Original: We use a small LSTM model for 10K, Base Transformer for 100K/640K, and Big Transformer for 3.9M. 8 Noisy ST is repeated for three iterations.
Revised: We use a small LSTM model for 10K, Base Transformer for 100K/640K, and Big Transformer for 3.9M. 8 Noisy ST is repeated for three iterations.

Original: We see that in all cases noisy ST is able to improve upon the baseline, while the performance gain is larger for intermediate value of the size of the parallel dataset, as expected.
Revised: We see that in all cases noisy ST is able to improve upon the baseline, while the performance gain is larger for intermediate value of the size of the parallel dataset, as expected.

Original: Monolingual data size.
Revised: Monolingual data size.

Original: We ﬁx the parallel data size to 100K samples, and use the rest 3.8M
Revised: We ﬁx the parallel data size to 100K samples, and use the rest 3.8M

Original: English sentences from the parallel data as monolingual data.
Revised: English sentences from the parallel data as monolingual data.

Original: We sample from this set 100K, 500K, 1.5M, and 3.8M sentences.
Revised: We sample from this set 100K, 500K, 1.5M, and 3.8M sentences.

Original: We also include another point that uses 20M monolingual sentences from a subset of News Crawl dataset.
Revised: We also include another point that uses 20M monolingual sentences from a subset of News Crawl dataset.

Original: We report performance at the ﬁrst iteration of noisy ST.
Revised: We report performance at the ﬁrst iteration of noisy ST.

Original: [Table] Figure 3(b) illustrates that the performance keeps improving as the monolingual data size increases, albeit with diminishing returns.
Revised: Figure 3(b) illustrates that the performance keeps improving as the monolingual data size increases, albeit with diminishing returns.

Original: Noise level.
Revised: Noise level.

Original: We have shown that noisy ST outperforms ST, but intuitively larger noise must not always be better since at some point it may destroy all the information present in the input.
Revised: We have shown that noisy ST outperforms ST, but intuitively larger noise must not always be better since at some point it may destroy all the information present in the input.

Original: We adopt the WMT100K setting with 100K parallel data and 3.8M monolingual data, and set the word blanking probability in the synthetic noise (Lample et al., 2018) to 0.2 (default number), 0.4, 0.6, and 0.8.
Revised: We adopt the WMT100K setting with 100K parallel data and 3.8M monolingual data, and set the word blanking probability in the synthetic noise (Lample et al., 2018) to 0.2 (default number), 0.4, 0.6, and 0.8.

Original: We also include the baseline ST without any synthetic noise.
Revised: We also include the baseline ST without any synthetic noise.

Original: Figure 3(c) demonstrates that performance is quite sensitive to noise level, and that intermediate values work best.
Revised: Figure 3(c) demonstrates that performance is quite sensitive to noise level, and that intermediate values work best.

Original: It is still unclear how to select the noise level a priori , besides the usual hyper-parameter search to maximize
Revised: It is still unclear how to select the noise level a priori , besides the usual hyper-parameter search to maximize

Original: BLEU on the validation set.
Revised: BLEU on the validation set.

Original: 5. N OISE P ROCESS ON P ARALLEL D ATA O NLY
Revised: 5. N OISE P ROCESS ON P ARALLEL D ATA O NLY

Original: In this section, we justify whether the proposed noisy self-training process would help the supervised baseline alone without the help of any monolingual data.
Revised: In this section, we justify whether the proposed noisy self-training process would help the supervised baseline alone without the help of any monolingual data.

Original: Similar to the training process on the monolingual data, we ﬁrst train the model on the noisy source data (pseudo-training), and then ﬁnetune it on clean parallel data.
Revised: Similar to the training process on the monolingual data, we ﬁrst train the model on the noisy source data (pseudo-training), and then ﬁnetune it on clean parallel data.

Original: Different from using monolingual data, there are two variations here in the “pseudo-training” step: we can either train with the fake target predicted by the model as on monolingual data, or train with the real target paired with noisy source.
Revised: Different from using monolingual data, there are two variations here in the “pseudo-training” step: we can either train with the fake target predicted by the model as on monolingual data, or train with the real target paired with noisy source.

Original: We denote them as “parallel + fake target” and “parallel + real target” respectively, and report the performance on WMT100K in
Revised: We denote them as “parallel + fake target” and “parallel + real target” respectively, and report the performance on WMT100K in

Original: Table 6.
Revised: Table 6.

Original: We use the same synthetic noise as used in previous experiments.
Revised: We use the same synthetic noise as used in previous experiments.

Original: When applying the same noise process to parallel data using fake target, the smoothing effect is not signiﬁcant since it is restricted into the limited parallel data space, producing marginal improvement over the baseline (+0.4 BLEU).
Revised: When applying the same noise process to parallel data using fake target, the smoothing effect is not signiﬁcant since it is restricted into the limited parallel data space, producing marginal improvement over the baseline (+0.4 BLEU).

Original: As a comparison, 100K monolingual data produces +1.0 BLEU and the effect is enhanced when we increase the monolingual data to 3.8M, which leads to +3.
Revised: As a comparison, 100K monolingual data produces +1.0 BLEU and the effect is enhanced when we increase the monolingual data to 3.8M, which leads to +3.

Original: BLEU.
Revised: BLEU.

Original: Interestingly, pairing the noisy source with real target results in much worse performance than the baseline (-4.3 BLEU), which implies that the use of fake target predicted by the model (i.e. distillation) instead of real target is important for the success of noisy self-training, at least in the case where parallel data size is small.
Revised: Interestingly, pairing the noisy source with real target results in much worse performance than the baseline (-4.3 BLEU), which implies that the use of fake target predicted by the model (i.e. distillation) instead of real target is important for the success of noisy self-training, at least in the case where parallel data size is small.

Original: Intuitively, the distilled fake target is simpler and relatively easy for the model to ﬁt, but the real target paired with noisy source makes learning even harder than training with real target and real source, which might lead to a bad starting point for ﬁne-tuning.
Revised: Intuitively, the distilled fake target is simpler and relatively easy for the model to ﬁt, but the real target paired with noisy source makes learning even harder than training with real target and real source, which might lead to a bad starting point for ﬁne-tuning.

Original: This issue would be particularly severe when the parallel data size is small, in that case the model would have difﬁculties to ﬁt real target even with clean source.
Revised: This issue would be particularly severe when the parallel data size is small, in that case the model would have difﬁculties to ﬁt real target even with clean source.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Self-training belongs to a broader class of “pseudo-label” semi-supervised learning approaches.
Revised: Self-training belongs to a broader class of “pseudo-label” semi-supervised learning approaches.

Original: These approaches all learn from pseudo labels assigned to unlabelled data, with different methods on how to assign such labels.
Revised: These approaches all learn from pseudo labels assigned to unlabelled data, with different methods on how to assign such labels.

Original: For instance, co-training (Blum & Mitchell, 1998) learns models on two independent feature sets of the same data, and assigns conﬁdent labels to unlabeled data from one of the models.
Revised: For instance, co-training (Blum & Mitchell, 1998) learns models on two independent feature sets of the same data, and assigns conﬁdent labels to unlabeled data from one of the models.

Original: Co-training reduces modeling bias by taking into account conﬁdence scores from two models.
Revised: Co-training reduces modeling bias by taking into account conﬁdence scores from two models.

Original: In the same spirit, democratic co-training (Zhou & Goldman, 2004) or tri-training (Zhou & Li, 2005) trains multiple models with different conﬁgurations on the same data feature set, and a subset 
Revised: In the same spirit, democratic co-training (Zhou & Goldman, 2004) or tri-training (Zhou & Li, 2005) trains multiple models with different conﬁgurations on the same data feature set, and a subset of the models act as teachers for others. [Table]

Original: Another line of more recent work perturb the input or feature space of the student’s inputs as data augmentation techniques.
Revised: Another line of more recent work perturb the input or feature space of the student’s inputs as data augmentation techniques.

Original: Self-training with dropout or noisy self-training can be viewed as an instantiation of this.
Revised: Self-training with dropout or noisy self-training can be viewed as an instantiation of this.

Original: These approaches have been very successful on classiﬁcation tasks (Rasmus et al., 2015;
Revised: These approaches have been very successful on classiﬁcation tasks (Rasmus et al., 2015;

Original: Miyato et al., 2017; Laine & Aila, 2017; Miyato et al., 2018; Xie et al., 2019) given that a reasonable amount of predictions of unlabeled data (at least the ones with high conﬁdence) are correct, but their effect on language generation tasks is largely unknown and poorly understood because the pseudo language targets are often very different from the ground-truth labels.
Revised: Miyato et al., 2017; Laine & Aila, 2017; Miyato et al., 2018; Xie et al., 2019) given that a reasonable amount of predictions of unlabeled data (at least the ones with high conﬁdence) are correct, but their effect on language generation tasks is largely unknown and poorly understood because the pseudo language targets are often very different from the ground-truth labels.

Original: Recent work on sequence generation employs auxiliary decoders (Clark et al., 2018) when processing unlabeled data, overall showing rather limited gains.
Revised: Recent work on sequence generation employs auxiliary decoders (Clark et al., 2018) when processing unlabeled data, overall showing rather limited gains.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we revisit self-training for neural sequence generation, and show that it can be an effective method to improve generalization, particularly when labeled data is scarce.
Revised: In this paper we revisit self-training for neural sequence generation, and show that it can be an effective method to improve generalization, particularly when labeled data is scarce.

Original: Through a comprehensive ablation analysis and synthetic experiments, we identify that noise injected during self-training plays a critical role for its success due to its smoothing effect.
Revised: Through a comprehensive ablation analysis and synthetic experiments, we identify that noise injected during self-training plays a critical role for its success due to its smoothing effect.

Original: To encourage this behaviour, we explicitly perturb the input to obtain a new variant of self-training, dubbed noisy selftraining.
Revised: To encourage this behaviour, we explicitly perturb the input to obtain a new variant of self-training, dubbed noisy selftraining.

Original: Experiments on machine translation and text summarization demonstrate the effectiveness of this approach in both low and high resource settings.
Revised: Experiments on machine translation and text summarization demonstrate the effectiveness of this approach in both low and high resource settings.

Revised: A CKNOWLEDGEMENTS

Revised: We want to thank Peng-Jen Chen for helping set up the FloRes experiments, and Michael Auli,

Revised: Kyunghyun Cho, and Graham Neubig for insightful discussion about this project

Original: A E XPERIMENTS DETAILS A. S ETUP D ETAILS
Revised: A E XPERIMENTS DETAILS A. S ETUP D ETAILS

Original: For all experiments, we optimize with Adam (Kingma & Ba, 2014) using β 1 = 0 .
Revised: For all experiments, we optimize with Adam (Kingma & Ba, 2014) using β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 98 , (cid:15) = 1 e − 8 .
Revised: 98 , (cid:15) = 1 e − 8 .

Original: All implementations are based on fairseq (Ott et al., 2019), and we basically use the same learning rate schedule and label smoothing as in fairseq examples to train the transformers.
Revised: All implementations are based on fairseq (Ott et al., 2019), and we basically use the same learning rate schedule and label smoothing as in fairseq examples to train the transformers.

Original: 9 Except for the toy sum dataset which we runs on a single GPU and each batch contains 32 examples, all other experiments are run on 8 GPUs with an effective batch size of 33K tokens.
Revised: 9 Except for the toy sum dataset which we runs on a single GPU and each batch contains 32 examples, all other experiments are run on 8 GPUs with an effective batch size of 33K tokens.

Original: All experiments are validated with loss on the validation set.
Revised: All experiments are validated with loss on the validation set.

Original: For self-training or noisy self-training, the pseudo-training takes 300K synchronous updates while the ﬁne-tuning step takes 100K steps.
Revised: For self-training or noisy self-training, the pseudo-training takes 300K synchronous updates while the ﬁne-tuning step takes 100K steps.

Original: We use the downloading and preprocessing scripts in fairseq to obtain the WMT 2014 EnglishGerman dataset, 10 which hold out a small fraction of the original training data as the validation set.
Revised: We use the downloading and preprocessing scripts in fairseq to obtain the WMT 2014 EnglishGerman dataset, 10 which hold out a small fraction of the original training data as the validation set.

Original: The model architecture for the toy sum dataset is a single-layer LSTM with word embedding size 32, hidden state size 32, and dropout rate 0.3.
Revised: The model architecture for the toy sum dataset is a single-layer LSTM with word embedding size 32, hidden state size 32, and dropout rate 0.3.

Original: The model architecture of WMT10K baseline in
Revised: The model architecture of WMT10K baseline in

Original: Figure 3(a) is a single layer LSTM with word embeddings size 256, hidden state size 256, and dropout rate 0.3.
Revised: Figure 3(a) is a single layer LSTM with word embeddings size 256, hidden state size 256, and dropout rate 0.3.

Original: A. J USTIFICATION OF THE WMT100K B ASELINE
Revised: A. J USTIFICATION OF THE WMT100K B ASELINE

Original: We provide more details and evidence to show that our baseline model on WMT100K dataset is trained properly.
Revised: We provide more details and evidence to show that our baseline model on WMT100K dataset is trained properly.

Original: In all the experiments on WMT100K dataset including baseline and self-training ones, we use Adam optimizer with learning rate 0.0005, which is defaulted in fairseq.
Revised: In all the experiments on WMT100K dataset including baseline and self-training ones, we use Adam optimizer with learning rate 0.0005, which is defaulted in fairseq.

Original: We do not use early stop during training but select the best model in terms of the validation loss.
Revised: We do not use early stop during training but select the best model in terms of the validation loss.

Original: We train with 30K update steps for the baseline model and (300K pseudo-training + 100K ﬁne-tuning) update steps for self-training.
Revised: We train with 30K update steps for the baseline model and (300K pseudo-training + 100K ﬁne-tuning) update steps for self-training.

Original: In both cases we veriﬁed that the models are trained sufﬁciently to fully converge through observing the increase of validation loss.
Revised: In both cases we veriﬁed that the models are trained sufﬁciently to fully converge through observing the increase of validation loss.

Original: Figure 4 shows the validation curve of the baseline model.
Revised: Figure 4 shows the validation curve of the baseline model.

Original: Note that the model starts to overﬁt, and we select the model checkpoint at the lowest point.
Revised: Note that the model starts to overﬁt, and we select the model checkpoint at the lowest point.

Original: We also varied the learning rate hyperparameter as 0.0002, 0.0005, and 0.001, which produced BLEU score 15.0, 15.6 (reported in the paper), and 15.5 respectively – our baseline model in previous sections obtained the best performance.
Revised: We also varied the learning rate hyperparameter as 0.0002, 0.0005, and 0.001, which produced BLEU score 15.0, 15.6 (reported in the paper), and 15.5 respectively – our baseline model in previous sections obtained the best performance.

Original: [Figure]
Revised: [Figure]

Original: B C OMPARISON REGARDING SEPARATE TRAINING , JOINT TRAINING , AND FILTERING
Revised: B C OMPARISON REGARDING SEPARATE TRAINING , JOINT TRAINING , AND FILTERING

Original: In the paper we perform self-training with separate pseudo-training and ﬁne-tuning steps and always use all monolingual data.
Revised: In the paper we perform self-training with separate pseudo-training and ﬁne-tuning steps and always use all monolingual data.

Original: However, there are other variants such as joint training or iteratively adding conﬁdent examples.
Revised: However, there are other variants such as joint training or iteratively adding conﬁdent examples.

Original: Here we compare these variants on WMT100K dataset, noisy self-training uses paraphrase as the perturbation function.
Revised: Here we compare these variants on WMT100K dataset, noisy self-training uses paraphrase as the perturbation function.

Original: For joint training, we tune the upsampling ratio of parallel data just as in back-translation (Edunov et al., 2018).
Revised: For joint training, we tune the upsampling ratio of parallel data just as in back-translation (Edunov et al., 2018).

Original: We perform noisy self-training for 3 iterations, and for ﬁltering experiments we iteratively use the most conﬁdent 2.5M, 3M, and 3.8M monolingual data respectively in these 3 iterations.
Revised: We perform noisy self-training for 3 iterations, and for ﬁltering experiments we iteratively use the most conﬁdent 2.5M, 3M, and 3.8M monolingual data respectively in these 3 iterations.

Original: Table 7 shows that the ﬁltering process helps joint training but still underperforms separate-training methods by over 1.5 BLEU points.
Revised: Table 7 shows that the ﬁltering process helps joint training but still underperforms separate-training methods by over 1.5 BLEU points.

Original: Within separate training ﬁltering produces comparable results to using all data.
Revised: Within separate training ﬁltering produces comparable results to using all data.

Original: Since separate training with all data is the simplest method and produces the best performance, we stick to this version in the paper.
Revised: Since separate training with all data is the simplest method and produces the best performance, we stick to this version in the paper.

Original: [Table]
Revised: [Table]

Original: C A DDITIONAL RESULTS ON THE TOY SUM DATASET
Revised: C A DDITIONAL RESULTS ON THE TOY SUM DATASET

Original: We additionally show the error heat maps of the entire data space on the toy sum datasets for the ﬁrst two iterations.
Revised: We additionally show the error heat maps of the entire data space on the toy sum datasets for the ﬁrst two iterations.

Original: Here the model at pseudo-training step is initialized as the model from last iteration to clearly examine how the decodings change due to injected noise.
Revised: Here the model at pseudo-training step is initialized as the model from last iteration to clearly examine how the decodings change due to injected noise.

Original: As shown in Figure 5, for each iteration the pseudo-training step smooths the space and ﬁne-tuning step beneﬁts from it and greatly reduces the errors
Revised: As shown in Figure 5, for each iteration the pseudo-training step smooths the space and ﬁne-tuning step beneﬁts from it and greatly reduces the errors

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original:  [Figure]
Revised: [Table] [Table] [Figure] [Figure]