Original: C ROSS - MODEL B ACK - TRANSLATED D ISTILLATION FOR U NSUPERVISED M ACHINE T RANSLATION
Revised: C ROSS - MODEL B ACK - TRANSLATED D ISTILLATION FOR U NSUPERVISED M ACHINE T RANSLATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent unsupervised machine translation (UMT) systems usually employ three main principles: initialization, language modeling and iterative back-translation, though they may apply them differently.
Revised: Recent unsupervised machine translation (UMT) systems usually employ three main principles: initialization, language modeling and iterative back-translation, though they may apply them differently.

Original: Crucially, iterative back-translation and denoising auto-encoding for language modeling provide data diversity to train the UMT systems.
Revised: Crucially, iterative back-translation and denoising auto-encoding for language modeling provide data diversity to train the UMT systems.

Original: However, the gains from these diversiﬁcation processes has seemed to plateau.
Revised: However, the gains from these diversiﬁcation processes has seemed to plateau.

Original: We introduce a novel component to the standard UMT framework called Cross-model Back-translated Distillation (CBD), that is aimed to induce another level of data diversiﬁcation that existing principles lack.
Revised: We introduce a novel component to the standard UMT framework called Cross-model Back-translated Distillation (CBD), that is aimed to induce another level of data diversiﬁcation that existing principles lack.

Original: CBD is applicable to all previous UMT approaches.
Revised: CBD is applicable to all previous UMT approaches.

Original: In our experiments, it boosts the performance of the standard UMT methods by 1.5-2.0 BLEU.
Revised: In our experiments, it boosts the performance of the standard UMT methods by 1.5-2.0 BLEU.

Original: In particular, in
Revised: In particular, in

Original: WMT’14 English-French, WMT’16 German-English and English-Romanian, CBD outperforms cross-lingual masked language model (XLM) by 2.3, 2.2 and 1.
Revised: WMT’14 English-French, WMT’16 German-English and English-Romanian, CBD outperforms cross-lingual masked language model (XLM) by 2.3, 2.2 and 1.

Original: BLEU, respectively.
Revised: BLEU, respectively.

Original: It also yields 1.5–3.3 BLEU improvements in IWSLT EnglishFrench and English-German tasks.
Revised: It also yields 1.5–3.3 BLEU improvements in IWSLT EnglishFrench and English-German tasks.

Original: Through extensive experimental analyses, we show that CBD is effective because it embraces data diversity while other similar variants do not.
Revised: Through extensive experimental analyses, we show that CBD is effective because it embraces data diversity while other similar variants do not.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Machine translation (MT) is a core task in natural language processing that involves both language understanding and generation.
Revised: Machine translation (MT) is a core task in natural language processing that involves both language understanding and generation.

Original: Recent neural approaches (Vaswani et al., 2017; Wu et al., 2019) have advanced the state of the art with near human-level performance (Hassan et al., 2018).
Revised: Recent neural approaches (Vaswani et al., 2017; Wu et al., 2019) have advanced the state of the art with near human-level performance (Hassan et al., 2018).

Original: However, they continue to rely heavily on large parallel data.
Revised: However, they continue to rely heavily on large parallel data.

Original: As a result, the search for unsupervised alternatives using only monolingual data has been active.
Revised: As a result, the search for unsupervised alternatives using only monolingual data has been active.

Original: While Ravi & Knight (2011) and Klementiev et al.
Revised: While Ravi & Knight (2011) and Klementiev et al.

Original: (2012) proposed various unsupervised techniques for statistical MT (SMT), Lample et al.
Revised: (2012) proposed various unsupervised techniques for statistical MT (SMT), Lample et al.

Original: (2018a;c) established a general framework for modern unsupervised MT (UMT) that works for both SMT and neural MT (NMT) models.
Revised: (2018a;c) established a general framework for modern unsupervised MT (UMT) that works for both SMT and neural MT (NMT) models.

Original: The framework has three main principles: model initialization, language modeling and iterative back-translation.
Revised: The framework has three main principles: model initialization, language modeling and iterative back-translation.

Original: Model initialization bootstraps the model with a knowledge prior like word-level transfer (Lample et al., 2018b).
Revised: Model initialization bootstraps the model with a knowledge prior like word-level transfer (Lample et al., 2018b).

Original: Language modeling, which takes the form of denoising auto-encoding (DAE) in NMT (Lample et al., 2018c), trains the model to generate plausible sentences in a language.
Revised: Language modeling, which takes the form of denoising auto-encoding (DAE) in NMT (Lample et al., 2018c), trains the model to generate plausible sentences in a language.

Original: Meanwhile, iterative back-translation (IBT) facilitates cross-lingual translation training by generating noisy source sentences for original target sentences.
Revised: Meanwhile, iterative back-translation (IBT) facilitates cross-lingual translation training by generating noisy source sentences for original target sentences.

Original: The recent approaches differ on how they apply each of these three principles.
Revised: The recent approaches differ on how they apply each of these three principles.

Original: For instance, Lample et al.
Revised: For instance, Lample et al.

Original: (2018a) use an unsupervised word-translation model (Lample et al., 2018b) for model initialization, while
Revised: (2018a) use an unsupervised word-translation model (Lample et al., 2018b) for model initialization, while

Original: Conneau & Lample (2019) use a pretrained cross-lingual masked language model (XLM).
Revised: Conneau & Lample (2019) use a pretrained cross-lingual masked language model (XLM).

Original: In this paper, we focus on a different aspect of the UMT framework, namely, its data diversiﬁcationprocess.
Revised: In this paper, we focus on a different aspect of the UMT framework, namely, its data diversiﬁcationprocess.

Original: If we look from this view, the DAE and IBT steps of the UMT framework also perform some form of data diversiﬁcation to train the model.
Revised: If we look from this view, the DAE and IBT steps of the UMT framework also perform some form of data diversiﬁcation to train the model.

Original: Speciﬁcally, the noise model in the DAE process generates new , but noised, versions of the input data, which are used to train the model with a reconstruction objective.
Revised: Speciﬁcally, the noise model in the DAE process generates new , but noised, versions of the input data, which are used to train the model with a reconstruction objective.

Original: Likewise, the IBT step involves the same UMT model to create synthetic parallel pairs (with the source being synthetic), which are then used to train the model.
Revised: Likewise, the IBT step involves the same UMT model to create synthetic parallel pairs (with the source being synthetic), which are then used to train the model.

Original: Since the
Revised: Since the

Original: NMT model is updated with DAE and IBT simultaneously, the model generates fresh translations in each back-translation step.
Revised: NMT model is updated with DAE and IBT simultaneously, the model generates fresh translations in each back-translation step.

Original: Overall, thanks to DAE and IBT, the model gets better at translating by iteratively training on the newly created and diversiﬁed data whose quality also improves over time.
Revised: Overall, thanks to DAE and IBT, the model gets better at translating by iteratively training on the newly created and diversiﬁed data whose quality also improves over time.

Original: This argument also applies to statistical UMT, except for the lack of the DAE (Lample et al., 2018c).
Revised: This argument also applies to statistical UMT, except for the lack of the DAE (Lample et al., 2018c).

Original: However, we conjecture that these diversiﬁcation methods may have reached their limit as the performance does not improve further the longer we train the UMT models.
Revised: However, we conjecture that these diversiﬁcation methods may have reached their limit as the performance does not improve further the longer we train the UMT models.

Original: In this work, we introduce a fourth principle to the standard UMT framework: Cross-model Backtranslated Distillation or CBD (§3), with the aim to induce another level of diversiﬁcation that the existing UMT principles lack.
Revised: In this work, we introduce a fourth principle to the standard UMT framework: Cross-model Backtranslated Distillation or CBD (§3), with the aim to induce another level of diversiﬁcation that the existing UMT principles lack.

Original: CBD initially trains two UMT agents (models) using existing approaches.
Revised: CBD initially trains two UMT agents (models) using existing approaches.

Original: Then, one of the two agents translates the monolingual data from one language s to another t in the ﬁrst level.
Revised: Then, one of the two agents translates the monolingual data from one language s to another t in the ﬁrst level.

Original: In the second level, the generated data are back-translated from t to s by the other agent .
Revised: In the second level, the generated data are back-translated from t to s by the other agent .

Original: In the ﬁnal step, the synthetic parallel data created by the ﬁrst and second levels are used to distill a supervised MT model.
Revised: In the ﬁnal step, the synthetic parallel data created by the ﬁrst and second levels are used to distill a supervised MT model.

Original: CBD is applicable to any existing UMT method and is more efﬁcient than ensembling approaches (Freitag et al., 2017) (§5.3).
Revised: CBD is applicable to any existing UMT method and is more efﬁcient than ensembling approaches (Freitag et al., 2017) (§5.3).

Original: In the experiments (§4), we have evaluated CBD on the WMT’14 English-French, WMT’
Revised: In the experiments (§4), we have evaluated CBD on the WMT’14 English-French, WMT’

Original: English-German and WMT’16 English-Romanian unsupervised translation tasks.
Revised: English-German and WMT’16 English-Romanian unsupervised translation tasks.

Original: CBD shows consistent improvements of 1.0-2.0 BLEU compared to the baselines in these tasks.
Revised: CBD shows consistent improvements of 1.0-2.0 BLEU compared to the baselines in these tasks.

Original: It also boosts the performance on IWSLT’14 English-German and IWSLT’13 English-French tasks signiﬁcantly.
Revised: It also boosts the performance on IWSLT’14 English-German and IWSLT’13 English-French tasks signiﬁcantly.

Original: In our analysis, we explain with experiments why other similar variants (§5.1) and other alternatives from the literature (§5.4) do not work well and cross-model back-translation is crucial for our method.
Revised: In our analysis, we explain with experiments why other similar variants (§5.1) and other alternatives from the literature (§5.4) do not work well and cross-model back-translation is crucial for our method.

Original: We further demonstrate that CBD enhances the baselines by achieving greater diversity as measured by back-translation BLEU (§5.2).
Revised: We further demonstrate that CBD enhances the baselines by achieving greater diversity as measured by back-translation BLEU (§5.2).

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Ravi & Knight (2011) were among the ﬁrst to propose a UMT system by framing the problem as a decipherment task that considers non-English text as a cipher for English.
Revised: Ravi & Knight (2011) were among the ﬁrst to propose a UMT system by framing the problem as a decipherment task that considers non-English text as a cipher for English.

Original: Nonetheless, the method is limited and may not be applicable to the current well-established NMT systems (Luong et al., 2015;
Revised: Nonetheless, the method is limited and may not be applicable to the current well-established NMT systems (Luong et al., 2015;

Original: Vaswani et al., 2017; Wu et al., 2019).
Revised: Vaswani et al., 2017; Wu et al., 2019).

Original: Lample et al.
Revised: Lample et al.

Original: (2018a) set the foundation for modern UMT.
Revised: (2018a) set the foundation for modern UMT.

Original: They propose to maintain two encoder-decoder networks simultaneously for both source and target languages, and train them via denoising auto-encoding, iterative back-translation and adversarial training.
Revised: They propose to maintain two encoder-decoder networks simultaneously for both source and target languages, and train them via denoising auto-encoding, iterative back-translation and adversarial training.

Original: In their follow-up work, Lample et al.
Revised: In their follow-up work, Lample et al.

Original: (2018c) formulate a common UMT framework for both PBSMT and NMT with three basic principles that can be customized.
Revised: (2018c) formulate a common UMT framework for both PBSMT and NMT with three basic principles that can be customized.

Original: The three principles are:
Revised: The three principles are:

Original: • Initialization : A non-randomized cross- or multi-lingual initialization that represents a knowledge prior to bootstrap the UMT model.
Revised: • Initialization : A non-randomized cross- or multi-lingual initialization that represents a knowledge prior to bootstrap the UMT model.

Original: For instance, Lample et al.
Revised: For instance, Lample et al.

Original: (2018a) and Artetxe et al.
Revised: (2018a) and Artetxe et al.

Original: (2019) use an unsupervised word-translation model MUSE (Lample et al., 2018b) as initialization to promote word-to-word cross-lingual transfer.
Revised: (2019) use an unsupervised word-translation model MUSE (Lample et al., 2018b) as initialization to promote word-to-word cross-lingual transfer.

Original: Lample et al.
Revised: Lample et al.

Original: (2018c) use a shared jointly trained sub-word (Sennrich et al., 2016b) dictionary.
Revised: (2018c) use a shared jointly trained sub-word (Sennrich et al., 2016b) dictionary.

Original: On the other hand, Conneau & Lample (2019) use a pretrained cross-lingual masked language model (XLM) to initialize the unsupervised NMT model.
Revised: On the other hand, Conneau & Lample (2019) use a pretrained cross-lingual masked language model (XLM) to initialize the unsupervised NMT model.

Original: • Language modeling : Training a language model on monolingual data helps the UMT model to generate ﬂuent texts.
Revised: • Language modeling : Training a language model on monolingual data helps the UMT model to generate ﬂuent texts.

Original: The neural UMT approaches (Lample et al., 2018a;c; Conneau & Lample, 2019) use denoising auto-encoder training to achieve language modeling effects in the neural model.
Revised: The neural UMT approaches (Lample et al., 2018a;c; Conneau & Lample, 2019) use denoising auto-encoder training to achieve language modeling effects in the neural model.

Original: Meanwhile, the PBSMT variant proposed by Lample et al.
Revised: Meanwhile, the PBSMT variant proposed by Lample et al.

Original: (2018c) uses the KenLM smoothed n-gram language models (Heaﬁeld, 2011).
Revised: (2018c) uses the KenLM smoothed n-gram language models (Heaﬁeld, 2011).

Original: • Iterative back-translation : Back-translation (Sennrich et al., 2016a) brings about the bridge between source and target languages by using a backward model that translates data from target to source.
Revised: • Iterative back-translation : Back-translation (Sennrich et al., 2016a) brings about the bridge between source and target languages by using a backward model that translates data from target to source.

Original: The (source and target) monolingual
Revised: The (source and target) monolingual

Original: data is translated back and forth iteratively to progress the UMT model in both directions.
Revised: data is translated back and forth iteratively to progress the UMT model in both directions.

Original: During training, the initialization step is conducted once, while the denoising and back-translation steps are often executed in an alternating manner.
Revised: During training, the initialization step is conducted once, while the denoising and back-translation steps are often executed in an alternating manner.

Original: 3 It is worth noting that depending on different implementations, the parameters for backward and forward components may be separate (Lample et al., 2018a) or shared (Lample et al., 2018c; Conneau & Lample, 2019).
Revised: 3 It is worth noting that depending on different implementations, the parameters for backward and forward components may be separate (Lample et al., 2018a) or shared (Lample et al., 2018c; Conneau & Lample, 2019).

Original: A parameter-shared cross-lingual NMT model has the capability to translate from either source or target, while a UMT system with parameter-separate models has to maintain two models.
Revised: A parameter-shared cross-lingual NMT model has the capability to translate from either source or target, while a UMT system with parameter-separate models has to maintain two models.

Original: Either way, we deem a standard
Revised: Either way, we deem a standard

Original: UMT system to be bidirectional, i.e. it is capable of translating from either source or target language.
Revised: UMT system to be bidirectional, i.e. it is capable of translating from either source or target language.

Original: Our proposed cross-model back-translated distillation (CBD) works outside this well-established framework.
Revised: Our proposed cross-model back-translated distillation (CBD) works outside this well-established framework.

Original: It employs two UMT agents to create extra diversiﬁed data apart from what existing methods already offer, rendering it a useful add-on to the general UMT framework.
Revised: It employs two UMT agents to create extra diversiﬁed data apart from what existing methods already offer, rendering it a useful add-on to the general UMT framework.

Original: Furthermore, different implementations of UMT as discussed above can be plugged into the CBD system to achieve a performance boost, even for future methods that may potentially not employ the three principles.
Revised: Furthermore, different implementations of UMT as discussed above can be plugged into the CBD system to achieve a performance boost, even for future methods that may potentially not employ the three principles.

Original:  C ROSS - MODEL B ACK - TRANSLATED D ISTILLATION
Revised:  C ROSS - MODEL B ACK - TRANSLATED D ISTILLATION

Original: Let X s and X t be the sets of monolingual data for languages s and t , respectively.
Revised: Let X s and X t be the sets of monolingual data for languages s and t , respectively.

Original: We ﬁrst train two UMT agents independently with two different parameter sets θ 

Revised: We ﬁrst train two UMT agents independently with two different parameter sets θ

Original: and θ 2 using existing methods (Lample et al., 2018a;c; Conneau & Lample, 2019).
Revised: and θ 2 using existing methods (Lample et al., 2018a;c; Conneau & Lample, 2019).

Original: Since a UMT agent with parameter set θ is deemed bidirectional in our setup, we denote y t ∼ P ( ·| x s , θ ) to be the translation sample from language s to t of input sentence x s using parameters θ .
Revised: Since a UMT agent with parameter set θ is deemed bidirectional in our setup, we denote y t ∼ P ( ·| x s , θ ) to be the translation sample from language s to t of input sentence x s using parameters θ .

Original: In addition, for each language direction s → t (and vice versa for t → s ), we introduce a supervised MT model with parameters ˆ θ that will be trained through our CBD.
Revised: In addition, for each language direction s → t (and vice versa for t → s ), we introduce a supervised MT model with parameters ˆ θ that will be trained through our CBD.

Original: With these notations, we denote the following training loss L with respect to input variables θ α , θ β ∈ { θ 1 , θ 2 } , the supervised parameters ˆ θ and the learning rate η .
Revised: With these notations, we denote the following training loss L with respect to input variables θ α , θ β ∈ { θ 1 , θ 2 } , the supervised parameters ˆ θ and the learning rate η .

Original: Figurealso illustrates the translation ﬂows of function L .
Revised: Figurealso illustrates the translation ﬂows of function L .

Original: [Equation]
Revised: [Equation]

Original: In the CBD strategy (Algorithm 1, Eq. 1), at each step, each agent θ ∈ { θ 1 , θ 2 } samples translations from the monolingual data of both languages s and t to acquire the ﬁrst level of synthetic parallel data ( x s , y t ) and ( x t , y s ) .
Revised: In the CBD strategy (Algorithm 1, Eq. 1), at each step, each agent θ ∈ { θ 1 , θ 2 } samples translations from the monolingual data of both languages s and t to acquire the ﬁrst level of synthetic parallel data ( x s , y t ) and ( x t , y s ) .

Original: In the second level , the other agent { θ 1 , θ 2 } \ θ is used to sample the translation z s of the translation y t of x s (and similarly for z t from the translation y s of x t ).
Revised: In the second level , the other agent { θ 1 , θ 2 } \ θ is used to sample the translation z s of the translation y t of x s (and similarly for z t from the translation y s of x t ).

Original: This process is basically back-translation, but with the backward model coming from a different regime than that of the forward model.
Revised: This process is basically back-translation, but with the backward model coming from a different regime than that of the forward model.

Original: The fact that the ﬁrst level agent must be different from the second level agent is crucial to achieve the desirable level of diversity in data generation.
Revised: The fact that the ﬁrst level agent must be different from the second level agent is crucial to achieve the desirable level of diversity in data generation.

Original: After this, we update the ﬁnal model ˆ θ using all the synthetic pairs ( x, y ) and ( y, z ) with maximum likelihood estimation (MLE).
Revised: After this, we update the ﬁnal model ˆ θ using all the synthetic pairs ( x, y ) and ( y, z ) with maximum likelihood estimation (MLE).

Original: Algorithm
Revised: Algorithm

Original: -model Back-translated Distillation:
Revised: -model Back-translated Distillation:

Original: Given monolingual data X s and X t of languages s and t , return a s → t UMT model with parameters ˆ θ .
Revised: Given monolingual data X s and X t of languages s and t , return a s → t UMT model with parameters ˆ θ .

Original: [Equation]
Revised: [Equation]

Original: loss function L in Eq.
Revised: loss function L in Eq.

Original: Variable set ( θ α , θ β ) is replaced with ( θ 1 , θ 2 ) and ( θ 2 , θ 1 ) iteratively in Algorithm 1.
Revised: Variable set ( θ α , θ β ) is replaced with ( θ 1 , θ 2 ) and ( θ 2 , θ 1 ) iteratively in Algorithm 1.

Original: Parallel pairs ( x s , y t ) , ( z s , y t ) , ( y s , x t ) and ( y s , z t ) are used to train the ﬁnal model ˆ θ .
Revised: Parallel pairs ( x s , y t ) , ( z s , y t ) , ( y s , x t ) and ( y s , z t ) are used to train the ﬁnal model ˆ θ .

Original: In this way, the model ˆ θ ﬁrst gets trained on the translated products ( x − y ) of the UMT teachers, making it as capable as the teachers.
Revised: In this way, the model ˆ θ ﬁrst gets trained on the translated products ( x − y ) of the UMT teachers, making it as capable as the teachers.

Original: Secondly, the model ˆ θ is also trained on the second-level data ( y − z ), which is slightly different from the ﬁrst-level data.
Revised: Secondly, the model ˆ θ is also trained on the second-level data ( y − z ), which is slightly different from the ﬁrst-level data.

Original: Thus, this mechanism provides extra data diversiﬁcation to the system ˆ θ in addition to what the UMT teachers already offer, resulting in our ﬁnal model outperforming the UMT baselines (§4).
Revised: Thus, this mechanism provides extra data diversiﬁcation to the system ˆ θ in addition to what the UMT teachers already offer, resulting in our ﬁnal model outperforming the UMT baselines (§4).

Original: However, one may argue that since θ 1 and θ 2 are trained in a similar fashion, z will be the same as x , resulting in a duplicate pair.
Revised: However, one may argue that since θ 1 and θ 2 are trained in a similar fashion, z will be the same as x , resulting in a duplicate pair.

Original: In our experiments, on the contrary, the back-translated dataset contains only around 14% duplicates across different language pairs, as shown in our analysis on data diversity in §5.2.In Appendix §8.1, we provide a more generalized version of CBD with n ( ≥ 2 ) UMT agents, where we also analyze its effectiveness on the IWSLT translation tasks.
Revised: In our experiments, on the contrary, the back-translated dataset contains only around 14% duplicates across different language pairs, as shown in our analysis on data diversity in §5.2.In Appendix §8.1, we provide a more generalized version of CBD with n ( ≥ 2 ) UMT agents, where we also analyze its effectiveness on the IWSLT translation tasks.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We present our UMT experiments on the WMT tasks (§4.1) followed by IWSLT (§4.2).
Revised: We present our UMT experiments on the WMT tasks (§4.1) followed by IWSLT (§4.2).

Original: 4. WMT EXPERIMENTS
Revised: 4. WMT EXPERIMENTS

Original: Setup.
Revised: Setup.

Original: We use the default data setup and codebase of Conneau & Lample (2019)’s code repository 5 .
Revised: We use the default data setup and codebase of Conneau & Lample (2019)’s code repository 5 .

Original: Speciﬁcally, we use the News Crawl 2007-2008 datasets for English (En), French (Fr) and German (De), and News Crawl 2015 dataset for Romanian (Ro), and limit the total number of sentences per language to 5M.
Revised: Speciﬁcally, we use the News Crawl 2007-2008 datasets for English (En), French (Fr) and German (De), and News Crawl 2015 dataset for Romanian (Ro), and limit the total number of sentences per language to 5M.

Original: This results in a total 10M, 10M and 7M sentences of combined monolingual data for
Revised: This results in a total 10M, 10M and 7M sentences of combined monolingual data for

Original: En-Fr, En-De and En-Ro, respectively.
Revised: En-Fr, En-De and En-Ro, respectively.

Original: Note that in their paper, Lample et al.
Revised: Note that in their paper, Lample et al.

Original: (2018c)
Revised: (2018c)

Original: and Conneau & Lample (2019) conducted their UMT experiments on very large datasets of 274M, 509M and 195M sentences (combination of source and target monolingual data) for En-Fr, En-De and En-Ro respectively, which are derived from News Crawl 2007-2017 datasets.
Revised: and Conneau & Lample (2019) conducted their UMT experiments on very large datasets of 274M, 509M and 195M sentences (combination of source and target monolingual data) for En-Fr, En-De and En-Ro respectively, which are derived from News Crawl 2007-2017 datasets.

Original: Unfortunately, we were unable to reproduce these (very) large scale experiments because we encountered out-of-CPU-memory error when attempting to load the large datasets, even though our system has 128GB RAM.
Revised: Unfortunately, we were unable to reproduce these (very) large scale experiments because we encountered out-of-CPU-memory error when attempting to load the large datasets, even though our system has 128GB RAM.

Original: Instead, we rerun the NMT and PBSMT models (Lample et al., 2018c) and the pretrained XLM (Conneau &
Revised: Instead, we rerun the NMT and PBSMT models (Lample et al., 2018c) and the pretrained XLM (Conneau &

Original: Lample, 2019) on the default setup to acquire the corresponding NMT, PBSMT and XLM baselines.
Revised: Lample, 2019) on the default setup to acquire the corresponding NMT, PBSMT and XLM baselines.

Original: For the NMT models, we follow Lample et al.
Revised: For the NMT models, we follow Lample et al.

Original: (2018c) to train the UMT agents with a parametershared Transformer (Vaswani et al., 2017) that has 6 layers and 512 model dimensions and a batch size of 32 sentences.
Revised: (2018c) to train the UMT agents with a parametershared Transformer (Vaswani et al., 2017) that has 6 layers and 512 model dimensions and a batch size of 32 sentences.

Original: We use joint Byte-Pair Encoding (BPE) (Sennrich et al., 2016b) and train fastText (Bojanowski et al., 2017) on the BPE tokens to initialize the token embeddings.
Revised: We use joint Byte-Pair Encoding (BPE) (Sennrich et al., 2016b) and train fastText (Bojanowski et al., 2017) on the BPE tokens to initialize the token embeddings.

Original: For the
Revised: For the

Original: PBSMT (Koehn et al., 2003) models, following Lample et al.
Revised: PBSMT (Koehn et al., 2003) models, following Lample et al.

Original: (2018c)
Revised: (2018c)

Original: , we use MUSE (Lample et al., 2018b) to generate the initial phrase table and run 4 iterations of back-translation.
Revised: , we use MUSE (Lample et al., 2018b) to generate the initial phrase table and run 4 iterations of back-translation.

Original: We subsample 500K sentences from the 5M monolingual sentences at each iteration to train the PBSMT models.
Revised: We subsample 500K sentences from the 5M monolingual sentences at each iteration to train the PBSMT models.

Original: To ensure randomness in the PBSMT agents, we use different seeds for MUSE training and randomly subsample different sets of data during PBSMT training.
Revised: To ensure randomness in the PBSMT agents, we use different seeds for MUSE training and randomly subsample different sets of data during PBSMT training.

Original: For XLM (Conneau & Lample, 2019), we also follow their setup.
Revised: For XLM (Conneau & Lample, 2019), we also follow their setup.

Original: For each agent, we ﬁrst pretrain a big XLM model (6 layers, 1024 model dimensions) on the default 10M data with 4K tokens per batch.
Revised: For each agent, we ﬁrst pretrain a big XLM model (6 layers, 1024 model dimensions) on the default 10M data with 4K tokens per batch.

Original: Then we initialize the encoder-decoder system of the UMT agent with the XLM and train the model for 30 epochs.
Revised: Then we initialize the encoder-decoder system of the UMT agent with the XLM and train the model for 30 epochs.

Original: 6 The NMT and XLM agents are trained with a 4-GPU setup.
Revised: 6 The NMT and XLM agents are trained with a 4-GPU setup.

Original: We use the big Transformer (Ott et al., 2018) as the ﬁnal supervised model.
Revised: We use the big Transformer (Ott et al., 2018) as the ﬁnal supervised model.

Original: We choose the best model based on validation loss and use beam size of 5.
Revised: We choose the best model based on validation loss and use beam size of 5.

Original: [Table]
Revised: [Table]

Original: Results
Revised: Results

Original: [Table]
Revised: [Table]

Original: from previous papers on large scale datasets
Revised: from previous papers on large scale datasets

Original: [Table]
Revised: [Table]

Original: 2018c)
Revised: 2018c)

Original: [Table]
Revised: [Table]

Original: XLM
Revised: XLM

Original: [Table]
Revised: [Table]

Original: Conneau & Lample, 2019)
Revised: Conneau & Lample, 2019)

Original: [Table]
Revised: [Table]

Original: Results from our runs on smaller datasets
Revised: Results from our runs on smaller datasets

Original: [Table]
Revised: [Table]

Original: NMT (Lample et al., 2018c) 24.
Revised: NMT (Lample et al., 2018c) 24.

Original: + CBD 26.
Revised: + CBD 26.

Original: PBSMT (Lample et al., 2018c) 17.
Revised: PBSMT (Lample et al., 2018c) 17.

Original: + CBD 21.
Revised: + CBD 21.

Original: XLM (Conneau & Lample, 2019) 33.
Revised: XLM (Conneau & Lample, 2019) 33.

Original: + CBD 35.
Revised: + CBD 35.

Original: Results.
Revised: Results.

Original: Table 1 shows the experimental results of different UMT approaches with and without
Revised: Table 1 shows the experimental results of different UMT approaches with and without

Original: CBD.
Revised: CBD.

Original: First of all, with the datasets that are 30-50 times smaller than the ones used in previous papers, the baselines perform around 2 to 3 BLEU worse than the reported BLEU.
Revised: First of all, with the datasets that are 30-50 times smaller than the ones used in previous papers, the baselines perform around 2 to 3 BLEU worse than the reported BLEU.

Original: As it can be seen, the
Revised: As it can be seen, the

Original: CBD-enhanced model with the pretrained XLM achieves 35.3 and 33.0 BLEU on the WMT’14 En-Fr and Fr-En tasks respectively, which are 2.3 and 1.5 BLEU improvements over the baseline.
Revised: CBD-enhanced model with the pretrained XLM achieves 35.3 and 33.0 BLEU on the WMT’14 En-Fr and Fr-En tasks respectively, which are 2.3 and 1.5 BLEU improvements over the baseline.

Original: It also surpasses Conneau & Lample (2019) by 1.9 BLEU, despite the fact that the SOTA was trained with 274M sentences (compared to our setup of 10M sentences).
Revised: It also surpasses Conneau & Lample (2019) by 1.9 BLEU, despite the fact that the SOTA was trained with 274M sentences (compared to our setup of 10M sentences).

Original: CBD also boosts the scores for XLM in
Revised: CBD also boosts the scores for XLM in

Original: En-De, De-En, En-Ro, Ro-En by around 2.0 BLEU.
Revised: En-De, De-En, En-Ro, Ro-En by around 2.0 BLEU.

Original: For the NMT systems, CBD also outperforms the baselines by 1 to 2 BLEU.
Revised: For the NMT systems, CBD also outperforms the baselines by 1 to 2 BLEU.

Original: More interestingly, PBSMT models are known to be deterministic, but
Revised: More interestingly, PBSMT models are known to be deterministic, but

Original: CBD is still able to improve data diversity and provide performance boost by up to 4.0 BLEU points.
Revised: CBD is still able to improve data diversity and provide performance boost by up to 4.0 BLEU points.

Original: 4. IWSLT EXPERIMENTS
Revised: 4. IWSLT EXPERIMENTS

Original: We also demonstrate the effectiveness of CBD on relatively small datasets for IWSLT En-Fr and
Revised: We also demonstrate the effectiveness of CBD on relatively small datasets for IWSLT En-Fr and

Original: En-De translation tasks.
Revised: En-De translation tasks.

Original: The IWSLT’13 En-Fr dataset contains 200K sentences for each language.
Revised: The IWSLT’13 En-Fr dataset contains 200K sentences for each language.

Original: We use the IWSLT15.TED.tst2012 set for validation and the IWSLT15.TED.tst2013 set for testing.
Revised: We use the IWSLT15.TED.tst2012 set for validation and the IWSLT15.TED.tst2013 set for testing.

Original: The IWSLT’14 En-De dataset contains 160K sentences for each language.
Revised: The IWSLT’14 En-De dataset contains 160K sentences for each language.

Original: We split it into 95% for training and 5% for validation, and we use IWSLT14.TED.{dev2010, dev2012, tst2010,tst1011, tst2012} for testing.
Revised: We split it into 95% for training and 5% for validation, and we use IWSLT14.TED.{dev2010, dev2012, tst2010,tst1011, tst2012} for testing.

Original: For these experiments, we use the neural UMT method (Lample et al., 2018c) with a Transformer of 5 layers and 512 model dimensions, and trained using only 1 GPU.
Revised: For these experiments, we use the neural UMT method (Lample et al., 2018c) with a Transformer of 5 layers and 512 model dimensions, and trained using only 1 GPU.

Original: From the results in Table 2, we can see that CBD improves the performance in all the four tasks by 2-3 BLEU compared to the NMT baseline of Lample et al.
Revised: From the results in Table 2, we can see that CBD improves the performance in all the four tasks by 2-3 BLEU compared to the NMT baseline of Lample et al.

Original: (2018c).
Revised: (2018c).

Original: [Table]
Revised: [Table]

Original:  U NDERSTANDING CBD 5. C ROSS - MODEL BACK - TRANSLATION IS KEY
Revised:  U NDERSTANDING CBD 5. C ROSS - MODEL BACK - TRANSLATION IS KEY

Original: As mentioned, crucial to our strategy’s success is the cross-model back-translation, where the agent operating at the ﬁrst level must be different from the one in the second level.
Revised: As mentioned, crucial to our strategy’s success is the cross-model back-translation, where the agent operating at the ﬁrst level must be different from the one in the second level.

Original: To verify this, we compare CBD with similar variants that do not employ the cross-model element in the WMT tasks.
Revised: To verify this, we compare CBD with similar variants that do not employ the cross-model element in the WMT tasks.

Original: We refer to these variants commonly as back-translation distillation (BD).
Revised: We refer to these variants commonly as back-translation distillation (BD).

Original: The ﬁrst variant BD (1st level,1 agent) has only 1 UMT agent that translates the monolingual data only once and uses these synthetic parallel pairs to distill the model ˆ θ .
Revised: The ﬁrst variant BD (1st level,1 agent) has only 1 UMT agent that translates the monolingual data only once and uses these synthetic parallel pairs to distill the model ˆ θ .

Original: The second variant BD (1st level, 2 agents) employs
Revised: The second variant BD (1st level, 2 agents) employs

Original: UMT agents, similar to CBD, to produce 2 sets of synthetic parallel data from the monolingual data and uses both of them for distillation.
Revised: UMT agents, similar to CBD, to produce 2 sets of synthetic parallel data from the monolingual data and uses both of them for distillation.

Original: Finally, the third variant BD (2nd level, 2 agents) uses 2 UMT agents to sample translations from the monolingual data in forward and backward directions using the same respective agents.
Revised: Finally, the third variant BD (2nd level, 2 agents) uses 2 UMT agents to sample translations from the monolingual data in forward and backward directions using the same respective agents.

Original: In other words, BD (2nd level, 2 agents) follows similar procedures in
Revised: In other words, BD (2nd level, 2 agents) follows similar procedures in

Original: Algorithm 1, except that it optimizes the following loss with θ α being alternated between θ 1 and θ 2 :
Revised: Algorithm 1, except that it optimizes the following loss with θ α being alternated between θ 1 and θ 2 :

Original: [Equation]
Revised: [Equation]

Original: From the comparison in Table 3, we see that none of the BD variants noticeably improves the performance across the language pairs, while CBD provides consistent gains of 1.0-2.0 BLEU.
Revised: From the comparison in Table 3, we see that none of the BD variants noticeably improves the performance across the language pairs, while CBD provides consistent gains of 1.0-2.0 BLEU.

Original: In particular, the BD (1st level, 1 agent) variant fails to improve as the distilled model is trained on the same synthetic data that the UMT agent is already trained on.
Revised: In particular, the BD (1st level, 1 agent) variant fails to improve as the distilled model is trained on the same synthetic data that the UMT agent is already trained on.

Original: The variant BD (1st level, 2 agents) is in fact similar in sprit to (Nguyen et al., 2020), which improves supervised and semi-supervised
Revised: The variant BD (1st level, 2 agents) is in fact similar in sprit to (Nguyen et al., 2020), which improves supervised and semi-supervised

Original: MT.
Revised: MT.

Original: However, it fails to do so in the unsupervised setup, due to the lack of supervised agents.
Revised: However, it fails to do so in the unsupervised setup, due to the lack of supervised agents.

Original: The variant BD (2nd level, 2 agents) also fails because the 2nd level synthetic data is already optimized during iterative back-translation training of the UMT agents, leaving the distilled model with no extra information to exploit.
Revised: The variant BD (2nd level, 2 agents) also fails because the 2nd level synthetic data is already optimized during iterative back-translation training of the UMT agents, leaving the distilled model with no extra information to exploit.

Original: On the other hand, cross-model back-translation enables CBD to translate the second-level data by an agent other than the ﬁrst-level agent.
Revised: On the other hand, cross-model back-translation enables CBD to translate the second-level data by an agent other than the ﬁrst-level agent.

Original: In this strategy, the second agent produces targets that the ﬁrst agent is not aware of, while the second agent receives as input the sources that are foreign to it.
Revised: In this strategy, the second agent produces targets that the ﬁrst agent is not aware of, while the second agent receives as input the sources that are foreign to it.

Original: This process creates corrupted but new information, which the supervised MT model can leverage to improve the overall MT performance through more data diversity.
Revised: This process creates corrupted but new information, which the supervised MT model can leverage to improve the overall MT performance through more data diversity.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: CBD PRODUCES DIVERSE DATA
Revised: CBD PRODUCES DIVERSE DATA

Original: Having argued that cross-model back-translation creates extra information for the supervised MT model to leverage on
Revised: Having argued that cross-model back-translation creates extra information for the supervised MT model to leverage on

Original: [Table]
Revised: [Table]

Original: we hypothesize that such extra information can be measurable by the diversity of the generated data.
Revised: we hypothesize that such extra information can be measurable by the diversity of the generated data.

Original: To measure this
Revised: To measure this

Original: [Table]
Revised: [Table]

Original: we compute the reconstruction BLEU and compare the scores for BD and CBD (both 2nd level, 2 agents) in the WMT En-Fr, En-De and En-Ro tasks.
Revised: we compute the reconstruction BLEU and compare the scores for BD and CBD (both 2nd level, 2 agents) in the WMT En-Fr, En-De and En-Ro tasks.

Original: The scores are obtained by using the ﬁrst agent to translate the available monolingual data in language s to t and then the second agent to translate those translations back to language s .
Revised: The scores are obtained by using the ﬁrst agent to translate the available monolingual data in language s to t and then the second agent to translate those translations back to language s .

Original: After that, a BLEU score is measured by comparing the reconstructed text with the original text.
Revised: After that, a BLEU score is measured by comparing the reconstructed text with the original text.

Original: In BD, the ﬁrst and second agents are identical, while they are distinct for CBD.
Revised: In BD, the ﬁrst and second agents are identical, while they are distinct for CBD.

Original: From the results in Table 4, we observe that the reconstruction BLEU scores of CBD are more than 10 points lower than those of BD, indicating that the newly generated data by CBD are more diverse and different from the original data.
Revised: From the results in Table 4, we observe that the reconstruction BLEU scores of CBD are more than 10 points lower than those of BD, indicating that the newly generated data by CBD are more diverse and different from the original data.

Original: [Table]
Revised: [Table]

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we further report the ratio of duplicate source-target pairs to the amount of synthetic parallel data created by CBD.
Revised: we further report the ratio of duplicate source-target pairs to the amount of synthetic parallel data created by CBD.

Original: We sample 30M synthetic parallel data using the CBD strategy and examine the amount of duplicate pairs for the WMT En-Fr, En-De and En-Ro tasks.
Revised: We sample 30M synthetic parallel data using the CBD strategy and examine the amount of duplicate pairs for the WMT En-Fr, En-De and En-Ro tasks.

Original: We can notice that across the language pairs, only around 14% of the parallel data are duplicates.
Revised: We can notice that across the language pairs, only around 14% of the parallel data are duplicates.

Original: Given that only about 5M (3.5M for En-Ro) sentences are real sentences and the remaining 25M sentences are synthetic, this amount of duplicates is surprisingly low.
Revised: Given that only about 5M (3.5M for En-Ro) sentences are real sentences and the remaining 25M sentences are synthetic, this amount of duplicates is surprisingly low.

Original: This fact also explains why CBD is able to exploit extra information better than any standard UMT to improve the performance
Revised: This fact also explains why CBD is able to exploit extra information better than any standard UMT to improve the performance

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 5. C OMPARISON WITH ENSEMBLES OF MODELS AND ENSEMBLE KNOWLEDGE DISTILLATION
Revised: 5. C OMPARISON WITH ENSEMBLES OF MODELS AND ENSEMBLE KNOWLEDGE DISTILLATION

Original: Since CBD utilizes outputs from two UMT agents for supervised distillation, it is interesting to see how it performs compared to an ensemble of UMT models and ensemble knowledge distillation (Freitag et al., 2017).
Revised: Since CBD utilizes outputs from two UMT agents for supervised distillation, it is interesting to see how it performs compared to an ensemble of UMT models and ensemble knowledge distillation (Freitag et al., 2017).

Original: To perform ensembling, we average the probabilities of the two UMT agents at each decoding step.
Revised: To perform ensembling, we average the probabilities of the two UMT agents at each decoding step.

Original: For ensemble distillation, we generate synthetic parallel data from an ensemble of UMT agents, which is then used to train the supervised model.
Revised: For ensemble distillation, we generate synthetic parallel data from an ensemble of UMT agents, which is then used to train the supervised model.

Original: [Table]
Revised: [Table]

Original: From the results on the WMT translation tasks in Table
Revised: From the results on the WMT translation tasks in Table

Original: [Table]
Revised: [Table]

Original: we observe that ensembles of models improve the performance only by 0.5-1.0 BLEU, while CBD provides larger gains (1.0-
Revised: we observe that ensembles of models improve the performance only by 0.5-1.0 BLEU, while CBD provides larger gains (1.0-

Original: [Table]
Revised: [Table]

Original: 0 BLEU)
Revised: 0 BLEU)

Original: across all the tasks.
Revised: across all the tasks.

Original: These results demonstrate that CBD is capable of leveraging the potentials of multiple UMT agents better than how an ensemble of agents does.
Revised: These results demonstrate that CBD is capable of leveraging the potentials of multiple UMT agents better than how an ensemble of agents does.

Original: This is in contrast to data diversiﬁcation (Nguyen et al., 2020), which is shown to mimic and perform similarly to model ensembling.
Revised: This is in contrast to data diversiﬁcation (Nguyen et al., 2020), which is shown to mimic and perform similarly to model ensembling.

Original: More importantly, during inference, an ensemble of models requires more memory and computations (twice in this case) to store and execute multiple models.
Revised: More importantly, during inference, an ensemble of models requires more memory and computations (twice in this case) to store and execute multiple models.

Original: In contrast, CBD can throw away the UMT teacher agents after training and needs only one single model for inference.
Revised: In contrast, CBD can throw away the UMT teacher agents after training and needs only one single model for inference.

Original: Meanwhile, ensemble knowledge distillation (Freitag et al., 2017), which performs well with supervised agents, performs poorly in unsupervised MT tasks.
Revised: Meanwhile, ensemble knowledge distillation (Freitag et al., 2017), which performs well with supervised agents, performs poorly in unsupervised MT tasks.

Original: The reason could be that the UMT agents may not be suitable for the method originally intended for supervised learning.
Revised: The reason could be that the UMT agents may not be suitable for the method originally intended for supervised learning.

Original: Further inspection in Appendix §8.2 suggests that many samples in the ensemble translations contain incomprehensible repetitions.
Revised: Further inspection in Appendix §8.2 suggests that many samples in the ensemble translations contain incomprehensible repetitions.

Original: 5. C OMPARISON WITH OTHER POTENTIAL ALTERNATIVES
Revised: 5. C OMPARISON WITH OTHER POTENTIAL ALTERNATIVES

Original: In this section, we compare CBD with other alternatives in the text generation literature that also attempt to increase diversity.
Revised: In this section, we compare CBD with other alternatives in the text generation literature that also attempt to increase diversity.

Original: While many of these methods are generic, we adopt them in the UMT framework and compare their performance with our CBD technique in the WMT En-Fr, Fr-en, En-De, and De-en tasks, taking the XLM (Conneau & Lample, 2019) as the base model.
Revised: While many of these methods are generic, we adopt them in the UMT framework and compare their performance with our CBD technique in the WMT En-Fr, Fr-en, En-De, and De-en tasks, taking the XLM (Conneau & Lample, 2019) as the base model.

Original: One major group of alternatives is sampling based methods, where the model samples translations following multinomial distributions during iterative back-translation.
Revised: One major group of alternatives is sampling based methods, where the model samples translations following multinomial distributions during iterative back-translation.

Original: Speciﬁcally, we compare the
Revised: Speciﬁcally, we compare the

Original: CBD with ( i ) sampling with temperature 0.3 (Edunov et al., 2018; Fan et al., 2018), ( ii ) top-k sampling (Radford et al., 2019), and ( iii ) nucleus or top-p
Revised: CBD with ( i ) sampling with temperature 0.3 (Edunov et al., 2018; Fan et al., 2018), ( ii ) top-k sampling (Radford et al., 2019), and ( iii ) nucleus or top-p

Original: sampling (Holtzman et al., 2020).
Revised: sampling (Holtzman et al., 2020).

Original: Plus, we compare
Revised: Plus, we compare

Original: CBD with target noising , where we add random noises to the translations of the UMT model during iterative back-translation.
Revised: CBD with target noising , where we add random noises to the translations of the UMT model during iterative back-translation.

Original: Finally, multi-agent dual learning (Wang et al., 2019) is also considered as another alternative, where multiple unsupervised agents are used train the end supervised model.
Revised: Finally, multi-agent dual learning (Wang et al., 2019) is also considered as another alternative, where multiple unsupervised agents are used train the end supervised model.

Original: The results are reported in Table 7.
Revised: The results are reported in Table 7.

Original: We can see that while the sampling based methods indeed increase the diversity signiﬁcantly, they do not improve the performance as much as CBD does.
Revised: We can see that while the sampling based methods indeed increase the diversity signiﬁcantly, they do not improve the performance as much as CBD does.

Original: The reason could be that the extra data generated by (stochastic) sampling are noisy and their quality is not as good as deterministic predictions from the two UMT agents via cross-model back-translation.
Revised: The reason could be that the extra data generated by (stochastic) sampling are noisy and their quality is not as good as deterministic predictions from the two UMT agents via cross-model back-translation.

Original: On the other hand, target noising does not provide a consistent improvement while multi-agent dual learning achieves less impressive gains compared to CBD.
Revised: On the other hand, target noising does not provide a consistent improvement while multi-agent dual learning achieves less impressive gains compared to CBD.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: There has long been a desire to make use of vast monolingual data to boost MT quality. The ﬁrst step towards this direction is semi-supervised training.
Revised: The ﬁrst step towards utilizing the vast monolingual data to boost MT quality is through semisupervised training.

Original: Back-translation (Sennrich et al., 2016a; Edunov et al., 2018) is an effective approach to exploit target-side monolingual data.
Revised: Back-translation (Sennrich et al., 2016a; Edunov et al., 2018) is an effective approach to exploit target-side monolingual data.

Original: Hoang et al.

Original: (2018) propose iterative back-translation to further improve it.

Original: Dual learning (He et al., 2016; Wang et al., 2019), meawhile, is an attractive alternative where the backward and forward modelsare trained [Table] concurrently and intertwined with each other.
Revised: Dual learning (He et al., 2016; Wang et al., 2019), meanwhile, trains backward and forward models concurrently and intertwines them together.

Original: Recently, Zheng et al.
Revised: Recently, Zheng et al.

Original: (2020) proposed a variational method to couple the translation and language models  through a shared latent space to train from both parallel and monolingual data simultaneously.
Revised: (2020) proposed a variational method to couple the translation and language models [Table] through a shared latent space.

Original: There have also been attempts in solving low-resource translation problems with limited parallel data (Gu et al., 2018; Irvine & Callison-Burch, 2014;
Revised: There have also been attempts in solving low-resource translation problems with limited parallel data (Gu et al., 2018; Irvine & Callison-Burch, 2014;

Original: Guzmán et al., 2019).
Revised: Guzmán et al., 2019).

Original: In the realm of SMT, cross-lingual dictionaries have been leveraged to reduce parallel data reliance (Irvine & Callison-Burch, 2016; 2014; Klementiev et al., 2012)
Revised: In the realm of SMT

Original: [Table]
Revised: [Table]

Revised: cross-lingual dictionaries have been used to reduce parallel data reliance (Irvine & Callison-Burch, 2016; Klementiev et al., 2012).

Original: In recent years
Revised: In recent years

Original: [Table]
Revised: [Table]

Original: unsupervised word-translation via cross-lingual word embedding has seen a huge success (Lample et al., 2018b; Artetxe et al., 2017; 2018a).
Revised: unsupervised word-translation via cross-lingual word embedding has seen a huge success (Lample et al., 2018b; Artetxe et al., 2017; 2018a).

Original: This opened the door for unsupervised NMT methods that employ the three principles described in §2.
Revised: This opened the door for UMT methods that employ the three principles described in §2.

Original: Lample et al.
Revised: Lample et al.

Original: (2018a) and Artetxe et al.
Revised: (2018a) and Artetxe et al.

Original: (2018c) were among the ﬁrst of this kind, who use denoising autodecoder for language modeling and iterative back-translation.
Revised: (2018c) were among the ﬁrst of this kind, who use denoising autodecoder for language modeling and iterative back-translation.

Original: Lample et al.
Revised: Lample et al.

Original: (2018a) use MUSE (Lample et al., 2018b) as the initialization to bootstrap theUMT model, while Artetxe et al.
Revised: (2018a) use MUSE (Lample et al., 2018b) as the initialization to bootstrap the model, while Artetxe et al.

Original: (2018c) use cross-lingual word embeddings (Artetxe et al., 2017).
Revised: (2018c) use cross-lingual word embeddings (Artetxe et al., 2017).

Original: Lample et al.
Revised: Lample et al.

Original: (2018c) laterestablished the three principles formally and suggested the use of BPE (Sennrich et al., 2016b) and fastText (Bojanowski et al., 2017) to initialize theUMT model and the sharing of all parameters.
Revised: (2018c) later suggested the use of BPE (Sennrich et al., 2016b) and fastText (Bojanowski et al., 2017) to initialize the model and the parameters sharing.

Original: Pretrained language models (Devlin et al., 2019) are then used to initialize the Transformer encoder and decoder (Conneau & Lample, 2019).
Revised: Pretrained language models (Devlin et al., 2019) are then used to initialize the entire network (Conneau & Lample, 2019).

Original: Song et al.
Revised: Song et al.

Original: (2019) proposed to pretrain an encoder-decoder model while Artetxe et al.
Revised: (2019) proposed to pretrain an encoder-decoder model while Artetxe et al.

Original: (2018b) suggested a combination of PBSMT and NMT with subword information.
Revised: (2018b) suggested a combination of PBSMT and NMT with subword information.

Original: 7 Plus, pretraining BART (Lewis et al., 2020) on multi-lingual corpora improves the initialization process (Liu et al., 2020).
Revised: 7 Plus, pretraining BART (Lewis et al., 2020) on multi-lingual corpora improves the initialization process (Liu et al., 2020).

Original: Our proposed CBD works outside the three-principle UMT framework and is considered as an add-on to any underlying UMT system.
Revised: Our proposed CBD works outside the three-principle UMT framework and is considered as an add-on to any underlying UMT system.

Original: There exist some relevant approaches to CBD.
Revised: There exist some relevant approaches to CBD.

Original: First, it is similar to Nguyen et al.
Revised: First, it is similar to Nguyen et al.

Original: (2020), which generates a diverse set of data from multiple supervised MT agents.
Revised: (2020), which generates a diverse set of data from multiple supervised MT agents.

Original: Despite being effective in supervised and semi-supervised settings, a direct implementation of it in UMT underperforms due to lack of supervised signals (§5.1).
Revised: Despite being effective in supervised and semi-supervised settings, a direct implementation of it in UMT underperforms due to lack of supervised signals (§5.1).

Original: In order to successfully exploit unsupervised agents, CBD requires cross-model back-translation which is the key to its effectiveness.
Revised: In order to successfully exploit unsupervised agents, CBD requires cross-model back-translation which is the key to its effectiveness.

Original: Second, CBD can be viewed as an augmentation technique (Fadaee et al., 2017; Wang et al., 2018).
Revised: Second, CBD can be viewed as an augmentation technique (Fadaee et al., 2017; Wang et al., 2018).

Original: Although the denoising autoencoding built in a typical UMT system also performs augmentation, the noising process is rather naive, while CBD augments data by well-trained agents.
Revised: Although the denoising autoencoding built in a typical UMT system also performs augmentation, the noising process is rather naive, while CBD augments data by well-trained agents.

Original: Third, CBD is related to ensemble knowledge distillation (Kim & Rush, 2016; Freitag et al., 2017).
Revised: Third, CBD is related to ensemble knowledge distillation (Kim & Rush, 2016; Freitag et al., 2017).

Revised: These methods use multiple models to perform ensemble inference to generate one-way synthetic targets from the original source data, which are then used to distill the ﬁnal model.

Original: However, these distillation schemes only apply to supervised settings and the agents are used to jointly produce high quality translations rather than diverse candidates.
Revised: However, these distillation schemes only apply to supervised settings and the agents are used to jointly produce high quality translations rather than diverse candidates.

Original: Multi-agent dual learning (Wang et al., 2019)is also relevant as it seeks to minimize the reconstruction losses using multiple agents.
Revised: Similar to our method, multi-agent dual learning (Wang et al., 2019) also uses multiple models in both forward and backward directions, but to minimize the reconstruction losses instead of to generate diverse synthetic data.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have proposed cross-model back-translated distillation (CBD) - a method that works outside the three existing principles for unsupervised MT and is applicable to any UMT methods.
Revised: We have proposed cross-model back-translated distillation (CBD) - a method that works outside the three existing principles for unsupervised MT and is applicable to any UMT methods.

Original: CBD improves the performance of various underlying UMT approaches in the WMT’14 English-French, WMT’
Revised: CBD improves the performance of various underlying UMT approaches in the WMT’14 English-French, WMT’

Original: English-German and English-Romanian translation tasks by 1.0-2.0 BLEU.
Revised: English-German and English-Romanian translation tasks by 1.0-2.0 BLEU.

Original: It also outperforms the baselines in the IWSLT’14 German-English and IWSLT’13 English-French tasks by up to 3.0 BLEU.
Revised: It also outperforms the baselines in the IWSLT’14 German-English and IWSLT’13 English-French tasks by up to 3.0 BLEU.

Original: Our analysis shows that CBD embraces data diversity and extracts more model-speciﬁc intrinsic information than what an ensemble of models would do.
Revised: Our analysis shows that CBD embraces data diversity and extracts more model-speciﬁc intrinsic information than what an ensemble of models would do.

Original:  A PPENDIX
Revised:  A PPENDIX

Original: In the following supplementary material, we provide the generalized version of our method crossmodel back-translated distillation, or GCBD, and measure its effectiveness in the IWSLT EnglishGerman, German-English, English-French and French-English unsupervised tasks (§8.1).
Revised: In the following supplementary material, we provide the generalized version of our method crossmodel back-translated distillation, or GCBD, and measure its effectiveness in the IWSLT EnglishGerman, German-English, English-French and French-English unsupervised tasks (§8.1).

Original: In addition, we investigate why ensemble knowledge distillation (Freitag et al., 2017), which boosts the performance in a supervised setup, fails to do so in an unsupervised setup where we replace the supervised agents used in the method with the UMT agents (§8.2).
Revised: In addition, we investigate why ensemble knowledge distillation (Freitag et al., 2017), which boosts the performance in a supervised setup, fails to do so in an unsupervised setup where we replace the supervised agents used in the method with the UMT agents (§8.2).

Original: Lastly, We also show that our method is uneffected by the translationese effect (Edunov et al., 2020) (§8.3).
Revised: Lastly, We also show that our method is uneffected by the translationese effect (Edunov et al., 2020) (§8.3).

Original: 8. G ENERALIZED VERSION
Revised: 8. G ENERALIZED VERSION

Original: In this section, we describe a generalized version of our CBD, which involves multiple UMT agents instead of just two.
Revised: In this section, we describe a generalized version of our CBD, which involves multiple UMT agents instead of just two.

Original: Then, we test this method in the IWSLT experiments to demonstrate its effectiveness and characteristics.
Revised: Then, we test this method in the IWSLT experiments to demonstrate its effectiveness and characteristics.

Original: Speciﬁcally, in addition to the input monolingual data X s and X t of languages s and t and the supervised model ˆ θ , we introduce another hyper-parameter n to indicate the number of unsupervised agents used to perform cross-model back-translation.
Revised: Speciﬁcally, in addition to the input monolingual data X s and X t of languages s and t and the supervised model ˆ θ , we introduce another hyper-parameter n to indicate the number of unsupervised agents used to perform cross-model back-translation.

Original: The generalized cross-model back-translated distillation (GCBD) strategy is presented in Algorithm 2.
Revised: The generalized cross-model back-translated distillation (GCBD) strategy is presented in Algorithm 2.

Original: In this method, instead of training only two agents, the method trains a set of n UMT agents Θ = { θ 1 , ..., θ n } .
Revised: In this method, instead of training only two agents, the method trains a set of n UMT agents Θ = { θ 1 , ..., θ n } .

Original: During training, we iteratively select two orderly distinct agents θ i and θ j from Θ and use them to perform cross-model back-translation and train the model ˆ θ .
Revised: During training, we iteratively select two orderly distinct agents θ i and θ j from Θ and use them to perform cross-model back-translation and train the model ˆ θ .

Original: To evaluate GCBD in comparison with CBD, we conduct experiments with the IWSLT’13 EnglishFrench (En-Fr) and IWSLT’14 English-German (En-De) tasks.
Revised: To evaluate GCBD in comparison with CBD, we conduct experiments with the IWSLT’13 EnglishFrench (En-Fr) and IWSLT’14 English-German (En-De) tasks.

Original: The setup for these experiments are identical to the experiment in §4.2, except that we vary the hyper-parameter n = (2 , 3 , 4) to determine the optimal number of agents.
Revised: The setup for these experiments are identical to the experiment in §4.2, except that we vary the hyper-parameter n = (2 , 3 , 4) to determine the optimal number of agents.

Original: The results are reported in Table 8.
Revised: The results are reported in Table 8.

Original: As it can be seen, increasing the number of agents n to 3 adds an additional 0 .
Revised: As it can be seen, increasing the number of agents n to 3 adds an additional 0 .

Original: 4 − 1 . 

Revised: 4 − 1 .

Original: BLEU improvement compared to the standard CBD.
Revised: BLEU improvement compared to the standard CBD.

Original: Moreover, using 4 UMT agents does not improve the performance over using just 3 UMT, despite that this setup still outperforms the standard CBD.
Revised: Moreover, using 4 UMT agents does not improve the performance over using just 3 UMT, despite that this setup still outperforms the standard CBD.

Original: The results indicate that increasing the system complexity further is not always optimal and diminishing return is observed as we add more agents to the system.
Revised: The results indicate that increasing the system complexity further is not always optimal and diminishing return is observed as we add more agents to the system.

Original: [Table]
Revised: [Table]

Original: + GCBD ( n = 2 ) or CBD
Revised: + GCBD ( n = 2 ) or CBD

Original: [Table]
Revised: [Table]

Original: + GCBD ( n = 3 )
Revised: + GCBD ( n = 3 )

Original: [Table]
Revised: [Table]

Original: + GCBD ( n = 4 )
Revised: + GCBD ( n = 4 )

Original: [Table]
Revised: [Table]

Original: 8. A NALYSIS OF DEGENERATION IN ENSEMBLE KNOWLEDGE DISTILLATION
Revised: 8. A NALYSIS OF DEGENERATION IN ENSEMBLE KNOWLEDGE DISTILLATION

Original: Ensemble knowledge distillation (Freitag et al., 2017) has been used to enhance supervised machine translation.
Revised: Ensemble knowledge distillation (Freitag et al., 2017) has been used to enhance supervised machine translation.

Original: It uses multiple strong (supervised) teachers to generate synthetic parallel data from both sides of the parallel corpora by averaging the decoding probabilities of the teachers at each step.
Revised: It uses multiple strong (supervised) teachers to generate synthetic parallel data from both sides of the parallel corpora by averaging the decoding probabilities of the teachers at each step.

Original: The synthetic data are then used to train the student model.
Revised: The synthetic data are then used to train the student model.

Original: Having seen its effectiveness in the supervised setup, we apply this same tactic to unsupervised MT tasks by replacing the supervised teachers with unsupervised MT agents.
Revised: Having seen its effectiveness in the supervised setup, we apply this same tactic to unsupervised MT tasks by replacing the supervised teachers with unsupervised MT agents.

Original: However, the method surprisingly causes drastic performance drop in the
Revised: However, the method surprisingly causes drastic performance drop in the

Original: WMT’14 En-Fr, WMT’16 En-De and En-Ro unsupervised MT tasks.
Revised: WMT’14 En-Fr, WMT’16 En-De and En-Ro unsupervised MT tasks.

Original: By manual inspection, we found that many instances of the synthetic data are incomprehensible and contain repetitions, which is a degeneration behavior.
Revised: By manual inspection, we found that many instances of the synthetic data are incomprehensible and contain repetitions, which is a degeneration behavior.

Original: We then quantitatively measure the percentage of sentences in the synthetic data containing tri-gram repetitions by counting the number of sentences where a word/sub-word is repeated at least three consecutive times.
Revised: We then quantitatively measure the percentage of sentences in the synthetic data containing tri-gram repetitions by counting the number of sentences where a word/sub-word is repeated at least three consecutive times.

Original: As reported in Table 9, from 30% to 86% of the synthetic data generated by the ensemble knowledge distillation method are incomprehensible and contain repetitions.
Revised: As reported in Table 9, from 30% to 86% of the synthetic data generated by the ensemble knowledge distillation method are incomprehensible and contain repetitions.

Original: Relative to the performance of CBD, the performance drop in ensemble distillation is also more dramatic for language pairs with higher percentage of degeneration (En-Ro and En-De).
Revised: Relative to the performance of CBD, the performance drop in ensemble distillation is also more dramatic for language pairs with higher percentage of degeneration (En-Ro and En-De).

Original: This explains why the downstream student model fails to learn from these corrupted data.
Revised: This explains why the downstream student model fails to learn from these corrupted data.

Original: The results indicate that UMT agents are unable to jointly translate through ensembling strategy the monolingual data that they were trained on.
Revised: The results indicate that UMT agents are unable to jointly translate through ensembling strategy the monolingual data that they were trained on.

Original: This phenomenon may require further research to be fully understood.
Revised: This phenomenon may require further research to be fully understood.

Original: On the other hand, with less than 0.1% tri-gram repetitions, CBD generates little to no repetitions, which partly explains why it is able to improve the performance.
Revised: On the other hand, with less than 0.1% tri-gram repetitions, CBD generates little to no repetitions, which partly explains why it is able to improve the performance.

Original: tasks.
Revised: tasks.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: T HE TRANSLATIONESE EFFECT
Revised: T HE TRANSLATIONESE EFFECT

Original: It can be seen that our cross-model back-translation method is indeed a modiﬁed version of backtranslation (Sennrich et al., 2016a).
Revised: It can be seen that our cross-model back-translation method is indeed a modiﬁed version of backtranslation (Sennrich et al., 2016a).

Original: Therefore, it is necessary to test if this method suffer from the translationese effect (Edunov et al., 2020).
Revised: Therefore, it is necessary to test if this method suffer from the translationese effect (Edunov et al., 2020).

Original: As pointed out in their work, back-translation only shows performance gains with translationese source sentences but does not improve when the sentences are natural text.
Revised: As pointed out in their work, back-translation only shows performance gains with translationese source sentences but does not improve when the sentences are natural text.

Original: 8 Nguyen et al.
Revised: 8 Nguyen et al.

Original: (2020) shows that the translationese effect only exhibits in semisupervised setup, where there are both parallel and monolingual data.
Revised: (2020) shows that the translationese effect only exhibits in semisupervised setup, where there are both parallel and monolingual data.

Original: However, while they show that their supervised back-translation technique is not impacted by the translationese effect, they left out the question whether unsupervised counterparts are affected.
Revised: However, while they show that their supervised back-translation technique is not impacted by the translationese effect, they left out the question whether unsupervised counterparts are affected.

Original: Therefore, we test our unsupervised CBD method against the translationese effect by conducting the same experiment.
Revised: Therefore, we test our unsupervised CBD method against the translationese effect by conducting the same experiment.

Original: More precisely, we compare the BLEU scores of our method versus the XLM baseline (Conneau & Lample, 2019) in the WMT’14 English-German test sets in the three setups devised by Edunov et al.
Revised: More precisely, we compare the BLEU scores of our method versus the XLM baseline (Conneau & Lample, 2019) in the WMT’14 English-German test sets in the three setups devised by Edunov et al.

Original: • Natural source → translationese target ( X → Y ∗ ).
Revised: • Natural source → translationese target ( X → Y ∗ ).

Original: • Translationese source → natural target ( X ∗ → Y ) • Translationese of translationese of source to translationese of target ( X ∗∗ → Y ∗ ).
Revised: • Translationese source → natural target ( X ∗ → Y ) • Translationese of translationese of source to translationese of target ( X ∗∗ → Y ∗ ).

Original: Table 10 shows that our method outperforms the baseline signiﬁcantly in the natural source → translationese target scenario ( X → Y ∗ ), while it may not improve the translationese source scenario ( X ∗ → Y ) considerably.
Revised: Table 10 shows that our method outperforms the baseline signiﬁcantly in the natural source → translationese target scenario ( X → Y ∗ ), while it may not improve the translationese source scenario ( X ∗ → Y ) considerably.

Original: The results demonstrate that our method behaves differently than what the translationese effect indicates.
Revised: The results demonstrate that our method behaves differently than what the translationese effect indicates.

Original: More importantly, the translations of the natural source sentences are improved, which indicates the practical usefulness of our method.
Revised: More importantly, the translations of the natural source sentences are improved, which indicates the practical usefulness of our method.

Original: Furthermore, in combination with the ﬁndings in Nguyen et al.
Revised: Furthermore, in combination with the ﬁndings in Nguyen et al.

Original: , the experiment shows that the translationese effect may only exhibit in the semi-supervised setup, but not in supervised or unsupervised setups.
Revised: , the experiment shows that the translationese effect may only exhibit in the semi-supervised setup, but not in supervised or unsupervised setups.

Original: [Table]
Revised: [Table]