Original: I NCORPORATING BERT INTO N EURAL M ACHINE T RANSLATION
Revised: I NCORPORATING BERT INTO N EURAL M ACHINE T RANSLATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The recently proposed BERT (Devlin et al., 2019) has shown great power on a variety of natural language understanding tasks, such as text classiﬁcation, reading comprehension, etc.
Revised: The recently proposed BERT (Devlin et al., 2019) has shown great power on a variety of natural language understanding tasks, such as text classiﬁcation, reading comprehension, etc.

Original: However, how to effectively apply BERT to neural machine translation (NMT) lacks enough exploration.
Revised: However, how to effectively apply BERT to neural machine translation (NMT) lacks enough exploration.

Original: While BERT is more commonly used as ﬁne-tuning instead of contextual embedding for downstream language understanding tasks, in NMT, our preliminary exploration of using BERT as contextual embedding is better than using for ﬁne-tuning.
Revised: While BERT is more commonly used as ﬁne-tuning instead of contextual embedding for downstream language understanding tasks, in NMT, our preliminary exploration of using BERT as contextual embedding is better than using for ﬁne-tuning.

Original: This motivates us to think how to better leverage BERT for NMT along this direction.
Revised: This motivates us to think how to better leverage BERT for NMT along this direction.

Original: We propose a new algorithm named BERT-fused model, in which we ﬁrst use BERT to extract representations for an input sequence, and then the representations are fused with each layer of the encoder and decoder of the NMT model through attention mechanisms.
Revised: We propose a new algorithm named BERT-fused model, in which we ﬁrst use BERT to extract representations for an input sequence, and then the representations are fused with each layer of the encoder and decoder of the NMT model through attention mechanisms.

Original: We conduct experiments on supervised (including sentence-level and document-level translations), semi-supervised and unsupervised machine translation, and achieve state-of-the-art results on seven benchmark datasets.
Revised: We conduct experiments on supervised (including sentence-level and document-level translations), semi-supervised and unsupervised machine translation, and achieve state-of-the-art results on seven benchmark datasets.

Original: Our code is available atan anonymous Github page https://github.com/bert-nmt/bert-nmt .
Revised: Our code is available at https://github.com/bert-nmt/bert-nmt .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently, pre-training techniques, like ELMo (Peters et al., 2018), GPT/GPT-2 (Radford et al., 2018; 2019), BERT (Devlin et al., 2019), cross-lingual language model (brieﬂy, XLM) (Lample & Conneau, 2019),
Revised: Recently, pre-training techniques, like ELMo (Peters et al., 2018), GPT/GPT-2 (Radford et al., 2018; 2019), BERT (Devlin et al., 2019), cross-lingual language model (brieﬂy, XLM) (Lample & Conneau, 2019),

Original: XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019) have attracted more and more attention in machine learning and natural language processing communities.
Revised: XLNet (Yang et al., 2019b) and RoBERTa (Liu et al., 2019) have attracted more and more attention in machine learning and natural language processing communities.

Original: The models are ﬁrst pre-trained on large amount of unlabeled data to capture rich representations of the input, and then applied to the downstream tasks by either providing context-aware embeddings of an input sequence (Peters et al., 2018), or initializing the parameters of the downstream model (Devlin et al., 2019) for ﬁne-tuning.
Revised: The models are ﬁrst pre-trained on large amount of unlabeled data to capture rich representations of the input, and then applied to the downstream tasks by either providing context-aware embeddings of an input sequence (Peters et al., 2018), or initializing the parameters of the downstream model (Devlin et al., 2019) for ﬁne-tuning.

Original: Such pre-training approaches lead to signiﬁcant improvements on natural language understanding tasks.
Revised: Such pre-training approaches lead to signiﬁcant improvements on natural language understanding tasks.

Original: Among them, BERT is one of the most powerful techniques that inspires lots of variants like XLNet, XLM, RoBERTa and achieves state-of-the-art results for many language understanding tasks including reading comprehension, text classiﬁcation, etc (Devlin et al., 2019).
Revised: Among them, BERT is one of the most powerful techniques that inspires lots of variants like XLNet, XLM, RoBERTa and achieves state-of-the-art results for many language understanding tasks including reading comprehension, text classiﬁcation, etc (Devlin et al., 2019).

Original: Neural Machine Translation (NMT) aims to translate an input sequence from a source language to a target language.
Revised: Neural Machine Translation (NMT) aims to translate an input sequence from a source language to a target language.

Original: An NMT model usually consists of an encoder to map an input sequence to hidden representations, and a decoder to decode hidden representations to generate a sentence in the target language.
Revised: An NMT model usually consists of an encoder to map an input sequence to hidden representations, and a decoder to decode hidden representations to generate a sentence in the target language.

Original: Given that BERT has achieved great success in language understanding tasks, a question worthy studying is how to incorporate BERT to improve NMT.
Revised: Given that BERT has achieved great success in language understanding tasks, a question worthy studying is how to incorporate BERT to improve NMT.

Original: Due to the computation resource limitation, training a BERT model from scratch is unaffordable for many researchers.
Revised: Due to the computation resource limitation, training a BERT model from scratch is unaffordable for many researchers.

Original: Thus, we focus on the setting of leveraging a pre-trained BERT model (instead of training a BERT model from scratch) for NMT.
Revised: Thus, we focus on the setting of leveraging a pre-trained BERT model (instead of training a BERT model from scratch) for NMT.

Original: Given that there is few work leveraging BERT for NMT, our ﬁrst attempt is to try two previous strategies: (1) using BERT to initialize downstream models and then ﬁne-tuning the models, and (2) using BERT as context-aware embeddings for downstream models.
Revised: Given that there is limited work leveraging BERT for NMT, our ﬁrst attempt is to try two previous strategies: (1) using BERT to initialize downstream models and then ﬁne-tuning the models, and (2) using BERT as context-aware embeddings for downstream models.

Original: For the ﬁrst strategy, following Devlin et al.
Revised: For the ﬁrst strategy, following Devlin et al.

Original: (2019), we initialize the encoder of an NMT model with a pre-trained BERT model, and then ﬁnetune the NMT model on the downstream datasets.
Revised: (2019), we initialize the encoder of an NMT model with a pre-trained BERT model, and then ﬁnetune the NMT model on the downstream datasets.

Original: Unfortunately, we did not observe signiﬁcant improvement.
Revised: Unfortunately, we did not observe signiﬁcant improvement.

Original: Using a pre-trained XLM (Lample & Conneau, 2019) model, a variant of
Revised: Using a pre-trained XLM (Lample & Conneau, 2019) model, a variant of

Original: BERT for machine translation, to warm up an NMT model is another choice.
Revised: BERT for machine translation, to warm up an NMT model is another choice.

Original: XLM has been veriﬁed to be helpful for WMT’16 Romanian-to-English translation.
Revised: XLM has been veriﬁed to be helpful for WMT’16 Romanian-to-English translation.

Original: But when applied to a language domain beyond the corpus for training XLM (such as IWSLT dataset (Cettolo et al., 2014), which is about spoken languages) or when large bilingual data is available for downstream tasks, no signiﬁcant improvement is observed neither.
Revised: But when applied to a language domain beyond the corpus for training XLM (such as IWSLT dataset (Cettolo et al., 2014), which is about spoken languages) or when large bilingual data is available for downstream tasks, no signiﬁcant improvement is observed neither.

Original: For the second strategy, following the practice of (Peters et al., 2018), we use BERT to provide context-aware embeddings for the NMT model.
Revised: For the second strategy, following the practice of (Peters et al., 2018), we use BERT to provide context-aware embeddings for the NMT model.

Original: We ﬁnd that this strategy outperforms the ﬁrst one (please refer to Section 3 for more details).
Revised: We ﬁnd that this strategy outperforms the ﬁrst one (please refer to Section 3 for more details).

Original: This motivates us to go along this direction and design more effective algorithms.
Revised: This motivates us to go along this direction and design more effective algorithms.

Original: We propose a new algorithm, BERT-fused model , in which we exploit the representation from BERT by feeding it into all layers rather than served as input embeddings only.
Revised: We propose a new algorithm, BERT-fused model , in which we exploit the representation from BERT by feeding it into all layers rather than served as input embeddings only.

Original: We use the attention mechanism to adaptively control how each layer interacts with the representations, and deal with the case that BERT module and NMT module might use different word segmentation rules, resulting in different sequence (i.e., representation) lengths.
Revised: We use the attention mechanism to adaptively control how each layer interacts with the representations, and deal with the case that BERT module and NMT module might use different word segmentation rules, resulting in different sequence (i.e., representation) lengths.

Original: Compared to standard NMT, in addition to BERT, there are two extra attention modules, the BERT-encoder attention and BERT-decoder attention.
Revised: Compared to standard NMT, in addition to BERT, there are two extra attention modules, the BERT-encoder attention and BERT-decoder attention.

Original: An input sequence is ﬁrst transformed into representations processed by BERT.
Revised: An input sequence is ﬁrst transformed into representations processed by BERT.

Original: Then, by the BERTencoder attention module, each NMT encoder layer interacts with the representations obtained from
Revised: Then, by the BERTencoder attention module, each NMT encoder layer interacts with the representations obtained from

Original: BERT and eventually outputs fused representations leveraging both BERT and the NMT encoder.
Revised: BERT and eventually outputs fused representations leveraging both BERT and the NMT encoder.

Original: The decoder works similarly and fuses BERT representations and NMT encoder representations.
Revised: The decoder works similarly and fuses BERT representations and NMT encoder representations.

Original: We conduct 14 experiments on various NMT tasks to verify our approach, including supervised, semi-supervised and unsupervised settings.
Revised: We conduct 14 experiments on various NMT tasks to verify our approach, including supervised, semi-supervised and unsupervised settings.

Original: For supervised NMT, we work on ﬁve tasks of IWSLT datasets and two WMT datasets.
Revised: For supervised NMT, we work on ﬁve tasks of IWSLT datasets and two WMT datasets.

Original: Speciﬁcally, we achieve 36 .
Revised: Speciﬁcally, we achieve 36 .

Original: BLEU score on IWSLT’14 Germanto-English translation, setting a new record on this task.
Revised: BLEU score on IWSLT’14 Germanto-English translation, setting a new record on this task.

Original: We also work on two document-level translations of IWSLT, and further boost the BLEU score of German-to-English translation to 36 . 69 .
Revised: We also work on two document-level translations of IWSLT, and further boost the BLEU score of German-to-English translation to 36 . 69 .

Original: On WMT’14 datasets, we achieve 30 .
Revised: On WMT’14 datasets, we achieve 30 .

Original: 75 BLEU score on English-to-German translation and 43 .
Revised: 75 BLEU score on English-to-German translation and 43 .

Original: 78 on
Revised: 78 on

Original: English-to-French translation, signiﬁcantly better over the baselines.
Revised: English-to-French translation, signiﬁcantly better over the baselines.

Original: For semi-supervised NMT, we boost BLEU scores of WMT’16 Romanian-to-English translation with back translation (Sennrich et al., 2016b), a classic semi-supervised algorithm, from 37 .
Revised: For semi-supervised NMT, we boost BLEU scores of WMT’16 Romanian-to-English translation with back translation (Sennrich et al., 2016b), a classic semi-supervised algorithm, from 37 .

Original: 73 to 39 .
Revised: 73 to 39 .

Original: 10 , achieving the best result on this task.
Revised: 10 , achieving the best result on this task.

Original: Finally, we verify our algorithm on unsupervised English ↔ French and unsupervised
Revised: Finally, we verify our algorithm on unsupervised English ↔ French and unsupervised

Original: English ↔ Romanian translations and also achieve state-of-the-art results.
Revised: English ↔ Romanian translations and also achieve state-of-the-art results.

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND AND RELATED WORK

Original: We brieﬂy introduce the background of NMT and review current pre-training techniques.
Revised: We brieﬂy introduce the background of NMT and review current pre-training techniques.

Original: NMT aims to translate an input sentence from the source language to the target one.
Revised: NMT aims to translate an input sentence from the source language to the target one.

Original: An NMT model usually consists of an encoder, a decoder and an attention module.
Revised: An NMT model usually consists of an encoder, a decoder and an attention module.

Original: The encoder maps the input sequence to hidden representations and the decoder maps the hidden representations to the target sequence.
Revised: The encoder maps the input sequence to hidden representations and the decoder maps the hidden representations to the target sequence.

Original: The attention module is ﬁrst introduced by Bahdanau et al.
Revised: The attention module is ﬁrst introduced by Bahdanau et al.

Original: (2015), which is used to better align source words and target words.
Revised: (2015), which is used to better align source words and target words.

Original: The encoder and decoder can be specialized as
Revised: The encoder and decoder can be specialized as

Original: LSTM (Hochreiter & Schmidhuber, 1997; Sutskever et al., 2014; Wu et al., 2016), CNN (Gehring et al., 2017) and Transformer (Vaswani et al., 2017).
Revised: LSTM (Hochreiter & Schmidhuber, 1997; Sutskever et al., 2014; Wu et al., 2016), CNN (Gehring et al., 2017) and Transformer (Vaswani et al., 2017).

Original: A Transformer layer consists of three sublayers, a self-attention layer that processes sequential data taking the context of each timestep into consideration, an optional encoder-decoder attention layer that bridges the input sequence and target sequence which exists in decoder only), and a feed-forward layer for non-linear transformation.
Revised: A Transformer layer consists of three sublayers, a self-attention layer that processes sequential data taking the context of each timestep into consideration, an optional encoder-decoder attention layer that bridges the input sequence and target sequence which exists in decoder only, and a feed-forward layer for non-linear transformation.

Original: Transformer achieves the state-of-the-art results for NMT (Barrault et al., 2019).
Revised: Transformer achieves the state-of-the-art results for NMT (Barrault et al., 2019).

Original: In this work, we will use Transformer as the basic architecture of our model.
Revised: In this work, we will use Transformer as the basic architecture of our model.

Original: Pre-training has a long history in machine learning and natural language processing (Erhan et al., 2009; 2010).
Revised: Pre-training has a long history in machine learning and natural language processing (Erhan et al., 2009; 2010).

Original: Mikolov et al.
Revised: Mikolov et al.

Original: and Pennington et al.
Revised: and Pennington et al.

Original: (2014) proposed to use distributional representations (i.e., word embeddings) for individual words.
Revised: (2014) proposed to use distributional representations (i.e., word embeddings) for individual words.

Original: Dai & Le (2015) proposed to train a language model or an auto-encoder with unlabeled data and then leveraged the obtained model to ﬁnetune downstream tasks.
Revised: Dai & Le (2015) proposed to train a language model or an auto-encoder with unlabeled data and then leveraged the obtained model to ﬁnetune downstream tasks.

Original: Pre-training has attracted more and more attention in recent years and achieved great improvements when the data scale becomes large and deep neural networks are employed.
Revised: Pre-training has attracted more and more attention in recent years and achieved great improvements when the data scale becomes large and deep neural networks are employed.

Original: ELMo was proposed in Peters et al.
Revised: ELMo was proposed in Peters et al.

Original: based on bidirectional LSTMs and its pre-trained models are fed into downstream tasks as context-aware inputs.
Revised: based on bidirectional LSTMs and its pre-trained models are fed into downstream tasks as context-aware inputs.

Original: In GPT (Radford et al., 2018), a Transformer based language model is pre-trained on unlabeled dataset and then ﬁnetuned on downstream tasks.
Revised: In GPT (Radford et al., 2018), a Transformer based language model is pre-trained on unlabeled dataset and then ﬁnetuned on downstream tasks.

Original: BERT (Devlin et al., 2019) is one of the widely adopted pre-training approach for model initialization.
Revised: BERT (Devlin et al., 2019) is one of the widely adopted pre-training approach for model initialization.

Original: The architecture of BERT is the encoder of Transformer (Vaswani et al., 2017).
Revised: The architecture of BERT is the encoder of Transformer (Vaswani et al., 2017).

Original: Two kinds of objective functions are used in BERT training: (1) Masked language modeling (MLM) , where 15% words in a sentence are masked and BERT is trained to predict them with their surrounding words.
Revised: Two kinds of objective functions are used in BERT training: (1) Masked language modeling (MLM) , where 15% words in a sentence are masked and BERT is trained to predict them with their surrounding words.

Original: (2) Next sentence prediction (NSP) : Another task of pre-training BERT is to predict whether two input sequences are adjacent.
Revised: (2) Next sentence prediction (NSP) : Another task of pre-training BERT is to predict whether two input sequences are adjacent.

Original: For this purpose, the training corpus consists of tuples ( [cls] , input 1, [sep] , input 2, [sep] ), with learnable special tokens [cls] to classify whether input 1 and input 2 are adjacent and [sep] to segment two sentences, and with probability 50%, the second input is replaced with a random input.
Revised: For this purpose, the training corpus consists of tuples ( [cls] , input 1, [sep] , input 2, [sep] ), with learnable special tokens [cls] to classify whether input 1 and input 2 are adjacent and [sep] to segment two sentences, and with probability 50%, the second input is replaced with a random input.

Original: Variants of BERT have been proposed:
Revised: Variants of BERT have been proposed:

Original: In XLM (Lample & Conneau, 2019), the model is pre-trained based on multiple languages and NSP task is removed; in RoBERTa (Liu et al., 2019), more unlabeled data is leveraged without
Revised: In XLM (Lample & Conneau, 2019), the model is pre-trained based on multiple languages and NSP task is removed; in RoBERTa (Liu et al., 2019), more unlabeled data is leveraged without

Original: NSP task neither; in XLNet (Yang et al., 2019), a permutation based modeling is introduced.
Revised: NSP task neither; in XLNet (Yang et al., 2019b), a permutation based modeling is introduced.

Revised: Recently, there are some works leveraging knowledge distillation to combine pre-trained models with

Revised: NMT (Yang et al., 2019a; Chen et al., 2019).

Original:  A PRELIMINARY EXPLORATION
Revised:  A PRELIMINARY EXPLORATION

Original: While a few pieces of work (Lample & Conneau, 2019; Song et al., 2019) design speciﬁc pretraining methods for NMT, they are time and resource consuming given that they need to pre-train large models from scratch using large-scale data, and even one model for each language pair.
Revised: While a few pieces of work (Lample & Conneau, 2019; Song et al., 2019) design speciﬁc pretraining methods for NMT, they are time and resource consuming given that they need to pre-train large models from scratch using large-scale data, and even one model for each language pair.

Original: In this work, we focus on the setting of using a pre-trained BERT model.
Revised: In this work, we focus on the setting of using a pre-trained BERT model.

Original: Detailed model download links can be found in Appendix D.
Revised: Detailed model download links can be found in Appendix D.

Original: Considering that pre-trained models have been utilized in two different ways for other natural language tasks, it is straightforward to try them for NMT.
Revised: Considering that pre-trained models have been utilized in two different ways for other natural language tasks, it is straightforward to try them for NMT.

Original: Following previous practice, we make the following attempts.
Revised: Following previous practice, we make the following attempts

Original: (I) Use pre-trained models to initialize the NMT model.
Revised: (I) Use pre-trained models to initialize the NMT model.

Original: There are different implementations for this approach.
Revised: There are different implementations for this approach.

Original: (1) Following (Devlin et al., 2019), we initialize the encoder of an NMT model with a pretrained BERT.
Revised: (1) Following (Devlin et al., 2019), we initialize the encoder of an NMT model with a pretrained BERT.

Original: (2) Following (Lample & Conneau, 2019), we initialize the encoder and/or decoder of an NMT model with XLM.
Revised: (2) Following (Lample & Conneau, 2019), we initialize the encoder and/or decoder of an NMT model with XLM.

Original: (II) Use pre-trained models as inputs to the NMT model.
Revised: (II) Use pre-trained models as inputs to the NMT model.

Original: Inspired from (Peters et al., 2018), we feed the outputs of the last layer of BERT to an NMT model as its inputs.
Revised: Inspired from (Peters et al., 2018), we feed the outputs of the last layer of BERT to an NMT model as its inputs.

Original: We conduct experiments on the IWSLT’14 English → German translation, a widely adopted dataset for machine translation consisting of 160 k labeled sentence pairs.
Revised: We conduct experiments on the IWSLT’14 English → German translation, a widely adopted dataset for machine translation consisting of 160 k labeled sentence pairs.

Original: We choose Transformer (Vaswani et al., 2017) as the basic model architecture with transformer iwslt de en conﬁguration (a six-layer model with 36 .
Revised: We choose Transformer (Vaswani et al., 2017) as the basic model architecture with transformer iwslt de en conﬁguration (a six-layer model with 36 .

Original: 7 M parameters).
Revised: 7 M parameters).

Original: The translation quality is evaluated by BLEU (Papineni et al., 2002) score; the larger, the better.
Revised: The translation quality is evaluated by BLEU (Papineni et al., 2002) score; the larger, the better.

Original: Both BERT base and XLM models are pre-trained and we get them from the Web.
Revised: Both BERT base and XLM models are pre-trained and we get them from the Web.

Original: More details about the experimental settings are included in Appendix A.2.
Revised: More details about the experimental settings are included in Appendix A.2.

Original: [Table]
Revised: [Table]

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: We have several observations: (1) Using BERT to initialize the encoder of NMT can only achieve 27 .
Revised: We have several observations: (1) Using BERT to initialize the encoder of NMT can only achieve 27 .

Original: 14 BLEU score, which is even worse than standard Transformer without using BERT.
Revised: 14 BLEU score, which is even worse than standard Transformer without using BERT.

Original: That is, simply using BERT to warm up an NMT model is not a good choice.
Revised: That is, simply using BERT to warm up an NMT model is not a good choice.

Original: (2) Using XLM to initialize the encoder or decoder respectively, we get 28 .
Revised: (2) Using XLM to initialize the encoder or decoder respectively, we get 28 .

Original: 22 or 26 .
Revised: 22 or 26 .

Original: 13 BLEU score, which does not outperform the baseline.
Revised: 13 BLEU score, which does not outperform the baseline.

Original: If both modules are initialized with XLM, the BLEU score is boosted to 28 . 99 , slightly outperforming the baseline.
Revised: If both modules are initialized with XLM, the BLEU score is boosted to 28 . 99 , slightly outperforming the baseline.

Original: Although XLM achieved great success on WMT’16 Romanian-to-English, we get limited improvement here.
Revised: Although XLM achieved great success on WMT’16 Romanian-to-English, we get limited improvement here.

Original: Our conjecture is that the
Revised: Our conjecture is that the

Original: XLM model is pre-trained on news data, which is out-of-domain for IWSLT dataset mainly about spoken languages and thus, leading to limited improvement.
Revised: XLM model is pre-trained on news data, which is out-of-domain for IWSLT dataset mainly about spoken languages and thus, leading to limited improvement.

Original: (3) When using the output of BERT as context-aware embeddings of the encoder, we achieve 29 .
Revised: (3) When using the output of BERT as context-aware embeddings of the encoder, we achieve 29 .

Original: 67 BLEU, much better than using pretrained models for initialization.
Revised: 67 BLEU, much better than using pretrained models for initialization.

Original: This shows that leveraging BERT as a feature provider is more effective in NMT.
Revised: This shows that leveraging BERT as a feature provider is more effective in NMT.

Original: This motivates us to take one step further and study how to fully exploit such features provided by pre-trained BERT models.
Revised: This motivates us to take one step further and study how to fully exploit such features provided by pre-trained BERT models.

Original:  A LGORITHM
Revised:  A LGORITHM

Original: In this section, we ﬁrst deﬁne the necessary notations, then introduce our proposed BERT-fused model and ﬁnally provide discussions with existing works.
Revised: In this section, we ﬁrst deﬁne the necessary notations, then introduce our proposed BERT-fused model and ﬁnally provide discussions with existing works.

Original: Notations Let X and Y denote the source language domain and target language domain respectively, which are the collections of sentences with the corresponding languages.
Revised: Notations Let X and Y denote the source language domain and target language domain respectively, which are the collections of sentences with the corresponding languages.

Original: For any sentence x ∈ X and y ∈ Y , let l x and l y denote the number of units (e.g., words or sub-words) in x and y .
Revised: For any sentence x ∈ X and y ∈ Y , let l x and l y denote the number of units (e.g., words or sub-words) in x and y .

Original: The i -th unit in x / y is denoted as x i / y i .
Revised: The i -th unit in x / y is denoted as x i / y i .

Original: Denote the encoder, decoder and BERT as Enc , Dec and BERT respectively.
Revised: Denote the encoder, decoder and BERT as Enc , Dec and BERT respectively.

Original: For ease of reference, we call the encoder and decoder in our work as the NMT module .
Revised: For ease of reference, we call the encoder and decoder in our work as the NMT module .

Original: W.l.o.g.
Revised: W.l.o.g.

Original: , we assume both the encoder and decoder consists of L layers.
Revised: , we assume both the encoder and decoder consists of L layers.

Original: Let attn ( q, K, V ) denote the attention layer, where q , K and V indicate query, key and value respectively (Vaswani et al., 2017).
Revised: Let attn ( q, K, V ) denote the attention layer, where q , K and V indicate query, key and value respectively (Vaswani et al., 2017).

Original: We use the same feed-forward layer as that used in (Vaswani et al., 2017) and denote it as
Revised: We use the same feed-forward layer as that used in (Vaswani et al., 2017) and denote it as

Original: FFN .
Revised: FFN .

Original: Mathematical formulations of the above layers are left at Appendix E.
Revised: Mathematical formulations of the above layers are left at Appendix E.

Original: 4. BERT- FUSED MODEL
Revised: 4. BERT- FUSED MODEL

Original: An illustration of our algorithm is shown in Figure 1.
Revised: An illustration of our algorithm is shown in Figure 1.

Original: Any input x ∈ X is progressively processed by the BERT, encoder and decoder.
Revised: Any input x ∈ X is progressively processed by the BERT, encoder and decoder.

Original: [Figure]
Revised: [Figure]

Original: Step-1 : Given any input x ∈ X , BERT ﬁrst encodes it into representation H B = BERT ( x ) .
Revised: Step-1 : Given any input x ∈ X , BERT ﬁrst encodes it into representation H B = BERT ( x ) .

Original: H B is the output of the last layer in BERT .
Revised: H B is the output of the last layer in BERT .

Original: The h B,i ∈ H B is the representation of the i -th wordpiece in x .
Revised: The h B,i ∈ H B is the representation of the i -th wordpiece in x .

Original: Step-2 : Let H lE denote the hidden representation of l -th layer in the encoder, and let H 0 E denote word embedding of sequence x .
Revised: Step-2 : Let H lE denote the hidden representation of l -th layer in the encoder, and let H 0 E denote word embedding of sequence x .

Original: Denote the i -th element in H lE as h li for any i ∈
Revised: Denote the i -th element in H lE as h li for any i ∈

Original: [ l x ] .
Revised: [ l x ] .

Original: In the l -th layer, l ∈
Revised: In the l -th layer, l ∈

Original: [ L ] ,
Revised: [ L ] ,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where attn S and attn B are attention models (see Eqn.(6)) with different parameters.
Revised: where attn S and attn B are attention models (see Eqn.(6)) with different parameters.

Original: Then each ˜ h li is further processed by FFN ( · ) deﬁned in Eqn.(7) and we get the output of the l -th layer: H lE =
Revised: Then each ˜ h li is further processed by FFN ( · ) deﬁned in Eqn.(7) and we get the output of the l -th layer: H lE =

Original: ( FFN (˜ h l 1 ) , · · · , FFN (˜ h ll x )) .
Revised: ( FFN (˜ h l 1 ) , · · · , FFN (˜ h ll x )) .

Original: The encoder will eventually output H LE from the last layer.
Revised: The encoder will eventually output H LE from the last layer.

Original: Step-3 : Let S l<t denote the hidden state of l -th layer in the decoder preceding time step t , i.e.,
Revised: Step-3 : Let S l<t denote the hidden state of l -th layer in the decoder preceding time step t , i.e.,

Original: S l <t =
Revised: S l<t =

Original: ( s l 1 , · · · , s lt − 1 ) .
Revised: ( s l 1 , · · · , s lt − 1 ) .

Original: Note s 01 is a special token indicating the start of a sequence, and s 0 t is the embedding of the predicted word at time-step t − 1 .
Revised: Note s 01 is a special token indicating the start of a sequence, and s 0 t is the embedding of the predicted word at time-step t − 1 .

Original: At the l -th layer, we have
Revised: At the l -th layer, we have

Original: [Equation]
Revised: [Equation]

Original: The attn S , attn B and attn E represent self-attention model, BERT-decoder attention model and encoder-decoder attention model respectively.
Revised: The attn S , attn B and attn E represent self-attention model, BERT-decoder attention model and encoder-decoder attention model respectively.

Original: Eqn.(2) iterates over layers and we can eventually obtain s Lt .
Revised: Eqn.(2) iterates over layers and we can eventually obtain s Lt .

Original: Finally s Lt is mapped via a linear transformation and softmax to get the t -th predicted word ˆ y t .
Revised: Finally s Lt is mapped via a linear transformation and softmax to get the t -th predicted word ˆ y t .

Original: The decoding process continues until meeting the end-of-sentence token.
Revised: The decoding process continues until meeting the end-of-sentence token.

Original: In our framework, the output of BERT serves as an external sequence representation, and we use an attention model to incorporate it into the NMT model.
Revised: In our framework, the output of BERT serves as an external sequence representation, and we use an attention model to incorporate it into the NMT model.

Original: This is a general way to leverage the pre-trained model regardless of the tokenization way.
Revised: This is a general way to leverage the pre-trained model regardless of the tokenization way.

Original: 4. D ROP - NET TRICK
Revised: 4. D ROP - NET TRICK

Original: Inspired by dropout (Srivastava et al., 2014), which can regularize the network training, we propose a drop-net trick to ensure that the features output by BERT and the conventional encoder are fully utilized.
Revised: Inspired by dropout (Srivastava et al., 2014) and drop-path (Larsson et al., 2017), which can regular- ize the network training, we propose a drop-net trick to ensure that the features output by BERT and the conventional encoder are fully utilized.

Original: The drop-net will effect Eqn.(1) and Eqn.(2).
Revised: The drop-net will effect Eqn.(1) and Eqn.(2).

Original: Denote the drop-net rate as p net ∈ [0 , 1] .
Revised: Denote the drop-net rate as p net ∈ [0 , 1] .

Original: At each training iteration, for any layer l , we uniformly sample a random variable U l from [0 , 1] , then all the ˜ h li in Eqn.(1) are calculated in the following way:
Revised: At each training iteration, for any layer l , we uniformly sample a random variable U l from [0 , 1] , then all the ˜ h li in Eqn.(1) are calculated in the following way:

Original: [Equation]
Revised: [Equation]

Original: where I ( · ) is the indicator function.
Revised: where I ( · ) is the indicator function.

Original: For any layer, with probability p net / 2 , either the BERT-encoder attention or self-attention is used only; w.p. (1 − p net ) , both the two attention models are used.
Revised: For any layer, with probability p net / 2 , either the BERT-encoder attention or self-attention is used only; w.p. (1 − p net ) , both the two attention models are used.

Original: For example, at a speciﬁc iteration, the ﬁrst layer might uses attn S only while the second layer uses attn B only.
Revised: For example, at a speciﬁc iteration, the ﬁrst layer might uses attn S only while the second layer uses attn B only.

Original: During inference time, the expected output of each attention model is used, which is
Revised: During inference time, the expected output of each attention model is used, which is

Original: E U ∼ uniform [0 , 1] (˜ h li, drop-net ) .
Revised: E U ∼ uniform [0 , 1] (˜ h li, drop-net ) .

Original: The expectation is exactly Eqn.(1).
Revised: The expectation is exactly Eqn.(1).

Original: Similarly, for training of the decoder, with the drop-net trick, we have
Revised: Similarly, for training of the decoder, with the drop-net trick, we have

Original: [Equation]
Revised: [Equation]

Original: For inference, it is calculated in the same way as Eqn.(2).
Revised: For inference, it is calculated in the same way as Eqn.(2).

Original: Using this technique can prevent network from overﬁtting (see the second part of Section 6 for more details).
Revised: Using this technique can prevent network from overﬁtting (see the second part of Section 6 for more details).

Original: 4. D ISCUSSION
Revised: 4. D ISCUSSION

Original: Comparison with ELMo As introduced in Section 2, ELMo (Peters et al., 2018) provides a contextaware embeddings for the encoder in order to capture richer information of the input sequence.
Revised: Comparison with ELMo As introduced in Section 2, ELMo (Peters et al., 2018) provides a contextaware embeddings for the encoder in order to capture richer information of the input sequence.

Original: Our approach is a more effective way of leveraging the features from the pre-trained model: (1) The output features of the pre-trained model are fused in all layers of the NMT module, ensuring the well-pre-trained features are fully exploited; (2) We use the attention model to bridge the NMT module and the pre-trained features of BERT, in which the NMT module can adaptively determine how to leverage the features from BERT.
Revised: Our approach is a more effective way of leveraging the features from the pre-trained model: (1) The output features of the pre-trained model are fused in all layers of the NMT module, ensuring the well-pre-trained features are fully exploited; (2) We use the attention model to bridge the NMT module and the pre-trained features of BERT, in which the NMT module can adaptively determine how to leverage the features from BERT.

Original: Limitations We are aware that our approach has several limitations.
Revised: Limitations We are aware that our approach has several limitations.

Original: (1) Additional storage cost: our approach leverages a BERT model, which results in additional storage cost.
Revised: (1) Additional storage cost: our approach leverages a BERT model, which results in additional storage cost.

Original: However, considering the BLEU improvement and the fact that we do not need additional training of BERT, we believe that the additional storage is acceptable.
Revised: However, considering the BLEU improvement and the fact that we do not need additional training of BERT, we believe that the additional storage is acceptable.

Original: (2) Additional inference time: We use BERT to encode the input sequence, which takes about 45% additional time (see Appendix C for details).
Revised: (2) Additional inference time: We use BERT to encode the input sequence, which takes about 45% additional time (see Appendix C for details).

Original: We will leave the improvement of the above two limitations as future work.
Revised: We will leave the improvement of the above two limitations as future work.

Original:  A PPLICATION TO SUPERVISED NMT AND SEMI - SUPERVISED NMT
Revised:  A PPLICATION TO SUPERVISED NMT AND SEMI - SUPERVISED NMT

Original: We ﬁrst verify our BERT-fused model on the supervised setting, including low-resource and richresource scenarios.
Revised: We ﬁrst verify our BERT-fused model on the supervised setting, including low-resource and richresource scenarios.

Original: Then we conduct experiments on document-level translation to verify our ap- proach.
Revised: Then we conduct experiments on document-level translation to verify our approach.

Original: Finally, we combine BERT-fused model with back translation (Sennrich et al., 2016b) to verify the effectiveness of our method on semi-supervised NMT.
Revised: Finally, we combine BERT-fused model with back translation (Sennrich et al., 2016b) to verify the effectiveness of our method on semi-supervised NMT.

Original: 5. S ETTINGS
Revised: 5. S ETTINGS

Original: Dataset For the low-resource scenario, we choose IWSLT’14 English ↔ German
Revised: Dataset For the low-resource scenario, we choose IWSLT’14 English ↔ German

Original: (En ↔ De),
Revised: (En ↔ De),

Original: English → Spanish (En → Es), IWSLT’17 English → French (En → Fr) and English → Chinese (En → Zh) translation.
Revised: English → Spanish (En → Es), IWSLT’17 English → French (En → Fr) and English → Chinese (En → Zh) translation.

Original: There are 160 k , 183 k , 236 k , 235 k bilingual sentence pairs for En ↔ De,
Revised: There are 160 k , 183 k , 236 k , 235 k bilingual sentence pairs for En ↔ De,

Original: En → Es, En → Fr and En → Zh tasks.
Revised: En → Es, En → Fr and En → Zh tasks.

Original: Following the common practice (Edunov et al., 2018), for
Revised: Following the common practice (Edunov et al., 2018), for

Original: En ↔ De, we lowercase all words.
Revised: En ↔ De, we lowercase all words.

Original: All sentences are preprocessed by BPE (Sennrich et al., 2016c).
Revised: All sentences are preprocessed by BPE (Sennrich et al., 2016c).

Original: The model conﬁguration is transformer iwslt de en , representing a six-layer model with embedding size 512 and FFN layer dimension 1024 .
Revised: The model conﬁguration is transformer iwslt de en , representing a six-layer model with embedding size 512 and FFN layer dimension 1024 .

Original: For the rich-resource scenario, we work on
Revised: For the rich-resource scenario, we work on

Original: WMT’

Revised: WMT’

Original: En → De and En
Revised: En → De and En

Original: → Fr, whose corpus sizes are 4 .
Revised: → Fr, whose corpus sizes are 4 .

Original: 5 M and 36 M respectively.
Revised: 5 M and 36 M respectively.

Original: We concatenate newstest2012 and newstest2013 as the validation set and use newstest2014 as the test set.
Revised: We concatenate newstest2012 and newstest2013 as the validation set and use newstest2014 as the test set.

Original: The model conﬁguration is transformer big , another six-layer network with embedding sizeand FFN layer dimension 4096 .
Revised: The model conﬁguration is transformer big , another six-layer network with embedding sizeand FFN layer dimension 4096 .

Original: More details about data and model are left in Appendix A.1.
Revised: More details about data and model are left in Appendix A.1.

Original: We choose BERT base for IWSLT tasks and BERT large for WMT tasks, which can ensure that the dimension of the BERT and NMT model almost match.
Revised: We choose BERT base for IWSLT tasks and BERT large for WMT tasks, which can ensure that the dimension of the BERT and NMT model almost match.

Original: The BERT models are ﬁxed during training.
Revised: The BERT models are ﬁxed during training.

Original: Detailed BERT information for each task is in Appendix D.
Revised: Detailed BERT information for each task is in Appendix D.

Original: The drop-net rate p net is set as 1 . 0 .
Revised: The drop-net rate p net is set as 1 . 0 .

Original: Training Strategy We ﬁrst train an NMT model until convergence, then initialize the encoder and decoder of the BERT-fused model with the obtained model.
Revised: Training Strategy We ﬁrst train an NMT model until convergence, then initialize the encoder and decoder of the BERT-fused model with the obtained model.

Original: The BERT-encoder attention and BERTdecoder attention are randomly initialized.
Revised: The BERT-encoder attention and BERTdecoder attention are randomly initialized.

Original: Experiments on IWSLT and WMT tasks are conducted on 1 and 8 M40 GPUs respectively.
Revised: Experiments on IWSLT and WMT tasks are conducted on 1 and 8 M40 GPUs respectively.

Original: The batchsize is 4 k tokens per GPU.
Revised: The batchsize is 4 k tokens per GPU.

Original: Following (Ott et al., 2018), for WMT tasks, we accumulate the gradient for 16 iterations and then update to simulate a 128 -GPU environment.
Revised: Following (Ott et al., 2018), for WMT tasks, we accumulate the gradient for 16 iterations and then update to simulate a 128 -GPU environment.

Original: It takes 1 , 8 and 14 days to obtain the pre-trained NMT models, and additional 1 ,and 10 days to ﬁnish the whole training process.
Revised: It takes 1 , 8 and 14 days to obtain the pre-trained NMT models, and additional 1 ,and 10 days to ﬁnish the whole training process.

Original: The optimization algorithm is Adam (Kingma &
Revised: The optimization algorithm is Adam (Kingma &

Original: Ba, 2014) with initial learning rate 0 . 0005 and inverse sqrt learning rate scheduler (Vaswani et al., 2017).
Revised: Ba, 2014) with initial learning rate 0 . 0005 and inverse sqrt learning rate scheduler (Vaswani et al., 2017).

Original: For WMT’

Revised: For WMT’

Original: En → De, we use beam search with width 4 and length penalty 0 .
Revised: En → De, we use beam search with width 4 and length penalty 0 .

Original: 6 for inference following (Vaswani et al., 2017).
Revised: 6 for inference following (Vaswani et al., 2017).

Original: For other tasks, we use width 5 and length penalty 1 . 0 .
Revised: For other tasks, we use width 5 and length penalty 1 . 0 .

Original: Evaluation We use multi-bleu.perl to evaluate IWSLT’14 En ↔ De and WMT translation tasks for fair comparison with previous work.
Revised: Evaluation We use multi-bleu.perl to evaluate IWSLT’14 En ↔ De and WMT translation tasks for fair comparison with previous work.

Original: For the remaining tasks, we use a more advance implementation of BLEU score, sacreBLEU for evaluation.
Revised: For the remaining tasks, we use a more advance implementation of BLEU score, sacreBLEU for evaluation.

Original: Script urls are in Appendix A.1.
Revised: Script urls are in Appendix A.1.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: The results of IWSLT translation tasks are reported in Table 2.
Revised: The results of IWSLT translation tasks are reported in Table 2.

Original: We implemented standard Transformer as baseline.
Revised: We implemented standard Transformer as baseline.

Original: Our proposed BERT-fused model can improve the BLEU scores of the ﬁve tasks by 1 . 88 , 1 . 47 , 2 . 4 , 1 . 9 and 2 . 8 points respectively, demonstrating the effectiveness of our method.
Revised: Our proposed BERT-fused model can improve the BLEU scores of the ﬁve tasks by 1 . 88 , 1 . 47 , 2 . 4 , 1 . 9 and 2 . 8 points respectively, demonstrating the effectiveness of our method.

Original: The consistent improvements on various tasks shows that our method works well for low-resource translations.
Revised: The consistent improvements on various tasks shows that our method works well for low-resource translations.

Original: We achieved state-of-the-art results on IWSLT’14 De
Revised: We achieved state-of-the-art results on IWSLT’14 De

Original: → En translation, a widely investigated baseline in machine translation.
Revised: → En translation, a widely investigated baseline in machine translation.

Original: The comparison with previous methods are shown in
Revised: The comparison with previous methods are shown in

Original: Appendix B.4 due to space limitation.
Revised: Appendix B.4 due to space limitation.

Original: [Table]
Revised: [Table]

Original: The results of WMT’14 En → De and En → Fr are shown in Table 3.
Revised: The results of WMT’14 En → De and En → Fr are shown in Table 3.

Original: Our reproduced Transformer matches the results reported in Ott et al.
Revised: Our reproduced Transformer matches the results reported in Ott et al.

Original: and we can see that our BERT-fused model can improve these two numbers to 30 .
Revised: and we can see that our BERT-fused model can improve these two numbers to 30 .

Original: 75 and 43 .
Revised: 75 and 43 .

Original: 78 , achieving 1 . 63 and 0 .
Revised: 78 , achieving 1 . 63 and 0 .

Original: 82 points improvement.
Revised: 82 points improvement.

Original: Our approach also outperforms the well-designed model DynamicConv (Wu et al., 2019) and a model obtained through neural architecture search (So et al., 2019).
Revised: Our approach also outperforms the well-designed model DynamicConv (Wu et al., 2019) and a model obtained through neural architecture search (So et al., 2019) [Table] [Table]

Original: 5. T RANSLATION WITH DOCUMENT - LEVEL CONTEXTUAL INFORMATION
Revised: 5. T RANSLATION WITH DOCUMENT - LEVEL CONTEXTUAL INFORMATION

Original: BERT is able to capture the relation between two sentences, since the next sentence prediction (NSP) task is to predict whether two sentences are adjacent.
Revised: BERT is able to capture the relation between two sentences, since the next sentence prediction (NSP) task is to predict whether two sentences are adjacent.

Original: We can leverage this property to improve translation with document-level contextual information (Miculicich et al., 2018), which is brieﬂy denoted as document-level translation.
Revised: We can leverage this property to improve translation with document-level contextual information (Miculicich et al., 2018), which is brieﬂy denoted as document-level translation.

Original: The inputs are a couple of sentences extracted from a para- [Table] graph/document, x d 1 , x d 2 , · · · , x dT , where the T x ’s are contextually correlated.
Revised: The inputs are a couple of sentences extracted from a paragraph/document, x d 1 , x d 2 , · · · , x dT , where the T x ’s are contextually correlated.

Original: We want to translate them into target language by considering the contextual information.
Revised: We want to translate them into target language by considering the contextual information.

Original: Algorithm
Revised: Algorithm

Original: In our implementation, to translate a sentence x to target domain, we leverage the contextual information by taking both x and its preceding sentence x prev as inputs.
Revised: In our implementation, to translate a sentence x to target domain, we leverage the contextual information by taking both x and its preceding sentence x prev as inputs.

Original: x is fed into Enc , which is the same as sentence-level translation.
Revised: x is fed into Enc , which is the same as sentence-level translation.

Original: For the input of BERT , it is the concatenation of two sequences: ( [cls] , x prev , [sep] , x , [sep] ), where both [cls] and [sep] are special tokens of BERT .
Revised: For the input of BERT , it is the concatenation of two sequences: ( [cls] , x prev , [sep] , x , [sep] ), where both [cls] and [sep] are special tokens of BERT .

Original: Setting We use IWSLT’14 En ↔ De dataset as introduced in Section 5.1.
Revised: Setting We use IWSLT’14 En ↔ De dataset as introduced in Section 5.1.

Original: The data is a collection of TED talks, where each talk consists of several sequences.
Revised: The data is a collection of TED talks, where each talk consists of several sequences.

Original: We can extract the adjacent sentences for training, validation and test sets.
Revised: We can extract the adjacent sentences for training, validation and test sets.

Original: The training strategy, hyperparameter selection and evaluation metric are the same for sentence-level translation.
Revised: The training strategy, hyperparameter selection and evaluation metric are the same for sentence-level translation.

Original: [Table]
Revised: [Table]

Original: Baselines We use two baselines here.
Revised: Baselines We use two baselines here.

Original: (1) To demonstrate how BERT works in our model, we replace BERT by a Transformer with conﬁguration transformer iwslt de en , which is randomly initialized and jointly trained.
Revised: (1) To demonstrate how BERT works in our model, we replace BERT by a Transformer with conﬁguration transformer iwslt de en , which is randomly initialized and jointly trained.

Original: Another baseline is proposed by Miculicich et al.
Revised: Another baseline is proposed by Miculicich et al.

Original: (2018), where multiple preceding sentences in a document are leveraged using a hierarchical attention network.
Revised: (2018), where multiple preceding sentences in a document are leveraged using a hierarchical attention network.

Original: Results The results are shown in Table 4.
Revised: The results are shown in Table 4.

Original: We can see that introducing contextual information from an additional encoder can boost the sentence-level baselines, but the improvement is limited ( 0 .for
Revised: We can see that introducing contextual information from an additional encoder can boost the sentence-level baselines, but the improvement is limited ( 0 .for

Original: En → De and 0 .
Revised: En → De and 0 .

Original: 31 for De → En).
Revised: 31 for De → En).

Original: For Miculicich et al.
Revised: For Miculicich et al.

Original: (2018), the best results we obtain are 27 .and
Revised: (2018), the best results we obtain are 27 .and

Original: 33 . 97 respectively, which are worse than the sentence-level baselines.
Revised: 33 . 97 respectively, which are worse than the sentence-level baselines.

Original: Combining BERT-fused model and document-level information, we can eventually achieve 31 .
Revised: Combining BERT-fused model and document-level information, we can eventually achieve 31 .

Original: 02 for En → De and 36 .
Revised: 02 for En → De and 36 .

Original: 69 for
Revised: 69 for

Original: De → En.
Revised: De → En.

Original: We perform signiﬁcant test 1 between sentence-level and document-level translation.

Original: Our document-level BERT-fused model signiﬁcantly outperforms sentence-level baseline with p -value less than 0 .
Revised: We document-level BERT-fused model signiﬁcantly outperforms sentence-level baseline with p -value less than 0 .

Original: This shows that our approach not only works for sentence-level translation, but can also be generalized to document-level translation.
Revised: This shows that our approach not only works for sentence-level translation, but can also be generalized to document-level translation.

Original: 5. A PPLICATION TO SEMI - SUPERVISED NMT
Revised: 5. A PPLICATION TO SEMI - SUPERVISED NMT

Original: We work on WMT’16 Romanian → English (Ro → En) translation to verify whether our approach can still make improvement over back translation (Sennrich et al., 2016b), the standard and powerful semi-supervised way to leverage monolingual data in NMT.
Revised: We work on WMT’16 Romanian → English (Ro → En) translation to verify whether our approach can still make improvement over back translation (Sennrich et al., 2016b), the standard and powerful semi-supervised way to leverage monolingual data in NMT.

Original: The number of bilingual sentence pairs for Ro → En is 0 .
Revised: The number of bilingual sentence pairs for Ro → En is 0 .

Original: 6 M .
Revised: 6 M .

Original: Sennrich et al.
Revised: Sennrich et al.

Original: (2016a) provided 2 M back translated data 2 .
Revised: (2016a) provided 2 M back translated data 2 .

Original: We use newsdev2016 as validation set and newstest2016 as test set.
Revised: We use newsdev2016 as validation set and newstest2016 as test set.

Original: Sentences were encoded using BPE with a shared source-target vocabulary of about 32 k tokens.
Revised: Sentences were encoded using BPE with a shared source-target vocabulary of about 32 k tokens.

Original: We use transformer big conﬁguration.
Revised: We use transformer big conﬁguration.

Original: Considering there is no Romanian BERT, we use the cased multilingual BERT (please refer to Appendix D) to encode inputs.
Revised: Considering there is no Romanian BERT, we use the cased multilingual BERT (please refer to Appendix D) to encode inputs.

Original: The drop-net rate p net is set as 1 . 0 .
Revised: The drop-net rate p net is set as 1 . 0 .

Original: The translation quality is evaluated by multi-bleu.perl .
Revised: The translation quality is evaluated by multi-bleu.perl .

Original: [Table]
Revised: [Table]

Original: The results are shown in Table 5.
Revised: The results are shown in Table 5.

Original: The
Revised: The

Original: Transformer baseline achieves 33 .
Revised: Transformer baseline achieves 33 .

Original: 12 BLEU score.
Revised: 12 BLEU score.

Original: With back-translation, the performance is boosted to 37 .
Revised: With back-translation, the performance is boosted to 37 .

Original: We use the model obtained with back-translation to initialize BERT-fused model, and eventually reach 39 .
Revised: We use the model obtained with back-translation to initialize BERT-fused model, and eventually reach 39 .

Original: 10 BLEU.
Revised: 10 BLEU.

Original: Such a score surpasses the previous best result 38 .achieved by XLM (Lample & Conneau, 2019) and sets a new record.
Revised: Such a score surpasses the previous best result 38 .achieved by XLM (Lample & Conneau, 2019) and sets a new record.

Original: This demonstrates that our proposed approach is effective and can still achieve improvement over strong baselines.
Revised: This demonstrates that our proposed approach is effective and can still achieve improvement over strong baselines.

Original:  A BLATION STUDY
Revised:  A BLATION STUDY

Original: We conduct two groups of ablation studies on IWSLT’

Revised: We conduct two groups of ablation studies on IWSLT’

Original: En → De translation to better understand our model.
Revised: En → De translation to better understand our model.

Original: [Table]
Revised: [Table]

Revised: Jointly tune

Revised: [Table]

Revised: encoder

Revised: [Table]

Revised: Feed

Revised: [Table]

Revised: all

Revised: [Table]

Revised: without attention

Revised: [Table]

Revised: Replace BERT output with random vectors

Revised: [Table]

Revised: Replace BERT with the encoder of another Transformer model

Revised: [Table]

Revised: BERT-encoder attention

Revised: [Table]

Revised: Remove BERT-decoder attention

Revised: [Table]

Original: Study for training strategy and network architecture
Revised: Study for training strategy and network architecture

Original: We conduct ablation study to investigate the performance of each component of our model and training strategy.
Revised: We conduct ablation study to investigate the performance of each component of our model and training strategy.

Original: Results are reported in Table 6: (1) We randomly initialize the NMT module (i.e., encoder and decoder) of BERT-fused model instead of using a warm-start one as introduced in the training strategy of Section 5.1.
Revised: Results are reported in Table 6: (1) We randomly initialize the NMT module (i.e., encoder and decoder) of BERT-fused model instead of using a warm-start one as introduced in the training strategy of Section 5.1.

Original: In this way, we can only achieve 27 .
Revised: In this way, we can only achieve 27 .

Original: 03 BLEU score, which cannot catch up with the baseline.
Revised: 03 BLEU score, which cannot catch up with the baseline.

Original: We also jointly train
Revised: We also jointly train

Original: BERT model with the NMT module.
Revised: BERT model with the NMT module.

Original: Although it can also boost the baseline from 28 .
Revised: Although it can also boost the baseline from 28 .

Original: 57 to 28 .
Revised: 57 to 28 .

Original: 87 , it is not as good as ﬁxing the BERT part, whose BLEU is 30 . 45 .
Revised: 87 , it is not as good as ﬁxing the BERT part, whose BLEU is 30 . 45 .

Original: (2) We feed the output of BERT into all layers of the encoder without attention models.
Revised: (2) We feed the output of BERT into all layers of the encoder without attention models.

Original: That is, the Eqn.(1) is revised to ˜ h
Revised: That is, the Eqn.(1) is revised to ˜ h

Original: li = 12 (cid:0) attn S ( h l − 1 i , H l − 1 E , H l − 1 E )
Revised: li = 12 (cid:0) attn S ( h l − 1 i , H l − 1 E , H l − 1 E )

Original: + W lB h l − 1 i ) (cid:1) , where W lB is learnable.
Revised: + W lB h l − 1 i ) (cid:1) , where W lB is learnable.

Original: In this case, the encoder and BERT have to share the same vocabulary.
Revised: In this case, the encoder and BERT have to share the same vocabulary.

Original: The BLEU score is 29 . 61 , which is better than the standard Transformer but slightly worse than leveraging the output of BERT as embedding.
Revised: The BLEU score is 29 . 61 , which is better than the standard Transformer but slightly worse than leveraging the output of BERT as embedding.

Original: This shows that the output of BERT should not be fused into each layer directly, and using the attention model to bridge the relation is better than using simple transformation.
Revised: This shows that the output of BERT should not be fused into each layer directly, and using the attention model to bridge the relation is better than using simple transformation.

Original: More results on different languages are included in Appendix B.3.
Revised: More results on different languages are included in Appendix B.3.

Original: To illustrate the effectiveness of our method, we choose another two kinds of ways to encode the input sequence rather than using BERT: (1) Using a ﬁxed and randomly initialized embedding; (2) Using the encoder from another NMT model.
Revised: To illustrate the effectiveness of our method, we choose another two kinds of ways to encode the input sequence rather than using BERT: (1) Using a ﬁxed and randomly initialized embedding; (2) Using the encoder from another NMT model.

Original: Their BLEU scores are 28 .
Revised: Their BLEU scores are 28 .

Original: 91 and 28 .
Revised: 91 and 28 .

Original: 99 respectively, indicating that the BERT pre-trained on large amount of unlabeled data can provide more helpful features to NMT.
Revised: 99 respectively, indicating that the BERT pre-trained on large amount of unlabeled data can provide more helpful features to NMT.

Original: (3) To verify where the output of BERT should be connect to, we remove the BERT-encoder attention (i.e., attn B in Eqn.(1)) and the BERT-decoder attention (i.e,, attn B in Eqn.(2)) respectively.
Revised: (3) To verify where the output of BERT should be connected to, we remove the BERT-encoder attention (i.e., attn B in Eqn.(1)) and the BERT-decoder attention (i.e,, attn B in Eqn.(2)) respectively.

Original: Correspondingly, the BLEU score drops from 30 .
Revised: Correspondingly, the BLEU score drops from 30 .

Original: 45 to 29 .
Revised: 45 to 29 .

Original: 87 and 29 .
Revised: 87 and 29 .

Original: This indicates that the output of BERT should be leveraged by both encoder and decoder to achieve better performances.
Revised: This indicates that the output of BERT should be leveraged by both encoder and decoder to achieve better performances.

Original: At last, considering that there are two stacked encoders in our model, we also choose ensemble models and deeper NMT models as baselines.
Revised: At last, considering that there are two stacked encoders in our model, we also choose ensemble models and deeper NMT models as baselines.

Original: Our approach outperforms the above baselines.
Revised: Our approach outperforms the above baselines.

Original: The results are left in Appendix B.2 due to 
Revised: The results are left in Appendix B.2 due to space limitation.

Original: Study on drop-net
Revised: Study on drop-net

Original: To investigate the effect of drop-net, we conduct experiments on IWSLT’

Revised: To investigate the effect of drop-net, we conduct experiments on IWSLT’

Original: En → De dataset with different drop-net probability, p net ∈ { 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 . 0 } .
Revised: En → De dataset with different drop-net probability, p net ∈ { 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 . 0 } .

Original: The results are shown in Figure 2.
Revised: The results are shown in Figure 2.

Original: As can been seen, although larger p net leads to larger training loss, it leads to smaller validation loss and so better BLUE scores.
Revised: As can been seen, although larger p net leads to larger training loss, it leads to smaller validation loss and so better BLUE scores.

Original: This shows that the drop-net trick can indeed improve the generalization ability of our model.
Revised: This shows that the drop-net trick can indeed improve the generalization ability of our model.

Original: We ﬁx p net = 1 . 0 in other experiments unless specially speciﬁed.
Revised: We ﬁx p net = 1 . 0 in other experiments unless specially speciﬁed.

Original: [Figure]
Revised: [Figure]

Original:  A PPLICATION TO UNSUPERVISED NMT
Revised:  A PPLICATION TO UNSUPERVISED NMT

Original: We work on unsupervised En ↔ Fr and En ↔ Ro translation.
Revised: We work on unsupervised En ↔ Fr and En ↔ Ro translation.

Original: The data processing, architecture selection and training strategy is the same as Lample & Conneau (2019).
Revised: The data processing, architecture selection and training strategy is the same as Lample & Conneau (2019).

Original: Settings For En ↔ Fr, we use 190 M monolingual English sentences and 62 M monolingual French sentences from WMT News Crawl datasets.
Revised: Settings For En ↔ Fr, we use 190 M monolingual English sentences and 62 M monolingual French sentences from WMT News Crawl datasets, which is the same as that used in (Song et al., 2019).

Original: For unsupervised En ↔ Ro translation, we use 50 M English sentences from News Crawl  and collect 2 .
Revised: For unsupervised En ↔ Ro translation, we use 50 M English sentences from News Crawl (sampled from the data for En → Fr) and collect 2 .

Original: 9 M sentences for Romanian by concatenating News Crawl data sets and WMT’16 Romanian monolingual data following Lample et al.
Revised: 9 M sentences for Romanian by concatenating News Crawl data sets and WMT’16 Romanian monolingual data following Lample et al.

Original: The data is preprocessed in the same way as Lample & Conneau (2019).
Revised: The data is preprocessed in the same way as Lample & Conneau (2019).

Original: We use the same model conﬁguration as Lample & Conneau (2019), with details in Appendix A.3.
Revised: We use the same model conﬁguration as Lample & Conneau (2019), with details in Appendix A.3.

Original: The BERT is the pre-trained XLM model (see Appendix D).
Revised: The BERT is the pre-trained XLM model (see Appendix D).

Original: We ﬁrst train an unsupervised NMT model following Lample & Conneau (2019) until convergence.
Revised: We ﬁrst train an unsupervised NMT model following Lample & Conneau (2019) until convergence.

Original: Then we initialize our BERT-fused model with the obtained model and continue training.
Revised: Then we initialize our BERT-fused model with the obtained model and continue training.

Original: We train models on 8 M40 GPUs, and thebatchsize is 2000 tokens per GPU.
Revised: We train models on 8 M40 GPUs, and the

Original: We use the same optimization hyper-parameters as that described in Lample & Conneau (2019).

Original: [Table]
Revised: [Table]

Original: Results
Revised: Results

Original: The results of unsupervised NMT are shown in Table 7.
Revised: The results of unsupervised NMT are shown in Table 7.

Original: Our reproduced baseline numbers of the four translation tasks are 37 .

Original: 33 and 32 . 85 respectively, which are signiﬁcant better than the results reported in Lample & Conneau (2019).

Original: This is because that we use better training data 3 .

Original: After combined with BERT-fused model, we can reach 38 .
Revised: With our proposed BERT-fused model, we can achieve 38 .

Original: 02 and 33 . 20 BLEU scores on the four tasks, setting state-of-the-art results on these tasks.
Revised: 02 and 33 . 20 BLEU scores on the four tasks, setting stateof-the-art results on these tasks.

Original: Therefore, our BERT-fused model also beneﬁts unsupervised NMT.
Revised: Therefore, our BERT-fused model also beneﬁts unsupervised NMT.

Original:  C ONCLUSION AND FUTURE WORK
Revised:  C ONCLUSION AND FUTURE WORK

Original: In this work, we propose an effective approach, BERT-fused model, to combine BERT and NMT, where the BERT is leveraged by the encoder and decoder through attention models.
Revised: In this work, we propose an effective approach, BERT-fused model, to combine BERT and NMT, where the BERT is leveraged by the encoder and decoder through attention models.

Original: Experiments on supervised NMT (including sentence-level and document-level translations), semi-supervised NMT and unsupervised NMT demonstrate the effectiveness of our method.
Revised: Experiments on supervised NMT (including sentence-level and document-level translations), semi-supervised NMT and unsupervised NMT demonstrate the effectiveness of our method.

Original: For future work, there are many interesting directions.
Revised: For future work, there are many interesting directions.

Original: First, we will study how to speed up inference time.
Revised: First, we will study how to speed up inference time.

Original: Second, we can apply such an algorithm to more applications, like questioning and answering.
Revised: Second, we can apply such an algorithm to more applications, like questioning and answering.

Original: Third, how to compress BERT-fused model into a light version is another interesting direction
Revised: Third, how to compress BERT-fused model into a light version is another interesting direction. A CKNOWLEDGEMENTS

Revised: This work was supported in part by NSFC under Contract 61836011.

Original: A E XPERIMENT S ETUP A. IWSLT’14 & WMT’14 S ETTINGS
Revised: A E XPERIMENT S ETUP A. IWSLT’14 & WMT’14 S ETTINGS

Original: We mainly follow the scripts below to preprocess the data: https://github.com/pytorch/ fairseq/tree/master/examples/translation .
Revised: We mainly follow the scripts below to preprocess the data: https://github.com/pytorch/ fairseq/tree/master/examples/translation .

Original: Dataset For the low-resource scenario, we choose IWSLT’14 English ↔ German
Revised: Dataset For the low-resource scenario, we choose IWSLT’14 English ↔ German

Original: (En ↔ De),
Revised: (En ↔ De),

Original: English → Spanish (En → Es), IWSLT’17 English → French (En → Fr) and English → Chinese (En → Zh) translation.
Revised: English → Spanish (En → Es), IWSLT’17 English → French (En → Fr) and English → Chinese (En → Zh) translation.

Original: There are 160 k , 183 k , 236 k , 235 k bilingual sentence pairs for En ↔ De,
Revised: There are 160 k , 183 k , 236 k , 235 k bilingual sentence pairs for En ↔ De,

Original: En → Es, En → Fr and En → Zh tasks.
Revised: En → Es, En → Fr and En → Zh tasks.

Original: Following the common practice (Edunov et al., 2018), for
Revised: Following the common practice (Edunov et al., 2018), for

Original: En ↔ De, we lowercase all words, split 7 k sentence pairs from the training dataset for validation and concatenate dev2010 , dev2012 , tst2010 , tst2011 , tst2012 as the test set.
Revised: En ↔ De, we lowercase all words, split 7 k sentence pairs from the training dataset for validation and concatenate dev2010 , dev2012 , tst2010 , tst2011 , tst2012 as the test set.

Original: For other tasks, we do not lowercase the words and use the ofﬁcial validation/test sets of the corresponding years.
Revised: For other tasks, we do not lowercase the words and use the ofﬁcial validation/test sets of the corresponding years.

Original: For rich-resource scenario, we work on WMT’14 En → De and En → Fr, whose corpus sizes are 4 .
Revised: For rich-resource scenario, we work on WMT’14 En → De and En → Fr, whose corpus sizes are 4 .

Original: 5 M and 36 M respectively.
Revised: 5 M and 36 M respectively.

Original: We concatenate newstest2012 and newstest2013 as the validation set and use newstest2014 as the test set.
Revised: We concatenate newstest2012 and newstest2013 as the validation set and use newstest2014 as the test set.

Original: We apply BPE (Sennrich et al., 2016c) to split words into sub-units.
Revised: We apply BPE (Sennrich et al., 2016c) to split words into sub-units.

Original: The numbers of BPE merge operation for IWSLT tasks, WMT’

Revised: The numbers of BPE merge operation for IWSLT tasks, WMT’

Original: En → De and En → Fr are 10 k , 32 k and 40 k respectively.
Revised: En → De and En → Fr are 10 k , 32 k and 40 k respectively.

Original: We merge the source and target language sentences for all tasks to build the vocabulary except En → Zh.
Revised: We merge the source and target language sentences for all tasks to build the vocabulary except En → Zh.

Original: Model Conﬁguration For IWSLT tasks, we use the transformer iwslt de en setting with dropout ratio 0 .
Revised: Model Conﬁguration For IWSLT tasks, we use the transformer iwslt de en setting with dropout ratio 0 .

Original: In this setting, the embedding dimension, FFN layer dimension and number of layers are 512 , 1024 and 6 .
Revised: In this setting, the embedding dimension, FFN layer dimension and number of layers are 512 , 1024 and 6 .

Original: For WMT’

Revised: For WMT’

Original: En → De and En → Fr, we use transformer big setting (short for transformer vaswani wmt en de big ) with dropout 0 .
Revised: En → De and En → Fr, we use transformer big setting (short for transformer vaswani wmt en de big ) with dropout 0 .

Original: 3 and 0 .
Revised: 3 and 0 .

Original: 1 respectively.
Revised: 1 respectively.

Original: In this setting, the aforementioned three parameters are 1024 , 4096 and 6 respectively.
Revised: In this setting, the aforementioned three parameters are 1024 , 4096 and 6 respectively.

Original: Evaluation We use multi-bleu.perl 4 to evaluate IWSLT’14 En ↔ De and WMT translation tasks for fair comparison with previous work.
Revised: Evaluation We use multi-bleu.perl 4 to evaluate IWSLT’14 En ↔ De and WMT translation tasks for fair comparison with previous work.

Original: For the remaining tasks, we use a more advance implementation of BLEU score, detokenized sacreBLEU for evaluation 5 .
Revised: For the remaining tasks, we use a more advance implementation of BLEU score, detokenized sacreBLEU for evaluation 5 .

Original: A. D ETAILED E XPERIMENT S ETTING IN S ECTION
Revised: A. D ETAILED E XPERIMENT S ETTING IN S ECTION

Original: The IWSLT’14 English-to-German data and model conﬁguration is introduced in Section A.1.
Revised: The IWSLT’14 English-to-German data and model conﬁguration is introduced in Section A.1.

Original: For the training stategy, we use Adam (Kingma & Ba, 2014) to optimize the network with β 1 = 0 .
Revised: For the training stategy, we use Adam (Kingma & Ba, 2014) to optimize the network with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 98 and weight-decay = 0 .
Revised: 98 and weight-decay = 0 .

Original: The learning rate scheduler is inverse sqrt , where warmup-init-lr = 10 − 7 , warmup-updates = 4000 and max-lr = 0 . 0005 .
Revised: The learning rate scheduler is inverse sqrt , where warmup-init-lr = 10 − 7 , warmup-updates = 4000 and max-lr = 0 . 0005 .

Original: A. D ETAILED MODEL CONFIGURATION IN U NSUPERVISED NMT
Revised: A. D ETAILED MODEL CONFIGURATION IN U NSUPERVISED NMT

Original: We leverage one Transformer model with GELU activation function to work on translations of two directions, where each language is associated with a language tag.
Revised: We leverage one Transformer model with GELU activation function to work on translations of two directions, where each language is associated with a language tag.

Original: The embedding dimension, FFN layer dimension and number of layer are 1024 , 4096 and 6 .
Revised: The embedding dimension, FFN layer dimension and number of layer are 1024 , 4096 and 6 .

Original: The BERT is initialized by the pretrained XLM model provided by (Lample & Conneau, 2019).
Revised: The BERT is initialized by the pretrained XLM model provided by (Lample & Conneau, 2019).

Original: B M ORE EXPERIMENT RESULTS B. M ORE RESULTS ON PRELIMINARY EXPLORATION OF LEVERAGING BERT
Revised: B M ORE EXPERIMENT RESULTS B. M ORE RESULTS ON PRELIMINARY EXPLORATION OF LEVERAGING BERT

Original: We use XLM to initialize the model for WMT’14 English → German translation task, whose training corpus is relative large.
Revised: We use XLM to initialize the model for WMT’14 English → German translation task, whose training corpus is relative large.

Original: We eventually obtain 28 .
Revised: We eventually obtain 28 .

Original: after 90 epochs, which is still underperform the baseline, 29 .
Revised: after 90 epochs, which is still underperform the baseline, 29 .

Original: 12 as we got.
Revised: 12 as we got.

Original: Similar problem is also reported in https://github.com/ facebookresearch/XLM/issues/32 .
Revised: Similar problem is also reported in https://github.com/ facebookresearch/XLM/issues/32 .

Original: We leave the improvement of supervised NMT with
Revised: We leave the improvement of supervised NMT with

Original: XLM as future work.
Revised: XLM as future work.

Original: B. M ORE ABLATION STUDY
Revised: B. M ORE ABLATION STUDY

Original: Part I: A different way to deal with multiple attention models
Revised: Part I A different way to deal with multiple attention models

Original: Junczys-Dowmunt & Grundkiewicz (2018) proposed a new way to handle multiple attention models.
Revised: Junczys-Dowmunt & Grundkiewicz (2018) proposed a new way to handle multiple attention models.

Original: Instead of using Eqn.(2), the input is processed by self-attention, encoder-decoder attention and
Revised: Instead of using Eqn.(2), the input is processed by self-attention, encoder-decoder attention and

Original: BERT-decoder attention sequentially.
Revised: BERT-decoder attention sequentially.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: The BLEU score is 29 .
Revised: The BLEU score is 29 .

Original: 35 for this setting, not as good as our proposed method.
Revised: 35 for this setting, not as good as our proposed method.

Original: Part II: More results on IWSLT’14 En → De translation
Revised: Part II: More results on IWSLT’14 En → De translation

Original: Since our BERT-fused model contains two stacked encoders, we carry our two groups of additional baselines:
Revised: Since our BERT-fused model contains two stacked encoders, we carry out two groups of additional baselines:

Original: (1) Considering that stacking the BERT and encoder can be seen as a deeper model, we also train another two NMT models with deeper encoders, one with 18 layers (since BERT base consists oflayers) and the other with 12 layers (which achieved best validation performance ranging from 6 to 18 layers).
Revised: (1) Considering that stacking the BERT and encoder can be seen as a deeper model, we also train another two NMT models with deeper encoders, one with 18 layers (since BERT base consists oflayers) and the other with 12 layers (which achieved best validation performance ranging from 6 to 18 layers).

Original: (2) We also compare the results of our approach with ensemble methods.
Revised: (2) We also compare the results of our approach with ensemble methods.

Original: To get an M -model ensemble, we independently train M models with different random seeds ( M ∈ Z + ).
Revised: To get an M -model ensemble, we independently train M models with different random seeds ( M ∈ Z + ).

Original: We ensemble both standard Transformers and our BERT-fused models, which are denoted as M -model ensemble (standard) and M -model ensemble (BERT-fused) respectively.
Revised: We ensemble both standard Transformers and our BERT-fused models, which are denoted as M -model ensemble (standard) and M -model ensemble (BERT-fused) respectively.

Original: Please note that when we aggregate multiple BERT-fused models, we only need to store one replica of the BERT model because the
Revised: Please note that when we aggregate multiple BERT-fused models, we only need to store one replica of the BERT model because the

Original: BERT part is not optimized.
Revised: BERT part is not optimized.

Revised: [Table]

Revised: [Table]

Revised: 3-model ensemble (standard)

Original: [Table]
Revised: [Table]

Original:  ) 29 .3-model ensemble (standard) 30 .4-model ensemble (standard) 30 .2-model ensemble (BERT-fused) 31 .3-model ensemble (BERT-fused) 31 .4-model ensemble (BERT-fused) 31 .
Revised:  4-model ensemble (standard) 30 .2-model ensemble (BERT-fused) [Table]  3-model ensemble (BERT-fused) [Table]

Revised:  4-model ensemble (BERT-fused)

Revised: [Table]

Original: The results are shown in Table 8.
Revised: The results are shown in Table 8.

Original: We have the following observations:
Revised: We have the following observations:

Original: Adding more layers can indeed boost the baseline, but still not as good as BERT-fused model.
Revised: Adding more layers can indeed boost the baseline, but still not as good as BERT-fused model.

Original: According to our experiments, when increasing the number of layers to 12 , we achieve the best BLEU score, 29 . 27 .
Revised: According to our experiments, when increasing the number of layers to 12 , we achieve the best BLEU score, 29 . 27 .

Original: We also compare our results to ensemble methods.
Revised: We also compare our results to ensemble methods.

Original: Indeed, ensemble signiﬁcantly boost the baseline by more than one point.
Revised: Indeed, ensemble signiﬁcantly boost the baseline by more than one point.

Original: However, even if using ensemble of four models, the BLEU score is still lower than our BERT-fused model (30.18 v.s. 30.45), which shows the effectiveness of our method.
Revised: However, even if using ensemble of four models, the BLEU score is still lower than our BERT-fused model (30.18 v.s. 30.45), which shows the effectiveness of our method.

Original: We want to point out that our method is intrinsically different from ensemble.
Revised: We want to point out that our method is intrinsically different from ensemble.

Original: Ensemble approaches usually refer to “independently train several different models for the same task, and then aggregate the output of each model to get the eventually task.
Revised: Ensemble approaches usually refer to “independently” train several different models for the same task, and then aggregate the output of each model to get the eventually task.

Original: In BERT-fused model, although we include a pre-trained BERT into our model, there is still only  model serving for the translation task.
Revised: In BERT-fused model, although we include a pre-trained BERT into our model, there is still only one model serving for the translation task.

Original: In this sense [Table] we can also combine our BERT-fused model with ensemble.
Revised: In this sense, we can also combine our BERT-fused model with ensemble.

Original: Our approach beneﬁts from ensemble too.
Revised: Our approach beneﬁts from ensemble too.

Original: When ensembling two models, we can achieve 31 .
Revised: When ensembling two models, we can achieve 31 .

Original: BLEU score.
Revised: BLEU score.

Original: When adding the number of models to four, we eventually achieve 31 .
Revised: When adding the number of models to four, we eventually achieve 31 .

Original: 85 BLEU score, which is 1 .
Revised: 85 BLEU score, which is 1 .

Original: 67 point improvement over the ensemble of standard Transformer.
Revised: 67 point improvement over the ensemble of standard Transformer.

Original: Part III: More results on IWSLT’14 De → En translation
Revised: Part III: More results on IWSLT’14 De → En translation

Original: We report the ensemble results on IWSLT’14 De
Revised: We report the ensemble results on IWSLT’14 De

Original: → En translation in Table 9.
Revised: → En translation in Table 9.

Original: We can get similar conclusion compared to that of IWSLT’

Revised: We can get similar conclusion compared to that of IWSLT’

Original: En → De.
Revised: En → De.

Original: [Table]
Revised: [Table]

Original: . M ORE RESULTS ON FEEDING BERT OUTPUT TO NMT MODULE
Revised: B. M ORE RESULTS ON FEEDING BERT OUTPUT TO NMT MODULE

Original: The ablation study on more languages on shown in Table 10.
Revised: The ablation study on more languages is shown in Table 10.

Original: Our method achieves the best results compared to all baselines.
Revised: Our method achieves the best results compared to all baselines.

Original: [Table]
Revised: [Table]

Original: Our
Revised: Our

Original: [Table]
Revised: [Table]

Original: IWSLT’14 D E → E N TRANSLATION
Revised: IWSLT’14 G ERMAN - TO -E NGLISH TRANSLATION

Original: We summarize the BLEU scores on IWSLT’14 De
Revised: We summarize the BLEU scores on IWSLT’14 De

Original: → En of existed works and our BERT-fused model approach in Table 11.
Revised: → En of existed works and our BERT-fused model approach in Table [Table]

Original: [Table]
Revised: [Table]

Original: B. C OMPARISON WITH BACK TRANSLATION
Revised: B. C OMPARISON WITH BACK TRANSLATION

Original: When using unlabeled data to boost machine learning systems, one of the most notable approaches is back translation (brieﬂy, BT) (Sennrich et al., 2016b):
Revised: When using unlabeled data to boost machine learning systems, one of the most notable approaches is back translation (brieﬂy, BT) (Sennrich et al., 2016b):

Original: We ﬁrst train a reversed translation model, use the obtained model to translate the unlabeled data in the target domain back to source domain, obtain a synthetic dataset where the source data is back-translated and ﬁnally train the forward model on the augmented dataset.
Revised: We ﬁrst train a reversed translation model, use the obtained model to translate the unlabeled data in the target domain back to source domain, obtain a synthetic dataset where the source data is back-translated and ﬁnally train the forward model on the augmented dataset.

Original: Our method has two main differences with BT method.
Revised: Our method has two main differences with BT method.

Original: In BT, the monolingual data from the target side is leveraged.
Revised: In BT, the monolingual data from the target side is leveraged.

Original: In our proposed approach, we use a BERT of the source language, which indirectly leverages the monolingual data from the source side.
Revised: In our proposed approach, we use a BERT of the source language, which indirectly leverages the monolingual data from the source side.

Original: In this way, our approach and BT are complementary to each other.
Revised: In this way, our approach and BT are complementary to each other.

Original: In Section 5.4, we have already veriﬁed that our method can further improve the results of standard BT one Romanian-to-English translation.
Revised: In Section 5.4, we have already veriﬁed that our method can further improve the results of standard BT on Romanian-to-English translation.

Original: To use BT, we have to train a reversed translation model and then back translate the monolingual data, which is time-cost due to the decoding process.
Revised: To use BT, we have to train a reversed translation model and then back translate the monolingual data, which is time-cost due to the decoding process.

Original: In BERT-fused model, we only need to download a pre-trained BERT model, incorporate it into our model and continue training.
Revised: In BERT-fused model, we only need to download a pre-trained BERT model, incorporate it into our model and continue training.

Original: Besides, the BERT module is ﬁxed during training.
Revised: Besides, the BERT module is ﬁxed during training.

Original: On IWSLT’14, we also implement BT on wikipedia data, which is a subset of the corpus of training
Revised: On IWSLT’14, we also implement BT on wikipedia data, which is a subset of the corpus of training

Original: BERT.
Revised: BERT.

Original: The model used for back translation are standard Transformer baselines introduced in Section 5, whose BLEU scores are 28 .
Revised: The model used for back translation are standard Transformer baselines introduced in Section 5, whose BLEU scores are 28 .

Original: 57 and 34 .
Revised: 57 and 34 .

Original: 64 respectively.
Revised: 64 respectively.

Original: We back translate 1 M, 2 M, 5 M, 15 M and 25 M randomly selected German sentences.
Revised: We back translate 1 M, 2 M, 5 M, 15 M and 25 M randomly selected German sentences.

Original: The results are reported in Table 12.
Revised: The results are reported in Table 12.

Original: The rows started with BT( · ) represent the results of BT, and the numbers in the brackets are the number of sentences for back translation.
Revised: The rows started with BT( · ) represent the results of BT, and the numbers in the brackets are the number of sentences for back translation.

Original: [Table]
Revised: [Table]

Original: IWSLT dataset is a collection of spoken language, and the bilingual training corpus is small ( 160 k ).
Revised: IWSLT dataset is a collection of spoken language, and the bilingual training corpus is small ( 160 k ).

Original: In Wikipedia, the sentences are relatively formal compared to the spoken language, which is outof-domain of spoken languages.
Revised: In Wikipedia, the sentences are relatively formal compared to the spoken language, which is outof-domain of spoken languages.

Original: We can see that when using 1 M or 2 M monolingual data for BT, the BLEU scores can indeed improve from 28 .
Revised: We can see that when using 1 M or 2 M monolingual data for BT, the BLEU scores can indeed improve from 28 .

Original: 57 to 29 .
Revised: 57 to 29 .

Original: However, simply adding more wikipedia data for BT does not result in more improvement.
Revised: However, simply adding more wikipedia data for BT does not result in more improvement.

Original: There is even a slight drop when adding more than 15 M monolingual sentences.
Revised: There is even a slight drop when adding more than 15 M monolingual sentences.

Original: However, our BERT-fused model can achieve better performances than BT with wikipedia data.
Revised: However, our BERT-fused model can achieve better performances than BT with wikipedia data.

Original: C C OMPARISON OF INFERENCE TIME
Revised: C C OMPARISON OF INFERENCE TIME

Original: [Table]
Revised: [Table]

Original: We compare the inference time of our approach to the baselines.
Revised: We compare the inference time of our approach to the baselines.

Original: The results are shown in Table 13, where from the second column to the last column, the numbers are the inference time of standard
Revised: The results are shown in Table 13, where from the second column to the last column, the numbers are the inference time of standard

Original: Transformer, BERT-fused model, and the increase of inference time.
Revised: Transformer, BERT-fused model, and the increase of inference time.

Original: Indeed, introducing BERT to encode the input brings additional inference time, resulting in about 40% to 49% increase.
Revised: Indeed, introducing BERT to encode the input brings additional inference time, resulting in about 40% to 49% increase.

Original: But considering the signiﬁcant improvement of BLEU score, it is acceptable of such extra cost.
Revised: But considering the signiﬁcant improvement of BLEU score, it is acceptable of such extra cost.

Original: We will study how to reduce inference time in the future.
Revised: We will study how to reduce inference time in the future.

Original: D D OWNLOAD LINK OF PRE - TRAINED BERT MODELS
Revised: D D OWNLOAD LINK OF PRE - TRAINED BERT MODELS

Original: We leverage the pre-trained models provided by PyTorch-Transformers 6 .
Revised: We leverage the pre-trained models provided by PyTorch-Transformers 6 .

Original: For IWSLT’14 tasks, we choose BERT base model with 12 layers and hidden dimension 768 .
Revised: For IWSLT’14 tasks, we choose BERT base model with 12 layers and hidden dimension 768 .

Original: 1. IWSLT

Revised: 1. IWSLT

Original: En →{ De, Es, Fr, Zh } , we choose bert-base-uncased . 2.
Revised: En →{ De, Es, Fr, Zh } , we choose bert-base-uncased . 2.

Original: IWSLT

Revised: IWSLT

Original: De → En, we choose bert-base-german-cased .
Revised: De → En, we choose bert-base-german-cased .

Original: For WMT14 En → Fr, De } , 
Revised: For WMT14 En →{ Fr, De } , we choose bert large uncased , which is a BERT

Original: we
Revised: large

Original: choose
Revised: model

Original: bert-large-uncased , which is a BERT large model with 24 layers and hidden dimension1024 .
Revised: with 24 layers and hidden dimension

Original: For WMT16 Ro → En, we choose bert-base-multilingual-cased , because there is no
Revised: For WMT16 Ro → En, we choose bert-base-multilingual-cased , because there is no

Original: BERT specially trained for the Romanian.
Revised: BERT specially trained for the Romanian.

Original: For unsupervised En ↔ Fr and unsupervised En ↔ Ro, we choose xlm-mlm-enfr1024 and xlm-mlm-enro1024 respectively.
Revised: For unsupervised En ↔ Fr and unsupervised En ↔ Ro, we choose xlm-mlm-enfr1024 and xlm- mlm enro1024 respectively.

Original: The download links are summarized as follows:
Revised: The download links are summarized as follows:

Original: • bert-base-uncased: https://s3.amazonaws.com/models.huggingface.co/ bert/bert-base-uncased.tar.gz .
Revised: • bert-base-uncased: https://s3.amazonaws.com/models.huggingface.co/ bert/bert-base-uncased.tar.gz .

Original: • bert-large-uncased: https://s3.amazonaws.com/models.huggingface.
Revised: • bert-large-uncased: https://s3.amazonaws.com/models.huggingface.

Original: co/bert/bert-large-uncased.tar.gz .
Revised: co/bert/bert-large-uncased.tar.gz .

Original: • bert-base-multilingual-cased: https://s3.amazonaws.com/models.
Revised: • bert-base-multilingual-cased: https://s3.amazonaws.com/models.

Original: huggingface.co/bert/bert-base-multilingual-cased.tar.gz .
Revised: huggingface.co/bert/bert-base-multilingual-cased.tar.gz .

Original: • bert-base-german-cased: https://int-deepset-models-bert.s3.
Revised: • bert-base-german-cased: https://int-deepset-models-bert.s3.

Original: eu-central-1.amazonaws.com/pytorch/bert-base-german-cased.
Revised: eu-central-1.amazonaws.com/pytorch/bert-base-german-cased.

Original: tar.gz .
Revised: tar.gz .

Original: • xlm-mlm-enfr1024: https://s3.amazonaws.com/models.huggingface.
Revised: • xlm-mlm-enfr1024: https://s3.amazonaws.com/models.huggingface.

Original: co/bert/xlm-mlm-enfr-1024-pytorch_model.bin .
Revised: co/bert/xlm-mlm-enfr-1024-pytorch_model.bin .

Original: • xlm-mlm-enro1024: https://s3.amazonaws.com/models.huggingface.
Revised: • xlm-mlm-enro1024: https://s3.amazonaws.com/models.huggingface.

Original: co/bert/xlm-mlm-enro-1024-pytorch_model.bin .
Revised: co/bert/xlm-mlm-enro-1024-pytorch_model.bin .

Original: E D ETAILS OF THE NOTATIONS
Revised: E D ETAILS OF THE NOTATIONS

Original: Let attn ( q, K, V ) denote the attention layer, where q , K and V indicate query, key and value respectively.
Revised: Let attn ( q, K, V ) denote the attention layer, where q , K and V indicate query, key and value respectively.

Original: Here q is a d q -dimensional vector ( d ∈ Z ), K and V are two sets with | K | = | V | .
Revised: Here q is a d q -dimensional vector ( d ∈ Z ), K and V are two sets with | K | = | V | .

Original: Each k i ∈ K and v i ∈ V are also d k / d v -dimensional ( d q , d k and d v can be different) vectors, i ∈
Revised: Each k i ∈ K and v i ∈ V are also d k / d v -dimensional ( d q , d k and d v can be different) vectors, i ∈

Original: [ | K | ] .
Revised: [ | K | ] .

Original: The attention model works as follows:
Revised: The attention model works as follows:

Original: [Equation]
Revised: [Equation]

Original: where W q , W k and W v are the parameters to be learned.
Revised: where W q , W k and W v are the parameters to be learned.

Original: In Vaswani et al.
Revised: In Vaswani et al.

Original: (2017), attn is implemented as a multi-head attention model and we omit the details here to increase readability.
Revised: (2017), attn is implemented as a multi-head attention model and we omit the details here to increase readability.

Original: Following Vaswani et al.
Revised: Following Vaswani et al.

Original: (2017), we deﬁne the non-linear transformation layer as
Revised: (2017), we deﬁne the non-linear transformation layer as

Original: [Equation]
Revised: [Equation]

Original: where x is the input; W 1 , W 2 , b 1 , b 2 are the parameters to be learned; max is an element-wise operator.
Revised: where x is the input; W 1 , W 2 , b 1 , b 2 are the parameters to be learned; max is an element-wise operator.

Original: Layer normalization is also applied following Transformer (Vaswani et al., 2017).
Revised: Layer normalization is also applied following Transformer (Vaswani et al., 2017).