BERT , m BERT , or B i BERT ? A Study on Contextualized Embeddings for Neural Machine Translation (2021)

Abstract: The success of bidirectional encoders using masked language models, such as BERT, on numerous natural language processing tasks has prompted researchers to attempt to incorporate these pre-trained models into neural machine translation (NMT) systems. However, proposed methods for incorporating pre-trained models are non-trivial and mainly focus on BERT, which lacks a comparison of the impact that other pre-trained models may have on translation performance. In this paper, we demonstrate that simply using the output (contextualized embeddings) of a tailored and suitable bilingual pre-trained language model (dubbed BiBERT) as the input of the NMT encoder achieves state-of-the-art translation performance. Moreover, we also propose a stochastic layer selection approach and a concept of a dual-directional translation model to ensure the sufficient utilization of contextualized embeddings. In the case of without using back translation, our best models achieve BLEU scores of 30.45 for En→De and 38.61 for De→En on the IWSLT’14 dataset, and 31.26 for En→De and 34.94 for De→En on the WMT’14 dataset, which exceeds all published numbers.

## introduction pre-trained language models (lms), trained on a large-scale unlabeled data to capture rich representations of the input, such as elmo @xcite , bert @xcite , xlnet @xcite and xlm @xcite have increasingly attracted attention in various nlp tasks. either utilizing context-aware representations of input tokens @xcite or fine-tuning the pre-trained parameters @xcite both lead to significant improvement for downstream tasks. figure 1 : the overview of methods: a series of additive improvements to the use of contextualized embeddings on iwslt'14 dataset. experimenting over various pretrained language models, we show that our bibert, a bilingual english-german language model, vastly outperforms all other methods (section 2). adding stochastic layer selection to bibert improves performance (section 3). finally, innovative dual-directional training and fine-tuning with the previous two methods yield around 2 bleu point gains over the previous state-of-the-art result @xcite (section 4). inspired by the superior performance of bert on many other tasks, researchers have investigated leveraging using this pre-trained masked language model to enhance translation models, e.g., initializing the parameters of the model's encoder with bert parameters @xcite , and incorporating the output of bert to each layer of the encoder @xcite . in this paper, we demonstrate simply using the output of a pre-trained language model as the input of nmt systems can achieve state-of-the-art results on iwlst'14 @xcite and wmt'14 @xcite english↔german (en↔de) translation tasks in the case of without using back translation @xcite 3 . after conducting a thor-ough evaluation of numerous pre-trained language models, we demonstrate that specialized bilingual models perform the best. we then introduce two further refinements, stochastic layer selection and dual-directional training that yield further improvements. the overview of methods are shown in figure 1 . overall, our best systems beat published state-of-the-art bleu scores by around 2 points. our main contributions are listed as follows: ## one model, dual-directional translations in this section, different from ordinary oneway translation models, we introduce our dualdirectional translation models, i.e., a model can translate both @xmath0. the model architecture is the same as the one in section 3. one of the biggest advantages of the shared english-german vocabulary of bibert en-de is that our encoder has the capability of receiving contextualized embeddings of both source and target tokens. during the training step, we feed source sentences to the model and expect the generation of a target translation, yet also, inversely, feed target sentences and expect translations in the source language. the motivation behind the dual-directional translation model is that we expect the contextualized representations of source and target sentences could enhance each other to build a better encoder for the translation model. from the aspect of data augmentation, the target sentences play a role in augmented data in the task of translating from the source language to the target language, vice versa. with the method of swapping source and target sentences once as an additional dataset, our experiments show superior performance for both directional translations. two advantages of this method are 1) obtaining improvement without extra bitexts, and 2) only slight modification for data preprocessing and no changes for the model architecture. ## conclusion we have shown that our bibert trained on a large amount of mixed texts of the source and target languages can better help nmt models improve translation performance compared with other existing pre-trained language models and achieve state-ofthe-art results by simply using the output of the last layer. moreover, we introduce the stochastic layer selection method and demonstrated its effectiveness in improving translation performance. finally, experiments on the dual-directional translation model illustrate that source and target data can augment each other to further boost performance.