Document-Level Neural Machine Translation Using BERT as Context Encoder (2020)

Abstract: Large-scale pre-trained representations such as BERT have been widely used in many natural language understanding tasks. The methods of incorporating BERT into document-level machine translation are still being explored. BERT is able to understand sentence relationships since BERT is pre-trained using the next sentence prediction task. In our work, we leverage this property to improve document-level machine translation. In our proposed model, BERT performs as a context encoder to achieve document-level contextual information, which is then integrated into both the encoder and decoder. Experiment results show that our proposed method can significantly outperform strong document-level machine translation baselines on BLEU score. Moreover, the ablation study shows our method can capture document-level context information to boost translation performance.

## introduction recent years have witnessed the great success of neural machine translation (nmt) @xcite @xcite . nmt systems have even achieved human parity on resource-rich language pairs @xcite . however, standard nmt systems perform translation only at the sentence level, which ignores the dependencies among sentences when translating entire documents. to address the above challenges, various document-level nmt models, have been proposed to extract contextual information from surrounding sentences and have achieved substantial improvements in generating consistent translations @xcite @xcite @xcite . large-scale pre-trained text representations like gpt-2 @xcite , bert @xcite , roberta @xcite , albert @xcite , have been widely used in many natural language understanding tasks. among them, bert is one of the most effective representations that has inspired many other representations such as roberta, albert. it significantly boosts the performance of many natural language understanding tasks, including text classification, question answering, etc @xcite . there have been few recent attempts to incorporate bert into nmt models @xcite @xcite . intuitively, as one of bert's pre-training tasks is the binarized next sentence prediction (nsp) task, a natural assumption is that the nsp task has enabled the model to understand the relationship between two sentences, the relationship information is helpful to model the context information for documentlevel machine translation. in this work, we propose to extend the transformer model to take advantage of bert document-level contextual representation. we use the pre-trained bert as a context encoder to achieve document-level representation, which is then integrated into both the encoder and the decoder of transformer nmt model. we use a multihead attention mechanism and context gate to control how each layer interacts with bert context representations adaptively. we conducted experiments on two documentlevel machine translation datasets. experimental results show that our proposed model can outperform transformer baselines and previous state-of-theart document-level nmt models on bleu score. also, we perform an ablation study showing that the bert context encoder can capture documentlevel context representation to improve translation performance. 2 approach 2.1 problem statement formally, denote @xmath0, x 2 , . . . , x n as a sourcelanguage document with n source sentences. the corresponding target-language document is denoted by @xmath1, y 2 , . . . , y n . we assume that (x i , y i ) is a parallel sentence pair. following @xcite , we omit the target-side document-level context @xmath2, and source side document-level context @xmath3. therefore, the probability can be approximated as: ## conclusion we have presented a method for leveraging bert to capture contextual information for documentlevel neural machine translation. experiments on two document-level machine translation tasks demonstrate the effectiveness of our approach. besides, we have shown that our approach can really capture the context information to improve the translation performance. for future work, we plan to compress our model into a light version to leverage more context sen-