A Simple and Effective Unified Encoder for Document-Level Machine Translation (2020)

Abstract: Most of the existing models for document-level machine translation adopt dual-encoder structures. The representation of the source sentences and the document-level contexts are modeled with two separate encoders. Although these models can make use of the document-level contexts, they do not fully model the interaction between the contexts and the source sentences, and can not directly adapt to the recent pre-training models (e.g., BERT) which encodes multiple sentences with a single encoder. In this work, we propose a simple and effective unified encoder that can outperform the baseline models of dual-encoder models in terms of BLEU and METEOR scores. Moreover, the pre-training models can further boost the performance of our proposed model.

## introduction thanks to the development of the deep learning methods, the machine translation systems have achieved good performance that is even comparable with human translation in the news domain @xcite . however, there are still some problems with machine translation in the documentlevel context @xcite . therefore, more recent work @xcite @xcite @xcite @xcite is focusing on the document-level machine translation. most of the existing models @xcite @xcite for document-level machine translation use two encoders to model the source sentences and the document-level contexts. figure 1a illustrates the structure of these models. they extend the standard 1 in this work, document-level contexts denote the surrounding sentences of the current source sentence. transformer model with a new context encoder, and the encoder for source sentences is conditioned on this context encoder. however, they do not fully model the interaction between the contexts and the source sentences because the self-attention layers are performed inside each encoder separately. moreover, it cannot be directly adapted to the recent pre-training models @xcite @xcite @xcite , which encodes multiple sentences with a single encoder. different from the dual-encoder structure, the uni-encoder structure takes the concatenation of contexts and source sentences as the input (as shown in figure 1b ). therefore, when modeling the contexts, it can make full use of the interaction between the source sentences and the contexts, while the dual-encoder model fails to exploit this information. moreover, the uni-encoder structure is identical to the recent pre-training models (e.g., bert). however, the previous uni structure suffers from two problems for document-level machine translation. first, the attention is distracted due to longer sequences. second, the source sentences and the contexts are modeled equally, which is contrary to the fact that the translation is more related to the current source sentences. to address these problems, we propose a novel flat structure with a unified encoder called flat-transformer. it separates the encoder of standard transformers into two parts so that the attention can concentrate at both the global level and the local level. at the bottom of the encoder blocks, the self-attention is applied to the whole sequence. at the top of the blocks, it is only implemented at the position of the source sentences. we evaluate this model on three document-level machine translation datasets. experiments show that it can achieve better performance than the baseline models of dual-encoder structures in terms of bleu and meteor scores. moreover, the pre-training models can further boost the performance of the proposed structure. ## flat-transformer in this section, we introduce our proposed flat structured model, which we denote as flat-transformer. ## experiments we evaluate the proposed model and several stateof-the-art models on three document-level machine translation benchmarks. we denote the proposed model as flat-transformer. ## related work here we summarize the recent advances in document-level neural machine translation. some work focuses on improving the architectures of the document machine translation models. tiedemann and scherrer (2017) and @xcite explore possible solutions to exploit the crosssentence contexts for neural machine translation. @xcite extends the transformer model with a new context encoder to represent documentlevel context. @xcite and @xcite propose two different hierarchical attention models to model the contexts. @xcite introduces a capsule network to improve these hierarchical structures. there are also some works analyzing the contextual errors @xcite @xcite and providing the test suites @xcite . more recently, @xcite explores the approaches to incorporate the mono-lingual data to augment the document-level bi-lingual dataset. different from these works, this paper mainly discusses the comparison between dual-encoder models and uniencoder models and proposes a novel method to improve the uni-encoder structure. ## conclusions in this work, we explore the solutions to improve the uni-encoder structures for document-level machine translation. we propose a flat-transformer model with a unified encoder, which is simple and can model the bi-directional relationship between the contexts and the source sentences. besides, our flat-transformer is compatible with the pretraining model, yielding a better performance than both the existing uni-encoder models and the dualencoder models on two datasets.