Context-Interactive Pre-Training for Document Machine Translation (2021)

Abstract: Document machine translation aims to translate the source sentence into the target language in the presence of additional contextual information. However, it typically suffers from a lack of doc-level bilingual data. To remedy this, here we propose a simple yet effective context-interactive pre-training approach, which targets benefiting from external large-scale corpora. The proposed model performs inter sentence generation to capture the cross-sentence dependency within the target document, and cross sentence translation to make better use of valuable contextual information. Comprehensive experiments illustrate that our approach can achieve state-of-the-art performance on three benchmark datasets, which significantly outperforms a variety of baselines.

## introduction document machine translation (doc-mt) aims at utilizing the surrounding contexts of the source sentence to tackle some linguistic consistency problems (e.g., deixis, ellipsis, and lexical cohesion) in translation @xcite . however, due to the introduction of extra contexts, it also presents several intractable challenges: to tackle the above three challenges, here we propose a simple yet effective context-interactive pre-training approach for doc-mt. the proposal consists of three pre-training tasks, whose sketch is presented in figure 1 . specifically, the cross sentence translation task (cst in figure 1 (a) ) strives to generate the target sentence in the absence of the source sentence and only based on the source contexts. with such a goal, the model is encouraged to maximize the utilization of extra contexts. to capture interactions between multiple sentences in the target document so that the discourse phenomena can be modeled, we conduct inter sentence generation (isg in figure 1 (b) ) that aims to predict the inter sentence based on the target surrounding contexts. this task can be regarded as discourse language modeling that injects the cross-sentence dependency within the target document into the decoder of the translation model. we also introduce parallel sentence translation (pst in figure 1 (c )) to alleviate the lack of doc-level bilingual corpora and achieve knowledge transfer from abundant sentlevel parallel data to limited doc-level parallel data. in order to avoid the catastrophic forgetting of pretrained model in downstream fine-tuning, elastic weight consolidation (ewc) regularization is introduced to further enhance the model performance. transformer encoder it's good. </@xmath0. </@xmath1. </@xmath2, which is able to outperform a variety of baselines. uni-encoder structure. this line of research aims at performing doc-mt based on a universal transformer, which takes the concatenation of the additional contexts and the source sentence as the input. @xcite explores multiple different concatenation strategies and proves that the translation with extended source achieves the best performance. @xcite presents several new discourse testsets, which aims to evaluate the ability of the models to exploit previous source and target sentences. @xcite utilizes dynamic or topic cache to model coherence for doc-mt by capturing contextual information either from recently translated sentences or the entire document. going a step further, they @xcite presents an inter-sentence gate model to encode two adjacent sentences and controls the amount of information flowing from the preceding sentence to the translation of the current sentence with an inter-sentence gate. @xcite augments translation model with a cache-like memory network that stores recent hidden representations as translation history. yang et al. (2019) introduce a query-guided capsule networks into document-level translation to capture high-level capsules related to the current source sentence. @xcite proposes a unified encoder to process the concatenated source information that only attends to the source sentence at the top of encoder blocks. dual-encoder structure. this line of work tends to adopt two encoders or another components to model the source sentences and the document-level contexts. @xcite summarize the source history in a hierarchical way and then integrate the historical representation into translation model with multiple strategies. @xcite takes both source and target document context into account using memory networks, which modeling doc-mt as a structured prediction problem with inter-dependencies among the observed and hidden variables. zhang et al. (2018) et al. (2019) presents a hierarchical context-aware translation model, which selectively focus on relevant sentences in the document context and then attends to key words in those sentences. ## methodology following prior work @xcite , we translate the i-th source sentence x i into the i-th target sentence y i in the presence of extra source contexts @xmath3, x i+1 ), where x i-1 and x i+1 refer to the predecessor and successor of x i respectively. we adopt transformer as the model architecture of pre-training and machine translation. the model is trained by minimizing the negative log-likelihood of target sequence y conditioned on the source sequence x, i.e., @xmath4. readers can refer to @xcite for more details. we introduce our approach based on @xmath5.