Breaking the Corpus Bottleneck for Context-Aware Neural Machine Translation with Cross-Task Pre-training (2021)

Abstract: Context-aware neural machine translation (NMT) remains challenging due to the lack of large-scale document-level parallel corpora. To break the corpus bottleneck, in this paper we aim to improve context-aware NMT by taking the advantage of the availability of both large-scale sentence-level parallel dataset and source-side monolingual documents. To this end, we propose two pre-training tasks. One learns to translate a sentence from source language to target language on the sentence-level parallel dataset while the other learns to translate a document from deliberately noised to original on the monolingual documents. Importantly, the two pre-training tasks are jointly and simultaneously learned via the same model, thereafter fine-tuned on scale-limited parallel documents from both sentence-level and document-level perspectives. Experimental results on four translation tasks show that our approach significantly improves translation performance. One nice property of our approach is that the fine-tuned model can be used to translate both sentences and documents.

## introduction document-level context-aware neural machine translation (nmt) aims to translate sentences in a document under the guidance of document-level context. recent years have witnessed great improvement in context-aware nmt with extensive attempts at effectively leveraging document-level context @xcite @xcite , to name a few). however, the performance of contextaware nmt still suffers from the size of parallel document dataset. on the one hand, unlike sentence-level translation models which could be well trained on large-scale sentence-level parallel datasets, the translation models of context-aware nmt may result in insufficient training. on the other hand, with only scale-limited source-side documents, the context encoders may fail to effectively extract useful context from the whole document. 2 on the contrary, large-scale of parallel sentence corpora, and especially monolingual document corpora are much easier to find. in this paper, our goal is to break the corpus bottleneck for context-aware nmt by leveraging both largescale sentence-level parallel dataset and monolingual documents. specifically, we aim to use the former to boost the performance of translation models while employ the latter to enhance the context encoders' capability of capturing useful context information. there have been several attempts to boost context-aware nmt performance in the scenarios where the document-level parallel dataset is scale-limited, or even not available. on the one hand, sentence-level parallel dataset is a natural resource to use. for example, @xcite propose a two-stage training strategy for context-aware nmt by pre-training the model on a sentencelevel parallel dataset. on the other hand, junczys-dowmunt (2019) leverage large-scale source-side monolingual documents, in which they simply concatenate sentences within a document into a long sequence and explore multi-task training via the bert-objective @xcite on the encoder. due to that different models are usually required to model sentences and documents, however, it is challenging to effectively take them both in a single model. in order to effectively and simultaneously model both sentence-level parallel dataset and monolingual documents, in this paper we propose a novel cross-task pre-training approach. as shown in figure 1 , we define two pre-training tasks. one learns to translate a sentence from source language to target language while the other learns to translate a document from deliberately noised to original. importantly, the two pre-training tasks are jointly learned via the same model synchronously. then we use document-level parallel dataset to fine-tune the properly pre-trained models. similarly to the pre-training, we can fine-tune the models from both sentence-level and document-level perspectives. experimental results on four document-level translation tasks show that our approach significantly improves translation performance, suggesting the effectiveness of our approach in modeling both sentence-level parallel dataset and monolingual documents. one nice property of our approach is that the fine-tuned models can be used to translate both sentences and documents. in the following, we first describe our pre-training tasks defined upon sentence-level parallel dataset and large-scale monolingual documents (section 2.1). then we detail our model which caters such pre-training tasks (section 2.2). finally, we present our joint pre-training (section 2.3). ## experimentation to test the effect of our approach in leveraging sentence-level parallel dataset and monolingual documents, we carry out experiments on chineseto-english (zh-en) and english-to-german (en-de) translation. ## discussion next we use zh-en translation to analyze more on how our approach affects translation performance. see appendix b for parameter analysis and statistics of the pre-trained models. ## related work we describe related studies in the following two perspectives. ## conclusion in order to leverage both large-scale sentence-level parallel dataset and source-side monolingual documents for context-aware nmt, in this paper, we have proposed a novel cross-task pre-training approach, which simultaneously learns to translate a sentence from source language to target language while denoising a document from deliberately noised to original. upon the pre-trained models, we fine-tune them with document-level parallel dataset from both sentence-level and documentlevel perspectives. experimental results on multiple document-level translation tasks have demonstrate the effectiveness of our approach. finally, we also provide insights on how context-aware nmt benefits from our approach.