Improving Document-Level Neural Machine Translation with Domain Adaptation (2020)

Abstract: Recent studies have shown that translation quality of NMT systems can be improved by providing document-level contextual information. In general sentence-based NMT models are extended to capture contextual information from large-scale document-level corpora which are difficult to acquire. Domain adaptation on the other hand promises adapting components of already developed systems by exploiting limited in-domain data. This paper presents FJWU’s system submission at WNGT, we specifically participated in Document level MT task for German-English translation. Our system is based on context-aware Transformer model developed on top of original NMT architecture by integrating contextual information using attention networks. Our experimental results show providing previous sentences as context significantly improves the BLEU score as compared to a strong NMT baseline. We also studied the impact of domain adaptation on document level translationand were able to improve results by adaptingthe systems according to the testing domain.

## introduction in past few years, machine translation systems have witnessed remarkable growth due to increasing amount of multilingual information. neural machine translation (nmt) has become one of the powerful and de-facto approaches recognized for its generality and effectiveness @xcite . due to better accuracy of deep neural models, it has quickly achieved state of the art performance in machine translation @xcite . standard neural machine translation model works on individual sentences and focuses on short context windows for improving translation quality while ignoring cross-sentence links and dependencies @xcite . sentence-by-sentence translation of well-formed documents may generate an incoherent target text which is unable to span the entire document. this largely limits the success of nmt, as document context is totally ignored. intuitively, to generate coherent translation of source document, machine learning models expect cross-sentence dependencies and linkages. to this end, several models @xcite @xcite @xcite have been proposed for document-wide translation. adapting nmt models for context-aware translations has the biggest challenge of limited availability of bilingual document-level corpora. since, only few resources are available for training, the application domain of nmt may greatly vary from domain of training data. consequently, the performance of nmt system may quickly degrade as soon as the testing conditions deviate from training conditions. domain adaptation has been an active research topic in the field of machine translation to improve translation performance often for low resource settings @xcite . the quality of neural machine translation heavily depends upon domain-specificity of test data and the amount of parallel training data. the demand for high quality domain specific mt systems has significantly increased over the years but the bilingual corpora for relevant languages still lack in quantity @xcite . in this work, we aim to demonstrate performance optimization in a particular domain by training document-level models on large out-of domain parallel corpus combined with small in-domain corpus using domain adaptation techniques. our experiments on german-english data using documentlevel translation model @xcite an extension of standard nmt transformer @xcite reveals the importance of contextual information and domain adaptation on translation quality. by comparing the performance with standard nmt baseline models trained on bilingual data, we show that nmt models exposed to random and actual contextual information are more sensitive to translation quality. we also demonstrate the impact of domain adaptation on translation quality by adapting document-level system to the testing domain. ## models we use two types of models, sentence level and document-level context aware model, both built using transformer architecture @xcite . for our primary submission we train documentlevel models on parallel corpus with document boundaries. the document level model consists of hierarchical attention encoder and decoder to capture both source and target side contextual information during training and testing. ## experimental setup the baseline and document-level models are trained on english-german parallel data of differ-ent domains (i.e. news, press and sports) provided by wmt19 foot_0 and wngt20 foot_1 . ## experimental results we present the results of experimentation from our models on german-english translation in tables 2, 3 and 4. ## conclusion in this study, we present methods to improve document-level neural machine translation. following recently reported results on the task, our experiments also reiterate the fact that incorporating context in translation helps considerably improve the quality. taking a strong transformer based baseline model trained on substantial corpus (a concatenation of four corpora rotowire, rapid, euro and news commentary), context aware document models result in significant improvement in bleu points. we have also experimented with the effects of corpus replication to adapt to the domain of test corpus. we find it an effective method to improve translation quality and domain adaptation. we have submitted results of our best model (han encoder) for german-english direction as reported in