Domain Adaptation for NMT via Filtered Iterative Back-Translation (2021)

Abstract: Domain-specific Neural Machine Translation (NMT) model can provide improved performance, however, it is difficult to always access a domain-specific parallel corpus. Iterative Back-Translation can be used for fine-tuning an NMT model for a domain even if only a monolingual domain corpus is available. The quality of synthetic parallel corpora in terms of closeness to in-domain sentences can play an important role in the performance of the translation model. Recent works have shown that filtering at different stages of the back translation and weighting the sentences can provide state-of-the-art performance. In comparison, in this work, we observe that a simpler filtering approach based on a domain classifier, applied only to the pseudo-training data can consistently perform better, providing performance gains of 1.40, 1.82 and 0.76 in terms of BLEU score for Medical, Law and IT in one direction, and 1.28, 1.60 and 1.60 in the other direction in low resource scenario over competitive baselines. In the high resource scenario, our approach is at par with competitive baselines.

## introduction neural machine translation (nmt) @xcite systems heavily rely on the availability of the parallel corpora to produce good quality translations @xcite . even for high resource language pairs, in-domain parallel corpora are scarce. @xcite ; @xcite address this challenge of domain adaptation with the objective of improving the performance of an nmt system by exploiting the in-domain monolingual corpora and out-of-domain parallel corpora for a given language pair. in the current work, we build on top of the existing data centric approaches for domain adaptation @xcite , i.e., back-translation (bt) @xcite and iterative back-translation (ibt) @xcite . ibt is a variant of bt, which leverages both source and target-side monolingual corpora along with the out-of-domain parallel corpora and train n m t @xmath0, where n m t @xmath1. the performance of the nmt is influenced by the quality of synthetic parallel corpora as noted by @xcite ; @xcite . hence, for the domain adaptation task, @xcite proposed a curriculum-based approach (ddswibt) for sentence selection from the in-domain monolingual corpora and use junczys-dowmunt (2018) for weight assignment to synthetic parallel corpora. in initial iterations of ibt, ddswibt prefer simple sentences over representative in-domain sentences, in later iterations they use more representative sentences as compared to simple sentences. meanwhile, @xcite use an in-domain language model (sent-lm) and "round-trip bleu" score for synthetic parallel corpora filtering. in this paper, we propose a "classifier augmented filtered iterative back-translation" (cfibt) for the domain adaptation task. we train two convolutional neural network (cnn) @xcite based binary classifiers, one in source and the other in the target language on the combination of in-domain and out-of-domain corpora. we use ibt for synthetic parallel corpora generation and classifierbased filtering to remove the pair of sentences where the synthetic sentence in the pair does not belong to the domain. this entire procedure is depicted in figure 1 . we do not employ sentence selection over the monolingual corpora, or a weighting mechanism for the synthetic corpora, and neither utilize any "round-trip" criteria for scoring the synthetic parallel corpora. in the current work, we present our domain adaptation results for the german (de) -english (en) language pair on three different domains -medical, law and it under low and high resource scenarios. in the low resource scenario, our proposed method cfibt outperforms all the baselines in every domain. in the high resource settings, cfibt outperforms the baselines in most of the scenarios, whereas it performs competitively with the best baseline results in the rest of the scenarios. the rest of the paper is organized as follows: we describe related work in section 2, our problem statement in section 3, the proposed approach in section 4. we present the results of the proposed and other baseline approaches in section 5 and conclude in section 6. ## related work in the current work, our objective is to improve the performance of the nmt model on in-domain sentences given out-of-domain parallel corpora and indomain monolingual corpora in both source and target language, which is known as domain-adaptation for nmt @xcite . approaches for domain-adaptation @xcite are categorized into data-centric and modelcentric. data-centric approaches for domain adaptation focuses on the use of in-domain monolingual corpora @xcite , synthetic corpora @xcite @xcite , or parallel corpora @xcite along with the out-of-domain parallel corpora. on the other hand, model-centric approaches modify the nmt architecture to include domain information i.e. domain-tags @xcite , domain embedding with word embeddings @xcite and assign higher weights to in-domain sentences as compared to out-of-domain sentences @xcite . in our current work, we use the data-centric approach for domain adaptation to generate synthetic parallel data via iterative back-translation. @xcite ; @xcite ; foster et al. (2010); søgaard (2011); sgaard (2013) have proposed different instance weighting based approaches for domain adaptation in nlp, where in-domain instances are assigned more weight as compared to the out-of-domain instances. in nmt, @xcite ; fadaee and monz (2018) observe that noisy sentences in synthetic parallel corpora can affect the performance of the translation model. round-trip bleu score @xcite between the authentic and synthetic versions of the same sentence is used by @xcite to filter noisy synthetic corpora. and they also use the language model trained on indomain monolingual data to filter the noisy sentences. similarly, @xcite ## problem description given out-of-domain parallel corpora d p , and indomain monolingual corpora m s , m t in source and target language respectively. our objective is to create a pair of in-domain nmt models n m t @xmath2, n m t @xmath3. similar to edunov et al. (2018) , we use more d p in high resource scenario as compared to low resource scenario and the same amount of m s , m t in both the scenario. ## proposed approach we describe our proposed approach "classifier augmented filtered iterative back-translation" (cfibt) in algorithm 1. we assume that we have access to out-of-domain parallel corpora and in-domain ## results and discussion here, we describe the datasets and the training details of our experiments. we also discuss and analyze the results and key observations. ## conclusion in the context of domain adaptation for nmt, we propose a simple and effective approach for filtering the synthetic parallel corpus, which is as good as more involved approaches for the same task. in the low resource scenario, the proposed approach outperforms all the existing baselines whereas we get similar results to baselines in the high resource scenario. as a part of future work, we would like to validate our findings on different language pairs and multiple domains and also would like to explore the combination of different filtering techniques. and instead of training different filtering models for source and target language, we would like to use a single multilingual filtering model.