Generalised Unsupervised Domain Adaptation of Neural Machine Translation with Cross-Lingual Data Selection (2021)

Abstract: This paper considers the unsupervised domain adaptation problem for neural machine translation (NMT), where we assume the access to only monolingual text in either the source or target language in the new domain. We propose a cross-lingual data selection method to extract in-domain sentences in the missing language side from a large generic monolingual corpus. Our proposed method trains an adaptive layer on top of multilingual BERT by contrastive learning to align the representation between the source and target language. This then enables the transferability of the domain classifier between the languages in a zero-shot manner. Once the in-domain data is detected by the classifier, the NMT model is then adapted to the new domain by jointly learning translation and domain discrimination tasks. We evaluate our cross-lingual data selection method on NMT across five diverse domains in three language pairs, as well as a real-world scenario of translation for COVID-19. The results show that our proposed method outperforms other selection baselines up to +1.5 BLEU score.

## introduction unsupervised domain adaptation (uda) aims to generalise mt models trained on domains with typically large-scale bilingual parallel text to new domains without parallel data @xcite . most prior works in uda of nmt assume the availability of either non-parallel texts of both languages or only the target-language monolingual text in the new domain to adapt the nmt model. the adaptation is achieved by modifying the model architecture and joint training with other auxiliary tasks @xcite @xcite , or constructing a parallel corpus for the new domain from a general-domain parallel text using data-selection methods @xcite . however, very little attention has been paid to the uda problem with only the source-language monolingual text in the new domain. in practice, this setting is not very rare, e.g. building a translation system from english to shona (a low-resource african language) in a specific domain such as healthcare and disaster. while it would be very time consuming to collect in-domain text in shona, english corpora are more accessible. in this paper, we consider the generalised problem of uda in nmt where we assume the availability of monotext in only one language, either the source or target, in the new domain. we propose a generalised approach to the problem using crosslingual data selection to extract sentences in the new domain for the missing language side from a large monolingual generic corpus. our proposed data selection method trains an adaptive layer on top of multilingual bert by contrastive learning @xcite , such that the representations of source and target language are aligned. the aligned representations enable the transferability of a domain classifier trained on one language side to the other language for in-domain data detection. previous works have explored filtering data of the same language for mt @xcite @xcite ; however, utilising data in one language to detect in-domain data in the other language is under-explored. with selected sentences in the new domain of the missing language side, the original adaptation problem is transformed to the usual setting of uda problem, and can be approached by the existing uda methods. in this paper, we extend the discriminative domain mixing method for supervised domain adaptation @xcite which jointly learns domain discrimination and translation to the unsupervised setting. more specifically, the nmt model jointly learns to translate with the translation loss on pseudo bitext, and captures the characteris-tics of the new domain by the domain discrimination loss on data from the old and new domains. our contributions can be summarised as follows: ## generalised unsupervised domain adaptation domain adaptation is an important problem in nmt as it is very expensive to obtain training data that are both large and relevant to all possible domains. supervised adaptation problem requires the existence of out-of-domain (ood) bitext and indomain bitext. unsupervised domain adaptation problem assumes ood and in-domain monotext, usually in the target language. a domain is defined as a distribution p (x, y ) where x ranges over sentences in the source language s, and y is its translation in the target language t. we define the generalised unsupervised domain adaptation (guda) for nmt as the problem of adapting an nmt model trained on an old domain p old (x, y ) to a new domain p new (x, y ), where only either the source or target language text is available in the new domain. since p (x, y ) = p s (x)p s,t (y |x), let us consider p s old (x) which is the distribution over sequences on the source language s in the old domain. it is usually much richer (i.e., containing diverse categories such as news, politics, etc.) than p s new (x) which is typically a much more specific category where we aim to adapt the nmt model. the conditional distribution p s,t old (y |x) specifies the encoder-decoder nmt network to be adapted to the new domain. given parallel bitext d @xmath0, y y y i )} in the old domain, we consider two settings in guda: crucially, in both cases we do not require any parallel text in the new domain, hence the term unsupervised domain adaptation. the goal is to adapt an nmt model, parametrised by θ θ θ, trained on the old domain bitext d old to the new domain. in the setting involving y new , it can be used to create pseudo-parallel data via back-translation @xcite , or to adapt the decoder via multi-task learning @xcite . this setting is the usual formulation in uda for nmt @xcite . in contrast, the setting involving the source monotext x new is not well explored in the literature. our approach for addressing guda is to create in-domain monotext for the language side, where the data in the new domain is missing. that is, if given x new , we build a classifier to select indomain monotext y new in the target language from the generic monotext d t . we perform a similar procedure for the other case where only in-domain y new is present. we then adapt the nmt model based on the bitext from the old domain as well as the source and target language monotext in the new domain. the challenge, however, is how to train a classifier for data selection for the languageside with missing data. we address this problem in section 3, then mention how to adapt the nmt model to the new domain in section 4. ## cross-lingual in-domain data selection aharoni and goldberg (2020) have shown that the emergent domain clusters via bert @xcite can be used to select in-domain bitext for nmt. inspired from that observation, we leverage the sentence representations produced by the multilingual bert (mbert) for cross-lingual monotext selection. we first align the source and target language representation space while preserving the domain clustering characteristics in each space. using the available monotext in one language, we train a binary classifier to detect old and new domains on the aligned semantic spaces. this classifier is then transferred to pick in-domain sentences in the other language (fig. 1 ). ## nmt adaptation to the new domain given the parallel data in the old domain d old and monolingual data in the new domain for both the source language x new and target language y new , we adapt the nmt model by minimising the loss, bitext loss. we create pseudo-bitext d new by back-translating y new using a reverse-direction translation model trained on d old . the quality of the pseudo-bitext depends on the quality of the reverse-direction nmt model in the new domain. we further mix the pseudo-bitext d new with the olddomain bitext d old to form the bitext loss function ## experiments we evaluate our proposed approach for guda on the three language pairs covering five domains, and a real-world translation task, namely, tico-19. ## conclusion we have proposed a cross-lingual data selection method to the guda problem for nmt where only monolingual data from one language side is available in the new domain. we first learn an adaptive layer to align the bert representation of the source and target languages. we then utilise a domain classifier trained on one language to select in-domain data for another. experiments on translation tasks of several language pairs and domains show the effectiveness of our method over other baselines.