Can Domains Be Transferred across Languages in Multi-Domain Multilingual Neural Machine Translation? (2022)

Abstract: Previous works mostly focus on either multilingual or multi-domain aspects of neural machine translation (NMT). This paper investigates whether the domain information can be transferred across languages on the composition of multi-domain and multilingual NMT, particularly for the incomplete data condition where in-domain bitext is missing for some language pairs. Our results in the curated leave-one-domain-out experiments show that multi-domain multilingual (MDML) NMT can boost zero-shot translation performance up to +10 gains on BLEU, as well as aid the generalisation of multi-domain NMT to the missing domain. We also explore strategies for effective integration of multilingual and multi-domain NMT, including language and domain tag combination and auxiliary task training. We find that learning domain-aware representations and adding target-language tags to the encoder leads to effective MDML-NMT.

## introduction multilingual nmt (mnmt), which enables a single model to support translation across multiple directions, has attracted a lot of interest both in the research community and industry. the gap between mnmt and bilingual counterparts has been reduced significantly, and even for some settings, it has been shown to surpass bilingual nmt @xcite . mnmt enables knowledge sharing among languages, and reduces model training, deployment, and maintenance costs. on the other hand, multi-domain nmt aims to build robust nmt models, providing high-quality translation on diverse domains. while multilingual and multidomain nmt are highly appealing in practice, they are often studied separately. to accommodate the domain aspect, previous mnmt works focus on learning a domain-specific mnmt by finetuning a general nmt model on the domain of interest @xcite . @xcite propose to unify multilingual and multi-domain nmt into a holistic system by stacking language-specific and domain-specific adapters with a two-phase training process. thanks to the plug-and-play ability of adapters, their system can handle translation across multiple languages and support multiple domains. however, as each domain adapter is learned independently, their adapter-based model lacks the ability of effective knowledge sharing among domains. in this paper, we take a step further toward unifying multilingual and multi-domain nmt into a single setting and model, i.e., multi-domain multilingual nmt (mdml-nmt), and enable effective knowledge sharing across both domains and languages. unlike the complete data assumption in the multi-domain single language-pair setting where training data is available in all domains, we assume the existence of bitext in all domains for only a subset of language-pairs, as illustrated in figure 1 (a). in fact, it is highly improbable to obtain in-domain bitext for all domains and all language pairs in many real-life settings. depending on the availability of parallel data, we categorise a translation task from a source to a target language into four categories based on the following dimensions: please note the domain and language-pair matrix in figure 1 @xcite . in this figure, parallel data available in the training set specifies the group a, the in-domain seen tasks. given this training dataset, most mnmt research focuses on crosslingual transfer to in-domain unseen translation tasks (@xmath0, while the studies on multi-domain nmt and domain adaptation seek to generalise to out-of-domain seen translation tasks (@xmath1. integrating domain and language aspects in the incomplete data condition gives rise to an interesting and more challenging setting that transfers to outof-domain unseen translation tasks (@xmath2. we hypothesise that the out-of-domain "seen and unseen" translation tasks (@xmath3. specifically, we ask the following research questions: (1) do out-of-domain translation tasks benefit from the out-of-domain and in-domain bitext in other seen translation pairs? and (2) what is effective method to handle the composition of domains and languages? furthermore, beyond the crosslingual transfer (@xmath4, we also consider the challenging setting where the translation direction of interest may not have any bitext in any domain, i.e. the zero-shot setting (@xmath5. in general, we can vary the degree of domain transfer based on the number of domains in which parallel data for a translation task is available. combining with the number of language pairs of interest, there are large numbers of incomplete data conditions, even for our toy examples in figure 1 . in this study, we assume the highest degree of domain transfer and carefully design controlled experiments where one domain is left out for some language pairs ( ## multi-domain multilingual nmt in this section, we first provide the necessary background on multilingual nmt (mnmt) and multidomain nmt individually. we then describe effective modelling approaches for the integration of multi-domain and multilingual nmt (mdml-nmt). ## experiments in this section, we evaluate the mdml-nmt approaches and seek to answer the following research questions (rqs): we explore the benefits of having a single mdml model trained on all available training data from multiple languages and domains over the multi-domain bilingual (mdbl) and the single domain multilingual (sdml) models learned on a subset of training data from a single language pair or domain. we carefully design controlled experiments to build incomplete data conditions and study the translation quality of the unified mdml-nmt model on both seen and unseen (zero-shot) translation directions. we hypothesise that the translation involving the out-of-domain languages can be beneficial from the in-domain languages thanks to the knowledge sharing across domain and languages. we investigate strategies for effective integration of existing multi-domain and multilingual nmt methods, including the use of language and domain tags and auxiliary task training. ## conclusion we study the problem of mdml-nmt for which a single nmt can support multiple translation directions and domains. we investigate whether the tagging and auxiliary task learning method can be combined for mdml-nmt. our empirical results reveal a positive transfer from in-domain to out-ofdomain languages, especially in the zero-shot scenario. this study provides insights into the synergy of the domain and language aspects of training an mdml-nmt model. the main findings include: (i) it is crucial to make the encoder domain-aware; and (ii) it is best to prepend the target language tag to the encoder in mdml. these findings lay the groundwork for future research in this direction.