Multilingual Domain Adaptation for NMT : Decoupling Language and Domain Information with Adapters (2021)

Abstract: Adapter layers are lightweight, learnable units inserted between transformer layers. Recent work explores using such layers for neural machine translation (NMT), to adapt pre-trained models to new domains or language pairs, training only a small set of parameters for each new setting (language pair or domain). In this work we study the compositionality of language and domain adapters in the context of Machine Translation. We aim to study, 1) parameter-efficient adaptation to multiple domains and languages simultaneously (full-resource scenario) and 2) cross-lingual transfer in domains where parallel data is unavailable for certain language pairs (partial-resource scenario). We find that in the partial resource scenario a naive combination of domain-specific and language-specific adapters often results in ‘catastrophic forgetting’ of the missing languages. We study other ways to combine the adapters to alleviate this issue and maximize cross-lingual transfer. With our best adapter combinations, we obtain improvements of 3-4 BLEU on average for source languages that do not have in-domain data. For target languages without in-domain data, we achieve a similar improvement by combining adapters with back-translation. Supplementary material is available at https://tinyurl.com/r66stbxj .

## introduction multilingual neural machine translation (nmt) has made a lot of progress recently @xcite @xcite @xcite and is now widely adopted by the community and mt service providers. multilingual nmt models handle multiple language directions at once and allow for knowledge transfer to low-resource languages. machine translation systems often need to be adapted to specific domains like legal or medical text. however, when adapting multilingual systems, in-domain data for most language pairs might not exist. we would like to be able to leverage data in a subset of language pairs to transfer domain knowledge to other languages. straightforward methods of domain adaptation include fine-tuning @xcite or usage of domain tags @xcite for different domains. for these methods each new domain request would require retraining the whole model, which is a costly procedure. and naive training on a subset of languages typically reduces performance on all other languages @xcite , a phenomenon known as 'catastrophic forgetting' @xcite ). an alternative technique for adapting such models to new language-pairs or domains are 'adapter layers' @xcite , lightweight, learnable units inserted between transformer layers. a previously trained large multilingual model can be adapted to each language-pair by learning only these small units, and keeping the rest of the model frozen. this procedure also allows for the incremental adding of new language pairs and/or domains to the pre-trained model, reducing the cost of adaptation. previous studies have shown it is possible to combine language-specific (as opposed to language-pair specific) adapters @xcite , or language and task adapters @xcite trained independently, enabling zero-shot compositions of adapters. our ultimate goal is, for ease of deployment and storage, a single model that can handle all languages and domains. in this work we analyse how to combine language adapters with domain adapters in multilingual nmt, and study to what extent the domain knowledge can be transferred across languages. first, we show it is hard to decouple language knowledge from domain knowledge when finetuning multilingual mt systems on new domains. in section 5.2 we demonstrate that adapters learnt on a subset of language pairs fail to generate into languages not in that subset. such generation into the wrong language is referred to as 'off-target' translation. we additionally find combinations of domain and language adapters not seen at training time lead to bad performance. we examine how adapter placement and other techniques can improve the compositionality of language and domain adapters when dealing with source or target languages that do not have in-domain data (which we refer to throughout this work as "out-of-domain languages"). our key contributions are: domain adaptation in nmt domain adaptation has been discussed extensively for bilingual nmt models. a typical approach is to fine-tune a model trained on a large corpus of 'generic' data on a smaller in-domain corpus @xcite . a common technique to make use of monolingual in-domain data is to do back-translation @xcite @xcite . although effective, it is expensive to create back-translated data, especially when one needs to cover multiple language pairs. multi-domain models can be trained with domain tags @xcite @xcite that can encode domain-specific information. however, domain tags do not allow incrementally adding new domains to a model: each new domain adaptation requires retraining the full model (as opposed to adapter layers that can be trained independently for each language/domain). there are a number of works @xcite @xcite trying to explicitly decouple domainspecific representations from domain independent representations in bilingual settings. in our work we try to decouple language and domain specific representations through adapter layers. adapter layers @xcite introduce adapter layers for nmt as a lightweight alternative to fine-tuning. they study both adding language-pair specific adapters to multilingual nmt models to match the performance of a bilingual version, and domain-specific adapters for parameter-efficient domain adaptation. @xcite train adapters for each language instead of language-pair and show that composing such adapters improves zero-shot translation in englishcentric settings, and can adapt a model to all language directions in a scalable way. @xcite study adapter layers for pre-trained language models evaluated on nlu tasks. they show it is possible to compose language and task adapters. combining language adapters trained with a masked language modelling objective for language x and task adapters trained on a classification task in language y can transfer to classification in language x. we have a similar objective to pfeiffer et al. ( 2020 ), but for nmt where in addition to encoding sentences we need to generate text for new language and domain combinations. to the best of our knowledge none of the works above study composing language and domain adapters for generation tasks (such as translation) which is the goal of this work. ## composing adapter modules adapter modules @xcite are randomly initialised modules inserted between the layers of a pre-trained network and fine-tuned on new data. an adapter layer is typically a down projection to a bottleneck dimension followed by an up projection to the initial dimension, which we write as ffn(h) = w up f (w down h), with f (•) a non-linearity. the bottleneck controls the parameter count of the module; typically nmt requires slightly larger parameter counts than classification to match fine-tuning @xcite . with a residual connection and a nearidentity initialization the original model is (approximately) retained at the beginning of optimization, keeping at least the performance of the parent model. ## results and discussion first, in section 5.1 we discuss the results of experiments testing the domain adaptation capacity of various models, assuming access to data for all language pairs. in section 5.2 we analyse domain transfer across languages with adapters and other methods. we first demonstrate problems with cross-lingual generalisation during domain adaption for 'naive' methods, and then propose potential solutions. note we concentrate on the medical domain and a particular language subset for convenience. appendix d has results in ## conclusion in this work we studied multilingual domain adaptation both in the full resource setting where indomain parallel data is available for all the language pairs, as well as the partial resource setting, where in-domain data is only available for a small set of languages. in particular, we study how to better compose language and domain adapter modules in the context of nmt. we find that while adapters for encoder architectures like bert can be safely composed, this is not true for nmt adapters: domain adapters learnt in the partial resource scenario struggle to generate into languages they were not trained finally we note that a model fine-tuned with domain tags serves as a very competitive baseline for multilingual domain adaptation. on the other hand, domain adaptation with adapters offers modularity, and allows incrementally adapting to new domains without retraining the full model. future research directions could explore multi-task training combining parallel and monolingual in-domain data in other ways to alleviate the need for backtranslation. our work is the first attempt to combine domain adapters and language adapters for a generation task (nmt). although such combinations have shown to be successful for nlu tasks, obtaining good representations for generating unseen target languages proves to be a difficult problem. we believe a fine-grained study of where to use language or domain-specific capacity could lead to better cross-lingual domain transfer in future. finally, we provide supplementary material to facilitate reproducibility. 10 a data and hyper-parameters for bilingual domain adaptation we use a transformer base @xcite model trained for 12 epochs on german to english wmt20 data (47m parallel lines), with a joint bpe @xcite vocabulary of size 24k with inline casing @xcite (i.e. wordpieces are put in lowercase with a special token indicating their case.). for bilingual domain adaption we use the same datasets as aharoni and goldberg (2020), namely parallel text in german and english from five diverse domains: koran, medical, it, law and subtitles. for multilingual settings we use the following hyper-parameters. we share embeddings between encoder and decoder. we use the adam optimizer (kingma and ba, 2014) with an inverse square root learning rate schedule for pre-training, and a fixed learning rate schedule for training adapters. we speed up training with 16 bit floating point arithmetic. we use label smoothing 0.1 and dropout 0.1. we train for either 20 epochs or 1 million updates, whichever corresponds to the smallest number of training updates. we use early stopping, checking performance after each epoch or every 100,000 training steps, and use average validation negativelog-likelihood on all of the training data (but not out-of-domain language data) as our criteria for choosing the best model. we otherwise use default fairseq @xcite parameters. we train all models on a single nvidia v100 gpu, and training takes between 8 and 36 hours depending on dataset size. in order to create validation and test splits that had no overlap with training data in any language, we first set aside a number of english sentences. then we aligned all language pairs to these sentences, i.e. the german to french test set is composed of german and french sentences that share the same english sentence. finally we remove all sentences in any language from the train splits of all parallel data if those sentences are aligned with any english sentences in the subset we set aside for validation/test splits. both validation sets and test sets contain around 2000 examples for every domain and language-pair.