Domain Generalisation of NMT : Fusing Adapters with Leave-One-Domain-Out Training (2022)

Abstract: Generalising to unseen domains is under-explored and remains a challenge in neural machine translation. Inspired by recent research in parameter-efficient transfer learning from pretrained models, this paper proposes a fusion-based generalisation method that learns to combine domain-specific parameters. We propose a leave-one-domain-out training strategy to avoid information leaking to address the challenge of not knowing the test domain during training time. Empirical results on three language pairs show that our proposed fusion method outperforms other baselines up to +0.8 BLEU score on average.

## introduction building robust machine translation (mt) models that can perform well on a test set outside the domain of training examples is highly desired in real-world scenarios. despite recent great progress in neural machine translation (nmt) research, nmt models have been found sensitive to distribution shift and adversarial examples @xcite @xcite . while improving an nmt model to a new domain has been studied extensively in domain adaptation settings where in-domain parallel or monolingual data is given @xcite , generalising nmt models to unseen domains is under-explored @xcite . domain generalisation is a problem setting in machine learning that tackles the challenge of learning a robust model for unseen domains from multiple existing domains. this problem is closely related to several settings such as multi-task learning, transfer learning, and domain adaptation in terms of learning models from one or more given tasks/domains to enhance performance on some target tasks/domains. the main difference and challenge in domain generalisation is that the test do- * work done while doing internship at ebay inc. mains are unknown in advance. previous works on domain generalisation focused on learning invariant features by minimising the difference in the representations of the given domains for the classification tasks @xcite @xcite . learning a domain-invariant representation is applicable to the classification problems where such invariances may be sufficient to predict the target classes. however, it may be inadequate for translation tasks. a good translation should not only preserve the invariant features such as syntax and grammar, but also be able to maintain the domain-specific features such as style of the source sentence. in this paper, we propose a fusion-based approach to the domain generalisation problem for nmt. our method comprises two training stages. the first stage is to learn domain-specific features through adapter modules added to the pretrained encoder-decoder model. previous works have shown that the task-specific adapter is an effective alternative method to fine-tuning. it allows fast adaptation of pretrained language models to downstream tasks @xcite , and multilingual nmt models to new language pairs @xcite . in the second stage, we propose to use an adapterfusion module @xcite and train it to effectively combine features of the existing domains in order to handle unseen domains. unlike @xcite who trains the adapterfusion module for transfer learning (from existing tasks to a seen target task), we do not have access to the test domain during training. to address this challenge, we propose a novel leave-onedomain-out (lodo) training strategy by creating homogeneous mini-batches consisting of training examples from a single domain and disabling the corresponding domain adapter when optimising the fusion layer. this training strategy is related to model selection for domain generalisation (gulra-figure 1 : leave-one-out adapter fusion training strategy jani and lopez-paz, 2020), where the aim is to maximise the expected performance w.r.t an unknown meta-distribution over domains. unlike (gulrajani and lopez-paz, 2020), we use lodo to train the adapterfusion module instead of model selection. our contributions can be summarised as follows: (i) we extend adapterfusion for domain generalisation, where the target domain is not available during training; (ii) we propose a novel leave-oneout training strategy to avoid over-fitting of the fusion layer to the given training domains; (iii) we demonstrate the efficacy of our proposed fusion method on three language pairs and four unseen domains. empirical results show that our approach outperforms the learning invariant feature baseline on most of unseen test domains with an improvement up to +0.8 bleu score on average foot_0 . ## our approach problem formulation. we define domain generalisation for nmt as the problem of learning an nmt model on training datasets from multiple domains @xmath0, ..., d k } such that it performs well at some unseen test domain d k+1 . a dataset domain-specific parameter learning. we insert a small adapter module in transformer layers of the algorithm 1 lodo training of adapterfusion function: trainfusion input: training data @xmath1, ..., d k }, nmt model θ, adapters {ω1, ..., ωk } output: fusion layer ψ 1: while not converge do 2: shuffle d 3: {b k j j } j @xmath2, y y y) ∈ d k j |d k @xmath3: for @xmath4: dj ← {1, ..., k} \ {kj} // active domains 6: ## experiments we evaluate our proposed approach to generalise a pretrained nmt model to unseen domains on three language pairs english-german (en-de), english-french (en-fr), and english-polish (en-pl). ## conclusion in this paper, we propose a fusion-based approach to the domain generalisation problem for nmt. our method first captures domain-specific features via adapters, then learns to combine them with leaveone-out strategy training. experiments show the effectiveness of our methods without retraining the nmt backbone. hence, it is a potential method to quickly incorporate newly arriving domains into the existing nmt systems.