A Study of Residual Adapters for Multi-Domain Neural Machine Translation (2020)

Abstract: Domain adaptation is an old and vexing problem for machine translation systems. The most common approach and successful to supervised adaptation is to fine-tune a baseline system with in-domain parallel data. Standard fine-tuning however modifies all the network parameters, which makes this approach computationally costly and prone to overfitting. A recent, lightweight approach, instead augments a baseline model with supplementary (small) adapter layers, keeping the rest of the mode unchanged. This has the additional merit to leave the baseline model intact, and adaptable to multiple domains. In this paper, we conduct a thorough analysis of the adapter model in the context of a multidomain machine translation task. We contrast multiple implementations of this idea on two language pairs. Our main conclusions are that residual adapters provide a fast and cheap method for supervised multi-domain adaptation; our two variants prove as effective as the original adapter model, and open perspective to also make adapted models more robust to label domain errors.

## introduction owing to multiple improvements, neural machine translation (nmt) @xcite @xcite nowadays delivers useful outputs for many language pairs. however, as many deep learning models, nmt systems need to be trained with sufficiently large amounts of data to reach their best performance. therefore, the quality of the translation of nmt models is still limited in low-resource language or domain conditions @xcite @xcite . while many approaches have been proposed to improve the quality of nmt models in low-resource domains (see the recent survey of @xcite ), full fine-tuning @xcite of a generic baseline model remains the dominant supervised approach when adapting nmt models to specific domains. under this view, building adapted systems is a two-step process: (a) one first trains nmt with the largest possible parallel corpora, aggregating texts from multiple, heterogeneous sources; (b) assuming that in-domain parallel documents are available for the domain of interest, one then adapts the pre-trained model by resuming training with the sole in-domain corpus. it is a conjecture that the pretrained model constitutes a better initialization than a random one, especially when adaptation data is scarce. indeed, studies of transfer learning for nmt such as @xcite ; @xcite have confirmed this claim in extensive experiments. full fine-tuning, that adapts all the parameters of a baseline model usually significantly improves the quality of the nmt for the chosen domain. however, it also yields large losses in translation quality for other domains, a phenomenon referred to as "catastrophic forgetting" in the neural network literature @xcite . therefore, a fully fine-tuned model is only useful to one target domain. as the number of domains to handle grows, training, and maintaining a separate model for each task can quickly become tedious and resource-expensive. several recent studies (e.g. @xcite @xcite ) have proposed more lightweight schemes to perform domain adaptation, while also preserving the value of pre-trained models. our main inspiration is the latter work, whose proposal relies on small adapter components that are plugged in each hidden layer. these adapters are trained only with the in-domain data, keeping the pre-trained model frozen. because these additional adapters are very small compared to the size of the baseline model, their use significantly reduces the cost of training and maintaining fine-tuned models, while delivering a performance that remains close to that of full fine-tuning. in this paper, we would like to extend this architecture to improve nmt in several settings that still challenge automatic translation, such as translating texts from multiple topics, genre, or domains, in the face of unbalanced data distributions. furthermore, as the notion of "domains" is not always well established, another practical setting is the translation of texts mixing several topics/domains. an additional requirement is to translate texts from domains unseen in training, based only on the unadapted system, which should then be made as strong as possible. in this context, our main contribution is a thorough experimental study of the use of residual adapters for multi-domain translation. we notably explore ways to adjust and/or regularize adapter modules to handle situations where the adaptation data is very small. we also propose and contrast two new variants of the residual architecture: in the first one (highway residual adapters), adaptation still affects each layer of the architecture, but its effect is delayed till the last layer, thus making the architecture more modular and adaptive; our second variant (gated residual adapters) exploits this modularity and enables us to explore ways to improve performance in the face of train-test data mismatch. we experiment with two language pairs and report results that illustrate the flexibility and effectiveness of these architectures. ## residual adapters in this section, we describe the basic version of the residual adapter architectures @xcite , as well as two novel variants of this model. ## related work training with data from multiple, heterogeneous sources is a common scenario in natural language processing @xcite . it is thus no wonder that the design of multi-domain systems has been proposed for many tasks. in this short survey, we exclusively focus on machine translation; it is likely that similar methods (parameter sharing, instance selection/weighting, adversarial training, etc) have also been proposed for other tasks. early approaches to multi-domain mt were proposed for statistical mt, either considering multiple data sources (eg. @xcite ; clark et al. (2012); sennrich et al. (2013); huck et al. (2015) ) or domains containing several topics @xcite . two main strategies emerge: feature-based methods, where domain labels are integrated through supplementary features; and instance-based methods, involving a measure of similarity between train and test domains. the former approach has also been adapted to nmt: @xcite ; @xcite use an additional domain feature in an rnn model, in the form of an extra domain-token or of additional domain-features associated with each word. @xcite apply domain control on the target side, using a topic vector to describe the ## conclusion and outlook in this paper, we have performed an experimental study of the residual adapter architecture in the context of multi-domain adaptation, where the goal is to build one single system that (a) performs well for domain seen in training, ideally as well as full fine-tuning; (b) is also able to robustly handle translations for new, unseen domains. we have shown that this architecture allowed us to quickly adapt a model to a specific domain, delivering bleu performance than are much better than the generic, mixed domain baseline, and close the gap with the full-finetuning approach, at a modest computa-tional cost. several new variants have been introduced and evaluated for two language directions: if none that able to clearly surpass the baseline, residual adapter models, they provide directions for improving this model in practical settings: unbalanced data condition, noise in label domains, etc. in our future work, we would like to continue the development of the gated variant, which, it seems to us, provides a flexible and robust tool to address the various challenges of multi-domain machine translation.