Improving the Quality Trade-Off for Neural Machine Translation Multi-Domain Adaptation (2021)

Abstract: Building neural machine translation systems to perform well on a specific target domain is a well-studied problem. Optimizing system performance for multiple, diverse target domains however remains a challenge. We study this problem in an adaptation setting where the goal is to preserve the existing system quality while incorporating data for domains that were not the focus of the original translation system. We find that we can improve over the performance trade-off offered by Elastic Weight Consolidation with a relatively simple data mixing strategy. At comparable performance on the new domains, catastrophic forgetting is mitigated significantly on strong WMT baselines. Combining both approaches improves the Pareto frontier on this task.

## introduction the quality of neural machine translation (nmt) has improved considerably in recent years, mostly due to improvements in model architecture @xcite @xcite . training nmt models typically involves collecting parallel training data from multiple sources to achieve high translation quality and generalize well to unseen data @xcite . however, translation quality depends strongly on the relevance of the training data to the input text, which is why performance varies across target domains @xcite . a popular method for domain adaptation of nmt models is fine-tuning generic models on indomain data to yield a domain-specific model @xcite . when high quality output on more than one target domain is required, multi-domain adaptation methods aim to produce a single system that performs well on multiple domains @xcite @xcite . ## related work most previous work on multi-domain adaptation focuses on a scenario with fixed training data. for example, @xcite use knowledge distillation to build a single model from expert models optimized for the training domains, @xcite train models that better distinguish between the training domains and @xcite learn domain-specific word embeddings for the domains present in the training data. in contrast, we focus on the adaptation setting where additional domain data becomes available over time. @xcite apply ewc for adaptation to a single new domain while @xcite use it for sequentially adapting to two new domains. both report positive results but at the same time show a performance trade-off which our work tries to address further. mixing out-of-domain and in-domain data for fine-tuning was proposed by @xcite who use tags to distinguish domains at test time while our models are domain-agnostic. data mixing is also related to work on episodic memories for continual learning. for example, @xcite show that a random sample of previous task data can outperform ewc for image recognition. ## multi-domain adaptation our goal is to optimize translation quality for several new domains represented by small amounts of parallel data while maintaining the performance of a high-quality, general-purpose nmt model. we focus on the scenario where only small amounts of additional data are available since it is suitable for an adaptation setup. for large amounts of additional data, retraining the model from scratch might be a more suitable approach. @xcite study the problem of catastrophic forgetting in sequential machine learning settings. they propose ewc as a method to preserve model performance during sequential learning of task b by selectively slowing down learning on the weights that are important for the original task a learned by the model. this goal is achieved by adding a loss term to the training objective as shown in equation 1 : ## experiments we evaluate multi-domain adaptation on top of two strong wmt baselines: @xmath0. ## conclusion we investigated techniques to mitigate catastrophic forgetting during nmt model adaptation in order to optimize for new domains while maintaining the quality of already deployed systems. we found that data mixing provides a favourable quality trade-off and improves the pareto frontier when combined with ewc. we showed that data mixing is robust to random sampling and sample size and that our reported gains persist for different learning rates.