Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation (2021)

Abstract: Domain Adaptation is widely used in practical applications of neural machine translation, which aims to achieve good performance on both general domain and in-domain data. However, the existing methods for domain adaptation usually suffer from catastrophic forgetting, large domain divergence, and model explosion. To address these three problems, we propose a method of “divide and conquer” which is based on the importance of neurons or parameters for the translation model. In this method, we first prune the model and only keep the important neurons or parameters, making them responsible for both general-domain and in-domain translation. Then we further train the pruned model supervised by the original whole model with knowledge distillation. Last we expand the model to the original size and fine-tune the added parameters for the in-domain translation. We conducted experiments on different language pairs and domains and the results show that our method can achieve significant improvements compared with several strong baselines.

## introduction neural machine translation (nmt) models @xcite @xcite @xcite are datadriven and hence require large-scale training data to achieve good performance @xcite . in practical applications, nmt models usually need to produce translation for some specific domains with only a small quantity of in-domain data available, so domain adaptation is applied to address the problem. a typical domain adaptation scenario as discussed in @xcite is that an nmt model have been trained with large-scale general-domain data and then is adapted to specific domains, hoping the model can fit in-domain data well meanwhile the performance will not degrade too much on the general domain. towards this end, many researchers have made their attempts. the fine-tuning method @xcite performs in-domain training based on the general-domain model by first training the model on general-domain data and then continuing to train on in-domain data. despite its convenience for use and high-quality for in-domain translation, this method suffers from catastrophic forgetting which leads to poor performance in the previous domains. regularization-based methods @xcite @xcite instead introduce an additional loss to the original objective so that the translation model can trade off between general-domain and in-domain. this kind of methods usually has all the parameters shared by general-domain and in-domain, with the assumption that the optimal parameter spaces for all the domains will overlap with each other, and retaining these overlapped parameters can balance over all the domains. this assumption is feasible when the domains are similar, but when the divergence of the domains is large, it is not reasonable anymore. in contrast, the methods with domain-specific networks @xcite @xcite can be often (but not always) immune to domain divergence as it can capture domain-specific features. but unfortunately, as the number of domains increases, the parameters of this kind of methods will surge. besides, the structure of these networks needs to be carefully designed and tuned, which prevents them from being used in many cases. given the above, we propose a method of domain adaptation that can not only deal with large domain divergence during domain transferring but also keep a stable model size even with multiple domains. inspired by the analysis work on nmt @xcite @xcite , we find that only some important parameters in a well-trained nmt model play an important role when generating the translation and unimportant parameters can be erased without affecting the translation quality too much. according to these findings, we can preserve important parameters for generaldomain translation, while tuning unimportant parameters for in-domain translation. to achieve this, we first train a model on the general domain and then shrink the model with neuron pruning or weight pruning methods, only retaining the important neurons/parameters. to ensure the model can still perform well on general-domain data, we adjust the model on in-domain data with knowledge distillation where the original whole model is used as the teacher and the pruned model as the student. finally, we expand the model to the original size and fine-tune the added parameters on the in-domain data. experimental results on different languages and domains show that our method can avoid catastrophic forgetting on general-domain data and achieve significant improvements over strong baselines on multiple in-domain data sets. our contributions can be summarized as follows: ## method the main idea of our method is that different neurons or parameters have different importance to the translation model and hence different roles in domain adaptation. based on this, we distinguish them into important and unimportant ones and make important neurons or parameters compromise between domains while unimportant ones focus on in-domain. specifically, our method involves the following steps shown in figure 1 . first, we train a model on the general domain and then evaluate the importance of different neurons or parameters. then we erase the unimportant neurons or parameters and only keep the ones that are related to the general domain so that our method will not be subjected to domain divergence. next, we further adjust our model under the framework of knowledge distillation @xcite on the in-domain with the unpruned model as the teacher and the pruned model as the student. in this way, the pruned model can regain some of its lost performance because of pruning. finally, we expand the pruned model to the original size and fine-tune the added parameters for the in-domain. ## conclusion in this work, we propose a domain adaptation method based on the importance of neurons and parameters of the nmt model. we make the important ones compromise between domains while unimportant ones focus on in-domain. based on this, our method consists of several steps, namely, model pruning, knowledge distillation, model expansion, and fine-tuning. the experimental results on different languages and domains prove that our method can achieve significant improvements with model capacity fixed. further experiments prove that our method can also improve the overall performance under the multi-domain scenario.