Domain Specific Sub-network for Multi-Domain Neural Machine Translation (2022)

Abstract: This paper presents Domain-Specific Sub-network (DoSS). It uses a set of masks obtained through pruning to define a sub-network for each domain and finetunes the sub-network parameters on domain data. This performs very closely and drastically reduces the number of parameters compared to finetuning the whole network on each domain. Also a method to make masks unique per domain is proposed and shown to greatly improve the generalization to unseen domains. In our experiments on German to English machine translation the proposed method outperforms the strong baseline of continue training on multi-domain (medical, tech and religion) data by 1.47 BLEU points. Also continue training DoSS on new domain (legal) outperforms the multi-domain (medical, tech, religion, legal) baseline by 1.52 BLEU points.

## introduction neural machine translation (nmt) has witnessed significant advances based on transformer models @xcite . these models are typically trained on large amounts of data from different sources, i.e. general data, from a single language pair or multiple languages @xcite . the fact that the models are trained on general data usually leads to poor, or less than average, performance on specific domains. this has a lot of practical implication since many users of machine translation are interested in the performance on some specific domain(s). therefore, improving the performance of nmt on specific domains has become an active area of research. we refer the reader to @xcite for a review. broadly speaking, the proposed techniques could be divided into data-centric and model-centric approaches. the goal of the former methods is to acquire, often automatically, monolingual and bilingual data that is representative of the domain of interest. the latter techniques, on the other hand, focus on modifying the model to perform well on the domain of interest without sacrificing the performance on general data. finetuning of the model parameters using domain data is perhaps one of the earliest and most popular techniques for domain adaptation @xcite . parallel domain data is usually limited and to avoid overfitting different techniques as model interpolation @xcite , regularization @xcite and mixing domain and general data @xcite are used. also other methods that introduce additional parameters in a controllable way have been successfully introduced such as adapters @xcite and low-rank adaptation (lora) @xcite . in @xcite it is shown that identifying sub-networks by pruning a large network, referred to as winning tickets, and retraining them leads to equal accuracy to the original network. this idea is explored for multilingual neural machine translation (mnmt) using the so-called language specific sub-networks (lass) @xcite . here we further explore the idea for domain finetuning and refer to it as domain specific @xcite . the basic idea is to identify a sub-network per domain via pruning and masking. the sub-network has both shared parameters with other domains as well as domain-specific parameters. it should be noted that the mask can overlap for multiple domains which results in some parameters shared by multiple domains. we also explore using constrained masks where we ensure that each mask represents only one domain. the latter is expected to work better for adding unseen domains. in contrast to language, domain information may not be necessarily known at inference time. in this work, similar to common domain fientuning setups, we assume the domain information is known but using a domain classifier at runtime should be straight forward. given the domain information, inference can be carried with the trained model and the domain mask. the paper is organized as follows. section 2 gives a detailed description of the proposed method followed by the experimental results in section 3. finally, the conclusion is given in section 4. ## method we present the doss method in this section as shown in figure 1 . we focus on the bilingual setting and defer the multilingual case to future work. assume we have an initial model λ 0 that is trained on large amounts of general data. we also have the data sets {d i } n @xmath0, y j ).typically, the initial model is finetuned for each domain resulting in n domain models. here, we first create a mask for each domain using pruning then train a domain sub-network using the resulting masks. we will explain the two steps below. ## experiments and results we evaluate the performance of doss on german to english translation, and we consider three domains: medicine, religion, and technology. the baseline model was a german to english model trained on 32.13m parallel sentences that were provided by the wmt19 news translation shared task foot_0 . all domain and baseline data are filtered to remove sentences longer than 250 tokens, as well as sentences with a source to target length ratio smaller than 0.67 or exceeding 1.5. fasttext @xcite language identification was also applied to both sides of the bitext to reduce the garbage @xcite . ## conclusion in this paper, we propose a new efficient method for multi-domain adaptation by learning domainspecific sub-network (doss). doss can efficiently generalize to new domains while preserving the performance of existing domains. for our experiments on de-en machine translation doss outperforms the strong baseline of continue training on multi-domain (medical, tech, religion) data by 1.47 bleu points. also for the interesting scenario of extension to new domains it outperforms continue training on multi-domain data (medical, tech, religion, legal) by 1.52 bleu points. in future work we plan to explore adding more domains, using domain classifiers during inference, experimenting with multi-lingual and multidomain setup and looking into new ways of defining constrained masks. we could also explore applying the method on sparse architectures.