Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixing (2020)

Abstract: Many multi-domain neural machine translation (NMT) models achieve knowledge transfer by enforcing one encoder to learn shared embedding across domains. However, this design lacks adaptation to individual domains. To overcome this limitation, we propose a novel multi-domain NMT model using individual modules for each domain, on which we apply word-level, adaptive and layer-wise domain mixing. We first observe that words in a sentence are often related to multiple domains. Hence, we assume each word has a domain proportion, which indicates its domain preference. Then word representations are obtained by mixing their embedding in individual domains based on their domain proportions. We show this can be achieved by carefully designing multi-head dot-product attention modules for different domains, and eventually taking weighted averages of their parameters by word-level layer-wise domain proportions. Through this, we can achieve effective domain knowledge sharing and capture fine-grained domain-specific knowledge as well. Our experiments show that our proposed model outperforms existing ones in several NMT tasks.

## introduction neural machine translation (nmt) has made significant progress in various machine translation tasks @xcite @xcite @xcite . the success of nmt heavily relies on a huge amount of annotated parallel sentences as training data, which is often limited in certain domains, e.g., medical domain. one approach to address this is to explore unparalleled corpora, such as unsupervised machine translation @xcite . another approach is to train a multi-domain nmt model and this is the focus of this paper. the simplest way is to build a unified model by directly pooling all training data from multiple domains together, as the languages from different domains often share some similar semantic traits, e.g., sentence structure, textual style and word usages. for domains with less training data, the unified model usually shows significant improvement. researchers have proposed many methods for improving multi-domain nmt. though certain semantic traits are shared across domains, there still exists significant heterogeneity among languages from different domains. for example, @xcite show that for a domain with sufficient training data, a unified model may lead to weaker performance than the one trained solely over the domain; @xcite ; @xcite ; @xcite ; @xcite also show that to improve the translation performance over certain domains, fine-tuning the unified model is often needed, but at the expense of sacrificing the performance over other domains. this indicates that a unified model might not well exploit the domain-specific knowledge for each individual domain. to overcome this drawback, two lines of recent research focus on developing new methods by exploiting domain-shared and domain-specific knowledge to improve multi-domain nmt @xcite @xcite @xcite @xcite @xcite @xcite . one line of research focuses on instance weighting, which assigns domain related weights to different samples during training. for example, @xcite consider sentence weighting and domain weighting for nmt. the sentence weight is determined by the bilingual cross-entropy of each sentence pair based on the language model of each domain. the domain weight can be modified by changing the number of sentences from that domain in a mini-batch. @xcite propose a cost weighting method, where the weight of each pair of sentences is evaluated by the output probability of a domain classifier on the encoder embedding. @xcite propose a dynamic training method to adjust the sentence selection and weighting during training. we remark that many of these methods are complementary to our proposed model, and can be applied to improve the training of our model. another line of research attempts to design specific encoder-decoder architectures for nmt models. for example, @xcite consider domain-aware embedding given by the encoder, and then jointly train a domain classifier, taking the embedding as input to incorporate the domain information. @xcite ; @xcite further extend their approach by separating the domainshared and domain-specific knowledge within the embedding. in addition, @xcite and @xcite propose a maximum weighted likelihood estimation method, where the weight is obtained by word-level domain aware masking to encourage the model to pay more attention to the domain-specific words. the aforementioned methods, however, have a notable limitation: they enforce one single encoder to learn shared embedding across all domains, which often lacks adaptivity to each individual domain. to better capture domain-shared knowledge beyond shared embedding from a single encoder, we propose a novel multi-domain nmt model using individual modules for each domain, on which we apply word-level, adaptive and layer-wise domain mixing. our proposed model is motivated by the observation that although every sentence of the training data has a domain label, the words in the sentence are not necessarily only related to that domain. for instance, the word "article" appears in the domains of laws and business. therefore, we expect the knowledge for translating the word "article" to be shared between these two domains. our proposed model assigns a context-dependent domain proportion foot_0 to every word in the sentence. the domain proportions of the words can be naturally integrated into the transformer model for capturing domain-shared/specific knowledge, as the multi-head dot-product attention mechanism is applied at the word-level. specifically, we carefully design multi-head dot-product attention modules for different domains, and eventually mix these modules by taking weighted averages of their parameters by their layer-wise domain proportions. compared with existing models, ours has the following two advantages: we evaluate our proposed model in several multidomain machine translation tasks, and the empirical results show that our proposed model outperforms existing ones and improves the translation performance for all domains. the rest of the paper is organized as follows: section 2 introduces the background; section 3 describes our proposed model in detail; section 4 presents numerical experiments on en-de, en-fr and zh-en datasets; section 5 discusses the connection to word disambiguation. transformer is one of the most popular nmt models @xcite @xcite . the encoder and decoder in transformer contain stacked self-attention and point-wise, fully connected layers without any explicit recurrent structure, which is different from existing rnn-based nmt models. specifically, @xcite propose a new attention function using the scaled dot-product as the alignment score, which takes the form, eventually, the final output applies a point-wise linear transformation w @xmath0: an illustrative example of the multihead attention architecture is provided in figure 1 . in addition to the above multi-head attention modules, each layer in the encoder and decoder in transformer contains a point-wise two-layer fully connected feed-forward network. ## model we present our transformer-based multi-domain neural machine translation model with word-level layer-wise domain mixing. ## experiment we conduct experiments on three different machine translation tasks: the sizes of training, validation, and testing sets for different language pairs are summarized in ## discussions one major challenge in multi-domain machine translation is the word ambiguity in different domains. for example, the word "article" has different meanings in the domains of laws and media. when translating "article" into chinese, the translated words are "条款" and "文章" , meaning a separate clause of a legal document and a piece of writing. our proposed word-level layer-wise domain mixing approach tends to reduce the word ambiguity. as mentioned in section 3.3, our model extracts different representations of each word from contexts at different layers. accordingly, the domain proportion of each word evolves from bottom to top layers, and can eventually help identify the corresponding domains. moreover, as mentioned in section 3.2, the positional embedding also contributes to the word disambiguation in multi-domain translation. for example, in the law domain, we find that "article" often appears at the beginning of a sentence, while in the media domain, the word "article" may appear in other positions. therefore, varying domain proportions for different positions can help with word disambiguation. we remark that word disambiguation across domains actually requires d(x) to be powerful for predicting the domain of the word. however, a powerful d(x) tends to yield skewed domain proportions and is not flexible enough for domain knowledge sharing. to trade off between strength and flexibility of d(x), the smoothing parameter of d(x) (see section 3.1) needs to be properly set. ## conclusions we present a novel multi-domain nmt with wordlevel layer-wise domain mixing, which can adaptively exploit the domain knowledge. unlike the existing work, we construct multi-head dot-product modules for each domain and then combine them by the layer-wise domain proportion of every word. the proposed method outperforms the existing embedding based methods. we also show mixing method can be combined with embedding based methods to make further improvement. moreover, we remark that our approach can be extended to other multi-domain or multi-task nlp problems.