Exploring Domain-shared and Domain-specific Knowledge in Multi-Domain Neural Machine Translation (2023)

Abstract: Currently, multi-domain neural machine translation (NMT) has become a significant research topic in domain adaptation machine translation, which trains a single model by mixing data from multiple domains. Multi-domain NMT aims to improve the performance of the low-resources domain through data augmentation. However, mixed domain data brings more translation ambiguity. Previous work focused on domain-general or domain-context knowledge learning, respectively. Therefore, there is a challenge for acquiring domain-general or domain-context knowledge simultaneously. To this end, we propose a unified framework for learning simultaneously domain-general and domain-specific knowledge, we are the first to apply parameter differentiation in multi-domain NMT. Specifically, we design the differentiation criterion and differentiation granularity to obtain domain-specific parameters. Experimental results on multi-domain UM-corpus English-to-Chinese and OPUS German-to-English datasets show that the average BLEU scores of the proposed method exceed the strong baseline by 1.22 and 1.87, respectively. In addition, we investigate the case study to illustrate the effectiveness of the proposed method in acquiring domain knowledge.

## introduction in recent years, neural machine translation (nmt) has shown excellent performance in various translation tasks, as evidenced by state-of-the-art (sota) results reported in studies such as @xcite @xcite @xcite , among which multi-domain nmt aims to construct a single nmt model with the ability to translate sentences across different domains @xcite . mixed-domain data can improve cross-domain knowledge on low-resource domains by data augmentation. however, word ambiguity increases when we mix data from multiple domains. therefore, a challenge remains in how to learn the domain-shared and domain-specific knowledge for multi-domain nmt. to address the above problem, researchers design domain-shared @xcite @xcite promotion centers and technology enterprise incubation base reference 促进中心和科技企业孵化✓基地 mixed 促进中心和技术企业孵化✓基地 single 促进中心和技术企业潜伏×基地 mdnmt @xcite ) 促进中心和技术企业培养×基地 ## related work according to the domain representation learning strategy, we divide it into domain-shared and domain-specific knowledge methods: domain-shared knowledge learning: mixed domain data is a simple and convenient method to obtain domain-shared knowledge. additionally, @xcite designed the domain-shared discriminator to learn cross-domain features. @xcite proposed isolating domain-agnostic from domain-specific lexical representations while sharing most of the network across domains. furthermore, @xcite proposed two complementary supervision signals by leveraging the power of knowledge distillation and adversarial learning. domain-specific knowledge learning: from a sentence-level perspective, training a discriminator to detect and embed the domain tag for a sentence has become the mainstream approach @xcite @xcite @xcite . @xcite and @xcite propose a maximum weighted likelihood estimation method, where the weight is obtained by masking the domain-aware word level to encourage the model to pay more attention to the domain-specific representation of words. recent work proposes domain proportion to improve the adaptability of each word @xcite @xcite . some works propose the domain proportion of words for mdnmt, where each word in the sentence has a corresponding proportion in each domain @xcite @xcite . however, this approach may also affect the performance of the domain discriminator in the target language to some extent, potentially leading to translation ambiguity. compared with the previous approaches, there are two salient features in our methods: (1) our method can capture domain-shared and domain-specific knowledge simultaneously within the framework of multi-domain nmt instead of separately. (2) our method learns domain-shared and domain-specific knowledge from the perspective of parameter learning, rather than utilizing domain discriminators. ## our model multi-domain nmt task: the objective of this task is to create a unified model using mixeddomain data, aiming to maximize performance across all domains @xcite . specifically, there are j subsets, denoted as d 1 , d 2 , ..., d j . each subset d j consists of pairs of input-output sequences, represented as d @xmath0, y m j ]} mj @xmath1, where j indicates the domain and m denotes the index within the domain, m j represents the number of all sentences in the j-th domain. the training objective can be formulated as follows: ## experiments in our experiments, we aim to investigate the following research problems: (1) what is the improved performance of our method against previous work? (3) can our model learn the more effective domain-shared and domain-specific knowledge? ## 5 analysis and discussion in this section, we first examine the effectiveness of differentiation. then, we visualize the domain distribution and analyze the case study. it is worth noting that we mainly verify the english-to-chinese translation task. ## conclusion and future work in this paper, we explore domain-shared and domain-specific knowledge in multi-domain nmt. our method can simultaneously learn domain-shared and domain-specific parameters to resolve word ambiguity. experimental results on two translation tasks show that our method can bring significant improvements. further analyses confirm that our method can improve word ambiguity between domains. in future work, we will improve the gradient similarity method to further improve the accuracy of domain-specific parameters.