Factorized Transformer for Multi-Domain Neural Machine Translation (2020)

Abstract: Multi-Domain Neural Machine Translation (NMT) aims at building a single system that performs well on a range of target domains. However, along with the extreme diversity of cross-domain wording and phrasing style, the imperfections of training data distribution and the inherent defects of the current sequential learning process all contribute to making the task of multi-domain NMT very challenging. To mitigate these problems, we propose the Factorized Transformer, which consists of an in-depth factorization of the parameters of an NMT model, namely Transformer in this paper, into two categories: domain-shared ones that encode common cross-domain knowledge and domain-specific ones that are private for each constituent domain. We experiment with various designs of our model and conduct extensive validations on English to French open multi-domain dataset. Our approach achieves state-of-the-art performance and opens up new perspectives for multi-domain and open-domain applications.

## introduction recent advances in neural machine translation (nmt) @xcite have led to significant improvement in terms of translation quality @xcite , opening new perspectives for machine translation in real-world scenarios. in order to deliver trust-worthy translations for end users, an nmt system is often required to meet expert-level translation quality in one or multiple related target domains, while performing well enough on a range of generic subjects, just like human experts do. however, requiring a single nmt system to perform well on multiple distant domains simultaneously is a very challenging task. first, languages are highly polysemous: the same words or expressions may have different meanings in different contexts. also wording and syntactic style may significantly vary depending on the domains. second, a multi-domain nmt system in general suffers from two major issues: domain bias and catastrophic forgetting @xcite @xcite . while the former biases the model toward well-represented domains to the detriment of the low-resource ones, the latter makes the sequential learning process difficult as the model keeps forgetting previously learned knowledge when exposed to the new training examples. most of the existing nmt systems rely on the same network to model all domains, which means the same word embedding to represent all the meanings of a word and the same set of parameters to model its depending contexts. this type of configuration in general maximizes the knowledge transfer, but overlooks the specificity of each domain @xcite ). an obvious solution for this problem is to dedicate an individual model to each constituent domain, which is unrealistic in practice as it dramatically increases the number of model parameters. moreover, the recent success of multilingual applications @xcite show that a single nmt model where all parameters are shared can handle translation between hundred of language pairs, suggesting that model capacity may not be the key weakness of the current nmt models to deal with multi-domain problems. thus, the need for a compact architecture with better parameter efficiency is appealing. we propose the factorized transformer framework to deal with the multi-domain nmt problem. the factorized transformer consists in factorizing partially or fully basic components (embedding, attention and ffn layers) of a conventional transformer architecture into domain-specific blocks and domain-shared blocks. this dual structure has several advantages: 1) it allows the model to leverage all available data, labeled or unlabeled, to build a generic model at an early stage of domain-agnostic training; 2) domain singularities could be effectively learned by using domain-specific components and the respective in-domain training data during the stage of domain-aware training. the domain bias issue naturally disappears; 3) domainspecific components are independently optimized, without any interference between target domains. the original performance of the generic model on un-adapted source domains is also preserved, overcoming the limit of catastrophic forgetting. 4) the design of factorized transformer is orthogonal to any data-driven approach, so that the benefit of both approaches can be combined. our contributions can be summarized as follows: ## related work multi-domain nmt has been an active research area. prior work in this area can be divided into two main categories: data-driven and model-driven, although they are usually complementary. ## approach all basic components (embedding, attention and ffn matrices) of a conventional transformer are factorized into multiple domain-specific blocks (figure 1 ), one for each domain (colored ones) and a domain-shared block (white ones), common across all domains. it's worth to notice that domain information is necessary for both training and inference, which could be obtained via external sources. nevertheless, the domain prediction is not the main purpose of this work and we suppose in the whole paper, except otherwise mentioned, that domain information is known and passed as input to the model during training and inference. ## towards open-domain nmt in many real-world scenarios, the domain information is unknown at inference time, and even worse, the test inputs may also be out-of-domain, which means the model has never seen data from the same domains during training. for such unknown domains, nmt systems are known to have poor performance, especially adapted ones @xcite . model ensembling is a reasonable approach to deal with unknown domains @xcite . the compact and unified architecture of factorized transformer makes it ideal for this purpose as at each step all domain-specific representations can be computed in parallel and feed-forward to obtain multiple domain-specific word prediction probabilities (logits). we consider in this section the unseen it domain as a new unknown test domain. the test set is drawn from the gnome corpus from the opus website. under the open domain paradigm, we do not use any development or training data. we experiment with 3 simple variants of model ensembling based on the parallel attention design of our approach (see the results ( ## conclusion in this paper, we propose the factorized transformer framework to overcome the limits of traditional multi-domain nmt approaches in modeling all domain knowledge within a single shared set of parameters. by factorizing wisely the parameters of the transformer model into domain-shared and domain-specific parts, we significantly improve the model's parameter efficiency and provide new perspectives for open domain applications.