Multi-Domain Neural Machine Translation (2018)

Abstract: We present an approach to neural machine translation (NMT) that supports multiple domains in a single model and allows switching between the domains when translating. The core idea is to treat text domainsasdistinctlanguagesandusemultilingual NMT methods to create multi-domain translation systems; we show that this approach results in significant translation quality gains over fine-tuning. We also explore whether the knowledge of pre-specified text domains is necessary; turns out that it is after all, but also that when it is not known quite high translation quality can be reached, and even higher than with known domains in some cases.

## introduction data-driven machine translation (mt) systems depends on the text domain of their training data. in a typical in-domain mt scenario the amount of parallel texts from a single domain is not enough to train a good translation system, even more so for neural machine translation (nmt; @xcite ; thus models are commonly trained on a mixture of parallel texts from different domains and then later fine-tuned to in-domain texts @xcite . in-domain fine-tuning has two main shortcomings: it depends on the availability of sufficient amounts of in-domain data in order to avoid overfitting and it results in degraded performance for all other domains. the latter means that for transc 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. lating multiple domains one has to run an individual nmt system for each domain. in this work we treat text domains as distinct languages: for example, instead of english-to-estonian translation we see it as translating english news to estonian news. we test two multilingual nmt approaches @xcite in a bilingual multidomain setting and show that both outperform single-domain fine-tuning on all the text domains in our experiments. however, this only works when the text domain is known both when training and translating. in some cases the text domain of the input segment is unknown -for example, web mt systems have to cope with a variety of text domains. also, some parallel texts do not have a single domain while they are either a mix of texts from different sources (like crawled corpora) or naturally constitute a highly heterogeneous mix of texts (like subtitles or wikipedia articles). we address these issues by replacing known domains with automatically derived ones. at training time we cluster parallel sentences and then applying the multi-domain approach to these clusters. when translating, the input segments are classified as belonging to one of these clusters and translated with this automatically derived information. in the following we review related work in section 2, then present our methodology of multidomain nmt and sentence clustering in section 3. after that, we describe our experiments in sections 4 and 5 and discuss the results in section 6. section 7 concludes the paper. ## related work the baseline to which we compare our work is fine-tuning nmt systems to a single text domain @xcite . there, the nmt system is first trained on a mix of parallel texts from different domains and then fine-tuned via continued training on just the in-domain texts. the method shows improved performance on indomain test data but degrades performance on other domains. in @xcite the nmt system is parametrized with one additional input feature (politeness), which is included as part of the input sequence, similarly to one of our two approaches (in our work -the domain tag approach). however, their goal is different from ours. in @xcite additional word features are used for specifying the text domain together with the same approach as @xcite . although both methods overlap with the first part or our work (domain features and domain tags), they only test these methods on pre-specified domains, while we include automatic domain clustering and identification. also, they use in-domain trained nmt systems as baselines even for small parallel corpora and do experiments with a different nmt architecture. finally, their results show very modest improvements, while in our case the improvements are much greater. other approaches also define a mixture of domains, for example @xcite . however, both define custom nmt methods and also limit the experiments to the cases where the text domain is known. ## methodology in the following we describe two different approaches to treating text domains as distinct languages and using multi-lingual methods, resulting in multi-domain nmt models. the first approach is inspired by google's multilingual nmt @xcite and the second one by the crosslingual language models @xcite . then we describe our methods of unsupervised domain segmentation used in our experiments in comparison with the pre-specified text domains. ## experiments with known domains in the experiments we use mixed-domain parallel data consisting of europarl @xcite , opensubtitles2016 @xcite , parallel data extracted from english-estonian wikipedia articles and some more mixed parallel corpora from the opus collection @xcite . the size of the corpora is shown in ## experiments with automatic domains since the results on the full parallel data show that both of multi-domain approaches are on-par, or superior to the single-domain baseline, we apply the methods in a setting where we do not assume beforehand knowledge of the origin domain of source sentences. here we take the domain tagging approach: even though domain features show better results, domain tags are more generic and compatible with any nmt architecture. we experiment with two data settings. in the first one, we have a single heterogeneous text domain. we explore both supervised and unsupervised tagging of single text domain based on sentence vectors. in the second one, we have texts from several domains but we ignore the pre-specified text domains and replace them with automatic clustering based on sentence embeddings. ## discussion the results from the experiments -en-et and et-en direction parallel translation, wikipedia data translation, and unsupervised sentence tagging -show that both of the two chosen multidomain approaches outperform regular approach of uniform translation and domain-tuning. this indicates the hypothesis that the parameter sharing effect discussed in google's zero-shot article would benefit domain translation holds. the translation scores even outperform domaintuning approach, which could be explained by the same parameter sharing. in tuning we tune the model to translate sentences characteristic to the model we are tuning to. this means that domaincharacteristic sentences get translated really well. on the other hand, the not-so-characteristic sentences get neglected. the parameter sharing effect of the multi-domain approach helps negate the negative effect by the support of other domains while still learning to more effectively represent each domain by the additional domain info. furthermore, the results indicate that adding domains as an input feature can have even stronger effect on the translation scores. this shows that concatenating the domain feature embedding with word embedding at each timestep -basically remembering the source domain equally throughout the sequence improves model performance. this could be explained by the fact that in tag prepending case, the neural net may "forget" for longer sequences what the input tag was, making the effect of it weaker. the results also show that for highly quality dependent settings the domain feature concatenation with word embedding is the more suitable option. however, the differences in scores are not drastically different from the domain tag prepending. this means that for the sake of data simplicity, model simplicity and efficiency the tag prepending approach could prove more reasonable of the two for in-production settings. finally, the performance of unsupervised domain tagged model indicates that there is grounds to substitute the pre-defined domain approach with automatically assigned domain approach. the unsupervised certainly serves as an improvement in less homogenous single domain settings, where the effect of the detection of underlying "domains" was shown on the example of wikipedia. no less important are the facts that the unsupervised tagging approach ensures better domain assignment to each new sentence and can efficiently incorporate new data from various small domains to fortify each of the learned "domain" (clusters). tuned is achieved by tuning these models with the specific corpus. tag is trained with data that has domain tag prepended to each source sentence. unsup is trained with data that has domain tags assigned to each sentence in an previously described unsupervised manner. p-values are given for significance against tag and unsup respectively, separated with /. ## conclusions in this article we tested two approaches to improve multi-domain neural translation. one approach involves prepending domain tags to source sentences, the other adding domain embeddings as an input feature to each source sentence word. we showed that both ways of adding domain information to source sentences in bilingual neural translation improves translation scores considerably compared to both regular baseline translation and finetuning. these improvements in source sentence tagging case can be obtained with mere data manipulation. we also showed that the domain tagging approach can be successfully coupled with unsupervised sentence clustering to add a "domain dimension" to a previously single-domain corpus. this approach produces better results as opposed to using the corpus as a single domain. the results indi-cate that unsupervised or semi-supervised training data clustering can be effectively used to improve neural machine translation. finally, to bring the two experiments together, we apply unsupervised domain tagging to full parallel data and show that it can serve as a viable alternative to the pre-defined domain approach. for future work the clustering in fully unsupervised tagging approach should be improved to see if this gives a visible improvement in translation scores. secondly, a more comprehensive sweep on number of clusters should be done. it would be interesting to see for how many clusters the effect still persists. this however would need more extensive computational resources and should probably be done with some model dataset. the differences of the two approaches -source sentence tagging and adding domain info as an input feature -deserve to be looked into more deeply. more precisely, the result profiles of the two in different cases of domain granularity. finally, in this work domains are still treated as nominal values; it would be interesting to explore the estimation of domain embeddings at transla-