Multi-Domain Adaptation in Neural Machine Translation Through Multidimensional Tagging (2021)

Abstract: Production NMT systems typically need to serve niche domains that are not covered by adequately large and readily available parallel corpora. As a result, practitioners often fine-tune general purpose models to each of the domains their organisation caters to. The number of domains however can often become large, which in combination with the number of languages that need serving can lead to an unscalable fleet of models to be developed and maintained. We propose Multi Dimensional Tagging, a method for fine-tuning a single NMT model on several domains simultaneously, thus drastically reducing development and maintenance costs. We run experiments where a single MDT model compares favourably to a set of SOTA specialist models, even when evaluated on the domain those baselines have been fine-tuned on. Besides BLEU, we report human evaluation results. MDT models are now live at Booking.com, powering an MT engine that serves millions of translations a day in over 40 different languages.

## introduction neural machine translation (nmt) has achieved remarkable results in recent years. a strong testament to its success and efficacy is the increasingly widespread industrial adoption of nmt solutions @xcite ; @xcite ; @xcite . model parameter estimation in nmt architectures @xcite @xcite is still largely a supervised learning problem which requires large amounts of translated sentence pairs (parallel data). obviously, acquiring a sufficient number of high quality parallel sentences in order to train a functional domain-specific nmt system can be prohibitively expensive; especially, if one needs to develop such systems for several domains across different language pairs. on the other hand, large quantities of untranslated in-domain content (monolingual data) are often readily available. various domain adaptation strategies have been developed to address the low-resource setting of niche domains @xcite . some of the more popular approaches involve figure 1 : schematic diagram of mdt in our setting. we use generic parallel data to train a base source-target and a reverse target-source models. we then back-translate target language monolingual in-domain data using the reverse model, and mix it with upsampled indomain parallel data to fine-tune the base model. the data is tagged with two special tokens: <@xmath0,1}>, @xmath1,messaging,descriptions}>. in real-world scenarios practitioners often need to deploy translation engines for several closely related, yet different sub-domains. for example, an online travel marketplace needs to translate offering descriptions, user-generated reviews and customer service communications, all related to travel, but all having different linguistic nuances. this fragmentation is further compounded by the company's need to provide services across many distinct languages. it can be very expensive or outright impossible to develop and maintain separate translation pipelines for every combination of language and sub-domain. we propose a new method for training models which are simultaneously fine-tuned on several closely related, yet succinctly different sub-domains. we show that those models achieve competitive (and often superior) results to single domain fine-tuned baselines while effectively serving n use cases, therefore cutting development and maintenance costs by a factor of n . ## related work our work builds on a growing body of domain adaptation research, mainly related to fine-tuning through tagged back-translation. ## multidimensional tagging as discussed in section 2, introducing special tokens in the training data has been independently useful at passing content-specific information (e.g. domain, voice, gender, etc.) and data-specific information (e.g. whether a given data point is synthetic). the current work extends this idea into the multidimensional setting. whenever several meaningful dimensions describing the data are available at inference and training time, we can encode that information with special tokens indicating the values along each of the dimensions (figure 1 ). a real-world multi-domain adaptation setting lends itself very naturally to the mdt approach. for example, domain or topic is one such dimension, whether or not the data is synthetic is another. the definition of a synthetic sample may also differ between applications. back-translation as used in this work is an obvious way of generating such samples, but so can be pseudo-alignment @xcite . a hybrid dataset may include samples from all three origins (genuine, machine translated and pseudo-aligned) and a tag can help the model differentiate between them. lastly, multilingual models where the source languages are not trivially different, can be boosted with a language tag foot_1 . it is therefore clear that although our experiments only cover a two-dimensional setting with the attributes mentioned above (data domain and source), multidimensional tagging can be extended to cover other data aspects. ## experimental setup this section describes our data sources, model architecture, and synthetic data generation and mixing strategies that we employ in our experiments. our principal goal is to evaluate mdt fine-tuning approach as a scalable alternative to state-of-the-art domain fine-tuning for nmt. ## conclusions in this work we introduce multidimensional tagging and demonstrate that it can be a scalable solution for multi-domain adaptation in a realistic resource-constrained setting. somewhat surprisingly we find that mdt models in fact outperform on average our best alternative fine-tuning technique (top10 from @xcite ), even though the alternative method trains a custom model for each sub-topic. although the present work offers limited empirical evaluations of mdt (two dimensions: 3 sub-domains and 2 data sources; three language pairs), we think that the technique can prove useful in a broader setting. we believe it to be particularly well suited to many real-world scenarios in which practitioners develop solutions for multiple related domains, while leveraging data from different sources, both genuine and synthetic. all experimental results reported in this work follow rigorous human evaluations in addition to the standard bleu scores assessments. all aspects of the review are comprehensible.