Revisiting Multi-Domain Machine Translation (2021)

Abstract: When building machine translation systems, one often needs to make the best out of heterogeneous sets of parallel data in training, and to robustly handle inputs from unexpected domains in testing. This multi-domain scenario has attracted a lot of recent work that fall under the general umbrella of transfer learning. In this study, we revisit multi-domain machine translation, with the aim to formulate the motivations for developing such systems and the associated expectations with respect to performance. Our experiments with a large sample of multi-domain systems show that most of these expectations are hardly met and suggest that further work is needed to better analyze the current behaviour of multi-domain systems and to make them fully hold their promises.

## introduction data-based machine translation (mt), whether statistical or neural, rests on well-understood machine learning principles. given a training sample of matched source-target sentence pairs (f , e) drawn from an underlying distribution d s , a model parameterized by θ (here, a translation function h θ ) is trained by minimizing the empirical expectation of a loss function ℓ(h θ (f ), e). this approach ensures that the translation loss remains low when translating more sentences drawn from the same distribution. owing to the great variability of language data, this ideal situation is rarely met in practice, warranting the study of an alternative scenario, where the test distribution d t differs from d s . in this setting, domain adaptation (da) methods are in order. da has a long history in machine learning in general (e.g., @xcite @xcite and in nlp in particular (e.g., @xcite @xcite . various techniques thus exist to handle both the situations where a (small) training sample drawn from d t is available in training, or where only samples of source-side (or target-side) sentences are available (see @xcite ; @xcite ; @xcite ; for proposals from the statistical mt era, or @xcite for a recent survey of da for neural mt). a seemingly related problem is multi-domain (md) machine translation @xcite @xcite @xcite where one single system is trained and tested with data from multiple domains. md machine translation (mdmt) corresponds to a very common situation, where all available data, no matter its origin, is used to train a robust system that performs well for any kind of new input. if the intuitions behind mdmt are quite simple, the exact specifications of mdmt systems are rarely spelled out: for instance, should mdmt perform well when the test data is distributed like the training data, when it is equally distributed across domains or when the test distribution is unknown? should mdmt also be robust to new domains? how should it handle domain labeling errors? a related question concerns the relationship between supervised domain adaptation and multidomain translation. the latter task seems more challenging as it tries to optimize mt performance for a more diverse set of potential inputs, with an additional uncertainty regarding the distribution of test data. are there still situations where md systems can surpass single domain adaptation, as is sometimes expected? in this paper, we formulate in a more precise fashion the requirements that an effective mdmt system should meet (section 2). our first contribution is thus of methodological nature and consists of lists of expected properties of mdmt systems and associated measurements to evaluate them (section 3). in doing so, we also shed light on new problems that arise in this context, regarding, for instance, the accommodation of new domains in the course of training, or the computation of automatic domain tags. our second main contribution is experimental and consists in a thorough reanalysis of eight recent multi-domain approaches from the literature, including a variant of a model initially introduced for da. we show in section 4 that existing approaches still fall short to match many of these requirements, notably with respect to the handling of a large amount of heterogeneous domains and to dynamically integrating new domains in training. ## requirements of multi-domain mt in this section, we recap the main reasons for considering a multi-domain scenario and discuss their implications in terms of performance evaluation. ## challenging multi-domain systems in this section, we propose seven operational requirements that can be expected from an effective multi-domain system, and discuss ways to evaluate whether these requirements are actually met. all these evaluations will rest on comparison of translation performance, and do not depend on the choice of a particular metric. to make our results comparable with the literature, we will only use the bleu score @xcite in section 4, noting it may not be the best yardstick to assess subtle improvements of lexical choices that are often associated with domain adapted systems @xcite . other important figures of merit for mdmt systems are the computational training cost and the total number of parameters. ## conclusion and outlook in this study, we have carefully reconsidered the idea of multi-domain machine translation, which seems to be taken for granted in many recent studies. we have spelled out the various motivations for building such systems and the associated expectations in terms of system performance. we have then designed a series of requirements that mdmt systems should meet, and proposed a series of associated test procedures. in our experiments with a representative sample of mdmts, we have found that most requirements were hardly met for our experimental conditions. if mdmt systems are able to outperform the mixed-domain baseline, at least for some domains, they all fall short to match the performance of fine-tuning on each individual domain, which remains the best choice in multisource single domain adaptation. as expected however, mdmts are less brittle than fine-tuning when domain frontiers are uncertain, and can, to a certain extent, dynamically accommodate additional domains, this being especially easy for feature-based approaches. our experiments finally suggest that all methods show decreasing performance when the number of domains or the diversity of the domain mixture increases. two other main conclusions can be drawn from this study: first, it seems that more work is needed to make mdmt systems make the best out of the variety of the available data, both to effectively share what needs to be shared while at the same time separating what needs to be kept separated. we notably see two areas worthy of further exploration: the development of parameter sharing strategies when the number of domains is large; and the design of training strategies that can effectively handle a change of the training mixture, including an increase in the number of domains. both problems are of practical relevance in industrial settings. second, and maybe more importantly, there is a general need to adopt better evaluation methodologies for evaluating mdmt systems, which require systems developers to clearly spell out the testing conditions and the associated expected distribution of testing instances, and to report more than comparisons with simple baselines on a fixed and known handful of domains. training for each domain, we create train/ dev/test sets by randomly splitting each corpus. we maintain the size of validation sets and of test sets equal to 1,000 lines for every domain. the learning rate is set as in @xcite . for the fine-tuning procedures used for ftfull and ft-res, we continue training using the same learning rate schedule, continuing the incrementation of the number of steps. all other mdmt systems reported in tables 3 and 4 use a combined validation set comprising 6,000 lines, obtained by merging the six development sets. for the results in