Additive Interventions Yield Robust Multi-Domain Machine Translation Models (2022)

Abstract: Additive interventions are a recently-proposed mechanism for controlling target-side attributes in neural machine translation by modulating the encoder’s representation of a source sequence as opposed to manipulating the raw source sequence as seen in most previous tag-based approaches. In this work we examine the role of additive interventions in a large-scale multi-domain machine translation setting and compare its performance in various inference scenarios. We find that while the performance difference is small between intervention-based systems and tag-based systems when the domain label matches the test domain, intervention-based systems are robust to label error, making them an attractive choice under label uncertainty. Further, we find that the superiority of single-domain fine-tuning comes under question when training data is scaled, contradicting previous findings.

## introduction multi-domain machine translation (mdmt) is the paradigm in which a single model is trained to service many domains by training on multiple corpora covering disparate labeled domains. the goal of mdmt is not only to provide high quality general machine translation enabled by knowledge transfer across domains, but also to enable high quality domain-specific machine translation when a model is provided cues about the target domain, used to control the generation. though an intuitive task, the expectations surrounding the task were only recently formalized by @xcite in which the authors provided both a set of functional requirements demanded of successful mdmt models and an experimental framework under which those requirements can be tested. @xcite explored several mechanisms for controlling domain, ranging from simple tag-based approaches to meta-learning based mechanisms. according to the functional requirements outlined by the authors, no method meets all the expectations demanded of effective multi-domain machine translators, though the experiments were run on a relatively small dataset of only in-domain data. the primary remaining expectations, according to the authors, are the superiority of fine-tuning based methods as compared to these methods which can control the target domain, and the ability to accommodate fuzzy or uncertain domains. this framework is useful, but the authors leave open several other questions regarding the state of mdmt. the first of these is data size. previous experiments focused only on relatively small, indomain data in an otherwise high-resource setting of english-french and found that most models pale in comparison to models fine-tuned on a single domain. we wonder whether this fine-tuning superiority conclusion holds under a more realistic paradigm in which models trained on large, out-ofdomain datasets are fine-tuned on in-domain data. while pretraining and fine-tuning on in-domain data can yield strong in-domain performance-as observed by the authors-this is likely to be at the cost of general domain performance, calling into question the transferability under mdmt. next, we wonder if new methods might help with the issue of domain control in mdmt. the authors examine reasonable mechanisms for controlling the domain which were known at the time. since then, new methods have been developed which we hope to investigate under the prescribed framework. we hypothesize that additive interventions @xcite , which learn tag embeddings separately from the encoder, may be harder to ignore, and that the learned interventions may be able to absorb target-side properties more easily, while freeing the encoder to learn strong representations purely for translation. in this work we scale the original experimental framework presented in @xcite by including a significantly larger, more realistic dataset. we also experiment with additive interventions as an alternative to domain tagging. we find that: ## method as a baseline, we inject domain metadata using the tag-based approach. in this scheme, a token representing the target-side attribute, t, is prepended to source segment x and fed to the encoder e whose hidden representation is finally exposed to decoder d in a "normal" fashion: while the original work only introduces the interventions to the top-most decoder layers in order to allow for partially freezing pretrained networks, we simplify by applying the intervention to the top layer of the encoder, such that it affects all decoder layers. further, the authors report that improved general performance can be promoted by randomly inducing a zero-vector intervention. as such, we can specify that t is randomly replaced by ⟨pad⟩ with some probability with the same effect. we report 20% masking in this paper, though we experiment with 0% masking and find no significant differences between the two. ## related work incorporating extra-sentential information has a rich history in nmt. aside from controlling for the domain, @xcite use a politeness tag at training and inference time to accommodate coarse politeness control in machine translation. additionally, @xcite use tags to afford users the ability to vary binary gender in the translations of gender-neutral inputs, hoping to address gender bias in mt. at the sub-sequence level, hoang et al. (2016) and @xcite included linguistically-informed word-level "source factors", such as part-of-speech tags and dependency relations, as additional feature factors to be concatenated to form a full encoder representation with the goal of reducing ambiguity and sparseness issues. perhaps more relatedly, several works have explored the impacts of incorporating domain information into training using various methods. @xcite explore two methods: a tag-based approach which concatenates a special token to the end of the source sequence, and a "source factors"style approach which concatenates domain-level embeddings to each token embedding in the source. @xcite explore few-shot domain adaptation, rather than domain control, through the lens of meta-learning and show that a meta-learning based approach is generally stronger than other adaptation approaches, though we note that adaptation and control address different needs. finally, @xcite frame machine translation with document-context as an unsupervised domain adaptation problem and incorporate do-main embeddings within the encoder, summed with positional and word embeddings, yielding strong improvements over competitive baseline models. ## conclusion in this work we examined the relative impact of additive interventions in a large-scale mdmt setting. we find that typically there are no significant differences between additive interventions and tagbased approaches when the provided domain label matches the test set, but find that additive interventions exhibit much more desirable degradation properties when the domain label is unknown or incorrectly provided. in addition, we find that models first trained on a large, general corpus and then fine-tuned on a single-domain-a realistic baseline in machine translation-rarely perform significantly better than approaches which are trained or fine-tuned only on in-domain data, which is in contrast to their generally superior performance in low-resource settings. in future work we consider developing extensions to additive interventions which can further improve their performance in mdmt settings. additionally, studying additive interventions in other tasks where tag-based approaches are dominant, such as multi-lingual machine translation, could be an interesting avenue for exploration.