Improving Both Domain Robustness and Domain Adaptability in Machine Translation (2022)

Abstract: We consider two problems of NMT domain adaptation using meta-learning. First, we want to reach domain robustness, i.e., we want to reach high quality on both domains seen in the training data and unseen domains. Second, we want our systems to be adaptive, i.e., making it possible to finetune systems with just hundreds of in-domain parallel sentences. We study the domain adaptability of meta-learning when improving the domain robustness of the model. In this paper, we propose a novel approach, RMLNMT (Robust Meta-Learning Framework for Neural Machine Translation Domain Adaptation), which improves the robustness of existing meta-learning models. More specifically, we show how to use a domain classifier in curriculum learning and we integrate the word-level domain mixing model into the meta-learning framework with a balanced sampling strategy. Experiments on English-German and English-Chinese translation show that RMLNMT improves in terms of both domain robustness and domain adaptability in seen and unseen domains.

## introduction the success of neural machine translation (nmt; @xcite heavily relies on large-scale high-quality parallel data, which is difficult to obtain in some domains. we study two major problems in nmt domain adaptation. first, models should work well on both seen domains (the domains in the training data) and unseen domains (domains which do not occur in the training data). we call this property domain robustness. second, with just hundreds of in-domain sentences, we want to be able to quickly adapt to a new domain. we call this property domain adaptability. previous work on nmt domain adaptation has usually focused on only one aspect of domain adaptation at the expense of the other one, and our motivation is to consider both of the two properties. there are a few works attempting to solve domain adaptability. the most basic approach is fine-tuning, in which an out-of-domain model is continually trained on in-domain data @xcite . although fine-tuning is effective, it can suffer from so-called catastrophic forgetting @xcite , resulting in deteriorated model performance in general domains @xcite . another efficient method is meta-learning @xcite , which trains models which can be later rapidly adapted to new scenarios using only a small amount of data. it works for many natural language processing (nlp) tasks @xcite @xcite @xcite , especially in low-resource scenarios @xcite . as a result, meta-learning is often used for nmt domain adaptation. for example, @xcite and @xcite fast adapt nmt models to new domains with meta-learning using a small amount of training data. @xcite improve meta-learning-based nmt models with a curriculum-based @xcite sampling strategy. meta-learning works well for adapting to new domains, however, previous work tends to neglect the problem of robustness towards domains unseen at training time. @xcite defined the concept of domain robustness and propose to improve the domain robustness by subword regularization @xcite , defensive distillation @xcite , reconstruction @xcite and neural noisy channel reranking @xcite . @xcite proposed using individual modules for each domain with a word-level domain mixing strategy, which they showed has domain robustness on seen domains. the work on domain robustness, however, tends to neglect the adaptability of the models for new domains. to address both domain adaptability and domain robustness at the same time, we propose rmlnmt (robust meta-learning nmt), a more robust meta-learning-based nmt domain adaptation framework. we first train a word-level domain mixing model to improve the robustness on seen domains, and show that, surprisingly, this improves robustness on unseen domains as well. then, we train a domain classifier based on bert @xcite to score training sentences; the score measures similarity between out-of-domain and general-domain sentences. this score is used to determine a curriculum to improve the meta-learning process. finally, we improve domain adaptability by integrating the domain-mixing model into a meta-learning framework with the domain classifier using a balanced sampling strategy. in summary, we make the following contributions: i) we propose rmlnmt, which shows better domain robustness and domain adaptability than all previous baseline systems; ii) we show that unseen domains can be very effectively handled with domain-robust models, even though post-hoc adaptation with domain-specific data still delivers the best overall translation quality; iii) experiments on @xmath0. to the best of our knowledge, this is the first work that considers both domain adaptability and domain robustness in nmt domain adaptation, a combination which we suggest the community pay more attention to. ## preliminaries neural machine translation. the goal of the nmt model is to model the conditional distribution of translated sentence @xmath1, ..., y n ) given a source sentence @xmath2, ..., x m ). current stateof-art nmt models (transformers; @xcite model the multi-head attention mechanism to focus on information in different representation subspaces from different positions and w @xmath3. for the i-th head h i , m is the number of heads, and d is the dimension of the model output. in some of our experiments (see section 3.1), we modify the multihead attention to do domain mixing @xcite . meta-learning for nmt. the goal of meta-learning is training a teacher model that using previous experience can be better finetuned for new tasks, including handling different domains in nmt domain adaptation @xcite @xcite . the idea of nmt domain adaptation with meta-learning is to use a small set of source tasks {t 1 , . . . , t n } (which correspond to domains) to find the initialization of model parameters θ from which finetuning for task t 0 would require only a small number of training examples. these meta-learning algorithms consist of three main steps: (i) split the seen domain corpus into small tasks t containing a small amount of data as d meta-train and d meta-test to simulate the low-resource scenarios. data for each task t i is decomposed into two sub-sets: a support set t support used for training the model and a query set t query used for evaluating the model; (ii) leverage a metalearning policy to adapt model parameters to different small tasks using d meta-train datasets. we use maml, proposed by @xcite , to create adaptable nmt systems which will be useful for different domains; (iii) finetune the model using the support set of d meta-test . ## method in our initial experiments, we observed that the standard meta-learning approach for nmt domain adaptation sacrifices the domain robustness on seen domains in order to improve the domain adaptability on unseen domains. to address these issues, we propose a novel approach, rmlnmt, which combines meta-learning with a word-level domainmixing system (for improving domain robustness) in a single model. rmlnmt consists of three parts: word-level domain mixing, domain classification, and online meta-learning. figure 1 illustrates rmlnmt. ## experiments datasets. we experiment with @xmath4. for the en2de task, we use the same corpora as @xcite . the data consists of corpora in nine domains (bible, books, ecb, emea, glob-alvoices, jrc, kde, ted, wmt-news) publicly available on opus foot_1 @xcite and the covid-19 corpus foot_2 . for en2zh, we use um-corpus (tian et al., 2014) containing eight domains: education, microblog, science, subtitles, laws, news, spoken, thesis. we use wmt14 (en2de) and wmt18 (en2zh) corpus published on the wmt website foot_3 as our general domain corpora. we use wmt19 english monolingual corpora to train the lm model so that we can reproduce results from previous work. ## results ## analysis in this section, we conduct additional experiments to better understand the strengths of rmlnmt. we analyze the contribution of different components in rmlnmt, through an ablation study. ## conclusion we presented rmlnmt, a robust meta-learning framework for low-resource nmt domain adaptation reaching both high domain adaptability and domain robustness (both in the seen domains and unseen domains). we found that domain robustness dominates the results compared to domain adaptability in meta-learning based approaches. the results show that rmlnmt works best in setups that require high robustness in low-resource scenarios.