Learning a Multi-Domain Curriculum for Neural Machine Translation (2020)

Abstract: Most data selection research in machine translation focuses on improving a single domain. We perform data selection for multiple domains at once. This is achieved by carefully introducing instance-level domain-relevance features and automatically constructing a training curriculum to gradually concentrate on multi-domain relevant and noise-reduced data batches. Both the choice of features and the use of curriculum are crucial for balancing and improving all domains, including out-of-domain. In large-scale experiments, the multi-domain curriculum simultaneously reaches or outperforms the individual performance and brings solid gains over no-curriculum training.

## introduction in machine translation (mt), data selection, e.g., @xcite , has remained as a fundamental and important research topic. it has played a crucial role in domain adaptation by selecting domain-matching training examples, or data cleaning (aka denoising) by selecting high-quality examples. so far, the most extensively studied scenario assumes a single domain to improve. it becomes both technically challenging and practically appealing to build a large-scale multidomain neural machine translation (nmt) model that performs simultaneously well on multiple domains at once. this requires addressing research challenges such as catastrophic forgetting @xcite at scale and data balancing. such a model can easily find potential use cases, i.e., as a solid general service, for downstream transfer learning, for better deployment efficiency, or for transfer learning across datasets. unfortunately, existing single-domain dataselection methods do not work well for multiple domains. for example, improving the translation accuracy of one domain will often hurt that of another @xcite , and improving model generalization across all domains by clean-data selection @xcite may not promise optimization of a particular domain. multiple aspects need to be considered for training a multi-domain model. this paper presents a dynamic data selection method to multi-domain nmt. things we do differently from previous work in mixing data are the choice of instance-level features and the employment of a multi-domain curriculum that is additionally able to denoise. these are crucial for mixing and improving all domains, including outof-domain. we experiment with large datasets at different noise levels and show that the resulting models meet our requirements. ## related work in mt, research that is most relevant to our work is data selection and data mixing, both being concerned with how to sample examples to train an mt model, usually for domain adaptation. static data selection for a single domain. @xcite select in-domain data for n-gram language model (lm) training. it is later generalized by @xcite to select parallel data for training mt models. @xcite ; @xcite use classifiers to select domain data. clean-data selection @xcite @xcite reduces harmful data noise to improve translation quality across domains. all these works select a data subset for a single "domain" foot_0 and treat the selected data as a static/flat distribution. ## curriculum learning for nmt we first introduce curriculum learning (cl) @xcite , which serves as a formulation for sota single-domain dynamic data selection and which our method is built upon and generalizes. in cl, a curriculum, c, is a sequence of training criteria over training steps. a training criterion, q t (y|x), at step t is associated with a set of weights, w t (x, y), foot_2 over training sentence pairs (x, y) in a parallel dataset d, where y is the translation for x. q t (y|x) is a re-weighting of the original training distribution p (y|x): hence, for t maximum training steps, c is a sequence: at step t, an online learner randomly samples a data batch from q t to fine-tune model m t-1 into m t . therefore, c corresponds to a sequence of models, m is the final model that the entire curriculum has been optimizing towards. intermediate models, m t , serve as "stepping stones" to m , to transfer knowledge through them and regularize the training for generalization. a performance metric p(c) evaluates m on a development or test set, after training on c. in nmt, cl is used to implement dynamic data selection. first, a scoring function (section 4.3) is employed to measure the usefulness of an example to a domain and sort data. then mini-batch sampling, e.g., @xcite , is designed to realize the weighting w t , to dynamically evolve the training criteria q t towards in-domain. figure 1 (1)-( 4 ) illustrates the basic idea of the curriculum we use. (1) shows three sentence pairs, s 1 , s 2 , s 3 , each having three scores, respectively representing usefulness to three domains. a greydomain training curriculum, for example, relies on the data order in (2), gradually discards least useful examples according to w t (x, y) (eq. 1) in ## conclusion existing curriculum learning research in nmt focuses on a single domain. we present a multidomain curriculum learning method. we carefully introduce instance-level features and learn a training curriculum to gradually concentrate on multi-domain relevant and noise-reduced data batches. end-to-end experiments and ablation studies on large datasets at different noise levels show that the multi-domain curriculum simultaneously reaches or outperforms the individual performance and brings solid gains over no-curriculum training, on in-domain and out-ofdomain testsets.