Learning Curricula for Multilingual Neural Machine Translation Training (2021)

Abstract: Low-resource Multilingual Neural Machine Translation (MNMT) is typically tasked with improving the translation performance on one or more language pairs with the aid of high-resource language pairs. In this paper and we propose two simple search based curricula – orderings of the multilingual training data – which help improve translation performance in conjunction with existing techniques such as fine-tuning. Additionally and we attempt to learn a curriculum for MNMT from scratch jointly with the training of the translation system using contextual multi-arm bandits. We show on the FLORES low-resource translation dataset that these learned curricula can provide better starting points for fine tuning and improve overall performance of the translation system.

## introduction curriculum learning @xcite @xcite hypothesizes that presenting training samples in a meaningful order to machine learners during training may help improve model quality and convergence speed. in the field of neural machine translation (nmt) most curricula are hand designed e.g., fine-tuning @xcite and data selection @xcite @xcite . another common curriculum is one based on ordering samples from easy to hard using linguistic features and auxiliary model scores @xcite but these are hard to tune, relying to extensive trial and error to find the right hyperparameters. attempts to learn a curriculum jointly with the nmt training setup @xcite can suffer from observation sparsity, where a single training run does not provide enough training samples for an external agent to learn a good curriculum policy. our nmt task of choice in this paper is low-resource multi-lingual nmt (mnmt). while standard nmt systems typically deal with a language pair, the source and the target, an mnmt model may have multiple languages as source and/or target. most large-scale mnmt models are trained using some form of model parameter sharing @xcite @xcite . the notion of how input data should be presented to the mnmt system during training only finds prominence in the case of low-resource mnmt. a typical low-resource task will try to leverage a high-resource language pair to aid the training of an nmt system for a low-resource (very small or no parallel data available) and related language-pair of interest. typical approaches for low resource mnmt involve pivoting and zero-shot training @xcite and transfer learning via fine-tuning @xcite . @xcite attempt to meta-learn parameter initialization for child models using trained-high resource parent models for this task. in this paper, we build upon the framework for learning curricula introduced in @xcite and attempt to alleviate the problem of observation sparsity by learning more robust policies from multiple training runs. we use contextual multi-arm bandits for our agents which learn multilingual data sampling policies jointly with the training of the nmt system. additionally, we explore some simple policy search methods to our list of baselines; specifically, we try and find the best policies using the expensive grid search and pruned-tree search methods. we use state-of-the-art hand-designed curricula as our baselines to beat. building upon the task and datasets established by @xcite , in this paper, we will attempt to learn a curriculum to train an nmt system for the nepali-english language pair while leveraging the high resource hindi-english pair. the agent will learn to choose between mini-batches containing either hindi-english or nepali-english data at each time step during nmt training to maximize the expected reward (improvement in validation set performance). the learned curriculum will hence condition on the state of the nmt system during training and determine whether to expose it to a batch of nepali-english or hindi-english data. we start by presenting our methods for obtaining search-based and learned curricula in section 2. we present our experiment setup in section 3 and results in section 4. ## methods the procedure for learning a multi-lingual training curriculum uses multiple multi-arm bandits as agents which explore independent of each other in randomly initialized environments (nmt systems) and effectively learn their own policies. the stochastic nature of their exploration policy ensures that they explore different action-reward spaces (the agent executes an action on the nmt environment and receives a reward associated with this action). figure 1 shows an overview of this interface. the training data for all agents is pooled at the end of the training of individual agents and one final agent is trained using this data which determines the final policy we use as our multi-lingual curriculum. we provide more details about this method of learning and the associated baselines below. ## experiment setup we use fairseq @xcite for all our nmt experiments and the our nmt systems are configured to replicate the setup described in @xcite . the grid search experiments search over the the range [0, 1] for sampling in increments of 0.1. the pruned tree-search uses a beam width of 1. the phase duration for tree-search is set to one epoch of nmt training. we use either 5 or 10 concurrent contextual mabs which are implemented as two 256-dimensional feed forward neural networks trained using rmsprop with a learning rate of 0.00025 and a decay of 0.95. rewards for the agent (validation delta-perplexity) are provided every ten training steps. to create the observations, we sample 32 prototype sentences from each bin to create a prototype dataset sentences tokens nepali-english 563k 6.8m hindi-english 1.6m 16.7m ## results our results are presented in our non-mab search-based curriculum baselines are: from ## conclusion in this paper, we present techniques which learn curricula for multilingual nmt training from multiple training runs and agents. on the task of low-resource multilingual nmt training, we use conditional multi-arm bandits which condition on the state of the nmt system and learn policies which determine whether to train on a batch of a high-resource (hindi-english) or the low-resource (nepali-hindi) language pair per step in training. in addition, we introduce some simple search-based methods for policy search (grid search and pruned tree search) for this task. we show that both these simple learned curricula and the ones derived from the mabs can match the state-of-the-art hand-designed multilingual baselines. however, continued training on models trained using these learned curricula yields better results, indicating that they may serve as good starting models for fine-tuning.