Optimizing the Training Schedule of Multilingual NMT using Reinforcement Learning (2025)

Abstract: Multilingual NMT is a viable solution for translating low-resource languages (LRLs) when data from high-resource languages (HRLs) from the same language family is available. However, the training schedule, i.e. the order of presentation of languages, has an impact on the quality of such systems. Here, in a many-to-one translation setting, we propose to apply two algorithms that use reinforcement learning to optimize the training schedule of NMT: (1) Teacher-Student Curriculum Learning and (2) Deep Q Network. The former uses an exponentially smoothed estimate of the returns of each action based on the loss on monolingual or multilingual development subsets, while the latter estimates rewards using an additional neural network trained from the history of actions selected in different states of the system, together with the rewards received. On a 8-to-1 translation dataset with LRLs and HRLs, our second method improves BLEU and COMET scores with respect to both random selection of monolingual batches and shuffled multilingual batches, by adjusting the number of presentations of LRL vs. HRL batches.

## introduction multilingual neural machine translation (nmt)is particularly effective to enable the translation of low-resource languages (lrls) when they are accompanied, in the training data, by related highresource languages (hrls) @xcite . including hrls in the training data reduces the chance of overfitting to the lrls and improves translation quality. many-to-one nmt systems can be trained either with monolingual or with multilingual batches. monolingual batches include a single language on the source side, while multilingual batches have their source side sampled from several source languages. using multilingual batches helps avoiding catastrophic forgetting @xcite , but the mixture of languages in each batch may be ineffective at early stages of training. here, we focus on monolingual batches, as they enable us to define the training schedule of a nmt system as the order of presentation of languages, but we also compare our results to those obtained with multilingual batches. we propose to use reinforcement learning (rl) to optimize the training schedule of many-to-one nmt systems, i.e. to improve the training process and the resulting system compared to a fixed sampling strategy. we enable our systems to select the source language of the batch at each training step, based on a learned estimate of the model's competence on each language in terms of loss on a development set. unlike fixed strategies, such as training on the hardest language, we leverage rl to let the model find better strategies. we make the following contributions: foot_0 when the teacher model is in charge of the training schedule of the student, it may use reinforcement learning (rl), with the student model playing the role of the environment @xcite . rl has proved particularly useful at training large language models to follow instructions @xcite , initially using ppo (proximal policy optimization, @xcite and then other algorithms @xcite , but these methods are not designed to optimize training schedules. while it is possible to use curriculum learning to train rl-based models @xcite , e.g. by presenting them with increasingly difficult problems, we focus here on the use of rl to train a teacher model, in the field of multilingual nmt. training schedules for nmt. optimizing the training schedule of an nmt system depends in particular on its architecture. for a system with a single input language and domain, the training sentences can be presented by order of estimated difficulty, or by order of translation reliability or noisiness. when multiple domains must be considered, additional decisions must be made about which domain to use first, or how to mix them based on sizes of available data. similar decisions must be made if there are multiple input languages, as in our case, or if one must train a multi-task system including nmt along with other tasks such as language modeling. we briefly review here previous work along these lines. static scheduling in multilingual nmt. @xcite study the upsampling of the lrl data when building minibatches, and observe that keeping the original proportions of hrl and lrl performs marginally better. however, @xcite and @xcite sample each batch uniformly from a concatenation of all language pairs. arivazhagan et al. (2019) compare simple concatenation with uniform balancing, and observe better results for lrls when using temperaturebased upsampling, which was favored afterwards @xcite . the translation capabilities of large language models (llms) have also been explored: @xcite compares several recent llms and shows that they can achieve state-of-the-art results when translating hrls, but highlights their limitations in translating lrls compared to nmt models. one of the leading open-weights llms for mt, tower instruct @xcite , is fine-tuned on a large set of translation-related tasks in 10 hrls, with no particular scheduling of the fine-tuning data, and no reinforcement learning. ## two rl algorithms for optimizing training schedules in the rl framework, an agent observes the state s t of the environment at each time step t, selects an action a t based on its policy π, executes the action, and receives a reward r t from the environment. using the observed states, actions, and rewards, the goal is to learn an optimal policy, i.e. one that maximizes the cumulative reward over time. bandit problems are those where the agent selects actions without considering state transitions. in this study, as we use for training only monolingual batches, the set of possible actions a is simply the set of source languages. the states s of the system are the values of the parameters of the neural network and of the optimizer. however, these are too numerous to be sensibly observed at each step. drawing inspiration from @xcite , we compute the current state of the model as the vector of cross-entropy loss values obtained from the nmt system over a development batch of sentences. 2 we use the score x t of the model at time step t to compute the reward r t as the decrease of the loss of the nmt system between the last two time steps: r @xmath0. the loss values are computed on a development minibatch of data selected from the current language, or, in some experiments, on a multilingual minibatch. ## experimental settings data. experiments were conducted using a subset of the multilingual ted corpus collected by @xcite , with four hrls and four lrls. 4 for comparability with prior research on multilingual nmt @xcite @xcite , we consider a 8-to-1 translation task with english as the target language. we are especially interested in the translation quality of the four lrls of the dataset: belarusian (be), azerbaijani (az), galician (gl) and slovak (sk), which are respectively paired with a hrl from the same family: russian (ru), turkish (tr), portuguese (pt) and czech (cs). three language families are thus represented (romance, slavic and turkic) but all scripts are latin-based. the numbers of sentences of the training and testing sets for each of the 8 languages are shown in preprocessing. the original data is already tokenized into words. we use byte pair encoding (bpe) for subword extraction and vocabulary construction @xcite . 5 a vocabulary of 32k subwords is generated over a multilingual corpus obtained by combining 10k random lines from the training data of each language, with upsampling for az and be which have fewer than 10k lines. for source language identification by the nmt model, each sentence is prefixed with a language tag. nmt models. we experiment with transformer models from the opennmt-py library version 3.4.3 @xcite . 6 all models are trained for 150k steps. the hyperparameter values are the default ones from the transformer-base model @xcite : 6 layers for the encoder and 6 for the decoder, 8 attention heads, label smoothing of 0.1, hidden layers with 512 units, and feedforward networks with 2,048 units. the adam optimizer (kingma and ba, 2014) is used. following atrio and popescu-belis (2022), we use a batch size of 8k tokens and the regularization parameters are: dropout rate of 0.3, scaling factor of 10, and gradients are re-normalized if their norm exceeds 5. in experiments with warm-up, there are 16k steps during which the learning rate increases from 0 to its maximum. ## conclusion and perspectives in this paper, we presented two algorithms for optimizing the training schedule of multilingual nmt models when a mixture of hrls and lrls must be learned on the source side. the tscl algorithm models the expected return of each action by smoothing past observations, while dqn trains a neural network to perform this estimation and to select the optimal action. both algorithms strike a balance between a uniform distribution of training batches across lan-guages and a distribution purely based on the respective frequencies of these languages in the actual data. the algorithms increase the proportions of lrls and reduce those of hrls, while still enabling cross-lingual transfer from hrls to related lrls. the better balance of hrls and lrls avoids too great a focus on the more abundant hrl data (which would sacrifice lrls) or too great a focus on lrls (which would lead to overfitting). without such algorithms, it would be difficult to find extrinsic criteria to optimize the presentation frequencies of batches. moreover, the optimized training schedules lead to improved macro-average bleu and comet scores. we leave for future work the study of other ways to construct batches. one option is to use multilingual batches -though, as shown above, shuffled batches underperform with respect to an optimized balance of lrls and hrls. another option is to define actions as specific batches or groups of batches, which would enable the model to prioritize certain batches over others, but would also increase the number of possible actions and hence the learning complexity of the rl agent. the relevance of our proposal should be tested with additional datasets combining hrls and related lrls, and with other neural architectures for which cross-lingual transfer may be important to ensure acceptable performance on lrls, particularly llms fine-tuned on translation tasks @xcite . in such cases, an optimized training schedule across available resources may also be beneficial. algorithm 1: tscl algorithm for nmt. require: actions a ← {a 1 , .., a k }, number of training steps ts, number of consecutive actions n, number of warm-up steps w, ϵ-greedy policy exploration parameter ϵ, smoothing coefficient α 1 initialize nmt model 2 initialize action index i ← 1 3 initialize unvisited actions indexes u ← {2, .., k} 4 initialize estimated return q(a k ) ← 0 for all k actions 5 initialize rewards history h(a k ) ← 0 for all k actions 6 for t ← 1, . . . , ts do 7 sample mini-batch b t from action a i 8 train nmt model using mini-batch b t 9 if t mod @xmath1