Bandits Don’t Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits (2021)

Abstract: Training data for machine translation (MT) is often sourced from a multitude of large corpora that are multi-faceted in nature, e.g. containing contents from multiple domains or different levels of quality or complexity. Naturally, these facets do not occur with equal frequency, nor are they equally important for the test scenario at hand. In this work, we propose to optimize this balance jointly with MT model parameters to relieve system developers from manual schedule design. A multi-armed bandit is trained to dynamically choose between facets in a way that is most beneficial for the MT system. We evaluate it on three different multi-facet applications: balancing translationese and natural training data, or data from multiple domains or multiple language pairs. We find that bandit learning leads to competitive MT systems across tasks, and our analysis provides insights into its learned strategies and the underlying data sets.

## introduction parallel training data for machine translation (mt) is commonly sourced and combined from multiple large sub-corpora to obtain the maximum number of training examples. the wmt shared tasks, for example, provide a number of distinct training corpora since @xcite . such corpora are multi-faceted in nature, consisting of a generally unbalanced mixture of data sources that differ from each other in word distribution, domain or other traits. examples of such differences could range from strongly heterogeneous data like distinct languages for training multi-lingual systems @xcite @xcite to rather subtle variations in data provenance (e.g. human-generated vs. machineproduced data crawled from web), through a midstrength variation in multi-domain mt @xcite @xcite . the nature of data facets and their identity is known at training time, either from the data sources directly, for example meta-data from data collection pipelines, or can be provided by dedicated classifiers-but this important information is discarded when mixing and shuffling them for training @xcite . thus, the optimal balance of facets needs to be decided beforehand, and with the requirements at test time in mind. at test time, facets may be equally important, but they might not all have the same amounts of training data. these data balancing decisions are time-consuming and expensive as they often require multiple iterations for striking the right balance between, on the one hand, robust performance at test time on underrepresented facets and, on the other hand, preserving valuable linguistic and lexical information contained in the higherrepresented ones. for additional complication, potential positive and negative transfer between facets should be taken into account @xcite . the complexity of these decisions exacerbate as training data grows. even with established data balancing heuristics in place (e.g. upsampling with a tuned temperature τ foot_0 @xcite ), different balances might be needed at different stages of training. this realization kick-started a development of training curricula @xcite which, despite efforts in neural mt, have yet to produce a recipe applicable to concrete data at hand @xcite . existing curricula presuppose fixed notions of difficulty and come with hand-crafted schedules, often inspired by the human learning process @xcite @xcite . such approaches are brittle in that they may not generalize well across tasks, and there has been evidence that even the reverse of the initially hypothesized order works well @xcite @xcite . this suggests that our human intuitions about difficulty and data succession may not correspond to the optimization process of an nmt system @xcite . in this paper we argue for automatically learned and adaptive data curricula, where the learning system explicitly chooses a facet at each point in training, and does not depend on presupposed schedules. this has three major advantages: first, it relieves system developers from lots of manual work. second, it can improve quality by ignoring irrelevant, redundant or already learned data. third, it can directly optimize for a uniform performance objective to maintain quality on all facets. as a side effect, post-training analyses may improve data interpretability and efficiency @xcite . however, outsourcing data decisions to an auxiliary ml model sacrifices some of control and understanding. in particular, multiple training data selection strategies can lead to models of comparable quality, especially when measured by crude metrics like bleu. we formulate multi-faceted training as a multi-armed bandit learning problem, where the arms/actions correspond to the available facets in the training data. at each training step, the bandit chooses one facet for the mt system to train on and receives a reward signal whether this choice was beneficial for the training progress (see figure 1 ). we implement the exp3 algorithm @xcite as proposed for automated curriculum learning @xcite ( §2), and evaluate it on three different multi-facet applications for machine translation. these require balancing training data that is natural or translationese ( §4.1), comes from a variety of domains ( §4.2), or from many different languages ( §4.3). to the best of our knowledge this is the first study that addresses these problems jointly and provides a competitive solution to all of them. we analyze the effects of different reward signals and chosen facets over time, shedding new light on the importance of different facets for each of the tasks. ## learning to select data with bandits learning a data curriculum can be framed as a multi-arm bandit problem, where the decision to train on a particular subset of data is outsourced to a bandit algorithm that is learned alongside the main task @xcite . after the bandit chooses a facet, the nmt system is updated on a uniformly sampled batch of data from this facet. the system then provides a reward to the bandit, telling how successful this selected batch of data was in terms of overall training progress (see figure 1 ). formally, the bandit selects actions from a set a which is a discrete set of ids. in each round t, the bandit selects an action a @xmath0, y @xmath1, where y t is the complete but unobserved loss vector for each possible action. the bandit parameters are updated to minimize the regret @xmath2. we operate in a fully adversarial setup assuming that reward vectors y t can be arbitrary, i.e., they can depend on the full history, data etc., but cannot be adaptive to the selected action a t . with a collection of subsets of training data (facets), covering the full training data, ∪d @xmath3, the exp3 algorithm proceeds as follows @xcite : input :nmt model θ 0 , number of facets n, exploration rate γ, bandit learning rate µ, training facets d a result: sequence of arms {a 1 , a 2 , . . . , a t } 1 initialize weights @xmath4, . . . , t do 3 π t (a) := (1 -γ) exp(wa) a exp(wa) the regret r behaves as o( √ t ln d) @xcite , so in the limit the bandit will do as good as the best arm from a, i.e., r/@xmath5. @xcite used a slightly modified algorithm exp3.s @xcite that competes against any sequence of actions to reflect dynamic changes. in practice, we found the performance of the vanilla exp3 sufficient for nmt. measuring learning progress @xcite propose a variety of reward functions for measuring learning progress. in this work, we focus on rewards that are functions of the loss value: they can be evaluated on the training batch b t or on a development batch b t dev (denoted with prefix dev-). due to the ambiguity of what can lead to high losses on the training set-such as noisy, unseen or untypical examples-rewards calculated on batches sampled from dev sets (that in our work contain an equal mix of facets) proved to be more successful in our experiments. this also allowed us to inject the equal importance of facets into the rewards. we calculate rewards on randomly sampled batches, since it is cheaper than on the full dev set @xcite . this is not an issue for exp3, as it allows rewards to be non-deterministic: it provably converges for adversarial feedback, and so for identically distributed random rewards too. we linearly re-scale the rewards to [0; 1] clipping them between the 20th and the 80th quantiles of the most recent 5k rewards @xcite . ## results for each task we evaluate whether the banditdirected training schedules can outperform the zeroeffort "take-it-all" approach where datasets are concatenated and training examples are presented in random order. in addition, we compare it to taskspecific best practices, and investigate which strategies are learned by the bandit schedules. ## related work model-based data selection @xcite reported first empirical success of handcrafted schedules for data from different domains which are chosen according to cross-entropy scores of rnn-nmt models. @xcite proposed an online data denoising approach, where noise is measured as the difference of logprobabilities between a learning model and the same model fine-tuned on small set of trusted data. batches are composed of sentences with the highest contrastive data scores (cds) corresponding to the least noisy sentences. our approach is similar to the above in that the multi-armed bandit acts on the online learning success of the mt model, but it is significantly cheaper since it does not require contrastive models nor a pre-defined schedule. furthermore, the requirement of trusted data is lifted. difficulty-based curricula @xcite apply the idea of curriculum learning @xcite to rnn nmt by simple ordering data in buckets corresponding to increased difficulty. zhang et al. (2018) combine non-reusable buckets of difficulties with a manual schedule and achieve small improvements on small data with rnns. platanios et al. (2019) apply a competence-based schedule with lengths and rarity to transformer nmt that re-samples already used examples as long as they fall under the current competency. many works on manually designed curricula note that presenting examples in the reverse order (hard-to-easy) works comparably well (ben @xcite @xcite , which may be a sign of flawed intuitions. our proposed solution groups data into facets rather than difficulty levels and reveals counterintuitive but effective schedules. learned curricula apart from @xcite , whose curriculum learning bandits we adapt for nmt, @xcite is closest to our work. they frame the data selection task as an rl problem and define actions as data clusters corresponding to bins of cds scores @xcite . the same idea of representative batches is reused for multi-armed bandits enhanced with state representations in @xcite . in @xcite another rl algorithm is deployed for optimizing a distribution over training examples using the alignment of training and development gradients as rewards, requiring two backward passes on every step and an additional forward pass on an auxiliary neural net. in contrast to the rl-based approaches, we use light-weight bandits without state representations, which reduces memory and time complexities drastically. bandit learning in mt multi-armed bandits were used in mt to improve general quality, either from online simulated user feedback @xcite @xcite @xcite or from offline logs @xcite for domain adaptation. @xcite applied bandit algorithm to select the best nmt system for a particular translation task, when maintaining of multiple such systems is possible. more generally, rl approaches also seek to improve quality by focusing on more task-informed objectives @xcite and improved approximations to the nmt policies @xcite . unlike these approaches, we treat the nmt model as a black box and do not intervene with its inner workings (see figure 1 ). translationese vs. natural @xcite have shown that the original language a sentence has been written in has a big impact on translation quality, i.e., translating a sentence originally written in the source language is more difficult than translating (back) a sentence that was originally written in the target language and then translated into the source language. this second condition is 'unnatural' for the actual use case of translation systems, but occurs frequently in translation evaluations, if the same dataset is used for evaluating both translation directions. to avoid such artifacts, source sentences for evaluation should be been written originally in the source language @xcite . recently, @xcite showed that mt outputs present lower lexical diversity than human produced texts. mt systems generating outputs closer in style to the original target text are preferred by human judges @xcite . hence our motivation ( §4.1) to produce more natural sounding translations. multi-faceted mt multi-task learning (caruana, 1997) for nmt was introduced by @xcite with the motivation to support a primary tasks with auxiliary data from related tasks. when understanding languages as tasks @xcite , one single mt model can be used to translate between a multitude of languages and in particular also between translation pairs that were not in the training set @xcite . to address the problem of data imbalance @xcite ; @xcite proposed temperature sampling to upsample low-resource languages and downsample higher-resource ones (with τ > 1), that has since turned into the go-to data weighting strategy @xcite . while it is a convenient solution and often outperforms uniform weighting (τ = 1), it reduces the characteristics of languages to their size and reflects the assumption of a zero-sum game in joint training @xcite , ignoring more complex interactions @xcite . our experiments reveal that even very unbalanced and counter-intuitive schedules can lead to improved results across the board thanks to more intricate and automated sampling. closest to our work are recent approaches to schedule data based on inter-facet gradient similarity @xcite , which are more computationally expensive. ## discussion and conclusion we showed that a simple application of the exp3 algorithm @xcite to the training of a black-box nmt system is a cheaper and non-invasive alternative to taskspecific expensively hand-crafted curricula and to heavy rl-based approaches. bandit-optimized data usage leads to improved performance compared to the baselines across the board, and sometimes even faster convergence. on the difficult task of improving naturalness of translations we gained +0.5-0.9 bleu on natural on average; on the multi-domain task up to 1.7 points on certain domains using 72% of the baseline's time to converge; on the multilingual mt task on average-by +1.2 points for translations of 94 languages into english, and by +0.6 points for the reverse. we found intuitive explanations for the learned policies on some of the tasks, but our ability to interpret bandit actions with human reasoning is very limited especially when the number of facets and training steps grow, and also defeats the purpose of replacing possibly flawed human intuitions with learned curricula. as opposed to the expensive development cycles ("train-interpret-retrain") of post-training data interpretability methods @xcite the bandits directly act on their understanding of what is beneficial for the task at hand. after training we can report for each model how much each facet actually mattered, which would increase the transparency of model reporting @xcite , especially for large-scale models @xcite . finally, there are a few limitations of our approach: being stateless, unlike rl approaches (kumar et al., 2019), bandits might be short-sighted and keen on exploiting easy data first. our experiments, though, show that this, with a sufficiently large exploration rate, does not seem to be the case for the tested applications and is not an obstacle to practical use. another limitation are additional hyperparameters to be set (learning and exploration rates, and reward definitions). again, we found it relatively easy to navigate in practice by stopping unpromising runs early (∼50k steps in our runs, cf. figure 2 ); moreover, the hyperparameters tend to generalize across tasks. we believe that the flexibility provided by the reward definitions would allow to inject domain knowledge and/or signals from potentially multiple objectives, and prior knowledge of the data imbalance could be reflected in the exploration rate. our implementation of exp3 samples facets in homogeneous batches, but the current sota models use heterogeneous ones @xcite . this introduces a limitation and a potential hindrance for optimization @xcite , that we hope to address in future work by learning a sampling distribution over individual sentences. with steadily growing training data from more and more sources @xcite , it would also be desirable to model facet hierarchies or intersectionalities, e.g., differentiating between domains and translationese vs. natural within each language pair for a multilingual model. non-printing chars, normalizing punctuation, tokenizing, truecasing and length filtering) to all splits of the data, including the test set. the subtitles part was limited to 500k sentences and concatenated data was preprocessed jointly with 32,000 bpe merges @xcite , resulting in a 32,298 vocabulary entries. maximum training length was 100 post-bpe tokens. we used the same configuration as the base transformer model from @xcite . training on tpuv2 used learning rate 0.01, warmup 4,000, label smoothing 0.1, dropout 0.2 and nominal batch size 256. decoding was done with beam size 5 and maximum length 256 during beam search. bleu score were calculated with sacrebleu on debpe'ed and detokenized sentences w.r.t. similarly preprocessed references. the bandits used the learning rate of 0.1 and exploration rate of 0.25, found by grid search over the range [0.001, 0.01, 0.1] and [0.5, 0.25, 0.1] respectively. a.4 multilingual ted for ted we train the models on 4 v100 gpus with a batch size of 64 sentences for 50k steps, a warmup period of 4k steps for a learning rate schedule with linear increase and square-root decay and a base learning rate of 0.0625. training sentences up to a length of 512 tokens are considered. for inference, beam width is set to 4. models are validated every 2k steps. for opus100 5. bandit learning and exploration rate were tuned over a grid search over the range [0.001, 0.01, 0.01] and [0.1, 0.2, 0.3, 0.4, 0.5] respectively, with training up to 10k training steps. for the diverse task the best setting was (0.1, 0.3) and for the related task (0.01, 0.2).