Data Selection Curriculum for Neural Machine Translation (2022)

Abstract: Neural Machine Translation (NMT) models are typically trained on heterogeneous data that are concatenated and randomly shuffled. However, not all of the training data are equally useful to the model. Curriculum training aims to present the data to the NMT models in a meaningful order. In this work, we introduce a two-stage training framework for NMT where we fine-tune a base NMT model on subsets of data, selected by both deterministic scoring using pre-trained methods and online scoring that considers prediction scores of the emerging NMT model. Through comprehensive experiments on six language pairs comprising low- and high-resource languages from WMT’21, we have shown that our curriculum strategies consistently demonstrate better quality (up to +2.2 BLEU improvement) and faster convergence (approximately 50% fewer updates).

## introduction the notion of a curriculum came from the human learning experience; we learn better and faster when the learnable examples are presented in a meaningful sequence rather than a random order @xcite . in the case of machine learning, curriculum training hypothesizes presenting the data samples in a meaningful order to machine learners during training such that it imposes structure in the task of learning @xcite . in recent years, neural machine translation (nmt) has shown impressive performance in highresource settings @xcite . typically, training data of the nmt systems are a heterogeneous collection from different domains, sources, topics, styles, and modalities. the quality of the training data also varies a lot, so as their linguistic difficulty levels. the usual practice of training nmt systems is to concatenate all available data into a single pool and randomly sample * work done while tasnim was interning at meta ai training examples. however, not all of them may be useful, some examples may be redundant, and some data might even be noisy and detrimental to the final nmt system performance @xcite . so, nmt systems have the potential to benefit greatly from curriculum training in terms of both speed and quality. in this work, we propose a two-stage training framework for nmt -model warm-up and model fine-tuning, where we apply the data-selection curriculum in the later stage. we initially train a base nmt model in the warm-up stage on all available data. in the fine-tuning stage, we adapt the base model on selected subsets of the data. the subset selection is performed by considering data quality and/or usefulness at the current state of the model. we explore two sets of data-selection curriculum strategies -deterministic and online. the deterministic curriculum uses external measures which require pretrained models for selecting the data subset at the beginning of the model fine-tuning stage and continues training on the selected subset. in contrast, the online curriculum dynamically selects a subset of the data for each epoch without requiring any external measure. specifically, it leverages prediction scores of the emerging nmt model which are the training by-product. for picking the data subset in the online curriculum, we investigate two approaches of dataselection window -static and dynamic. even though the size of the data-selection window is constant throughout the training in the static approach, the samples in the selected subset vary from epoch-to-epoch due to the change in their prediction scores. in contrast, we change the dataselection window size in the dynamic approach by either expanding or shrinking. comprehensive experiments on six language pairs (12 translation directions) comprising lowand high-resource languages from wmt'21 @xcite reveal that our curricu-lum strategies consistently demonstrate better performance compared to the baseline trained on all the data (up to +2.2 bleu). we observe bigger gains in the high-resource pairs compared to the low-resource ones. interestingly, we find that the online curriculum approaches perform on par with the deterministic approaches while not using any external pretrained models. our proposed curriculum training approaches not only exhibit better performance but also converge much faster requiring approximately 50% fewer updates. ## proposed framework we propose a two-stage training framework where in model warm-up stage we train m on general domain parallel data d g for k number of gradient updates; k is generally smaller than the total number of updates m requires for convergence. then in model fine-tuning stage, we adapt m on selected subsets of in-domain parallel data d d . based on the intuition: "not all of the training data are useful or non-redundant, some samples might be irrelevant or even detrimental to the model", we hypothesize that there exists a d s ⊂ d d , fine-tuning on which m will exhibit improved performance. our goal is to design a ranking of the training samples which will eventually help us to extract d s from d d . for this, we investigate two sets of data-selection curriculum strategies -deterministic and online. both strategies require a measure of data quality and/or usefulness at the current state of the model to extract d s . while the deterministic curriculum uses external measures that require pretrained models, the online curriculum leverages the prediction scores of the emerging nmt models. ## experimental setup datasets we conduct experiments on six language pairs: three high-resource including english (en) to/from german (de), hungarian (hu), and estonian (et); and three low-resource including english (en) to/from hausa (ha), tamil (ta), and malay (ms). we use the dataset provided in wmt 2021 foot_2 -de and ha are from news shared task, while the remaining four pairs are from large-scale multilingual mt shared task. for en↔de, we use newstest2019 as validation set and report test results on newstest2020. for en↔ha, we randomly split the provided dev set into validation and test set. for the other language pairs, we use the official evaluation data (dev and devtest) as validation and test sets. baselines we compare our methods with the converged model, which is a standard nmt model trained on all the general-domain data (d g ) until convergence. additionally, we compare both the deterministic and online curriculum approaches with the traditional fine-tuning approach where we fine-tune the base model from the warm-up stage with all the in-domain train data (d d ) until convergence. ## results the main results for the low-and high-resource languages are shown in tables 3 and 4 , respectively. for low-resource languages, we train the warm-up stage models for 20k updates, while the converged models are trained for approximately 50k updates. for high-resource languages, we train for 50k and 100k updates for the warm-up and converged models, respectively. in traditional fine-tuning (traditional ft. row in the tables), we use all the available in-domain data (d d ) in each fine-tuning epoch. on the other hand, for both deterministic and online curricula, we use at most 40% of the available in-domain data (d s ⊂ d d ) in each fine-tuning epoch. comparing the performance of traditional finetuning with the converged model on low-resource languages ( for high-resource languages ( ## discussion and analysis 5.1 hybrid curriculum to benefit from both deterministic and online curricula, we combine the two strategies. specifically, we consider three subsets of data comprising of the top 50% of d d ranked by each of the three bitext scoring methods in §2.1 and keep the common sentence pairs (intersection of three subsets). we then apply the static window data-selection curriculum on these sentence pairs, where we discard the top 10% and bottom 10% pairs (ranked by the emerging model's prediction scores) and fine-tune the base model from the warm-up stage on the remaining bitext. depending on the language pairs, the data percentage for the fine-tuning stage (d s ) becomes 15-20% of d d . despite being a smaller subset of data for fine-tuning, performances of the hybrid curriculum strategy are better on 10 out of 12 translation tasks compared to the baseline ( ## related work curriculum learning inspired by human learners, @xcite argues that optimization of neural network training can be accelerated by gradually increasing the difficulty of the concepts. @xcite were the first to use the term "curricu-lum learning" to refer to the easy-to-hard training strategies in the context of machine learning. using an easy-to-hard curriculum based on increasing vocabulary size in language model training, they achieved performance improvement. recent work @xcite @xcite shows that manoeuvring the sequence of training data can improve both training efficiency and model accuracy. several studies show the effectiveness of the difficulty-based curriculum learning in a wide range of nlp tasks including task-specific word representation learning @xcite , natural language understanding tasks @xcite , reading comprehension @xcite , and language modeling @xcite . several studies show the effectiveness of the difficultybased curriculum learning in a wide range of nlp tasks @xcite . ## conclusion we have presented a two-stage training framework for nmt where we apply a data-selection curriculum in the model fine-tuning stage. our novel online curriculum strategy utilizes the emerging models' prediction scores for the selection of a better data subset. experiments on six low-and high-resource language pairs show the efficacy of our proposed framework. our curriculum training approaches exhibit better performance as well as converge much faster by requiring fewer updates compared to the baselines.