Uncertainty-Aware Curriculum Learning for Neural Machine Translation (2020)

Abstract: Neural machine translation (NMT) has proven to be facilitated by curriculum learning which presents examples in an easy-to-hard order at different training stages. The keys lie in the assessment of data difficulty and model competence. We propose uncertainty-aware curriculum learning, which is motivated by the intuition that: 1) the higher the uncertainty in a translation pair, the more complex and rarer the information it contains; and 2) the end of the decline in model uncertainty indicates the completeness of current training stage. Specifically, we serve cross-entropy of an example as its data difficulty and exploit the variance of distributions over the weights of the network to present the model uncertainty. Extensive experiments on various translation tasks reveal that our approach outperforms the strong baseline and related methods on both translation quality and convergence speed. Quantitative analyses reveal that the proposed strategy offers NMT the ability to automatically govern its learning schedule.

## introduction neural machine translation (nmt) has advanced the state-of-the-art on various translation tasks @xcite . a wellperformed nmt is trained using an end-to-end framework @xcite that profits from large-scale training corpus and various optimization tricks @xcite @xcite . these techniques boost the translation quality, in the meanwhile, leading to massive hyper-parameters to be tuned and expensive development costs @xcite . recent studies @xcite @xcite have proven that feeding training examples in a meaningful order rather than considering them randomly can accelerate the model figure 1 : the change of confidence in an area during the learning. humans (red) experience the process of overconfidence⇒despair⇒enlightenment (dunning-kruger curve), while prior work that exploits cl in nmt assumes a monotonically increased curve @xcite . interestingly, our model automatically draws a similar tendency as humans (blue). there exists two open problems in the integration of cl with nmt, i.e. the assessment of data difficulty and the programme of learning schedule. considering the former, prior studies @xcite intuitively treat human linguistic knowledge, e.g. either sentence length or word rarity, as the measure of difficulty. nevertheless, each linguistic feature merely considers an aspect of sentences which fails to fully cope with the data difficulty for a model @xcite . for the latter, existing methods pre-define the duration of curriculum based on an assumption that the model confidence monotonically increases with the training @xcite . we argue that this assumption does not conform to human behavior, i.e. dunning-kruger curve (figure 1 , @xcite , and limits the adaptability and flexibility of curriculum learning. in response to these problems, we propose to strengthen cl for nmt through determining the data difficulty and scheduling the curriculum according to model ability rather than human intuitions. we introduce a novel uncertainty-aware curriculum learning framework, which serves uncertainty as its principle to order the input examples and control the duration of each training stage. specifically, we measure the data uncertainty of a sentence pair according to its joint distribution that is estimated by a language model pre-trained on the training corpus. the intuition behind is that the higher the cross-entropy and uncertainty have in an example, the harder it is to learn and translate @xcite . besides, we calculate the model uncertainty using the variance of the distribution over the network presented by bayesian neural networks @xcite . accordingly, the model uncertainty reflects whether our model can best describe the data distribution @xcite , and the stop of its decline indicates the completeness of the current training stage. one principle in our work is to maintain the simplicity and efficiency in cl. several researchers may doubt that the use of bayesian inference over the training corpus may significantly raise the computational cost. to this end, we apply monte carlo dropout @xcite to approximate bayesian inference. besides, we categorize examples into subsets according to their difficulty, which is then be progressively added into the training set at different training stages, namely baby step @xcite . the model uncertainty can be calculated after each epoch using the samples randomly selected from the current training set, thus avoiding affect training efficiency. we evaluate the effectiveness of our methods on wmt16 english-to-german, iwslt15 english-to-vietnamese, and wmt17 chinese-to-english translation tasks. the experimental results demonstrate that the proposed model consistently improves translation performance over the strong trans-former @xcite baseline and related methods that exploit cl into nmt. extensive analyses confirm that: 1) our approach significantly speeds up the model convergence; 2) using data uncertainty to present the translation difficulty surpasses its sentence length and word rarity counterparts, and this superiority can be further expanded by exploiting a language model that is trained on large-scale external data, i.e. bert @xcite ; 3) the model uncertainty performs a selfadaptive manner to assess the model competence regardless the pre-defined patterns. ## preliminary nmt uses a single, large neural network to build translation model, aiming to maximize the conditional distribution of sentence pairs using parallel corpus @xcite @xcite . formally, the learning objective is to minimize the following loss function over the training corpus @xmath0, y n )} n @xmath1, with the size being n : related work on exploring cl several studies have shown the effectiveness of cl in the field of computer vision @xcite @xcite , as well as a range of nlp tasks, including math word problem @xcite , sentiment analysis @xcite , and natural answer generation @xcite . they point out that cl can solve the problem in some tasks that is hard to train through presenting training data in an easy-to-hard order. @xcite first apply cl into nmt and suggest two sticking points, i.e. data difficulty and learning schedule. partially inspired by their findings, @xcite al. (2018), zhang et al. (2019), wang et al. (2019b 2019 ) pay attention to the schedule that determines the duration of each curriculum. they introduce monotonically increased curves, e.g. either linear or square root, to represent the changes of the model ability across the training process. these early successes presuppose the limited heuristic knowledge on both the data difficulty and the tendency of model competence. to this end, we approach these problems from the model perspective. in this section, we first introduce data uncertainty to quantify the translation difficulty for each training example (section 3.1). then, we propose to predict the model uncertainty at the training time which is a self-adaptive manner to govern curriculum by the model itself (section 3.2). finally, we describe how to exploit the proposed two factors in nmt training (section 3.3). the proposed framework is illustrated in figure 2 . ## experiments we examine our method upon advanced trans-former @xcite and conduct experiments on widely used translation tasks: iwslt15 english-to-vietnamese (en⇒vi), wmt16 englishto-german (en⇒de) and wmt17 chinese-to-english (zh⇒en). foot_1 ## analysis we conduct extensive analyses on en⇒de task to better understand our model. we investigate three problems: 1) whether the proposed model indeed speeds up the model convergence; 2) how different are between difficulty measures; and 3) how the model uncertainty exactly changes during training. ## conclusion and future work we propose a novel uncertainty-aware framework to improve the two key components in cl for nmt, i.e. data difficulty measurement and curriculum arrangement. our contributions are mainly in: as our model is not limited to machine translation, it is interesting to validate the proposed framework into other nlp tasks that need to exploit cl. another promising direction is to design more powerful training strategies to replace the baby step.