Self-Guided Curriculum Learning for Neural Machine Translation (2021)

Abstract: In supervised learning, a well-trained model should be able to recover ground truth accurately, i.e. the predicted labels are expected to resemble the ground truth labels as much as possible. Inspired by this, we formulate a difficulty criterion based on the recovery degrees of training examples. Motivated by the intuition that after skimming through the training corpus, the neural machine translation (NMT) model “knows” how to schedule a suitable curriculum according to learning difficulty, we propose a self-guided curriculum learning strategy that encourages the NMT model to learn from easy to hard on the basis of recovery degrees. Specifically, we adopt sentence-level BLEU score as the proxy of recovery degree. Experimental results on translation benchmarks including WMT14 English-German and WMT17 Chinese-English demonstrate that our proposed method considerably improves the recovery degree, thus consistently improving the translation performance.

## introduction inspired by the learning behavior of humans, curriculum learning (cl) for neural network training starts from a basic idea of "starting small", namely better to start from easier aspects of a task and then progress towards aspects with increasing level of difficulty @xcite . @xcite achieves significant performance boost on several tasks by forcing models to learn training examples following an order from "easy" to "difficult". they further explain cl method with two important constituents: how to rank training examples by learning difficulty and how to schedule the presentation of training examples based on that rank. figure 1 : the nmt model is well-trained on parallel corpus d, {(x 1 , y 1 ), (x 2 , y 2 )} ∈ d. ŷi is translated from x i . the distance between the ground truth y i and the nmt generated hypothesis ŷi represents the recovery degree (dashed arrows), which is computed by sentence-level bleu in our case. blue-and greencolored examples represent the nmt learned distribution and the empirical distribution, respectively. taking x 1 and x 2 as the input, the training example (x 1 , y 1 ) shows a better recovery degree, which means it's easier to be mastered than (x 2 , y 2 ). in the field of neural machine translation (nmt), empirical studies have shown that cl strategies contribute to both convergence speed and model performance @xcite @xcite @xcite . these cl strategies vary by difficulty criteria and curriculum schedules. early difficulty criterion depends on manually crafted features and prior knowledge such as sentence length and word rarity @xcite . the drawback lies in the fact that humans understand learning difficulty differently from nmt models. recent works choose to derive difficulty criteria based on the probability distribution of training examples to approximate the perspective of nmt models. for instance, @xcite turn discrete numerical difficulty scores into relative probabilities and then construct the difficulty criterion, while others derive difficulty criterion from independently trained language model @xcite @xcite and word embedding model @xcite . @xcite derive difficulty criterion from the nmt model in the training process. and these difficulty criteria are applied to either fixed curriculum schedule @xcite or dynamic one @xcite @xcite . a well-trained nmt model estimates the optimal probability distribution mapping from the source language to the target language, which is assumed to be able to recover the ground truth translations accurately @xcite . however, if we perform inference on the training set, many of the predictions are inconsistent with the references. it reflects the distribution shift between the nmt model leaned distribution and the empirical distribution of training corpus, as figure 1 illustrated. for a training example, a high recovery degree between prediction and ground-truth target sentence means it's easier to be mastered by the nmt model while a lower recovery degree means it's more difficult @xcite . to this end, we employ this recovery degree as the difficulty criterion, where the recovery degree is computed by the sentence-level bleu. we put forward an analogy of this method that humans can schedule a personal and effective curriculum after skimming over a textbook, namely self-guided curriculum. in this work, we cast the recovery degree of each training example as its learning difficulty, enforcing the nmt model to learn from examples with higher recovery degrees to those with lower degrees. also, we implement our proposed recovery-based difficulty criterion with fixed and dynamic curriculum schedules. experimental results on two machine translation benchmarks, i.e., wmt14 en-de and wmt17 zh-en, demonstrate that our proposed self-guided cl can alleviate the distribution shift problem in vanilla nmt models, thus consistently boosting the performance. ## problem definition for a better interpretation of curriculum learning for neural machine translation, we put the discussion of various cl strategies into a probabilistic perspective. such perspective also motivates us to derive this recovery-based difficulty criterion. ## methodology as mentioned above, due to the distribution shift problem, predictions made by a well-trained vanilla nmt model can be inconsistent with the references when performing inference on the training set. training examples with higher recovery degrees are easier to be masted by the nmt model while those with lower recovery degrees are likely to be more difficult. ## results we train our implemented baseline of transformer base and proposed cl models for 300k steps. for both sgcl fixed and sgcl dynamic methods, we observe superior performances over the strong baseline on all three test sets of two benchmarks, which agree with existing approaches that curriculum learning can facilitate the nmt model. and if we compare the two scheduling methods, sgcl dynamic outperforms sgcl fixed. a possible reason is that the dynamic schedule encourages the cl model to spend more steps on the more difficult subset. encouragingly, we observe considerable gains over other curriculum learning counterparts. subset range average d 1 17.72 -100.00 35.62 d 2 9.18 -17.72 12.77 d 3 5.16 -9.18 6.97 d 4 0.00 -5.16 3.35 we conduct experiments on en-de for further analysis of the proposed cl methods. as described in section 3, we adopt sentencelevel bleu score to measure the recovery degrees of all examples in the training corpus with a vanilla nmt model. when making predictions with the vanilla model, we set the beam size to 1 for simplicity. so the recovery degrees could be lower than test results of a strong baseline. if we look at the distribution in terms of bleu score on all training examples, as figure 3 illustrated, the distribution is very dense in the region with lower scores. specifically, more than 53.9% training examples get a recovery degree lower than 10. it reflects the distribution shift problem of well-trained vanilla nmt mode, that the model learned distribution and empirical distribution on training corpus are inconsistent. ## conclusion in this work, we propose a self-guided cl strategy for neural machine translation. the intuition behind it is that after skimming through all training examples, the nmt model naturally learns how to schedule a curriculum for itself. we discuss exist-ing difficulty criteria for curriculum learning from a probabilistic perspective, which also explains our motivation for deriving a difficulty criterion based on recovery degree. moreover, we corporate this recovery-based difficulty criterion with both fixed and dynamic curriculum schedules. empirical results show that with a self-guided cl strategy, the nmt model achieves better performance over the strong baseline on translation benchmarks. in the future, we will corporate recovery-based difficulty criterion with other dynamic scheduling methods. also, it will be interesting to apply our proposed cl strategy to different scenarios, e.g., non-autoregressive generation @xcite @xcite .