Self-Paced Learning for Neural Machine Translation (2020)

Abstract: Recent studies have proven that the training of neural machine translation (NMT) can be facilitated by mimicking the learning process of humans. Nevertheless, achievements of such kind of curriculum learning rely on the quality of artificial schedule drawn up with the handcrafted features, e.g. sentence length or word rarity. We ameliorate this procedure with a more flexible manner by proposing self-paced learning, where NMT model is allowed to 1) automatically quantify the learning confidence over training examples; and 2) flexibly govern its learning via regulating the loss in each iteration step. Experimental results over multiple translation tasks demonstrate that the proposed model yields better performance than strong baselines and those models trained with human-designed curricula on both translation quality and convergence speed.

## introduction neural machine translation (nmt) has achieved promising results with the use of various optimization tricks @xcite @xcite @xcite . in spite of that, these techniques lead to increased training time and massive hyper-parameters, making the development of a well-performed system expensive @xcite . as an alternative mitigation, curriculum learning @xcite has shown its effectiveness on speeding up the convergence and stabilizing the nmt model training @xcite . cl teaches nmt model from easy examples to complex ones rather than equally considering all samples, where the keys lie in the definition of "diffi-culty" and the strategy of curricula design @xcite . existing studies artificially determine data difficulty according to prior linguistic knowledge such as sentence length (sl) and word rarity (wr) @xcite @xcite , and manually tune the learning schedule @xcite . however, neither there exists a clear distinction between easy and hard examples @xcite , nor these human intuitions exactly conform to effective model training @xcite . instead, we resolve this problem by introducing self-paced learning @xcite , where the emphasis of learning can be dynamically determined by model itself rather than human intuitions. specifically, our model measures the level of confidence on each training example @xcite , where an easy sample is actually the one of high confidence by the current trained model. then, the confidence score is served as a factor to weight the loss of its corresponding example. in this way, the training process can be dynamically guided by model itself, refraining from human predefined patterns. we evaluate our proposed method on iwslt15 en⇒vi, wmt14 en⇒de, as well as wmt17 zh⇒en translation tasks. experimental results reveal that our approach consistently yields better translation quality and faster convergence speed than transformer @xcite baseline and recent models that exploit cl @xcite . quantitative analyses further confirm that the intuitive curriculum schedule for a human does not fully cope with that for model learning. ## experiments we evaluate our method upon transformer-base/big model @xcite and conduct experiments on iwslt15 english-to-vietnamese (en⇒vi), wmt14 english-to-german (en⇒de) and wmt17 chinese-to-english (zh⇒en) tasks. for fair comparison, we use the same experimental setting as @xcite for en⇒vi and follow the common configuration in @xcite for en⇒de and zh⇒en. during training, we apply 0.3 dropout ratio and batch size as 4,096 for en⇒vi task, and experiments are conducted upon one nvidia gtx1080ti gpu device. for en⇒de and zh⇒en task, we use 32,768 as batch size, and use four nvidia v100 gpu devices for experiments. we use beam size as 4, 5, 10, and decoding alpha as 1.5, 0.6, 1.35 for each task, respectively @xcite . we compare our models with two baselines: ## analysis in this section, we further investigate how the proposed method exactly affects the nmt model training by conducting experimental analyses on 1) convergence speed, 2) self-paced adjustment and 3) sequential bucketing. ## conclusion in this paper, we propose a novel self-paced learning model for nmt in which the learning schedule is determined by model itself rather than being intuitively predefined by humans. experimental results on three translation tasks verify the universal effectiveness of our approach. quantitative analyses confirm that exploiting self-paced strategy presents a more flexible way to facilitate the model convergence than its cl counterparts. it is interesting to combine with other techniques @xcite to further improve nmt. besides, as this idea is not limited to machine translation, it is also interesting to validate our model in other nlp tasks, such as low-resource nmt model training @xcite and neural architecture search @xcite .