Norm-Based Curriculum Learning for Neural Machine Translation (2020)

Abstract: A neural machine translation (NMT) system is expensive to train, especially with high-resource settings. As the NMT architectures become deeper and wider, this issue gets worse and worse. In this paper, we aim to improve the efficiency of training an NMT by introducing a novel norm-based curriculum learning method. We use the norm (aka length or module) of a word embedding as a measure of 1) the difficulty of the sentence, 2) the competence of the model, and 3) the weight of the sentence. The norm-based sentence difficulty takes the advantages of both linguistically motivated and model-based sentence difficulties. It is easy to determine and contains learning-dependent features. The norm-based model competence makes NMT learn the curriculum in a fully automated way, while the norm-based sentence weight further enhances the learning of the vector representation of the NMT. Experimental results for the WMT’14 English-German and WMT’17 Chinese-English translation tasks demonstrate that the proposed method outperforms strong baselines in terms of BLEU score (+1.17/+1.56) and training speedup (2.22x/3.33x).

## introduction the past several years have witnessed the rapid development of neural machine translation (nmt) based on an encoder-decoder framework to translate natural languages @xcite @xcite . since nmt benefits from a massive amount of training data and works in a cross-lingual setting, it becomes much hungrier for training time than other natural language processing (nlp) tasks. ## background nmt uses a single large neural network to construct a translation model that translates a source sentence x into a target sentence y. during training, given a parallel corpus @xmath0, y n } n @xmath1, nmt aims to maximize its log-likelihood: cl supposes that if mini-batches are bucketed in a particular way (e.g. with examples from easy to difficult), this would boost the performance of nmt and speed up the training process as well. that is, upgrading the sample(•) to ## related work the norm of a word embedding has been sufficiently validated to be highly correlated with word frequency. @xcite and @xcite train a simple word embedding model @xcite on a monolingual corpus, and find that the norm of a word vector is relevant to the frequency of the word and its context sensitivity: frequent words and words that are insensitive to context will have word vectors of low norm values. for language generation tasks, especially nmt, there is still a correlation between word embedding and word frequency. @xcite observe that the word embedding of nmt contains too much frequency information, considering two frequent and rare words that have a similar lexical meaning to be far from each other in terms of vector distance. @xcite regard this issue as a representation degeneration issue that it is hard to learn expressive representations of rare words due to the bias in the objective function. @xcite observe a similar issue during nmt decoding: given two word candidates with similar lexical meanings, nmt chooses the more frequent one as the final translation. they attribute this to the norm of word vector, and find that target words with different frequencies have different norms, which affects the nmt score function. in the present paper, for the sake of obtaining an easy and simple word vector norm requirement, we use the norm derived from a simple word embedding model. in the future, we would like to test norms of various sorts. there are two main avenues for future research regarding cl for nmt: sentence difficulty criteria and curriculum training strategies. regarding sentence difficulty, there are linguistically motivated features @xcite and model-based features @xcite @xcite . both types of difficulty criteria have their pros and cons, while the proposed norm-based sentence difficulty takes the best of both worlds by considering simplicity and effectiveness at the same time. regarding the training strategy, both deterministic @xcite and probabilistic strategies @xcite @xcite can be better than the other, depending on the specific scenario. the former is easier to control and explain, while the latter enables nmt to benefit from the randomization of mini-batch training. however, both kinds of strategy need to carefully tune the cl-related hyperparameters, thus making the training process somewhat costly. in the present paper, we have designed a fully automated training strategy for nmt with the help of vector norms, removing the need for manual setting. ## conclusion we have proposed a novel norm-based curriculum learning method for nmt by: 1) a novel sentence difficulty criterion, consisting of linguistically motivated features and learning-dependent features; 2) a novel model competence criterion enabling a fully automatic learning framework without the need for a task-dependent setting of a feature; and 3) a novel sentence weight, alleviating any bias in the objective function and further improving the representation learning. empirical results on the medium-and large-scale benchmarks confirm the generalizability and usability of the proposed method, which provides a significant performance boost and training speedup for nmt.