Token-wise Curriculum Learning for Neural Machine Translation (2021)

Abstract: Existing curriculum learning approaches to Neural Machine Translation (NMT) require sampling sufficient amounts of “easy” samples from training data at the early training stage. This is not always achievable for low-resource languages where the amount of training data is limited. To address such a limitation, we propose a novel token-wise curriculum learning approach that creates sufficient amounts of easy samples. Specifically, the model learns to predict a short sub-sequence from the beginning part of each target sentence at the early stage of training. Then the sub-sequence is gradually expanded as the training progresses. Such a new curriculum design is inspired by the cumulative effect of translation errors, which makes the latter tokens more challenging to predict than the beginning ones. Extensive experiments show that our approach can consistently outperform baselines on five language pairs, especially for low-resource languages. Combining our approach with sentence-level methods further improves the performance of high-resource languages.

## introduction neural machine translation (nmt) has achieved significant progress in recent years @xcite @xcite , mainly in the scenarios where the parallel training corpora are abundant. however, training corpora can be limited in some domains (e.g., spoken language @xcite ) and languages (e.g., african languages) due to the high cost of data acquisition. @xcite ; @xcite show that nmt models do not perform well in such data-limited settings. to improve nmt with limited data, researchers resort to large amounts of auxiliary data. one line of research leverages the knowledge from highresource parallel corpora. for examples, some works pre-train nmt models on high-resource data, and then fine-tune them on low-resource data @xcite @xcite @xcite ; others train multilingual or multitask nmt models jointly on both high-resource and low-resource datasets @xcite @xcite . the other line exploits high-resource monolingual data as auxiliary data to train nmt models in a semi-supervised manner @xcite @xcite . aside from previous approaches, curriculum learning @xcite is proposed to address the data insufficiency issue via utilizing the limited data more efficiently @xcite . the idea of curriculum learning is to sample training data in an order of increasing difficulty. the "easy" samples of such a curriculum can be beneficial to the training of the models at the early stage. there have been multiple designs of curriculum for nmt in the recent literatures @xcite @xcite @xcite @xcite . all these methods sample complete sentence pairs for training from a selected subset, which expands as training progresses. we refer to them as the "sentence-level" curricula. however, such a sentence-level design is not necessarily effective for nmt when data is limited. in the early stage of training, the selected subset is usually limited to a small portion of total training samples. in the low-resource setting, this subset contains even fewer samples. to better measure this effect, we use figure 1 to show the diversity of the samples selected in the early training stage under low-resource and high-resource settings 1 . specifically, we count the number of unique trigrams in the sentence pairs used for training up-till a certain training iteration. we observe that the selected samples in low resource setting are less diverse than in high-resource setting, especially in the early curriculum (i.e. up-till 25% of total updates in curriculum). consequently, the sentence-level curriculum slows down the learning progress in low-resource setting although this is not an issue for high-resource setting, as shown in figure 2 . ## method we introduce a token-wise curriculum learning approach for nmt. ## nmt experiments to demonstrate the effectiveness of our token-wise curriculum design, we present experimental results on nmt tasks. ## language modeling experiments to demonstrate our token-wise curriculum can be applied to other sequence generation task, we presents experimental results on language modeling. ## conclusion in this paper, we introduce a novel token-wise curriculum learning method for nmt. we show its superiority in low-resource setting, and is beneficial in high-resource setting. different from existing works, we only consider a vanilla curriculum schedule, where the created sub-sequences expand linearly, as our focus is to validate the idea of tokenwise design. we leave other potential scheduler design, e.g., training adaptive scheduler @xcite , as future discussion.