Competence-based Curriculum Learning for Multilingual Machine Translation (2021)

Abstract: Currently, multilingual machine translation is receiving more and more attention since it brings better performance for low resource languages (LRLs) and saves more space. However, existing multilingual machine translation models face a severe challenge: imbalance. As a result, the translation performance of different languages in multilingual translation models are quite different. We argue that this imbalance problem stems from the different learning competencies of different languages. Therefore, we focus on balancing the learning competencies of different languages and propose Competence-based Curriculum Learning for Multilingual Machine Translation, named CCL-M. Specifically, we firstly define two competencies to help schedule the high resource languages (HRLs) and the low resource languages: 1) Self-evaluated Competence, evaluating how well the language itself has been learned; and 2) HRLs-evaluated Competence, evaluating whether an LRL is ready to be learned according to HRLs’ Self-evaluated Competence. Based on the above competencies, we utilize the proposed CCL-M algorithm to gradually add new languages into the training set in a curriculum learning manner. Furthermore, we propose a novel competence-aware dynamic balancing sampling strategy for better selecting training samples in multilingual training. Experimental results show that our approach has achieved a steady and significant performance gain compared to the previous state-of-the-art approach on the TED talks dataset.

## introduction with the development of natural language processing and deep learning, multilingual machine translation has gradually attracted the interest of researchers @xcite . moreover, the multilingual machine translation model demands less space than multiple bilingual unidirectional machine translation models, making it more popular among developers @xcite @xcite . however, existing multilingual machine translation models face imbalance problems. on the one hand, various sizes of training corpora for different language pairs cause imbalance. typically, the training corpora size of some high resource languages (hrls) is hundreds or thousands of times that of some low resource languages (lrls) @xcite , resulting in lower competence of lrl learning. on the other hand, translation between different languages has different difficulty, which also leads to imbalance. in general, translation between closely related language pairs is more effortless than that between distant language pairs, even if the training corpora is of the same size @xcite . this would lead to low learning competencies for distant languages compared to closely related languages. therefore, multilingual machine translation is inherently imbalanced, and dealing with this imbalance is critical to advancing multilingual machine translation @xcite . to address the above problem, existing balancing methods can be divided into two categories, i.e., static and dynamic. 1) among static balancing methods, temperature-based sampling @xcite is the most common one, compensating for the gap between different training corpora sizes by oversampling the lrls and undersampling the hrls. 2) researchers have also proposed some dynamic balancing methods @xcite . @xcite introduce an adaptive scheduling, oversampling the languages with poorer results than their respective baselines. in addition, multidds-s @xcite focus on learning an optimal strategy to automatically balance the usage of training corpora for different languages at multilingual training. nevertheless, the above methods focus too much on balancing lrls, resulting in lower competencies for hrls compared to that trained only on bitext corpora. consequently, the performance on the hrls by the multilingual translation model is inevitably worse than that of bitext models by a large margin @xcite . besides, knowledge learned by related hrls is also beneficial for lrls @xcite , while is neglected by previous approaches, limiting the performance on lrls. therefore, in this paper, we try to balance the learning competencies of languages and propose a competence-based curriculum learning approach for multilingual machine translation, named ccl-m. specifically, we firstly define two competence-based evaluation metrics to help schedule languages, which are 1) self-evaluated competence, for evaluating how well the language itself has been learned; and 2) hrls-evaluated competence, for evaluating whether an lrl is ready to be learned by the lrl-specific hrls' self-evaluated competence. based on the above two competencebased evaluation metrics, we design the ccl-m algorithm to gradually add new languages into the training set. furthermore, we propose a novel competence-aware dynamic balancing sampling method for better selecting training samples at multilingual training. we evaluate our approach on the multilingual transformer @xcite and conduct experiments on the ted talks foot_0 to validate the performance in two multilingual machine translation scenarios, i.e., many-to-one and one-to-many ("one" refers to english). experimental results show that our approach brings in consistent and significant improvements compared to the previous state-ofthe-art approach @xcite on multiple translation directions in the two scenarios. our contributions foot_1 are summarized as follows: ## methodology in this section, we first define a directed bipartite language graph, on which we deploy the languages to train. then, we define two competence-based evaluation metrics, i.e., the self-evaluated competence c and the hrls-evaluated competence ĉ, to help decide which languages to learn. finally, we elaborate the entire ccl-m algorithm. ## related work curriculum learning was first proposed by @xcite with the idea of learning samples from easy to hard to get a better optimized model. as a general method for model improvement, curriculum learning has been widely used in a variety of machine learning fields @xcite @xcite @xcite there are also some previous curriculum learning researches for machine translation. for example, @xcite divide the training corpus into smaller buckets using some features such as sentence length or word frequency and then train the buckets from easy to hard according to the predefined difficulty. platanios et al. (2019) propose competence-based curriculum learning for machine translation, which treats the model competence as a variable in training and samples the training corpus in line with the competence. in detail, they believes that competence is positively related to the training steps, and uses linear or square root functions for experiments. we bring the concept of competence and redefine it in this paper with a multilingual context. further, we define self-evaluated competence and hrls-evaluated competence as the competence of each language pair to capture the model's multilingual competence more accurately. ## conclusion in this paper, we focus on balancing the learning competencies of different languages in multilingual machine translation and propose a competencebased curriculum learning framework for this task. the experimental results show that our approach brings significant improvements over baselines and the previous state-of-the-art balancing sampling method, multidds-s. furthermore, the ablation study on sampling methods verifies the great generalibility of our curriculum learning framework. biao zhang, philip williams, ivan titov, and rico sennrich. 2020. improving massively multilingual neural machine translation and zero-shot translation. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 1628-1639, online. association for computational linguistics.