CLLE : A Benchmark for Continual Language Learning Evaluation in Multilingual Machine Translation (2022)

Abstract: Continual Language Learning (CLL) in multilingual translation is inevitable when new languages are required to be translated. Due to the lack of unified and generalized benchmarks, the evaluation of existing methods is greatly influenced by experimental design which usually has a big gap from the industrial demands. In this work, we propose the first Continual Language Learning Evaluation benchmark CLLE in multilingual translation. CLLE consists of a Chinese-centric corpus — CN-25 and two CLL tasks — the close-distance language continual learning task and the language family continual learning task designed for real and disparate demands. Different from existing translation benchmarks, CLLE considers several restrictions for CLL, including domain distribution alignment, content overlap, language diversity, and the balance of corpus. Furthermore, we propose a novel framework COMETA based on Constrained Optimization and META-learning to alleviate catastrophic forgetting and dependency on history training data by using a meta-model to retain the important parameters for old languages. Our experiments prove that CLLE is a challenging CLL benchmark and that our proposed method is effective when compared with other strong baselines. Due to the construction of the corpus, the task designing and the evaluation method are independent of the centric language, we also construct and release the English-centric corpus EN-25 to facilitate academic research.

## introduction training a multilingual neural machine translation (nmt) model jointly in all the directions re-quires collecting in advance the parallel corpus of all the languages, which is less practical due to the continuously occurrence of the translation requirement of the new languages. adding new languages to a well-trained multilingual nmt model is a resource-saving method compared with training from scratch. however, directly finetuning on new languages will result in catastrophic forgetting of historical languages. continual language learning (cll) methods @xcite @xcite @xcite , focus on gradually extending the language capacity of multilingual nmt models without forgetting old languages which is the major challenge of cll tasks. the existing multilingual nmt evaluation benchmarks @xcite @xcite focus more on multi-task nmt or continual domain learning @xcite but put little emphasis on cll restrictions. hence, most of the existing cll methods @xcite are evaluated on the traditional multilingual nmt evaluation benchmarks and conducted on a simple experiment (e.g. training a multilingual nmt model and adding a specific new language) which has a big gap from the realistic cll. in the industrial demands @xcite , there are usually more new languages and families are required to be continually learned in more continual learning stages. due to the lack of cll benchmark, there are no rigorous evaluations of the cll methods for the number of languages, language family distribution, and learning order. the existed methods' @xcite @xcite @xcite experiment settings such as the selection of the new and old languages, the availability of historical training data, and the growth of model parameters are not unified. however, the catastrophic forgetting observed is sensitive to the experimental design more than any inherent modeling limitations @xcite . hence, it is urgent and necessary to design a benchmark to unify the configurations of the cll tasks. in this work, we propose the first cll benchmark -clle for cll in the multilingual nmt scenario. clle consists of a chinese-centric and domain distribution-consistent multilingual parallel corpus -cn-25, which is collected by extracting and refining the cc-matrix corpus @xcite . specifically, cn-25 includes 25 languages aligned with chinese, 23 of which have more than 650k sentence pairs. the corpus refinement is processed with the text-based filter rules and the labse @xcite multilingual model. the content domain distribution of each language is adjusted to be similar by adjusting the number of samples for each topic clustered by kmeans. we design the close-distance language continual learning (clcl) task and the language family continual learning (lfcl) task associated with the clle benchmark to verify the cll method on disparate experiment settings. to be specific, in the clcl task, the new languages are from the learned languages families, and the addition of new languages is needed at only one learning stage. in the more challenging lfcl task, the new languages are from the unseen languages families, and more learning stages are introduced. for the cll tasks, we propose the cometa framework based on the constrained optimization @xcite and the meta-learning (de masson d' @xcite @xcite . we train a cnn-based meta-model to predict the performance of the nmt model in old languages according to its parameters. then we use the metamodel to calculate the importance weights to retain the language-specific embeddings @xcite @xcite of old languages. compared with standard constrained optimization methods such as ewc @xcite and @xcite , cometa retains the knowledge of the old languages without accessing the historical training data. and the importance weights can be dynamically updated, which is more flexible for the cll process. the main contributions of this work include: benchmarks in multilingual neural machine translation in this section, we focus on multilingual nmt benchmarks including chinese. wmt series @xcite @xcite @xcite corpus includes the commonly used of high-quality languages and most of them are aligned with english. the corpus updates each year, and the multilingual low-resource translation corpus is now added such as the indo-european (akhbardeh et al., 2021) translation. wat series @xcite @xcite @xcite @xcite provide a multilingual multidomain parallel corpus between asian languages and english, and the patent task includes a chinese-japanese parallel corpus. flores-101 @xcite includes the multilingual parallel corpus of 101 languages, in which 3001 sentences were extracted from english wikipedia and translated into 101 languages by professional translators. the corpus covers a variety of different topics and domains, but it only has the test part. opus-100 @xcite ) is an english-centric dataset sampled from the opus collection @xcite and covers a large variety of topics. cwmt series @xcite corpus provides the corpus of china's ethnic minorities' languages such as mongolian, tibetan, and uyghur aligned with simplified chinese. however, the language number is not enough to support chinesecentric cll research. ted talks corpus @xcite has 3540 (60×59) language pairs that rely on volunteers to provide translations for public texts. the content domain of ted corpus includes but not limited to technology, entertainment, and design. although many language corpora are aligned with chinese, the content's overlap of different languages corpus can not meet the chinese-centric multilingual wmt models' training requirements, which are experimentally proved in section 6.2. the above benchmarks are designed for multitask multilingual nmt training or evaluation. due to limitations on language and content distributions, they can not meet the cll requirements. in sec-tion3.4, we compare cn-25 with the above benchmarks in detail. ## the cn-25 corpus in this section, we introduce the domain-aligned corpus cn-25 and compare it with existing multilingual corpora. the processing pipeline of cn-25 is shown in figure 1 . in the first step, we utilize the multilingual encoding model labse and textbased filter rules to refine the cc-matrix corpus. then we use a model-based method to evaluate the refined corpus and the refinement process is regulated according to the evaluation results. to align the topic distribution of different languages, we utilize the central language as the agent and cluster the whole corpus into 100 topics through k-means clustering. in each topic, the sentence number is regulated according to the median value and labse score. ## the continual language learning tasks we then introduce two challenging tasks for cll in our benchmark with different language families and cll stage settings. both the language families and the number of learning stages have an impact on the catastrophic forgetting, as the former is empirically analyzed in appendix a and the latter is shown by @xcite . cll models are evaluated on the newly and historically learned languages by the averaged bleu score after each learning stage. clcl task: close-distance language continual learning. in this scenario, the whole cll task consists of two stages: training a multilingual nmt model at the first stage and then continuing learning new languages at the second stage. the learning sequence in this scenario is relatively short and no new language families are added to the subsequent stage. the languages learned at the second stage belong to the language families of the previous stage. for instance, participants are required to train a multilingual nmt model on three language families including germanic (dutch, german, swedish), romance (portuguese, spanish), and slavic (russian, czech) at the first stage, and then continuously train on three languages including english (germanic), french (romance), and polish (slavic). lfcl task: language family continual learning. in this more challenging scenario, we set more learning stages and introduce new language families to add difficulty to cll approaches. for example, the germanic family (dutch, german, english, and swedish), the romance family (french portuguese, and spanish), and the slavic family (polish, russian and czech) are learned in sequence. in the long sequence of learning, the catastrophic forgetting problem could be serious. existing ccl models which try to retain knowledge from previous stages to alleviate catastrophic forgetting may suffer more time and memory consumption. take ewc, a classical continual learning approach, for example, as the learning stage increases, the time of traversing old training sets and computing will significantly increase when calculating the fisher matrix for the parameters. performance metric. we use the average bleu from and to chinese (the centrical language) to evaluate the performance of methods on cll tasks, which is the commonly used metrics in cll task @xcite . let l i represent the set of historical learned languages up to stage-i, @xmath0, -→ b i,j and ←b i,j be the test bleus of zh->j and j->zh after the model has finished learning stage-i, the average bleus from and to chinese are respectively defined as the average bleu b -1 of the last stage represents the performance on the cll task. ## cometa framework for continual language learning tasks we propose the cometa framework which is based on constrained optimization and metalearning. the core idea is to utilize a new network to evaluate the importance of the nmt model's parameters for old languages, where we call the new network as meta-model. the meta-model is trained to learn the training process of the nmt ln fc max pool 2 * soft+ 3 * 4 * concat figure 4: the structure of meta-model. ## experiments in this section, we evaluate the cn-25 corpus and the cometa method. all the experiments are evaluated under sacrebleu (post, 2018) metric. ## conclusion we propose the first cll benchmark -clle with the cn-25 corpus and two cll tasks -clcl and lfcl. compared with existing multilingual benchmarks, clle considers several restrictions for cll, including domain distribution alignment, content overlap, language diversity, and the balance of corpus. for the cll tasks, we introduce a novel method cometa based on constrained optimization and meta-learning to retain the important parameters for old languages through the meta-model. the experiments prove that cn-25 is a high-quality corpus, that the cll tasks are challenging and that our proposed method outperforms other strong baselines.