Learning to Solve NLP Tasks in an Incremental Number of Languages (2021)

Abstract: In real scenarios, a multilingual model trained to solve NLP tasks on a set of languages can be required to support new languages over time. Unfortunately, the straightforward retraining on a dataset containing annotated examples for all the languages is both expensive and time-consuming, especially when the number of target languages grows. Moreover, the original annotated material may no longer be available due to storage or business constraints. Re-training only with the new language data will inevitably result in Catastrophic Forgetting of previously acquired knowledge. We propose a Continual Learning strategy that updates a model to support new languages over time, while maintaining consistent results on previously learned languages. We define a Teacher-Student framework where the existing model “teaches” to a student model its knowledge about the languages it supports, while the student is also trained on a new language. We report an experimental evaluation in several tasks including Sentence Classification, Relational Learning and Sequence Labeling.

## introduction in natural language processing (nlp), multilingualism refers to the capability of a single model to cope with multiple languages. recently, different transformer-based architectures have been extended to operate over multiple languages, as in @xcite ; @xcite ; @xcite . despite these models can be applied in the zero-shot setting @xcite , in many practical applications their quality will not be satisfactory. instead, fine-tuning over annotated material in each target language is needed to obtain competitive results, as the experimental results in @xcite ; @xcite suggest. having annotated material for all the languages is not always possible, especially when the model has to support an incremental number of new languages over time. in fact, the original fine-tuning material may no longer be available for storage, business or privacy constraints. for example, in a real-world application, customers may request deletion of their data, or the service itself may provide specific data retention policies, or the adopted model may be provided by a third party that did not release the training data @xcite . in these cases, new language support can be added in a continual learning (cl) setting @xcite , that is fine-tuning the model only using the annotated material for the new language(s). however, this approach is vulnerable to the catastrophic forgetting (cf) @xcite of previously learned languages, a well-documented concern discussed in @xcite : when a model is incrementally fine-tuned on new data distributions, it risks forgetting how to treat instances of the previously learned ones. in this paper, we propose a cl strategy for updating a model over an incremental number of languages, so that at each step the model requires only annotated examples of the new language(s). our goal is to remove the dependency on the original fine-tuning material and reduce the need for annotated data at each training step. we propose a teacher-student framework inspired by the knowledge distillation (kd) literature @xcite . although this technique is traditionally used for the purpose of model compression @xcite , recent works in computer vision applied kd to incrementally learn image processing tasks @xcite . here, we adopt kd to miti-gate cf when incrementally training transformerbased architectures @xcite for semantic processing tasks. the existing model (here the teacher) imparts knowledge to a (student) model about the languages it already supports, while this is trained on new languages. we evaluated our approach using multilingual bert-based models on three semantic processing tasks, involving sentence classification, paraphrase identification and sequence tagging. results suggest that the model can progressively learn new languages, while maintaining or even improving its quality over previously observed ones. ## related work continual learning (cl) @xcite studies how to train a machine from a stream of data, which can evolve over time by changing the input distribution or by incorporating new tasks. cl aims to gradually extend the knowledge in a model @xcite , while avoiding catastrophic forgetting @xcite . previous work has mostly focused on computer vision @xcite @xcite by using knowledge distillation (kd) @xcite as the base framework. cl in nlp, as opposed to computer vision, is still nascent @xcite . this reflects in the small number of proposed methods to alleviate cf, as discussed in @xcite . in this context, some works focus on the online learning aspect of the cl @xcite . in nlp, kd has been mainly adopted to compress models @xcite , and was only recently applied for cl in named entity recognition @xcite . in the context of multilingual analysis, most of the works leverage domain adaptation techniques within machine translation @xcite @xcite @xcite in order to apply a machine translation model to an increasing set of languages. to the best of our knowledge, this is the first work adopting cl to mitigate cf when training transformer-based models in an incremental number of languages for semantic processing tasks. ## cl for multilingual processing multilingual continual learning. in the targeted scenario, we have a multilingual neural model, namely m l a , originally pre-trained on a set of languages l @xmath0, l 2 , . . .} (such as multilingual bert @xcite ) and already fine-tuned to solve a task t (such as sentence classification) on a given set of languages l a ⊂ l p . the scope is to extend such model to solve t on a set of new languages l b ⊂ l p , with l a ∩ l @xmath1. in the rest of the discussion, without loss of generality, we assume that l @xmath2, i.e., we support only one new language at a time. in case @xmath3, a sequence of n model extensions can be performed. in our setting, we assume that: (i) a new annotated dataset s {lnew} for task t in language l new is available; (ii) the examples used to fine-tune m l a are not available anymore; (iii) unlabeled examples are available in each language from l a . since l @xmath4, i.e., the original pre-training stage included l new , the model could already operate in a zero-shot setting (i.e., without any fine-tuning stage involving l new data). however, the performance of the zero-shot setting is typically non-satisfactory and a dedicated fine-tuning on l new is generally required. a naive cl strategy consists of finetuning m l a over s {lnew} . however, even though this schema is supposed to produce an effective model for l new instances, it is not guaranteed that the resulting model would still be competitive on languages l a , due to cf @xcite ). an alternative greedy solution consists of adopting self-training as in @xcite : m l a is used to annotate some unlabeled examples in languages l a so that the resulting pseudo-labeled dataset sl a can be used together with s {lnew} to fine-tune m l a and mitigate cf. unfortunately, this can also reinforce the errors of m l a , as discussed in @xcite . preventing catastrophic forgetting. cf is typically caused by the model's weights, which are pushed towards fitting the data of the latest finetuning stage. if the model is not trained using examples in languages l a , it risks forgetting how to treat them. to overcome cf, we propose a method based on knowledge distillation (kd). we define a teacher-student framework where m l a acts as the teacher, while the student is a clone of m l a which is fine-tuned using the multi-loss function l @xmath5. the term l t is the taskspecific loss, computed on the annotated examples from s {lnew} . l kd is a distillation loss computed on u l a , a set of unlabeled examples written in the previous languages l a and here processed by the teacher model. l t thus pushes the model to learn how to solve t in the new language l new . l kd helps the model maintaining a consistent performance on the languages l a , by forcing the student to mimic the teacher predictions on data resembling the data distribution observed in l a . in order to define l kd consistently with @xcite , let us define d i (x) as the output logits of the model's last layer when applied to an example x. the logits are converted into a class-probability distribution using the temperature-softmax: using l kd instead of the self-training procedure preserves the uncertainty of the teacher's model and prevents the student from amplifying the teacher's errors, as demonstrated in @xcite . ## experimental evaluation this section presents the results of the proposed cl strategy over three semantic processing tasks, involving text classification and sequence tagging. in particular, we report the mean absolute error (mae) over the multilingual amazon review corpus (marc) @xcite , i.e., a 5 category sentiment analysis task in 6 languages. we report the accuracy over a sentence-pair classification task, i.e., paraphrase identification on the paws-x dataset @xcite in 6 languages foot_0 . finally, we report the f1 for the named entity recognition (ner) in 4 languages by merging the conll 2002 @xcite and 2003 @xcite datasets. additional details about the datasets are in appendix. experimental setup. we foresee a setting where a bert-based model is incrementally trained using annotated datasets in multiple languages. at each step, the model is fine-tuned using a dataset in one specific language, while the annotated material used up to that point is discarded. we reasonably assume that a set of unlabeled data is available for the languages already observed. in order to simulate this scenario, we designed a data splitting procedure such that each annotated example is observed only in one step. let us assume we observe languages in the order l 1 →, . . . , → l n . for each language l i , its training set d {l i } is divided into n -i + 1 equal slices, i.e., (d (i) as a sort of upper-bound, we report the results by adopting a non-continual learning strategy, i.e., multi-last, where the model is trained from scratch using an annotated dataset in all languages we want to support at step k. more formally, ## conclusions this paper investigated a continual learning strategy, based on knowledge distillation, for training transformer architectures in an incremental number of languages. we demonstrated that with our approach the model maintains its robustness in processing already acquired languages without having access to annotated data for them, while learning new languages. future work will apply our methodology to other nlp tasks, such as qa.