Multilingual Continual Learning Approaches for Text Classification (2023)

Abstract: Multilingual continual learning is important for models that are designed to be deployed over long periods of time and are required to be updated when new data becomes available. Such models are continually applied to new unseen data that can be in any of the supported languages. One challenge in this scenario is to ensure consistent performance of the model throughout the deployment lifecycle, beginning from the moment of first deployment. We empirically assess the strengths and shortcomings of some continual learning methods in a multilingual setting across two tasks.

## introduction there is a substantial amount of research in continual learning that studies how large language models can be trained in multiple steps. taskincremental learning @xcite @xcite , class-incremental learning @xcite , domain-incremental learning @xcite and language-incremental learning @xcite @xcite have been extensively studied (van de ven and tolias, 2019) in the realm of continual learning. the motivation is for the same model to be trained on various tasks/classes/domains/languages not only to avoid having individual models for each task, class, domain, or language but also to improve the overall model performance. one of the challenges in continual learning is to counteract the tendency of large language models to "forget" previously learned information when trained on new data. in multilingual models, the performance of the model on languages fine-tuned in the past tends to decrease or can even result in catastrophic forgetting @xcite . task-incremental learning, class-incremental learning, and domain-incremental learning research typically focus on few-step continual learning, where the model is incrementally fine-tuned with a few tasks, classes, or domains. while language-incremental learning or multilingual continual learning offers an opportunity to do continual learning over dozens of steps. we will use the term "multilingual continual learning" in this paper. this work addresses the performance of training multilingual models over many fine-tuning steps. we analyze the main existing approaches, provide a summary of their performance on two multilingual datasets and provide an analysis of the trade-offs for selecting one approach over another. for this work, we develop a test scenario that is representative of the practical setting: we update multilingual classification models over a large number of steps with small amounts of training data, and the training data is in different languages. we consider three dimensions in which the approaches differ: the amount of previously seen data that is used in fine-tuning steps, the training time, and the fine-tuning of data in different languages. the main contributions are as follows: ## related work there are multiple approaches to languageincremental learning such as praharaj and matveeva ( 2022 in this paper, we provide a comparison of the main methods in application to continual learning over many steps and discuss considerations for selecting one approach over another. one group of existing studies considered how to construct the training data at each step: joining all training data from all previous fine-tuning steps vs using only the training data from the current step. joint training of all languages (m'hamdi et al., 2022) or domains @xcite has been shown to work better than sequential training using only the current data. since joint tuning does not comply with possible privacy constraints, we consider both approaches in our study, joint fine-tuning with all training data and sequential fine-tuning with only current training data. for sequential fine-tuning, we consider @xcite . they present an analysis of sequential training with multilingual bert @xcite and show that a combination of translation augmentation and specialized training methodology facilitates stable continual learning performance over many multilingual steps. we consider their approach for our study because they also used a large number of fine-tuning steps. we adapt their approach to generating long training sequences with some modifications. adapter-based methods such as @xcite al. (2019); ke et al. (2021); pfeiffer et al. (2020) were proposed as another methodology for robust continual learning. more recently, @xcite have proposed a parameter modularization approach in pre-training (x-mod) that enables positive transfer between languages while also reducing negative interference between them. we include x-mod as one of the approaches in our analysis. memory-based approaches such as @xcite ; @xcite ; @xcite have also been explored to mitigate forgetting. such methods make use of an episodic memory or a cache that stores a subset of data from previous tasks. these examples are then used for training along with the current examples in the current optimization step. we don't consider them in our study. to represent approaches that assume access to the previous training data, we use joint fine-tuning. ## compared approaches we focus on four approaches in this survey. joint training (joint), joint-incremental training (joint-inc), sequential fine-tuning with a specialized optimization regime (seqft-so) @xcite and pre-trained adapters for individual languages (ada-seqft) @xcite . out of the four approaches we consider here, only seqft-so was studied for continual multi-lingual learning. therefore we don't have many related research results. we implemented each approach and used the best practice parameters from the literature for each of them. this section outlines how we implemented these approaches. joint training. (joint) training the base model on all data in the language sequence simultaneously in only one step. previous literature on continual learning has shown that joint training outperforms most sequential training methods. this is a non-incremental baseline since there is no continual learning aspect here. ## results we provide a comparison of the performance of the methods over the marc and mtop datasets in figure 1 . for each dataset, we generated three sequences. for these plots, we took the average accuracy at each step across all three sequences, and we show the average accuracy results for all four approaches. the first important observation is that the different training sequences for each dataset show similar performance trends. this suggests that the approaches are largely stable and show only a small performance variation due to the variation in the order of the languages in the training data, with the exception of ada-seqft on the mtop dataset. this is important because the order of the training data in each sequence is different. for marc, each sequence has a different order of languagecategory combinations, and for mtop, the order of languages is different. the sequence details are provided in we also investigate the role of the size of the training data. joint is the baseline for when all data is available, and as expected, it performs higher in both datasets. joint-inc performance improves with every step as the training set size grows for both datasets. this is in line with expectations and existing results from the literature. on the marc dataset, the performance of all methods seems to converge after 20 steps. seqft-so performance is just below joint-inc and ada-seqft after the initial 15 steps performs the same as the joint baseline and slightly outperforms the other two sequential approaches. on the mtop dataset, we see a wide difference between the approaches. in this case, the joint-inc outperforms the other two sequential approaches and reaches the same accuracy as the joint baseline at step 22, before all training data that is used for joint is available to it. the performance of seqft-so, on the other hand, improves much more gradually. another set of interesting results is about the performance of ada-seqft. adapters are supposed to provide better handling for each language, and this can lead to significant improvements. as we mentioned, ada-seqft has the best performance on the marc data set and even performs the same as the joint baseline. however, ada-seqft performs worst on the mtop dataset. we see that after a brief upward trend, the average accuracy starts dropping. this phenomenon was observed in all three sequences, even though the drop begins at different points: at step 8 for sequence 1, at step 18 for sequence 2, and at step 13 for sequence 3. this explains the wider confidence interval for ada-seqft after step 8. it seems that x-mod is prone to catastrophic forgetting when there are many classes in the data. another interesting observation about ada-seqft is on the marc dataset, the average accuracy after the first non-ipt step collapses to as low as 50%. this was observed in all three sequences. although the performance recovers at the next step, in a practical model deployment, even a one-time drop is undesirable. on the other hand, for the 113-class mtop dataset, combining all available training data results in much higher accuracy. if privacy is not a consideration and training data can be stored and used throughout the multi-step fine-tuning, it is beneficial to use the joint-inc approach for multi-class datasets. we provide the following recommendation. we recommend using seqft-so for two class data. it has a shorter training time because it uses only current training data and is compliant with privacy considerations. if there are data privacy considerations or training time considerations with multiclass data, seqft-so is the recommended robust approach that improves model performance over time. if there are no privacy constraints, joint-inc is expected to provide better performance for multiclass data. ada-seqft requires more research as it exhibits unstable performance in step 3 on the two-class dataset and catastrophic forgetting on the multiclass dataset. ## conclusion we provided a comprehensive study of approaches to multilingual continual learning. we carry out multi-step training using a two-class and a 113class dataset. we consider three dimensions of their difference: the amount of previously seen data that is used in fine-tuning steps, the training time, and handling the fine-tuning for data in different languages. the main result is that all approaches are stable with respect to the order and type of languages in multi-step training data sequences. the adapters approach needs more research to be reliably used in multilingual continual learning, especially for multiclass data. joining all previous training data results in the best performance. however, if there are privacy constraints or training time constraints, sequential incremental learning is a robust alternative.