TL - CL : Task And Language Incremental Continual Learning (2024)

Abstract: This paper introduces and investigates the problem of Task and Language Incremental Continual Learning (TLCL), wherein a multilingual model is systematically updated to accommodate new tasks in previously learned languages or new languages for established tasks. This significant yet previously unexplored area holds substantial practical relevance as it mirrors the dynamic requirements of real-world applications. We benchmark a representative set of continual learning (CL) algorithms for TLCL. Furthermore, we propose Task and Language-Specific Adapters (TLSA), an adapter-based parameter-efficient fine-tuning strategy. TLSA facilitates cross-lingual and cross-task transfer and outperforms other parameter-efficient fine-tuning techniques. Crucially, TLSA reduces parameter growth stemming from saving adapters to linear complexity from polynomial complexity as it was with parameter isolation-based adapter tuning. We conducted experiments on several NLP tasks arising across several languages. We observed that TLSA outperforms all other parameter-efficient approaches without requiring access to historical data for replay.

## introduction a deployed nlp model may need to be periodically updated for various reasons, such as adapting to data shifts or accommodating newly available training data. learning new tasks or languages is crucial for real-world nlp models @xcite . for example, a customer service that handles specific tasks like order tracking and returns in english needs to extend its support for additional languages or handle more tasks like catering to payment issues. growing demand for multilingual support makes developing robust multilingual nlp models cruwhile (2) language incremental continual learning focuses on sequentially learning the same tasks in new languages. (3) task and language incremental continual learning enable both by learning new tasks for known languages, learning a new language for a known task or learning completely new tasks in a new language sequentially. cial. a multilingual model @xcite is fundamentally a singular deep neural network trained across multiple languages. this unified approach simplifies the training and maintenance of nlp systems by consolidating multilingual capabilities into a single model. it optimizes performance across languages through shared learning architectures. like monolingual models, multilingual models must be periodically updated to adapt to new tasks or languages. if these updates are not carefully managed, training on new data can reduce the model's ability to perform previously learned tasks or handle earlier languages effectively. this phenomenon is known as catastrophic forgetting @xcite . to prevent catastrophic forgetting, a machine learning paradigm called continual learn-ing (cl) @xcite . cl aims to address the challenges of adapting a pre-trained language model (plm) to a new task, language or domain while retaining previously obtained knowledge. most previous works deal with the challenge of forgetting mitigation for task incremental continual learning (ticl) @xcite @xcite . another line of research deals with cl over language sequences but is mostly limited to language incremental learning for neural machine translation @xcite . there is scarce literature on language incremental continual learning (licl) on other downstream tasks. m'hamdi et al. (2023) studied cross-lingual cl on individual downstream tasks from knowledge preservation, accumulation and generalization perspective. @xcite proposed a parameter-efficient fine-tuning approach using adapters and language similarity metrics. the intertwined complexities of task and language incremental continual learning (tlcl) remain unexplored despite the need. we fill that gap in this paper. in the tlcl setting, a pre-trained, multilingual model must be continually updated with a new task for an already-seen language or a new language for an already-seen task. given a shared multilingual model, the goal of updating is to adapt new task-language 1 pair while retaining knowledge obtained from previously learned tasklanguage data. figure 1 shows three different types of incremental learning. we propose task and language-specific adapters (tlsa), a novel parameter-efficient finetuning approach that leverages adapters for continual learning (cl) over tasks and languages. this method enables cross-lingual and cross-task transfer between different task-language pairs. additionally, tlsa reduces the number of adapters that need to be stored, compared to standard parameter isolation-based fine-tuning using adapters. we conduct experiments using four different tasks, namely classification (cls), natural language inference (nli), context-based question answering (qa) and summarization (summ), each in four languages: english (en), spanish (es), hindi (hi) and arabic (ar). we comprehensively analyze a wide variety of successful cl algorithms from previous 1 throughout paper task-language term refers to a task in particular language literature investigated in a cl context including (a) regularization @xcite , (b) experience replay @xcite and (c) parameter isolation @xcite techniques. the experiments demonstrate the superior performance of tlsa over other parameter-efficient fine tuning approaches. our main contributions are: ## related works from an experimental setup perspective, existing cl in nlp @xcite ) can be broadly divided into ticl and licl. researchers have broadly used replay-based techniques (de masson d' @xcite @xcite that store a subset of samples for future rehearsal via experience replay. additionally, regularisation-based strategies @xcite which restrict changes in model parameters to avoid interference with already learned tasks. architecturebased @xcite ) cl methods learn separate sets of parameters for each task. task incremental continual learning ticl @xcite aims to develop algorithms to accumulate knowledge on non-stationary data for different tasks. some previous works for ticl in nlp include lamol @xcite , which proposes a pseudo-replay-based approach where it learns downstream tasks. it also learns to generate training data for downstream tasks simultaneously. parameter isolation-based methods such as @xcite learn a small set of private parameters for each task with a shared and frozen pretrained model, while @xcite proposes training separate sets of adapters for each downstream task. progressive prompts @xcite trains new soft prompts for each task and sequentially concatenates them with the previously learned prompts. @xcite also trains a separate set of adapters for task-oriented dialogue systems. language incremental continual learning (licl) also known as multilingual continual learning (mcl) @xcite , studies multilingual modelling from a cl perspective. many researchers have studied language incremental cl for neural machine translation (nmt) @xcite tasks. @xcite ; @xcite ; @xcite and @xcite propose a method by replacing shared vocabulary with small language-specific vocabulary and finetuning embeddings on the new language parallel data. @xcite suggests using knowledge distillation for continual nmt. m'hamdi et al. (2023) explores cross-lingual continual learning from knowledge preservation and cross-lingual generalization perspective. @xcite proposes a sparse fine-tuning approach for licl. parameter-efficient finetuning: parameterefficient fine-tuning (peft) has recently garnered attention from many researchers, especially for large language models. recent work on peft has shown that training a subset of parameters can achieve full model performance @xcite @xcite . peft methods like adapters have shown promising results for zero-shot cross-lingual transfer @xcite as well as continual learning @xcite . ## problem definition and background this section formally defines task and language incremental continual learning (tlcl). to the best of our knowledge, we are the first to study tlcl, where the focus is on continually learning both new tasks and new languages. ## methodology in continual learning, a significant challenge is forgetting, which arises due to updates in models' parameters, which may overwrite information that was crucial for previous task-language pairs. traditional cl methods like er @xcite , and ewc @xcite mitigate this to an extent but require fine-tuning the entire model for each new task-language pair, which is computationally expensive. parameter isolation (pi) based parameter efficient fine-tuning (peft) meth-ods like adapters address forgetting and computational costs by training only a subset of parameters. to address our problem setup, we can consider an adapter-based fine-tuning approach on a multilingual pre-trained model, where we train separate adapters for each task-language pair. however, this can limit transfer learning opportunities and lead to a polynomial increase in the number of adapters that need to be stored when a new task or language is added. to address these issues, we propose task and language-specific adapters (tlsa), which support cross-lingual and cross-task transfer while reducing number of adapter sets that need to be stored from polynomial to linear if a new task or language is introduced. ## conclusion and future work we introduce and study the problem of task and language incremental continual learning. we provide the first benchmark to compare the effectiveness of different cl methods for the tlcl case. we further introduce tlsa, a parameter and memory-efficient fine-tuning method for tlcl, and its variant tlsa p i , eliminating forgetting by parameter isolation. tlsa p i outperforms all other cl methods for complete and partial tasklanguage sequence, while er performs better in single-language constraint-based scenarios. tlsa outperforms all other peft methods and performs on par with er without having access to historical data. the selection of tlsa vs tlsa p i can be made based on a tradeoff between memory and performance. we also explore their capabilities for zero-shot generalization to unseen task-language pairs. however, we find them to struggle with zeroshot generalization. therefore, a key future direction is to improve zero-shot transfer capabilities within cl to unseen task-language pairs. limitations since this is one of the first studies on task and language incremental continual learning, the paper focuses on laying the groundwork by establishing the experimental setting on a set of nlp tasks and languages. possible limitations of our experimental setup are the restrictions on evaluating the mt5-small checkpoint and fixing the number of samples for each task-language data. however, in real-life scenarios, one may have access to different amounts of data for each task-language pair. in our setup for evaluation, the language chosen (en, es, hi, ar) is diverse but still a relatively high resource; extending the ticl setup on underrepresented (or even absent) during the pretraining of the model may provide interesting insights. mengjie zhao and hinrich schütze. 2021. discrete and soft prompting for multilingual models. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 8547-8555, online and punta cana, dominican republic. association for computational linguistics.