Rehearsal-Free Modular and Compositional Continual Learning for Language Models (2024)

Abstract: Continual learning aims at incrementally acquiring new knowledge while not forgetting existing knowledge. To overcome catastrophic forgetting, methods are either rehearsal-based, i.e., store data examples from previous tasks for data replay, or isolate parameters dedicated to each task. However, rehearsal-based methods raise privacy and memory issues, and parameter-isolation continual learning does not consider interaction between tasks, thus hindering knowledge transfer. In this work, we propose MoCL, a rehearsal-free **Mo**dular and **C**ompositional Continual **L**earning framework which continually adds new modules to language models and composes them with existing modules. Experiments on various benchmarks show that MoCL outperforms state of the art and effectively facilitates knowledge transfer.

## introduction to effectively deploy machine learning (ml) models in real-world settings, they need to adopt continual learning (cl), i.e., incrementally acquire, update and accumulate knowledge to evolve continually and stay effective over time @xcite . however, cl often suffers from catastrophic forgetting @xcite ): the knowledge learned at early stages of training is overwritten by subsequent model updates. a commonly used strategy to mitigate catastrophic forgetting is to store training samples from prior tasks along the continual learning process and train the model jointly with samples from prior and current tasks (rehearsal) @xcite . however, training samples of prior tasks are not always available due to storage or privacy constraints @xcite . another line of work allocates task-specific parameters to overcome catastrophic forgetting, often referred to as parameter isolation-based cl. although inter-task interference leads to catastrophic forgetting @xcite , knowledge transfer across tasks could be promising. however, those approaches do not enable effective knowledge transfer. recent parameter isolation-based methods either separately train task-specific modules, completely excluding knowledge transfer @xcite , or progressively concatenate all previous task-specific modules with the current task module @xcite , without considering if the interaction between tasks is "positive" (knowledge transfer boosting performance) or "negative" (knowledge interference hurting performance). to address these challenges, we introduce mocl, a modular and compositional continual learning framework for language models. mocl avoids catastrophic forgetting without storing additional data and facilitates effective knowledge transfer via module composition. specifically, mocl allocates task-specific parameters using parameterefficient fine-tuning (peft) modules. 1 during training, mocl continually adds new task-specific modules to language models. to avoid catastrophic forgetting, the task-specific module is frozen once the training on the respective task is finished. additionally, mocl facilitates knowledge transfer across tasks by composing existing and new modules based on task matching weights while learning the new task. in our evaluation on near-domain and far-domain continual learning benchmarks, mocl outperforms state-of-the-art methods under the task-incremental learning setting where the task identities are available during testing. it further demonstrates strong abilities to transfer knowledge of previous tasks to the new tasks. furthermore, the task matching strategy of mocl enables task composition during testing. as a result, mocl effectively addresses the continual learning problem in the challenging class-incremental setting where task identities are not provided during testing. the code base for mocl is available online. 2 ## related work in line with previous work @xcite @xcite , we group cl strategies into three categories. (i) regularization-based methods add explicit regularization terms to preserve the knowledge of previous tasks @xcite @xcite . as regularizing knowledge tends to have suboptimal performance, it is often used in combination with other methods. (ii) rehearsal-based methods address catastrophic forgetting by saving old training samples in a memory buffer @xcite @xcite , or training generative models to provide pseudo samples of previous tasks @xcite for future rehearsal. since rehearsal-based methods raise memory and data privacy issues, we focus on rehearsal-free cl methods. mocl falls into the category of parameter isolation-based continual learning, i.e., we allocate task-specific parameters to avoid knowledge interference. in contrast to related work, we additionally encourage knowledge transfer considering the relatedness across tasks. specifically, we denote the sequence of tasks as {t 1 , . . . , t n }. each task t n contains a set of input samples {(x i n , y i n )}, where x i n is the input text, y i n is the ground-truth label, and @xmath0, . . . , n } is the task identity. a cl model aims to solve the series of tasks which arrive sequentially. the overarching goal is to optimize the model's average performance across all tasks after learning them in the sequence. as we focus on rehearsal-free continual learning, data from earlier tasks is not available when training later tasks, i.e., our model does not suffer from the aforementioned shortcomings of rehearsal-based methods, such as memory issues. while in many benchmark settings, the task identity n is provided, it is not a realistic assumption that task identities are available in realworld setups. thus, we consider two setups: taskincremental learning (til) and class-incremental learning (cil). in til, the task identities are available in both training and testing. in cil, the task identities are only provided during training. 3 3 ## method we propose mocl, a novel cl approach for language models to tackle catastrophic forgetting and enhance knowledge transfer at the same time. avoiding catastrophic forgetting. we utilize two representative peft methods, prefix-tuning @xcite and @xcite for allocating task-specific parameters to lms, avoiding catastrophic forgetting without storing data samples. in particular, mocl adds a set of trainable peft parameters (prefix or lora) to the frozen pretrained language model (plm) for downstream task fine-tuning. instead of updating the whole model, only a small number of the peft parameters are trained. as illustrated in figure 1 , mocl optimizes the task-specific modules and keeps the plm frozen. for each task t n in the sequence, we initialize a trainable module p n for fine-tuning. after the training on one task is finished, the corresponding peft parameters are frozen to preserve the task-specific knowledge in the following training process, thus avoiding catastrophic forgetting. enabling knowledge transfer. mocl introduces task feature vectors for task matching and composes old and new modules for learning. this composition strategy facilitates effective knowledge transfer, which is often ignored by prior work. in particular, while learning on t n , the previously acquired knowledge, which is encoded in the respective peft module (p 1 , . . . , p n-1 ), is reused via a weighted summation, denoted as here, p k is the module specific to the k th task and α k is the weight determining the contribution of p k for new task learning. we detail its computation below. finally, the composed module p ′ n is combined with the plm, consisting of all the module components up to the current task. to calculate the contribution weights α k of each task-specific module, we introduce trainable task feature vectors @xmath1. note that each task-specific vector @xmath2.e., the embeddings from the plm encoder). then, we calculate the cosine similarity between the input embeddings x n and feature vectors up to the current n th task v [: n] as task matching scores α [: n] = cos(x n , v [: n]). training and inference. the training objective for the n th task is to find the peft module p n and the task feature vector v n that minimize the crossentropy loss of training examples, and, at the same time, maximize the cosine similarity between v n and the corresponding task input embeddings x n : during inference, as the task identities are available in the til setting, we directly select the taskspecific module for inference. in the cil setting, we use the matching scores between task inputs and the feature vectors for module composition. the resulting module is combined with the plm for inference. ## experimental setup in this section, we describe our experimental setup. following @xcite , we distinguish benchmarks according to the domain similarity of tasks. as near-domain benchmarks, we use the web-of-science (wos) document classification dataset @xcite consisting of 7 tasks, and afrisenti @xcite , a multilingual sentiment analysis dataset with 12 african languages. as far-domain benchmark, we use the widely adopted mtl5 dataset (de masson d'autume et al., 2019), including 5 text classification tasks. we summarize the details of mtl5 in ## experimental results in this section, we discuss our experimental results. ## analysis in this section, we analyze mocl's forward transfer capability and its matching weights distribution. ## conclusion in this paper, we introduced mocl, a modular and compositional continual learning framework for language models, effectively addressing the critical challenges of catastrophic forgetting and knowledge transfer in continual learning. our broad evaluations across various benchmarks demonstrated mocl's superior performance compared to existing state-of-the-art methods and showed its proficiency in knowledge transfer from previous tasks.