Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Models (2024)

Abstract: Parameter-efficient fine-tuning (PEFT) methods are increasingly used with pre-trained language models (PLMs) for continual learning (CL). These methods typically involve training a PEFT module for each new task and employing similarity-based selection to route modules during inference. However, they face two major limitations: 1) interference during module training with already learned modules and 2) suboptimal routing when composing modules. In this paper, we present L2R, a method that isolates the training of new PEFT modules to ensure their task specialization. L2R then learns to compose the learned modules by training a network of routers that leverages a small memory containing examples of previously seen tasks. We evaluate our method in two CL setups using various benchmarks. Our results demonstrate that L2R provides an effective composition of PEFT modules, leading to improved generalization and performance compared to other methods.

## introduction cl aims to continuously learn new tasks without forgetting previously learned knowledge from old tasks @xcite . recent advancements in plms and peft methods have shown potential in cl for nlp @xcite . peft methods, like prompts or adapters, are lightweight modules that can be trained for downstream tasks while keeping the plm frozen. recent work shows that these methods can be competitive with, or even superior to, full fine-tuning of plms @xcite . additionally, research @xcite @xcite demonstrates their effectiveness in cl. however, peft approaches for cl have two significant limitations: 1) when a new task arises, a new peft module is added, and the previously learned ones remain frozen but activated @xcite . previous modules may inter-fere with optimal task knowledge acquisition by the current peft module. 2) they rely on a similaritybased selection mechanism @xcite that may not accurately represent a routing function that effectively composes the peft modules. to address these issues, we introduce learning to route for dynamic peft composition (l2r). l2r is a simple but effective method that trains peft modules in isolation to avoid interference from previous knowledge when learning new tasks @xcite . then, it uses previous examples stored in a memory to learn a routing function for dynamically composing peft modules during inference, a process analogous to local adaptation @xcite . our extensive evaluations show that l2r is competitive against other peftbased methods across benchmarks and cl setups. unlike existing methods that focus on developing routing mechanisms within the training phase, our method learns to route adapters prior to testtime inference. from a practitioner's perspective, the ultimate goal of cl is to develop a model that can learn new tasks while maintaining high performance @xcite . our method addresses this by incrementally adding peft modules for new tasks. additionally, in real-world applications, systems are primarily constrained by computational and time budgets rather than storage @xcite . by leveraging a memory along peft, our approach achieves a better trade-off between performance and efficiency. our work focuses on the peft setup. we rely on the parameter isolation strategy @xcite to let the plm learn taskspecific modules independently. we also adopt a memory as in replay-based methods, not to rehearse examples but to learn a routing function for effective module composition before inference. ## method we consider a cl setup for nlp @xcite , where a model needs to learn from a sequence of t tasks: @xmath0, ..., d t }. each task t consists of a new data distribution d @xmath1, y t ), where x t are the input instances and y t are the labels. a model in this setup consists of a plm that processes input x and a classifier for predicting class y. we extend this setup with peft modules, specifically adapters, which consistently achieve superior predictive performance than other techniques in cl @xcite . task-specific adapters l2r employs a set of adapters @xmath2, ..., a t } that learn task-specific knowledge from a data stream (figure 1a ). specifically, an adapter a t is activated and trained when a new task d t emerges. previous adapters are deactivated, allowing a t to learn in isolation and specialize in task t @xcite . we keep the backbone frozen and train only the adapter a t on the current task t. to simplify the explanation, we omit the fact that each task uses l adapters that match the number of plm layers. adapters offer significant benefits @xcite . their robust modeling capabilities allow them to effectively capture distribution-specific details from each task and the high parameter ef-ficiency enables efficient training of our method whenever a new task emerges. ## experimental setup benchmarks we adopt two cl setups: 1) class-incremental learning (cil) and 2) task-incremental learning (til). both setups aim to incrementally learn new tasks (and thus new classes), but til always has access to task identity, making cil a more challenging and realistic scenario. based on this, we consider three benchmarks. mtl5 (d' @xcite : 5 text classification tasks. wos @xcite : 7 document classification tasks. afrisenti @xcite : 12 (multilingual) sentiment analysis tasks. more details and training orders are in appendix a. baselines we focus our experimentation on comparing peft-based cl methods. lower-bound trains task-specific adapters, summing their outputs during inference. upper-bound trains task-specific adapters, using only the corresponding adapter for inference. progprompt @xcite progressively adds new prompts for new tasks. all prompts are used for inference (only for til). dam @xcite learns task-specific adapters and creates task vectors by averaging training example representations. at inference, it uses similarity scores between the input and task vectors to route (only for cil). epi @xcite learns task-specific prompts and records task distributions. at inference, it selects the nearest task's prompt based on input comparison with all distributions. mocl @xcite progressively adds new prompts and learns a task vector. at inference, it uses task vectors to compute similarity scores to combine the prompts. implementation details we use bert @xcite for mtl5 and wos while afroxlmr (alabi et al., 2022) for afrisenti. to implement our adapters, we adopt lora (hu et al., 2022) due to its efficiency and performance. our memory is similar to that of (d'autume et al., 2019), but we accumulate only 10% of each task's, training dataset a negligible amount when using efficient storage formats. our router network consists of a linear transformation followed by a gumbel-sigmoid @xcite . in cil, routers are trained using all memory examples to route inputs universally across tasks. in til, leveraging task identity, routers are trained with task-specific memory elements. for more details, see appendix b. ## results in this section, we present our results, summarized in results on cil results on til in the wos benchmark, our methods perform competitively but fall behind mocl. we attribute this to the amount of data used to train the routing function. wos consists of very small datasets, resulting in a memory populated with approximately 100 elements per task, which may not be sufficient for router learning. in fact, @xcite have shown that as the number of examples increases, the routing function reaches its optimal performance. we explore this in the next section. ## conclusion we introduced l2r, a method for routing in cl that enables plms to dynamically combine adapters, improving generalization and performance. experiments across benchmarks demonstrate l2r's effectiveness in both til and the challenging cil setting. we also show that larger memory enhances routing function learning and that l2r effectively leverages diverse adapter combinations.