Convolutional L o RA Aggregation for Unseen Tasks Adaptation (2025)

Abstract: Recent studies have increasingly explored the combination of existing LoRA modules for effective adaptation to unseen tasks in data-scarce scenarios. However, current LoRA selection methods typically rely on a few task samples, making it difficult to capture the full scope of task-relevant information. Furthermore, even after selection, a knowledge gap remains between the selected LoRA modules and the target task, which existing coarse-grained LoRA aggregation strategies struggle to bridge. To address these challenges, we propose Selection and Convolution for LoRA aggregation (SC-LoRA), a two-stage framework that first selects appropriate LoRA modules based on parameter clustering and then aggregates them using a convolutional LoRA aggregator. Our LoRA selection strategy ensures comprehensive coverage of task-relevant LoRA modules by leveraging their distance in the parameter space. Building on this, the convolutional LoRA aggregator extracts useful knowledge in a fine-grained manner, seamlessly bridging the gap to the target task. Our experiments demonstrate that SC-LoRA excels in aggregating multiple LoRA modules for effective adaptation to unseen tasks.

## introduction fine-tuning with low-rank adaptation (lora) @xcite has become a prevalent paradigm for adapting large language models (llms) @xcite to downstream tasks. lora freezes the base model and trains lightweight parameter matrices, enabling task-specific knowledge to be encapsulated in a modular structure. therefore, existing lora modules are naturally reusable for adapting to unseen tasks without requiring extensive labeled data. based on this, recent studies have explored how to combine existing lora modules to adapt to un-* corresponding author seen tasks @xcite @xcite . however, existing approaches are still faced with significant challenges in selecting appropriate lora modules and aggregating them. among lora selection methods, random selection @xcite often leads to the inclusion of lora modules that are irrelevant to the new task. @xcite compute the similarity between the training data of each lora module and a few samples of the target task, which are insufficient to capture the full distribution of the new task. as a result, increasing the number of selected lora modules leads to a performance drop, as shown in figure 1 . beyond the top-ranked lora modules that may capture the overall characteristics of the target task, additional ones are selected based on partial task information reflected by a few samples, which may resemble features encoded in irrelevant lora modules and lead to their incorrect selection, resulting in limited coverage of effective knowledge for the target task. crucially, after the lora modules are selected, the combination process still suffers from a gap between the knowledge encapsulated in lora modules and the target task, since they are trained on upstream tasks that inherently differ from the unseen task. most existing methods adopt coarse-grained composition strategies @xcite @xcite that operate at the level of entire parameter matrices or hidden vectors, failing to disentangle useful signals from redundant information within lora modules. meanwhile, these strategies result in a relatively narrow range of representation space due to simplistic combination methods such as linear combination, making them inadequate for handling complex tasks. therefore, if these lora modules are composed in a coarse-grained manner, the knowledge gap may be further exacerbated. to address these challenges, we propose selection and convolution for lora aggregation (sc-lora), a novel framework that first selects appropriate lora modules based on parameter clustering and then integrates their representations in a fine-grained manner through a convolutional lora aggregator. specifically, sc-lora clusters lora modules in the parameter space and identifies taskrelevant clusters. this approach broadens the scope of useful knowledge by exploiting parameter-level correlations among lora modules, thereby overcoming the limited knowledge coverage inherent in sample-based selection methods. the convolutional lora aggregator employs convolutional kernels to scan the semantic feature map, which formed by concatenating the lora representations, enabling fine-grained extraction of task-relevant signals across lora modules while pruning redundant feature. moreover, convolutional operation facilitates deep reconstruction of lora outputs, enhancing representational capacity and promoting tighter alignment with the target task to seamlessly bridge the knowledge gap. in summary, our main contributions are summarized as follows: ## related work the plug-and-play nature of lora modules has inspired extensive research to reuse multiple lora adapters for new tasks @xcite @xcite ) from various perspectives. lora selection. existing methods for selecting lora modules adopt several strategies. lorahub @xcite uses random sampling, while others @xcite rely on task-specific samples to assess module adaptability to new tasks. evaluation metrics include target loss @xcite , semantic similarity to source training data @xcite , and matching probabilities from a trained embedding model @xcite . however, in few-shot settings, these sample-based lora selection strategies depend on scarce task samples that fail to reflect the full task distribution, leading to inadequate knowledge coverage. ## preliminaries lora lora @xcite is a parameterefficient fine-tuning technique designed to adapt llms to downstream tasks. specifically, given the linear transformation matrix @xmath0, lora introduces a lowrank decomposition to approximate the parameter update to w , which is formulated as: problem statement given a set of tasks @xmath1, t 2 , ..., t n } and respective training sets @xmath2, d 2 , ..., d n }, the trained lora modules l are obtained by fine-tuning the language model m with lora on each task t i using its corresponding training set d i , for @xmath3, ..., n. we consider an unseen task t / ∈ t , accompanied by only a few labeled examples t of this task. given the inherent gap between training distributions d and the target task t , our goal is to explore how to use trained lora modules l, in conjunction with the few-shot examples, to bridge the knowledge gap. ## sc-lora in this section, we elaborate the methodological details of sc-lora, which selects task-relevant lora modules by clustering them in the parameter space and employs cnn-based scanning to capture transferable knowledge in a fine-grained manner, as shown in figure 2 . ## experiments this section presents an overview of the sc-lora experiments, including the setup, implementation details, and analysis of its effectiveness. base models and benchmark. our experiments are conducted using the publicly available lora modules released by lorahub and loraretriever. to ensure compatibility and preserve the effectiveness of the lora modules, we reuse them to the same backbone models they were originally trained with: flan-t5-large (chung et al., 2024) for lo-rahub, and llama-2-7b, 13b @xcite for loraretriever. we follow the same benchmark and evaluation protocol as used in lorahub @xcite , specifically evaluating on big-bench hard (bbh) @xcite , a challenging benchmark for language model that encompasses 27 tasks from multiple domains. we report exact match (em) as the evaluation metric. ## conclusion in this paper, we propose sc-lora, a novel framework that enables leveraging the knowledge of trained lora modules to tackle unseen tasks under data-scarce conditions. sc-lora first performs parameter-based clustering and selects lora modules at the cluster level, overcoming the limited coverage of knowledge in sample-based selection. it then employs a convolutional aggregator to integrate the representations of the selected lora modules in a fine-grained manner, seamlessly bridging the knowledge gap between existing knowledge and unseen tasks. experiments demonstrate that sc-lora consistently enables robust knowledge transfer across diverse tasks.