S u L o RA : Subspace Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (2025)

Abstract: As the scale of large language models (LLMs) grows and natural language tasks become increasingly diverse, Parameter-Efficient Fine-Tuning (PEFT) has become the standard paradigm for fine-tuning LLMs. Among PEFT methods, LoRA is widely adopted for not introducing additional inference overhead. However, existing LoRA’s shared parameter space paradigm introduces parameter interference, leading to a gap in generalization performance for specific tasks compared to full fine-tuning. To address this issue, we propose a parameter-separated low-rank adapter, called Subspace Low-Rank Adaptation (SuLoRA). The core idea of SuLoRA is to account for task differences by decomposing LoRA’s parameter matrix into multiple independent subspaces and assigning them differentially to distinct tasks. This prevents interference across tasks and enhances the effectiveness of low-rank adaptation. Additionally, SuLoRA achieves higher rank expansion by freezing the A matrix, further improving generalization capability. We conduct extensive experiments on various NLP tasks, demonstrating that SuLoRA significantly outperforms LoRA in trainable parameter efficiency and overall model performance. Furthermore, we validate SuLoRA’s effectiveness in domain generalization and multi-modal tasks, showcasing its strong generalization ability.

## introduction large language models (llms), such as gpt (radford, 2018), llama @xcite , and deepseek @xcite , have emerged as the standard in natural language processing (nlp) owing to their exceptional comprehension capabilities @xcite . however, as the scale of these models and the complexity of tasks continue to grow, full fine-tuning (ft) has become impractical due to its substantial computational cost. in * corresponding author. response, parameter-efficient fine-tuning (peft) methods @xcite , including low-rank adaptation (lora) @xcite , adapters @xcite , and prompt-based approaches @xcite have been successfully applied across a diverse range of nlp tasks. among these methods, lora @xcite ) is widely employed because it does not incur additional inference overhead. as illustrated in fig. 1 (a) (left), lora approximates the update of pretrained weights (w) by decomposing them into a low-rank matrix (a) and a high-rank matrix (b). by applying low-rank decomposition to the incremental updates of the pre-trained weights, lora significantly reduces the memory overhead associated with fine-tuning. however, compared to full fine-tuning, low-rank approximations may result in a considerable generalization performance gap @xcite . consequently, enhancing the performance of low-rank approximations while minimizing computational overhead has emerged as a critical challenge. recently, asylora @xcite proposed a method that requires no additional parameters by leveraging the inherent asymmetry of lora matrices. specifically, the a matrix captures fundamental features, while the b matrix utilizes these features to generate task-specific outputs. consequently, fine-tuning b is inherently more effective than fine-tuning a. this method enhances the effective rank by freezing the a matrix during finetuning. however, despite successfully expanding the rank, it yields minimal performance improvement. this motivates a deeper investigation into the functional role of the b matrix in lora. in fig. 1 (b), we partition the parameter space of the lora b matrix and visualize the activations of these parameter groups (g) across different classes. it can be observed that distinct classes exhibit significant differences in their parameter preferences; for instance, the parameter groups predominantly activated by 'c-1' and 'c-2' are nearly mutually exclusive. however, traditional lora does not account for such inter-task preference differences, as it employs a shared parameter space paradigm. this approach may lead to interference among the parameter spaces of different classes during finetuning @xcite , ultimately affecting the performance of lora. how can we effectively distinguish the parameter preferences across tasks to mitigate such interference and enhance the model's generalization capability? given our objective to segregate the parameter space during fine-tuning to accommodate the distinct preferences of different tasks (e.g., varying categories or datasets) and mitigate inter-task interference, we propose a low-rank adapter that partitions the parameter space, termed subspace low-rank adaptation (sulora). as shown in fig. 1 (a), we first freeze matrix a, which is insensitive to fine-tuning performance, to improve its effective rank and thereby expand the trainable parameter space of matrix b @xcite . in contrast to traditional lora, which shares a single low-rank adaptation matrix across all task instances, sulora partitions the parameter space of matrix b into n subspaces (@xmath0. during training, sulora dynamically selects r subspaces based on the parameter preferences of each task, constructing a dedicated matrix b for each task to mitigate inter-task interference. experimental results demonstrate that this strategy yields significant performance improvements. to validate the effectiveness and generalization capability of sulora, we conducted extensive experiments across various nlp tasks and extended our evaluation to domain generalization and multimodal tasks. experimental results demonstrate that sulora consistently outperforms lora while utilizing fewer parameters and a lower rank, and it even surpasses full-parameter fine-tuning across multiple tasks. in summary, our method makes the following main contributions: notably, asylora @xcite has revealed an asymmetry between the roles of the a and b matrices in lora. the study demonstrates that the performance of a randomly initialized a matrix is comparable to that of a fine-tuned one. consequently, a strategy was proposed to extend the rank by freezing the a matrix and exclusively fine-tuning the b matrix. however, this work did not delve deeply into the parameter space of the b matrix, resulting in limited performance improvements. furthermore, melora @xcite achieves efficient inference by integrating multiple sub-loras. however, these sub-loras are shared across different tasks, which is fundamentally distinct from our approach. ## analysis 6.1 analysis of different design strategies for sulora in this section, we will delve into the impact of different design strategies. we conducted experiments on various benchmarks, and the results on glue and instructeval are shown in therefore, there is no need to apply the subspace strategy to a. additionally, we found that training b is more effective than not training it, further supporting the idea that the b matrix in lora is indeed associated with task-specific capabilities. thus, implementing parameter space separation for lora's b matrix to enhance model performance is logically sound. ## conclusion in this paper, we propose a novel parameterefficient fine-tuning method, sulora. this