Mixture-of-Subspaces in Low-Rank Adaptation (2024)

Abstract: In this paper, we introduce a subspace-inspired Low-Rank Adaptation (LoRA) method, which is computationally efficient, easy to implement, and readily applicable to large language, multimodal, and diffusion models. Initially, we equivalently decompose the weights of LoRA into two subspaces, and find that simply mixing them can enhance performance. To study such a phenomenon, we revisit it through a fine-grained subspace lens, showing that such modification is equivalent to employing a fixed mixer to fuse the subspaces. To be more flexible, we jointly learn the mixer with the original LoRA weights, and term the method as Mixture-of-Subspaces LoRA (MoSLoRA). MoSLoRA consistently outperforms LoRA on tasks in different modalities, including commonsense reasoning, visual instruction tuning, and subject-driven text-to-image generation, demonstrating its effectiveness and robustness.

## introduction large language models (llms), such as gpt-4 (openai, 2023), llama 3 (ai@meta, 2024), and internlm2 @xcite , have demonstrated remarkable performance across diverse disciplines @xcite . such strong capability is often attributed to the increased scale of training data and model parameters. however, it also brings increasing challenges to adapting these llms for downstream tasks via fully fine-tuning all the parameters. to tackle this issue, parameter-efficient finetuning (peft) has been developed @xcite @xcite to minimize the number of optimized parameters while achieving comparable performance as much as possible. among these methods, lora @xcite has gained increasing popularity due to its simplicity and efficacy, which proposes to update the decompose !" figure 1 : comparison between vanilla lora and proposed moslora. in moslora, we employ learnable weights to mix more subspaces with negligible parameters (i.e., (d 1 + d 2 + r)r vs (d 1 + d 2 )r and d 1 + d 2 ≫ r typically). in this paper, we first define subspaces in lora as the parallel components with smaller rank values, similar to the subspace in multi-head attention (mha) design @xcite . after that, we can decompose the vanilla lora into several subspaces via structural re-parameterization @xcite . figure 2 indicates the process of decomposing into two subspaces. interestingly, we find that simply mixing these two subspaces performs better in the commonsense reasoning tasks. motivated by the observation, we further revisit the two-subspaces-mixing strategy in a more finegrained (@xmath0. in short, such a strategy equals inserting a mixer matrix between a and b, which is a fixed butterfly factor @xcite . meanwhile, vanilla lora can be considered as a special case with a fixed identity matrix being the mixer. therefore, we propose moslora, a simple yet effective method, which employs a learnable mixer to fuse more subspaces and more flexibly. as shown in figure 1 , we adapt the mixer w to fuse all the possible subspaces (i.e., a i b j ). compared to lora, moslora requires negligible extra parameters since d 1 + d 2 ≫ r. similarly to lora, moslora can also be merged into the original weights, and thus introduce no latency during inference. we perform experiments on various downstream tasks, including commonsense reasoning tasks finetuning llama 3 (ai@meta, 2024), visual instruction tuning on llava-1.5 @xcite series models, and subject-driven text-to-image generation on stable diffusion xl (sdxl) model @xcite . experimental results indicate that the proposed moslora consistently outperforms lora and other baselines, demonstrating its effectiveness and robustness. our contributions can be concluded as follows: ## conclusion this work proposes a novel moslora method for parameter-efficient fine-tuning. we first decompose the lora into subspaces and find that simply mixing the half-rank subspaces would lead to better performance. after that, we revisit vanilla lora and two-subspaces-mixing strategy in a more finegrained view (i.e., @xmath1, thus unifying both methods as employing an extra fixed mixer. therefore, we propose moslora, which employs a learnable mixer to fuse more information and more flexibly. the mixer requires negligible extra parameters and computing costs. experimental results on commonsense reasoning tasks, visual instruction tuning tasks, and subject-driven generation tasks demonstrate the effectiveness and robustness of the proposed moslora. for future work, we would consider applying moslora for more tasks. finding a task-specific way to initialize the mixer for faster convergence would be another interesting topic.