R - L o RA : Randomized Multi-Head L o RA for Efficient Multi-task Learning (2025)

Abstract: Fine-tuning large language models (LLMs) is computationally expensive, and Low-Rank Adaptation (LoRA) provides a cost-effective solution by approximating weight updates through low-rank matrices. In real-world scenarios, LLMs are fine-tuned on data from multiple domains to perform tasks across various fields, embodying multi-task learning (MTL). LoRA often underperforms in such complex scenarios. To enhance LoRA’s capability in multi-task learning, we propose R-LoRA, which incorporates Multi-Head Randomization. Multi-Head Randomization diversifies the head matrices through Multi-Head Dropout and Multi-Head Random Initialization, enabling more efficient learning of task-specific features while maintaining shared knowledge representation. Our approach not only improves performance in MTL but also reduces GPU memory usage and training time. Experiments show that R-LoRA’s gains stem from increased diversity in the head matrices, demonstrating its effectiveness for multi-task learning. The code is open-sourced.

## introduction in recent years, large language models (llms) have manifested unprecedentedly superior performance in various natural language processing (nlp) tasks @xcite @xcite . due to their impressive capabilities in language understanding and generation, llms have gained extensive interest from both academia and industry. despite their high generalizability, llms still require fine-tuning for specific domains or updating the knowledge base @xcite . supervised fine-tuning (sft) is crucial for aligning large language models (llms) with human * corresponding author instructions, which trains the model with a small yet high-quality set of labeled data @xcite . the vast number of parameters in llms poses significant challenges regarding computational efficiency and memory consumption during full fine-tuning (ft), which updates all parameters. to address the issue of hardware requirements for llm adaptation, a solution called parameter-efficient fine-tuning (peft) has been proposed @xcite . peft methods reduce vram usage of cached optimizer states by only optimizing a fraction of model parameters while keeping the rest frozen. various peft methods have been widely studied @xcite @xcite @xcite . among these methods, lora has emerged as the mainstream alternative to full parameter fine-tuning. instead of updating the original parameter matrix directly, lora approximates the updated parameters using the product of two smaller matrices. during inference, the output obtained from the original parameter matrix is combined with the output from the updated parameter matrices. however, lora does not perform well in multi-task scenarios, particularly in dealing with complex datasets. recent lora variants have improved multi-task learning by employing multiple lora adapters, including multi-lora @xcite , lora moe @xcite , mixlora @xcite , and moelora @xcite . we refer to this extended framework as the multi-adapter lora architecture, which consists of multiple down-projection matrices (a) and their corresponding head matrices (b), enabling task-specific adaptation through diverse parameter sets. notably, lora-moe and moelora further enhance this architecture by introducing a mixture of experts (moe) mechanism to aggregate adapter outputs. @xcite and @xcite observes that in the multi-adapter lora architecture, the parameters of the down-projection matrices a are relatively consistent, while the differences between the head matrices b are more pronounced, which aids in capturing task-specific knowledge. to leverage this property, ma-lora @xcite and hydralora @xcite are proposed to feature an asymmetric architecture with one shared down-projection matrix a and multiple task-specific head matrices b. additionally, hydralora also employs an moe mechanism to aggregate the outputs of the head matrices. this design achieves a good balance between training performance and parameter efficiency. in this work, we explicitly define asymmetric architecture as a multi-head structure, and introduce multi-head randomization to improve llms' performance on multi-task learning. the mathematical formalization of the multi-head structure is detailed in section 2.2. figure 1 illustrates the differences among the aforementioned structures. however, in the multi-head architecture, the parameter similarity among head matrices remains high, hindering task-specific knowledge learning. this is due to the zero initialization of head matrices b, leading to similar update directions. to address this limitation, r-lora employs multi-head randomization, combining random initialization with multi-head dropout. this approach diversifies both the starting points and inputs of the head matrices, enabling more effective task-specific learning by breaking initial symmetry and promoting distinct optimization trajectories. our work makes the following key contributions: ## motivation in this section, we analyze the parameter similarity between different head matrices in the multi-head lora architecture. to achieve our objectives, we focus on hydralora @xcite and use cosine similarity and the t-sne method to observe the parameters of the head matrices. we fine-tune qwen2.5-3b (qwen team, 2024) with hydralora @xcite on five different tasks. the details of the dataset can be referred to appendix c.1. first, we flatten the head matrices into vectors and then calculate the cosine similarity between the vectors to obtain a similarity matrix. the average value of the matrix is regarded as the similarity of the head matrix corresponding to the parameter matrix. additionally, we perform t-sne analysis @xcite on all the head matrices in figure 6 of appendix b. as shown in figure 2 , the average similarity between different head matrices still reaches around 80%. with such a high similarity, the knowledge learned between different head matrices is also quite similar, which hinders the learning of taskspecific knowledge. to the best of our knowledge, this is due to the zero initialization of the head matrices. tuning a pretrained llm essentially becomes optimizing in a much smaller parameter space around the local optimum of pretrained models. after receiving the outputs from the shared down-projection matrix a, the outputs of the head matrices are highly similar in the early stages of training, leading to highly similar update directions during gradient updates. research question: is there a simple yet effective approach to differentiate head matrices such that they capture distinct task-specific knowledge for efficient multi-task learning? ## method in this work, we propose r-lora, which leverages multi-head randomization to assist the model in learning distinct knowledge. multi-head randomization consists of two components: multi-head dropout and random initialization. an overview of r-lora is illustrated in figure 3 research objective: to exploit randomization to differentiate the head matrices, thereby facilitating the optimization of their parameters to distinct regions and enhancing the diversity among the head matrices. ## experiments in this section, we validate the effectiveness of r-lora across various models and settings. specifically, we evaluate r-lora's multi-task adaptability using qwen2.5 @xcite in setting 1 and its multi-task generalization capability using llama-2 @xcite in setting 2. model sizes range from 3b to 13b. an extensive ablation study further demonstrates the effectiveness of multi-head randomization in r-lora. ## discussion in this section, we discuss the mechanisms behind r-lora's multi-head randomization for enhanced multi-task learning. as in the motivation section, traditional multi-head lora suffers from high similarity among head matrices due to zeroinitialization, confining heads to symmetric states and limiting exploration of sparse, critical subspaces (e.g., for rare syntactic relationships), leading to overlapping subspaces and poor task-specific adaptation. tokens' semantic logics (semantic, syntactic, contextual) exist in multiple subspaces of highdimensional embeddings. the multi-head mechanism decomposes these logics into distinct subspaces for independent processing, and r-lora follows this via two innovations: multi-head dropout: enables heterogeneous feature learning to capture complementary embedding space aspects. multi-head random initialization: avoids head convergence to overlapping subspaces by decoupling trajectories. knowledge diversification, is more important in knowledge-intensive tasks, specifically commonsense reasoning (task 5) and closed-book qa (task 8). on task 5 (commonsense reasoning), removing mi caused a larger performance drop (1.66 vs 1.18), as it breaks symmetry to support wider knowledge learning. md, key for capturing semantic relationships, is more vital in tasks requiring nuanced semantic understanding, such as natural language inference (task 3). on task 3 (natural language inference), removing md led to a greater performance loss (1.79 vs 0.67), as it provides varied inputs to capture diverse semantic patterns. as shown in figure 4 , r-lora effectively reduces the similarity among head matrices, promoting diverse feature learning across tasks. empirical results in ## conclusion in this work, we first analyze the multi-head structure of lora, revealing excessive similarity among head matrices that limits task-specific learning. to address this, r-lora introduces multi-head randomization, a simple yet effective approach that differentiates head matrices, enabling the model to learn diverse knowledge across tasks. this innovation enhances both performance and efficiency, reducing gpu memory usage and training time. extensive experiments validate r-lora's superiority. the performance gains stem primarily from increased diversity in the parameter spaces of the head matrices, confirming the effectiveness of r-lora for efficient multi-task learning.