C o LA : Collaborative Low-Rank Adaptation (2025)

Abstract: The scaling law of Large Language Models (LLMs) reveals a power-law relationship, showing diminishing return on performance as model scale increases. While training LLMs from scratch is resource-intensive, fine-tuning a pre-trained model for specific tasks has become a practical alternative. Full fine-tuning (FFT) achieves strong performance; however, it is computationally expensive and inefficient. Parameter-efficient fine-tuning (PEFT) methods, like LoRA, have been proposed to address these challenges by freezing the pre-trained model and adding lightweight task-specific modules. LoRA, in particular, has proven effective, but its application to multi-task scenarios is limited by interference between tasks. Recent approaches, such as Mixture-of-Experts (MOE) and asymmetric LoRA, have aimed to mitigate these issues but still struggle with sample scarcity and noise interference due to their fixed structure. In response, we propose CoLA, a more flexible LoRA architecture with an efficient initialization scheme, which introduces three collaborative strategies to enhance performance by better utilizing the quantitative relationships between matrices A and B . Our experiments demonstrate the effectiveness and robustness of CoLA, outperforming existing PEFT methods, especially in low-sample scenarios. Our data and code are fully publicly available: https://github.com/zyy-2001/CoLA.

## introduction the scaling law @xcite of large language models (llms) describes a power-law relationship between the performance of a deep learning model and its scale (e.g., number of parameters, computation, and data) as the model size increases. as the model scale grows, the rate of performance improvement gradually flattens. despite the impressive under-standing and expressive capabilities of llms, training such a model from scratch is costly, which hinders the application of the scaling law, especially for small companies or institutions. finetuning a single llm for different downstream tasks or knowledge domains has become a common paradigm in various vertical fields @xcite @xcite , and previous studies @xcite indicate that the scaling law also applies to fine-tuning. however, this approach, i.e., full fine-tuning (fft), requires entire pre-trained weights of the llm to be involved in heavy gradient computation, demanding substantial computational resources and energy consumption, thus hindering further exploration of the scaling law. in response, parameter-efficient fine-tuning (peft) methods have been proposed, where the backbone model's parameters are frozen, and only a small number of additional parameters or external modules customized for specific tasks or multi-task learning are modified. common methods include lora @xcite , adapters @xcite @xcite , and other variants @xcite @xcite , which offer a solution for companies and researchers with limited computational resources. as a peft method, lora has gained significant attention due to its simplicity, effectiveness, and generality, leading to many promising works, including exploration of better lora architectures. as shown in figure 1 (a) and (b), unlike full finetuning, which completely unfreezes the pre-trained weight matrix w , lora freezes the pre-trained matrix w and approximates the incremental update ∆w of the pre-trained weights with two trainable low-rank matrices a and b. these matrices are inserted into each layer of the pre-trained model, achieving comparable or even superior performance to full fine-tuning. however, a single lora module projects the features of different tasks into the same dense low-dimensional space, causing interference between tasks and failing to effectively separate the knowledge of different tasks, limiting adaptability in multi-task scenarios. recent research @xcite @xcite has introduced the mixture-of-experts (moe) idea to decouple multi-task information in lora, as shown in figure 1 (c ). in this design, multiple experts are separately designed to learn task-shared and task-specific knowledge, with each expert consisting of a pair of low-rank matrices. this design allows the knowledge of multiple tasks to be effectively learned while maintaining parameter efficiency. meanwhile, some studies @xcite have found that multiple smaller lora heads are more effective than a single lora, as matrix a tends to learn the commonality across all data, while matrix b focuses on the unique aspects of each intrinsic component. this leads to the design of an asymmetric lora architecture, as shown in figure 1 (d ). however, a single matrix a may struggle to capture commonality in limited samples and is prone to interference from noisy data. moreover, these lora variant structures consistently initialize matrices a and b with gaussian noise and zeros, which may result in small or random gradients early in training, slowing the fine-tuning process or causing the model to get stuck in suboptimal local minimum points. as a result, various alternative initialization schemes for lora have been explored @xcite @xcite , e.g., pissa @xcite , which directly updates the model's principal components during fine-tuning through singular value decomposition, thereby accelerating convergence and improving performance. despite significant progress in the model parameter efficiency of different lora variant architectures, the scaling law for model fine-tuning remains limited in real-world scenarios due to the fact that sample labeling is expensive and private data is scarce. in particular, current lora methods, due to their fixed structure, fail to effectively capture the more complex inherent diversity in scarce samples. for example, as shown in figure 3 in sec. 4.2, lora's generalization ability deteriorates sharply when the sample size starts to decrease below 300. therefore, it is crucial to further explore the numerical and collaborative relationships between matrices a and b in lora, freeing it from a monotonous structural design. to address this, we introduce cola, a more flexible lora architecture, and extend the efficient pissa initialization scheme to cola, as shown in figure 2 in sec.3. cola does not enforce strict numerical relationships between matrices a and b. to fully leverage their collaborative potential, we have designed three collaborative strategies with different energy consumption ( computation ): (i) fully collaborative cola ⊺ : different matrices a and b interact and learn from each other, with deep parameter sharing. (ii) random collaborative cola † : a single matrix is randomly selected, without relying on specific combinations, allowing more diverse parameter learning. (iii) heuristic collaborative cola ‡ : a combination of the two structures, integrating the different advantages of lora structures. the details of these three collaborative strategies are described in sec.3.3. we have conducted extensive experiments to validate the effectiveness and robustness of cola, with experiments performed on two recent llama models with different parameter sizes, using fine-tuning data that reflect realistic scenarios. four interesting observations are made, as presented in sec. 4.2. ## cola in this section, we introduce the proposed cola, a flexible lora achitecture as illustrated in figure 2 . we then present the extended pissa initialization scheme applied to cola and provide three collaborative strategies for the matrices a and b. ## experiments in this section, we present the setup and details of the experiments. then, we share our findings and provide concise explanations. ## conclusion in conclusion, this paper introduces cola, a flexible lora architecture designed to address the limitations of current parameter-efficient fine-tuning methods, particularly in scenarios with scarce data. by decoupling the rigid numerical relationship between matrices a and b, cola enables more effective collaboration through three various strategies. through extensive experimentation on multiple llama models, we demonstrate that cola significantly improves generalization and robustness, especially in resource-constrained environments.