M i L o RA : Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning (2025)

Abstract: Efficient finetuning of large language models (LLMs) aims to adapt the LLMs with reduced computational and memory costs. Previous LoRA-based approaches initialize the low-rank matrices with Gaussian distribution and zero values while keeping the original weight matrices frozen. However, the trainable model parameters optimized in an unguided subspace might interfere with the well-learned subspace of the pretrained weight matrices. In this paper, we propose MiLoRA, a simple yet effective LLM finetuning approach that only updates the minor singular components of the weight matrix while keeping the principal singular components frozen. It is observed that the minor matrix corresponds to the noisy or long-tail information, while the principal matrix contains important knowledge. The MiLoRA initializes the low-rank matrices within a subspace that is orthogonal to the principal matrix, thus the pretrained knowledge is expected to be well preserved. During finetuning, MiLoRA makes the most use of the less-optimized subspace for learning the labeled dataset. Extensive experiments on commonsense reasoning, math reasoning, instruction following and visual instruction following benchmarks present the superior performance of our method.

## introduction large language models @xcite @xcite have demonstrated superior performance on various tasks @xcite , such as math reasoning @xcite and question answering @xcite . these models are pretrained with the next token prediction task @xcite on large web-scale data, then finetuned with instruction data as well as human preference data @xcite @xcite for different downstream tasks. fully finetuning is commonly employed to unlock the complete potential of llms, however, optimizing all model parameters necessitates substantial and restrictive computational resources @xcite , which hampers the utilization of llms across diverse scenarios. parameter-efficient finetuning @xcite @xcite @xcite aims at reducing the computational and gpu memory cost for finetuning of pretrained models. low-rank adaptation @xcite is one of the most widely used peft methods for llm finetuning. it assumes the change of linear model weights to be low-rank @xcite . for each selected weight matrix, it only updates two low-rank matrices while keeping the pretrained weight frozen. during inference, the low-rank matrices are merged into the pretrained linear weights, thus no additional computational or memory cost is introduced. recently, researchers have explored different lora-based variants for efficient llm finetuning @xcite @xcite @xcite . however, most existing lora-based methods randomly initialize the low-rank matrices and optimize the trainable parameters in an unguided subspace. we suspect this strategy may override important pretrained features, thus degrading the performance of low-rank adaptation methods @xcite . in this paper, we propose minor singular component based low rank adaptation (milora) for efficient llm finetuning. milora has a similar framework as lora but employs a different initialization schedule. specifically, a weight matrix w is decomposed with the singular value decomposition (svd) algorithm. based on the magnitude of the singular values, we divide w into two components: the principal matrix w p corresponding to large singular values and the minor matrix w m corresponding to small singular values. we argue that the principal matrix captures the essence of the pretrained knowledge, whereas the minor matrix is suboptimal with noisy or long-tail information. it is supported by previous works @xcite @xcite that principal low-rank approximation can achieve comparable or even better performance to full parameters. motivated by these observations, we freeze the principal matrix w p and adapt the minor singular components during finetuning. the low-rank matrices a and b in lora framework are initialized with the minor matrix w m . since the trainable lowrank matrices are initialized in a subspace orthogonal to the principal matrix, milora is expected to effectively learn from finetuning tasks while better preserving and utilizing the pretrained knowledge. to maintain the capability of the pretrained model at the start of finetuning, vanilla lora explicitly initializes b with zeros. in contrast, milora naturally satisfies this requirement, as the pretrained weight matrix w equals the frozen principal part w p plus the low-rank part w @xmath0. we conduct extensive experiments on commonsense reasoning, math reasoning, instruction following and visual instruction following benchmarks. the experimental results show that milora consistently outperforms lora without sacrificing training or inference efficiency, such as commonsense reasoning (+1.6/+1.1 on llama2-7b/llama3-8b), math reasoning (+2.0 on llama2-7b), instruction following (+2.9 on llama2-7b) and visual instruction following (+1.4 on llava1.5-7b). 1 2 preliminaries singular value decomposition given a matrix @xmath1, its singular value decomposition is denoted as @xmath2, where the columns of u are the left singular vectors, and the columns of v are the right singular vectors. the σ ∈ r m×n is a diagonal matrix containing the singular values of w in descending order. without loss of generality, we suppose @xmath3. the ## methodology the proposed milora is a simple yet effective peft approach that selects the minor singular components of the weight matrices for optimization. as observed in laser @xcite , the minor singular components of weight matrices contain noisy or long-tail information, while the principal singular components are responsible for important features across tasks. therefore, milora is designed to effectively learn from the finetuning dataset by adapting minor singular components, while maintaining the knowledge encoded in the pretrained model. on the contrary, vanilla lora fails to constrain the optimization subspace as its low-rank matrices are randomly initialized. specifically, suppose @xmath4, milora splits each linear weight matrix into two matrices according to the corresponding singular values: principal matrix w p and minor matrix w m (see figure 1 ): the decomposition can also be reformulated in a matrix form. the u matrix in the svd of w can be reformulated as @xmath5, u m ], where during the finetuning process, instead of freezing the entire pretrained weight matrix, we just keep the principal singular components w p fixed to preserve the pretrained knowledge. the minor matrix w m is used to initialize low-rank matrices a m and b m (see figure 1 ): this strategy has two benefits: 1) it encourages the model to learn in the less-optimized subspace spanned by the minor singular vectors, thus reducing the interference with the well-learned pretrained knowledge encoded in the principal singular components. 2) unlike vanilla lora which requires tuning the scaling factor α and the initialization hyperparameters, ours does not require any hyperparameter except the rank r. one potential variant of milora is to impose strict constraints on the orthogonality between w m and w p during fine-tuning. however, we adopt a softer approach by guiding the optimization direction solely through initialization, resulting in a method that is simpler, more training-efficient, and more flexible. our experiment results show that our scheme works quite well in balancing learning from the finetuning dataset and preserving pretrained knowledge. ## experiments to evaluate milora against other pefts, we conduct experiments on both nlp and multimodal tasks covering 18 datasets. we benchmark against lora @xcite and pissa @xcite • lora @xcite reparameterizes the weight update ∆w with two trainable low-rank matrices a and b, while freezing the pretrained weight w . they use a random gaussian initialization for a and zero for b, so ∆@xmath6. given that the principal components capture the essence of a matrix, pissa is expected to better approximate full finetuning by changing the essential parts while freezing the "noisy" parts. ## understanding milora in this section, we conduct experiments to further understand milora. by default, we use the finetuned llama2-7b model on math reasoning from our main experiments for analyses. ## related work parameter-efficient finetuning of llms. after pretraining, llms are finetuned with instruction data or human preference data to adapt to different downstream tasks @xcite @xcite . parameter-efficient finetuning (lialin et al., 2023, peft) explores effective approaches to reduce the computational resources required during finetuning while managing to match the performance of full finetuning. previous peft methods can be grouped into three lines: adapter-, lora-and prompt-based methods. adapter-based methods insert additional trainable feedforward submodules into each transformer layer @xcite . however, they introduce additional computational costs during inference. lora-based methods @xcite @xcite model the changes of selected linear weights as low-rank matrices. during finetuning, given a linear matrix, only two low-rank matrices are optimized while the pretrained weight matrix is frozen. prompt-based methods @xcite @xcite add additional soft prompts to the input tokens. during training, only soft prompts are updated while the pretrained model parameters are fixed. among these peft methods, the lora-based approaches are widely used for llm fine-tuning because they are easy to implement and do not introduce computational overhead during inference. lora and its variants lora @xcite reparameterizes the weight update with two trainable low-rank matrices, while freezing the pretrained weights. with this lightweight decomposition, lora reduces storage and task-switching overhead by sharing the pretrained models across multiple tasks. since then, researchers have explored and proposed different lora variants for peft. adalora @xcite and alora @xcite , adaptively determine the rank of lora module in each weight matrix according to the importance score. the rslora @xcite modifies the lora with the appropriate scaling factor to improve the performance of large ranks. the dora @xcite ) method decomposes the pretrained weight into the magnitude and directional components, then finetunes both for better performance. concurrent with our work, pissa @xcite and lora-ga @xcite propose to better approximate full finetuning by only updating the principal singular components or aligning the gradients of low-rank updates to that of full finetuning. ## conclusion in this paper, we introduce milora, a simple yet effective low-rank adaption method for llms. milora effectively learns on finetuning tasks while better preserving the pretrained knowledge by adapting the minor singular components of pretrained weight matrices. we investigate the effectiveness of milora on a wide range of llm evaluation benchmarks, including commonsense reasoning, math reasoning and instruction-following, and vision-language model evaluation benchmarks. experiment results demonstrate that milora consistently outperforms lora and pissa without sacrificing training or inference efficiency. we hope that our work will inspire future research on parameter-efficient finetuning of llms.