Sensitivity- L o RA : Low-Load Sensitivity-Based Fine-Tuning for Large Language Models (2025)

Abstract: Large Language Models (LLMs) have transformed both everyday life and scientific research. However, adapting LLMs from general-purpose models to specialized tasks remains challenging, particularly in resource-constrained environments. Low-Rank Adaptation (LoRA), a prominent method within Parameter-Efficient Fine-Tuning (PEFT), has emerged as a promising approach to LLMs by approximating model weight updates using low-rank decomposition. However, LoRA is limited by its uniform rank ( r ) allocation to each incremental matrix, and existing rank allocation techniques aimed at addressing this issue remain computationally inefficient, complex, and unstable, hindering practical applications. To address these limitations, we propose Sensitivity-LoRA, an efficient fine-tuning method that dynamically allocates ranks to weight matrices based on both their global and local sensitivities. It leverages the second-order derivatives (Hessian Matrix) of the loss function to effectively capture weight sensitivity, enabling optimal rank allocation with minimal computational overhead. Our experimental results have demonstrated robust effectiveness, efficiency and stability of Sensitivity-LoRA across diverse tasks and benchmarks.

## introduction large language models (llms) have become transformative tools across a wide spectrum of tasks and applications @xcite @xcite @xcite @xcite @xcite @xcite @xcite . despite these advancements, fine-tuning remains a critical technique for effectively adapting llms from general-purpose models to specialized applications, especially in resource-constrained environments. however, full-parameter fine-tuning can be prohibitively resource-intensive, requiring significant computational power and gpu capacity. to address this limitation, the research community introduced parameter-efficient fine-tuning (peft) @xcite @xcite @xcite , which aims to balance accuracy and efficiency by selectively updating a subset of model parameters. lora @xcite , a prominent peft method, approximates model weight updates using low-rank decomposition, leveraging the low intrinsic dimension of over-parameterized models @xcite . during training, the update of the weight matrix (∆w ) can be approximated as the product of two smaller matrices b and a, expressed as: to address this limitation, dynamic rank allocation has emerged as a key solution by allocating the rank r to each different module or layer according to its specific requirements. existing methods achieve this through three main approaches: singular value decomposition (svd), single-rank decomposition (srd), and rank sampling. svdbased methods @xcite @xcite decompose matrix ba into pretrained weight (frozen) (trainable) a (trainable) b + dim out dim global metric local metric r rank allocation global metric local metric weight sensitivity via hessian matrix h w @xmath0, h @xmath1. . . . . . . . . . . . h w n1 h w n2 • • • h w nn 3 7 7 7 5 an svd form and selectively truncate the singular values in order to allocate the matrix rank. however, this process is computationally expensive and requires additional memory to store singular values and vectors. srd-based methods @xcite @xcite decompose matrix ba into single-rank components and allocate the ranks by selecting the proper components. however, optimizing single-rank components and the pruning process can increase computational complexity, potentially offsetting efficiency gains. rank sampling-based methods @xcite allocate ranks directly by random sampling. however, the randomness introduced by sampling could increase training instability. in order to design a dynamic rank allocation method that introduces extremely low overhead and ensures stability, we propose sensitivity-lora, which can rapidly allocate rank to the weight matrix based on the sensitivity of the parameters, without incurring a significant computational load. specifically, we utilize the second derivatives of the loss function with respect to the parameters (hessian matrix) to ascertain the sensitivity of each parameter within the weight matrix. to comprehensively evaluate the sensitivity of the parameter matrix, we employ metrics such as the trace of the hessian matrix, topk and effective rank to measure its global and local sensitivities respectively. by integrating various metrics, we determine the rank allocation weights corresponding to the weight matrices to achieve rank allocation. the efficiency, stability, and generality of our approach have been validated through extensive experiments on various tasks, such as sentiment analysis, natural language inference, question answering, and text generation. in summary, the main contributions of our paper are listed as follows: ## related work existing peft approaches can be classified into four main types in terms of memory efficiency, storage efficiency, and inference overhead, as follows: ## methodology in this section, we firstly introduce the concept of weight sensitivity with a formal definition of global and local sensitivity metrics of weight matrices. next, we propose effective allocation strategies to optimize the dynamic rank allocation process based on these sensitivity metrics. the pipeline of our method is presented in figure 1 . ## conclusion in this work, we introduce sensitivity-lora, a method that efficiently allocates ranks to weight matrices based on their sensitivity, without a significant computational burden. sensitivity-lora first performs sensitivity utilization by analyzing both global and local sensitivities. it utilizes the second-order derivatives (hessian matrix) of the loss function to accurately capture parameter sensitivity. next, it optimizes rank allocation by aggregating global and local sensitivities, ensuring a comprehensive and fair evaluation metric. extensive experiments consistently demonstrate the efficiency, effectiveness and stability of our method.