Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for L o RA (2025)

Abstract: Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of large language models by decomposing weight updates into low-rank matrices, significantly reducing storage and computational overhead. While effective, standard LoRA lacks mechanisms for uncertainty quantification, leading to overconfident and poorly calibrated models. Bayesian variants of LoRA address this limitation, but at the cost of a significantly increased number of trainable parameters, partially offsetting the original efficiency gains. Additionally, these models are harder to train and may suffer from unstable convergence.In this work, we propose a novel parameter-efficient Bayesian LoRA via subspace inference, demonstrating that effective uncertainty quantification can be achieved in very low-dimensional parameter spaces. The proposed method achieves strong performance with improved calibration and generalization while maintaining computational efficiency. Our empirical findings show that, with the appropriate projection of the weight space: (1) uncertainty can be effectively modeled in a low-dimensional space, and (2) weight covariances exhibit low ranks.

## introduction lora (low-rank adaptation) @xcite reduces computational overhead by decomposing the update weights of pre-trained models into low-rank matrices, enabling efficient adaptation to downstream tasks. minimizing the number of trainable parameters reduces memory and storage requirements, making large-scale model adaptation feasible. reducing computational overhead speeds up training time and makes adaptation possible in resource-constrained settings. lora swag-lora lora-xs our (@xmath0. 4 k 5 k 6 k 2 5 k 6 0 k 7 4 k 0 . 2 m 0 . 3 m 0 . 7 m 0 . 8 m 2 . 4 m 9 . 4 m # parameters 0.050 0.075 ece figure 1 : performance averaged over multiple glue datasets (individual results in fig. 3 ). our method achieves superior calibration (ece) and competitive predictive performance (brier) while maintaining computational efficiency. for example, at r " 8 (ĳ), we reduce ece by half with only 1/10th lora parameters. models (e.g., llms) often become overconfident and poorly calibrated @xcite @xcite , especially when trained on limited data. this hinders their usability for applications where uncertainty-aware decisions are performed. bayesian treatment is then frequently proposed to address overconfidence in neural networks @xcite @xcite . consequently, recently proposed bayesian variants of lora @xcite @xcite address the aforementioned challenges by introducing uncertainty estimation directly into the fine-tuning process. during training, these models continuously adjust both the mean and covariance of fine-tuned parameters to achieve better generalization and uncertainty quantification. learning the posterior covariance matrix is necessary for modeling epistemic uncertainty. however, its size grows quadratically with the number of parameters, which can easily cancel out the benefits of lora, in addition to making learning significantly harder. using low-rank, kronecker-factored, or diagonal-only covariances partially alleviates pretrained weights 𝑊 2 ∈ ℝ m × @xmath1, 𝜎 2 ) @xmath2, 𝜎 2 ) @xmath3, computed from svd remaining layers random @xmath4 we propose a method that learns bayesian posteriors for weights projected onto a low-dimensional manifold, hence maintaining parameter efficiency. the thoughtfully selected projection allows for the effective representation of the covariances between weights through covariances between representations in the lower-dimensional space. in this design, we follow the work of @xcite , who recently proposed a strategy for finding such projections with svd. we prove that they are effective for learning bayesian models as well. operating in such a reduced parameter space significantly improves the feasibility of bayesian inference. we show that correlations between weights can be represented very efficiently -unlike in the original weight space, we can use covariance matrices with ranks as low as k " 2. thanks to the low number of parameters, training is also more stable. finally, the method achieves superior calibration and accuracy at low budgets (e.g., see fig. 1 ). a key contribution of our work lies in introducing bayesian learning within a low-rank projected subspace derived from pre-trained weights. while the individual components of our method, such as svd projections and bayesian inference, are established techniques, their synergistic application to learn bayesian posteriors within a compressed subspace constitutes a meaningful conceptual innovation. our approach enables uncertainty-aware fine-tuning with strong parameter efficiency, yielding improved uncertainty quantification with minimal computational overhead. in the appendix, we supplement the results presented in the paper with a discussion of related work, a detailed overview of the experimental setup, and the exact numeric values for the figures in the main text. the source code is available online foot_0 . bayesian loras obtain the posterior for t∆w l u through the learned posterior for θ " y l ta l y b l u, where l indexes the weight updates (layers). the posterior itself is approximated either using a set of particles or a closed-form distribution. due to its superior performance, we rely on the latter and assume ppθ|dq « n pµ, σq, where µ is the vector of means (of size equal to the number of learned parameters) and σ is the covariance matrix, whose size grows quadratically with the total number of parameters. notably, we aim to model cross-layer interdependencies, requiring covariance estimation also across weights in different layers tlu. this however results in an impractically large number of parameters. consequently, we explore methods to reduce this cost by representing distributions ppt∆w l u|dq differently, e.g., using svd-based projections. in lora-xs @xcite , the adaptation matrices a and b are initialized using the truncated svd of the corresponding pre-trained weight matrices w 0 . this initialization captures the most informative singular components of the original weights. under the assumption that the fine-tuned task is similar to the original task, these projections retain the functional properties also for downstream adaptations. lora-xs then freezes a and b and inserts a small trainable matrix r p r rˆr between them, reducing the number of trainable parameters to r 2 (r 2 ! pn `mq ¨r) per weight matrix. then, the fine-tuning update is: h " xw 0 `x∆w " xw 0 `xarb, where a p r mˆr and b p r rˆn are low-rank matrices obtained from the truncated svd of w 0 , specifically a " u r s r and b " v t r . b-lora-xs, proposed in this paper, leverages the frozen projections a and b for effective and efficient bayesian learning. its core idea is to apply the bayesian treatment in the extremely compressed parameter space given by svd-based projections, making bayesian inference tractable and highly efficient. although this specific idea is novel, it can be related to the framework of subspace bayesian inference. subspace inference (si) @xcite ) was proposed as a remedy for the intractability of full-dimensional posteriors in modern networks. low-dimensional affine manifold, carefully centred on a well-trained solution, already contains a rich family of high-performing weight vectors. performing bayesian integration restricted to that manifold restores calibrated uncertainty without revisiting the entire parameter space. in particular, given a well-trained reference point w p r d and k ! d orthonormal basis vectors-e.g. the leading pca directions of an sgd trajectory -the learning subspace is s :" tw " w `p z | z p r k u, p p r dˆk , where bayesian parameters are the low-dimensional representations θ " tzu. b-lora-xs defines a layer-local manifold that leaves the pretrained backbone untouched. for each frozen weight matrix w 0 l , an svd w 0 l " u l s l v t l is computed once, and the top r singular directions yield fixed projectors a l " u l,r s l,r and b l " v t l,r . note that, whereas si learns p from the training dynamics, b-lora-xs projects onto directions already favored by the pretrained backbone. then, all task-specific variability is captured by these square adapters r l p r rˆr . vectorising and stacking these layer-projections defines w " w `pb z b , p b " blockdiagpb t l b a l q, where w " rvecpw 1 q t , . . . , vecpw l q t , . . .s t , w " rvecpw 0 1 q t , . . . , vecpw 0 l q t , . . .s t , and p b is a block-diagonal matrix with blocks b t l b a l , one for each layer l. thus b-lora-xs explores an affine subspace s b :" tw " w 0 `pb z b |z b p r ř l r 2 u, whose dimension scales with r 2 per adapted layer. in sec. 3, we empirically demonstrate that a l and b l , obtained from the svd of the pre-trained weights, are not only effective for point-wise finetuning but also enable effective uncertainty quantification for t∆w l u through modeling covariances for tr l u. although we never compute it explicitly, the covariance matrix for individual ∆w is expressed as in practice, we simply learn the joint posterior ppθ " y l r l |dq « n pµ, σq (only) for the inner matrices r. the covariance matrix σ captures both inter-layer and intra-layer dependencies, allowing the model to learn complex relationships. at inference, similar to lora, we use samples of r along with the respective projections a and b to obtain h, as realized through samples of ∆w , however without ever computing it explicitly. the parameters µ and σ are learned efficiently using swag (maddox et al., 2019) (though variational inference or the laplace approximation could also be used). after a burn-in phase (a fixed 10 or 25 epochs) of the gradient-based optimization, the algorithm maintains μ -a running average of θ -along with k vectors of differences dlast " θ last ´μ for the last k values of θ, and a running average of θ 2 . based on these averages, we estimate the variances σ2 for individual parameters and approximate the covariance as σ « 1 2 p d ¨d t `diagpσ 2 qq, which constitutes a rank-k approximation to the covariance matrix. we illustrate b-lora-xs method in fig. 2 . our method uses the total of |θ| ¨pk `2q parameters, where |θ| " ř l r 2 . performance analysis: fig. 3 compares accuracy, expected calibration error (ece), and negative log-likelihood (nll) against total parameter count across 4 datasets. our main claim is that b-lora-xs improves overall model performance, with a particular focus on calibration metrics. indeed, figure 3 (middle and right) demonstrates that b-lora-xs consistently yields lower ece and nll compared to standard lora across all parameter scales. regarding accuracy (figure 3 : left), while standard lora shows marginally better results for a few configurations at moderate parameter scales, the majority of configurations show b-lora-xs matching or exceeding the accuracy of standard lora. more importantly, in no setting does standard lora significantly outperform b-lora-xs in terms of calibration, which is a primary focus of our work. bayesian variants, including b-lora-xs and swag-lora, generally outperform their non-bayesian counterparts in ece and nll. however, our model achieves these strong calibration results with 5-15 times fewer parameters than swag-lora. moreover, while swag-lora sometimes performs well, its results vary significantly between runs. in contrast, b-lora-xs exhibits stable and consistent convergence. finally, as results for mrpc and cola suggest, its performance remains robust across different values of k, whereas swag-lora's ece deteriorates significantly at k " 2. 5 and 6 in the appendix. cola 0.05 0.00 acc change all data avg performance 0.00 0.05 ece change 0 1 nll change lora @xmath5.1 0.0 acc change all data avg performance -90% -75% -50% training data size reduction 0.00 0.05 ece change -90% -75% -50% training data size reduction 0 1 nll change lora-swag @xmath6: figure 4 compares the sensitivity of the bayesian lora variants to changes in covariance matrix rank k. markers indicate model sizes (e.g., swag-lora " b-lora-xs). as expected, swag-lora deteriorates proportionally as rank decreases. on the other hand, b-lora-xs maintains its performance across a wide range of k. significant degradation occurs only when off-diagonal covariance values are entirely ignored (i.e., at k " 0). notably, b-lora-xs achieves the best calibration at low ranks, particularly at k " 2 or k " 5. this demonstrates that the svd-based projection effectively disentangles parameters, enabling low-dimensional uncertainty modeling. data size reduction analysis: figure 5 compares how accuracy, ece, and nll degrade when training data is subsampled. all methods predictably lose accuracy as data size decreases, with little difference between the various lora-based approaches. we conclude that bayesian learning does not improve robustness in this case. however, we note variations across datasets in terms of accuracy. for example, in mrpc, the decline is more pronounced, likely due to the dataset smaller size. ## conclusion b-lora-xs addresses the lack of uncertainty quantification in lora fine-tuning while maintaining parameter efficiency. it utilizes truncated svd to project model updates into a lower-dimensional space and leverages the bayesian framework to enhance uncertainty estimation. our method's primary strength lies in its calibration capabilities; it consistently achieves lower expected calibration error and negative loglikelihood compared to standard lora and lora-xs across various parameter scales. while standard lora may exhibit marginally better accuracy in a few specific configurations, b-lora-xs generally matches or exceeds its accuracy in most settings, and critically, always provides superior or equal calibration. compared to the bayesian lora baseline, b-lora-xs matches or surpasses its accuracy and calibration performance while using significantly fewer parameters, exhibiting greater training stability, and relying on simpler, lower-rank covariance representations. a related work peft: as large language models continue to grow, parameter-efficient fine-tuning (peft) has become a popular approach to reducing computational and storage costs. among various methods @xcite @xcite , lora @xcite has emerged as one of the most widely used. building on its success, several approaches have been proposed to enhance different aspects of @xcite @xcite . one such method, lora-xs @xcite , further optimizes parameter efficiency by enabling flexible control over the number of trainable parameters per adaptation module. b-lora-xs reuses the idea of svd-based projections to reduce the parameter space dimensionality.