R iemannian Optimization for L o RA on the Stiefel Manifold (2025)

Abstract: While powerful, large language models (LLMs) present significant fine-tuning challenges due to their size. Parameter-efficient fine-tuning (PEFT) methods like LoRA provide solutions, yet suffer from critical optimizer inefficiencies; notably basis redundancy in LoRA’s B matrix when using AdamW, which fundamentally limits performance. We address this by optimizing the B matrix on the Stiefel manifold, imposing explicit orthogonality constraints that achieve near-perfect orthogonality and full effective rank. This geometric approach dramatically enhances parameter efficiency and representational capacity. Our Stiefel optimizer consistently outperforms AdamW across benchmarks with both LoRA and DoRA, demonstrating that geometric constraints are the key to unlocking LoRA’s full potential for effective LLM fine-tuning.

## introduction large language models (llms) have recently led to significant progress in the field of natural language processing (nlp), achieving near or superhuman performance across diverse tasks @xcite . however, the substantial computational overhead and memory footprint associated with llms, which possess parameters numbering hundreds of billions, cause serious restrictions on their wide adoption and efficient fine-tuning @xcite . to solve these real-world problems and effectively adapt llms for various downstream tasks, parameter-efficient fine-tuning (peft) techniques have emerged, which involve fine-tuning only a minimal subset of the original model weights (lialin et al., 2023; lester et al., 2021; li and liang, * equal contribute † corresponding author ‡ {jyoung.park, mjae.kang, sbae.lee, hgang.lee, swan.kim, jaeho.lee}@opt-ai.kr 2021). notably, low-rank adaptation (lora) @xcite represents the weight update ∇w as the product of two low-rank matrices, @xmath0.e.,∇@xmath1, thereby drastically reducing the total number of trainable parameters. this approach efficiently achieves performance analogous or competitive to full model fine-tuning, establishing lora as one of the most adopted peft methodologies at present. despite the success of lora, a fundamental question arises: are we truly leveraging this constrained resource r, in the most effective way? common lora approaches mostly train the update matrices a and b within a standard euclidean space using conventional gradient descent-based optimization algorithms, for example, adamw, without imposing explicit structural constraints @xcite . while this approach offers straightforward implementation, it may overlook potential inefficiencies stemming from the inherent low-dimensional structure. for instance, during the training process, the column vectors of matrix b, which are the basis directions of the update, may exhibit increased similarity, leading to redundancy. alternatively, the update directions might manifest unstable dynamics. such phenomena can result in an underutilization of the representational capacity afforded by the fixed rank r. this, in turn, can cause slower convergence, suboptimal final performance, or require a higher rank r, and consequently more parameters, to achieve satisfactory performance levels @xcite . however, in other fields of deep learning research, using appropriate geometric structures, especially orthogonality or unitarity constraints, on learnable parameter matrices has been consistently reported to be highly effective in enhancing model performance and training stability. for example, in convolutional neural networks (cnns), enforcing orthogonality constraints on weight matrices has been demonstrated to bolster the orthogonality of feature representations, thereby enriching their expressiveness and consequently improving the model's generalization capabilities and training stability @xcite @xcite . in recurrent neural networks (rnns), the application of unitary matrix constraints, which are orthogonal matrices in complex space, has proven effective in mitigating the vanishing and exploding gradient problems encountered during the learning of long-term dependencies @xcite . these diverse success stories strongly suggest that introducing suitable geometric structural constraints into the model parameter space, aligned with the characteristics of the data or the learning objectives, can exert a positive influence on the model's representational power, learning dynamics, and ultimate performance. this raise an important question: could such geometric constraints be effectively applied to the low-rank update matrices in lora, the cornerstone of peft techniques, to overcome the aforementioned limitations and unlock latent performance capabilities? this research aims to solve this key question. we propose stiefel-lora, a novel finetuning framework that explicitly imposes orthogonality constraints on the update matrix b, a core component of lora. specifically, we constrain the column vectors of matrix b to be orthonormal, which is equivalent to b residing on the stiefel manifold st(d, r) = @xmath2: b ⊤ @xmath3. under this constraint, the lora update ∇@xmath4. we anticipate that the orthogonality constraint imposed in stiefel-lora will maximize the representational efficiency of the lora update, eliminate unnecessary redundancies, and stabilize learning dynamics, thereby eliciting the maximum potential performance of lora under a given parameter budget (rank r). this will ultimately lead to achieving comparable or superior performance with fewer parameters (r), or attaining faster convergence and higher final performance with the same number of parameters. this paper describes the theoretical foundation of the stiefel-lora framework and provides an indepth analysis of how the proposed geometric constraint contributes to improving the inherent limitations of standard lora, particularly concerning parameter efficiency and training stability. furthermore, through extensive experimentation across diverse llms fine-tuning benchmarks, we successfully demonstrate that stiefel-lora consistently achieves superior performance, significantly faster convergence rates, and enhanced parameter efficiency compared to standard lora. in conclusion, this research introduces a novel perspective of geometric optimization to parameter-efficient finetuning techniques, thereby paving new avenues for optimizing the efficiency and performance of llms fine-tuning and is anticipated to contribute 20972 to the advancement of related research. ## methodology this chapter explains the core concepts of the proposed stiefel-lora methodology. the basic concepts of lora, the stiefel manifold, and riemannian optimization are described in appendix c.1. then, algorithm 1 presents the core mechanism and the overall algorithm for efficiently applying the stiefel manifold constraint to lora updates. ## conclusion this study introduced stiefel-lora, a novel optimization approach for lora that leverages geo-metric constraints by optimizing its b matrix on the stiefel manifold. this explicit orthogonality enforcement aimed to enhance representational efficiency and overcome limitations of standard euclidean optimization. extensive experiments across diverse benchmarks and llms scale demonstrated that stiefel-lora consistently outperformed conventional lora trained with adamw. key contributions include the proposal and experimental validation of stiefel-lora, and an analysis of internal metrics (orthogonality, effective rank) revealing the mechanism of improved representation efficiency. these findings highlight the importance of geometric constraints in peft design and suggest stiefel manifold optimization as a potent method to significantly enhance peft performance.