G e L o RA : Geometric Adaptive Ranks For Efficient L o RA Fine-tuning (2025)

Abstract: Fine-tuning large language models (LLMs) is computationally expensive because it requires updating all model parameters. Low-Rank Adaptation (LoRA) reduces this cost by modifying a subset of weights, but selecting the appropriate rank introduces a trade-off: lower ranks improve efficiency at the expense of expressivity, while higher ranks enhance performance but increase computational burden. Existing adaptive LoRA methods lack a theoretical foundation to guide this trade-off optimally. We propose Geometric Low-Rank Adaptation (GeLoRA), a principled approach that estimates the intrinsic dimensionality of hidden data representations to adaptively select LoRA ranks. We show theoretically and empirically that the intrinsic dimension serves as a lower bound for the optimal rank of LoRA matrices, enabling a balance between efficiency and expressivity. Extensive experiments on GLUE, SQuAD (with DeBERTa), and MT-Bench (with LLaMA) demonstrate that GeLoRA consistently outperforms recent adaptive LoRA methods by up to +1.0%, while simultaneously reducing computational time by 13.5% to 64.2%, depending on the baseline, under the same parameter budget.

## introduction llms excel in natural language processing (nlp) tasks but require fine-tuning for effective personalization. however, fine-tuning the entire model is computationally expensive in terms of time and memory. parameter-efficient fine-tuning (peft) offers a solution by adjusting only a subset of the model parameters @xcite . among peft methods, lora decomposes weight updates into low-rank components, significantly reducing computational costs while achieving performance comparable to full fine-tuning @xcite . lora and its variants assume that pre-trained models exhibit low "intrinsic dimensionality" @xcite , implying that weight updates should similarly be lowrank. however, a key challenge lies in determining the optimal rank values, which involves balancing expressivity and computational efficiency. expressivity refers to the model's ability to capture complex patterns in the data, while computational efficiency pertains to the speed and resource requirements for fine-tuning. lower ranks reduce expressivity but enhance memory efficiency and speed, while higher ranks increase expressivity at the cost of more resources and computation time. the trade-off is evident: lower ranks reduce expressivity but enhance memory efficiency and computational speed, whereas higher ranks increase expressivity at the cost of greater memory usage, longer computation times, and most likely more data to learn weights reliably. typically, ranks are set uniformly across all layers, with practitioners relying on trial-and-error to achieve a balance between expressivity and efficiency. this is time-consuming and may not always yield optimal results. recent studies @xcite show that hidden representations of transformer models exhibit low intrinsic dimensionality, suggesting potential connections between the manifold of data representations and the manifold of model parameters. this raises the question: is there a connection between the manifold of data representations and the manifold of model parameters? we theoretically examine the relationship between the dimensionality of data representations and the ranks of weight updates, deriving a lower bound for the optimal rank as a function of the intrinsic dimensionalities of each transformer block's input and output representations. these intrinsic dimensionalities are specific to each transformer block and depend on the underlying model and dataset. although determining the exact intrinsic dimension of each hidden representation is impractical, reliable estimates can be obtained using ex-isting methods. in this work, we adopt the two nearest neighbors (twonn) method @xcite , a robust and widely used estimator for estimating the intrinsic dimension of data representations in neural networks. @xcite @xcite @xcite . building on this, we develop a principled approach to improve lora methods. specifically, we introduce geometric low-rank adaptation (gelora), a method that dynamically adjusts the rank of weight updates for each transformer block as a function of the intrinsic dimensionalities of its input and output data manifold representations, achieving a more efficient and expressive tradeoff between model performance and computational cost. our work makes three key contributions: ## related work in this section, we review peft methods for llms, examine their limitations and recent developments in lora techniques. we then explore the concept of intrinsic dimensionality, a key theoretical concept that we leverage in our proposed methodology. ## methodology figure 2 provides a schematic overview of the gelora methodology, which begins by computing the hidden representations of the data using a language model and determining the intrinsic dimensions of these hidden representations. for each transformer block i, the intrinsic dimension of the data manifold at the input and d i represent the intrinsic dimension at the output. the minimum rank d θ i-1 required for each block is determined based on the condition we derived in corollary 3.4: to estimate the intrinsic dimension d of the hidden representations, we employ the twonn method @xcite . more details are provided in appendix a.1. in high-dimensional settings, the twonn method tends to provide a conservative estimate, often serving as a lower bound for the true intrinsic dimension. to illustrate this, we conducted experiments on established benchmark datasets comparing the estimated values to the ground truth. the results of these experiments are summarized in appendix c.2. to mitigate the risk of underestimating the intrinsic dimension, sometimes resulting in an inaccurate value of zero rank, we add a small offset of 1 to each rank lower bound. moreover, we compute a lower bound on the rank of each transformer block, rather than for the individual key, query, value and output projection matrices within the block. since the distribution of effective tunable parameters across these matrices is unknown, we adopt a uniform allocation strategy: the rank of each matrix in the block is set equal to the computed lower bound (i.e., the rank of the transformer block). although this approach implicitly assumes equal contribution from each component, it provides a tractable approximation in the absence of a fine-grained estimation method. ## conclusion in this work, we introduced gelora, a geometryaware method for adaptive low-rank fine-tuning. by leveraging intrinsic dimensionality estimates of hidden representations, gelora assigns principled lora ranks per layer, balancing expressivity and efficiency. our theoretical framework connects data representation geometry to parameter optimization needs, and our empirical results show consistent gains over existing peft methods on glue, squad, and mt-bench, with significantly fewer trainable parameters.