L o RMA : Low-Rank Multiplicative Adaptation for LLM s (2025)

Abstract: Large Language Models have shown remarkable capabilities in the NLP domain. Their effectiveness can mainly be attributed to their ability to adapt to an array of downstream tasks. However, generally, full fine-tuning is a computationally expensive job. To mitigate this, many techniques have been developed that prime efficiency, a prominent one being Low-Rank Adaptation (LoRA). However, LoRA and its variants employ re-parametrized additive updates. In this paper, we propose Low-Rank Multiplicative Adaptation (LoRMA), which shifts the paradigm of additive updates to a richer space of matrix multiplicative transformations. We tackle challenges such as computational complexity and rank bottleneck of matrix multiplication by effectively re-ordering operations and introducing rank inflation strategies. We conduct extensive experiments to demonstrate the effectiveness of our approach in terms of various evaluation metrics.

## introduction large language models (llms) have demonstrated strong performance across various nlp benchmarks @xcite . though llms have shown impressive generalization capabilities (for example, via in-context learning @xcite ), sometimes these tend to have lower performance on some niche or low-resource tasks, thus requiring task-specific fine-tuning. llms usage follows a pre-train and fine-tune paradigm @xcite , where the model is trained on a massive amount of text in an unsupervised fashion, and subsequently, the model is fine-tuned for some specific tasks/domains in a supervised setting. given the size of these models (order of billions of parameters), it may not always be feasible to fine-tune the entire model due to high computational costs. in recent years, a new class of techniques (referred * equal contribution to as peft (parameter efficient fine tuning)) has been proposed to address large computational costs associated with fine-tuning. various peft techniques previously have been devised @xcite ; however, they often introduce trade-offs such as lack of parallelism, increased inference latency (e.g., adaptors @xcite ), or restricted sequence lengths @xcite . consequently, re-parametrizationbased techniques such as low-rank adaptation (lora) @xcite based fine-tuning methods have gained popularity. typically, during finetuning, the weights (in the form of the weight matrix, e.g., query/key/value matrix) of llms are updated using additive update rule, i.e., w 0 + ∆w, where ∆w is the update in the weights obtained due to fine-tuning. the main idea behind lora is to approximate the update matrix ∆@xmath0, where @xmath1, k), α r is a scaling factor, leading to @xmath2. it is based on the study that the additional information required for taskspecific updates has a smaller intrinsic rank and lies on a much smaller manifold compared to the entire space of d × k matrices @xcite . the current lora-based approaches @xcite have employed additive transformations, where the low-rank update matrix can be added to the original weight matrix during inference. however, a similar transformation could also be achieved via multiplicative updates. for example, consider a weight vector w (fig. 1 ) and we would like to transform it to vector w, this could be accomplished via the addition of a vector v, or it could also be done by rotating w by angle θ (done via rotation matrix r θ ) and subsequently by scaling it by scalar α. inspired by this, we propose low-rank multiplicative adaptation (lorma) for efficiently finetuning llms on new tasks. lorma applies lowrank multiplicative update to a weight matrix, i.e., @xmath3, where α r is a scalar and @xmath4, k). however, this simple multiplicative update faces two new challenges: an increase in computational complexity due to additional matrix multiplication operations and a restriction on the maximum rank of w due to the property: r(ab) ≤ min(r(a), r(b)), where r(•) denotes the rank of a matrix. we employ appropriate ordering of matrix multiplication to address the issue of computational complexity ( §3). additionally, to counteract the issue of rank inhibition caused by matrix multiplication, we introduce rank inflation strategies and demonstrate their effectiveness (fig. 2 ). on average, the proposed techniques have better performance than lora ( §4). moreover, it has a much faster convergence rate (hence lower training time) as compared to lora ( §5). in a nutshell, we make the following contributions: ## related work llms are generally fine-tuned using parameter efficient fine tuning (peft) methods. existing peft techniques typically fall into three categories @xcite : (1) additive methods (these involve the inclusion of a small set of additional trainable parameters/modules, e.g., adaptors @xcite , prefix-tuning (li and liang, 2021)); (2) selective methods (these involve selecting a smaller subset of parameters/modules (e.g., bias in the case of bitfit (ben zaken et al., 2022)) and fine-tuning only those (via application of binary masks), e.g., diff pruning (guo et al., 2021)); (3) re-parametrization techniques (these involve re-parameterization of existing weight update matrix via low-rank approximation, e.g., lora (hu et al., 2022)). in this paper, we focus on reparameterization-based approaches. several variants of lora have been proposed @xcite , each focusing on different aspects of the method @xcite . here, we describe some of the prominent ones; for more details, please refer to the survey by @xcite . dylora @xcite dynamically searches for optimal ranks for different weight matrices of the model rather than using a fixed rank across all layers. methods like au-tolora @xcite and adalora @xcite adaptively allocate the parameter budget across the model matrices by determining an importance score. relora @xcite introduces aggregated low-rank updates to large neural networks during the training phase with a jagged learning rate scheduler, which depends on the interval in which updates are made to the weight matrix. dora @xcite improves convergence by splitting magnitude and directional updates, enabling weight updates close to traditional fine-tuning. vera @xcite further reduces storage requirements by using fixed matrices a and b across layers and introducing trainable diagonal matrices. svft @xcite performs a singular value decomposition of the weight matrix and modifies the singularvalue matrix using a trainable diagonal (svft p -plain svft) or by selecting trainable elements randomly (svft r d ). prolora @xcite ) introduces re-using parameters within the lora adapter matrix by replicating chunks across rows and columns. the paper introduces a rotation enhancement operation involving chunks in the adapter matrices to recover the expressivity in ba lost due to replicating parameters and add a set of trainable parameters to further enhance expressivity. our work is different from prolora; we introduce operations at the row level to inflate the rank of the matrix. most of these methods discussed are additive in nature, with the exception of svft. we explore the effect of replacing additive modules with multiplicative transformations. by investigating multiplicative updates, we aim to address some of the limitations of additive approaches while maintaining the efficiency and effectiveness of peft. multiplicative updates offer a more expressive mechanism for modifying weight matrices. by leveraging matrix multiplication, we can encode richer transformations, which may better capture several complex relationships. we propose a generic multiplicative variant of the additive lora. our proposed variant is orthogonal to many of these variants, thus enabling one to further improve the strategy's effectiveness by combining our variant with existing variants in the literature. for example, analogous to efficient rank-allocation strategies like autolora for additive lora, an equivalent multiplicative variant like autolorma can be devised by transforming the weight update to be multiplicative and using the rank-allocation strategy of autolora. similarly, approaches can be devised for qlorma, adalorma, etc. given this motivation, we primarily benchmark our proposed approach of lorma against lora to show that it has a competitive performance. ## experimentation we conduct a comprehensive set of experiments across a diverse set of tasks within the domain of natural language understanding and generation, involving widely used language models with a range of sizes from roberta (base: 125m params, large: 355m params) @xcite and gpt-2 (medium: 355m params) @xcite to gemma-2b (2.5b params) @xcite and llama3-8b (8b params) @xcite . lorma has been evaluated against various baselines, including lora and its variants like dora and svft and other peft strategies like bitfit and adapters ( §2). we also report the full finetuning results for comparison. overall, lorma demonstrates competitive performance to existing approaches. ## conclusion in this work, we proposed lorma, a novel approach for updating the weights of a language model via multiplicative updates. we mathematically proved the existence of multiplicative updates. further, to overcome the limitations of the naive approach of multiplicative updates, we propose two methods to inflate the rank of the update matrix via permutation and additive identity. extensive experiments demonstrate the competitive performance and training efficacy of the proposed approach. in the future, we plan to experiment with combining lorma with existing lora-based enhancements like autolora, dylora, etc.