L o R a DA : Low-Rank Direct Attention Adaptation for Efficient LLM Fine-tuning (2025)

Abstract: As the parameter size of language models becomes extremely large, fine-tuning them with limited resources has become a challenging task. Latest advancements in parameter-efficient fine-tuning (PEFT) techniques allow for adjustments to only a minor fraction of the parameters of these LLMs. Yet, most of PEFT methods may suffer from the following limitations: (1) As the rank decreases sharply, PEFT methods like LoRA and Adapter tuning will exhibit significant performance degradation in downstream tasks. (2) An accuracy gap between these methods and full fine-tuning (Full-FT) still exists. To tackle these problems, we propose a Low-Rank Direct Attention Adaptation (LoRaDA) method for efficient LLM fine-tuning. Specifically, we introduce a novel Low-rank Multi-head Attention Map Module (LMAM), which can bring negative attention to self-attention modules and learn low-rank attention weights directly, capturing the characteristics of downstream tasks. Furthermore, LMAM can serve as a plug-in to existing methods, such as LoRA and Adapter, providing state-of-the-art performance even with extreme low rank setting.Extensive experiments on various downstream tasks demonstrate the superior performance of our LoRaDA method. Specifically, LoRaDA even outperforms the full fine-tuning method by up to 2.1% on GLUE benchmark. As a plug-in, LMAM boosts the accuracy of LoRA by up to 27.7% with LLaMA-7B on Commonsense Reasoning benchmark.

## introduction most traditional nlp tasks acquire domain knowledge from extensive amounts of text data, a naive solution is fine-tuning all model parameters. with the rapid expansion of model parameters and data volume, a multitude of large language models (llm) have emerged, such as gpt-3 (175b) @xcite , llama (7b-65b) @xcite , and opt (125m-175b) @xcite . the advent of these models has made full fine-tuning increasingly impractical due to the enormous resource consumption. to address this challenge, researchers consider tuning only a small subset of parameters of llm while keeping most parameters frozen. for example, lora @xcite employs the concept of low-rank approximation by updating only a subset of the parameters to fine-tuning the model. adapter @xcite introduces small, trainable modules into each layer of a pre-trained model, allowing for efficient fine-tuning with minimal changes to the original model parameters. all these methods are called parameter-efficient fine-tuning (peft) , which significantly reduce the trainable parameters and resource consumption. however, most peft methods @xcite , including their variants @xcite @xcite , achieve lower performance compared to full fine-tuning. in addition, the performance of the latest methods @xcite @xcite @xcite tend to degrade sharply when the rank is significantly reduced. we observe that most peft methods represent attention map values as positive numbers between 0 and 1, and such a positive attention mechanism in llms often allocates considerable attention to irrelevant contexts, degrading model performance, as illustrated in @xcite @xcite . to cure this problem, recent works @xcite @xcite stress that negative attention can cancel noise and boost the model's capability. for example, inspired by differential amplifiers @xcite in electrical engineering which output the difference between two signals to eliminate common-mode noise, diff transformer @xcite introduces negative attention via a differential attention mechanism, demonstrating the superior performance in longcontext modeling, key information retrieval, hallucination mitigation and in-context learning. po-laformer @xcite points out that the non-negative constraints on attention maps result in significant information loss, leading to less discriminative attention maps. yet, these methods aims to study new model architecture with negative attention, and such a model need to be trained from scratch. thus, these methods can not be directly applied in fine-tuning tasks. based on this, a naive approach of introducing negative attention to peft would be adding learnable attention maps with both negative and non-negative values to the original attention map. however, too many learnable parameters in attention maps will cause overfitting problems, as shown in @xcite . therefore we introduce a novel static plug-in module called the low-rank multi-head attention map (lmam), which learns two low-rank weights on attention maps, thus bring negative attention to standard self-attention modules. as shown in figure 1 , the color blue indicates negative attentions which suppress less important tokens (e.g., "is", "the", "berling", "acted"), while the color red represents salient tokens such as "exquisitely", "superbly", "deeply" and "quit". thus it can enhance model performance with a minimal rank and effectively capture the characteristics of downstream tasks. as a plug-in, lmam can enable state-ofthe-art peft methods to outperform full-ft in downstream tasks. to demonstrate the effectiveness of our model, we conducted experiments on multiple benchmarks such as commonsense reasoning (+1.2% performance), glue (+2.1% performance), xsum and mt (+1.24% performance). our contributions can be briefly summarized as follows: ## related works as the size of models increases, the limitations of full-parameter fine-tuning become apparent. the high economic cost and the issue of catastrophic forgetting are the two main challenges currently faced. these issues have prompted researchers and engineers to seek more efficient and cost-effective fine-tuning strategies. parameter efficient finetuning (peft) involves fine-tuning a small number of parameters in large models to reduce costs and avoid catastrophic forgetting. the current peft methods can be classified into four primary categories: lora-based methods @xcite @xcite @xcite , adapter-based methods @xcite @xcite @xcite , prefix-based methods @xcite @xcite , and others @xcite @xcite @xcite . the core idea of low-rank adaptation (lora) @xcite is to approximate the weight changes in the model through low-rank matrix decomposition. nevertheless, these methods exhibit considerable limitations, as their performance deteriorates significantly with rank reduction, ultimately failing to adequately capture the nuances of downstream tasks. the second methods of peft is adapter @xcite , which adapts to new tasks by inserting additional small network layers, known as adapter layers, into the architecture of the pretrained model, without the need to retrain the entire model. the third peft method is prefix-tuning @xcite , which adds a small, learnable prefix of parameters at each model layer. these prefixes serve as conditional information, guiding the model to generate outputs that align with specific tasks. finally, we will introduce the other methods. for instance, quanta @xcite introduce efficient high-rank fine-tuning to potential failures of lora's extreme low-rank approach in complex downstream tasks. @xcite introduce a variant of moe that trains experts orthogonally to encourage diversity. ## methodology in this section, we begin by presenting the preliminaries, including multi-head attention and differential attention. we then give details of our proposed low-rank multi-head attention map module (lmam). finally, we discuss the extra operation and memory consumption brought by lmam.