D o RA : Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution (2024)

Abstract: Fine-tuning large-scale pre-trained models is inherently a resource-intensive task. While it can enhance the capabilities of the model, it also incurs substantial computational costs, posing challenges to the practical application of downstream tasks. Existing parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) rely on a bypass framework that ignores the differential parameter budget requirements across weight matrices, which may lead to suboptimal fine-tuning outcomes. To address this issue, we introduce the Dynamic Low-Rank Adaptation (DoRA) method. DoRA decomposes high-rank LoRA layers into structured single-rank components, allowing for dynamic pruning of parameter budget based on their importance to specific tasks during training, which makes the most of the limited parameter budget. Experimental results demonstrate that DoRA can achieve competitive performance compared with LoRA and full model fine-tuning, and outperform various strong baselines with the same storage parameter budget. Our code is available at [github](https://github.com/MIkumikumi0116/DoRA)

## introduction pre-trained language models (plms) @xcite @xcite @xcite play a crucial role in natural language processing (nlp), offering substantial improvements in various downstream tasks @xcite @xcite . customizing these models for specific tasks typically involves fine-tuning them to adapt pretrained knowledge to particular requirements @xcite . however, with the increasing scale of plms, the cost of fullmodel fine-tuning becomes prohibitive @xcite . this has highlighted the demand for and * equal contributions. fortunately, there are methods like adalora @xcite that can alleviate the limitations of the prior peft methods by introducing a more nuanced parameter distribution strategy. training with adalora begins with a higher parameter budget and simulates an svd (singular value decomposition) decomposition process, progressively pruning smaller singular values and corresponding singular vectors. it opens the door to implementing adaptive allocation of parameter budgets. however, its dependence on orthogonal regularization for the simulated svd decomposition might restrict further improvements in fine-tuning efficiency. additionally, the pruning strategy of adalora focuses solely on singular values and does not fully exploit all the available information in projection matrices, potentially leading to less-than-optimal decisions. to address existing challenges, this work introduces the dynamic low-rank adaptation (dora) method, as depicted in figure 1 . different from lora approaches, dora decomposes high-rank lora layers into sums of single-rank components, evaluates the contribution of each component to overall performance, and prunes components with fewer contributions. this allows for the on-demand allocation of parameter budgets to modules of the plm, maximizing the use of limited parameter budgets. compared to existing methods of dynamic parameter allocation (e.g., adalora), dora can allocate parameter budgets more appropriately based on a richer set of information from projection matrices. to sum up, our contributions are as follows: ## background the emergence of plms such as bert (kenton and toutanova, 2019), gpt @xcite , and llama @xcite has meaningfully advanced the field of nlp. trained on extensive text datasets, plms capture intricate linguistic patterns, enabling superior performance in various nlp tasks such as text categorization, named entity recognition, and machine translation @xcite . their flexibility in adapting to specific datasets via fine-tuning renders them exceedingly versatile for addressing various linguistic challenges. plms predominantly leverage the transformer architecture @xcite which features stacked transformer blocks. each block comprises two key components: a multi-head attention (mha) mechanism and a feed-forward neural (ffn) network. in particular, mha effectively captures contextual relationships in text and is given by: where @xmath0, n is the sequence length and d is the hidden dimension. the mechanism consists h self-attention heads, each aiming to capture different aspects of information. for each head head i , there are three projection matrices: query w qi , key w ki , and value w vi , each with dimensions r d×d h , where d h is the dimension of each head, typically set to d/h. the output projection matrix w @xmath1. attention scores are calculated by normalizing the dot product of queries and keys through the softmax function and are given by: these scores determine the attention each sequence position pays to other positions. subsequently, these scores are multiplied by the value projection result to yield the output of each head. finally, the outputs of all heads are concatenated and multiplied by the output projection matrix w o , forming the final mha output. following mha, the ffn further processes the information: this allows for more complex interactions between the features extracted by the self-attention mechanism. each transformer block incorporates a residual connection that adds the input of the block directly to its output. this approach helps to alleviate the vanishing gradient problem and ensures a consistent information flow across the layers of the models. ## dynamic low-rank adaptation in this paper, we aim to optimize the use of a limited parameter budget in fine-tuning plms with lora. we make improvements based on lora @xcite and adalora @xcite , as shown in prune the components with smaller s(t) based on b(t), set their c to 0; 10: end for 11: output: the fine-tuned parameters {a, b, c}. dora that stands out due to its innovative approach, comprising three main strategies: a decomposition strategy that views a high-rank lora layer as a combination of multiple single-rank lora components, a dynamic rank allocation mechanism that adjusts these components based on their contribution to the overall performance of the model and a regularization penalty to ensure stable pruning throughout the process. the overall algorithm is shown in algorithm 1. ## conclusion in this paper, we introduce dynamic low-rank adaptation (dora), a novel method aiming at enhancing the efficiency of fine-tuning plms by dynamically adjusting parameter distribution. dora innovatively allocates parameter budgets based on their importance to specific tasks, demonstrating considerable improvements in nlp applications. experimental results indicate that dora surpasses baseline methods, highlighting its potential for broader adoption in model optimization efforts. the innovation of dora lies in its adoption of an adaptive parameter allocation strategy, which, unlike traditional uniform distribution, dynamically adjusts the distribution of parameter budgets based on their contribution. additionally, dora employs a component-wise decomposition approach for handling lora layers, treating high-rank lora layers as a combination of single-rank lora components. these components are adjusted through a dynamic rank allocation mechanism, pruned according to their contribution to the overall model performance. to ensure stable pruning throughout the process, dora incorporates a regularization penalty term focused on reducing component variance.