AL o RA : Allocating Low-Rank Adaptation for Fine-tuning Large Language Models (2024)

Abstract: Parameter-efficient fine-tuning (PEFT) is widely studied for its effectiveness and efficiency in the era of large language models. Low-rank adaptation (LoRA) has demonstrated commendable performance as a popular and representative method. However, it is implemented with a fixed intrinsic rank that might not be the ideal setting for the downstream tasks. Recognizing the need for more flexible downstream task adaptation, we extend the methodology of LoRA to an innovative approach we call allocating low-rank adaptation (ALoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. First, we propose a novel method, AB-LoRA, that can effectively estimate the importance score of each LoRA rank. Second, guided by AB-LoRA, we gradually prune abundant and negatively impacting LoRA ranks and allocate the pruned LoRA budgets to important Transformer modules needing higher ranks. We have conducted experiments on various tasks, and the experimental results demonstrate that our ALoRA method can outperform the recent baselines with comparable tunable parameters.

## introduction large language models (llms) have been emerging and achieving state-of-the-art (sota) results not only on a variety of natural language processing tasks @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite , but also many challenging evaluation tasks @xcite @xcite like question answering in special domains, reasoning, math, safety, instruction following. despite llms becoming general task solvers, fine-tuning still plays a vital role in efficient llm inference and controlling the style of the llms' generated contents. 1 fine-tuning such large models by full parameters is prohibitive since it requires a large amount of gpu memory and computations. thus, parameter-efficient finetuning (peft) @xcite @xcite has raised much attention in the research field since in peft, the tunable parameters are often less than 1% of the llms and the computation costs will be significantly decreased. many peft methods have been validated to be effective across various models and tasks, often yielding comparable results with full-parameter fine-tuning @xcite @xcite . among these peft methods, the reparameterization-based method low-rank adaptation (lora) @xcite is considered one of the most efficient and effective methods at present. lora is especially popular after open-sourced llms become ubiquitous @xcite . lora assumes that the change of the model's parameters for adaptation is intrinsically low-dimensional and performs adaptation by optimizing the matrix obtained from low-rank decomposition. since it is in the form of weight matrix reparameterization, lora parameters can be merged with the original llms and cause no forward propagation latency. although lora is effective and can bring stable performance with the original setting in @xcite , how to fully exploit its potential for downstream tasks still needs to be determined. first, how to determine the intrinsic rank for each model weight in the transformer block is still unclear. moreover, is it reasonable to set the same lora rank number for adapting the query, key, and value matrix? second, in practice, the optimal lora rank setting would vary according to multiple factors, such as the backbone model and the task. in order to improve the performance of downstream task adaptation of lora, we now propose the allocating lora (alora) framework (depicted in figure 1 ). first, lora modules with equal rank size are initialized at each transformer weight, with all rank gates set to one. during finetuning, we re-allocate the lora ranks by (a) identifying which lora ranks are abundant or have negative contributions and prune those ranks by setting the rank gates to 0; (b) adding the pruned rank budgets to model weights that receive no pruning, that is, important model weights will be assigned more lora ranks. in order to calculate the contribution score of each lora rank efficiently and accurately, we propose a novel method, ab-lora. our working procedure does not require re-training and does not require higher lora rank budgets at initialization or during training. we conduct extensive experiments on a wide collection of tasks, including sentiment classification, natural language inference, question answering, natural language generation under constraint, and instruction tuning, to demonstrate the effectiveness of our method. notably, our method can consistently outperform strong peft baselines with our contributions are summarized as follows: ## experiments in this section, we conduct a series of experiments to evaluate our alora method. ## conclusion this work presents the allocating low-rank adaptation (alora), an innovative method for parameter-efficient fine-tuning large language models. upon the hypothesis that the adaptation for different transformer modules could be of different tanks, we introduce a novel workflow for allocating lora ranks in the fine-tuning process. first, we propose a novel method, ab-dnas, to accurately evaluate the importance scores of lora ranks. second, guided by the ab-dnas method, our workflow allows the pruning of ranks at specific modules and considers allocating more ranks to essential modules. thus, our method does not require to set a more significant initial rank. our method is convenient to implement and off-theshelf. experiments on various tasks demonstrate that our alora method outperforms the baseline methods.