QD y L o RA : Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning (2024)

Abstract: Finetuning large language models requires huge GPU memory, restricting the choice to acquire Larger models. While the quantized version of the Low-Rank Adaptation technique, named QLoRA, significantly alleviates this issue, finding the efficient LoRA rank is still challenging. Moreover, QLoRA is trained on a pre-defined rank and, therefore, cannot be reconfigured for its lower ranks without requiring further fine-tuning steps. This paper proposes QDyLoRA -Quantized Dynamic Low-Rank Adaptation-, as an efficient quantization approach for dynamic low-rank adaptation. Motivated by Dynamic LoRA, QDyLoRA is able to efficiently finetune LLMs on a set of pre-defined LoRA ranks. QDyLoRA enables fine-tuning Falcon-40b for ranks 1 to 64 on a single 32 GB V100-GPU through one round of fine-tuning. Experimental results show that QDyLoRA is competitive to QLoRA and outperforms when employing its optimal rank.

## introduction the popularity of adopting large language models (llms) across a diverse range of downstream tasks has rapidly increased over the past two years. finetuning llms has become necessary to enhance their performance and introduce desired behaviors while preventing undesired outputs @xcite . however, as the size of these models increases, fine-tuning costs become more expensive. this has led to a large body of research that focuses on improving the efficiency of the fine-tuning stage @xcite @xcite @xcite . low-rank adapter (lora) @xcite ) is a well-known, parameter-efficient tuning (peft) method that reduces memory requirements during fine-tuning by freezing the base model and updating a small set of trainable parameters in form of low-rank matrix multiplication added to matrices in the base model. however, the memory demand during fine-tuning remains substantial due to the necessity of a backward pass through the frozen base model during stochastic gradient descent. recent research has thus focused on further reducing memory usage by designing new parameterefficient modules that can be tuned without necessitating gradients from the base models @xcite . alternatively, researchers have explored combining other efficiency strategies with parameter-efficient tuning methods @xcite . among these approaches, qlora @xcite stands out as a recent and highly efficient fine-tuning method that dramatically decreases memory usage. it enables fine-tuning of a 65-billion-parameter model on a single 48gb gpu while maintaining full 16-bit fine-tuning performance. qlora achieves this by employing 4bit normalfloat (nf4), double quantization, and paged optimizers as well as lora modules. however, another significant challenge when utilizing lora modules is the need to tune their rank as a hyperparameter. different tasks may require lora modules of varying ranks. in fact, it is evident from the experimental results in the lora paper that the performance of models varies a lot with different ranks, and there is no clear trend indicating the optimal rank. on the other hand, any hyperparameter tuning for finding the optimal rank contradicts the primary objective of efficient tuning and is not feasible for very large models. moreover, when deploying a neural network on diverse devices with varying configurations, the use of higher ranks can become problematic for highly sensitive devices due to the increased parameter count. to address this, one typically has to choose between training multiple models tailored to different device configurations or determining the optimal rank for each device and task. however, this process is costly and time-consuming, even when using techniques like lora. dylora @xcite , is a recent peft method that aims to address theses challenges by employing dynamic low-rank adapter (dylora). inspired by nested dropout, this method aims to order the representations of the bottleneck at low-rank adapter modules. instead of training lora blocks with a fixed rank, dylora extends training to encompass a spectrum of ranks in a sorted manner. the resulting low-rank peft modules not only provide increased flexibility during inference, allowing for the selection of different ranks depending on the context, but also demonstrate superior performance compared to lora, all without imposing any additional training time. in this paper, we employ the dylora peft method in conjunction with the quantization scheme utilized in the qlora work, resulting in qdylora. qdylora has all the aforementioned benefits of dylora but with significant memory reduction both during training and at inference through 4-bit quantization. we utilize qdylora for efficient fine-tuning of llama-7b, llama-13b, and falcon-40b models across ranks ranging from 1 to 64, all on a single 32gb v100 gpu. once tuned, we determine the optimal rank by inferring the model on the test set. our results reveal that the optimal rank can be quite low, yet it outperforms qlora. ## experiments and evaluation this section evaluates the efficiency and efficacy of qdylora through several instruct-fine-tuning tasks. the first experiment compares qdylora with qlora on massively multitask language understating (mmlu) benchmark @xcite , consisting of more than 50 different tasks, spanning from fundamental mathematics and u.s. history to computer science and law. as shown in the second experiment provides a more in-depth comparison between qlora and qdylora. in particular, we fairly finetuned falcon-40b on we-bglm @xcite and gsm8k @xcite benchmarks, and compared their test performances across different ranks. as described in ## on the semi-sorted behavior of qdylora as shown in ## conclusion qdylora offers an efficient and effective technique for lora-based fine-tuning llms on downstream tasks. eliminating the need for fine-tuning multiple models to find the optimal lora rank and offering the possibility of fine-tuning larger llms are two main advantages of qdylora. the experimental results demonstrated that the optimal rank for qdylora can be surprisingly low, yet it consistently outperforms qlora. qdylora provides greater flexibility for deploying llms in various contexts and represents a promising step towards making fine-tuning large language models more accessible and efficient.