QEFT : Quantization for Efficient Fine-Tuning of LLM s (2024)

Abstract: With the rapid growth in the use of fine-tuning for large language models (LLMs), optimizing fine-tuning while keeping inference efficient has become highly important. However, this is a challenging task as it requires improvements in all aspects, including inference speed, fine-tuning speed, memory consumption, and, most importantly, model quality. Previous studies have attempted to achieve this by combining quantization with fine-tuning, but they have failed to enhance all four aspects simultaneously. In this study, we propose a new lightweight technique called Quantization for Efficient Fine-Tuning (QEFT). QEFT accelerates both inference and fine-tuning, is supported by robust theoretical foundations, offers high flexibility, and maintains good hardware compatibility. Our extensive experiments demonstrate that QEFT matches the quality and versatility of full-precision parameter-efficient fine-tuning, while using fewer resources. Our code is available at https://github.com/xvyaward/qeft.

## introduction while the outstanding zero-shot performance of large language models (llms) @xcite @xcite significantly contributes to their popularity, their versatility and adaptability are also crucial factors in their widespread adoption. through transfer learning and fine-tuning, llms can be extended to handle unseen or complex tasks, including new data types, which opens up new possibilities across various applications @xcite . as efficient inference and finetuning become increasingly important, this paper explores methods to enhance the efficiency of both inference and fine-tuning for llms. examining past research, several methods have been proposed to enhance the efficiency of in-* these authors contributed equally. ference, including pruning @xcite , speculative decoding @xcite , kv caching @xcite , and, particularly, weight quantization @xcite @xcite @xcite . however, studies focusing on lightweight approaches for fine-tuning remain relatively limited. this is because when considering both finetuning and inference, the factors requiring optimization-such as inference speed, training speed, memory consumption, and accuracy-become significantly varied. balancing all these conditions simultaneously presents a substantial challenge. in this context, lora @xcite is a representative study enabling parameter-efficient finetuning (peft) @xcite @xcite by freezing the pre-trained weights while adding a decomposed path that undergoes updates. this approach opens up new opportunities for updating llms flexibly with limited resources, leading to numerous emerging applications @xcite @xcite . in addition, several studies have advanced this concept by attempting to harmonize weight quantization and lora to reap the benefits of both methods. for instance, in qlora @xcite and qa-lora @xcite , the pre-trained weights remain fixed after quantization, while only the fp16 low-rank path is added and updated exclusively. however, qlora exhibits slow inference speeds, and both methods still entail noticeable fine-tuning overhead. achieving improvements in all aspects of inference and fine-tuning is not easily accomplished by simply applying multiple optimizations in parallel. in this study, we propose a new quantization technique called quantization for efficient fine-tuning (qeft), designed to achieve optimal performance in both inference and training. this method employs the data format of owq @xcite , storing weak columns vulnerable to quantization in fp16 while storing the majority of weights in 4-bit or less, and updating only the weak columns during fine-tuning. this approach allows us to enjoy the benefits of quantization while implementing peft. however, qeft offers its own unique innovations. first, owq suffers from the irregular mixed precision of columns in the weights of linear layers, resulting in low hardware compatibility. in contrast, qeft achieves a structured mixed precision phenotype based on the novel offline global reordering (ogr), improving hardware compatibility and resulting in substantial speed improvements in both training and inference, as shown in fig. 1 . additionally, qeft provides a theoretical framework for selecting weak columns to minimize loss values after fine-tuning. lastly, despite being implemented differently from lora, we validate that qeft can replace and be applied to applications that previously used lora, demonstrating its flexibility. through various experiments, we showed that qeft is the state-of-the-art method in terms of inference speed, training speed, and model quality. while it consumes slightly more memory than owq, qeft outperforms it in every other aspect and surpasses other baselines in all areas. ## detailed overview of qeft in this study, we introduce qeft, a mixedprecision quantization that achieves higher speed for both inference and fine-tuning and better finetuned quality thanks to its hardware-friendly and expressiveness. we begin by detailing qeft in this section and explain fine-tuning capabilities in sec. 4. then, we discuss peft merging, an advanced example validating its versatility, in sec. 5. ## optimal weak column selection while qeft is efficient for both fine-tuning and inference, it also offers superior fine-tuning quality. in this section, we provide theoretical support by proving that selecting weak columns as mask for tunable parameters is an optimal strategy for minimizing the loss value after sparse peft, under the following conditions: 1. a fixed budget is allocated to each linear layer. 2. the selection is applied at a per-channel granularity. firstly, we formulate the sparse peft as follows: to maximize the effect of fine-tuning, we need to select an appropriate m that can minimize the loss. according to the second-order approximation method of @xcite , we can find out the optimal mask based on the magnitude of the gradient. ## advanced application: peft merging qeft is designed for efficient inference and finetuning, but it also needs to be general enough to be an alternative to the lora-based approach. to validate this, we applied qeft to an advanced application of lora known as peft merging @xcite . fig. 6 provides an overview of this process. as shown in the figure, the lora adapter is fine-tuned using the open-platypus dataset @xcite on the llama-2 model. after finetuning, the updated weights are transferred to the stablebeluga @xcite model, which was also initialized with llama-2 but fine-tuned on a different dataset, resulting in the creation of stable-platypus2. despite potential differences in semantics, stable-platypus2 surprisingly exhibits better quality than each model before the merge. to assess the merging ability of qeft, we also attempt a similar approach, called qeft merging. we generate the quantized llama-2 model and fine-tune the weak columns for the open-platypus dataset. the updates of the weak columns (∆ = b -a) are then merged into the stablebeluga model. if the target model is a full-precision model, we add the update according to the weak column index. for the quantized stablebeluga, we add the update to the corresponding weak columns. we show that qeft merging works surprisingly well in sec. 6.3, validating its generality. ## experiments 6.1 experiments setting to demonstrate the superiority of qeft, we conducted extensive analysis. the fine-tuning environment adheres to the setup of a baseline, platypus @xcite . we utilized the open-platypus dataset for fine-tuning, specially filtered to remove duplicates and redundancy among 11 open-source datasets. given that the open-platypus dataset focuses on stem and logic question domains, we also adopted their evaluation method, which includes few-shot tasks from the open-llmleaderboard @xcite . following the recent version of the leaderboard, we report scores for 6 tasks (mmlu @xcite , hellaswag @xcite , arcc @xcite , truthfulqa @xcite , winogrande @xcite , and gsm8k @xcite ), along with their average, to assess fine-tuning performance. additionally, we report the average score of four tasks (mmlu, hel-laswag, arc-c, and truthfulqa) used for evaluation in previous studies @xcite . we utilized lm-eval-harness @xcite to measure few-shot accuracy. we used adamw optimizer with batch size of 16. we used constant learning rate of 1e-5 and 5e-6 for @xmath0, respectively. we observed the overfitting issue of optq reconstruction in the 70b model, which is also found in the previous ## conclusion the storage and computation demands of llms present significant barriers to widespread adoption. while quantization and peft optimize inference and fine-tuning, respectively, their harmonization is overlooked. in this work, we introduce qeft, designed to ensure fine-tuning compatibility while prioritizing hardware compatibility. our experiments confirm that qeft achieves the fastest fine-tuning and yields the highest accuracy. moreover, qeft shows superior performance in inference, highlighting its excellence across all aspects.