DL - QAT : Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models (2024)

Abstract: Improving the efficiency of inference in Large Language Models (LLMs) is a critical area of research. Post-training Quantization (PTQ) is a popular technique, but it often faces challenges at low-bit levels, particularly in downstream tasks. Quantization-aware Training (QAT) can alleviate this problem, but it requires significantly more computational resources. To tackle this, we introduced Weight-Decomposed Low-Rank Quantization-Aware Training (DL-QAT), which merges the advantages of QAT while training only less than 1% of the total parameters. Specifically, we introduce a group-specific quantization magnitude to adjust the overall scale of each quantization group. Within each quantization group, we use LoRA matrices to update the weight size and direction in the quantization space. We validated the effectiveness of our method on the LLaMA and LLaMA2 model families. The results show significant improvements over our baseline method across different quantization granularities. For instance, for LLaMA-7B, our approach outperforms the previous state-of-the-art method by 4.2% in MMLU on 3-bit LLaMA-7B. Additionally, our quantization results on pre-trained models also surpass previous QAT methods, demonstrating the superior performance and efficiency of our approach.

## introduction large language models (llms) have demonstrated exceptional performance across a variety of natural language processing (nlp) tasks. with the growing deployment and use of these models, quantization has become an essential method for reducing memory usage and enhancing computational efficiency. in llm compression, a range of post-training quantization (ptq) techniques have been developed, such as weight-only and weightactivation quantization. these techniques generally use a small calibration dataset and apply learning or optimization strategies to quickly transform a pre-trained floating-point model into a quantized version. however, ptq methods struggle in low-bit quantization, especially in the downstream tasks. despite the potential benefits, the development of quantization-aware training (qat) algorithms has been constrained. this is primarily due to the significant data and computational resources required for comprehensive model fine-tuning, making it a costly endeavor. to address the high computational expense associated with training llms, the parameter-efficient fine-tuning (peft) methodology has been introduced. peft entails fine-tuning only a fraction of the model's parameters, as opposed to the entirety, thereby enabling the efficient adaptation of pretrained models to a diverse range of downstream applications. notably, the low-rank adaptation (lora) @xcite technique, which represents the current state-of-the-art in peft, has been shown to achieve performance on par with fully fine-tuned models across various downstream tasks, without necessitating alterations to the model's inference architecture. the conventional approach to generating a quantized model for downstream tasks involves a two-step process: first, the floatingpoint model is fine-tuned on the downstream tasks; second, ptq is applied to the fine-tuned model. however, this methodology is not without its drawbacks, as it can be cumbersome and may result in a substantial loss of accuracy. conversely, directly employing qat methods can lead to prohibitively high computational costs due to the requirement of end-to-end fine-tuning of all the model's parameters. the objective of our research is to devise a seamless, end-to-end process that yields a quantized model with parameter-efficient fine-tuning, thereby mitigating the aforementioned challenges and enhancing the overall efficiency and effectiveness of model adaptation for downstream tasks. building upon these considerations, we propose weight-decomposed low-rank quantization-aware training (dl-qat), a novel end-to-end method designed to enhance the efficiency and effectiveness of model quantization for downstream tasks. dl-qat decomposes the optimization of quantized weights into two processes: groupspecific magnitude training and weight fine-tuning within a predefined quantization space. by incorporating a magnitude term, we calibrate the overall scale for each quantization group, ensuring a more precise representation of the model's parameters. furthermore, we leverage low-rank matrices a and b to refine the quantized weights, thereby enhancing the model's adaptability to the specific requirements of the downstream tasks. to validate the efficacy of our approach, we conducted comprehensive experiments on the llama and llama2 model families. the results demonstrate a significant improvement over the baseline method, qa-lora @xcite , across various quantization granularities. specifically, our method surpasses qa-lora by +4.2% on the mmlu benchmark @xcite and by +5.5% on the lm-eval benchmark @xcite . additionally, when compared to the previous state-of-theart llm-qat method @xcite , our approach achieves lower perplexity on the wikitext-2 dataset @xcite and higher accuracy on the lm-eval benchmark, underscoring the superior performance of dl-qat. llm-qat requires finetuning the entire model parameters, while we only need to fine-tune less than 1% of the parameters to achieve better results. these findings not only highlight the effectiveness of dl-qat in achieving competitive accuracy levels but also emphasize its efficiency in terms of both parameters and memory usage. by requiring minimal parameter modifications, dl-qat offers a compelling alternative to traditional quantization methods, particularly for scenarios where computational resources are limited or where the need for rapid model adaptation is paramount. ## related work parameter-efficient fine-tuning. lora (low-rank adaptation) is a key method in parameter-efficient fine-tuning (peft), training a small number of parameters without altering the model inference process. to enhance its capabilities, variants like adalora @xcite and pissa @xcite enhance rank via singular value decomposition (svd), while plora @xcite accumulates low-rank updates progressively. further, studies like @xcite and lora+ @xcite delve into the update mechanisms of lora's a and b matrices. dora @xcite proposed a new optimization approach for lora, which decomposes lora updates into separate magnitude and direction updates to improve accuracy. inspired by this idea, we further decompose lora quantization-aware training into fine-tuning the magnitude for quantization groups and fine-tuning the weights within the quantization space. quantization of llm. quantization has been widely used in llm. based on whether training is required, quantization can be classified into post-training quantization(ptq) and quantization-aware training(qat). ptq methods requires only a small amount of calibration data to update the quantized weights. for instance, gptq @xcite utilizes merely 128 data samples to approximate second-order information and achieve the quantized weight. as outliers are crucial for llm, considerable research is dedicated to addressing outlier issues. smoothquant @xcite effectively shifts the quantization challenge from activations to weights through a mathematically equivalent transformation. quarot @xcite employs hadamard transformations on the weight matrices and attention modules to mitigate outlier effects. compared with ptq methods, qat methods require more training data and resources, but generally achieve better performance. llm-qat @xcite leverages data generated by pre-trained llms and achieves better performance compared with gptq. however, llm-qat requires significant training resources. methods combining lora and quantization. building upon lora, qlora @xcite was the first to propose a memory-efficient fine-tuning method by quantizing the pretrained model to low-bit and fine-tuning a high-precision lora component. this approach enables effective fine-tuning of llms within limited memory resources. subsequent methods such as loftq @xcite and lq-lora @xcite further optimized the initialization of the lora component and reduced the memory required for the quantized pretrained model. however, the combination of a low-bit pretrained model and a high-precision lora component still resulted in a high-precision weight after merging, which did not improve inference speed. to address this issue, qa-lora @xcite made further improvements on qlora by learning an additional high-precision group-wise bias for the quantized model, effectively reducing both time and memory consumption without compromising accuracy. however, qa-lora could only perform groupwise fine-tuning, resulting in significant accuracy degradation when the quantization granularity increased. ## experiments in this section, we assess our approach using both language generation and zero-shot few-shot tasks with open-source models llama-7b/13b @xcite and llama2-7b/13b @xcite to demonstrate its effectiveness. ## conclusion in this paper, we introduce weight-decomposed low-rank quantization-aware training (dl-qat), a novel end-to-end approach designed to improve the efficiency of qat for tasks downstream of llms. dl-qat optimizes quantized weights through two main processes: group-specific magnitude training and weight fine-tuning within a set quantization space. by employing low-rank adaptation (lora) matrices, we are able to update the weight magnitude and direction within the quantization space, thereby enabling precise adjustments to the model's parameters. dl-qat achieves remarkable results by training on less than 1% of the model's parameters, outperforming previous qat methods across established natural language processing benchmarks. this efficiency in parameter utilization is a testament to the effectiveness of dl-qat in achieving state-of-the-art performance while minimizing computational overhead.