Can Post-Training Quantization Benefit from an Additional QL o RA Integration? (2025)

Abstract: Large language models (LLMs) have transformed natural language processing but pose significant challenges for real-world deployment. These models necessitate considerable computing resources, which can be costly and frequently unavailable. Model compression techniques such as quantization are often leveraged to alleviate resource demand, but they may have a negative impact on the generation quality. In this study, we explore the integration of 4-bit Post-training Quantization (PTQ) with QLoRA to address these issues. We demonstrate through extensive experiments that this integration outperforms standard PTQ, and in some cases even 16-bit full-parameter fine-tuning on LLMs, validated across proprietary and public datasets with different quantization algorithms. The results demonstrate the efficacy of PTQ-QLoRA integration, offering a viable solution for deploying powerful LLMs in resource-constrained environments without compromising on performance.

## introduction large language models (llms) have undeniably revolutionized the field of natural language processing and keep growing in both popularity and size. however, the "large" in llms is both their benefit and their curse. as the models are becoming more powerful, they are increasingly harder to train, deploy and serve in real-life applications in industry. they require substantial computing resources which are not only expensive but also not always readily available. obtaining resources for training llms is a challenge of its own, but deploying llms in customerfacing applications poses a new set of challenges. specifically, llm inference in real-life scenarios comes with certain challenges. it must meet latency requirements to ensure a smooth user experience for end users. it is also subject to memory constraints from accessible hardware, which is not always optimized for llms. additionally, it needs to allow for frictionless scaling as the number of requests to llms grows with the number of users or features it serves. therefore, there exists a need for optimization techniques that would allow for deployment of the most powerful llms regardless of the number of parameters but also address these issues without significant loss in performance. one of the popular techniques to optimize memory usage and computational efficiency is quantization, which reduces the precision of the numerical representation of data and thereby the model's size and the computational resources required for inference by a large margin, but often results in meaningful accuracy loss @xcite . at the same time, quantized large models can outperform full-precision models of smaller size @xcite , making quantized models a potentially preferred option and recovering accuracy loss a particularly important task. in this study, we explore the integration of post-training quantization (ptq) and qlora @xcite , which utilizes parameterefficient fine-tuning (peft) on a quantized model, to mitigate the loss in accuracy due to quantization. we focus solely on 4-bit quantization because it provides an optimal balance of memory footprint, latency and accuracy for our specific use cases, where the model is deployed 1 to handle business conversations such as support calls or meetings. we show through extensive experiments that this integration outperforms simple ptq and in certain cases even the 16-bit fully fine-tuned model. our contributions are the following: ## background traditionally, deep neural network models utilize high-precision floating point numbers to represent weights and activations, which requires significant memory and computational resources. quantization has emerged as a powerful technique to address this challenge by quantizing floating-point representations into a lower bit-width, effectively reducing the model's memory footprint and computational cost. quantization techniques generally fall into two main categories: post-training quantization (ptq) and quantization-aware training (qat). the former quantizes a model after the training is complete, without the need for retraining. early work like @xcite proposed a quantization schema that uses integer arithmetic to approximate the floating point. @xcite computes a layer-wise local loss and optimizes this loss with a soft relaxation. @xcite proposed brecq framework which achieves a good balance between cross-layer dependency and generalization error by reconstructing at the block granularity. more recently, llm.int8() from @xcite demonstrated for the first time that multi-billion parameter transformers can be effectively quantized to int8. moreover, @xcite introduced gptq which can accurately quantize llms of billions of parameters to 3-4 bits per component. activation-aware weight quantization (awq) from @xcite employs perchannel scaling to reduce the quantization loss of salient weights. conversely, qat techniques typically involve retraining the model with quantized parameters so that the model can converge to a point with better loss @xcite . @xcite presented a standard qat pipeline that leads to near-floating-point accuracy results for a wide range of models. another efficient approach to adapting pre-trained models with minimal overhead is parameter-efficient fine-tuning (peft). one direction is the adapter-based method, which injects small adapter modules into pre-trained models @xcite . more recently, low-rank adaptation (lora) @xcite has become increasingly popular, greatly reducing the number of trainable parameters by introducing rank decomposition matrices. moreover, qlora @xcite backpropagates gradients through a quantized model into lora while preserving high task performance. although @xcite shows that qlora can match the accuracy of 16-bit full fine-tuning in t5 @xcite and roberta @xcite , the comparison of qlora and 16-bit tuning in other larger language models has not been studied to the best of our knowledge. ## conclusion in this study, we explore the ptq-qlora that integrates 4-bit post-training quantization with qlora to optimize the deployment of llms in resourcelimited environments. through extensive experimentation, we demonstrate that this integration can match or surpass the performance of 16-bit full parameter fine-tuning, across various base llms, quantization methods and tasks. the results highlight that combining ptq with qlora enhances model efficiency without sacrificing task-specific accuracy. this effective solution allows high-performing llms to be deployed with fewer resources. overall, our findings underscore the potential of this integration to improve the practical deployment of llms, offering a scalable approach for future applications.