LRQ : Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices (2025)

Abstract: With the commercialization of large language models (LLMs), weight-activation quantization has emerged to compress and accelerate LLMs, achieving high throughput while reducing inference costs. However, existing post-training quantization (PTQ) techniques for quantizing weights and activations of LLMs still suffer from non-negligible accuracy drops, especially on massive multitask language understanding. To address this issue, we propose Low-Rank Quantization (LRQ) - a simple yet effective post-training weight quantization method for LLMs that reconstructs the outputs of an intermediate Transformer block by leveraging low-rank weight-scaling matrices, replacing the conventional full weight-scaling matrices that entail as many learnable scales as their associated weights. Thanks to parameter sharing via low-rank structure, LRQ only needs to learn significantly fewer parameters while enabling the individual scaling of weights, thus boosting the generalization capability of quantized LLMs. We show the superiority of LRQ over prior LLM PTQ works under (i) 8-bit weight and per-tensor activation quantization, (ii) 4-bit weight and 8-bit per-token activation quantization, and (iii) low-bit weight-only quantization schemes. Our code is available at Software.

## introduction as chatgpt and gpt-4 (openai, 2023) have showcased unprecedented capabilities across various domains such as common sense reasoning, mathematical problem-solving, and coding proficiency, there has been an exponential surge in interest surrounding the development of large language models (llms). this surge in interest has culminated in the recent release of cutting-edge llms like llama @xcite , palm 2 (google et al., 2023), and llama 2 @xcite . accordingly, serving llms has rapidly * equal contribution emerged as a significant concern in both academia and industry. this stems from the substantial memory footprint and considerable computational cost incurred when operating these language models with tens or hundreds of millions of parameters in fp16 format. therefore, extensive efforts @xcite ) such as quantization or pruning are underway to compress llms and provide efficient deployment. in particular, quantization has garnered considerable interest among llm engineers and researchers because quantization aids in not just model compression but also inference acceleration. llm quantization techniques fall into two primary categories: weight-only quantization and weight-activation quantization. weight-only quantization concentrates on enhancing memory-bound operations like matrix-vector multiplication by quantizing weights of llms into low-bit integers (e.g., 2-4 bits). with activations kept in fp16, weight-only quantization exhibits marginal accuracy degradation but is only effective in accelerating text generation inference for small batch sizes (e.g., a single batch). in contrast, weight-activation quantization aims to expedite computationally intensive operations, such as matrix-matrix multiplication, typically by quantizing both weights and activations of llms into 8-bit integers and employing int8 gemm kernels. this comprehensive quantization approach enables llm serving for large batch sizes, thus enhancing llm throughput and expediting llm inference through integer matrix multiplication. yet, it comes with the trade-off of potential non-negligible accuracy drop. while each approach boasts its own set of strengths and weaknesses, we first focus on weight-activation quantization on the grounds that achieving highthroughput llm inference is important to handle a substantial volume of user requests in real time. recent studies @xcite @xcite @xcite 2023a) have attempted to quantize both weights and activations of llms. among these works, only smoothquant @xcite and flexround @xcite demonstrated the potential for a hardware-efficient per-tensor static activation quantization scheme that can reduce the inference latency and memory usage by up to two-thirds and half respectively compared to fp16 baselines as thoroughly elucidated in @xcite . given the compelling advantages of this scheme, we also stick mainly to per-tensor static activation quantization, with a primary focus on preventing nonnegligible performance degradation, one of its key drawbacks, from occurring. despite promising results that smoothquant and flexround yielded, they still possess inherent limitations on enhancing model accuracy when using per-tensor static activation quantization. although smoothquant is a potent technique for alleviating the difficulty of quantizing activation outliers, it uniformly divides activations in each channel and multiplies the weights in the corresponding input channel by some constant. since such an uniform per-channel smoothing transformation can only scale the weights collectively per channel, not individually, smoothquant may lead to non-negligible accuracy loss after quantization for certain models as in figure 1 . on the other hand, as flexround learns a separate scale for each weight and thus enables flexible weight quantization based on individual characteristics of each weight, flexround can show marginal zero-shot accuracy drop on common sense reasoning tasks in figure 1 (a). however, as depicted in figure 1 (b), flexround falls short in performing well on massive multitask language understanding (mmlu), which necessitates problemsolving skills, specialized knowledge, as well as basic knowledge across diverse subjects. we empirically confirm that this phenomenon is because flexround has to learn too many scales relative to limited calibration samples due to the assignment of an independent scale to every weight. to improve generalization performance on such a challenging benchmark, we propose a new posttraining weight quantization approach, "low-rank quantization (lrq)", as a middle ground between smoothquant and flexround. lrq is designed to minimize the mean squared error between the outputs of an intermediate fp16 transformer block and those of its quantized counterpart with respect to low-rank weight-scaling matrices instead of full weight-scaling matrices that involve as many scales as their associated weights. using such low-rank matrices, we can reduce the number of learnable parameters effectively while maintaining the concept of scaling weights individually by sharing learnable parameters via low-rank structure. as a result, lrq can attain comparable accuracy to fp16 baselines on both common sense reasoning tasks and mmlu for every llama model as seen in figure 1 . our main contribution is threefold: ## method in this section, we outline the post-training quantization (ptq) background that our method, lrq is based on, figure out the problem arising when quantizing llms, and formulate lrq. finally, we deepen an empirical understanding of how lrq can improve generalization in quantized llms. ## experiments in this section, we first explore the influence of the rank r in eq. 2 and the quantity of calibration samples on the performance of lrq. next, to verify the effectiveness of lrq, we compare lrq with existing state-of-the-art post-training quantization (ptq) methods for large language models (llms). we use just a single nvidia a100-80gb gpu to quantize llms via lrq. we randomly choose 512 calibration samples with a token length of 1024 from the training set of c4 @xcite . unless otherwise mentioned, lrq is applied to all linear layers in both attention and feed-forward modules, and the rank r in eq. 2 is set to 2048 for large language models beyond 30b parameters or to 1024 for smaller models so as to reduce the number of learnable parameters by approximately half compared to flexround. the exact ratio of the number of learnable parameters in lrq to the number of pre-trained weights for an intermediate transformer block of each llama model is given in appendix j. quantized models are evaluated on mmlu @xcite in the five-shot setting or common sense reasoning benchmarks: boolq @xcite , piqa @xcite , hellaswag @xcite , winogrande @xcite , arc easy and challenge @xcite , and openbookqa @xcite in the zero-shot setting. more details are deferred to appendix i.