A Comprehensive Evaluation of Quantization Strategies for Large Language Models (2024)

Abstract: Increasing the number of parameters in large language models (LLMs) usually improves performance in downstream tasks but raises compute and memory costs, making deployment difficult in resource-limited settings. Quantization techniques, which reduce the bits needed for model weights or activations with minimal performance loss, have become popular due to the rise of LLMs. However, most quantization studies use pre-trained LLMs, and the impact of quantization on instruction-tuned LLMs and the relationship between perplexity and benchmark performance of quantized LLMs are not well understood. Evaluation of quantized LLMs is often limited to language modeling and a few classification tasks, leaving their performance on other benchmarks unclear. To address these gaps, we propose a structured evaluation framework consisting of three critical dimensions: (1) knowledge & capacity, (2) alignment, and (3) efficiency, and conduct extensive experiments across ten diverse benchmarks. Our experimental results indicate that LLMs with 4-bit quantization can retain performance comparable to their non-quantized counterparts, and perplexity can serve as a proxy metric for quantized LLMs on most benchmarks. Furthermore, quantized LLMs with larger parameter scales can outperform smaller LLMs. Despite the memory savings achieved through quantization, it can also slow down the inference speed of LLMs. Consequently, substantial engineering efforts and hardware support are imperative to achieve a balanced optimization of decoding speed and memory consumption in the context of quantized LLMs.

## introduction in recent years, llms have seen substantial growth in the number of parameters, scaling up to billions or even trillions @xcite @xcite @xcite , yielding exceptional performance across various tasks and real-world applications @xcite @xcite @xcite @xcite . however, the huge number of parameters also results in significant compute and memory requirements, hindering their deployment on devices with limited resources. to mitigate these challenges, researchers have proposed various approaches to model quantization, which aim to optimize model inference and memory usage while minimizing performance degradation. the central idea of model quantization is representing the weights or activations of a model in a lower-precision format (such as 8-bit integers) rather than their original high-precision floatingpoint format (typically 16-bit or 32-bit) @xcite . quantization approaches can be broadly classified into two primary categories: quantization-aware training (qat) and post-training quantization (ptq). qat incorporates the quantization process into the training phase of the model, thereby allowing the model to adapt to lower-precision representations @xcite @xcite . conversely, ptq applies quantization techniques after the training phase has finished @xcite @xcite @xcite @xcite . despite the risk of performance degradation, ptq is more prevalent due to the prohibitive training costs associated with qat. however, several aspects pertaining to the evaluation of ptq require further exploration. firstly, the majority of ptq methods are evaluated solely by assessing the performance of the quantized pre-trained llms on benchmarks, leaving the performance of quantized llms that have undergone instruction tuning unclear -despite the latter being more commonly used in real-world scenarios @xcite @xcite . secondly, the evaluation of quantized models is limited to the language modeling task and a few classification tasks. this restricts our understanding of their performance on other benchmarks that are more closely related to real-world applications. lastly, while perplexity is predominantly employed as the evaluation metric for verifying the effectiveness of quantization methods and has been demonstrated as an indicator of the performance of llms on other benchmarks in previous studies @xcite , the correlation between the perplexity of quantized llms and their performance on other benchmarks remains poorly understood. in this paper, we conduct a comprehensive evaluation of the quantized llms that undergo instruction tuning, utilizing a diverse range of publicly available benchmarks. these benchmarks cover language understanding and generation, as well as two critical dimensions of llms: knowledge & capacity and alignment. additionally, we evaluate various quantization strategies for their efficiency in terms of generation speed and memory consumption. the comprehensive framework for this evaluation is illustrated in figure 1 , while ## related work llms quantization there are currently two main formalisms of model quantization: qat @xcite and ptq. ptq applies quantization after model training, while qat considers the effects of quantization during the training process, necessitating considerable resources and expertise, thereby restricting its broader application. consequently, our research primarily concentrates on ptq. concerning the identification and protection of outlier values, gpt3.int8() @xcite (also known as llm.int8()) identifies outliers by magnitude while spqr @xcite employs hessian matrix to identify outlier values. by equivalently scaling weights and activation values, smoothquant @xcite greatly reduces quantization error of activation and thus results in a great reduction in the quantization loss of the model. outlier suppression+ @xcite suppresses the outlier of weights by performing channel-wise shift and scale. qlora @xcite proposes to use the nf4 data format to reduce quantization rounding errors further. optq @xcite (generally known as gptq) adjusting the weights during the quantization process to reduce quantization errors. ## evaluation protocol the comprehensive evaluation of llms presents a long-standing challenge due to their versatility, widespread application, and poor explainability. to address this, we propose a structured evaluation framework that encompasses three critical dimensions: (1) knowledge and capacity, (2) alignment, and (3) efficiency. for the evaluation of knowledge and capacity, we consider two types of benchmarks: (i) those requiring llms to demonstrate extensive knowledge across various domains to achieve satisfactory performance, and (ii) those assessing the ability of llms to perform specific tasks such as language generation and understanding. in this context, we employ the mmlu @xcite and c-eval @xcite benchmarks for the former, covering diverse subjects including but not limited to history, chemistry, and economics. for the latter, we select the @xcite , cnn/dailymail @xcite , and xsum @xcite benchmarks, which focus on essential language generation tasks like translation and summarization, and the gsm8k @xcite and @xcite benchmarks for evaluating language understanding and reasoning capabilities. @xcite and cnn/dailymail @xcite benchmarks. the "average hsr/ssr/csl" represents the mean hard satisfaction rates (hsr), soft satisfaction rates (ssr) across five difficulty levels, and consistent satisfaction levels (csl) across five constraints on the followbench benchmark @xcite . the "average bias score" is shown as x/y, where x and y represent the mean bias score across various categories in ambiguous and unambiguous contexts, respectively, on the bbq benchmark @xcite . the "average perplexity" indicates the mean perplexity on wikitext2 @xcite , c4 @xcite , and ptb @xcite . the "memory" refers to the memory consumed (in gb) during inference when the input consists of 256 tokens and the output contains 512 tokens. the "speed" represents the number of tokens generated per second when the input consists of 256 tokens and the output contains 512 tokens. ## experiment results and discussion llms with a greater number of parameters generally outperform those with fewer parameters across most benchmarks. furthermore, we observe a downward trend in the performance of these llms across most benchmarks when they are quantized to fewer bits. here are the detailed observations: perplexity is a reliable performance indicator for quantized llms on evaluation benchmarks. figure 7 : perplexity of qwen-chat series models and their quantized counterparts on the wikitext2 dataset @xcite . identifying and isolating outlier weights is crucial for spqr to effectively quantize llms to an extreme level of 2 bits. experimental results indicate a sharp decline in the performance of llms quantized to 2 bits by gptq, to the extent that they fail to produce coherent text. in contrast, llms quantized to 2 bits by spqr exhibit a relatively moderate performance across all evaluated benchmarks. spqr introduces two innovative strategies to enhance the performance of quantized llms, distinguishing it from gptq: (1) the adoption of an extremely small group size coupled with bilevel quantization, and (2) the isolation of unstructured outlier weights, maintaining these weights at a higher precision (16-bit) during computations. to study the impact of these strategies, we conducted benchmark metric pearson correlation coefficient mmul accuracy -0.892 c-eval accuracy -0.930 flores-200 bleu (english to chinese) -0.884 bleu (chinese to english) -0.904 xsum rouge-1 -0.768 rouge-2 -0.493 rouge-l -0.222 cnn/dailymail rouge-1 -0.890 rouge-2 -0.849 rouge-l -0.885 gsm8k accuracy -0.911 snli accuracy -0.583 followbench hsr (hard satisfaction rates) -0.864 ssr (soft satisfaction rates) -0.899 csl (consistent satisfaction levels) -0.877 truthfulqa mc1 accuracy -0.789 bbq bias scores in ambiguous context -0.765 bias scores in disambiguated context 0.806 in practical scenarios, the application of lowbit quantization necessitates substantial engineering effort and hardware support. as illustrated in figure 8a , both the gptq and llm.int8() can effectively reduce memory consumption during llms inference, with the memory requirement diminishing as the number of quantized bits decreases. conversely, despite the impressive performance of spqr, it does not contribute to reducing memory consumption during llms infer- at similar levels of memory consumption, llms quantized to lower bit precision with a larger parameter scale can be preferred over llms with a smaller parameter scale, considering their performance capabilities. as illustrated in figure 8a , the memory consumption during inference for qwen-14b-chat with 8-bit or 4-bit quantization by gptq is similar to that of qwen-7b-chat. however, the former outperforms the latter in most of the benchmarks evaluated. additionally, qwen-14b-chat with 4-bit or 3-bit quantization can be competitive with qwen-7b-chat with 8-bit quantization. nonetheless, while quantized llms offer advantages in terms of memory efficiency, quantization can also result in reduced inference speed. therefore, these quantization approaches are most suitable for scenarios where memory is limited and inference speed is a secondary consideration. ## conclusion we have presented a comprehensive evaluation of quantization strategies for llms, demonstrating the trade-offs between model efficiency and performance degradation across various benchmarks. by employing a structured evaluation framework that assesses models in terms of knowledge & capacity, alignment, and efficiency, we aim to offer valuable insights into the scalability and practical application of quantized llms. experimental findings indicate that while 4-bit quantization maintains performance close to non-quantized counterparts, a notable performance discrepancy emerges as quantization decreases to 3 bits or lower. moreover, the results suggest that perplexity can be a reliable performance indicator for quantized llms on various evaluation benchmarks. spqr effectively quantizes llms to an extreme level of 2 bits by isolating outlier weights and maintaining high precision during computation. when memory constraints exist and inference speed is a secondary concern, llms quantized to lower bit precision with a larger parameter scale can be preferred over smaller models. additionally, we highlight the need for engineering effort and hardware support to efficiently deploy quantized llms in real-world scenarios.