LLMC : Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit (2024)

Abstract: Recent advancements in large language models (LLMs) are propelling us toward artificial general intelligence with their remarkable emergent abilities and reasoning capabilities. However, the substantial computational and memory requirements limit the widespread adoption. Quantization, a key compression technique, can effectively mitigate these demands by compressing and accelerating LLMs, albeit with potential risks to accuracy. Numerous studies have aimed to minimize the accuracy loss associated with quantization. However, their quantization configurations vary from each other and cannot be fairly compared. In this paper, we present LLMC, a plug-and-play compression toolkit, to fairly and systematically explore the impact of quantization. LLMC integrates dozens of algorithms, models, and hardware, offering high extensibility from integer to floating-point quantization, from LLM to vision-language (VLM) model, from fixed-bit to mixed precision, and from quantization to sparsification. Powered by this versatile toolkit, our benchmark covers three key aspects: calibration data, algorithms (three strategies), and data formats, providing novel insights and detailed analyses for further research and practical guidance for users. Our toolkit is available at https://github.com/ModelTC/llmc.

## introduction recently, llms such as @xcite have demonstrated unprecedented generative capabilities in the field of natural language processing (nlp) and also achieved widespread applications. however, their substantial computational and storage costs have impeded their further popularization among users. for instance, bloom @xcite , a multilingual llm with 176 billion parameters, requires a minimum of 350 gb space * equal contribution. to store model weights in full-precision (fp16) format. even worse, it requires 5×80gb a100 or 9×40gb a800 nvidia gpus to perform inference. therefore, reducing llms' serving cost is paramount to further enhance their application. for the aforementioned challenge, model quantization @xcite can be an effective solution. it maps weights and/or activations to a lower-bit data format to reduce memory footprint and accelerate model inference. existing quantization approaches can be categorized into two types: quantization-aware-training (qat) @xcite @xcite and posttraining quantization (ptq) @xcite . although with prominent high performance, the necessity for qat to undergo finetuning or retraining with substantial training data and training costs renders it unattainable for the majority of users. correspondingly, ptq compresses models without retraining, making it a preferred method for llms due to its minimal resource requirements. therefore, we do not mention some qat methods @xcite @xcite in this paper. however, current ptq methods always evaluate across distinct datasets in different quantization configurations and with simulated quantization. for example, awq @xcite employs pile (val) @xcite as calibration data, instead of @xcite in gptq @xcite . this situation would cause an inaccurate assessment of configurations for efficient and accurate llm quantization. to provide a comprehensive options menu for users and directions with insights for further research, we make a fair benchmark, which considers three key dimensions, e.g., calibration data, algorithms, and data formats. first, we systematically explore the effect of calibration data for higher model performance. then, we aim to investigate the effectiveness and underlying mechanisms of three primary algorithm strategies: transformation, clipping, and reconstruction. finally, we probe how to select types between the integer and float-point quantization for further accuracy improvements. all the aforementioned studies benefit from our llmc, a user-friendly, plug-and-play llm compression toolkit. this toolkit incorporates several distinct traits, as demonstrated in figure 1 , offering users the freedom to select options that best suit their needs. in a word, our main contributions can be described as follows: ## llmc: a versatile llm compression toolkit first and foremost, we have developed a comprehensive toolkit named llmc for llm compres-sion, characterized by the following key features, which are also exhibited in figure 1 . high extensibility. the toolkit is highly modular and extensible, allowing easy adaptation foot_0 from integer quantization to floating-point quantization, from llms to vlms @xcite , from quantization to sparsification, and from dense models to mixture-of-expert (moe) models @xcite . this modularity ensures users can extend and customize the toolkit to meet their needs. comprehensive evaluation. llmc enables comprehensive evaluation of quantized models, providing detailed performance metrics and analysis, e.g., ppl @xcite , and data visualization analysis, e.g., kurtosis value, quantization error, and outlier distribution. this thorough evaluation capability ensures that users can make informed decisions about the best quantization strategies for their models. ## benchmarking llm quantization powered by llmc toolkit, we explore the quantization of llms from three distinct perspectives: the calibration data in subsection 3.2, the algorithms in subsection 3.3, and the data format of quantization in subsection 3.4. more explorations, e.g., extendability of llmc, kv cache quantization, and inference speed can be found in the appendix. ## additional results and discussions impact of quantization for fine-tuning. we conduct experiments for quantization on llama-3-8b with supervised fine-tuning (sft) on evol-instruction-66k 10 to analyze the impact. we choose ms-swift @xcite as the finetuning framework. additionally, we set the learning rate to 2e-6 with a mini-batch size of 2 and trained the model for 1 epoch on 16 40g a800 gpus. after fine-tuning, we employ w4a16 naive quantization and awq to quantize the model. we choose humaneval @xcite and humaneval-x @xcite for evaluation. as illustrated in ## conclusion this paper introduces llmc, a user-friendly and versatile toolkit for llm compression. supported by the toolkit, a series of observations and analyses were conducted, providing valuable and novel insights and suggestions for the community. content 1 introduction 1 2 llmc: a versatile llm compression toolkit 2 3 benchmarking llm quantization 3 3.1 experimental settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3.2 impact of calibration data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3.3 dive into the quantization algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3.3.1 how does transformation influence activation and weight outlier? . . . . . . 4 3.3.2 when should we utilize the weight clipping? . . . . . . . . . . . . . . . . . . 4 3.3.3 should we combine transformation and reconstruction? . . . . . . . . . . . . 5 3.4 integer or floating-point quantization? . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4 additional results and discussions 6 5 conclusion 7 a appendix 12 a.1 preliminary for quantization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 a.2 more implementation details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 a.3 ppl alignment with the existing methods . . . . . . . . . . . . . . . . . . . . . . . . . 13 a.4 kv cache quantization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 a.5 extensibility of llmc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 a.6 inference speed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 a.7 detailed accuracy & ppl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 a.8 results for various model families . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 a appendix technique approach strategy eq. trans. algorithm transformation rule-based @xmath0, γ = 0.5, 0.75, ... ✓ smoothquant(xiao et al., 2023) q, where qq @xmath1., 2024) search-based @xmath2, grid search for γ ∈ [0, 1] ✓ awq(lin et al., 2023) @xmath3.0, max(x)/t), grid search for t ✓ os+(wei et al., 2023b) learnining-based @xmath4, s ← sη ∂l(s) ∂s ✓ omniquant(shao et al., 2023) clipping rule-based α = 1, β = 1 ✓ smoothquant(xiao et al., 2023), os+(wei et al., 2023b), gptq(frantar et al., 2022), quarot (ashkboos et al., 2024) search-based grid search for α = β ∈ [0, 1] ✗ awq(lin et al., 2023) @xcite