C lus C omp: A Simple Paradigm for Model Compression and Efficient Finetuning (2025)

Abstract: As large language models (LLMs) scale, model compression is crucial for edge deployment and accessibility. Weight-only quantization reduces model size but suffers from performance degradation at lower bit widths. Moreover, standard finetuning is incompatible with quantized models, and alternative methods often fall short of full finetuning. In this paper, we propose ClusComp, a simple yet effective compression paradigm that clusters weight matrices into codebooks and finetunes them block-by-block. ClusComp (1) achieves superior performance in 2-4 bit quantization, (2) pushes compression to 1-bit while outperforming ultra-low-bit methods with minimal finetuning, and (3) enables efficient finetuning, even surpassing existing quantization-based approaches and rivaling full FP16 finetuning. Notably, ClusComp supports compression and finetuning of 70B LLMs on a single A6000-48GB GPU.

## introduction large language models (llms) have garnered significant acclaim and success across various domains and applications @xcite @xcite . with ongoing advancements, the scope and complexity of released llms have witnessed exponential growth, with some llms @xmath0. this remarkable upscaling introduces considerable challenges, particularly when deploying these models or granting their accessibility to users with constrained resources. to address these challenges, weight-only post-training quantization (ptq) has emerged as an effective approach. ptq methods can generally be classified into three categories: statistic-based, gradient-based, and codebook-based approaches. statistic-based methods @xcite @xcite determine the quantization grid based on the weight value distribution, whereas gradient-based methods @xcite optimize the quantization grid with some calibration samples. codebook-based methods @xcite @xcite cluster similar weight elements to the shared quantized centroids, employing non-uniform quantization and pushing the limits to extremely low bit levels. however, these methods continue to struggle with low-bit quantization and the presence of outliers in weights, leading to significant performance degradation. another challenge ptqs encounter is their limited support for finetuning, which is crucial for adapting llms to various downstream tasks. finetuning llms is computationally expensive due to their large scale and the need to cache activations and store optimizer states. ptq, which compresses llms, appears to be a promising approach for finetuning as it reduces the memory requirements for loading llms. however, most quantization techniques use a round-to-nearest operation, which does not support gradient backpropagation. typically, parameter-efficient methods @xcite @xcite are employed to train the added parameters while keeping the quantized llms frozen, bypassing this limitation. however, this fine-tuning approach presents two major drawbacks: (1) freezing quantized llms prevents further reduction of quantization errors during finetuning; (2) the low-rank nature of most parameter-efficient methods restricts their expressiveness @xcite . in this paper, we propose a simple while effective paradigm that mainly applies clustering to compress llms, referred to as cluscomp. additionally, cluscomp can function as a parameter and memory-efficient finetuning method. our preliminary experiments reveal that llms are increasingly difficult to quantize, mainly due to the growing frequency of outliers in their weight matrices ( §3). based on this observation, we propose using clustering instead of quantization to compress llms, retaining all values in fp16 format to circumvent issues arising from outlier quantization ( §3.1). to further reduce compression errors, we minimize block-wise output discrepancies between the compressed and uncompressed blocks, using a limited set of calibration samples ( §3.3). in addition, by incorporating an inexpensive end-to-end recovery finetuning step, we can even push the compression rate to the 1-bit level. since all parameters remain in fp16 after compression, cluscomp fully supports standard neural training, thus allowing the seamless finetuning of compressed llms on various downstream tasks ( §3.4). we begin by evaluating the effectiveness of clus-comp in the context of model compression across 2 language modeling tasks and 6 zero-shot reasoning tasks. cluscomp consistently surpasses various baselines at 2-4 levels, even achieving a perplexity @xmath1.1). following recovery finetuning, cluscomp's performance at 2-bit and 1-bit levels approaches that of the fp16 model, with an accuracy of 57.8 vs 68.6 for the 2-bit llama-3-8b and 51.4 vs 75.4 for the 1-bit llama-3-70b ( §4.2). additionally, cluscomp demonstrates its utility as a parameterefficient (< 1%) and memory-efficient (42gb for llama-3-70b) finetuning method, outperforming quantization-based and memory-efficient finetuning approaches, while matching the performance of full finetuning ( §4.3). ## related works quantization refers to the process of converting floating-point values into discrete levels, thereby reducing the bit-width required and minimizing memory consumption during model loading. taking the symmetric uniform quantization as an example, a weight matrix w is quantized as follows: post-training quantization (ptq) methods, such as gptq @xcite , awq @xcite , and omniquant @xcite , apply quantization to a model after training with minimal computational resources. however, these approaches, which rely on uniform quantization, are significantly impacted by the presence of outliers in the weight matrices. recent methods @xcite left & middle: from llama-2 to llama-3, all methods struggle more with lower-bit quantization, though cluscomp -is the least affected. right: from llama-1 to llama-3, the average weight kurtosis across most layers increases. et al., 2024a) address this challenge by retaining salient weights in fp16 format, thereby maintaining strong performance at lower bit widths. nonetheless, these mixed-precision quantization techniques require specially optimized cuda kernels to either enhance or preserve inference speed. closely related to our proposed method, cluscomp, are works such as gptvq (van baalen et al., 2024), quip# (tseng et al., 2024) and squeezellm (kim et al., 2024) which implement quantized codebooks for non-uniform quantization, achieving state-ofthe-art performance for ultra-low-bit quantization. cluscomp, however, differs in two significant ways: (1) the codebook in cluscomp is stored in fp16, offering additional advantages for subsequent recovery training and finetuning; (2) while other methods face limitations similar to those in vae-like approaches (kingma and welling, 2014), where large and high-dimensional codebooks are infeasible due to mode collapse, cluscomp circumvents this issue. our fixed-code design allows us to utilize a codebook size of 2 16 in 4-16d without encountering such difficulty. finetuning is crucial for adapting llms to various domains and applications. quantization, which reduces model size, is theoretically more conducive to finetuning. however, directly finetuning a quantized model is not a standard approach, as rtn does not support gradient backpropagation. finetuning using a straight-through estimator (ste) @xcite is relatively underexplored and may lead to catastrophic forgetting @xcite . previous works @xcite @xcite propose freezing the quantized model while updating newly added loras @xcite . however, these approaches suffer from two key limitations: (1) freezing the quantized model prevents the mitigation of quantization errors during finetuning. moreover, not all quantization methods are suitable for finetuning. popular techniques like gptq and qlora, while widely used, exhibit significant quantization errors below 4-bit. (2) the expressiveness of lora is constrained by its bottleneck design @xcite . in contrast, cluscomp, where all parameters are maintained in high precision, inherently supports seamless finetuning. additionally, updating the codebook in cluscomp results in modifying all parameters in the weight matrices, even offering superior performance compared to full finetuning while maintaining a similar number of trainable parameters as lora. we hypothesize that the increasing quantization difficulty stems from a higher frequency of outliers in llama-3's linear layers. since these methods rely on uniform quantization, they are particularly sensitive to outliers in weight matrices. to test this, we analyzed the kurtosis of weight matrices-a well-established metric for detecting outliers @xcite . as shown in figure 2 (right), we observe: (1) higher kurtosis at the beginning and end of all models; (2) a consistent increase in kurtosis from llama-1 to llama-3 across most layers, indicating more frequent outliers. this trend likely explains the growing quantization challenge and suggests that future llama models may be even harder to quantize. 1 since quantization performance is affected by outliers, could an alternative compression approach involve storing all weight values in fp16 rather than applying quantization? the first idea that comes to mind is clustering, where similar weight values are represented by a single shared value. this approach enables model compression while maintaining all weight values in fp16. in the following, we introduce three cluscomp variants that leverage clustering for llm compression: cluscomp -, which applies clustering alone; cluscomp, which enhances cluscomp -with block-wise error minimization; and cluscomp + , which further refines the compressed model through next-token prediction. ## conclusion the newly introduced model compression technique, cluscomp, operates by (1) independently applying clustering to the weight matrices to produce both the codebook and corresponding codes, (2) reducing compression error through blockwise knowledge distillation, and (3) enhancing model performance via efficient recovery finetuning. comprehensive experiments demonstrate its effectiveness as a compression method at 1-4 bit levels, while also showcasing its parameter and memory efficiency for finetuning, with a competitive performance with full finetuning.