Self-calibration for Language Model Quantization and Pruning (2025)

Abstract: Quantization and pruning are fundamental approaches for model compression, enabling efficient inference for language models. In a post-training setting, state-of-the-art quantization and pruning methods require calibration data, a small set of unlabeled examples. Conventionally, this is randomly sampled web text, aiming to reflect the model training data. However, this poses two key problems: (1) unrepresentative calibration examples can harm model performance, and (2) organizations increasingly avoid releasing model training data. In this paper, we propose self-calibration as a solution. Our approach requires no external data, instead leveraging the model itself to generate synthetic calibration data, with a view to better approximating the pre-training data distribution. We extensively compare the performance of self-calibration with several baselines, across a variety of models, compression methods, and tasks. Our approach proves consistently competitive in maximizing downstream task performance, frequently outperforming even using real data.

## introduction large language models (llms) trained using vast corpora have delivered remarkable advances across a variety of domains and tasks @xcite @xcite . however, they demand extensive computational resources for inference @xcite , presenting a limiting factor for their practical use. consequently, this has prompted the development of an extensive collection of methods to improve inference efficiency @xcite . in particular, model compression aims to reduce the size of a model while retaining downstream task performance @xcite . quantization and pruning have emerged as prominent model compression approaches for llms @xcite . pruning removes less important weights from the model, while quantization represents the weights (and possibly activations) using fewer bits. both quantization and pruning can be effectively applied in a post-training setting, retaining comparable performance across a range of downstream tasks @xcite @xcite . post-training quantization and pruning typically depend upon calibration data, a small set of unlabeled examples @xcite used to generate layer activations throughout the model. conventionally, llm calibration data consists of randomly sampled web text @xcite @xcite , aiming to reflect the model training data distribution. however, recent work has questioned the influence of calibration data in llm compression. @xcite hint that the careful selection of calibration data may benefit high-sparsity pruning. concurrently, @xcite illustrate the impact of calibration data in quantization and pruning. finally, @xcite and @xcite highlight the role of languagespecific calibration data for multilingual models. to further complicate matters, organizations are increasingly reluctant to release model training data or disclose necessary replication details. in this paper, we propose self-calibration as a solution to concerns surrounding the availability and quality of calibration data. our approach removes the need for external calibration data sources, instead leveraging the model itself to automatically generate synthetic calibration data. we compare our approach to various real and synthetic datasets, including data sampled from a large mixture-ofexperts model. our approach is consistently competitive in maximizing the performance of compressed models, across a variety of models and compression methods. in many cases, we find that self-calibration can outperform even real data. ## self-calibration when the exact training data for a model is unavailable, sampling calibration data from an alternative distribution offers an approximation at best. even if the exact training data is available, individual examples may be noisy and deviate from the overall distribution. to address these limitations, we pro-pose self-calibration, a general-purpose adaptation to model compression that relies on calibration data from the model itself. our hypothesis is that sampling from the learned posterior distribution, which approximates the training data, offers more representative calibration examples. in turn, we expect that such calibration examples will enable greater preservation of downstream task performance following model compression. ## results self-calibration outperforms other synthetic datasets. we observe that the performance of self-calibration matches or exceeds other synthetic datasets in 17 out of 20 instances. for example, when quantizing gemma 2b with gptq, selfcalibration records a mean accuracy of 59.9%, compared to 58.5% with cosmopedia and 57.9% with vocabulary. similarly, when pruning llama 3.1 8b with sparsegpt, self-calibration offers a 2.9 point increase in mean accuracy compared to cosmopedia (53.8% versus 50.9%). this suggests that self-calibration may produce calibration data that is more representative of the training distribution of each model, compared to other synthetic datasets. self-calibration can outperform real-world data. opt 6.7b, self-calibration achieves the highest mean accuracy compared to all other datasets in all but one instance. the only exception is when pruning opt 6.7b with sparsegpt, where selfcalibration ranks second to c4 (52.7% with selfcalibration versus 52.8% with c4). although selfcalibration does not outperform real data for mistral 7b and llama 3.1 8b, we observe that the performance is as competitive with real data as cosmopedia (i.e. matches or outperforms cosmopedia in five out eight instances). these outcomes suggest that using self-calibration for model compression results in downstream performance that is at least comparable to that of real data. ## conclusion in this paper, we proposed self-calibration for llm quantization and pruning as a solution to mitigate concerns about the availability, quality, and representativeness of training data. our proposed approach is intuitive and requires no external data sources, instead relying on the model itself. we empirically demonstrated that self-calibration maintains comparable or greater downstream task performance across a variety of models and compression methods. surprisingly, our results also revealed that self-calibration can enable higher downstream task performance than using real data. we hope that our study will inspire further work on the application of synthetic data to llm compression.