On the Impact of Calibration Data in Post-training Quantization and Pruning (2024)

Abstract: Quantization and pruning form the foundation of compression for neural networks, enabling efficient inference for large language models (LLMs). Recently, various quantization and pruning techniques have demonstrated remarkable performance in a post-training setting. They rely upon calibration data, a small set of unlabeled examples that are used to generate layer activations. However, no prior work has systematically investigated how the calibration data impacts the effectiveness of model compression methods. In this paper, we present the first extensive empirical study on the effect of calibration data upon LLM performance. We trial a variety of quantization and pruning methods, datasets, tasks, and models. Surprisingly, we find substantial variations in downstream task performance, contrasting existing work that suggests a greater level of robustness to the calibration data. Finally, we make a series of recommendations for the effective use of calibration data in LLM quantization and pruning.

## introduction scaling is an essential component for unlocking new capabilities and improved performance in large language models (llms) @xcite @xcite . however, the pursuit of scale has led to models that demand significant energy and computational resources @xcite @xcite . consequently, llms can be challenging to deploy, especially in resource-constrained environments @xcite . these challenges have ultimately motivated a substantial body of research on model compression techniques, aiming to reduce computational demands while maintaining performance @xcite . quantization and pruning are two of the most popular model compression techniques @xcite . pruning aims to remove redundant weights, while quantization seeks to represent weights (and possibly activations) in lower precision. most recently, several quantization and pruning methods have demonstrated outstanding performance in a post-training setting @xcite @xcite . post-training compression techniques rely upon calibration data @xcite to determine the distribution of layer activations. this process requires only a small number of examples, with further examples offering diminishing gains @xcite . in the case of llms, the calibration set is routinely sampled from web text @xcite @xcite or model pre-training data @xcite . notably, the calibration examples are sampled randomly. this is because post-training model compression methods are considered robust to the specific distribution of calibration data @xcite @xcite . in this paper, we present the first empirical study on the impact of calibration data used in posttraining llm compression. we offer an extensive study with several quantization and pruning methods, across a range of tasks, datasets, and models. surprisingly, we find that downstream task performance can vary substantially according to the selected calibration data. this contrasts existing work, which suggests a high level of robustness. finally, we offer a series of recommendations for the effective use of calibration data. ## methodology our aim is to investigate the robustness of llm pruning and quantization methods to the calibration data. to this end, we experiment with four compression methods and nine llms (i.e. three different sizes from three model families). we vary only the calibration data, trialing five source datasets, each comprising ten distinct calibration sets. this provides a total of 1,800 compressed models. we then evaluate each model across 11 standard nlp tasks, resulting in a total of 19,800 model evaluations. ## results & analysis our principal aim is to investigate the impact of calibration data upon llm performance. to this end, we examine the model performance across calibration sets, their source datasets, and evaluation tasks. additionally, we explore trends in overall performance throughout the different configurations. ## conclusion in this paper, we presented the first extensive empirical study on calibration data for llm quantization and pruning. we examined the downstream task performance across a variety of models, compression methods, and calibration data sources. our results suggest that calibration data can substantially influence the performance of compressed llms. we supplement our findings with several recommendations for the effective use of calibration data. we hope that our work will inspire further research surrounding the use of calibration data in llm compression, an area which has seen limited attention. for future work, we are particularly interested in exploring how choices in the training protocol may influence the sensitivity to calibration data in llm compression @xcite .