A Survey on Model Compression for Large Language Models (2024)

Abstract: Large Language Models (LLMs) have transformed natural language processing tasks successfully. Yet, their large size and high computational needs pose challenges for practical use, especially in resource-limited settings. Model compression has emerged as a key research area to address these challenges. This paper presents a survey of model compression techniques for LLMs. We cover methods like quantization, pruning, and knowledge distillation, highlighting recent advancements. We also discuss benchmarking strategies and evaluation metrics crucial for assessing compressed LLMs. This survey offers valuable insights for researchers and practitioners, aiming to enhance efficiency and real-world applicability of LLMs while laying a foundation for future advancements.

## introduction large language models (llms) @xcite @xcite @xcite refer to transformer language models that contain billions (or more) of parameters, which are trained on massive text data. llms consistently exhibit remarkable performance across various tasks, but their exceptional capabilities come with significant challenges stemming from their extensive size and computational requirements. for instance, the gpt-175b model @xcite , with an impressive 175 billion parameters, demands a minimum of 350gb of memory in half-precision (fp16) format. furthermore, deploying this model for inference necessitates at least five a100 gpus, each featuring 80gb of memory, to efficiently manage operations. to tackle these issues, a prevalent approach known as model compression @xcite offers a solution. model compression involves transforming a large, resource-intensive model into a compact version suitable for deployment on resource-constrained devices. additionally, model compression can enhance llm inference speed and optimizes resource efficiency. in our paper, our primary objective is to illuminate the recent strides made in the domain of model compression techniques tailored specifically for llms. our work conducts an exhaustive survey of methodologies, metrics, and benchmarks of model compression for llms. figure 1 shows the taxonomy of model compression methods for llms, including quantization, pruning, knowledge distillation, and low-rank factorization. figure 2 further shows basic flow of these model compression methods for llms. furthermore, our study sheds light on prevailing challenges and offers a glimpse into potential future research trajectories in this evolving field. we advocate for collaborative efforts within the community to pave the way for an ecologically conscious, all-encompassing, and sustainable future for llms. while there were previous surveys on neural networks model compression @xcite and it has been lightly discussed in prior surveys on lms @xcite and llms @xcite , our work is the inaugural survey dedicated solely to model compression for llms. ## quantization quantization @xcite refers to the process of reducing the number of bits (i.e., precision) in the parameters of the model with minimal loss in inference performance. quantization can be categorized into two main approaches: quantization-aware training (qat), and post-training quantization (ptq). the primary distinction between the two approaches lies in ## knowledge distillation knowledge distillation (kd) @xcite is a technique aimed at transferring knowledge from a large and complex model (i.e., teacher model) to a smaller and simpler model (i.e., student model). we classify these methods into two clear categories @xcite : black-box kd, where only the teacher's outputs are accessible, typically from closed-source llms, and whitebox kd, where the teacher's parameters or output distribution are available, usually from opensource llms. ## conclusion in this survey, we have explored model compression techniques for llms. our coverage spanned compression methods, metrics, and benchmark datasets. by diving into llm compression, we've highlighted its challenges and opportunities. this survey aims to be a valuable reference, providing insights into the current landscape and promoting ongoing exploration of this pivotal topic.