The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models (2023)

Abstract: Compressing large language models (LLMs), often consisting of billions of parameters, provides faster inference, smaller memory footprints, and enables local deployment. The standard compression techniques are pruning and quantization, with the former eliminating redundant connections in model layers and the latter representing model parameters with as little as 4 bits. The key tradeoff is between the degree of compression and the impact on the quality of the compressed model. Existing research on LLM compression primarily focuses on performance in terms of general metrics like perplexity or downstream task accuracy. More fine-grained metrics, such as those measuring parametric knowledge, remain significantly underexplored. To help bridge this gap, we present a comprehensive analysis across multiple model families using the LAMA and LM-Harness benchmarks in order to systematically quantify the effect of commonly employed compression techniques on model performance. A particular focus is on tradeoffs involving parametric knowledge, with the goal of providing practitioners with practical insights to make informed decisions on compression.

## introduction large language models (llms) have demonstrated exceptional performance across diverse tasks. however, their deployment in real-world applications is hindered by their substantial size and the associated costs, even for inference @xcite . for instance, the llama-65b model @xcite , a pioneering open-sourced llm, uses approximately 130gb of ram for 16-bit inference. to address this challenge, recent research has focused on developing novel compression techniques that enable efficient local deployment and inference. notable examples of such techniques include sparsegpt @xcite and llm.int8() @xcite . the tradeoff between model compression and quality is typically studied either through general metrics like perplexity @xcite or standardized benchmark task accuracy @xcite on, e.g., glue @xcite . furthermore, much of the literature studies such tradeoffs for one model or a particular class of models. unfortunately, as a result, practitioners do not have access to reliable insights or rules-of-thumb to ensure they can make an informed decision for compression in their own models. this is because • metrics like perplexity are too general, while benchmark prediction metrics are too easy to fool. for example, recent findings suggest that distilled versions of foundational llms, known as imitation models, may exhibit stylistic similarities but potentially lack knowledge when compared to the models they seek to imitate @xcite . these difficulties suggest that there is a need for a more fine-grained understanding of the effects of compression schemes, comparing a variety of model families, compression techniques, and specialized measurements. we address these challenges, specifically focusing on the preservation of parametric knowledge, i.e., knowledge acquired during pretraining, that is stored in model weights. this is particularly crucial for tasks involving reasoning and for specialized applications. concretely, we examine the impact of different compression schemes on parametric knowledge across multiple model families (encoder, encoder-decoder and de-coder) where we apply pruning and quantization approaches and analyze the performance of such techniques on downstream reasoning tasks. to the best of our knowledge, this work represents one of the first large-scale investigations in this direction. among the crucial observations resulting from this study include: ## background in this section, we briefly discuss the various compression techniques we use in our study. ## experimental setup in this section, we present a comprehensive overview of our experimental setup, including the rationale behind our design choices, along with the selection of models and datasets. ## experimental results and insights to facilitate our discussion, we categorize pruning levels as follows: ## related work early works seeking to understand large language model behavior focused on contextual representations and how such models gain linguistic capabilities @xcite @xcite . more recently, some lines of work have steered towards understanding how these models acquire factual and commonsense knowledge. techniques such as probing evolved as a way to understand the knowledge capabilities of these models @xcite @xcite @xcite . previous works including gordon et al. (2020); @xcite pruned bert and showed that it is resilient to a medium level of pruning. for example, @xcite showed that after finetuning for a particular downstream task, it is possible to prune about 40% of the attention weights without any loss in performance. a particular fo-cus has been to understand the importance of the attention mechanism @xcite by pruning the heads. in a similar fashion, works such as zafrir et al. (2019); bai et al. (2020); zadeh et al. (2020); tao et al. (2022); prato et al. (2019); frantar et al. (2022); dettmers et al. ( 2023 ) pushed the limits of quantization on language models. most of these works have focused on one model class or a particular metric. in another line of work, a variety of approaches @xcite @xcite focus on alternatives to traditional finetuning of the model due to its scale. in contrast to these works, our paper primarily focuses on the in-built parametric knowledge present in the model. this means we do not finetune and instead seek to understand whether some of the previously described phenomenona are applicable to other models as well. also connected to this work are techniques that edit factual knowledge in models. the goal for such works is to avoid retraining or even finetuning models, instead seeking to directly change parameters connected to certain facts @xcite @xcite . however, given our focus on compression, the main theme of our work differs. nevertheless, it would be interesting to understand the impact of relying on compressed models when using such editing techniques. ## conclusion compression is crucial in deploying and using large language models. despite its importance, existing empirical studies predominantly rely on generic measurements such as perplexity or standardized benchmark metrics when investigating the effects of compression. these coarse measurements are challenging to interpret. as a result, it is difficult to use them to develop meaningful heuristics for practitioners. to address these limitations, we provided a large-scale study that focused on fine-grained effects on quantities like parametric knowledge. we studied a variety of compression choices across multiple model families, providing usable insights into what types of compression schemes have the least and most significant impact on models. we hope this work serves as a useful step towards developing users' intuition for rules-of-thumb when selecting appropriate compression techniques in large language models. for future work, we hope to add additional, more specialized techniques for large language model compression.