When Quantization Affects Confidence of Large Language Models? (2024)

Abstract: Recent studies introduced effective compression techniques for Large Language Models (LLMs) via post-training quantization or low-bit weight representation. Although quantized weights offer storage efficiency and allow for faster inference, existing works have indicated that quantization might compromise performance and exacerbate biases in LLMs.This study investigates the confidence and calibration of quantized models, considering factors such as language model type and scale as contributors to quantization loss.Firstly, we reveal that quantization with GPTQ to 4-bit results in a decrease in confidence regarding true labels, with varying impacts observed among different language models. Secondly, we observe fluctuations in the impact on confidence across different scales. Finally, we propose an explanation for quantization loss based on confidence levels, indicating that quantization disproportionately affects samples where the full model exhibited low confidence levels in the first place.We make our code and quantized models publicly available.

## introduction large language models (llms) are widely used in a variety of natural language generation applications @xcite @xcite @xcite . llms have been proven to achieve high performance in zero and few-shot prompting, providing results on par with fine-tuned baselines, especially in commonsense reasoning tasks @xcite @xcite . @xcite show that emerging abilities come with the scale increase, which makes wellperforming larger models less accessible and limits their practical usability. a range of efficient compression and acceleration methods, including quantization, have been developed that help to alleviate high latency and extensive storage demands @xcite . despite its efficacy as a compression technique, recent works show that quantization may degrade the initial performance and amplify the sensitivity of an llm to certain linguistic phenomena and stereotypes @xcite . however, less attention has been paid to explaining the compression loss, particularly its variance across different texts. in this paper, we extend the existing research on the compression loss estimation; in particular, we measure the impact of quantization on the confidence of llms that can be initially overconfident in both right and wrong predictions @xcite @xcite . our main contributions are the following: (i) we investigate how quantization with gptq @xcite influences the calibration and confidence of llms, (ii) we assess the confidence alignment between compressed and full llms at scale, (iii) we explain the quantization loss from the initial confidence perspective. our null hypothesis is that the compressed vs. full predictive probability distributions are indistinguishable since prior work discussed a negligible accuracy drop in performance after quantization @xcite @xcite . we analyze the relationship between models by comparing calibration scores-indicating a model's ability to accurately reflect true probabilities-before and after quantization. to the best of our knowledge, our research is the first attempt to explain the quantization loss through the lens of predictive probabilities. ## related work the pretrained knowledge embedded in very large models has paved the way to parameter-efficient adaptation for downstream tasks, such as prompting and few-shot learning, bypassing the necessity for fine-tuning @xcite . the inference of llms can be accelerated through a low-bit representation of trained weights (quantization) and effective tensor slicing across multiple gpus (deepspeed @xcite , @xcite , inter alia). prior studies have estimated compression efficiency through: (1) latency-related measures determining throughput and a multiple of the original model's inference speed-up, (2) the precision of weights approximation, and (3) performance decrease (gap) @xcite @xcite . re-cent comparative studies on interpreting compression loss have indicated that compression amplifies biases and stereotypes, highlighting a disparate quantization loss in multilingual llms across different architectures @xcite . in contrast, another line of research suggests that compression enhances fairness @xcite . altogether, existing studies commonly measure compression loss by observing the deviation in performance before and after quantization. in this project, we adopt the recent gptq quantization method for compressing model weights and concentrate on the disparities between predictive probability distributions instead. for the first time, our approach reveals the relationship between the initial level of predictive confidence and quantization loss. ## methodology we follow jiang et al., 2020 and consider a classification problem where inputs to the model are questions x paired with candidate answers y to constitute concatenated sequences. the generative model then processes these concatenated questionanswer pairs to predict the most probable answer ŷ from the provided choices y for a given x: here, the probability of the token sequence y is derived as the product of individual token y [i] probabilities within the sequence, conditioned on x and the preceding tokens y [1:i-1] : for the entailment generation benchmarks, we use texts concatenated with possible completions as inputs to the model. we compare the quantized and full-precision models with the difference in the probabilities of the sequences p lm (y|x), further referred to as confidences. ## results we conduct a series of experiments to estimate the impact of quantization on various aspects of llms' performance, including calibration error, prediction entropy, cases of maximum confidence change, and the distribution dissimilarities between full and compressed models. we find variance in quantization impact across different families of models and their sizes, suggesting that scale and pre-training directly affect the further quantization loss. calibration impact confidence impact ## conclusion this paper investigates the impact of quantization on the confidence and calibration of llms. we demonstrate that quantization leads to an increase in calibration error and statistically significant changes in confidence levels for correct predictions. through a detailed examination of confidence shifts, we identify instances of confidence change occurring in data where models lack confidence before quantization. overall, our findings provide insights into quantization loss and suggest a potential direction for future work, emphasizing the need to focus on calibrating llms, specifically on uncertain examples. for example, future work may focus on integrating the models' calibration, such as temperature scaling, into the quantization pipeline. also, we have demonstrated that different model families, including llama, mistral, bloom, and opt, exhibit varying degrees of susceptibility to quantization, as measured by changes in confidence levels. this suggests another direction for future research -benchmarking llms based on their response to quantization-induced confidence shifts.