I ntact KV : Improving Large Language Model Quantization by Keeping Pivot Tokens Intact (2024)

Abstract: Large language models (LLMs) excel in natural language processing but demand intensive computation. To mitigate this, various quantization methods have been explored, yet they compromise LLM performance. This paper unveils a previously overlooked type of outliers in LLMs. Such outliers are found to allocate most of the attention scores on initial tokens of input, termed as pivot tokens, which are crucial to the performance of quantized LLMs. Given that, we propose IntactKV to generate the KV cache of pivot tokens losslessly from the full-precision model. The approach is simple and easy to combine with existing quantization solutions with no extra inference overhead. Besides, IntactKV can be calibrated as additional LLM parameters to boost the quantized LLMs further with minimal training costs. Mathematical analysis also proves that IntactKV effectively reduces the upper bound of quantization error. Empirical results show that IntactKV brings consistent improvement over various quantization methods across different LLMs and downstream tasks, leading to the new state-of-the-art for LLM quantization. The codes are available at https://github.com/ruikangliu/IntactKV.

## introduction large language models (llms) have achieved remarkable progress in various tasks and benchmarks in natural language processing @xcite @xcite . nonetheless, the rise of llms also increases computational intensity and memory requirements. this motivates various research to decrease the inference cost of llms, e.g., quantizaiton @xcite @xcite , pruning @xcite @xcite ., * † corresponding author. in this paper, we discover a new type of outlier that is overlooked by previous quantization methods. these outliers exhibit extremely high values at only the [bos] and some other common tokens (e.g., "," and ".") at the beginning of the input, which is referred to as pivot tokens. we find the extreme values of these outliers make the self-attention concentrate on the pivot tokens, leaving the rest of the tokens untouched. this is also known as attention sinks @xcite , which is critical to the model performance @xcite . the effect of quantization on these pivot tokens should be carefully studied to improve the quantized llms. towards that end, we are motivated to propose intactkv, a simple strategy that is orthogonal to most existing quantization solutions. the key idea behind intactkv is to generate the lossless kv cache of pivot tokens from the full-precision model. by keeping the kv cache of pivot tokens intact, quantization error accumulated on the output of self-attention will be effectively alleviated in the rest of the decoding steps. the integration empirical results show that intactkv consistently improves the capability of different quantization methods (e.g. awq @xcite , gptq @xcite , omniquant @xcite and quarot (ashkboos et al., 2024)) on various open-sourced llms (e.g., llama and vicuna) across different tasks and benchmarks such as ppl, mmlu, commonsense qa, and mt-bench, achieving new state-of-the-art results for weightonly quantization as well as weight and activation quantization, e.g., lossless int4 weight-only quantization for vicuna-v1.3-13b on commonsense qa tasks. moreover, calibrating intactkv with int4 quantization even matches the full-precision model on aligning with human preferences, as evaluated by gpt-4 (bubeck et al., 2023) on mt-bench. ## method in this section, we introduce intactkv, a simple and easy-to-implement method to improve the quantized llms. the key idea behind this is to keep the kv cache of the pivot tokens intact, i.e., without any distortion raised by quantization. an overview of our method can be found in figure 3 . ## conclusions in this paper, we propose intactkv, a simple and easy-to-combine method to improve large language model quantization. the research is motivated by the previously overlooked outliers over pivot tokens, which lead to attention sinks that are critical to the performance of quantized llms. by generating intactkv with the full-precision model, the quantization error accumulated over the attention scores can be effectively alleviated. intactkv can also be calibrated as additional parameters to the llm backbone, further improving the quantized llms. experiments show that combining the proposed intactkv gives consistent improvement on various sizes of llms and across multiple downstream tasks, leading to new state-of-the-art results for large language model quantization.