More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression (2025)

Abstract: As large language models (LLMs) process increasing context windows, the memory usage of KV cache has become a critical bottleneck during inference. The mainstream KV compression methods, including KV pruning and KV quantization, primarily focus on either token or precision dimensions separately. However, these works have left the trade-off between these two orthogonal dimensions largely unexplored. In this paper, we leverage the Information Bottleneck principle to formulate KV cache compression within a unified theoretical framework. We demonstrate that a carefully managed token-precision trade-off can achieve an optimal point within the Information Bottleneck compared to standalone KV pruning or KV quantization. Experiments reveal that storing more tokens in the KV cache at lower precision—a strategy we term quantized pruning—can significantly enhance the long-context performance of LLMs. An in-depth analysis of this token-precision trade-off across key aspects shows that quantized pruning achieves substantial improvements in retrieval-related tasks and consistently performs well across varying input lengths. Furthermore, quantized pruning exhibits notable stability and effectiveness across different KV pruning methods, quantization strategies, and model scales. These findings offer valuable insights into optimizing KV cache compression through balanced token-precision trade-off strategies. Our code isavailable at https://github.com/zhzihao/QPruningKV.

## introduction current long-context large language models (llms) heavily depend on the key-value (kv) cache mechanism, which stores intermediate keys and values during text generation to avoid redundant computations @xcite . as the input sequence length increases, the kv cache * corresponding author size expands proportionally, leading to substantial memory overhead @xcite @xcite . take llama3-8b @xcite as an example, storing the kv cache of a single sequence with 100k tokens necessitates a substantial memory overhead of 20gb -exceeding the model's parameter memory of approximately 14gb. moreover, since the decoding process is highly dependent on gpu memory bandwidth, the large kv cache also results in a significant increase in decoding time @xcite . as a result, efficiently compressing the kv cache has become a critical challenge in advancing llm development @xcite . to optimize kv cache memory usage, two primary approaches, focusing on token and precision dimensions, are widely adopted: kv pruning and kv quantization. for the token dimension, kv pruning methods reduce the kv cache footprint by discarding less salient tokens, thereby maintaining a fixed cache size @xcite @xcite . for the precision dimension, kv quantization methods reduce memory usage by approximating kv cache with lower precisions, like 4-bit or even lower @xcite @xcite . from the perspective of the information bottleneck (ib) principle, we propose a unified theoretical framework to describe kv pruning and kv cache quantization, and find an exclusive focus on optimizing either pruning or quantization in isolation may lead to a suboptimal trade-off between model performance and compression efficacy. in this paper, we demonstrate that a strategy cooptimizing both pruning and quantization aspects can achieve a better optimal operating point within the information-theoretic trade-off space than optimizing either aspect in isolation. specifically, we uncovered an intriguing finding: storing more tokens in the kv cache with lower precision, a strat-egy we term quantized pruning, can significantly enhance the long-context performance of llms. for instance, storing 4× tokens in 4-bit precision outperforms storing 1× tokens in 16-bit precision across various kv cache budgets. notably, in extremely low-resource scenarios, quantized pruning effectively preserves performance, whereas relying solely on kv pruning or quantization often leads to a significant performance collapse. furthermore, we conduct in-depth analysis of the tokenprecision trade-off across critical aspects, including task type, input length, model scaling, quantization strategies, and layer-wise configurations. extensive experiments reveal that quantized pruning achieves considerable gains in retrieval tasks and maintains robust performance across varying input lengths. moreover, quantized pruning demonstrates strong stability across different kv pruning methods, quantization strategies, and model scales. in summary, our contributions are as follows: • we are the first to describe kv pruning and kv cache quantization from a unified theoretical framework, and find an optimal trade-off between model performance and compression efficacy leveraging token-precision trade-off. kv quantization kv quantization compress kv cache from the precision dimension by storing kv cache using a reduced number of bits. flexgen @xcite utilizes group-wise 4-bit quantization for both key and value cache. kivi @xcite applies per-channel quantization on key cache and per-token quantization on value cache. kvquant @xcite and cq @xcite use rope-related quantization, while kvquant also preverses outliers without quantization. atom @xcite and skvq @xcite reorders the outlier channels for fine-grained group quantization with mixed-precision. qaq @xcite and mikv @xcite , inspired by kv pruning methods, store discarded tokens from kv pruning methods using lower bit precision while retaining salient tokens in full precision. however, their memory usage scales proportionally with context length. in contrast, quantized pruning, fundamentally based on kv pruning, theoretically enables compression to a pre-defined memory budget. other kv compression methods compressing kv cache from other dimensions typically requires modifying the model architecture, which usually necessitates additional training for adaptation. for the layer dimension, lckv @xcite , cla @xcite and mlkv @xcite reduce memory usage by sharing the kv cache across adjacent layers. shortgpt @xcite and dynamicslicing @xcite achieve compression by eliminating redundant layers. yoco @xcite changes the model structure and shares a single global kv cache across layers. for the head dimension, mqa @xcite and gqa @xcite share the kv cache within each head groups. deepseek-v2 @xcite employs dimension-reduction techniques to compress all heads into a single low-rank vector. ## preliminaries during inference, kv cache is implemented within the self-attention module and operates in two distinct phases: i) the prefill phase, where the input prompt is used to generate kv cache for each transformer layer of llms; and ii) the decoding phase, where the model uses kv cache to generate the next token, and updates the kv cache with the new token. prefill phase. let @xmath0, where b is the batch size, l prompt is the length of the input prompt, and d is the model hidden size. for clarity, we omit the layer index here. the key and value tensors can be computed by: ## experimental setup benchmarks we evaluate the performance of quantized pruning using the longbench benchmark @xcite and needle-in-a-haystack test @xcite . to better distinguish the performance impacts related to input lengths and layerwise configurations (sections 7.1 and c), we further utilize ruler @xcite , a dataset with different input length and diverse types of needles across 4 task categories. detailed information on these datasets can be found in appendix a. ## optimal token-precision trade-off in this section, we aim to find the optimal tokenprecision trade-off in kv cache compression. we first examine the feasibility of integrating kv pruning and quantization(q1). subsequently, we explore the best optimal allocation strategy between precision and token under varying memory budgets(q2). q1. is it feasible to integrate kv pruning and quantization for a lower compression rate? we first evaluate the feasibility of integrating kv pruning and quantization as a prerequisite for exploring the token-precision trade-off. we employ llama-3-8b-instruct and evaluate a range of kv pruning methods on the longbench and niah. we report the results of quantizing the remaining tokens to different precision levels after applying kv pruning. from compared with precision, reducing the number of preserved tokens leads to more significant performance degradation. specifically, when the number of preserved tokens is reduced to 1/4 (from 2048 to 512), all kv pruning methods experience a noticeable performance drop. in contrast, when the precision is reduced to 1/4 (from 16-bit to 4-bit), which has the same memory budget as token dimension, the performance degradation is relatively mild. this suggests that, under the same memory budget, tokens might have a more significant impact on the results compared to precision. to conclude, kv pruning can be effectively integrated with kv quantization at a precision level of 4-bit without substantial performance degradation. q2. what is the optimal allocation strategy between precision and token under varying memory budgets? observing that kv pruning and kv quantization can be effectively integrated, we further investigate that, given a fixed memory budget, how to balance the trade-off between number of preserved tokens and precision to achieve optimal performance. to this end, we evaluate the performance of quantized pruning using two leading kv pruning methods, snapkv and pyramidkv, across different memory budgets on longbench and niah. as shown in figure 2 , we observe that quantized pruning, which preserves more tokens at a lower precision, consistently outperforms standalone kv pruning methods across various budgets. besides, the 1/4 kv cache budget of quantized pruning on 8-bit 2x kv tokens outperforms 4-bit 4x tokens which represents the standalone kv quantization methods. this conclusion demonstrate that quantized pruning can achieve a more favorable operating point within the information bottleneck than optimizing either aspect in isolation. for the niah task, the improvements from quantized pruning are particularly pronounced. this may be attributed to that quantized pruning can cover more tokens for retrieval under the same memory budget compared to standalone kv pruning. in high-budget scenarios, the 8-bit strategy tends to deliver slightly better performance, which may due to the number of tokens at this budget is already quite large. in low-budget scenarios, such as 1/128 kv cache budget, storing more tokens at 4-bit precision yields superior results, highlighting the importance of token coverage when resources are constrained. overall, using lower precision to preserve more tokens under a limited budget results in notable performance gains, compared to standalone kv pruning methods that use full precision to store fewer tokens. ## further analysis in this section, we further investigate series of key aspects regarding token-precision trade-off, including the impact of quantized pruning on various downstream task types and input lengths, model scaling effect and ablation on quantization strategies. we also explore fine-grained layer-wise quantized pruning in the appendix c. ## conclusion we investigate a series of critical yet unexplored questions regarding the effectiveness and feasibility of token-precision trade-off in kv cache compression. we leverage the information bottleneck principle to formulate kv cache compression within a unified theoretical framework and demonstrate that a carefully managed token-precision trade-off can achieve an optimal point within the information bottleneck. through comprehensive experiments, we demonstrate that storing more tokens in the kv cache with lower precision can significantly enhance the long-context performance of llms, and demonstrating robust performance across diverse input lengths, downstream tasks, with particularly significant gains in retrieval tasks. moreover, we show that quantized pruning demonstrates strong feasibility across different kv pruning methods, quantization strategies, and model scales. our analysis sheds light on the token-precision trade-off of kv cache memory optimization, offering valuable insights into designing more efficient compression strategies. we hope this work deepens the understanding of kv cache compression and inspires future research.