M ini KV : Pushing the Limits of 2-Bit KV Cache via Compression and System Co-Design for Efficient Long Context Inference (2025)

Abstract: State-of-the-art 2-bit KV cache quantization techniques achieve excellent results in accelerating LLM inference while retaining accuracy on long context tasks. However, further pushing the compression ratio fails to deliver performance gains. In this work, we revisit these approaches by considering, additionally, adaptive KV methods that retain LLM accuracy with only a subset of KV states. This leads us to propose a method based on 2-bit KV cache quantization with adaptive KV policies. In addition, we take an algorithm and system co-design approach by developing hardware-friendly kernels to accelerate LLM inference while making MiniKV compatible with existing memory-efficient attention techniques such as FlashAttention, effectively translating algorithmic improvements into system performance gains. Experiments on a wide range of long context tasks show that MiniKV effectively achieves >80% KV cache compression while retaining accuracy, outperforming state-of-the-art methods while achieving excellent latency, throughput, and memory consumption improvements in long context inference.

## introduction large language models (llms) have exhibited unique capabilities, such as instruction following, reasoning, and inference time scaling @xcite . however, efficiently serving llms is still a pressing concern. one of the main llm inference bottlenecks is the consumption of kv cache memory, which consumes memory in addition to widely studied bottlenecks such as model sizes @xcite . to address this challenge, one of the prevailing practices is to quantize the kv cache @xcite . studies show that fp8/int8 or even 4-bit quantization can be achieved for kv cache compression while preserving accuracy @xcite @xcite . state-of-the-art approaches, such as kivi and kvquant @xcite , show that the kv cache can be effectively quantized to sub 4-bit, e.g., 2 bits, while preserving most accuracy. however, further pushing down the compression ratio (e.g.,<2 bits) leads to a significant accuracy loss. in a separate line of research in the community, numerous work have explored adaptive kv, where the llm selects a small subset of kv states based on their importance @xcite @xcite . recent advances also introduce head-specific adaptive kv @xcite @xcite and layer-specific adaptive kv @xcite @xcite with the goal of evicting or merging kv pairs without compromising overall performance. however, following the work of @xcite , few studies have included studies on how adaptive kv policies work on quantized kv cache, despite quantized kv is widely used in practice @xcite . moreover, for long context inference, where kv cache memory becomes the major bottleneck, few adaptive kv work manage to achieve a compression ratio that exceeds 50% while maintaining accuracy in long context tasks @xcite . these two points of view (quantized kv and adaptive kv) consider the extreme sides of the spectrum of optimization points for kv cache memory. however, there has been very little work exploring how to consolidate these two lines of work to maximize the kv cache memory savings. the conventional wisdom is that these techniques can be combined. however, existing work aiming to combine 4-bit quantization and adaptive kv figure 1 : an overview of minikv. tensors colored red/blue indicate 16-bit/2-bit representation, and shaded tokens are evicted during inference. during the prefill phase, we employ pyramid kv with rectified token selection policy across layers to identify a sparse set of important tokens. for all the important tokens, we employ sub-channel key quantization and per-token value quantization to minimize the quantization errors while maintaining a compact kv cache data layout without introducing any irregular operations. to address the incompatibility issue between score-based kv pair selection policies and memory-efficient system optimizations such as flashattention, we develop a two-pass triton-based selective flash-attention kernel to output both the representation x o and the cumulative attention map a cumul , while still keeping the memory consumption of the attention calculation linear with respect to the sequence length. during decoding, we use a fused unpacking and multiplication kernel to compute both the attention map between the new query token t q and the quantized keys, as well as the product between the attention map and the quantized values. to answer the question, we revisit existing approaches on ultra low-bit quantized kv (e.g., 2-bit) and adaptive kv, together with a compression system co-design perspective, which is unexplored so far. our findings led us to develop minikv, which effectively compresses the kv cache through a synergistic combination of 2-bit quantization and adaptive kv to achieve minimal accuracy loss in long-context tasks while maximizing the compression ratio. specifically, on the algorithm side, we employ subchannel-wise key and token-wise value quantization, as well as pyramid kv with rectified token selection policy across layers to significantly push the kv compression ratio while keeping the algorithm still hardware-friendly without introducing any irregular computation. on the system side, we develop a two-pass triton @xcite kernel together with native fused kernels to accelerate the inference latency while resolving the incompatibility limitation from the attention score-based eviction policy and memory-efficient attention system optimizations such as flashattention @xcite . consequently, the resulting system maximizes the compression ratio on the kv cache while obtaining high accuracy and hardware efficiency in long context inference. to validate the approach, we compare minikv with existing kv cache compression techniques such as h2o, snapkv, and q-hitter, across three major models in longbench datasets. the results show that minikv effectively achieves 86% kv cache compression while retaining comparable accuracy on longbench, outperforming state-of-theart methods. furthermore, minikv enables prompt lengths up to 44k tokens and a maximum throughput that is 48% higher than its strongest baseline on a single nvidia a100 gpu. to our knowledge, our work is the first to show that it is possible to achieve @xmath0. ## related work numerous efforts have been made to improve the kv cache efficiency of llms. among them, quantization has been a prevailing technique employed in deployment to overcome kv memory overhead without retraining @xcite . many research has shown that fp8/int8/int4 quantization can be achieved for kv cache while preserving accuracy @xcite @xcite @xcite . however, further pushing the quantization limit to under 4-bit, e.g., 2-bit, leads to major performance loss. more recently, researchers have proposed advanced quantization techniques, such as kivi @xcite , to quantize kv cache into 2-bit without major loss in accuracy. while being effective, it still has one major limitation: its effectiveness against adaptive kv policies and its implication on system performance has not yet been studied. our results indicate that it is nontrivial to use 2-bit quantized kv together with adaptive kv policies in conjunction while achieving high compression ratio, accuracy, and system efficiency in long context inference, simultaneously. adaptive kv policies have also gained interest within the community, leading to various algorithms @xcite @xcite @xcite @xcite @xcite @xcite . however, many of those works either do not focus on long context inference @xcite @xcite , where the kv cache pressure is the most prominent, or introduce irregular operations or auxiliary scores that are not hardware-friendly (e.g., batch @xmath1. finally, most adaptive kv methods struggle to exceed a 50% compression rate in long context inference @xcite @xcite , suggesting that solely identifying important tokens may have limited improvements for adaptive kv. complementary to this line of work, our goal is to improve the compression ratio of kv cache via revising ultra-low quantized kv (e.g., 2bit) with adaptive kv policies, with an eye toward system co-design to maximize the performance of llm inference. we empirically show that this path can be more memory-efficient, especially on long context tasks. we provide a detailed summary of the comparison between minikv and previous approaches in appendix b. ## method in this section, we first focus on the compressibility of ultra low-bit quantized kv cache by considering adaptive kv policies, with an eye toward being able to still keep the final solution hardware friendly, which leads to the proposed algorithm in minikv. in addition, we introduce kernel optimiza-tion that addresses the composibility issue between score-based adaptive kv and memory-efficient attention implementation such as flashatttention. ## experiments we conduct experiments to evaluate the effectiveness of minikv in improving accuracy preserving and inference performance. ## conclusion in this work, we revisit kv cache optimization via compression and system co-design to accelerate the inference of llm. our empirical analysis indicates that it is challenging to directly compose state-of-the-art 2-bit quantized kv with existing adaptive kv policies while preserving both accuracy and system efficiency on long context tasks under a high compression ratio. to address this issue, we develop minikv to bridge the gap between ultra low-bit kv quantization and adaptive policies, as well as the gap between the compression algorithm and hardware. evaluation on a wide range of datasets and models shows that minikv preserves long context accuracy while significantly improving the efficiency of llm inference.