T ailor KV : A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization (2025)

Abstract: The Key-Value (KV) cache in generative large language models (LLMs) introduces substantial memory overhead. Existing works mitigate this burden by offloading or compressing the KV cache. However, loading the entire cache incurs significant latency due to PCIe bandwidth bottlenecks in CPU-GPU communication, while aggressive compression causes notable performance degradation. We identify that certain layers in the LLM need to maintain global information and are unsuitable for selective loading. In contrast, other layers primarily focus on a few tokens with dominant activations that potentially incur substantial quantization error. This observation leads to a key insight that loading dominant tokens and quantizing all tokens can complement each other. Building on this insight, we propose a hybrid compression method, TailorKV, which seamlessly integrates quantization and offloading. TailorKV develops an inference framework along with a hardware-friendly implementation that leverages these complementary characteristics. Extensive long-context evaluations exhibit that TailorKV achieves nearly lossless performance under aggressive compression settings, outperforming the state-of-the-art. Particularly, the Llama-3.1-8B with 128k context can be served within a single RTX 3090 GPU, reaching 82 ms per token during decoding.

## introduction large language models (llms) have demonstrated exceptional performance in tasks such as multi-turn dialogues @xcite and multi-document understanding @xcite . in response to the growing complexity of tasks, recent llms have expanded their context windows to over 128k tokens, e.g., gpt-4 @xcite and deepseek v3 @xcite . typically, the inference of llms is auto-regressive, with the key-value (kv) cache stored in memory to avoid recomputation. however, the size of kv cache grows linearly with sequence length, leading to much higher gpu memory consumption and inference latency. recent studies have proposed sparse attention mechanisms to reduce kv cache usage. these methods fall into two categories: irreversible eviction and recallable selection. irreversible eviction methods @xcite @xcite suffer from accuracy degradation due to permanently discarding tokens that may later become crucial, particularly in multi-turn dialogues. recallable selection methods adopt a different approach by maintaining the entire kv cache while selecting only a subset of tokens for processing. however, methods like quest @xcite and sparq @xcite encounter memory limitations when attempting to store all tokens on the gpu. although cpu offloading mitigates gpu memory limitations, existing approaches @xcite still require retrieving a substantial portion of tokens (around 20%), introducing significant decoding latency overheads due to slow data transfer between cpu ram and gpu ram. to optimize accuracy, memory, and latency simultaneously, we first analyze the compression preferences for the kv cache based on layer characteristics. prior researches @xcite applied different sparsity rates to different layers under the same compression strategy. however, our analyses demonstrate that performance degradation primarily stems from the application of unsuitable compression at the layer level (section 3). therefore, we suggest that shallow layers, which exhibit dense attention patterns and emphasize global information @xcite , are better suited for uniform compression like quantization. conversely, layers with a few dominant tokens and largely redundant information are wellsuited for sparsity, as performance can be maintained by retrieving only the dominant tokens. building upon these insights, we propose a novel framework, tailorkv, which employs hybrid compression techniques to reduce gpu memory usage. we introduce an identification metric to classify transformer layers into two distinct types: (i) quantization-friendly layers, which preserve global information from a macro perspective, and (ii) sparsity-friendly layers, which capture crucial information from a micro perspective. this design enables quantization-friendly layers to employ static quantization, achieving a high compression ratio (1-bit per floating-point number) while maintaining model quality. meanwhile, for sparsity-friendly layers, the system offloads the kv cache to cpu memory during prefilling and dynamically retrieves the top-k tokens during decoding. by aligning compression strategies with the characteristics of each layer, this tailored approach significantly reduces overall memory consumption. the accuracy and efficiency of tailorkv are evaluated on various backbone llms using longcontext benchmarks. the results demonstrate that tailorkv drastically reduces memory usage by quantizing 1 to 2 layers to 1-bit precision and loading only 1% to 3% of the tokens for the remaining layers while maintaining nearly lossless performance. specifically, tailorkv achieves a decoding latency of 82 ms for llama-3.1-8b with a 128kcontext on a single rtx 3090 (pcie 1.0) foot_0 , yielding a 53.7% reduction in peak gpu memory usage. the key contributions are summarized as follows. ## motivations and observations layers have compression preferences. in contrast to previous belief @xcite , we propose that not all layers are suitable for sparsity. to quantify the sparsity challenges during decoding, we define the sparse error e. let @xmath0, and let @xmath1, 1} n denote a binary mask that selects the top k elements of a. the sparse error e for each head is defined as: as shown in figure 1a , layers with dense attention distributions exhibit higher sparse errors compared to those with sparse distributions. additionally, we observe sparse error patterns across models and datasets. figure 1b shows that sparse errors are similar across models, with higher sparse errors in shallower layers (e.g., 0, 1). figure 1c shows that the distribution of sparse errors remains consistent across various datasets for the same model. similarly, not all layers are suitable for quantization. as shown in these findings highlight the need for a tailored kv cache compression strategy. we regard layers with dense distributions as quantization-friendly, which focus on global information, and layers with sparse distributions as sparsity-friendly, which prioritize crucial information. attention scores correlate with outliers. each channel in the key and query contributes to the attention scores through their dot product, as expressed by the formula qk ⊤ . figure 2 (top) illustrates that some channels have large magnitudes in the query and key. it follows from the dot product formula that attention scores correlate with these outliers. a recent method @xcite focuses on static channel sparsity, utilizing offline calibration technique to identify high-magnitude channels. however, we find that the sparsity of query and key channels is dynamic rather than static. as shown in figure 2 (bottom), outliers in the query and key do not consistently appear in fixed positions; instead, they may appear in any position. furthermore, dynamically selecting high-magnitude channels improves the recall of dominant tokens compared to using a static offline strategy. this claim is empirically validated in section 5.4. ## related work existing kv cache compression methods include eviction, selection, and quantization, with detailed comparisons in appendix a. eviction methods reduce kv cache size by evicting most tokens during inference. streamingllm @xcite identifies 'attention sinks' by retaining the initial and the most recent tokens. h2o @xcite , snapkv @xcite , and scissorhands @xcite estimate token importance based on historical attention scores. however, evicting dominant tokens may degrade accuracy in tasks like 'needle-in-the-haystack' and multi-turn dialogues. selection methods are more commonly used in sparse attention scenarios. quest @xcite retains the kv cache and utilizes paged keys for retrieving tokens, but it fails to reduce memory usage and suffers from lower recall. instead, kv cache offloading methods like pqcache @xcite and infinigen @xcite approximate attention scores for identifying and loading critical tokens from cpu to gpu, though they face challenges in balancing computation and communication due to large kv cache loads. some methods @xcite use lsh and knn to retrieve critical tokens, which are processed on the cpu and subsequently merged with gpu outputs; however, imbalanced computation times may result in gpu idle time. quantization is a common compression technique that converts high-precision floats into lowprecision integers. existing methods employ vari-ous solutions to minimize quantization error. for example, kvquant @xcite isolates outliers for mixed precision, gear @xcite utilizes svd to recover residuals, and kivi @xcite quantizes keys per channel and values per token. flexgen @xcite reduces i/o transfer latency by quantizing the kv cache to 4-bits. however, none of these methods reduce the kv cache to 1-bit. by contrast, we focus on exploring layer characteristics and selecting the most suitable compression strategy. ## conclusion in this paper, we propose tailorkv, an effective framework for kv cache management in llms. we begin by observing that different layers exhibit distinct compression preferences and categorize them into quantization-friendly and sparsityfriendly, each employing a tailored strategy. specifically, quantization-friendly layers aggressively quantize the kv cache to 1-bit. sparsity-friendly layers, on the other hand, dynamically retrieve dominant tokens based on large magnitudes in the query and key channels, integrating cpu-gpu codesign. experiments across long-context benchmarks show that tailorkv effectively minimizes the usage of the kv cache while maintaining model performance, with an acceptable latency cost. our hybrid framework demonstrates the potential of deploying llms on resource-limited gpus, extending the application of llms to more devices while maintaining efficiency.