E vol KV : Evolutionary KV Cache Compression for LLM Inference (2025)

Abstract: Existing key-value (KV) cache compression methods typically rely on heuristics, such as uniform cache allocation across layers or static eviction policies, however, they ignore the critical interplays among layer-specific feature patterns and task performance, which can lead to degraded generalization. In this paper, we propose EvolKV, an adaptive framework for layer-wise, task-driven KV cache compression that jointly optimizes the memory efficiency and task performance. By reformulating cache allocation as a multi-objective optimization problem, EvolKV leverages evolutionary search to dynamically configure layer budgets while directly maximizing downstream performance. Extensive experiments on 11 tasks demonstrate that our approach outperforms all baseline methods across a wide range of KV cache budgets on long-context tasks and surpasses heuristic baselines by up to 7 percentage points on GSM8K. Notably, EvolKV achieves superior performance over the full KV cache setting on code completion while utilizing only 1.5% of the original budget, suggesting the untapped potential in learned compression strategies for KV cache budget allocation.

## introduction key-value (kv) cache @xcite @xcite has become a cornerstone of efficient inference in large language models (llms) @xcite @xcite @xcite , allowing them to reuse previously computed hidden states and thus reduce redundant computation. however, the memory footprint of a kv cache scales linearly with input sequence length, and the quadratic complexity of self-attention makes long-range inference prohibitively slow when the full cache is retained. to address these challenges, existing kv cache compression methods predominantly rely on rule-based heuristics. current approaches can be categorized into three paradigms: (1) fixed-position retention across all layers @xcite @xcite , (2) uniform layer allocation with attentionweighted eviction @xcite @xcite @xcite , and (3) pyramidal strategies with predefined depth-wise attenuation @xcite . while effectively for memory reduction, these heuristics fail to account for two critical aspects: (1) the varying functional roles of transformer layers in information processing @xcite @xcite , (2) the dynamic relationship between cache and task performance. relying solely on rule-based allocation of kv cache budgets across layers can lead to suboptimal retention of task-relevant information. in response to these limitations, we employ evolutionary algorithms @xcite to directly search for optimal kv cache allocation based on the task performance, inspired by @xcite . we introduce evolkv, an evolutionary framework that adaptively allocates kv cache budgets across transformer layers, as shown in figure 1 . it formulates per-layer kv cache budgets as optimization variables, partitions them into groups, and employs an evolutionary algorithm @xcite to iteratively search for group-wise configurations that directly maximize downstream task fitness scores. by integrating task-driven optimization with layer-specific cache pruning, evolkv achieves fine-grained, performanceaware allocation aligned with the varying contributions of different layers. in contrast to rigid heuristics, evolkv provides a flexible and effective mechanism for layer-wise kv cache budget allocation guided by downstream task objectives. first, it formulates layer/group-wise cache budget as learnable parameters, in which we group layers into optimization units for efficient search. then, we directly maximize the performance of downstream tasks using black-box evolutionary optimization methods. by doing so, our approach enables task-aware, granular cache allocation that automatically adapts to each group or layer's functional contribution. specifically, it can accommodate diverse evaluation criteria, such as accuracy and f1 score, and discover non-uniform distributions (i.e., patterns deviated from heuristic fixed-length or pyramidal patterns) without predefined assumptions. we conduct comprehensive experiments on mistral-7b-instruct and llama-3-8b-instruct, evaluating evolkv across four distinct benchmarks (eleven tasks), covering long-context retrieval, long-context reasoning, and mathematical tasks. our results demonstrate that task-optimized kv cache allocation yields consistent improvements: (1) on the needle-in-a-haystack benchmark, evolkv achieves up to a 13% improvement over the best baseline. (2) in the ruler benchmark, evolkv delivers up to a 3.6% gain over the strongest baseline. (3) across longbench evaluations, it consistently outperforms all baseline methods across a wide range of target kv cache budgets (ranging from 128 to 2048), and remarkably exceeds the full model's performance while utilizing only 1.5% of its kv cache budget. (4) for gsm8k, evolkv achieves up to a 7 percentage points improvement in accuracy over the strongest baseline under a 128 kv cache budget, preserving up to 95.7% of the full-model performance, while the strongest baseline retains only up to 84.5% under a 512 kv cache budget. in conclusion, our key contributions are as follows: ## conclusion we introduce evolkv, a task-driven framework that leverages evolution algorithm methods, and delivers state-of-the-art performance under stringent cache constraints and scaling gracefully to larger target cache budgets. thus, evolkv provides a practical, plug-and-play solution for efficient inference in downstream tasks. in the future, it would be valuable to investigate the robustness of kv cache compression under different tokenization schemes, as well as tokenization-free approaches @xcite @xcite .