MEDA : Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference (2025)

Abstract: Long-context Multimodal Large Language Models (MLLMs) that incorporate long text-image and text-video modalities, demand substantial computational resources as their multimodal Key-Value (KV) cache grows with increasing input lengths, challenging memory and time efficiency. For multimodal scenarios, the cross-modal interactions inevitablely increase complexity, and prior methods for KV cache compression, in both text-only and multimodal LLMs, have neglected attention density variations across layers, often adopting uniform or progressive reduction strategis for layer-wise cache allocation. This results in precision loss and suboptimal performance. We propose MEDA , a novel approach specifically designed for the complexities of multimodal settings, dynamically allocating KV cache sizes based on attention entropy to better adapt to multimodal interactions.Through a dynamic multimodal KV cache allocation strategy, MEDA compresses the KV cache, adaptively retains sufficient multimodal information at each layer. Meanwhile, to mitigate the degradation of contextual information due to cache compression, we also integrate KV pairs merging techniques to maintain coherence. MEDA achieves up to 72% KV cache memory reduction and 2.82 faster decoding speeds in some cases, while maintaining or enhancing performance on various multimodal tasks in a long context, including multi-image and long video scenarios.

## introduction long-context multimodal large language models (mllms) have achieved remarkable progress in processing multimodal long context involving long text-image and text-video inputs, as exemplified by llava-next @xcite , gpt-4v @xcite and long-video mllms @xcite . these models are capable of handling complex multimodal patterns within their key-value (kv) caches, such as text accompanied by multiple in-figure 1 : a multimodal long-context sample from video-chatgpt @xcite , showing key information interactions between blue-boxed video frames and textual phrases. conventional kv cache methods designed for text-only llms are difficult to be directly adopted to long-context multimodal inputs because they do not account for the complex cross-modal interactions present in long-context settings. previous methods for kv cache compression in both text-only llms and mllms such as text-centric eviction-based methods @xcite @xcite , static progressive layer-wise reduction methods @xcite , and multimodal pruning methods @xcite have predominantly employed uniform cache size allocation across layers. however, these methods overlook the variations in attention density across different layers as illustrated in figure 2 . as a consequence, allocating a uniform kv cache size across different layers, without accounting for these variations, can not only lead to information loss in dense layers if less kv caches are allocated, resulting in reduced precision and suboptimal performance, but also cause significant inefficiency in sparse layers when more than enough kv caches are allocated. in this paper, we propose a dynamic layer-wise kv cache allocation method, which we refer to as meda (multimodal attention entropy-guided dynamic kv cache allocation), for efficient multimodal long-context inference. the key idea of meda is that it proposes to use cross-modal attention entropy to capture the diverse cross-modal attention patterns at different layers in mllms, and then dynamically allocates kv caches across layers so as to adapt to the unique layer-wise attention distributions. moreover, given the dynamically allocated kv cache size at each layer, meda employs a multimodal kv pair selection scheme to identify which kv pairs to be selected at each layer. for each kv pair that is not selected, meda incorporates a kv pair merging strategy that merges the selected and non-selected kv pairs to preserve information from the entire context despite the reduced kv cache size. in doing so, meda is able to achieve efficient kv cache usage for multimodal long-context inference. it is also worthwhile to note that meda does not require additional fine-tuning and can be seamlessly integrated as a plug-and-play solution, offering a dynamic kv cache allocation strategy tailored for multimodal contexts. we evaluate meda across various recent mllm backbones, including llava-v1.5-13b @xcite , llava-next-7b @xcite , and internvl-v1.5-7b @xcite for multi-images tasks, as well as llava-video-7b/32b @xcite , longva-7b @xcite , and longvila-8b @xcite for long-video tasks. we also evaluate meda on diverse mutlimodal long-context datasets including milebench @xcite , video-chatgpt @xcite , dream-1k @xcite , and worldqa @xcite . our results show that meda outperforms both state-of-the-art text-based and multimodal kv cache methods including h2o @xcite , snapkv @xcite , pyramidkv @xcite , look-m @xcite , and is able to achieve up to 2.82 times faster inference speed and reduce kv cache memory footprint by up to 72%, while maintaining or improving performance on the target tasks. post-training kv cache compression. posttraining kv cache compression methods fall into four categories: token-wise eviction, 0 5 10 15 20 25 30 layer index 2 3 4 5 6 7 layer entropy cross-modal attention entropy clevr-change wikivqa spot-the-diff movingdirection we analyze llava-next-7b @xcite across different sub-tasks @xcite . we observe varying multimodal interaction patterns: early layers (e.g., layer 1) exhibit dense attention weights with higher entropy, while deeper layers (e.g., layer 24) exhibit sparse attention weights with lower entropy, given that they focus on key tokens (red columns), similar to the blue areas and text in figure 1 . ## experiment results we present experiments demonstrating meda's effectiveness in optimizing multimodal kv cache across various models. evaluations on milebench @xcite highlight its benefits in long-context scenarios, while tests on long-video datasets confirm its generative reasoning capabilities. we analyze the impact of kv cache budgets at different compression ratios ρ on token generation and provide ablation studies for further insight into meda's performance. additionally, we evaluate computational efficiency by measuring runtime and kv cache load during decoding. ## conclusion in this paper, we introduced meda, an approach for dynamically allocating kv cache in longcontext mllms. by leveraging attention entropy, meda efficiently adapts to the complexities of cross-modal interactions, dynamically allocating layer-wise cache sizes and preserving critical information using an average merging technique, while significantly reducing memory usage and improving decoding speed. our experiments show that meda achieves substantial memory savings and faster inference without sacrificing performance, setting a new standard for efficient multimodal processing in tasks involving long text-image and textvideo inputs. these results affirm the effectiveness of meda in addressing the computational challenges inherent to multimodal models.