LOOK - M : Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference (2024)

Abstract: Long-context Multimodal Large Language Models (MLLMs) demand substantial computational resources for inference as the growth of their multimodal Key-Value (KV) cache, in response to increasing input lengths, challenges memory and time efficiency. Unlike single-modality LLMs that manage only textual contexts, the KV cache of long-context MLLMs includes representations from multiple images with temporal and spatial relationships and related textual contexts. The predominance of image tokens means traditional optimizations for LLMs’ KV caches are unsuitable for multimodal long-context settings, and no prior works have addressed this challenge.In this work, we introduce **LOOK-M**, a pioneering, fine-tuning-free approach that efficiently reduces the multimodal KV cache size while maintaining performance comparable to a full cache. We observe that during prompt prefill, the model prioritizes more textual attention over image features, and based on the multimodal interaction observation, a new proposed text-prior method is explored to compress the KV cache. Furthermore, to mitigate the degradation of image contextual information, we propose several compensatory strategies using KV pairs merging. **LOOK-M** demonstrates that with a significant reduction in KV Cache memory usage, such as reducing it by **80%** in some cases, it not only achieves approximately **1.3x** faster decoding but also maintains or even **enhances** performance across a variety of long context multimodal tasks.

## introduction large language models (llms) @xcite @xcite are progressively evolving into multimodal large language models (mllms) @xcite instruction: your objective is the main goal. evaluate your current environment and your past decisions, and decide your immediate course of action. figure 1 : a multimodal long-context sample contains multiple images from milebench @xcite showing comprehensive spatial relationships. 2023; @xcite , making significant advances in the processing of extensive multimodal contexts such as gpt-4v. despite the impressive capabilities of mllms, they still face significant challenges when dealing with long multimodal context inputs, such as temporal multi-image tasks and semantic multi-image tasks @xcite , or multi-turn multimodal dialogues @xcite in real-world applications. specifically, multimodal kv caches hinder the efficient processing of long multimodal inputs. during inference, the increased lengths of inputs linearly slow down the decoding process due to the attention computations across past multimodal kvs. furthermore, as depicted in figure 1 , in contrast to text-only llms' kv cache eviction methods @xcite , long multimodal inputs typically include multiple interrelated images, along with definitions or background descriptions relevant to the task. directly applying traditional text-centric kv cache eviction strategies @xcite @xcite to mllms overlooks the potential interactions between multimodal representations @xcite . specifically, figure 2 shows the attention visualization for multimodal long-context, the model exhibits greater attention to the textual components during the multimodal prompt encoding process. this observation demonstrates that the model tends to understand global visual content through textual knowledge, highlighting the necessity of preserving textual features and selectively pruning redundant image tokens in the multimodal kv cache to maintain the integrity of the multimodal context. in this paper, we introduce look-m, a pioneering and efficient framework that marks the first effort to compress kv caches specifically for multimodal long-context scenarios. the term look-once in our method implies that pruning occurs only once during multimodal long prompt encoding, and the model effectively sees the full image just once. look-m utilizes a text-prior technique that prioritizes the retention of textual kv pairs during the prompt encoding phase, given the insight from figure 2 . for visual representation, inspired by attention-based eviction strategies @xcite , our method prunes redundant visual kv pairs that show sparse patterns in attention visualizations, utilizing the metric of attention scores. furthermore, to preserve global contextual information in the compressed cache, we develop several merging strategies to merge the evicted kv tokens into conserved ones, addressing potential hallucinations and contextual inconsistencies @xcite during the decoding process. remarkably, look-m does not require any fine-tuning and can be applied in a plug-andplay manner with a look-once kv cache compression strategy. we evaluate our look-m with several strategies over four recent mllm backbones llava-v1.5-7b/13b @xcite , mobilevlm-v2 @xcite and internvl-v1.5 @xcite across several multimodal long-context tasks from milebench @xcite ): temporal multi-image tasks, semantic multiimage tasks, needle in a haystack task, and image retrieval tasks, respectively. compared to baselines, look-m achieves minimal performance drop with a fixed kv cache budget and improves the model inference decoding latency by 1.3x to 1.5x and reduces kv cache memory footprint by 80% to 95% while still maintaining performance on long context multimodal tasks, and even showing improved performance across various tasks. our analysis validates that combining text-prior and proposed merging strategies contributes to the multimodal kv cache compression effectiveness of look-m. kv cache compression for llms. kv cache compression primarily encompasses three strategies: eviction, quantization, and trainable compression. in eviction, techniques like mistral-7b @xcite and streamingllm @xcite only preserve key tokens for efficient sequence generation, while approaches like h 2 o @xcite and snapkv @xcite al., step#1: turn left and…. current step: vit proj llm layer prefill merge llm layer's kv cache • • • • • • text prior token • • • • • • visual token visual evicted token recent kept tokens pick up the mug that's in front of you at the coffee maker. visual important token 2024) focus on maintaining a small, influential set of tokens to enhance performance, though risk losing context with evicted kvs. quantization strategies such as kivi @xcite and gear @xcite reduce cache memory through advanced quantization techniques, balancing memory efficiency with precision. in trainable compression, methods like less @xcite and dmc @xcite adapt llms to compress kv caches by training on selected datasets, although they face challenges in generalization. however, our look-m utilizes a plugand-play approach that does not require additional training, ensuring wider applicability without the necessity for tuning specific to multimodal datasets. therefore, different from these text-centric kv cache compression methods, our look-m specifically targets long multimodal text scenarios and seeks to leverage attention map interactions between text and images to guide kv cache pruning. token merging. unlike token pruning @xcite @xcite in encoder-based backbones like vit @xcite or bert @xcite , which discards less significant tokens, token merging @xcite consolidates tokens into fewer, more meaningful units, preserving information integrity. consequently, token merging has become preferred over token pruning to reduce token count. existing methods like tps @xcite , mg-vit @xcite , and pumer @xcite have explored token merging and pruning techniques, primarily in computer vision tasks. in contrast, look-m is a pioneering effort to adapt token merging within the multimodal kv cache in long-context scenarios, enhancing ef-ficiency for auto-regressive tasks in mllms. ## methodology in section 3.1, we first review the basic implementation of generative inference utilizing a multimodal kv cache. subsequently, as shown in figure 3 , we detail the principal components of the look-m model, which includes text-prior kv pairs eviction strategy to facilitate precise pruning, discussed in section 3.2, and various strategies for merging kv pairs, such as averaged, pivotal, and weighted merging in section 3.3. ## experiment results in this section, we present experimental results demonstrating the effectiveness of our look-m strategy for kv cache optimization on the llava-v1.5-7b and 13b @xcite , internvl-v1.5-7b @xcite , and mobilevlm_v2-3b @xcite models. these models were tested across various subtasks of the milebench dataset @xcite , highlighting the advantages of our approach in multimodal long-context scenarios. we also examine the impact of kv cache compression on different model architectures, establishing its efficacy across diverse structures. additionally, we explore how varying kv cache budgets and compression ratios (α 1 and α 2 ) affect model performance. finally, we assess the computational efficiency of our method by measuring the time and computational load during the decoding phase of compressed models. ## conclusion in this work, we propose look-once optimization in kv for efficient multimodal long-context inference (look-m), the first framework is specifically designed to manage multimodal kv caches in multimodal large language models (mllms) efficiently. look-m integrates a novel kv cache eviction strategy with innovative merging techniques, such as averaged, weighted, and pivotal merging, to maintain essential contextual information without the need for fine-tuning. our findings reveal that the framework not only preserves the quality of generation in multimodal long-text scenarios but also ensures robust performance under significant kv cache compression. observations indicate that look-m prioritizes text over visual inputs during prompt prefilling, leading to the development of a text-prior method that further optimizes kv cache compression. looking ahead, we plan to expand look-m's capabilities by incorporating additional compression techniques like quantization, distillation, and efficient attention mechanisms to enhance both efficiency and efficacy.