A da R e T a K e: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understanding (2025)

Abstract: Multimodal Large Language Models (MLLMs) have revolutionized video understanding, yet are still limited by context length when processing long videos. Recent methods compress videos by leveraging visual redundancy uniformly, yielding promising results. Nevertheless, our quantitative analysis shows that redundancy varies significantly across time and model layers, necessitating a more flexible compression strategy. We propose **AdaReTaKe**, a training-free method that flexibly reduces visual redundancy by allocating compression ratios among time and layers with theoretical guarantees. Integrated into state-of-the-art MLLMs, AdaReTaKe improves processing capacity from 256 to 2048 frames while preserving critical information. Experiments on VideoMME, MLVU, LongVideoBench, and LVBench datasets demonstrate that AdaReTaKe outperforms existing methods by 2.3% and 2.8% for 7B and 72B models, respectively, with even greater improvements of 5.9% and 6.0% on the longest LVBench.

## introduction in pursuit of general intelligence, multimodal large language models (mllms) @xcite @xcite have revolutionized video understanding. however, current mllms require hundreds of tokens to represent a single image @xcite @xcite , limiting video lengths to less than 10 minutes @xcite . efforts to extend mllms' capabilities for long videos include: agent systems @xcite figure 1 : adaretake enables mllm to perceive longer with fixed context length for video-language understanding. which retrieve and interpret pre-segmented videos but remain constrained by single-model abilities. techniques like length extrapolation @xcite and multi-modal sequence parallelism @xcite enhance usable video context length but introduce more visual redundancy. rather than extending context length, compressionbased methods reduce video tokens into shorter sequences by leveraging visual redundancy @xcite . many approaches @xcite @xcite to condense videos guided by language or learnable query tokens. recent advancements @xcite integrate compression into mllm prefilling, yielding promising results. in this work, we push the boundaries of compression-based methods in two key ways: first, by optimizing the compression algorithm with insights from quantitative analysis; and second, by scaling the number of frames processed to capture more information from the video. to dive deeper into compression-based methods, we quantitatively analyze visual redundancy by examining the distribution of influential tokens (more likely to be preserved during compression) during mllm inference, revealing significant variations across video timestamps and llm layers. these findings show that previous methods with fixed compression ratios fail to capture the dynamic nature of visual redundancy, underscoring the need for a more flexible compression strategy. in light of this, we propose adaretake, a training-free adaptive video redundancy reduction method. it features two modules: temporaladaptive allocation, which adjusts compression ratios for video sequence features over time, and layer-adaptive allocation, which manages kv cache states across layers. for temporal allocation, we divide a long video into chunks and allocate a compression ratio for each chunk based on the similarity scores between adjacent frames. for layer allocation, we adjust compression ratios across layers based on video-prompt attention scores. our theoretical analysis demonstrates that this approach reduces the upper bound of l 1 compression loss compared to uniform allocation. the combination of the above allocation determines a specific compression ratio for each chunk in each llm layer. finally, we apply chunked prefilling for all chunks and the prompt. during this process, the kv caches of each chunk are compressed iteratively based on the accumulated attention scores. adaretake compresses long videos into shorter sequences, allowing to perceive more informative frames within a fixed gpu memory budget, thereby enhancing long video understanding. we integrate adaretake into cutting-edge mllms like qwen2-vl @xcite and llava-video @xcite , and conduct extensive experiments across various video understanding benchmarks, including @xcite , mlvu @xcite , longvideobench @xcite , and lvbench @xcite . the results show that adaretake significantly outperforms existing methods, achieving an average improvement of 2.3% and 2.8% across datasets for 7b and 72b models, respectively. on lvbench, the dataset with the longest average video length, the gains are even more pronounced, with improvements of 5.9% and 6.0%, respectively. additionally, the results on needle qa and temporal grounding tasks further demonstrate that our approach effectively preserves fine-grained temporal grounding capabilities. the ablation study validates the effectiveness of our temporal and layer-adaptive budget allocation methods. through comparison with other compression approaches, it further demonstrates the superiority of our method. in summary, our contributions are threefold: ## preliminary analysis in this section, we provide a quantitative analysis of the visual redundancy with mllm for long video understanding. intuitively, redundancy varies across dimensions: at the frame level, static scenes are more redundant than dynamic ones, and at the model level, deeper layers focus on more abstract features, leading to different attention patterns. to quantify this, we measure redundancy through the ratio of heavy-hitters @xcite , a set of influential tokens essential for generation. by identifying these across dimensions, we validate the varying redundancy levels, providing a strong motivation for our approach to achieve more flexible and efficient compression. ## conclusion we introduce adaretake, a training-free method for adaptive redundancy reduction in mllms. by dynamically allocating compression ratios across frames and model layers, adaretake achieves more efficient video token compression. this allows us to scale up to more frames and extract valuable information within the same computational budget. integrated into state-ofthe-art mllms, adaretake enables processing of up to 2,048 frames and outperforms existing methods on benchmarks such as videomme, while adaretake can be integrated into most mllms, it may also inherit their inherent limitations, such as factual inaccuracies, biases, and hallucinations.