P rune V id: Visual Token Pruning for Efficient Video Large Language Models (2025)

Abstract: We introduce PruneVid, a training-free visual token pruning method designed to enhance the efficiency of multimodal video understanding. While Large Language Models (LLMs) have shown promising performance on video tasks due to their advanced visual comprehension capabilities, the substantial redundancy inherent in video data poses significant computational challenges. To address this issue, PruneVid (1) reduces intrinsic video redundancy by merging temporally static and spatially similar tokens, and (2) leverages LLMs’ inherent ability to selectively prune visual tokens irrelevant to specific queries, thereby improving model efficiency. We validate our method across multiple video benchmarks, demonstrating that PruneVid can prune over 80% of tokens while maintaining competitive performance when combined with different video LLMs. Our results highlight PruneVid’s superior effectiveness and efficiency compared to existing pruning methods.

## introduction large language models (llms) @xcite @xcite have significantly advanced multi-modal understanding owing to their exceptional reasoning capabilities and proficiency in following instructions. within the realm of video understanding, recent studies @xcite @xcite @xcite @xcite have capitalized on the use of pre-trained llms as foundational models to address video question-answering tasks. however, the redundancy inherent in video content @xcite can lead to significant computational expenses for llms due to the quadratic complexity of attention mechanisms. consequently, effectively reducing the number of video tokens * corresponding author. existing approaches to video token pruning, nonetheless, face notable limitations. for instance, llama-vid @xcite compresses a video into context and content tokens but necessitates expensive pretraining processes. llava-prumerge @xcite utilizes clip @xcite token correlations but overlooks the relevance of tokens to specific questions. kv cache methods like look-m @xcite and elastic cache @xcite merge multimodal inputs by prioritizing text tokens, requiring full visual token encoding @xcite , which becomes inefficient for long sequences. while fastv @xcite employs llm attention patterns to prune tokens, it lacks video-specific optimizations and does not adequately address the reduction of video inputs. in light of these challenges, we identify three essential criteria that an optimal token pruning method for multi-modal video understanding should satisfy. first, the method should be trainingfree, allowing seamless integration with existing models without necessitating extensive retraining or fine-tuning. second, inherent video redundancy needs to be reduced to save computations on tokens with similar representations along both spatial and temporal dimensions. third, it must retain visual tokens that are specifically relevant to the given queries, ensuring that the model maintains high performance and mitigates the risk of hallucinations when lacking pertinent information @xcite . to achieve our objectives, we present prunevid, a training-free approach for pruning video tokens to achieve efficient video understanding. as illustrated in fig. 1 (a) , our method first tackles intrinsic video redundancy by identifying static regions-areas with minimal variation due to motion or camera movements, often corresponding to what happened after the person took the phone? vision encoder word embedding llm … attention layer … question temporal merging of static tokens spatial merging merged static tokens dynamic tokens … … attention-based token selection … input regions static regions dynamic regions … … … … … … figure 1: an illustration of prunevid. (a) prunevid begins by identifying static video regions with minimal temporal variation, reducing redundancy through temporal token merging. then, spatial redundancy is compressed via clustering and merging, followed by question-guided attention in the llm to select task-relevant visual tokens. (b) static regions correspond to areas with minimal change, whereas dynamic regions display object or camera motion; thus, static regions can be compressed together temporally. (c) attention map visualization across 32 layers shows persistent focus on question-relevant visual regions (e.g., hands and window). the background (see fig. 1 (b) ). we merge these static tokens along the temporal dimension, reducing computational burden by eliminating redundant temporal information. next, we employ a clustering technique @xcite to merge spatially similar tokens in both static and dynamic regions, further compressing the input. subsequently, within the llm, we utilize attention scores between the question and video tokens at an intermediate layer to distinguish and preserve discriminative visual tokens essential for answering the question, while pruning irrelevant ones. as depicted in fig. 1 (c ), attention visualizations reveal that the model consistently focuses on crucial features-such as hand movements and related objects (e.g., a window)-that are directly relevant to the question. this demonstrates that important visual regions can be effectively pinpointed using attention mechanisms, capitalizing on the llm's reasoning and instruction-following strengths. ad-ditionally, for the key-value caches from previous layers, we retain only the essential visual tokens, reducing computational demands during the decoding phase. we integrate prunevid with three video llms: pllava @xcite , st-llm @xcite , and llava-onevision @xcite , and evaluate their performance on multiple video benchmarks, including mvbench @xcite , video-mme @xcite , egoschema @xcite , and videochatgpt-bench @xcite . our extensive experiments demonstrate that prunevid can prune over 80% of visual tokens with only minimal performance degradation in certain cases. notably, our method can occasionally enhance model performance. furthermore, it achieves competitive results compared to the baseline model while reducing flops by 74% to 80%, and reducing gpu memory usage. the main contributions of this paper are as follows: (1) we introduce prunevid, a training-free framework that efficiently prunes video tokens for video understanding tasks, seamlessly integrating with off-the-shelf video llms. (2) our method minimizes video redundancy by merging static tokens over time and clustering spatially similar ones. additionally, we leverage attention scores between the question and video tokens within the llm to retain only the visual tokens pertinent to answering the questions. (3) through extensive experiments across multiple benchmarks, we demonstrate that prunevid consistently achieves superior efficiency and effectiveness with various video llms compared to existing approaches. ## method to efficiently process video data, our method strategically minimizes redundancy in visual tokens before inputting them into the llm and effectively identifies question-relevant visual tokens within the llm. in this section, we first introduce the necessary preliminaries and then provide a detailed explanation of our approach. ## conclusion we propose prunevid, a training-free visual token pruning method for efficient multi-modal video understanding. by merging redundant tokens and using llm attention to select relevant visual information, prunevid significantly reduces computational cost while maintaining or even improving performance. experiments show that prunevid can prune over 80% of visual tokens with minimal impact on accuracy. our method integrates easily with existing video llms and provides a practical solution for efficient long-video processing.