C o V i PAL : Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models (2025)

Abstract: Large Vision-Language Models (LVLMs) process multimodal inputs consisting of text tokens and vision tokens extracted from images or videos. Due to the rich visual information, a single image can generate thousands of vision tokens, leading to high computational costs during the prefilling stage and significant memory overhead during decoding. Existing methods attempt to prune redundant vision tokens, revealing substantial redundancy in visual representations. However, these methods often struggle in shallow layers due to the lack of sufficient contextual information. We argue that many visual tokens are inherently redundant even in shallow layers and can be safely and effectively pruned with appropriate contextual signals. In this work, we propose CoViPAL, a layer-wise contextualized visual token pruning method that employs a Plug-and-Play Pruning Module (PPM) to predict and remove redundant vision tokens before they are processed by the LVLM. The PPM is lightweight, model-agnostic, and operates independently of the LVLM architecture, ensuring seamless integration with various models. Extensive experiments on multiple benchmarks demonstrate that CoViPAL outperforms training-free pruning methods under equal token budgets and surpasses training-based methods with comparable supervision. CoViPAL offers a scalable and efficient solution to improve inference efficiency in LVLMs without compromising accuracy.

## introduction large vision-language models @xcite @xcite have recently demonstrated remarkable capabilities in understanding and generating content grounded in visual inputs, including both images and videos. to effectively capture the rich spatial and semantic details inherent in visual signals, these models often rely on generating hundreds or even thousands of visual tokens per image or video. for instance, llava-onevision @xcite explicitly allocates up to 7,290 visual tokens per image, leveraging a large corpus of high-quality images to maximize visual comprehension. although dense visual token representations enhance the model's capacity to understand finegrained visual content, they come at the cost of substantial computational and memory overhead @xcite . this leads to reduced inference efficiency and makes it difficult to apply lvlms in scenarios where resources are limited or real-time performance is required. to address this issue, prior work has explored reducing the number of visual tokens or compressing their corresponding key-value (kv) cache @xcite pruning from layer 0 guidance configuration: et al., 2022), highlighting the substantial redundancy present in visual representations. token eviction methods discard less informative tokens based on importance scores @xcite , while token merging approaches group similar tokens and consolidate them to reduce token number @xcite . empirical observations suggest that pruning visual tokens in shallow layers can significantly hurt performance and every visual token matters in these layers @xcite . despite their effectiveness to some extent, these methods largely fail to prune tokens in the shallow layers. visual token reduction is less effective in shallow layers, primarily because tokens in these layers interact with fewer transformer decoder layers, resulting in limited contextual information. this makes it challenging to identify unimportant tokens, leading to significant performance degradation when attempting to prune visual tokens at these stages @xcite . however, we observe that some visual tokens are inherently redundant and can be effectively and safely pruned when guided by appropriate contextual information. based on this insight, we propose covipal, a contextualized visual token pruning method that operates across all layers, as illustrated in figure 1 . covipal implements the ppm module using a small classifier trained on limited data to identify and remove less important tokens before they are passed to the base model of lvlm, thereby reducing the number of visual tokens while maintaining model performance. we conducted experiments on two models: llava-onevision and llava-video. for llava-onevision, we trained the classifier using only 0.46% of the pretraining dataset, while for llava-video, we extended its capabilities to handle video inputs using just 7.4% of the video instructionfollowing dataset. additionally, we performed extensive experiments on various image and video benchmarks. the results demonstrate that our method reduces the prefilling time by up to 60% compared to the original model, with only minimal performance degradation when pruning 75% of the visual tokens. furthermore, our approach outperforms both training-free methods, fastv and sparsevlm, and training-based method pyramid-drop, with the same visual token budget. ## conclusion we propose covipal, a novel contextualized visual token pruning method that efficiently reduces the computational and memory overhead of large vision-language models by leveraging a lightweight and plug-and-play pruning module. covipal identifies and removes redundant visual tokens across all layers with minimal supervision, achieving up to 50% reduction in pre-filling time and pruning 75% of visual tokens while maintaining competitive performance. our method out-performs both training-free and training-based approaches, offering a scalable and adaptable solution for efficient multimodal inference. this work provides new insights into visual token redundancy and paves the way for deploying lvlms in resource-constrained settings.