Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem? (2025)

Abstract: Multimodal large language models (MLLMs) have shown remarkable performance for cross-modal understanding and generation, yet still suffer from severe inference costs. Recently, abundant works have been proposed to solve this problem with token pruning, which identifies the redundant tokens in MLLMs and then prunes them to reduce the computation and KV storage costs, leading to significant acceleration without training. While these methods claim efficiency gains, critical questions about their fundamental design and evaluation remain unanswered: Why do many existing approaches underperform even compared to naive random token selection? Are attention-based scoring sufficient for reliably identifying redundant tokens? Is language information really helpful during token pruning? What makes a good trade-off between token importance and duplication? Are current evaluation protocols comprehensive and unbiased? The ignorance of previous research on these problems hinders the long-term development of token pruning. In this paper, we answer these questions one by one, providing insights into the design of future token pruning methods. Codes are available in the supplementary materials.

## introduction multi-modal language models (mllms) @xcite @xcite , especially the vision-language models have demonstrated powerful effectiveness in various tasks. however, the extremely high computational and storage costs have limited the application of mllms in real-time applications, which is caused by not only the enormous parameters inherited from llms but also a large number of tokens from the large visual information such as high-resolution images and multi-frame videos. to solve this problem, abundant efforts have been made in token pruning @xcite @xcite , which aims to reduce the number of input tokens in mllms. usually, token pruning methods first introduce a carefully-designed criterion to measure the importance of a vision token, and then prune the redundant tokens, or merge the redundant tokens into fewer tokens. as a result, the following computation of the pruned tokens or the merged tokens can be removed or reduced, bringing efficiency in both computation and storage. for instance, some recent works show that more than 70% tokens can be pruned with a tolerant loss in accuracy @xcite . most attractively, thanks to the natural ability of mllms to process tokens in different lengths, token pruning can be applied to most existing mllms with no need for additional training, and thus attracts great attention from both academic researchers and industrial developers. however, despite the popularity of token pruning, numerous foundational questions have long been overlooked and remain largely unexplored, giving rise to several surprising phenomena. for instance, figure 1 demonstrates the comparison between two classical token pruning methods in-cluding fastv @xcite and sparsevlm @xcite , and two naive baselines, including random token selection and direct average pooling on tokens. surprisingly, the two baselines outperform the two well-designed token pruning methods in most benchmarks by a clear margin. this counterintuitive phenomenon may demonstrate that the current understanding of socalled important tokens is far away from the truth. unfortunately, most recent works just focus on purchasing higher performance, while ignoring these questions, which may hinder the long-term development of token pruning. in this paper, we have conducted massive experiments and analyses to dive into the fundamental problems of token pruning, with the main takeaways as follows. compatibility with hardware has a significant influence on real acceleration performance. ## benchmarking we begin by presenting the datasets, models, and pruning methods included in our study, along with the rationale behind the selection. next, we outline the experimental setup and provide guidance on interpreting the results reported in our study. finally, we analyze the findings, emphasizing notable patterns and offering insights that may inform future research in this area. ## token pruning revisited: are simple methods better? when considering token pruning in multimodal large language models, two very basic methods naturally come to mind: random token pruning (hereafter referred to as random) and token pooling (hereafter referred to as pooling). comparison with these two simple baselines is reliable evidence to demonstrate the significance of a well-designed token pruning method, yet has been ignored by most previous works. with several recent token pruning methods (e.g., fastv and sparsevlm). as shown in when scaling up to larger models, the experimental results of llava-1.5-13b foot_1 and qwen2-vl-72b foot_2 in tables 2 and 3 also demonstrate the superior performance of the simple methods random and pooling. these surprising results shocked us since its inferior performance compared with random selection may demonstrate that we are on the wrong road toward the ideal token pruning methods. ## limitations and challenges in token pruning evaluation token pruning has emerged as a promising technique to improve the efficiency of mllms. however, despite its potential, the evaluation of token pruning methods remains fraught with challenges. in this section, we critically examine two key issues that hinder the accurate and meaningful assessment of token pruning techniques: (i) the over-reliance on flops as a proxy for speed gains, and (ii) the failure to account for training-aware compression in some advanced mllms. we argue that addressing these challenges is crucial for developing more robust and reliable token pruning approaches. ## conclusion our systematic investigation into token pruning for mllms reveals several critical yet overlooked issues. while existing methods prioritize attentionbased scoring and language-guided strategies, we demonstrate that naive spatial uniformity, achieved through random selection or pooling, often outperforms complex designs due to inherent positional biases in visual tokens. notably, the effectiveness of linguistic guidance depends on task alignment: it enhances performance in text-driven scenarios through cross-modal attention but risks degradation in vision-centric tasks. from an informationtheoretic perspective, we shed light on the core principles of token pruning, i.e., the pursuit of structural integrity versus prediction accuracy. furthermore, we challenge the conventional reliance on flops for efficiency evaluation, showing that latency serves as a more practical and meaningful metric. these findings provide a refined framework to guide the development of future token pruning methods, balancing simplicity, effectiveness, and task-specific adaptability.