A da TP : Attention-Debiased Token Pruning for Video Large Language Models (2025)

Abstract: Video Large Language Models (Video LLMs) have achieved remarkable results in video understanding tasks. However, they often suffer from heavy computational overhead due to the large number of visual tokens generated from multiple video frames. Existing visual token compression methods often rely on attention scores from language models as guidance. However, these scores exhibit inherent biases: global bias reflects a tendency to focus on the two ends of the visual token sequence, while local bias leads to an over-concentration on the same spatial positions across different frames. To address the issue of attention bias, we propose Attention-Debiased Token Pruning for Video Large Language Models(AdaTP), a novel token pruning pipeline for Video LLMs. AdaTP integrates two dedicated debiasing modules into the pipeline, targeting global attention bias and local attention bias, respectively. Without the need for additional training, our method significantly reduces the computational overhead of Video LLMs while retaining the performance of vanilla models. Extensive evaluation shows that AdaTP achieves state-of-the-art performance in various commonly used video understanding benchmarks. In particular, on LLaVA-OneVision-7B, AdaTP maintains performance without degradation while using only up to 27.3% FLOPs compared to the vanilla model. Our code will be released soon.

## introduction video large language models (video llms) @xcite @xcite @xcite have shown potential and effectiveness in video comprehension domain. building upon architectures of previous multi-modal large language models (li et al., 2023; liu et al., 2024b; zhu et al., 2023; team et al., 2023; alayrac et al., 2022; liu et al., 2024a, * equal contribution. video llms face significant computational challenge, which can be mainly attributed to the large number of visual tokens and the quadratic complexity of attention mechanism. to mitigate this, some previous works @xcite @xcite @xcite @xcite have sought to alleviate this through visual token compression, typically leveraging attention scores to guide token pruning or merging. however, these works often fail to accurately utilize attention scores for token compression process. as a result, the selected tokens may fail to retain a comprehensive representation of the visual content, and thus lead to obvious performance degradation. in this work, we first identify that the attention scores in the language models of video llms are inherently biased, which can be characterized from two perspectives: global attention bias. visual tokens with high attention scores tend to cluster at the beginning or end of the sequence, potentially overlooking intermediate important visual information. local attention bias. we conduct frame-wise analysis, and discovers that in shallow layers, attention scores are disproportionately concentrated on a few spatial positions, leading to over-retention of those tokens and reduced visual diversity. to provide a more intuitive understanding of attention biases, we visualize the average attention distribution in the llava-onevision-7b model @xcite . the visualization is computed across all samples in the videomme @xcite dataset and shown in fig. 1 . visualization results from other layers can be found in the appendix a.3. in fig. 1 , we can intuitively see that the red bars, 0 1000 2000 3000 4000 5000 6000 visual token index which represent high attention scores, are concentrated in the latter part of the sequence, showing global attention bias; the green bars, exhibit a periodic pattern throughout the entire sequence, representing local attention bias. to effectively leverage attention scores to guide visual token compression, we argue that it is essential to address the aforementioned biases. in this paper, we propose targeted modules to mitigate global and local attention bias respectively. building upon these insights, we introduce attention-debiased token pruning for video large language models (adatp), a training-free and plug-and-play acceleration pipeline for video llms. to be specific, we first partition adjacent similar frames into coherent segments, then introduce two debiasing modules: the global debiasing module helps the model focus on semantically meaningful segments by identifying visual contents that are closely related to textual information; the local debiasing module performs token pruning adaptively and independently within each segment, which encourages the preservation of diverse visual content and effectively reduces intra-segment redundancy. we integrate our adatp pipeline into two types of representative video llm architectures: llava-onevision @xcite and llava-video @xcite . experimental results show that our method achieves stateof-the-art results on several video understanding benchmarks including videomme @xcite , longvideobench @xcite , and mlvu @xcite , which are widely adopted to assess the comprehension ability of video llms, and contain diverse video types. on llava-onevision-7b, our method achieves comparable performance to the original model while keeping only 27.3% of its flops. our main contributions are as follows: ## methodology attention score in llm as token importance metric. in video llms, attention scores produced by llms can serve as a metric for token importance. in the self-attention module of the l-th layer in language models, by averaging across all attention heads, we get attention map a @xmath0. we extract the attention scores from text tokens to visual tokens by averaging the rows of the attention matrix a l corresponding to text tokens and slicing the columns corresponding to visual tokens, resulting in s @xmath1. in the (l + 1)-th layer, these scores can be used to guide the token compression process. however, directly selecting the visual tokens with the highest attention scores leads to significant performance degradation, as will be discussed in the next subsection. ## conclusion in this paper, we identified and thoroughly analyzed the attention bias phenomenon in video llms, and proposed adatp, a training-free token pruning pipeline. our adatp effectively addressed the issue of attention bias, relieved the heavy computational burden of video llms while retaining critical visual contents. extensive experiments on different video llms and video understanding benchmarks have demonstrated the effectiveness and robustness of our method.