A da V : Adaptive Text-visual Redirection for Vision-Language Models (2025)

Abstract: The success of Vision-Language Models (VLMs) often relies on high-resolution schemes that preserve image details, while these approaches also generate an excess of visual tokens, leading to a substantial decrease in model efficiency. A typical VLM includes a visual encoder, a text encoder, and an LLM. Recent studies suggest pruning visual tokens based on visual and textual priors to accelerate VLMs without additional training costs. However, these methods often overlook prompt semantics or suffer from biased self-attention in the LLM. Inspired by the efficient mechanisms of the human brain for multimodal understanding, we introduce AdaV, a novel training-free visual token pruning method. By emulating the neural pathways that preprocess visual and auditory information before the reasoning stage, we shift text-guided visual attention redirection to the pre-LLM stage, which reduces biased token pruning and enhances model robustness with a limited visual token budget. A Self-adaptive Cross-modality Attention Redirection (SCAR) module is further proposed that effectively merges and redirects visual attention with text-to-image attention. Extensive experiments on seven challenging benchmarks demonstrate that our AdaV achieves SOTA performance in training-free VLM acceleration and can be plug-and-play on various VLMs. We plan to open-source the code upon publication.

## introduction in recent years, vision-language models (vlms) have demonstrated exceptional performance in various visual-grounded tasks. despite their impressive achievements, the computational cost associated with vlms remains a significant challenge for practical deployment. a key factor contributing to this cost is the large number of visual tokens required. for instance, llava-next models @xcite et al., 2024) utilize 2,880 visual tokens for singleimage tasks, which may significantly exceed the number of tokens typically used in text prompts. many research efforts have focused on pruning redundant visual tokens to accelerate vlms without additional training. fastv @xcite observes that the distribution of attention weights among visual tokens tends to cluster, allowing for the ranking and retention of only the top-ranked tokens in the llm layers. sparsevlm @xcite selects the keywords from the text and reserves key visual tokens within the self-attention layers of the llm. however, fastervlm @xcite highlights that these methods suffer from biased text-to-image attention of the llm and may not accurately reflect the importance of visual tokens. to address this, fastervlm proposes utilizing class attention extracted from the visual encoder as a significance metric for visual token pruning. however, it fails to recall non-salient yet semantically relevant visual information. previous research has demonstrated that insights from the mechanisms of the human brain can inspire advancements in intelligent systems @xcite @xcite . the human brain tackles multimodal understand-ing through a series of steps: (i) processing visual and linguistic information separately within their respective cortexes, (ii) matching information and redirecting attention, primarily occurring in the temporoparietal junction (tpj), and (iii) engaging in higher-order thinking and response generation within the prefrontal cortex (pfc) @xcite @xcite . the tpj, situated at the convergence of the temporal and parietal lobes, is crucial for various cognitive functions, including the reorientation of attention and the matching of visual and auditory language inputs. the encoded visual and linguistic information undergoes initial cross-modal attention reorientation in regions such as the tpj. this stage of processing is distinct from the subsequent activities that occur in the pfc. after the tpj's involvement, the pfc engages in higher-order cognitive processes, including decision-making and judgment. this workflow enables the brain to concentrate on essential visual information guided by linguistic cues @xcite . inspired by these cognitive processes, we propose adav, a novel training-free acceleration method that emulates the mechanisms of the human brain. as depicted in fig. 2 , we decompose the vlm into four components corresponding to specific brain regions: (1) the visual encoder (red) mirrors the function of the visual cortex, (2) the text encoder (purple) aligns with the temporal lobe, responsible for comprehending language semantics, (3) the llm (green) parallels the prefrontal cortex (pfc), which is involved in cognitive processing and responses, and (4) the self-adaptive crossmodality attention redirection (scar) module (blue) in the pre-llm stage mimics the tpj's function to integrate multimodal information. first, we extract visual attention from the self-attention layers within the visual encoder. next, embedded text prompts query the visual embeddings to obtain text-to-image attention. we then measure the overall significance of potential visual token collections using a geometric average of both visual and text-to-image attention and employ a one-step optimization process to determine the optimal visual token collection. our validation experiments demonstrate that, compared with the text-to-image attention extracted from the self-attention layers of the llm, the attention in the pre-llm stage mitigates the attention bias, and effectively reflects the significance of visual tokens. additionally, ex-tensive experiments show that our adav achieves state-of-the-art (sota) performance on multiple benchmarks and is even comparable to fine-tuning methods such as visionzip @xcite . our contributions are summarized as follows: i. we propose adav, a novel training-free vlm acceleration method that effectively speeds up vlms while maintaining the model's effectiveness. ii. inspired by the human brain's multimodal information-processing pathways, we introduce visual attention redirection to the pre-llm stage and reveal the necessity and feasibility of this design. experiments demonstrate it significantly enhances the model's performance given a limited visual token budget. iii. we propose a self-adaptive cross-modality attention redirection (scar) module that effectively redirects visual attention via text-to-image attention for effective visual token pruning. iv. extensive experiments on seven benchmarks show that adav achieves sota performance and is plug-and-play on various vlms. significant progress has been made in the development of vlms. llava @xcite is the first approach to effectively combine large language models (llms) with foundational vision models. the initial models in the llava family only utilize a single image as input, resulting in 576 visual tokens for an image. however, this approach often led to significant information loss, thereby limiting model performance. to retain detailed vision information of the input, the subsequent vlms propose dynamic resolution schema to the input image, enhancing multi-modal capabilities @xcite @xcite . ## conclusion in this study, we introduce adav, a training-free approach designed to accelerate vlms by emulating the multimodal information processing pathways of the human brain. our method positions textguided visual attention redirection before the llm, effectively mitigating biased and text-agnostic token preservation. additionally, we present the self-adaptive cross-modality attention redirection (scar) module, which adaptively integrates and redirects visual attention in conjunction with text-to-image attention. extensive experiments demonstrate that adav achieves state-of-the-art performance compared to existing approaches for training-free vlm acceleration and is plug-andplay on various vlms.