R edundancy L ens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLM s (2025)

Abstract: Current Multimodal Large Language Model (MLLM) architectures face a critical tradeoff between performance and efficiency: decoder-only architectures achieve higher performance but lower efficiency, while cross-attention-based architectures offer greater efficiency but lower performance. The key distinction lies in how visual tokens are processed. Decoder-only architectures apply self-attention and FFN operations on visual tokens, while cross-attention architectures skip these computations. To investigate whether redundancy exists in this computationally expensive process, we propose a training-free framework for analyzing trained MLLMs. It consists of Probe-Activated Dynamic FFN and Hollow Attention, which enable adjustable reductions in computations for visual tokens, as well as a Layer Ranking Algorithm that prioritizes layers for these reductions. Extensive experiments demonstrate substantial, structured, and clustered redundancy unique to decoder-only MLLMs, offering valuable insights for future MLLM architecture design. Furthermore, by leveraging our reduction framework as a training-free inference acceleration approach, we achieve performance comparable to or better than state-of-the-art methods while remaining compatible with them. Code is available at https://github.com/L-Hugh/RedundancyLens.

## introduction large language models (llms) @xcite @xcite have seen rapid advancement in recent years, attracting attention for their strong capabilities in language comprehension and reasoning. in computer vision, researchers extend llms with visual abilities aimed at developing multimodal large language models (mllms) @xcite @xcite text tokens (t) 2024). these models hold significant potential for multimodal task solving and have become a prominent focus of current research. a key challenge in this area is designing effective architectures to integrate visual signals into llms. ## methodology the proposed framework consists of two components: (1) probe-activated dynamic ffn and hollow attention, which replace the original ffn and attention modules, enabling adjustable reductions in computations for visual tokens. (2) layer ranking algorithm, which assigns a rank to each layer. when selecting a subset of layers for computational reductions, those with the highest ranks are prioritized. ## conclusion in this paper, we present a systematic investigation into the redundancy of visual token processing, which plays a crucial role in the trade-off between performance and efficiency in mainstream mllm architectures. through careful analysis of existing mllms, we propose a new framework consisting of two key components: computational reductions for visual tokens and a layer ranking algorithm. these reductions are applied across various layer proportions to evaluate their impact on mllm performance. extensive experiments reveal that current decoder-only mllms exhibit significant redundancy in visual token processing within certain layers. this structured and clustered redundancy can be effectively leveraged, providing valuable insights for future architectural design. furthermore, this work opens new perspectives on training-free acceleration strategies for mllms, suggesting that future improvements in model efficiency might benefit from considering both token-level compression and computation-level optimization.