A dapt M erge: Inference Time Adaptive Visual and Language-Guided Token Merging for Efficient Large Multimodal Models (2025)

Abstract: Recent advances in Large Multimodal Models (LMMs) have showcased impressive visual understanding and vision-language reasoning capabilities, yet their computational cost hinders practical deployment, especially in resource-constrained settings. A key bottleneck is the large number of visual tokens generated by its vision encoders, which increases latency and memory demands. Existing token reduction methods often require costly fine-tuning or apply fixed token reduction ratios, ignoring image complexity and vision-language interactions. We propose AdaptMerge, a training-free, inference-time token merging strategy that adaptively reduces visual tokens by leveraging feature diversity and language-guided relevance. By dynamically adjusting to image complexity and ensuring multimodal coherence, AdaptMerge significantly lowers floating-point operations while improving performance. Extensive experiments on Google’s latest Gemma 3 models (4B and 12B parameters) across four challenging benchmarks demonstrate that AdaptMerge outperforms state-of-the-art token reduction techniques, achieving both reduced computational costs and improved performance, thereby providing a practical pathway to more efficient LMMs.

## introduction recent advances in large multimodal models (lmms) @xcite @xcite have demonstrated remarkable capabilities in visual understanding and vision-language reasoning. lmms typically consist of a vision encoder @xcite @xcite responsible for processing visual information such as images into visual tokens or embeddings and a large language model (llm) that processes the combined visual and text tokens to perform sophisticated tasks requiring complex multimodal understanding, such as visual question answering and image captioning. however, the computational demands of lmms during inference remain a significant barrier to their practical adoption and efficient deployment, particularly in resource-constrained environments. a primary contributor to this challenge is the large number of visual tokens generated by vision encoders, which scales with image resolution and model size, leading to increased latency and substantial computational requirements @xcite @xcite . to mitigate these challenges, token reduction techniques have been developed to streamline visual processing in lmms while preserving performance. however, existing methods face several limitations. many approaches require fine-tuning or calibration on large datasets, which is computationally intensive and impractical due to substantial gpu memory and training time requirements @xcite @xcite @xcite . additionally, most token reduction methods apply a fixed reduction ratio across all images, lacking adaptiveness to image complexity. this one-size-fits-all approach can lead to suboptimal outcomes: images with sparse details, such as natural scenes, could tolerate aggressive token reduction, whereas images with finegrained details, like wildlife or satellite images, require a more conservative strategy to preserve critical information. furthermore, the majority of current methods @xcite @xcite focus solely on visual information for token reduction, overlooking the multimodal nature of lmms and the relevance and relationship of visual tokens to the textual input, which is vital for optimizing performance in vision-language tasks. in this paper, we propose adaptmerge, a novel inference-time adaptive token reduction strategy that integrates visual and language guidance to enhance the efficiency of lmms. adaptmerge is entirely training-free, eliminating the need for costly retraining or fine-tuning. for the unimodal vision encoder, adaptmerge employs diversity-guided visual token merging, which dynamically reduces tokens by prioritizing significant visual tokens while merging redundant or less informative ones in each encoder layer based on their feature diversity, adapting to the complexity of input images. subsequently, it performs language-guided visual token merging on the visual tokens fed into the llm, prioritizing those most relevant to the text prompt to reduce llm input tokens. by dynamically consolidating visual tokens during inference, adaptmerge strikes an optimal balance between computational cost and performance, reducing floating-point operations (flops) while improving accuracy across multiple multimodal benchmarks compared to existing token reduction techniques (fig. 1 ). as a plug-and-play solution, adaptmerge paves the way for broader adoption of lmms in real-world applications. in summary, we make the following key contributions: ## our approach we present adaptmerge, an inference-time, plugand-play adaptive token reduction strategy that can be easily retrofitted into lmms. we begin by describing the general architecture of lmms, followed by a description of adaptmerge. ## experiments in this section, we first lay out our experimental settings and implementation details. then, we showcase the effectiveness of our method on several standard multimodal benchmarks. ## conclusion we propose adaptmerge, a novel inference-time, training-free adaptive visual token merging strategy that represents a significant advancement in the pursuit of efficient lmms, facilitating the broader adoption of multimodal ai in real-world applications. by addressing the computational challenges of lmms through an innovative integration of adaptiveness with both visual and language guidance, adaptmerge outperforms state-of-the-art token reduction methods, achieving lower computational cost while enhancing task performance across diverse image-language benchmarks.