TAMP : Token-Adaptive Layerwise Pruning in Multimodal Large Language Models (2025)

Abstract: Multimodal Large Language Models (MLLMs) have shown remarkable versatility in understanding diverse multimodal data and tasks. However, these capabilities come with an increased model scale. While post-training pruning reduces model size in unimodal models, its application to MLLMs often yields limited success. Our analysis discovers that conventional methods fail to account for the unique token attributes across layers and modalities inherent to MLLMs. Inspired by this observation, we propose TAMP, a simple yet effective pruning framework tailored for MLLMs, featuring two key components: (1) Diversity-Aware Sparsity, which adjusts sparsity ratio per layer based on diversities among multimodal output tokens, preserving more parameters in high-diversity layers; and (2) Adaptive Multimodal Input Activation, which identifies representative multimodal input tokens using attention scores to guide unstructured weight pruning. We validate our method on two state-of-the-art MLLMs: LLaVA-NeXT, designed for vision-language tasks, and VideoLLaMA2, capable of processing audio, visual, and language modalities. Empirical experiments across various multimodal evaluation benchmarks demonstrate that each component of our approach substantially outperforms existing pruning techniques. Our code is available at https://github.com/G-JWLee/TAMP

## introduction large language models (llms) have achieved remarkable success at billion-parameter scales @xcite , excelling in challenging tasks. building on this, multimodal large language models (mllms) @xcite @xcite , which extend llms to handle diverse modality inputs, have grown in size to address the complexities of multimodal tasks @xcite @xcite . while beneficial output token feature space mllm layer ⋯ ⋯ ⋯ biased input activation 𝐈𝐈 = 𝐖𝐖 � 𝐗𝐗 𝟐𝟐 weight pruning 𝑵𝑵 𝒂𝒂 𝐗𝐗 ⋯ ⋯ ⋯ 𝐙𝐙 𝑵𝑵 𝒗𝒗 𝑵𝑵 𝒍𝒍 unique output distributions of each modality 𝑵𝑵 𝒂𝒂 𝑵𝑵 𝒗𝒗 𝑵𝑵 𝒍𝒍 ## method in this section, we first present empirical studies that reveal key properties of multimodal tokens and their implications for pruning. based on these insights, we introduce the core components of token adaptive multimodal pruning (tamp). the overall framework is illustrated in figure 2 . ## conclusion in this paper, we investigate the critical challenges in multimodal large language model (mllm) pruning. in our comprehensive investigations, we find that different mllm layers have varying capabilities to encode multimodal output tokens. we also empirically observe that existing pruning methods fail to address varying modality reliance across blocks in mllms, resulting in suboptimal pruning outcomes. based on these observations, we introduce tamp, a novel pruning framework that adapts both layer-wise sparsity and input activations to each layer's multimodal token attributes. we validate our method on powerful mllms and extensive experiments demonstrate that tamp is effective in preserving diverse multimodal abilities, even at extreme model sparsity. we believe our work offers a strong foundation for future advancements in mllm pruning, enabling the deployment of recent mllms in resource-constrained scenarios.