MAFMO : Multi-modal Adaptive Fusion with Meta-template Optimization for Vision-Language Models (2025)

Abstract: Vision-language models like CLIP demonstrate exceptional generalization capabilities but face significant adaptation challenges due to parameter scale, prompt sensitivity, and cross-modal alignment difficulties. Existing approaches primarily focus on single-modality adjustments, leading to suboptimal alignment and limited generalization. We introduce MAFMO, a plug-and-play framework comprising: (1) a Harmonic Cross-Modal Adapter enabling efficient cross-modal knowledge transfer; (2) a Meta-Template Optimization module dynamically generating input-dependent templates; and (3) a Cross-Modal Knowledge Synthesis mechanism preserving critical structural relationships during adaptation. Extensive experiments across multiple fine-grained visual recognition benchmarks demonstrate MAFMO consistently improves existing methods’ performance on both novel classes and harmonic mean, while maintaining robustness under various challenging conditions with minimal computational overhead.

## introduction vision-language models (vlms) have emerged as a pivotal advancement in artificial intelligence research, demonstrating unprecedented capabilities in bridging the semantic gap between visual and textual information @xcite . these models leverage contrastive learning on vast corpora of image-text pairs harvested from the web to create aligned representations across modalities @xcite @xcite @xcite , enabling remarkable zero-shot generalization where models can classify images into categories they have never explicitly been trained to recognize @xcite @xcite . this paradigm shift has revolutionized the field, with models like clip @xcite , align @xcite , and florence @xcite demonstrating robust transfer capabilities across diverse downstream tasks without requiring task-specific labeled data. the core innovation underlying these vlms is their ability to learn generalizable visual concepts from natural language supervision at scale @xcite , leveraging the rich, compositional nature of language to learn nuanced and transferable visual representations that function as open-vocabulary classifiers @xcite @xcite . despite their impressive capabilities, effectively adapting pre-trained vlms to specific downstream tasks presents several significant challenges @xcite . the massive parameter scale of these models-often comprising hundreds of millions to billions of parameters-makes full finetuning computationally prohibitive and potentially vulnerable to overfitting, especially when labeled data is scarce @xcite @xcite . additionally, vlms exhibit a notable sensitivity to prompt engineering, where the specific phrasing used to describe visual concepts can significantly impact performance @xcite . furthermore, naive adaptation techniques may disrupt the delicate cross-modal alignment established during pre-training, potentially compromising the model's generalization capabilities @xcite @xcite . these challenges have spurred extensive research on parameter-efficient transfer learning approaches, with context optimization (coop) @xcite pioneering the concept of learning continuous prompt vectors optimized for specific tasks while updating only 0.01% of the model parameters. building on this foundation, conditional coop (co-coop) @xcite introduced image-conditional prompts that adapt based on the visual input, substantially improving generalization to novel classes. concurrent research has explored various other parameter-efficient adaptation strategies, including prompt learning (proda) @xcite , visual prompt tuning (vpt) @xcite , multi-modal prompt learning (maple) @xcite , and tip-adapter @xcite . however, existing approaches continue to face critical limitations. most prominently, the majority of these methods focus on adapting each modality independently or with limited cross-modal interaction @xcite . for instance, coop and vpt focus exclusively on the textual or visual branch, respectively, without considering their interdependence. even methods that adapt both modalities, such as maple, often do so in parallel without enabling deep interactions between modalities during adaptation. additionally, most approaches utilize fixed template structures that remain constant across all inputs once trained, constraining the model's ability to adapt to the diverse semantic requirements of different visual inputs @xcite @xcite . to address these persistent challenges, we propose multi-modal adaptive fusion with metatemplate optimization (mafmo), a comprehensive plug-and-play enhancement framework for vision-language models.unlike previous approaches that treat adaptation as a primarily modality-specific process, mafmo adopts a holistic perspective that emphasizes harmonized crossmodal interactions while maintaining parameter efficiency and computational tractability. through comprehensive experiments on six diverse finegrained visual recognition datasets, we demonstrate mafmo's effectiveness as a plug-and-play enhancement for various vision-language models, showing particular strength in novel class generalization, few-shot learning, and cross-domain transfer scenarios. our key contributions include: ## methodology recent research in vision-language adaptation has highlighted the importance of simultaneous tuning across modalities for optimal cross-modal alignment. however, existing approaches often adapt each modality independently or with limited crossmodal interaction, leading to suboptimal alignment and constrained generalization capabilities. to address these limitations, we propose multi-modal adaptive fusion with meta-template optimization (mafmo), a novel plug-and-play framework designed to enhance vision-language models through harmonized cross-modal adaptation. mafmo consists of three synergistic components that work together to improve performance while maintaining parameter efficiency, as illustrated in figure 1 . ## experiments we conduct comprehensive experiments to evaluate our proposed mafmo framework as a plug-andplay enhancement for vision-language models. our experiments are designed to answer the following research questions: ## conclusion we introduced mafmo, a plug-and-play framework enhancing vision-language models through harmonic cross-modal adaptation, meta-template optimization, and knowledge synthesis. comprehensive experiments across multiple fine-grained recognition datasets demonstrate mafmo's consistent performance improvements, with particularly strong enhancements for models with limited cross-modal interaction capabilities. our frame-work exhibits remarkable robustness under challenging conditions including few-shot learning scenarios, cross-dataset generalization, and adversarial attacks, while maintaining high parameter efficiency and minimal inference overhead. extended evaluations on diverse datasets confirm mafmo's effectiveness across various visual domains and its robustness to distribution shifts, highlighting its potential as an effective and practical approach for adapting vision-language models across diverse visual recognition tasks.