MEXA : Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregation (2025)

Abstract: Combining pre-trained expert models offers substantial potential for scalable multimodal reasoning, but building a unified framework remains challenging due to the increasing diversity of input modalities and task complexity. For instance, medical diagnosis requires precise reasoning over structured clinical tables, while financial forecasting depends on interpreting plot-based data to make informed predictions. To tackle this challenge, we introduce MEXA, a training-free framework that performs modality- and task-aware aggregation of multiple expert models to enable effective multimodal reasoning across diverse and distinct domains. MEXA dynamically selects expert models based on the input modality and the task-specific reasoning demands (i.e., skills). Each expert model, specialized in a modality task pair, generates interpretable textual reasoning outputs. MEXA then aggregates and reasons over these outputs using a Large Reasoning Model (LRM) to produce the final answer. This modular design allows flexible and transparent multimodal reasoning across diverse domains without additional training overhead. We extensively evaluate our approach on diverse multimodal benchmarks, including Video Reasoning, Audio Reasoning, 3D Understanding, and Medical QA. MEXA consistently delivers performance improvements over strong multimodal baselines, highlighting the effectiveness and broad applicability of our expert-driven selection and aggregation in diverse multimodal reasoning tasks.

## introduction the rapid advancement of multimodal learning has significantly improved ai systems' ability to understand, reason, and interact with the real world, benefiting diverse tasks such as visual question answering (antol et al., 2015; hudson and manning, 2019; marino et al., 2019; mathew et al., 2021; tanaka * equal contribution. et al., 2023; winata et al., 2024; wang et al., 2025; zhang et al., 2024b; guo et al., 2025), medical image diagnosis @xcite @xcite @xcite , and embodied ai @xcite @xcite @xcite . as ai systems become more integrated into complex, real-world applications, the ability to flexibly understand and reason upon heterogeneous multimodal inputs is increasingly essential. for example, a medical diagnostic system may need to interpret ct scans, extract structured information from clinical notes, and reason about temporal patterns in patient histories, while a financial forecasting system must effectively analyze and interpret plot data to make informed predictions about market trends and economic risks. however, despite recent advances in multimodal learning, the increasing diversity and complexity of multimodal data pose significant challenges for developing a flexible and unified framework that can reason effectively across modalities and generalize across tasks requiring diverse skills. existing multimodal architectures typically require training separate encoders tailored to each modality, along with designing complex cross-modal alignment mechanisms @xcite @xcite @xcite . while effective, these models require finetuning for each specific task, leading to substantial training overhead and limiting their adaptability to new modalities or tasks. moreover, previous approaches @xcite often implicitly fuse multimodal inputs at early stages, restricting transparency and interpretability of the reasoning process. instead, an ideal multimodal framework should seamlessly process inputs from any modality for any given task, dynamically directing these inputs to the most suitable multimodal expert models. this necessitates a modular design where each expert model is associated explicitly with skills corresponding to particular modalitytask, thereby ensuring precise skill matching and enhancing interpretability. in this paper, we propose multimodal expert aggregator (mexa), a novel training-free multiexpert aggregation framework that dynamically coordinates a pool of specialized experts. mexa selectively activates and aggregates reasoning outputs from the most relevant expert models for each input instance, guided by both the input modalities (e.g., image, audio, 3d, medical scan) and the required level of reasoning demands (i.e., skills) for that instance. mexa spans a wide spectrum of tasks, from low-level perceptual reasoning (e.g., object recognition or ocr) to high-level cognitive inference (e.g., temporal event understanding in video or diagnostic interpretation in medical imaging). unlike existing methods that rely on monolithic, end-to-end fine-tuned multimodal models or fixed, statically defined expert pipelines, mexa's modular expert coordination enables flexible, interpretable, and scalable multimodal reasoning. specifically, we first assemble a diverse pool of expert models, each designed to handle distinct aspects of modalities and tasks, to address mainstream multimodal challenges effectively. each expert specializes in extracting information unique to its respective skill and encoding it into a unified textual representation. then, to effectively coordinate these experts, we design an expert selection module that employs a multimodal large language model (mllm) as a versatile router within our framework. this router dynamically selects and activates the appropriate experts by analyzing the modality of the input data and identifying the specific skills required based on the input query and task requirements. finally, we introduce an aggregator that reasons over the outputs of the selected experts using a large reasoning model (lrm), which excels at long-context understanding and producing complex long cot reasoning during lrm inference. instead of relying on heuristic merging, the aggregator systematically integrates the complementary information from each expert and infers the final answer based on the full context of the target task. we perform extensive evaluations of our framework on a diverse set of challenging multimodal benchmarks containing various modalities and requiring diverse skills, including expert knowledge heavy video reasoning (video-mmmu @xcite ), audio qa (mmau @xcite ), 3d scene understanding (sqa3d @xcite ), and medical imaging-based qa (m3d @xcite ). experimental results show that our proposed method consistently outperforms strong multimodal baseline models across all evaluated benchmarks, with accuracy gains of +5.7% on video-mmmu, +12.2% on mmau, +1.7% on sqa3d, and +1.6% on m3d in the corresponding metrics, demonstrating the effectiveness, flexibility, and robustness of our expert-driven multimodal aggregation framework. we further conduct ablation studies on expert-selection router design and llm aggregator to provide more insights for future work. our contributions are summarized as follows: ## mexa: dynamic multi-modal expert aggregation we first highlight the challenges faced by generalpurpose multimodal reasoning systems in sec. 3.1 and introduce our skill-specialized expert aggregation framework as a solution in sec. 3.2. within sec. 3.2, we first describe the organization of a pool of expert modules and our strategy for extracting expertise-specific text representations from the specialized experts(sec. 3.2.1). next, we introduce an expert selection module that dynamically determines which experts to activate based on the input query, target modality, required skill, and reasoning complexity (sec. 3.2.2). finally, we present our expert aggregation strategy, which integrates the outputs from the selected expert to generate a final answer (sec. 3.2.3). ## conclusion in this paper, we introduce mexa, a dynamic multimodal reasoning framework that adaptively leverages a diverse set of expert modules with the skills for specific modalities and tasks. by effectively integrating an expert selection module and aggregation mechanism, our method achieves superior performance across a variety of challenging benchmarks, including video reasoning, audio qa, 3d situated reasoning, and medical qa. extensive experimental analyses and visualization results demonstrate the clear advantages of our dynamic and modality-aware design, highlighting the significance of selectively combining complementary expert skills. we believe that our framework provides a robust foundation for future research in generalized multimodal reasoning, enabling models to tackle increasingly complex real-world multimodal applications.