MAM : Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis via Role-Specialized Collaboration (2025)

Abstract: Recent advancements in medical Large Language Models (LLMs) have showcased their powerful reasoning and diagnostic capabilities. Despite their success, current unified multimodal medical LLMs face limitations in knowledge update costs, comprehensiveness, and flexibility. To address these challenges, we introduce the Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis (MAM). Inspired by our empirical findings highlighting the benefits of role assignment and diagnostic discernment in LLMs, MAM decomposes the medical diagnostic process into specialized roles: a General Practitioner, Specialist Team, Radiologist, Medical Assistant, and Director, each embodied by an LLM-based agent. This modular and collaborative framework enables efficient knowledge updates and leverages existing medical LLMs and knowledge bases. Extensive experimental evaluations conducted on a wide range of publicly accessible multimodal medical datasets, incorporating text, image, audio, and video modalities, demonstrate that MAM consistently surpasses the performance of modality-specific LLMs. Notably, MAM achieves significant performance improvements ranging from 18% to 365% compared to baseline models. Our code, data, and prompts are released at https://github.com/yczhou001/MAM .

## introduction large language models (llms) have recently demonstrated remarkable reasoning capabilities @xcite @xcite @xcite @xcite . beyond demonstrating impressive language reasoning and generation capabilities, llms are expanded to process diverse modalities, e.g., images, audio, and video @xcite @xcite . this formidable reasoning capacity holds significant promise for addressing problems in medical diagnostics. for medical practice, physicians are confronted with a deluge of heterogeneous medical data, encompassing textual reports, medical images, cardiac sounds, and even surgical video recordings. accurately extracting critical information from this complex data to arrive at precise diagnoses places a significant cognitive burden and challenge on clinicians. furthermore, the growth in the volume of medical diagnostic data provides a substantial foundation for the training of llms. consequently, the development of llms to enhance medical diagnostic workflows is crucial. however, many efforts are directed towards constructing unified multimodal medical large models @xcite @xcite . while these models have shown some progress in integrating multimodal information, they suffer from two limitations. firstly, for unified models, each knowledge update is cost, often requiring substantial computational resources to retrain the entire model. secondly, unified models lack modularity and flexibility, necessitating a single model to exhibit sufficient performance across various medical diagnostic tasks to satisfy demands. to explore the capabilities of existing domain-specific llms, we conducted an empirical study. our findings indicate that role assignment significantly enhances the diagnostic abilities of llms, and llms possess the potential to discern the correct diagnosis from multiple ones. to overcome the aforementioned limitations of unified multimodal medical llms and to better emulate the collaborative approach of human medical teams, we propose the multi-agent framework for multi-modal medical diagnosis (mam). instead of pursuing an "omnipotent" unified model, mam framework decomposes the medical diag-nostic process into several specialized roles and designs llm-based agents for each role. these agents include: a general practitioner agent responsible for initial triage, a specialist team agent providing domain-expert opinions, a radiologist agent specializing in medical image analysis, a medical assistant agent aiding information retrieval and knowledge management, and a director agent responsible for coordinating and synthesizing diagnostic opinions. the core advantages of the mam framework lie in its modular design and collaborative workflow. the modular agent design enables more granular and efficient knowledge updates and model maintenance, without requiring global retraining. the framework also allows mam to easily integrate and leverage various existing medical models and specialized knowledge bases. in the experiments, we evaluate our mam framework in multimodal medical diagnosis tasks through comprehensive experiments on several publicly available multimodal medical datasets. experimental results demonstrate that the mam framework consistently outperforms specific-modal llms across various medical datasets and data modalities. in addition, we conduct ablation studies, consistency analysis, and sensitivity analyses regarding the number of discussion rounds and roles to gain deeper insights into the roles of individual components and the operational mechanisms of the framework. ## method based on our preliminary empirical studies, we observed that augmenting large language models (llms) with specific medical roles significantly enhances their diagnostic performance. furthermore, llms demonstrate a notable capacity to reason and synthesize correct diagnoses from diverse diagnostic opinions. inspired by these findings, we propose the multi-agent medical (mam) framework, shown in figure 1 . this framework aims to transform multi-modal medical diagnosis into a collaborative endeavor, thereby amplifying the diagnostic capabilities of existing models. it comprises five key roles, each embodied by an llm-based agent, working synergistically within a defined workflow: ① general practitioner: responsible for initial disease type classification and referral to specialist. ② specialist team: charged with providing diagnostic opinions on specific medical conditions and actively participating in discussions. ③ radiologist: tasked with analyzing medical images and contributing to diagnostic discussions. ④ medical assistant: responsible for retrieving and summarizing relevant medical information from databases. ⑤ director: synthesizes discussion reports and reviews the quality of medical diagnoses. in our framework, multi-modal medical inputs are initially directed to the general practitioner, who performs disease classification and subsequently refers the case to the relevant specialist team. the specialist teams will decompose and anonymize the medical problem. the medical assistant then retrieves and summarizes pertinent information from medical databases based on the decomposed problem. subsequently, the director orchestrates discussions among the specialist team, where each specialist presents their diagnostic opinion. the director then synthesizes these opinions and the database summaries into a comprehensive report. the specialist team reviews this report and votes on whether to endorse it. in cases of disagreement, the process iteratively re-enters the specialist team discussion phase. otherwise, upon reaching a consensus, the director derives the final diagnosis based on the synthesized report. ## conclusion this study introduces the multi-agent framework for multi-modal medical diagnosis (mam), addressing the limitations of unified multimodal medical llms. mam employs a modular, collaborative approach, assigning specialized roles, i.e., general practitioner, specialist team, radiologist, medical assistant, and director, to distinct llmbased agents. this structure enhances knowledge updates, leverages specialized expertise, and adapts to diverse medical tasks and modalities. extensive evaluations on multimodal medical datasets demonstrate mam's superiority, outperforming modalityspecific llms by 18% to 365%. future work will integrate advanced knowledge retrieval and evaluate mam in real-world clinical settings.