MM ed A gent: Learning to Use Medical Tools with Multi-modal Agent (2024)

Abstract: Multi-Modal Large Language Models (MLLMs), despite being successful, exhibit limited generality and often fall short when compared to specialized models. Recently, LLM-based agents have been developed to address these challenges by selecting appropriate specialized models as tools based on user inputs. However, such advancements have not been extensively explored within the medical domain. To bridge this gap, this paper introduces the first agent explicitly designed for the medical field, named M ulti-modal Med ical Agent (MMedAgent). We curate an instruction-tuning dataset comprising six medical tools solving seven tasks across five modalities, enabling the agent to choose the most suitable tools for a given task. Comprehensive experiments demonstrate that MMedAgent achieves superior performance across a variety of medical tasks compared to state-of-the-art open-source methods and even the closed-source model, GPT-4o. Furthermore, MMedAgent exhibits efficiency in updating and integrating new medical tools.

## introduction multi-modal large language models (mllms) have made considerable progress across diverse tasks with inputs from different medical imaging modalities (e.g., magnetic resonance imaging, computed tomography, x-ray) in healthcare, including visual question answering (vqa) @xcite @xcite , image segmentation @xcite , and medical report generation (mrg) @xcite , etc. despite these advancements, mllms often exhibit limitations in seamlessly solving multiple tasks across different medical imaging modalities. although recent large medical models (zhang et al., 2023b; tu et al., * corresponding authors 2024; @xcite @xcite have attempted to address this challenge, they remain limited to handling a narrow range of tasks across a restricted set of imaging modalities and cannot be efficiently extended to new tasks or more imaging modalities. furthermore, these generalists typically do not provide expert-level responses comparable to those of specialized mllms customized for specific tasks. one way to address this issue is to build an ai agent, an ai system driven by large language models (llms) that integrates various domain expert models as tools. such a system can understand user instructions, make decisions, and select the appropriate tools to execute any specific task, thereby generating expert-level responses for any given request @xcite @xcite @xcite . despite the significant success of ai agents in the general image domain @xcite @xcite , there are currently few ai agents developed specifically for the medical domain. although several works @xcite @xcite in the medical field use the term "agent" in their methods, they focus on utilizing llms to play various roles and collaborate on complex tasks, in which an "agent" refers to a specific role. pathasst @xcite integrates tool utilization into their framework but specifically designed for pathology tasks. in this work, we aim to build the first ai agent specifically for the medical domain, termed as multi-modal medical agent (mmedagent). we choose llava-med @xcite as the backbone and aim to extend its capability to handle various language and multi-modal tasks, including grounding, segmentation, classification, mrg, and retrieval-augmented generation (rag). these tasks encompass multiple medical imaging modalities, such as mri, ct, and x-ray, allowing mmedagent to support a wide range of data types typically encountered in clinical practice. the first step to building mmedagent is to collect the stateof-the-art (sota) methods for each task, hereafter referred to as "tools". during this phase, we identify a lack of an effective tool for the grounding task, prompting us to fine-tune grounding dino @xcite specifically for medical applications. next, we build an instruction-based dataset that teaches the agent to select the proper tool(s) when encountering a user instruction and aggregate the outputs from tools to reply to users precisely and comprehensively. the core of our approach involves an end-to-end training regimen through visual instruction tuning @xcite . mmedagent has demonstrated promising results in various aspects. when evaluated on several complex medical tasks, mmedagent significantly outperforms sevaral open-source sota methods, including llava @xcite , flamingo-med @xcite , yi-vl-34b @xcite , qwen-vl-chat @xcite , llava-med @xcite and radfm @xcite , and even surpasses close-source method, gpt-4o (openai, 2024), on average. it also enhances mmedagent's backbone, i.e., llava-med, original capability in the vqa task, as well as exhibits efficient capability in learning new tools. our code and web ui are available at https: //github.com/wangyixinxin/mmedagent , and a demonstration of the user interface is provided in appendix figure 6 . our contributions can be summarized as: ## mmedagent multi-modal medical agent (mmedagent), a system based on an mllm, is designed to seamlessly manage diverse medical tasks by integrating various open-source medical models. mmedagent comprises two components: (1) an instructiontuned multi-modal llm that functions as an action planner and results aggregator, and (2) a collection of medical tools tailored to the agent, each targeting specific tasks in the medical domain. we first present the fundamental workflow of mmedagent in section 3.1, followed by a description of creating an instruction-tuning dataset for training the multi-modal llm as an action planner in section 3.2. the details of medical tasks and corresponding tools incorporated in mmedagent are described in section 3.3. ## experimental settings mmedagent is initialized with llava-med 60k-im, instruction-tuned using lora @xcite for 15 epochs, and conducted over approximately 72 hours on two 80g nvidia a100 gpus. the rank of lora is set to 128, and the training batch size is set to 48. we employ adamw @xcite as the optimizer alongside a cosine learning rate schedule peaking at 2e-4. we generate 48k instruction-tuning data, consisting of 15k augmented vqa instruction following the method from llava-plus @xcite derived from 60k inline mentions @xcite , 10k data points for detection, 3k for rag, 5k each for segmentation, classification, mrg, and g-seg. data sources are shown in ## experimentals we conduct experiments on mmedagent to answer three research questions: (1) what is the performance of mmedagent in addressing diverse medical tasks across various modalities (section 5.1)? ## conclusion we propose mmedagent, the first multi-modal medical ai agent that is capable of seamlessly utilizing various medical tools to handle a broad spectrum of medical tasks across different imaging modalities. we create an instruction-tuning dataset that mmedagent utilize to learn to invoke various medical tools and aggregate results from tools. comprehensive experiments demonstrate that mmedagent significantly outperforms opensource baselines and even surpasses gpt-4o across many medical tasks. furthermore, mmedagent efficiently integrates with new tools while remaining the capability to activate previously learned tools.