M ed A gents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2024)

Abstract: Large language models (LLMs), despite their remarkable progress across various general domains, encounter significant barriers in medicine and healthcare. This field faces unique challenges such as domain-specific terminologies and reasoning over specialized knowledge. To address these issues, we propose MedAgents, a novel multi-disciplinary collaboration framework for the medical domain. MedAgents leverages LLM-based agents in a role-playing setting that participate in a collaborative multi-round discussion, thereby enhancing LLM proficiency and reasoning capabilities. This training-free framework encompasses five critical steps: gathering domain experts, proposing individual analyses, summarising these analyses into a report, iterating over discussions until a consensus is reached, and ultimately making a decision. Our work focuses on the zero-shot setting, which is applicable in real-world scenarios. Experimental results on nine datasets (MedQA, MedMCQA, PubMedQA, and six subtasks from MMLU) establish that our proposed MedAgents framework excels at mining and harnessing the medical expertise within LLMs, as well as extending its reasoning abilities. Our code can be found at https://github.com/gersteinlab/MedAgents.

## introduction large language models (llms) @xcite @xcite @xcite have exhibited notable generalization abilities across a wide range of tasks and applications @xcite @xcite , with these capabilities stemming from their extensive training on vast comprehensive corpora covering diverse topics. however, in real-world scenarios, llms tend to encounter domain-specific tasks that * equal contribution. necessitate a combination of domain expertise and complex reasoning abilities @xcite @xcite . amidst this backdrop, a noteworthy research topic lies in the adoption of llms in the medical field, which has gained increasing prominence recently @xcite @xcite . two major challenges prevent llms from effectively handling tasks in the medical sphere: (i) limited volume and specificity of medical training data compared to the vast general text data, due to cost and privacy concerns @xcite . 1 (ii) the demand for extensive domain knowledge @xcite and advanced reasoning skills @xcite makes eliciting medical expertise via simple prompting challenging @xcite . although numerous attempts, particularly within math and coding, have been made to enhance prompting methods @xcite @xcite , strategies used in the medical field have been shown to induce hallucination @xcite @xcite , indicating the need for more robust approaches. meanwhile, recent research has surprisingly witnessed the success of multi-agent collaboration @xcite which highlights the simulation of human activities @xcite @xcite and optimizes the collective power of multiple agents @xcite @xcite . through such design, the expertise implicitly embedded within llms, or that the model has encountered during its training, which may not be readily accessible via traditional prompting, is effectively brought to the fore. this process subsequently enhances the model's reasoning capabilities throughout multiple rounds of interaction @xcite . motivated by these notions, we pioneer a multi-disciplinary collaboration framework (medagents) specifically tailored to the clinical domain. our objective centers on unveiling the intrinsic medical knowledge embedded in llms and reinforcing reasoning proficiency in a training-free manner. as shown in figure 1 , the medagents framework gathers experts from diverse disciplines and reaches consistent conclusions through collaborative discussions. based on our medagents framework, we conduct experiments on nine datasets, including medqa @xcite , medmcqa @xcite , pubmedqa @xcite and six medical subtasks from mmlu @xcite ). 2 to better align with real-world application scenarios, our study focuses on the zero-shot setting, which can serve as a plug-and-play method to supplement existing medical llms such as med-palm 2 @xcite . encouragingly, our proposed approach outperforms settings for both chain-of-thought (cot) and self-consistency (sc) prompting methods. most notably, our approach achieves better performance under the zero-shot setting compared with the 5-shot baselines. based on our results, we further investigate the influence of agent numbers and conduct human evaluations to pinpoint the limitations and issues prevalent in our approach. we find four common categories of errors: (i) lack of domain knowledge, (ii) mis-retrieval of domain knowledge, (iii) consistency errors, and (iv) cot errors. targeted refinements focused on mitigating these particular shortcomings would enhance the model's proficiency and reliability. our contributions are summarized as follows: ## method this section presents the details of our proposed multi-disciplinary collaboration medagents framework. figure 1 and 2 give an overview and an illustrative example of its pipeline, respectively. our proposed medagents framework works in five stages: (i) expert gathering: assemble experts from various disciplines based on the clinical question; (ii) analysis proposition: domain experts present their own analyses with their expertise; (iii) report summarization: develop a report summary on the basis of previous analyses; ## conclusion we present a novel medical qa framework that uses role-playing agents for multi-round discussions, offering greater reliability and clarity without prior training. our method surpasses zeroshot baselines and matches few-shot baselines across nine datasets. despite successes, humanbased evaluations of errors have highlighted areas for refinement. our approach differs from most existing methods by eliminating the dependency on knowledge bases, instead uniquely integrating medical knowledge through role-playing agents.