Agentic Medical Knowledge Graphs Enhance Medical Question Answering: Bridging the Gap Between LLM s and Evolving Medical Knowledge (2025)

Abstract: Large Language Models (LLMs) have greatly advanced medical Question Answering (QA) by leveraging vast clinical data and medical literature. However, the rapid evolution of medical knowledge and the labor-intensive process of manually updating domain-specific resources can undermine the reliability of these systems. We address this challenge with Agentic Medical Graph-RAG (AMG-RAG), a comprehensive framework that automates the construction and continuous updating of Medical Knowledge Graph (MKG), integrates reasoning, and retrieves current external evidence from the MKG for medical QA.Evaluations on the MEDQA and MEDMCQA benchmarks demonstrate the effectiveness of AMG-RAG, achieving an F1 score of 74.1% on MEDQA and an accuracy of 66.34% on MEDMCQA—surpassing both comparable models and those 10 to 100 times larger. By dynamically linking new findings and complex medical concepts, AMG-RAG not only boosts accuracy but also enhances interpretability for medical queries, which has a critical impact on delivering up-to-date, trustworthy medical insights.

## introduction medical knowledge is growing at an unprecedented rate: every day brings new research findings, revised clinical guidelines, and updated treatment protocols. recent work shows that large language models (llms) can already harness this ever-expanding corpus for medical question answering (qa) @xcite . despite their promise, llms face two persistent challenges. first, they must remain factually current in a field where knowledge can become obsolete almost overnight @xcite . second, they must correctly figure 1 : performance versus parameter count on the medqa and medmcqa benchmarks. our system, agentic medical graph-rag (amg-rag), attains an f1 of 74.1 % on medqa and an accuracy of 66.34 % on medmcqa, outperforming models that contain 10-100× more parameters. see tables 1 and 2 for details. model the intricate relationships among information entities. knowledge graph (kg) provides a structured and interconnected view of information that supports nuanced reasoning @xcite , yet creating and maintaining them by hand is costly-especially in medicine, where new insights rapidly invalidate older facts @xcite . we introduce an automated framework that constructs and continuously refines medical knowledge graphs (mkgs) for medical qa. our llm-driven agentic workflow, amg-rag, assisted by domain-specific search tools, generates graph entities enriched with metadata, confidence scores, and relevance indicators. this automation sharply reduces manual curation while keeping the graph aligned with the latest discoveries. in contrast to retrieval augmented generation (rag) systems that rely solely on vector similarity @xcite , our graph-centric retrieval leverages explicit relationships to synthesise information across domains such as drug interactions, clinical trials, patient histories, and guidelines. the amg-rag combines dynamically synthesised mkg with multi-step reasoning, guided by confidence scores and adaptive traversal strategies @xcite . this design yields more accurate and complete answers without incurring additional fine-tuning or inference costs. on the medqa and medmcqa benchmarks-both of which test evidence retrieval, complex reasoning, and multiple-choice comprehension-amg-rag achieves an f1 of 74.1% and an accuracy of 66.34%, respectively (fig. 1 ). these results surpass those of similarly sized rag approaches and even much larger state-of-the-art models, underscoring the benefits of a dynamically evolving mkg for medical qa. our findings highlight the potential of automated, relationally enriched knowledge retrieval to enhance clinical decision-making by delivering timely and trustworthy insights @xcite . contributions. our contributions are threefold: ## related work medical qa has progressed mainly through three complementary lines of research: domain-specific language models. biobert @xcite , pubmedbert @xcite , and medpalm @xcite adapt transformer pre-training to biomedical corpora, delivering strong gains on entity recognition, relation extraction, and multiple-choice qa @xcite . yet, even these specialised models struggle to synthesise multi-hop relations (e.g., rare comorbidities or drug-gene interactions) and must be re-trained to absorb new discoveries @xcite . ## method we propose our framework, agentic medical graph-rag (amg-rag), bridges these threads by dynamically generating a confidence-scored medical knowledge graph (mkg) that is tightly coupled to a retrieval augmented generation (rag)+cot pipeline. amg-rag features autonomous knowledge graph (kg) evolution through large language model (llm) agents extracting entities and relations from live sources with provenance tracking; graph-conditioned retrieval that maps queries onto the mkg to guide evidence selection; and reasoning over structured context where the answer generator utilizes both textual passages and traversed sub-graphs for transparent, multi-hop reasoning. ## experiments the medqa dataset is a free-form, multiplechoice open-domain qa dataset specifically designed for medical qa. derived from professional medical board exams, this dataset presents a significant challenge as it requires both the retrieval of relevant evidence and sophisticated reasoning to answer questions accurately. each question is accompanied by multiple-choice answers that demand a deep understanding of medical concepts and logical inference, often relying on evidence found in medical textbooks. for this study, the test partition of the medqa dataset, comprising approximately 1,200 samples, was used @xcite . the medmcqa dataset is another multiplechoice question-answering dataset tailored for medical qa. unlike medqa, which is derived from board exam questions, medmcqa offers a broader variety of question types, encompassing both foundational and clinical knowledge across diverse medical specialties. in this study, the medmcqa development set, containing approximately 4,000 questions, was used to benchmark against other models @xcite . this study employed the medqa and medm-cqa datasets to benchmark and evaluate medical qa systems. these datasets serve as challenging testbeds for open-domain qa tasks due to their demands for multi-hop reasoning and the integration of domain-specific knowledge. the relevance of medqa in the real world, together with the diverse question styles and extensive development set of medmcqa make them ideal for advancing the development of robust qa models capable of addressing medical inquiries. we utilize gpt-4omini as the backbone of the implementation for both mkg and amg-rag, leveraging its capabilities with approximately ∼ 8b parameters. this model serves as the core component, enabling advanced reasoning, rag, and structured knowledge integration. ## conclusion we introduce amg-rag, an advanced qa system that dynamically constructs mkg while integrating sophisticated structured reasoning for medical qa. the system demonstrates significant improvements in accuracy and reasoning capabilities, particularly for medical question-answering tasks, outperforming other approaches of similar model size or 10 to 100 times larger. 12690 despite amg-rag's advancements, our approach has certain limitations. firstly, it relies on external search tools which introduce latency during the creation of the mkg. however, this occurs only once, when the mkg is built from scratch for the first time. additionally, while the system performs exceptionally well in medical domains, its applicability to non-medical tasks remains unexplored. another limitation is the need for structured, authoritative sources of medical knowledge. currently, amg-rag retrieves information from diverse sources, including research articles and medical textbooks. however, as emphasized in clinical decision-making, treatment guidelines serve as essential references for standardized diagnosis and treatment protocols @xcite . future work on amg-rag should focus on integrating structured access to these sources to ensure compliance with evidence-based medicine.