C linical RAG : Enhancing Clinical Decision Support through Heterogeneous Knowledge Retrieval (2024)

Abstract: Large Language Models (LLMs) have revolutionized text generation across diverse domains, showcasing an ability to mimic human-like text with remarkable accuracy. Yet, these models frequently encounter a significant hurdle: producing hallucinations, a flaw particularly detrimental in the healthcare domain where precision is crucial. In this paper, we introduce ClinicalRAG, a novel multi-agent pipeline to rectify this issue by incorporating heterogeneous medical knowledge—both structured and unstructured—into LLMs to bolster diagnosis accuracy. ClinicalRAG can extract related medical entities from user inputs and dynamically integrate relevant medical knowledge during the text generation process. Comparative analyses reveal that ClinicalRAG significantly outperforms knowledge-deficient methods, offering enhanced reliability in clinical decision support. This advancement marks a pivotal proof-of-concept step towards mitigating misinformation risks in healthcare applications of LLMs.

## introduction large language models (llms) have demonstrated remarkable capabilities in generating human-like text across various domains @xcite @xcite . however, these models often produce hallucinations-generating inaccurate or entirely fictitious information. this issue is particularly critical in sensitive domains like healthcare, where misinformation can have dire repercussions @xcite . the underlying cause of such hallucinations largely stems from the model's insufficient domain-specific knowledge. medical domain is characterized by its vast array of knowledge, which includes both structured information (such as knowledge graphs, medical databases) and unstructured information (like online resources) @xcite . these knowledge are inherently heterogeneous, spanning various subfields and formats, which poses significant challenges for traditional models that rely on a one-size-fits-all approach to knowledge integration and application. as a result, the discrepancies among knowledge sources impede the models' ability to utilize knowledge prompts from all available sources effectively. in light of this, we aim to propose a method that can seamlessly integrate and accommodate all source of medical knowledge. retrieval-augmented generation (rag) offers a powerful approach for harnessing the implicit knowledge embedded within llms alongside diverse explicit knowledge sources @xcite . through real-time retrieval of pertinent information during the generation phase, rag models are adept at delivering outputs that are both precise and contextually relevant @xcite . consequently, it empowers the models to efficiently access domain-specific information, enhancing the quality of the generated content. in this paper, we introduce a clinical retrieval augmented generation (clinicalrag) pipeline (fig. 1 ), a novel framework designed to enhance clinical decision-making by leveraging medical knowledge from a variety of sources. our contributions are threefold and can be summarized as follows: 1) we develope a multi-agent integration approach, where each agent is responsible for a specific part of the clinicalrag process. this ensures the efficiency and robustness of the pipeline. 2) we design an effective solution for the extraction and integration of heterogeneous medical knowledge, which, compared to long text inputs, allows for the low-cost acquisition of high-quality information. 3) experimental results demonstrate that our clinicalrag pipeline outperforms traditional methods such as simple prompt learning and direct generation. it also provides relevant references, facilitating a more effective clinical decision support. ## related work recent literature on knowledge-enhanced clinical decision support (cds) systems showcases a plethora of innovative approaches aimed at leveraging technology to improve healthcare outcomes. @xcite implements ant colony optimization methods within cds systems to customize treatment plans for patients, thus enriching the knowledge base for making optimal clinical decisions. @xcite leverage a knowledge graph and an attribute graph to generate better medicine recommendations. recently, @xcite have introduced prompt learning methods for integrating heterogeneous medical knowledge. moreover, the development of llms enables a more precise and effective way to utilize current medical knowledge. one useful method is chain-of-thought @xcite which mimics human problem-solving processes by breaking down complex questions into simpler, manageable parts. based on this, tree-of-thought @xcite and graph-of-thought @xcite methods are proposed to deal with more complex question-solving flow. additionally, by integrating external knowledge, rag significantly improves the quality of the generated content, making it more informative and accurate across various tasks @xcite , demonstrating its effectiveness in enriching language model outputs with detailed and precise information. ## methods the detailed pipeline of clinicalrag is shown in fig. 2 . we employ a multi-agent strategy in clini-calrag, each agent is designed to carry out different task. ## conclusion in this paper, we presented clinicalrag, a novel multi-agent pipeline that significantly enhances the accuracy and reliability of clinical decision support provided by llms. by seamlessly integrating heterogeneous medical knowledge-ranging from structured knowledge graphs and to unstructured medical knowledge bases and online resources-clinicalrag addresses the critical challenge of hallucinations and inaccuracies in llmgenerated content within the healthcare domain. our comprehensive experiments have demonstrated the superior diagnosis performance of the clinicalrag pipeline over traditional methods.