Benchmarking Retrieval-Augmented Generation for Medicine (2024)

Abstract: While large language models (LLMs) have achieved state-of-the-art performance on a wide range of medical question answering (QA) tasks, they still face challenges with hallucinations and outdated knowledge. Retrieval-augmented generation (RAG) is a promising solution and has been widely adopted. However, a RAG system can involve multiple flexible components, and there is a lack of best practices regarding the optimal RAG setting for various medical purposes. To systematically evaluate such systems, we propose the Medical Information Retrieval-Augmented Generation Evaluation (MIRAGE), a first-of-its-kind benchmark including 7,663 questions from five medical QA datasets. Using MIRAGE, we conducted large-scale experiments with over 1.8 trillion prompt tokens on 41 combinations of different corpora, retrievers, and backbone LLMs through the MedRAG toolkit introduced in this work. Overall, MedRAG improves the accuracy of six different LLMs by up to 18% over chain-of-thought prompting, elevating the performance of GPT-3.5 and Mixtral to GPT-4-level. Our results show that the combination of various medical corpora and retrievers achieves the best performance. In addition, we discovered a log-linear scaling property and the “lost-in-the-middle” effects in medical RAG. We believe our comprehensive evaluations can serve as practical guidelines for implementing RAG systems for medicine.

## introduction large language models (llms) have revolutionized the way people seek information online, from searching to directly asking chatbots for answers. although recent studies have shown their state-ofthe-art capabilities of question answering (qa) in † equal contribution. § co-correspondence. 1 project homepage: https://teddy-xionggz.github. io/benchmark-medical-rag/ both general and medical domains @xcite @xcite @xcite , llms often generate plausible-sounding but factually incorrect responses, commonly known as hallucination @xcite . also, the training corpora of llms might not include the latest knowledge, such as recent updates of clinical guidelines. these issues can be especially dangerous in high-stakes domains such as healthcare @xcite , and will affect the overall performance of llms on domain-specific qa tasks. by providing llms with relevant documents retrieved from up-to-date and trustworthy collections, retrieval-augmented generation (rag) has the potential to address the above challenges @xcite . rag also improves the transparency of llms by grounding their reasoning on the retrieved documents. as such, rag has already been quickly implemented in various scientific and clinical qa systems @xcite . however, a complete rag system contains several flexible modules, such as document collections (corpora), retrieval algorithms (retrievers), and backbone llms, but the best practices for tuning these components are still unclear, hindering their optimal adoption in medicine. to systematically evaluate how different components in rag affect its performance, we first compile an evaluation benchmark termed mi-rage, representing medical information retrieval-augmented generation evaluation. mirage includes 7,663 questions from five commonly used qa datasets in biomedicine. to evaluate rag in realistic medical settings, mirage focuses on the zero-shot ability in rag systems where no demonstrations are provided. we also employ a questiononly setting for the retrieval phase of rag, as in real-world cases where no options are given. for a comprehensive comparison on mirage, we provide medrag, an easy-to-use toolkit that covers five corpora, four retrievers, and six llms including both general and domain-specific models. based on the mirage benchmark, we systematically evaluated different medrag solutions and studied the effects of each component on overall performance from a multidimensional perspective. for various llms, there is a 1% to 18% relative performance increase using medrag compared to chain-of-thought prompting @xcite . notably, with medrag, gpt-3.5 and mixtral @xcite can achieve comparable performance to @xcite on mirage. on the corpus dimension, we found different tasks have a preference over the retrieval corpus. while pointof-care articles and textbooks are solely helpful for examination questions, pubmed is a robust choice for all mirage tasks. our results also show that a combination of all corpora can be a more comprehensive choice. on the retriever dimension, bm25 @xcite and the domain-specific medcpt @xcite retriever display superior performance on our mirage benchmark. the performance can be further enhanced by combining multiple retrievers. beyond the evaluation results on mirage, we found a log-linear scaling relationship between model performance and the number of retrieved snippets. we also observed a "lost-in-the-middle" phenomenon @xcite between model performance and the position of the ground-truth snippet. finally, we provide several practical recommendations based on the results and analyses, which can guide the application and future research of rag in the biomedical domain. in summary, our contributions are three-fold: ## results we systematically evaluate medrag on our mi-rage benchmark, which provides us with a multidimensional analysis of different components in rag for medicine 6 . section 5.1 presents the results for different llms, and section 5.2 includes the results of different corpora and retrievers. based on the results, we provide practical recommendations for rag implementations in section 6.4. ## discussions 6.1 performance scaling we explore how the performance of medrag scales with the increase in the number of snippets used for medical qa. to study the scaling properties, we use gpt-3.5 as the backbone llm, rrf-4 as the retriever, and medcorp as the corpus. figure 3 shows the scaling curves of medrag on each task in mirage with different numbers of snippets @xmath0, 2, 4, ..., 64}. on mmlu-med, medqa-us, and medmcqa, we see roughly loglinear curves in the scaling plots for @xmath1. the results show that when k is small (@xmath2, medrag cannot provide enough useful information, which even hinders the llm from using its inherent knowledge to derive the correct answer. in general, the rag performance improves as k increases, indicating the existence of helpful knowledge from the retrieved snippets. however, the rag performance can drop when k is too large and the signal-noise-ratio begins to decrease. compared with the three examination tasks, pub-medqa* and bioasq-y/n can be relatively easier for medrag since the ground-truth supporting information can be found in pubmed. figure 3 reveals that medrag can achieve high accuracy on pubmedqa* with just @xmath3, and its performance drops with the increase of k as more irrelevant snippets are entered, which corresponds to the fact that 79.6% ground-truth snippets are successfully identified as the top-1 related context by the retrieval system. medrag also shows a dramatic increase in accuracy on bioasq-y/n when @xmath4, whose performance continues to grow as k gets larger. figure 4 shows the changes in model accuracy corresponding to different parts of context locations. from the figure, we can see a clear u-shaped decreasing-then-increasing pattern in the accuracy change concerning the position of ground-truth snippets, which sheds light on the arrangement of snippets for medical rag in future research. ## conclusion to evaluate rag systems in medicine, we introduced the mirage benchmark and the medrag toolkit. based on our comprehensive evaluations, we presented many novel observations and practical recommendations to guide the research and real-world deployments of medical rag systems.