Rationale-Guided Retrieval Augmented Generation for Medical Question Answering (2025)

Abstract: Large language models (LLM) hold significant potential for applications in biomedicine, but they struggle with hallucinations and outdated knowledge.While retrieval-augmented generation (RAG) is generally employed to address these issues, it also has its own set of challenges: (1) LLMs are vulnerable to irrelevant or unhelpful context, (2) medical queries are often not well-targeted for helpful information, and (3) retrievers are prone to bias toward the specific source corpus they were trained on. In this study, we present RAG ^2 (RAtionale-Guided RAG), a new framework for enhancing the reliability of RAG in biomedical contexts. RAG ^2 incorporates three key innovations: a small filtering model trained on perplexity-based labels of rationales, which selectively augments informative snippets of documents while filtering out distractors; LLM-generated rationales as queries to improve the utility of retrieved snippets; a structure designed to retrieve snippets evenly from a comprehensive set of four biomedical corpora, effectively mitigating retriever bias. Our experiments demonstrate that RAG ^2 improves the state-of-the-art LLMs of varying sizes, with improvements of up to 6.1%, and it outperforms the previous best medical RAG model by up to 5.6% across three medical question-answering benchmarks. Our code is available at https://github.com/dmis-lab/RAG2

## introduction large language models (llm) @xcite @xcite have demonstrated remarkable performance across various tasks in biomedicine, including usmle-style question-answering (qa) benchmarks @xcite . 1 despite their state-of-the-art performance, † corresponding authors. 1 the usmle or united states medical licensing examination is a standardized test that is required for obtaining a llms face challenges limiting their adoption in high-stakes areas. a key concern is hallucination, where the model produces information that sounds plausible but is incorrect @xcite . additionally, updating models' knowledge is resource-intensive, making it difficult to maintain current medical information for application @xcite . retrieval-augmented generation (rag) @xcite has emerged as a promising solution to address these limitations by reducing hallucinations and ensuring models provide up-to-date information through the integration of trustworthy documents into their input. however, applying rag in the biomedical domain presents unique challenges. first, irrelevant or incorrect context can mislead llms. @xcite . in biomedical texts, the complexity and specialized terminology heighten the risk of retrieving documents that are not relevant or unhelpful. second, medical queries often struggle to target useful information. some questions that include extensive patient details, such as medical histories and symptoms, may overwhelm retrieval systems, making it difficult to identify critical diagnostic clues. conversely, overly brief questions often lack necessary context and depend on implicit medical knowledge, further complicating effective retrieval. lastly, retriever bias @xcite also presents a significant challenge, as larger medical corpora often overshadow smaller, specialized ones that may contain critical, up-to-date information. for instance, medcpt @xcite may show a preference for pubmed documents over clinical guidelines and medical textbooks, a tendency likely stemming from its training on the pubmed corpus. building upon these insights, we propose rag 2 (rationale-guided retrieval augmented generation), a novel framework that improves the reliamedical license in the united states. figure 1 : our rag 2 framework comprises the following three steps. (1) rationale-based query formulation: the model-generated rationale is used as the query for evidence retrieval instead of the initial query. (2) balanced retrieval: evidence snippets are retrieved in equal amounts from four corpora: two large corpora, pubmed and pmc, and two smaller but crucial corpora, textbooks and clinical guidelines. subsequently, a reranker is used to rerank the retrieved snippets by cross-encoding the initial query and each snippet. (3) rationale-guided filtering: a filtering model determines whether incorporating the retrieved snippets into the llm prompt can help the model answer correctly and with higher confidence (as indicated by reduced perplexity). we evaluate our rag 2 framework using three closed-book medical qa benchmarks-medqa @xcite , medmcqa @xcite , and mmlu-med @xcite -where no oracle documents are present. our method significantly enhances the average accuracy of state-of-the-art llms of varying sizes, including llama-3-8b-instruct (ai@meta, 2024), meerkat-7b @xcite , and gpt-4o (openai, 2024). additionally, it consistently improves the performance of baseline llms by up to 6.1% across three medical qa datasets and outperforms four recently developed rag baselines, achieving a no our contributions are as follows: ## method we start by explaining the training objective of rag models. next, we provide a detailed breakdown of the pipeline, focusing on three key steps: ## conclusion in this study, we presented rag 2 , a novel framework designed to enhance the reliability and performance of llms in biomedical qa tasks. by integrating rationale queries, balanced retrieval, and a rationale-guided filtering method, we created a comprehensive approach that effectively refined and optimized the entire rag pipeline. our experimental results consistently demonstrated that rag 2 significantly improved the accuracy of various state-of-the-art llms across multiple medical qa benchmarks. this advancement marked an important step toward more reliable ai-assisted medical decision-making.