RAR ^2 : Retrieval-Augmented Medical Reasoning via Thought-Driven Retrieval (2025)

Abstract: Large Language Models (LLMs) have shown promising performance on diverse medical benchmarks, highlighting their potential in supporting real-world clinical tasks. Retrieval-Augmented Generation (RAG) has emerged as a key approach for mitigating knowledge gaps and hallucinations by incorporating external medical information. However, RAG still struggles with complex medical questions that require intensive reasoning, as surface-level input often fails to reflect the true knowledge needs of the task. Existing methods typically focus on refining queries without explicitly modeling the reasoning process, limiting their ability to retrieve and integrate clinically relevant knowledge. In this work, we propose RAR ^2 , a joint learning framework that improves both Reasoning-Augmented Retrieval and Retrieval-Augmented Reasoning. RAR ^2 constructs a thought process to uncover implicit knowledge requirements and uses it to guide retrieval and answer generation. We build a training dataset of mixed preference pairs and apply Direct Preference Optimization (DPO) to train the model. Moreover, we design two test-time scaling strategies to explore the boundaries of our framework. Experiments demonstrate the effectiveness of RAR ^2 across several biomedical question answering datasets, outperforming RAG baselines with or without fine-tuning.

## introduction the capabilities of large language models (llms) in medicine have long attracted substantial research attention @xcite . llms such as gpt-4o and baichuan-m1 have demonstrated strong performance across diverse medical benchmarks, highlighting their potential to support real-world clinical tasks @xcite @xcite . more recently, given the vast scope and high-stakes nature of the medical domain, retrieval-augmented generation (rag) has emerged as a key approach for mitigating knowledge gaps and hallucinations by leveraging external medical knowledge @xcite @xcite . while rag performs well in a range of medical tasks, it still struggles with complex medical questions requiring intensive reasoning @xcite . this difficulty arises because the surface-level information in these questions does not reflect their actual knowledge requirements. for example, in figure 1 , although the question clearly describes the patient's symptoms, such as "weak in upper body and arms" and "won't move legs", the knowledge necessary to generate the correct answer lies in clinical concepts like "autonomic dysfunction" and "infant botulism". these concepts must be inferred through detailed analytical reasoning. existing rag methods for complex medical reasoning primarily focus on direct query refinement, but fail to construct and optimize a reasoning process that comprehensively uncovers the underlying knowledge requirements @xcite @xcite . additionally, few studies have explored enhancing the medical reasoning capabilities of llms within the rag framework @xcite . reasoning with external knowledge is more challenging, as the retrieved content inevitably includes noise information. llms should learn to integrate relevant knowledge into the reasoning process while avoiding interference from irrelevant information. therefore, optimizing retrieval-augmented reasoning is essential for improving both the accuracy and robustness of generation outcomes. in this work, we construct a thought process to reason through medical questions and identify their implicit knowledge requirements. this thought process is directly used to retrieve relevant information, which is subsequently incorporated into reasoning to derive the final answer. to optimize both thought and answer generation, we propose a joint learning framework, rar foot_1 , that simultaneously improves reasoning-augmented retrieval and retrieval-augmented reasoning. specifically, we first construct a training dataset consisting of mixed preference pairs. one type is thought pairs, in which a sampled thought process is annotated based on the outcome of subsequent retrieval-augmented generation. the other type is answer pairs, in which a sampled answer is annotated according to its correctness. we then apply direct preference optimization (dpo) to fine-tune the llm @xcite . these preference pairs enable supervised preference learning, allowing the model to identify relevant knowledge and reason effectively with external information. extensive experiments across several biomedical question answering datasets demonstrate rar 2 's superiority over existing rag baselines. our further test-time scal-ing analysis validates the scalability of rar 2 . in summary, our contributions are as follows: ## reasoning before retrieval we begin with a preliminary study on the impact of reasoning prior to retrieval. rather than finetuning a model to generate or refine a query, we instead sample a thought process that could guide the model toward a final answer and use it directly as a query to retrieve relevant documents. the thought process is sampled in a zero-shot manner by prompting the llm with: "please reason step by step to identify the relevant knowledge that may be involved." for retrieval, we use the bm25 algorithm @xcite to retrieve the top-k documents from the corpus. these documents are then provided as context to the llm for answer generation. we evaluate this approach-referred to as thought-rag-against standard question-based retrieval (question-rag) using the qwen2.5-7b-instruct model @xcite with @xmath0. figure 2 shows the performance comparison on several medical question answering datasets. we can observe that thought-rag consistently outperforms question-rag, with an average improvement of 4.82% across all datasets. especially in challenging datasets like medbullets and gpqa, the gains are more significant. furthermore, for most datasets, the accuracy remains unchanged or even declines using question-rag. a similar phenomenon can be found in the mirage benchmark @xcite . these observations suggest that reasoning-intensive medical questions often do not explicitly state the specific information needs required for successful retrieval. in contrast, a thought process generated through prior reasoning more accurately captures the underlying knowledge requirements, thereby improving overall rag performance. reasoning first and then using the resulting thought process for retrieval is a straightforward yet powerful approach to collect relevant knowledge, but it has received limited attention within the rag community @xcite . therefore, further optimizing the generation of the thought process holds promise for enhancing retrieval quality and improving the rag performance. ## method in this section, we introduce the rar 2 framework, which constructs a mixed preference dataset consisting of thought and answer pairs, and applies dpo to jointly enhance both reasoning-augmented retrieval and retrieval-augmented reasoning for llms. as shown in figure 3 , we first sample a set of thought processes and answers. the thought processes are generated based on the medical question, while the answers are generated using the question along with the corresponding retrieved documents. all samples are annotated based on the correctness of the answer ( §4.1). we collect samples to form a mixed preference dataset and apply dpo to jointly optimize two types of reasoning processes ( §4.2). besides, we design two test-time scaling strategies to examine the boundaries of rar 2 ( §4.3). ## conclusion in this work, we propose rar 2 , a novel joint learning framework designed to enhance llm reasoning capabilities within the rag framework, specifically targeting complex medical questions. unlike existing rag approaches, which rely primarily on direct query refinement, rar 2 explicitly constructs and optimizes a thought-based reasoning process to uncover implicit knowledge requirements. we introduce a mixed preference dataset comprising thought pairs and answer pairs and leverage dpo for joint training. extensive experiments on multiple biomedical question answering datasets demonstrate that rar 2 outperforms state-of-the-art rag baselines, both with and without fine-tuning. moreover, our analysis of test-time scaling strategies validates the scalability and robustness of rar 2 , highlighting its potential to significantly improve medical reasoning and decision-making in clinical scenarios.