R3- RAG : Learning Step-by-Step Reasoning and Retrieval for LLM s via Reinforcement Learning (2025)

Abstract: Retrieval-Augmented Generation (RAG) integrates external knowledge with Large Language Models (LLMs) to enhance factual correctness and mitigate hallucination. However, dense retrievers often become the bottleneck of RAG systems due to their limited parameters compared to LLMs and their inability to perform step-by-step reasoning. While prompt-based iterative RAG attempts to address these limitations, it is constrained by human-designed workflows.To address these limitations, we propose \textbf{R3-RAG} , which uses \textbf{R} einforcement learning to make the LLM learn how to \textbf{R} eason and \textbf{R} etrieve step by step, thus retrieving comprehensive external knowledge and leading to correct answers. R3-RAG is divided into two stages. We first use cold start to make the model learn the manner of iteratively interleaving reasoning and retrieval. Then we use reinforcement learning to further harness its ability to better explore the external retrieval environment.Specifically, we propose two rewards for R3-RAG: 1) answer correctness for outcome reward, which judges whether the trajectory leads to a correct answer; 2) relevance-based document verification for process reward, encouraging the model to retrieve documents that are relevant to the user question, through which we can let the model learn how to iteratively reason and retrieve relevant documents to get the correct answer.Experimental results show that R3-RAG significantly outperforms baselines and can transfer well to different retrievers.

## introduction retrieval-augmented generation (rag) has become a prevailing mechanism to address large language models' (llms) hallucinations, as shown in figure 1a . it typically first retrieves relevant documents and then integrates them to supplement the * equal contribution. ## conclusion we propose r3-rag, a framework that leverages reinforcement learning to teach llms how to reason and retrieve step-by-step, optimizing the interleaved reasoning and retrieval process with both outcome rewards and fine-grained process rewards through relevance-based document verification. experimental results demonstrate that r3-rag significantly outperforms existing iterative rag methods across multiple benchmarks. further analyses show the effectiveness of r3-rag's reward design and its transferability across different retrievers. step 2: the director of "thendral veesum" is not mentioned in the provided documents. step 3: since "thendral veesum" is not mentioned in the provided documents, we cannot determine its director's birth year. the final answer: we cannot answer.