Bridging Relevance and Reasoning: Rationale Distillation in Retrieval-Augmented Generation (2025)

Abstract: The reranker and generator are two critical components in the Retrieval-Augmented Generation (i.e., RAG) pipeline, responsible for ranking relevant documents and generating responses. However, due to differences in pre-training data and objectives, there is an inevitable gap between the documents ranked as relevant by the reranker and those required by the generator to support answering the query. To address this gap, we propose RADIO, a novel and practical preference alignment framework with RAtionale DIstillatiOn. Specifically, We first propose a rationale extraction method that leverages the reasoning capabilities of large language models (LLMs) to extract the rationales necessary for answering the query. Subsequently, a rationale-based alignment process is designed to rerank the documents based on the extracted rationales, and fine-tune the reranker to align the preferences. We conduct extensive experiments on two tasks across three datasets to demonstrate the effectiveness of our approach compared to baseline methods. Our code is released online to ease reproduction.

## introduction large language models (llms), pretrained on massive datasets, have demonstrated exceptional reasoning and text generation capabilities, as evidenced by prior research @xcite @xcite @xcite . these models also adhere to the scaling laws, exhibiting improvements in performance and intelligence as the number of parameters increases @xcite . retrieval-augmented generation (rag) @xcite builds upon these capabilities by integrating information retrieval mechanisms (zhao et al., 2018a,b; liu et al., * equal contribution † corresponding author 1 https://github.com/applied-machine-learning-lab/ radio 2024) with generative models, such as llms. this approach not only mitigates the problem of hallucination in text generation but also enhances the system's adaptability to dynamically evolving information needs, making it a robust solution for tasks requiring both accuracy and contextual relevance @xcite . however, rag pipelines typically assemble components (e.g., the reranker and generator @xcite ) that have been pretrained separately. due to differences in their pretraining data and optimization objectives, these components often exhibit varying preferences, which can impact the overall effectiveness of the system. specifically, pretrained rerankers @xcite are designed to evaluate the relevance between queries and documents. however, the documents identified as relevant under this criterion may not provide the necessary support for reasoning to derive an accurate answer to the query. bridging this gap between the reranker's relevance measurement and the generator's reasoning requirements presents a significant challenge that must be addressed to improve the rag pipeline's performance. recent studies try to address this gap by training a bridge model @xcite , using llmbased scores @xcite or combining both llm-based and retrieval-based scores @xcite to fine-tune rag components. additionally, while some other methods are not explicitly designed for this problem, they can indirectly contribute to bridging the gap. these approaches can use response quality @xcite or perplexity distillation @xcite as signals to fine-tune the reranker. despite showing promise, these approaches face critical limitations: their alignment signals rely solely on the surfacelevel connection between the query/answer and document, failing to capture the deeper reasoning processes or more complex relationships involved. to address the above limitation, we propose ra-dio, a novel and practical preference alignment framework with rationale distillation in rag. ra-dio leverages rationale as a signal to bridge the reranker's relevance measurement with the generator's reasoning requirements for response generation. first, to efficiently extract the rationales needed to answer a query, we use the query and its ground truth answer as context and generate the rationales with llms. second, to mitigate the preference misalignment between the reranker and generator while ensuring the solution remains practical, we rerank the documents based on the extracted rationales and fine-tune the reranker. this step distillates rationales from generators to rerankers, and aligns the reranker with the generator's information needs for answering the query effectively. radio effectively addresses the preference inconsistency between rag components by first generating a comprehensive rationale and then finetuning the reranker based on the extracted rationale. this approach considers the deeper reasoning behind answers. we evaluate radio on three tasks across four datasets: open-domain qa (nq @xcite and trivi-aqa @xcite ), multi-choice questions (mmlu @xcite ), and multi-hop qa (musique @xcite ). the results validate the superiority of our method compared to other state-of-the-art baselines. our contributions can be summarized as follows: retrieval-augmented generation. large language models (llms) have demonstrated groundbreaking performance across numerous tasks @xcite but still face challenges such as hallucination and outdated knowledge @xcite . to address these issues, retrieval-augmented generation (rag) has been introduced @xcite @xcite . rag retrieves relevant information from external knowledge bases and incorporates it as contextual input to the generator (llm), enhancing the accuracy and reliability of the generated responses. the typical rag pipeline can be divided into several key components: query rewriter, retriever, reranker, and generator. the query rewriter @xcite modifies and expands @xcite the original query to improve retrieval recall, ensuring more relevant documents are retrieved. the retriever @xcite fetches relevant documents based on the query. dense retrievers generally outperform sparse retrievers in this step. to integrate contextual information more effectively and identify documents more relevant to the query, rerankers @xcite with larger models and greater complexity are introduced to reorder the retrieved documents compared to retrievers. finally, the generator-usually a powerful llm such as gpt-4 @xcite or llama @xcite -uses the query and the top-k documents from the reranker to generate the final response. in this work, we address the issue of preference misalignment among different components within the rag pipeline. we aim to leverage rationale as a signal to align these preferences and enhance the overall performance of the rag system. it is worth noting that ra-dio and methods like react @xcite are not mutually exclusive and can be used together to complement each other. while radio enhances the ranking of retrieved documents at each step, react focuses on task-solving through reasoning and action. preference alignment. to further improve llms, preference alignment is often performed after the initial pretraining phase @xcite . approaches such rlhf @xcite and dpo @xcite are proposed to align the output of llms more closely with human preferences. dpo transforms tasks into classification problems, achieving high computational efficiency and strong performance. in the context of rag, several works @xcite can be transformed to address the challenge of preference alignment between rag components. replug @xcite improves rag pipelines involving blackbox llms by using the probability of the llm generating the correct answer as a signal to determine document importance. similarly, rrr @xcite uses metrics based on the quality of the llm's generated response as a signal to evaluate a document's utility. on the other hand, arl2 @xcite prompts llms to generate self-guided relevance labels for fine-tuning retriever, and dpa-rag @xcite introduces a bidirectional alignment strategy to mitigate preference inconsistencies in rag pipelines. in this work, we focus on optimizing the reranker within rag. our goal is to enable the reranker to effectively identify supportive documents wellsuited for the generator, whether black-box or opensource, facilitating the production of accurate outputs. in addition, bgm @xcite trains a bridge model between retriever and llms to transform the retrieved information into the format llm's prefer. our method is theoretically compatible with bgm and could be combined to jointly enhance performance without any conflict. ## methodology in this section, we detail rationale distillation in rag. specifically, we first demonstrate the task definition of rag in section 3.1. then we give an overview of our proposed framework in section 3.2, introduce the rationale extraction method in section 3.3, and detail the rationale-based alignment in section 3.4 and optimization in section 3.5. ## conclusion in this paper, we propose a novel and practical preference alignment framework, radio, with rationale distillation in retrieval-augmented generation. first, we introduce a rationale extraction method to extract the rationales necessary for answering queries with llms. next, a rationale-based alignment is proposed to rerank documents based on extracted rationales and fine-tune rerankers. extensive experiments on three tasks across four datasets are conducted to validate the effectiveness of our proposed method against state-of-the-art baselines and demonstrate its strong transferability. we also release our code online to facilitate the following research and ease of reproduction.