PA - RAG : RAG Alignment via Multi-Perspective Preference Optimization (2025)

Abstract: The emergence of Retrieval-augmented generation (RAG) has alleviated the issues of outdated and hallucinatory content in the generation of large language models (LLMs), yet it still reveals numerous limitations. When a general-purpose LLM serves as the RAG generator, it often suffers from inadequate response informativeness, response robustness, and citation quality. Past approaches to tackle these limitations, either by incorporating additional steps beyond generating responses or optimizing the generator through supervised fine-tuning (SFT), still failed to align with the RAG requirement thoroughly. Consequently, optimizing the RAG generator from multiple preference perspectives while maintaining its end-to-end LLM form remains a challenge. To bridge this gap, we propose Multiple Perspective Preference Alignment for Retrieval-Augmented Generation (PA-RAG), a method for optimizing the generator of RAG systems to align with RAG requirements comprehensively. Specifically, we construct high-quality instruction fine-tuning data and multi-perspective preference data by sampling varied quality responses from the generator across different prompt documents quality scenarios. Subsequently, we optimize the generator using SFT and Direct Preference Optimization (DPO). Extensive experiments conducted on four question-answer datasets across three LLMs demonstrate that PA-RAG can significantly enhance the performance of RAG generators. Our code and datasets are available at https://github.com/wujwyi/PA-RAG.

## introduction pre-trained large language models (llms) demonstrate impressive question-answering capabilities but also reveal certain drawbacks, such as generating outdated or hallucinatory information @xcite . the emergence of retrieval-augmented generation (rag) @xcite has alleviated this issue by incorporating retrieved documents relevant to the question into the prompt, thereby providing the llms with information they may not know. however, while rag systems help improve factual grounding, the role of the generator in previous work is usually far from well-aligned. despite their impressive pre-training, off-the-shelf llms are not optimally suited for the specific requirements of rag tasks without proper task alignment. 1) helpfulness for rag application (response informativeness) @xcite : the generator should possess a refined ability to identify and utilize valuable information from the provided context. when presented with a mix of high-quality and less relevant information, it should actively and effectively leverage the valuable content while disregarding the rest. 2) harmlessness for rag application (response robustness) @xcite : a well-aligned generator should demonstrate robust noise resistance, capable of filtering out misleading information without being adversely affected. this ability ensures the generated responses remain accurate and coherent even when the input context is imperfect. 3) honesty for rag application (citation quality) @xcite : the generator should be capable of producing responses that are firmly rooted in the provided documents while maintaining appropriate attribution. this involves explicitly citing the source documents when necessary, and ensuring that the generated content is not only factually accurate but also traceable to the retrieved documents. to satisfy the aforementioned rag requirements, existing rag systems mainly fall into two categories: end-to-end architecture and pipeline architecture. these approaches attempt to bridge the gap between off-the-shelf llms and the specific demands of rag, but both face significant limitations. end-to-end architectures primarily rely on supervised fine-tuning (sft) of general llms @xcite . these approaches focus on constructing high-quality responses in rag scenarios. however, through further analysis, we found that the requirements for the generators in rag tasks are highly context-dependent and often interwoven. these varying and intertwined demands make it challenging for a model to meet all rag objectives through standard sft alone, as it does not incorporate the necessary preference information required for adapting to different retrieval scenarios. on the other hand, pipeline architectures introduce additional steps beyond generation, such as re-ranking retrieved documents, filtering irrelevant information, or employing post-hoc verification to ensure that citations support the claims @xcite @xcite . however, these additional steps can only satisfy certain rag requirements and still have a considerable gap in aligning with the global rag requirements. moreover, these additional steps introduce extra computational costs and time consumption, making it less practical for large-scale deployment. this further inspires us to think: is it possible to fully align the generator with the diverse rag requirements while retaining the simplicity and efficiency of an end-to-end architecture? to this end, in this paper, we propose multi-perspective preference alignment for retrieval-augmented generation (pa-rag), aiming to optimize the generator of rag systems to align comprehensively with specific rag requirements. as illustrated in figure 1 , pa-rag maintains the endto-end architecture of the generator while enabling it to learn multi-perspective preference information. the training of pa-rag is divided into two phases. the first phase is foundational capability training, where the generator acquires the basic ability to utilize and cite documents through instruction fine-tuning. to construct high-quality instruction fine-tuning data, we utilize chatgpt to generate complete and correct answers and employ a citation rewrite mechanism to ensure citation quality. the second phase is the multi-perspective preference optimization phase, in which the generator is trained using direct preference optimization (dpo) @xcite to learn preference information from different perspectives. this phase encompasses three sub-stages, sequentially enhancing the generator's response informativeness, response robustness, and citation quality. to construct high-quality preference data, we sample responses of varying quality from the generator across scenarios with different document qualities in prompt to build preference data for informativeness and robustness. additionally, we employ the citation rewrite mechanism to construct preference data for citation quality. we conducted extensive experiments on four qa datasets and three llms, demonstrating that pa-rag significantly enhances the generator's performance. the improvement achieved by pa-rag far surpasses the performance gains using only sft or additional steps. llms trained with pa-rag show an average absolute improvement of 13.97% in correctness, 49.77% in citation recall, and 39.58% in citation precision. our contributions can be summarized as follows: ## preliminaries in this section, we describe the optimization objectives of the rag generator, the motivation for multi-perspective optimization, and the rationale for choosing preference optimization over mere instruction fine-tuning. ## methodology after defining the optimization objectives, we trained the generator in two phases. the first phase enabled a general llm to acquire basic rag capabilities. the method for constructing the training data for this phase is detailed in § 3.1. the second phase involved multi-perspective preference optimization, further enhancing the generator's response informativeness, response robustness, and citation quality. the method for constructing the training data for this phase is described in § 3.2. ## conclusion in this work, we propose pa-rag, a method for optimizing the generator of rag systems to align with specific rag requirements comprehensively. the training process includes instruction fine-tuning and multi-perspective preference optimization. we conducted extensive experiments on four qa benchmarks and three llms, demonstrating that pa-rag can significantly enhance the generator's response informativeness, response robustness, and citation quality.