RAG - R eward B ench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment (2025)

Abstract: Despite the significant progress made by existing retrieval augmented language models (RALMs) in providing trustworthy responses and grounding in reliable sources, they often overlook effective alignment with human preferences. In the alignment process, reward models (RMs) act as a crucial proxy for human values to guide optimization. However, it remains unclear how to evaluate and select a reliable RM for preference alignment in RALMs. To this end, we propose RAG-RewardBench, the first benchmark for evaluating RMs in RAG settings. First, we design four crucial and challenging RAG-specific scenarios to assess RMs, including multi-hop reasoning, fine-grained citation, appropriate abstain, and conflict robustness. Then, we incorporate 18 RAG subsets, six retrievers, and 24 RALMs to increase the diversity of data sources. Finally, we adopt an LLM-as-a-judge approach to improve preference annotation efficiency and effectiveness, exhibiting a strong correlation with human annotations. Based on the RAG-RewardBench, we conduct a comprehensive evaluation of 45 RMs and uncover their limitations in RAG scenarios. Additionally, we also reveal that existing trained RALMs show almost no improvement in preference alignment, highlighting the need for a shift towards preference-aligned training.

## introduction retrieval augmented generation (rag) @xcite @xcite has emerged as a widely adopted approach for enabling large language models (llms) to access long-tailed and up-to-date knowledge by retrieving relevant information from external sources at inference. existing retrieval augmented language models (ralms) leverage rag to address the inherent knowledge limitations of llms, effectively reducing factual errors @xcite and providing better attributions @xcite . a direct approach @xcite to building ralms involves leveraging the in-context learning of llms to generate responses based on the retrieved documents. however, this plug-and-play method may cause the model to generate unfaithful responses or become distracted by noise. recent works @xcite @xcite have proposed constructing specialized rag datasets and applying supervised fine-tuning (sft) to further increase the usability of ralms. however, these sft-based methods may cause ralms to overly rely on and fit training data, lacking a feedback mechanism that enables the model to capture human preferences. as shown in figure 1 (a), the sft ralm may cite satirical content from the internet and generate harmful responses, or provide responses that lack sufficient information and fail to fully address the user's needs. to better integrate human preferences like help-ful and harmless @xcite into ralms, we argue that ralms should shift towards a new training paradigm, namely preference-aligned rag training. the alignment process, as illustrated in figure 1 (b), involves the reward model (rm) acting as a proxy for human values by providing feedback on the generated responses. based on the signals from the reward model, preference learning algorithms, such as ppo @xcite and dpo @xcite , optimize the policy model, ultimately resulting in the aligned ralm. reward models are central to this process. however, whether they can provide high-quality reward modeling for ralms remains underexplored. constructing a comprehensive benchmark for reward models in rag settings requires consideration of the following three key factors: (1) designing well-crafted rag scenarios: existing benchmarks for reward models primarily focus on evaluation in general scenarios. however, in rag scenarios, human preferences introduce new alignment requirements. for instance, privacy protection requires that ralms must not disclose any user privacy information from the private retrieval database @xcite . additionally, users often prefer generated responses that properly attribute information to the retrieved documents; (2) collecting diverse data sources: data collection should encompass a wide range of diverse sources, avoiding reliance on a single domain, retriever, or ralm, to prevent any biases in the evaluation of the reward model @xcite ; (3) providing high-quality preference judgments: compared to rewardbench @xcite with an average prompt length of 47, rag needs to incorporate a much larger number of retrieved documents in the prompt. this makes it challenging for human annotators to efficiently process the long context and provide reliable preference judgments. in this paper, we propose rag-rewardbench, a benchmark for systematically evaluating reward models in rag settings to facilitate the alignment of ralms. our rag-rewardbench is designed based on the three key factors mentioned above: based on the rag-rewardbench, we conduct a systematic evaluation of 45 reward models, including discriminative rms @xcite , generative rms @xcite and implicit rms @xcite . experimental results demonstrate that rag-rewardbench is highly challenging, with the top-ranked rm, skywork-critic-llama-3.1-70b @xcite , achieving only 78.3% accuracy. meanwhile, in the four rag-specific scenarios we designed, the rm's performance decreases to varying extents, underscoring the need for specialized rms tailored specifically for ralms. we obtain the following meaningful conclusions: (1) rms that perform well are generative or discriminative rms that have been specifically trained with 27b or 70b parameters, whereas those implicit rms tend to perform poorly on rag-rewardbench. (2) although state-of-theart trained ralms @xcite demonstrate significant improvements on certain rag datasets, their performance on rag-rewardbench shows only a minimal gain of 0.6% compared to the original llms. this suggests that the ralm training paradigm needs to shift towards preference-aligned rag training. (3) performance on rag-rewardbench shows a strong positive correlation with downstream rag task performance when using rm for best-of-n (bon) sampling. in summary, our key contributions are as follows: ## conclusion in this paper, we propose rag-rewardbench, the first benchmark for evaluating reward models in rag settings, including 1,485 high-quality preference pairs to facilitate the alignment of ralms. beyond helpfulness and harmlessness, we design four crucial and challenging rag-specific scenarios, including multi-hop reasoning, fine-grained citation, appropriate abstain, and conflict robustness. to increase the data source diversity, we adopt 18 datasets, six retrievers and 24 ralms. we conduct experiments with 45 rms, revealing the limitations of existing rms on rag-rewardbench. we find that current ralms show almost no improvement in preference alignment, highlighting the need for a shift towards preference-aligned training.