Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2025)

Abstract: Large Language Models (LLMs) have demonstrated significant performance improvements across various cognitive tasks. An emerging application is using LLMs to enhance retrieval-augmented generation (RAG) capabilities. These systems require LLMs to understand user queries, retrieve relevant information, and synthesize coherent and accurate responses. Given the increasing real-world deployment of such systems, comprehensive evaluation becomes crucial. To this end, we propose FRAMES (Factuality, Retrieval, And reasoning MEasurement Set), a high-quality evaluation dataset designed to test LLMs’ ability to provide factual responses, assess retrieval capabilities, and evaluate the reasoning required to generate final answers. While previous work has provided datasets and benchmarks to evaluate these abilities in isolation, FRAMES offers a unified framework that provides a clearer picture of LLM performance in end-to-end RAG scenarios. Our dataset comprises challenging multi-hop questions that require the integration of information from multiple sources. We present baseline results demonstrating that even state-of-the-art LLMs struggle with this task, achieving 0.40 accuracy with no retrieval. The accuracy is significantly improved with our proposed multi-step retrieval pipeline, achieving an accuracy of 0.66 (>50% improvement). We hope our work will help bridge evaluation gaps and assist in developing more robust and capable RAG systems.

## introduction recent advancements in large language models (llms) have significantly enhanced their capabilities across various natural language processing tasks, especially in systems that demand both factual accuracy and sophisticated reasoning for complex queries @xcite . retrieval-augmented generation (rag) techniques @xcite @xcite have become a powerful approach by leveraging the strengths of retrieval systems and the generative capabilities of llms. these techniques are particularly effective for tasks requiring multi-hop reasoning, factual grounding, and synthesizing information from diverse knowledge domains @xcite . however, despite this progress, the evaluation of rag systems remains fragmented and insufficient, as existing benchmarks typically assess components like retrieval, factual correctness, and reasoning in isolation @xcite . this piecemeal approach fails to capture the holistic performance of these systems in real-world applications @xcite . to bridge this gap, we introduce a novel evaluation framework, frames foot_0 (factuality, retrieval, and reasoning measurement set), designed to rigorously test llms on all three core capabilities-fact retrieval, reasoning across multiple constraints, and accurate synthesis of information into coherent responses. unlike existing datasets such as truthfulqa @xcite , hotpotqa @xcite , or gsm8k @xcite , which focus on isolated aspects of llm performance, frames provides an integrated evaluation that challenges models across these dimensions simultaneously. this approach offers a more accurate reflection of how these systems perform as end-to-end reasoning solutions, especially in scenarios requiring multi-document retrieval and complex reasoning. for example, our dataset includes questions like: "how many years earlier would punxsutawney phil have to be canonically alive to have made a groundhog day prediction in the same state as the us capitol?" this demands temporal and numerical reasoning based on information from multiple retrieved articles. figure 1 provides another such example. our work addresses a critical void in the current landscape by offering a challenging evaluation benchmark that not only tests the individual components of llms but also evaluates their performance in an end-to-end context. through our dataset, we simulate realistic, multi-document queries to assess the ability of llms to retrieve relevant facts, reason accurately, and synthesize information into coherent responses. additionally, we present empirical results on the performance of state-of-the-art models, highlighting both their strengths and the limitations in their reasoning capabilities. these findings pave the way for further research and development of more robust and efficient retrieval-augmented generation systems. our key contributions are as follows: ## frames frames (factuality, retrieval, and reasoning measurement set) is an evaluation set of 824 questions designed to provide an end-to-end evaluation of retrieval augmented generation (rag) systems. it assesses three key components of a rag system: factuality, retrieval, and reasoning. unlike most existing datasets and benchmarks that evaluate each of these rag components in isolation, frames offers a comprehensive test bed to gain a clear understanding of the overall quality of rag systems @xcite @xcite . this holistic approach allows for a more accurate reflection of how these systems perform in real-world scenarios. in this section, we first detail our data collection process, which involved both synthetic data generation attempts and human annotation. next, we present the dataset statistics, showcasing the diversity of topics and reasoning types covered. finally, we outline the rigorous quality checks implemented to ensure the dataset's reliability and challenging nature. by providing this end-to-end evaluation truthfulqa @xcite openbookqa @xcite hotpotqa @xcite hybridqa @xcite gsm8k @xcite multihop-rag @xcite morehopqa @xcite musique @xcite naturalquestions @xcite triviaqa @xcite eli5 @xcite ## empirical analysis after obtaining a high-quality test set, we evaluate state-of-the-art llms on their ability to answer questions that require proficiency in factuality, retrieval, and reasoning. our analysis is divided into two sections: (1) single-step evaluations (section 3.1): here, we evaluate the llms based on a single-shot inference, where the idea is to ask the question and assess the response after a single inference call. this evaluation is further divided into cases with and without retrieval to analyze the impact of retrieval on performance. ## related works evaluating retrieval-augmented generation (rag) systems is crucial as they combine retrieval and generative capabilities @xcite . existing benchmarks like naturalquestions @xcite , triviaqa @xcite , and eli5 @xcite focus on specific aspects but lack comprehensive assessment. naturalquestions tests retrieval precision, triviaqa emphasizes factual correctness, and eli5 targets explainability without rigorous multi-hop reasoning evaluation. frames addresses these limitations by offering a unified evaluation framework. it tests rag systems across factual retrieval, reasoning, and synthesis dimensions. unlike existing datasets, frames incorporates complex multi-hop queries requiring information retrieval and integration from various sources, while handling temporal disambiguation. it also assesses information synthesis into coherent responses, evaluating rag systems in realistic, multifaceted scenarios. this makes frames a more rigorous and comprehensive benchmark for guiding next-generation rag system development. additional dataset comparisons are provided in ## conclusion in this work, we introduced frames, a comprehensive evaluation dataset designed to test the capabilities of retrieval-augmented generation (rag) systems across factuality, retrieval accuracy, and reasoning. our experiments with state-of-the-art llms highlight the existing gaps in their ability to handle complex, multi-hop reasoning tasks. the baseline results showed that even advanced models struggle significantly with the challenging scenarios presented in frames, achieving only moderate improvements when multi-step retrieval and reasoning strategies were employed. the frames dataset addresses a critical need in the evaluation of rag systems by offering an integrated framework that tests these systems in a more holistic manner compared to existing benchmarks. by simulating realistic, multi-document queries, frames provides a clearer picture of the current capabilities and limitations of llms in real-world applications. our findings underscore the importance of further enhancing both the retrieval mechanisms and the reasoning capabilities of these models to improve their overall performance.