G a RAG e: A Benchmark with Grounding Annotations for RAG Evaluation (2025)

Abstract: We present GaRAGe, a large RAG benchmark with human-curated long-form answers and annotations of each grounding passage, allowing a fine-grained evaluation of whether LLMs can identify relevant grounding when generating RAG answers. Our benchmark contains 2366 questions of diverse complexity, dynamism, and topics, and includes over 35K annotated passages retrieved from both private document sets and the Web, to reflect real-world RAG use cases. This makes it an ideal test bed to evaluate an LLM’s ability to identify only the relevant information necessary to compose a response, or provide a deflective response when there is insufficient information. Evaluations of multiple state-of-the-art LLMs on GaRAGe show that the models tend to over-summarise rather than (a) ground their answers strictly on the annotated relevant passages (reaching at most a Relevance-Aware Factuality Score of 60%), or (b) deflect when no relevant grounding is available (reaching at most 31% true positive rate in deflections). The F _{1} in attribution to relevant sources is at most 58.9%, and we show that performance is particularly reduced when answering time-sensitive questions and when having to draw knowledge from sparser private grounding sources.

## introduction large language models (llms) have shown consistent improvements across many tasks requiring natural language understanding, coding, mathematical or logical reasoning and have been widely incorporated in user applications, especially in the form of retrieval augmented generation (rag) systems @xcite . rag is crucial in realworld applications, where users (a) have their own private documents to search through, and (b) require responses to be grounded on relevant, timely 1 the dataset can be found at https://github.com/ amazon-science/garage grounding relevant & up-to-date? what strategic value does the introduction of unrivaled, a new 3x3 women's basketball league, provide to wnba players and the broader landscape of women's sports? the introduction of unrivaled, a new 3x3 women's basketball league featuring 36 wnba players, provides strategic value by capitalizing on the rapidly growing interest and record-setting engagement in women's sports, particularly the momentum seen in 2024. @xcite the league positions itself as a response to the thriving women's sports industry, where increasing fan engagement is driving significant sponsorship, athlete earnings, and investor interest. web 5 2 8 kb ready for a good answer! question fast-changing? and attributable sources. for rag applications to be successful at satisfying these needs, llms must simultaneously exhibit strong performance at two core competencies: query generation and answer generation. for query generation, llms must be able to determine when to invoke the available external sources and generate the right query, or queries, that will likely return good search results (grounding information), conditioned on the available @xcite context. in answer generation, llms must be able to provide a fluent, helpful and possibly long-form response to the user question that is grounded on the available information. crucially, this task is not fulfilled with a simple summarisation step, because in reality the available grounding, obtained from either the user's private documents or the web, will contain a mixture of relevant and irrelevant passages @xcite . therefore, the llm needs to analyse the relevancy of each grounding snippet prior to generating the answer, which will typically include citation markers to the selected source documents for transparency and accountability. an example that reflects this process is presented in figure 1 . multihop rag @xcite the process of generating an accurate final answer remains crucial, regardless of the specific approach to query generation. whether utilizing single or multiple queries, implementing one or several passes, incorporating reasoning chains, or drawing from various information sources, the fundamental challenge persists: the llm must carefully distill relevant information from noise passages to provide precise, factual responses that address user needs while avoiding hallucinations. although there exist a large number of rag benchmarks in the literature, reliably evaluating each of these competencies remains a challenge, because query generation affects the available grounding and this process lacks human annotations. previous works resort to either: evaluating the final response directly @xcite , which conflates query generation and answer generation; or using fixed grounding from one or more documents whose relevancy is either guaranteed @xcite , unknown @xcite or synthetically mixed @xcite , missing the nuances of real-life retrieval from user databases and the web. in addition to this, they come with at least one of the following disadvantages, (1) unsupervised llm-based dataset creation @xcite @xcite raising concerns regarding question naturalness and answer reliability, (2) short or multi-choice examtype answers @xcite @xcite for simpler evaluation at the cost of departing from real user expectations, (3) automatic evaluations of faithfulness to the unannotated grounding which disregard whether the grounding is correct (e.g. @xcite , and (4) do not consider mixed grounding obtained from both user-specific document sets and the web. to overcome these limitations, we introduce garage, a benchmark with grounding annotations for rag evaluation, comprising 2.4k questions with over 35k manually-annotated grounding passages retrieved from both private documents and the web. fully annotated and validated by professional annotators, garage enables a fine-grained evaluation of llms answer generation capabilities, for different question types (different dynamism, complexity, etc.) and with controlled degrees of grounding noise. apart from grounding relevancy annotations for each snippet, the benchmark categorises questions along multiple dimensions: timesensitivity, topic popularity, complexity or domain, and provides manually-crafted long-form, comprehensive responses that contain attribution to the relevant grounding snippets and reflect the expectations of rag users nowadays. thanks to the grounding annotations, garage allows defining the relevance-aware factuality score (raf), which measures whether llm responses are grounded on strictly relevant passages, as opposed to summarising grounding of unknown value -extending factuality metrics from @xcite . furthermore, a subset of the dataset addresses the deflection capabilities of llms, where the provided grounding is annotated as insufficient to lead to a good answer, and the llm is expected to avoid hallucination. additionally, we report attribution metrics, leveraging our fine-grained annotations. using our benchmark and proposed metrics, we evaluate a diverse selection of state-of-the-art proprietary and open-weights llms and find that all models struggle at correctly detecting relevant grounding and overly summarise, reaching only 60% in raf, and in deflecting when the grounding is irrelevant, reaching at most 31% true positive rate. these problems become more evident when retrieving documents from sparser private knowledge bases (kb), or when having to process time-sensitive questions which require both relevancy and temporal reasoning. we release the dataset and the prompts used for evaluation, in order to encourage the community to evaluate and improve on the dimensions featured in the benchmark. topics complex question question filtering question annotation grounding annotation answer writing answer validation question with category annotation annotated grounding comprehensive, grounded answer with attribution markers question construction grounding collection human annotation kb web grounding query generation snippet retrieval question generation question filtering question decomposition passage retrieval passage reranking ## dataset construction our methodology for creating garage focuses on generating questions that challenge models with high complexity and reflect real-world rag scenarios. the dataset features diverse grounding sources and comprehensive human-curated long form answers. figure 2 illustrates the process, which comprises the following main elements: dynamic and complex questions. a diverse set of questions with varied level of temporal dynamism, complexity, and ranging both trending and tail topics. for temporal dynamism we construct fast-changing, slow-changing, and static type of questions. these different categories require distinct interactions with the grounding information and a variable leniency towards parametric knowledge. additionally, a question becomes challenging for rag when a simple extract from a retrieved document is not sufficient to answer it @xcite . this complexity can arise from various factors, such as when the answer demands multiple logical steps, or requires combining information from different retrieved snippets. following literature @xcite , we generate questions across dimensions such as comparisons, multi-hop and post-processing (see §2.1). ## evaluation framework thanks to the fine-grained annotations of garage at the question, grounding, and answer level, we have the opportunity to evaluate several dimensions of model performance. to this end, we follow jacovi et al. (2024) and report response eligibility and 10 there are on average 5 unique cited sources per answer. factuality. we build on top of the factuality metric, by defining metrics reflecting factuality to relevant passages. then, we evaluate deflective behaviour as opposed to hallucination when no relevant grounding is provided, and attribution ability to each of the provided relevant passages when generating the final response. we compute the metrics using gpt-4o foot_8 as a judge, unless otherwise specified. eligibility score measures whether the model sufficiently addresses the user request with respect to the human-written answer as a baseline response. the llm judge produces the labels: no issues, minor issue(s) or major issue(s) and the eligibility score is calculated as the percentage of datapoints having no major issue(s). the prompt used for this metric is reported in appendix b.2. unadjusted factuality score measures the percentage of answers that are completely supported by the provided grounding, by first splitting the model response into sentences and then judging whether each of those is supported, unsupported, contradictory, or not requiring factual attribution with respect to the provided grounding. finally, a model answer is factual only if every sentence is supported or does not require attribution. the prompt used for this metric is reported in appendix b.3. as garage provides relevancy annotations for each grounding piece, we sharpen this metric to calculate the model's ability to ground its answers only in the relevant passages provided as context, hereinafter unadjusted relevance-aware factuality score @xcite . notice that this is a more controlled metric that not only measures the factuality of the response, but also the relevancy of the information used to produce the final response. factuality score combines the first two metrics, and measures the percentage of answers that are both eligible and supported by the provided grounding. we similarly develop the relevance-aware factuality score (raf) by combining uraf with eligibility to calculate the percentage of answers that are both eligible and supported by the relevant passages. deflection score measures the percentage of deflective answers for the subset of the benchmark expecting a deflection, i.e., true positive, due to insufficient grounding information. additionally, we report the portion of false positive deflective answers on the rest of the dataset which has sufficient relevant information for providing a non-deflective response. the prompt used for this metric is reported in appendix b.4. attribution score measures the ability of the model to produce correct attribution to grounding passages from where the information was taken in the form of citations. for this metric, we leverage the citations in the human-written responses and calculate precision, recall and f 1 -score of the occurrence of citations in model response compared to those in the human-written response. ## results we evaluate strong proprietary models and leading open weights models of different sizes on the garage benchmark. in particular, we evaluate gpt-4o 12 and 32b models @xcite . we use greedy decoding and the same input prompt across the models as reported in appendix b.1. ## related work there have been an impressive number of benchmarks released in the last years in the domain of question answering, due to both the impact of the task and the complex and diverse reasoning required to have in order to solve it, spawning from extractive benchmark probing for reading comprehension like: natural questions @xcite , wikiqa @xcite or triv-iaqa @xcite to frameworks that are easily customizable to new data and models such as: ragas @xcite , ragnarok @xcite or ares @xcite . ## conclusion we introduce garage, a large rag benchmark with human-curated long-form answers and comprehensive annotations of individual grounding passages. due to the detailed annotation of the dataset, we were able to develop a thorough set of experiments and a new metric: reference-aware factuality (raf) score. this framework helped unveil multiple gaps in open and closed llm's abilities to deal with noisy grounding. furthermore, we conduct in-depth analysis on model performance across different dataset dimensions, increasing our understanding of model limitations.