GRADE : Generating multi-hop QA and fine-g RA ined Difficulty matrix for RAG Evaluation (2025)

Abstract: Retrieval-Augmented Generation (RAG) systems are widely adopted in knowledge-intensive NLP tasks, but current evaluations often overlook the structural complexity and multi-step reasoning required in real-world scenarios. These benchmarks overlook key factors such as the interaction between retrieval difficulty and reasoning depth. To address this gap, we propose GRADE, a novel evaluation framework that models task difficulty along two orthogonal dimensions: (1) reasoning depth, defined by the number of inference steps (hops), and (2) semantic distance between the query and its supporting evidence. We construct a synthetic multi-hop QA dataset from factual news articles by extracting knowledge graphs and augmenting them through semantic clustering to recover missing links, allowing us to generate diverse and difficulty-controlled queries. Central to our framework is a 2D difficulty matrix that combines generator-side and retriever-side difficulty. Experiments across multiple domains and models show that error rates strongly correlate with our difficulty measures, validating their diagnostic utility. GRADE enables fine-grained analysis of RAG performance and provides a scalable foundation for evaluating and improving multi-hop reasoning in real-world applications.

## introduction retrieval-augmented generation (rag) has become a widely adopted strategy for enhancing large language models (llms) with external knowledge sources @xcite . by retrieving relevant documents and conditioning generation on this evidence, rag systems aim to reduce hallucinations and improve factual accuracy in tasks such as open-domain qa, dialogue, and summarization @xcite . as llms continue to improve, the quality of generation in rag * corresponding author. current evaluation often lacks the granularity needed to assess query difficulty. tasks are typically treated as uniformly difficult, without considering structural aspects such as the number of reasoning steps (multi-hop reasoning) or the distribution of evidence across multiple documents (multichunk retrieval). moreover, many benchmarks rely on single-hop, single-chunk queries, and define difficulty using retriever-centric indicators such as lexical ambiguity or document length @xcite . furthermore, collapsing retrieval and reasoning onto a single axis complicates the attribution of errors to the retriever, the generator, or their interaction @xcite . these coarse-grained approaches hinder nuanced analysis of rag systems, making it difficult to isolate module-specific failure points-such as retrieval errors versus generation issues @xcite . recent work has begun to recognize the importance of task difficulty in evaluating rag systems. some studies have proposed taxonomies that categorize queries based on retriever-side metrics such as retrieval recall, query ambiguity, or corpus complexity @xcite . these approaches acknowledge that retrieval difficulty varies across queries and have led to more nuanced analyses of retriever performance. however, such efforts remain largely focused on retrieval alone, without fully considering the downstream reasoning challenges faced by rag models. recent work @xcite has highlighted the importance of evaluating rag systems in a unified manner that accounts for both retrieval and reasoning. by comparing naïve and multi-step rag systems using their proposed dataset, the authors demonstrate how evaluation design can directly influence our understanding of a model's reasoning capabilities. in particular, most prior work overlooks the structural complexity in real-world rag scenarios. many queries in practice require multi-hop reasoning across multiple semantically diverse sources, which poses significant challenges for retrieval and evidence integration. in practice, answering a complex question often requires synthesizing information scattered across multiple, semantically distant documents @xcite . the cognitive and computational cost of such synthesis grows not just with the number of reasoning steps but also with the semantic dispersion of the supporting evidence. in particular, reasoning across documents from different topical clusters is typically more demanding than connecting closely related passages. for example, answering a multihop question like "what legal implications has the use of facial recognition technology had in european countries?" requires synthesizing technical documents on facial recognition systems with legal texts or policy reports from eu jurisdictions. assessing rag systems ability to handle realworld complexity requires benchmarks that accurately capture multi-hop reasoning-demanding both retrieval of diverse, complementary documents and their integration through coherent, stepwise inference. while recent benchmarks have begun to include multi-hop queires in rag systems, many define "hops" merely in terms of the number of retrieved evidence chunks, rather than the logical compositionality of the reasoning required @xcite . such datasets may model retrieval under multi-evidence conditions, but often fall short of ensuring that the answer genuinely depends on connecting semantically distinct pieces of information in a structured reasoning chain. instead, many tasks can be solved by stitching together co-located or loosely related facts, which risks overestimating a system's reasoning capability @xcite . moreover, because these datasets rarely enforce semantic heterogeneity across retrieved documents, they may underestimate the cognitive and computational demands of real-world multi-hop scenarios-where information must be integrated across topically diverse sources such as technical reports, legal docu-ments, and policy analyses @xcite . as a result, although multi-hop datasets have become more prevalent, current evaluations still fall short in diagnosing the reasoning limitations of rag systems under practical applications, highlighting the need for more fine-grained and structurally aware evaluation frameworks that jointly account for retrieval difficulty and reasoning complexity. in this paper, we introduce grade, a finegrained difficulty framework that simultaneously models two orthogonal sources of complexity: (i) multi-hop depth-the minimum number of reasoning steps required to connect the question to the answer through explicit evidence chains, and (ii) query-evidence semantic distance-the degree of semantic divergence between the query and its retrieved supporting chunks. in grade, task difficulty is defined over a two-dimensional space combining graph-based hop count and query-evidence semantic distance, where each coordinate point (i.e., the intersection of a specific hop count and semantic distance level) corresponds to a group of queries considered to share the same difficulty level. this formulation enables a fine-grained characterization of query complexity and supports (1) the synthetic generation of difficulty-calibrated test suites with controllable proportions across different difficulty levels, and (2) module-level ablations that analyze how retrieval and generation components contribute to performance at varying difficulty levels. this enables precise troubleshooting and systematic diagnosis of rag system weaknesses. our main contributions are as follows: ## conclusion we present grade, a novel evaluation framework for retrieval-augmented generation (rag) that jointly considers reasoning depth (generator-side) and semantic distance (retriever-side) to better reflect the structural complexity of real-world multihop question answering. by defining a difficulty matrix based on hop count and retrieval difficulty, our approach enables fine-grained analysis of rag systems, disentangling the contributions of retrieval and generation. experimental results show that our difficulty metrics strongly correlate with error rates across multiple rag models in realistic settings. in addition, our knowledge graph augmentation-through missing link detection via semantic clustering-highlights the limitations of traditional triplet-based pipelines in supporting deeper reasoning. notably, the grade difficulty matrix itself serves as a unified difficulty space, where diagonal regions represent scenarios jointly challenging for both retrieval and generation. this holistic perspective enables more nuanced diagnoses of rag system behavior across the full complexity spectrum. overall, grade offers a scalable, interpretable tool for analyzing rag system failures and guiding future advancements in multi-hop reasoning architectures. future work will extend grade to enable module-level evaluation across the broader rag pipeline, including components such as query decomposition and document filtering.