Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage (2025)

Abstract: Evaluating retrieval-augmented generation (RAG) systems remains challenging, particularly for open-ended questions that lack definitive answers and require coverage of multiple sub-topics. In this paper, we introduce a novel evaluation framework based on sub-question coverage, which measures how well a RAG system addresses different facets of a question. We propose decomposing questions into sub-questions and classifying them into three types—core, background, and follow-up—to reflect their roles and importance. Using this categorization, we introduce a fine-grained evaluation protocol that provides insights into the retrieval and generation characteristics of RAG systems, including three commercial generative answer engines: You.com, Perplexity AI, and Bing Chat. Interestingly, we find that while all answer engines cover core sub-questions more often than background or follow-up ones, they still miss around 50% of core sub-questions, revealing clear opportunities for improvement. Further, sub-question coverage metrics prove effective for ranking responses, achieving 82% accuracy compared to human preference annotations. Lastly, we also demonstrate that leveraging core sub-questions enhances both retrieval and answer generation in a RAG system, resulting in a 74% win rate over the baseline that lacks sub-questions.

## introduction retrieval-augmented generation (rag) has emerged as a powerful solution for answering open-ended complex questions by combining the strengths of retrieval systems with the generative capabilities of large language models (llms). by retrieving relevant documents or knowledge chunks from external sources and using them as additional context when generating responses, rag-based models can provide more accurate and contextually grounded answers compared to models that rely solely on generative approaches. however, the evaluation of rag systems remains a challenging task, particularly in the context of open-ended non-factoid questions where the strict correctness of a long-form answer is difficult to define @xcite @xcite . one of the key issues in evaluating rag systems is the lack of a systematic approach to assess how well these models address the full scope of complex questions. most existing rag evaluations @xcite @xcite focus on surface-level metrics like faithfulness or relevance, without considering whether the generated response adequately covers the multi-dimensional nature of the question. complex questions are rarely monolithic in real-world applications. they often require answers that address multiple facets of the issue at hand, which can be thought of as sub-questions that together form a comprehensive answer. for instance, to fully answer the question "how climate change is affecting the earth?", we need to address multi-faceted information such as "what are the impacts of rising temperatures on polar ice caps and glaciers?" and "what effects do the oceanic changes have on marine life?". however, existing approaches @xcite leverage sub-questions as additional signals without fully considering how their relevance to the original query impacts response quality and user preferences. this paper proposes a novel evaluation framework that introduces the concept of sub-question coverage as a key metric for assessing rag performance (depicted in figure 1 ). we hypothesize that complex, open-ended questions can be decomposed into several interrelated sub-questions, and they can be further classified into core subquestions that are central to addressing the main query, background sub-questions that provide necgiven a question, rag's answer, and retrieved chunks, we decompose the question into sub-questions and classify them into three types: core, background, and follow-up; we measure the sub-question coverage rates of both the answer and retrieved chunks with categorized sub-questions and design a fine-grained evaluation protocol to assess three popular rag-based answer engines (section 3). we find the sub-question coverage as an answer quality metric can approximate human perception of answer quality well (section 4). we incorporate core sub-questions into different stages of the rag workflow and effectively improve its responses (section 5). there are currently no established datasets or frameworks for defining and classifying relevant sub-questions. although llms have been used for question decomposition, directly prompting them to generate sub-questions of each type results in low diversity and limited agreement with human classifications. to address these challenges, we propose a two-step approach: 1. we decompose complex, open-ended, non-factoid questions into a comprehensive set of sub-questions; 2. we classify these sub-questions into core, background, and follow-up categories based on their functional roles within the main question. this two-step prompting technique enhances the diversity of the generated sub-questions and achieves substantial agreement among human annotators and between human annotators and llms on type classification. next, we propose a fine-grained evaluation protocol based on sub-question coverage and analyze popular rag-based answer engines-you.com foot_0 , perplexity ai foot_1 , and bing chat foot_2 . we evaluate two key aspects: (1) the patterns of retrieval and answer generation coverages across different sub-question types, and (2) the areas where search engines fall short, identifying opportunities for improvement. we find that all three answer engines tend to prioritize core sub-questions over background or followup ones, which is the ideal behavior for generating high-quality answers. however, our analysis also reveals key shortcomings. none of the engines consistently retrieved knowledge chunks that fully addressed all core sub-questions. additionally, even when relevant core information was retrieved, the engines often failed to effectively utilize it in their generated answers. this reveals their key limitations in both retrieval completeness and the use of retrieved content to produce comprehensive responses. in addition to conducting a fine-grained analysis of answer comprehensiveness, we propose using sub-question classification to assess overall answer quality. leveraging human preference data from the webgpt comparisons dataset @xcite , we find that addressing core sub-questions correlates most positively with human preferences, while background sub-questions show a moderate correlation. in contrast, follow-up sub-questions negatively impact perceived answer quality. based on these findings, we introduce a weighted metric for evaluating answers that outperforms the conventional llm-as-a-judge approach @xcite and strongly correlates with human preferences for open-ended answers. both our studies reveal how sub-question types can enhance the analysis of retrieval coverage and answer comprehensiveness. a natural extension of this framework is improving rag systems by aligning retrieval and answer generation with core sub-questions. we evaluate four strategies for incorporating sub-question type information into different stages of rag systems, demonstrating that the most effective approach uses core sub-questions during chunk retrieval and reranking. our results show that prioritizing core information significantly improves response quality, resulting in more accurate and comprehensive answers. ## sub-question coverage: a new perspective for rag evaluation in this work, we focus on the rag evaluation on long-form answers to open-ended and non-factoid questions @xcite , instead of questions with short and factoid answers. such questions are typically more complex, requiring deeper exploration, analysis, and synthesis of information. they often lack a single correct answer, inviting subjective interpretations or multiple valid perspectives. the integration of diverse sources are essential for forming a well-rounded response. for instance, to address the question "are fresh or frozen vegetables healthier?" sufficiently, we might want to explore several sub-questions such as #1 "how does the freezing process affect the nutritional content of vegetables?", #2 "what are the common methods used to freeze vegetables?", and #3 "what are the cost and taste differences between fresh and frozen vegetables?". essentially, the multi-faceted information necessary for answering the given question is equivalent to the overall information that can be covered by multiple subquestions. nowadays, practical rag systems @xcite usually involve this general query decomposition step to improve their response quality and coverage. however, while gathering more information from these various sub-questions can be beneficial, we argue that not all should be treated equally, as their relevance and importance to the original question may vary. for example, sub-question #1 is the most crucial, #2 provides helpful context, and #3 encourages thinking one step ahead. motivated by this observation, we propose a new approach for automatic rag evaluation: subquestion coverage. we first decompose an openended non-factoid question into sub-questions, classifying them as core, background, or follow-up ( §3.1), based on their relevance and functional role in answering the main question. using this taxonomy, we develop a fine-grained evaluation protocol with metrics to assess the multi-dimensional effectiveness of rag systems ( §3.2). since retrieval is key in rag, our protocol examines the coverage of sub-questions in both the long-form answers and retrieved chunks. this allows us to analyze how retrieval affects sub-question coverage in the final answer. we then apply this evaluation to three popular rag systems: you.com, perplexity ai, and bing chat ( §3.3). in our experiments, we manually select 200 open-ended non-factoid questions with a high level of complexity from the researchy questions dataset @xcite and perform a question decomposition on them to get all corresponding sub-questions and their types. our criteria for question selection is that the question must be multifaceted and must require multi-hop reasoning to answer. in total, we have 200 questions and around 4000 sub-questions. ## an automatic answer quality metric the sub-question coverage enables a systematic evaluation of a rag system from a developer's standpoint, which spans both retrieval and generation. in contrast, end-users would perceive the effectiveness of a rag system directly from the quality of answers the rag system generates. they may evaluate the answer quality based on a variety of criteria such as completeness and relevance. while existing work @xcite tries to automate this evaluation by approximating human preferences with the llm as a judge, we believe that directly comparing two long answers poses a complex challenge for llms. therefore, by systematically identifying the types of sub-questions addressed in a given answer, we propose to stream- ## conclusions we introduced a novel evaluation framework for retrieval-augmented generation (rag) systems based on sub-question coverage. by decomposing complex, open-ended queries into core, background, and follow-up sub-questions, we provided a more refined approach to evaluating raggenerated responses. our results also show that incorporating core sub-questions into rag systems significantly improves the generation of more comprehensive and accurate answers. these findings open up new possibilities for evaluating and optimizing rag systems, particularly for complex, knowledge-intensive tasks. additionally, our sub-question-based answer quality metric can serve as a potential reward signal for training end-to-end rag models, offering a valuable direction for future research and system development.