Controlled Retrieval-augmented Context Evaluation for Long-form RAG (2025)

Abstract: Retrieval-augmented generation (RAG) enhances large language models by incorporating context retrieved from external knowledge sources. While the effectiveness of the retrieval module is typically evaluated with relevance-based ranking metrics, such metrics may be insufficient to reflect the retrieval’s impact on the final RAG result, especially in long-form generation scenarios. We argue that providing a comprehensive retrieval-augmented context is important for long-form RAG tasks like report generation and propose metrics for assessing the context independent of generation. We introduce CRUX, a C ontrolled R etrieval-a U gmented conte X t evaluation framework designed to directly assess retrieval-augmented contexts. This framework uses human-written summaries to control the information scope of knowledge, enabling us to measure how well the context covers information essential for long-form generation. CRUX uses question-based evaluation to assess RAG’s retrieval in a fine-grained manner. Empirical results show that CRUX offers more reflective and diagnostic evaluation. Our findings also reveal substantial room for improvement in current retrieval methods, pointing to promising directions for advancing RAG’s retrieval. Our data and code are publicly available to support and advance future research on retrieval for RAG. Github: https://github.com/DylanJoo/crux

## introduction with their emerging instruction-following capabilities @xcite , large language models (llms) have adopted retrievalaugmented generation (rag) @xcite to tackle more challenging tasks, such as ambiguous question answering (qa) @xcite and long-form response generation @xcite . the role of retrieval in rag is to access information from external sources and prompt it as plug-in knowledge 1 https://github.com/dylanjoo/crux open-ended @xmath0 ## controlled retrieval-augmented context evaluation (crux) this section introduces crux, a controlled evaluation framework for assessing the retrieval context in long-form rag. it comprises: (1) definitions of retrieval context and its sub-question answerability ( § 3.1); (2) curated evaluation datasets ( § 3.2) and (3) answerability-driven performance metrics: coverage and density ( § 3.3). ## 3 4 2 open-ended @xmath1 retrieval @xmath2 x u i u 7 q e e d g c a 8 w y u 8 o y / o i / p u f c x a c 0 4 2 c w x / 4 h z + a l u v j o @xmath3 ## experiments to validate crux's evaluation capability and usability, we begin with controlled experiments with empirical retrieval contexts to enable more diagnostic retrieval evaluation. next, we analyze metric correlations between the retrieval contexts z and the corresponding final results y. finally, we assess crux's usability through human annotations and examine other configuration impacts. ## conclusion we introduced crux, an evaluation framework for assessing retrieval in long-form rag scenarios. crux provides controlled datasets and metrics, enabling evaluation of the retrieval context's coverage of relevant information and of retrieval's impact on the final result. the framework serves as a diagnostic testbed for improving methods by tackling incomplete and redundant retrieval. our experiments demonstrate that existing retrieval methods have substantial room for improvement. by doing so, we present new perspectives for advancing retrieval in long-form rag scenarios and support exploration of retrieval context optimization as a key future direction.