I nspector RAG et: An Introspection Platform for RAG Evaluation (2025)

Abstract: Large Language Models (LLM) have become a popular approach for implementing Retrieval Augmented Generation (RAG) systems, and a significant amount of effort has been spent on building good models and metrics. In spite of increased recognition of the need for rigorous evaluation of RAG systems, few tools exist that go beyond the creation of model output and automatic calculation. We present InspectorRAGet, an introspection platform for performing a comprehensive analysis of the quality of RAG system output. InspectorRAGet allows the user to analyze aggregate and instance-level performance of RAG systems, using both human and algorithmicmetrics as well as annotator quality. InspectorRAGet is suitable for multiple use cases and is available publicly to the community.A live instance of the platform is available at https://ibm.biz/InspectorRAGet

## introduction the recent advances in large language models (llms) have led to an explosion of research on retrieval-augmented generation (rag): combining generative llms with data retrieval to provide responses grounded on authoritative document collections @xcite . rag systems have been deployed in diverse domains (see @xcite for a recent survey). there has been increased recognition of the importance of rag evaluation @xcite . this consists of 1) designing, 2) running, and 3) analyzing experiments (see section 2). the research community has mainly limited analyzing to aggregate metrics via benchmark datasets @xcite , evaluation metrics @xcite as well as evaluation frameworks (such as ragas @xcite and ares @xcite ). for any input (e.g., a question), a rag system runs a retriever and passes the input and retrieved passages to a generator. this rag output can then be evaluated on a variety of metrics. the output is affected by variations in retriever and generator models as well as different model configurations. improving system performance requires actionable analysis of these variations, specific to rag. we present a platform for performing a comprehensive analysis of the quality of rag output: holistic analysis: end-to-end analysis of rag system performance involves aggregate evaluations along several dimensions and configurations, enabling continuous benchmarking of models and datasets @xcite . a comprehensive set of rag evaluation metrics comprises algorithmic scores, llm judges @xcite and human judgements, and our platform enables comparison and correlation analysis to create a full picture of performance. quality analysis of all aspects of a rag experiment: while most evaluation works focus on the quality of model outputs, it is important to also analyze the quality of all other aspects of a rag experiment. these include (a) the annotator behavior (e.g. inter-annotator agreement and underperforming annotators) (b) the effectiveness of evaluation metrics at modeling the desired functionality (e.g., faithfulness to context(s)), and (c) the rag-specific properties of the employed dataset (e.g., relevant passages and question attributes). this analysis can be used to improve the experiment as well as providing context for the interpretation of the results. error analysis through instance-level inspection: actionable error analysis for rag requires going beyond the typical aggregate-level statistics into inspection of individual instances. this is essential for developers to identify the source of undesirable output, correct erroneous reference answers, clarify ambiguous instances, and form hypotheses for model improvement. inspectorraget rag models datasets experiment runner metric evaluators (human/algorithm) experiment results uation output using various ad-hoc data processing scripts and spreadsheet tools which are not sustainable, lack re-usability, and are difficult to interpret. we provide the first platform with endto-end capabilities for thorough rag analysis, an undersupported aspect of rag system development intended for researchers, model developers, and stakeholders. our contributions are as follows: ## the rag evaluation life cycle we briefly describe the life cycle of evaluating rag systems, and situating inspectorraget within it, as shown in figure 1 . evaluating rag systems involves three main steps: ## inspectorraget platform inspectorraget is a react web application built with nextjs 13 framework foot_3 . we use the carbon design system foot_4 for the user interface. the experiment results are provided as a json input file that is loaded on the platform. this enables our platform to be lightweight; it can easily be run on virtual machines or even personal laptops/desktops. to enable privacy, inspectorraget is a stateless application and does not retain any uploaded datasets. inspectorraget's frontend offers a series of views (presented as separate tabs), each tailored to a different aspect of the analysis process. excerpts of these views are shown in figure 2 . we describe each view along with hypotheses that can be formed, analyses it enables, and insights which may require further investigation. ## evaluating inspectorraget to showcase the value of inspectorraget, we evaluate its ability to yield actionable insights on two use cases: (1) analyzing rag model performance: we collected human judgements of model responses for a rag dataset. this allows us to show the full scope of our platform for comparing human and algorithmic metrics as well as multiple annotators. the experiment setting for each use case 6 (e.g. dataset, models, and evaluation metrics) is summarized in ## conclusion we present inspectorraget, a publicly available 8 platform to empower researchers, developers, and stakeholders to gain a deeper understanding of the strengths and limitations of rag systems. it includes aggregate-level and instance-level views as well as capabilities to explore human and algorithmic metrics and annotator behavior, allowing for a more holistic analysis. our evaluation shows inspectorraget's ability to yield concrete and actionable insights on two pertinent use casesanalyzing rag model performance and llm-as-ajudge performance. we publicly release our platform, input files used in this paper, and notebooks. in the future, we will explore augmenting inspec-torraget to facilitate comprehensive pattern discovery, as well as extending its analytical capabilities beyond rag to other popular llm tasks, such as summarization and code generation.