RAGA s: Automated Evaluation of Retrieval Augmented Generation (2024)

Abstract: We introduce RAGAs (Retrieval Augmented Generation Assessment), a framework for reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines. RAGAs is available at [https://github.com/explodinggradients/ragas]. RAG systems are composed of a retrieval and an LLM based generation module. They provide LLMs with knowledge from a reference textual database, enabling them to act as a natural language layer between a user and textual databases, thus reducing the risk of hallucinations. Evaluating RAG architectures is challenging due to several dimensions to consider: the ability of the retrieval system to identify relevant and focused context passages, the ability of the LLM to exploit such passages faithfully, and the quality of the generation itself. With RAGAs, we introduce a suite of metrics that can evaluate these different dimensions without relying on ground truth human annotations. We posit that such a framework can contribute crucially to faster evaluation cycles of RAG architectures, which is especially important given the fast adoption of LLMs.

## introduction language models (lms) capture a vast amount of knowledge about the world, which allows them to answer questions without accessing any external sources. this idea of lms as repositories of knowledge emerged shortly after the introduction of bert @xcite and became more firmly established with the introduction of ever larger lms @xcite . while the most recent large language models (llms) capture enough knowledge to rival human performance across a wide variety of question answering benchmarks @xcite , the idea of using llms as knowledge bases still has two fundamental limitations. first, llms are not able to answer questions about events that have happened after they were trained. second, even the largest models struggle to memorise knowledge that is only rarely mentioned in the training corpus @xcite . the standard solution to these issues is to rely on retrieval augmented generation (rag) @xcite @xcite . answering a question then essentially involves retrieving relevant passages from a corpus and feeding these passages, along with the original question, to the lm. while initial approaches relied on specialised lms for retrieval-augmented language modelling @xcite , recent work has suggested that simply adding retrieved documents to the input of a standard lm can also work well @xcite @xcite , thus making it possible to use retrievalaugmented strategies in combination with llms that are only available through apis. while the usefulness of retrieval-augmented strategies is clear, their implementation requires a significant amount of tuning, as the overall performance will be affected by the retrieval model, the considered corpus, the lm and the prompt formulation, among others. automated evaluation of retrieval-augmented systems is thus paramount. in practice, rag systems are often evaluated in terms of the language modelling task itself, i.e. by measuring perplexity on some reference corpus. however, such evaluations are not always predictive of downstream performance @xcite . moreover, this evaluation strategy relies on the lm probabilities, which are not accessible for some closed models (e.g. chatgpt and gpt-4). question answering is another common evaluation task, but usually only datasets with short extractive answers are considered, which may not be representative of how the system will be used. to address these issues, in this paper we present ragas, a framework for the automated assess-ment of retrieval augmented generation systems. we focus on settings where reference answers may not be available, and where we want to estimate different proxies for correctness, in addition to the usefulness of the retrieved passages. the ragas framework provides an integration with both llamaindex and langchain, the most widely used frameworks for building rag solutions, thus enabling developers to easily integrate ragas into their standard workflow. yet another strategy is to inspect the probabilities assigned to individual tokens, where we would expect the model to be less confident in hallucinated answers than in factual ones. for instance, bartscore @xcite estimates factuality by looking at the conditional probability of the generated text given the input. @xcite use a variation of this idea. starting from the observation that llms provide well-calibrated probabilities when answering multiple-choice questions, they essentially convert the problem of validating model generated answers into a multiple-choice question which asks whether the answer is true or false. rather than looking at the output probabilities, @xcite propose to train a supervised classifier on the weights from one of the hidden layers of the llm, to predict whether a given statement is true or not. while the approach performs well, the need to access the hidden states of the model makes it unsuitable for systems that access llms through an api. for models that do not provide access to token probabilities, such as chatgpt and gpt-4, different methods are needed. selfcheckgpt @xcite addresses this problem by instead sampling multiple answers. their core idea is that factual answers are more stable: when an answer is factual, we can expect that different samples will tend to be semantically similar, whereas this is less likely to be the case for hallucinated answers. ## evaluation strategies we consider a standard rag setting, where given a question q, the system first retrieves some context c(q) and then uses the retrieved context to generate an answer a s (q). when building a rag system, we usually do not have access to human-annotated datasets or reference answers. we therefore focus on metrics that are fully self-contained and reference-free. we focus in particular three quality aspects, which we argue are of central importance. first, faithfulness refers to the idea that the answer should be grounded in the given context. this is important to avoid hallucinations, and to ensure that the retrieved context can act as a justification for the generated answer. indeed, rag systems are often used in applications where the factual consistency of the generated text w.r.t. the grounded sources is highly important, e.g. in domains such as law, where information is constantly evolving. second, answer relevance refers to the idea that the generated answer should address the actual question that was provided. finally, context relevance refers to the idea that the retrieved context should be focused, containing as little irrelevant information as possible. this is important given the cost associated with feeding long context passages to llms. moreover, when context passages are too long, llms are often less effective in exploiting that context, especially for information that is provided in the middle of the context passage @xcite . we now explain how these three quality aspects can be measured in a fully automated way, by prompting an llm. in our implementation and experiments, all prompts are evaluated using the gpt-3.5-turbo-16k model, which is available through the openai api foot_1 . faithfulness we say that the answer a s (q) is faithful to the context c(q) if the claims that are made in the answer can be inferred from the context. to estimate faithfulness, we first use an llm to extract a set of statements, s(a s (q)). the aim of this step is to decompose longer sentences into shorter and more focused assertions. we use the following prompt for this step foot_2 : given a question and answer, create one or more statements from each sentence in the given answer. question: @xcite answer: @xcite where [question] and [answer] refer to the given question and answer. for each statement s i in s(a s (q)), the llm determines if s i can be inferred from c(q) using a verification function v(s i , c(q)). this verification step is carried out using the following prompt: consider the given context and following statements, then determine whether they are supported by the information present in the context. provide a brief explanation for each statement before arriving at the verdict (yes/no). provide a final verdict for each statement in order at the end in the given format. do not deviate from the specified format. statement: the final faithfulness score, f , is then computed as @xmath0, where |v | is the number of statements that were supported according to the llm and |s| is the total number of statements in s(a s (q)). ## experiments to put the results in context, we compare our proposed metrics (shown as ragas in faithfulness measures the information consistency of the answer against the given context. any claims that are made in the answer that cannot be deduced from context should be penalized. given an answer and context, assign a score for faithfulness in the range 0-10. context: @xcite answer: @xcite ties, where the same score is assigned by the llm to both answer candidates, were broken randomly. the second baseline, shown as gpt ranking, instead asks chatgpt to select the preferred answer/context. in this case, the prompt again includes a definition of the considered quality metric. for instance, to evaluate answer relevance, we used the following prompt: answer relevancy measures the degree to which a response directly addresses and is appropriate for a given question. it penalizes the present of redundant information or incomplete answers given a question. given an question and answer, rank each answer based on answer relevancy. question: [question] ## conclusions we have highlighted the need for automated reference-free evaluation of rag systems. in particular, we have argued the need for an evaluation framework that can assess faithfulness (i.e. is the answer grounded in the retrieved context), answer relevance (i.e. does the answer address the question) and context relevance (i.e. is the retrieved context sufficiently focused). to support the development of such a framework, we have introduced wikieval, a dataset which human judgements of these three different aspects. finally, we have also described ragas, our implementation of the three considered quality aspects. this framework is easy to use and can provide developers of rag systems with valuable insights, even in the absence of any ground truth. our evaluation on wikieval has shown that the predictions from ragas are closely aligned with human judgments, especially for faithfulness and answer relevance.