ARES : An Automated Evaluation Framework for Retrieval-Augmented Generation Systems (2024)

Abstract: Evaluating retrieval-augmented generation (RAG) systems traditionally relies on hand annotations for input queries, passages to retrieve, and responses to generate. We introduce ARES, an Automated RAG Evaluation System, for evaluating RAG systems along the dimensions of context relevance, answer faithfulness, and answer relevance. By creating its own synthetic training data, ARES finetunes lightweight LM judges to assess the quality of individual RAG components. To mitigate potential prediction errors, ARES utilizes a small set of human-annotated datapoints for prediction-powered inference (PPI). Across eight different knowledge-intensive tasks in KILT, SuperGLUE, and AIS, ARES accurately evaluates RAG systems while using only a few hundred human annotations during evaluation. Furthermore, ARES judges remain effective across domain shifts, proving accurate even after changing the type of queries and/or documents used in the evaluated RAG systems. We make our code and datasets publicly available on Github.

## introduction retrieval-augmented generation (rag) has become a prominent approach for building userfacing nlp applications, such as systems for question answering (qa), fact-checking, and customer support @xcite . typically, a rag system consists of a retriever and a downstream language model (lm). given a user question, the retriever finds relevant passages from a corpus and the lm uses these passages to generate a response. this formulation admits a multitude of choices: what retrieval model to use, how to divide the documents into retrieval chunks, and how to prompt or finetune the lm to use the retrieved information, to name only a few of the simplest design decisions. ## ares ares proceeds in three stages (figure 1 ). there are three required inputs: an in-domain passage set, a human preference validation set of approximately 150 annotated datapoints (or more), and few-shot examples of in-domain queries and answers (five or more examples), which are used for prompting llms in synthetic data generation. with our inputs prepared, we begin by generating synthetic queries (and their answers) from the passages in the target corpus. we then use these query-passage-answer triples to train llm judges. subsequently, we apply these judges to any rag system, scoring a sample of its in-domain query-document-answer triples, and use prediction-powered inference (ppi) with our human preference validation set to estimate a confidence interval for the quality of each rag system. ## conclusion in this work, we present ares, a novel automated evaluation framework for retrieval-augmented generation (rag). ares offers a novel training pipeline for fine-tuning lightweight llm judges on synthetically generated queries and answers. ares can evaluate each component of a rag system separately to help improve system understanding and create targeted solutions, and it requires only minimal human annotations. for the eight different datasets in kilt, superglue, and ais requiring rag-based solutions, we found that ares can accurately score and rank rag systems based on context relevance, answer faithfulness, and answer relevance scores, beating the existing ragas automated evaluation framework. ares is a flexible framework, and there may be variants of it that are even more powerful than the ones we explored here. avenues to explore include gpt-4 as a replacement for human labeling (