BERGEN : A Benchmarking Library for Retrieval-Augmented Generation (2024)

Abstract: Retrieval-Augmented Generation allows to enhance Large Language Models with external knowledge. In response to the recent popularity of generative LLMs, many RAG approaches have been proposed, which involve an intricate number of different configurations such as evaluation datasets, collections, metrics, retrievers, and LLMs. Inconsistent benchmarking poses a major challenge in comparing approaches and understanding the impact of each component in the pipeline. In this work, we study best practices that lay the groundwork for a systematic evaluation of RAG and present BERGEN, an end-to-end library for reproducible research standardizing RAG experiments. In an extensive study focusing on QA, we benchmark different state-of-the-art retrievers, rerankers, and LLMs. Additionally, we analyze existing RAG metrics and datasets.

## introduction with billions of learnable parameters, large language models (llms) hold the capacity to store vast amounts of the information contained in the pretraining data, transcending mere common sense knowledge @xcite @xcite @xcite @xcite . this knowledge, embedded in the model weights, can be accessed through model prompting after an alignment step @xcite , transforming llms into universal question answering (qa) tools and sparking an unprecedented surge in commercial and scientific interest. however, a major limitation of such llms is that their knowledge is static and can not be directly manipulated. consequently, inaccurately memorized or outdated information within the model's parameters cannot be easily identified, let alone updated, 4 ). and can lead to erroneous responses. therefore, ensuring factual accuracy has become a major concern when millions of users interact with llms or when addressing domain-specific qa scenarios where llms must rely on external information. such challenges are addressed by retrieval-augmented generation (rag) @xcite @xcite , where relevant information, retrieved from a given external collection, is explicitly provided as context to the llm to generate an answer that can go beyond its internal knowledge. due to their multi-step nature, rag pipelines are complex systems whose final performance is influenced by a myriad of possible configurations and design choices. new rag approaches are usually characterized by fragmented and often suboptimal experimental setups, e.g. using outdated retrievers or unreliable metrics. the importance of the evaluation metrics is even more important in zero-shot rag settings, where llm-generated answers are more verbose compared to standard qa short answers, and surface-matching metrics fail to capture whether the answer is correct. the described inconsistency between setups makes new methods hardly comparable, and the absence of a systematic our contribution. to address the challenges described above, we introduce bergen -short for benchmark on retrieval augmented-generationa python library for easy and reproducible end-toend rag experiments. through bergen, we conduct a comprehensive study benchmarking stateof-the-art retrievers, rerankers, and llms in 500+ experiments. by comparing a large number of prominent datasets and metrics, we derive best practices for testing rag approaches, laying the groundwork for comparable results and future advancements in this field. bergen also supports multilingual datasets to promote rag development beyond english. in a nutshell, our main findings are as follows: however, neither llamaindex, dspy, nor ragged offer sufficient flexibility for a research environment and focus on a limited selection of retrievers, datasets, and metrics. additionally, flashrag lacks an in-depth analysis of such components. furthermore, a reranking functionality is often overlooked and none of the works analyze or highlight enough the importance of retrieval quality. in contrast, our framework prioritizes flexibility and extensibility by simply writing configuration files for models and datasets to cover a wide range of supported configurations. inconsistent setups. amidst the growing interest in llms, numerous rag approaches have been introduced recently @xcite @xcite @xcite @xcite @xcite . among those works, the experimental setups are fragmented at best. works vary in the use of evaluation datasets, collections, evaluation metrics, retrieval systems, and llms. we present examples of experimental setups in ## task definition rag consists of a ranking system r and a parametric generative language model θ, where the ranking system can be multi-staged. first, the ranking system builds a search index i based on a collection. then, at request time, the index i is searched yielding context segments foot_0 c that are relevant to the user input x: @xmath0,r (x). next, the llm generates a response r based on the context c and user input x both embedded in a model-specific instruction template i: @xmath1, x, c). ## benchmarking library bergen we present bergen, an open-source python library that standardizes rag experiments foot_1 . bergen supports a wide range of model architectures as well as training and evaluation configurations and at its core is designed to be extendable with minimal code. the main goal is to simplify the currently fragmented experimental setup of rag research. our library allows reproducing experiments end-to-end including data download, preprocessing, indexing, retrieval, generation, and training for a wide range of state-of-the-art models with a simple command: to accommodate the fast-paced efforts in opensourcing models and datasets, bergen is built on top of the hugging face (hf) hub to handle datasets @xcite and models @xcite , allowing for a straightforward extension with all available resources hosted on the hub, as well as locally stored ones. bergen further includes a wide set of popular qa datasets, including multilingual datasets, as well as surfacebased and llm-based metrics for evaluation. for an overview of all features included, we refer to our github repository. the library supports zero-shot evaluation as well as different fine-tuning configurations. we rely on hydra @xcite to handle complex experiment configurations. for instance, adding a new llm to bergen is as simple as adding a yaml config file: we now give an overview of models, datasets, collections, evaluation metrics, and training currently supported in bergen. ## experiments to our knowledge, the experiments we conduct with bergen present the largest rag study yet, comparing a variety of different configurations of retrievers, rerankers, llms, datasets, and metrics -as (partly) summarized in ## conclusion in this work, we present bergen, a library for benchmarking rag systems. we conduct hundreds of experiments with various configurations allowing us to analyze each part of the rag pipeline, to derive recommendations for testing and provide strong baselines for future rag experiments. we highlight it is crucial to perform semantic evaluations, in addition to commonly used surfacematching metrics we show retrieval quality significantly impacts rag response generation, underscoring the importance of using state-of-the-art retrievers, specifically rerankers. we emphasize the need to review standard benchmarks for knowledgeintensive tasks in rag. additionally, we show that llms of any size can benefit from improved retrieval methods. to keep up with the rapid development of llms and the constant release of models, we plan to add more retrieval models, llms, and datasets in the future. additionally, by designing the library to be easily extendable, we make it straightforward for the research community to contribute. to conclude, we provide a modular framework, alongside data and runs, for systematically evaluating rag pipelines and contributing to better reproducibility and understanding of the effectiveness of current and future rag systems.