RAGLAB : A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation (2024)

Abstract: Large Language Models (LLMs) demonstrate human-level capabilities in dialogue, reasoning, and knowledge retention. However, even the most advanced LLMs face challenges such as hallucinations and real-time updating of their knowledge. Current research addresses this bottleneck by equipping LLMs with external knowledge, a technique known as Retrieval Augmented Generation (RAG). However, two key issues constrained the development of RAG. First, there is a growing lack of comprehensive and fair comparisons between novel RAG algorithms. Second, open-source tools such as LlamaIndex and LangChain employ high-level abstractions, which results in a lack of transparency and limits the ability to develop novel algorithms and evaluation metrics. To close this gap, we introduce RAGLAB, a modular and research-oriented open-source library. RAGLAB reproduces 6 existing algorithms and provides a comprehensive ecosystem for investigating RAG algorithms. Leveraging RAGLAB, we conduct a fair comparison of 6 RAG algorithms across 10 benchmarks. With RAGLAB, researchers can efficiently compare the performance of various algorithms and develop novel algorithms.

## introduction retrieval augmentation generation(rag) leverages external knowledge to mitigate hallucination issues, ensure real-time knowledge updates, and protect private data with no parametric knowledge @xcite @xcite . however, researchers face two main barriers to investigating new rag algorithms. on the one hand, many published works are either not opensource or have difficulty setting up the environment. while open-source works lack modular design, it is hard to develop new algorithms or extend 1 * equal contribution 2 † corresponding author new datasets for evaluation. researchers have to waste a lot of time developing new algorithms from scratch. on the other hand, a multitude of novel rag algorithms have merged, including iter-retgen @xcite , rrr @xcite , self-ask @xcite , active rag @xcite , self-rag @xcite , etc. however, these rag algorithms are not well aligned in their fundamental components and evaluation methodologies, making it difficult for researchers to accurately assess their improvements. as a result, the absence of a unified framework makes it difficult for researchers and engineers to select appropriate algorithms for varied contexts, potentially hindering the advancement of the field. various current works are investigating these questions, such as llamaindex @xcite , langchain @xcite , haystack @xcite , fastrag @xcite , ralle @xcite , localrqa @xcite , @xcite , and flashrag @xcite . llamaindex, langchain, and haystack are excessively encapsulated and lack transparency in internal operational mechanisms. consequently, even experienced experts abandon tools like langchain due to the lack of transparency @xcite . fastrag and ralle offer light and transparent frameworks that enable users to assemble their own rag systems using core components. autorag provides comprehensive metrics to assist users in selecting an optimal rag system for customized data. localrag provides a wide selection of model training algorithms and evaluation methods. however, localrag, fas-trag, autorag, and ralle do not reproduce published algorithms. researchers still need to invest time in replicating algorithms using the provided components. flashrag addressed this issue by reproducing a substantial number of existing algorithms. however, flashrag lacks training functionalities and fails to properly align generators to close this gap, we present raglab, a researcher-oriented rag toolkit for a fair comparison of existing rag algorithms and simplify the process of developing new algorithms. raglab provides a modular architecture for each component of the rag system, providing an ideal platform for fair comparison of algorithms. additionally, raglab designs an interactive mode and user-friendly interface, facilitating both educational purposes and demonstrations. in this paper, we introduce the raglab framework, giving an overview of core components and system workflows(section 2). we standardized key experimental variables: generator fine-tuning, instructions, retrieval configurations, knowledge bases, and benchmark. as a result, we present a comprehensive and fair comparison of 6 rag algorithms across 10 benchmarks(section 3). raglab is available on github under the mit license. ## raglab the overall architecture of raglab is illustrated in figure 1 . we first introduce the core classes and concepts, then demonstrate an experimental case using a concise 5-line code snippet. ## experiment one main aim of raglab is to facilitate fair comparisons among various advanced rag algorithms. to this end, we conducted comprehensive experi-ments by employing three distinct base models as generators while maintaining consistency across other fundamental components. experimental 1 generator: in experiment 1 , we select llama3-8b as the base language model. we utilize the open-source data provided by self-rag as training data. the resulting fine-tuned model is designated as selfrag-llama3-8b, which serves as the generator for the self-rag algorithm. to ensure a fair comparison, we removed all special tokens from the training data, then fullweighted fine-tuned another model named llama3-8b-baseline as the generator for other algorithms. for detailed training parameters, please refer to appendix a. experimental 2 generator: in experiment 2, we selected llama3-70b as the base language model. we select the qlora @xcite ) method to fine-tune selfrag-llama3-70b and llama3-70b-baseline. we use the same training data as in experiment 1. for detailed training parameters, please refer to appendix c. ## experimental result and discussion after analyzing the results from experiments 1, experiments 2, and experiments 3, we find several valuable insights. when utilizing selfrag-llama3-8b as the generator for the self-rag algorithm, its performance across 10 benchmarks did not significantly surpass other rag algorithms. however, when employing selfrag-llama3-70b as the generator, the self-rag algorithm significantly outperformed others in 10 benchmarks. we also find that naive rag, rrr, iter-retgen, and active rag demonstrate comparable performance across 10 datasets. notably, the iter-retgen algorithm exhibits superior performance in multi-hopqa tasks. furthermore, our findings indicate that rag systems underperform compared to direct llms in multiple-choice question tasks. this conclusion aligns with experimental results from other studies @xcite @xcite . a possible explanation for this phenomenon is that multiple-choice questions include both the question and candidate answers. additional retrieved information may mislead the generator. ## human evaluation of raglab to comprehensively evaluate the user experience of the raglab library, we implemented a user study. we developed a questionnaire comprising 12 questions, as shown in figure 6 . the study participants consisted of 20 nlp researchers, each having utilized raglab at least three days. the questionnaire was administered offline, achieving a 100% response rate. the results indicated that 85% of respondents perceived raglab as significantly enhancing their research efficiency, and 90% expressed willingness to recommend rablab to other researchers. additionally, we gathered valuable suggestions for improvement, which will guide future system development. ## conclusion we introduced raglab, an efficient and userfriendly library for the fair comparison of rag algorithms. with raglab, researchers can easily conduct fair comparisons of existing rag algorithms and develop new algorithms. we also conducted a fair comparison of 6 widely used rag algorithms across 10 benchmarks, finding several valuable insights. we believe raglab will become an essential research tool for the nlp community. limitations raglab provides a comprehensive and fair comparison of various rag algorithms performance. however, there remain potential improvement in future work. due to limited computational resources, raglab currently encompasses only 6 algorithms and 10 widely used benchmarks. however, there remains a need to include more algorithms and datasets. in future work, we will continue to follow the latest research developments and incorporate novel algorithms for fair comparison. during the implementation process, we found that different retriever models and external knowledge databases significantly impact the performance of rag algorithms. due to limited computational resources, we only processed wikipedia 2018 and wikipedia 2023. in future work, we plan to include a wider variety of knowledge databases and conduct experiments on how different retriever models and external knowledge databases influence the performance of rag algorithms. current research primarily focuses on improving the performance of algorithms, lacking a comprehensive evaluation of resource consumption and inference latency. at present, raglab incorporates only 3 classic metrics and 2 advanced metrics. in future work, we aim to expand our evaluation framework to include a more diverse range of metrics. we encourage the open-source community to address these limitations together. our objective is to continually refine the raglab framework, aiming to provide the most efficient and reliable evaluation platform and development tools.