R a LL e: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models (2023)

Abstract: Retrieval-augmented large language models (R-LLMs) combine pre-trained large language models (LLMs) with information retrieval systems to improve the accuracy of factual question-answering. However, current libraries for building R-LLMs provide high-level abstractions without sufficient transparency for evaluating and optimizing prompts within specific inference processes such as retrieval and generation. To address this gap, we present RaLLe, an open-source framework designed to facilitate the development, evaluation, and optimization of R-LLMs for knowledge-intensive tasks. With RaLLe, developers can easily develop and evaluate R-LLMs, improving hand-crafted prompts, assessing individual inference processes, and objectively measuring overall system performance quantitatively. By leveraging these features, developers can enhance the performance and accuracy of their R-LLMs in knowledge-intensive generation tasks.

## introduction large language models (llms) have shown great potential for natural language understanding and generation tasks @xcite @xcite . however, they face challenges when answering factual questions due to hallucinations (or confabulations) @xcite , outdated parametric knowledge @xcite , and memory efficiency of parametric knowledge (e.g., @xcite . to address these limitations, researchers have turned to the retrieval-augmented approach used in open-domain question answering (qa) @xcite , hereinafter referred to as retrieval-augmented llms or r-llms. in comparison to closed-book settings where language models generate answers without retrieval, * these authors contributed equally to this work. r-llms (open-book settings) enable the retrieval of relevant information from external databases or corpora @xcite , which has led to improved accuracy in opendomain qa @xcite . additionally, r-llms can acquire extended features even without additional training, such as explicit references, relief from fact hallucination @xcite , and easy updates to the knowledge source (e.g., @xcite . retrieval-augmented generation needs further research and development to reach its full potential. for example, even though the retriever-reader system has been trained on the natural questions (nq) dataset @xcite , its f1 score on the short answer task is 68.3 and still lags behind the oracle f1 score of 75.7 @xcite . this implies that further improvements can be made to the retrieval-augmented generation approach. additionally, users would be probably aware that the outputs generated by r-llms may contain factual errors, particularly when applied to knowledge-intensive tasks. however, there is currently a lack of accessible evaluation framework to assess their output quality. this makes it difficult to identify areas for improvement. furthermore, having effective tools for developing r-llms is crucial. these tools should enable the design of inference steps such as retrieve-thengenerate, selecting the combination of retrievers and llms, evaluating the performance of the entire system, and testing the prompts used in each inference step. currently available tools, such as the chatgpt retrieval plugin foot_0 , guidance foot_1 , and langchain foot_2 @xcite , offer a high degree of abstraction, making it challenging to verify the functionality of individual inference steps or optiexperimental setup and evaluation results can be tracked using mlflow. additionally, a simple chat interface can be built to test out the best practices from the development and evaluation stages in a practical setting. in this paper, we propose ralle, an accessible framework for retrieval-augmented large language model development and evaluation. we also present evaluation results of several r-llms that we have constructed by using open-source retrievers and llms. to the best of our knowledge, ralle is the first framework that empowers r-llm developers and open-domain qa researchers to efficiently develop, evaluate, and improve r-llms using objective metrics. ralle offers several key benefits: ## ralle usage figure 1 presents an overview of the key features of the proposed framework foot_3 . the primary development process involves three stages: (1) embedding and indexing the knowledge source documents, (2) designing an inference chain consisting of an r-llm with customized prompt templates for each action, and (3) benchmarking the developed r-llm. ## experimental settings in this section, we evaluate the performance of r-llms constructed with several combinations of open-source retrievers and llms on knowledgeintensive tasks. ## kilt benchmark results this section provides the downstream and retrieval performance of the r-llms developed and evaluated using ralle. ## conclusion this paper introduces ralle, an accessible framework for developing and evaluating r-llms. we also report evaluation results of several r-llms built using open-source retrievers and llms on knowledge-intensive tasks. overall, ralle offers a significant advancement in retrieval-augmented generation research, enabling efficient development, evaluation, and improvement of r-llms. we hope that ralle will contribute to the development of best practices for r-llms.