L ocal RQA : From Generating Data to Locally Training, Testing, and Deploying Retrieval-Augmented QA Systems (2024)

Abstract: Retrieval-augmented question-answering systems combine retrieval techniques with large language models to provide answers that are more accurate and informative. Many existing toolkits allow users to quickly build such systems using off-the-shelf models, but they fall short in supporting researchers and developers to customize the *model training, testing, and deployment process*. We propose LocalRQA, an open-source toolkit that features a wide selection of model training algorithms, evaluation methods, and deployment tools curated from the latest research. As a showcase, we build QA systems using online documentation obtained from Databricks and Faire’s websites. We find 7B-models trained and deployed using LocalRQA reach a similar performance compared to using OpenAI’s text-ada-002 and GPT-4-turbo.

## introduction retrieval-augmented question-answering (rqa) systems enhance large language models (llms) by enabling them to search through a large collection of documents before answering a user's query. these systems have shown improved performance in providing more accurate, informative, and factually grounded answers compared to using llms alone @xcite @xcite . many existing toolkits, such as llamaindex @xcite and langchain @xcite , allow users to quickly build such an rqa system using off-the-shelf models such as text-ada-002 (openai, 2022a) and @xcite . however, developers often find it costly to rely on these paid services, but also face difficulties to train/deploy smaller models with competitive performance. researchers face even greater hurdles: they need to modify models/training algorithms, compare against prior work, and obtain human evaluation for their rqa system -all of which are largely neglected by existing toolkits. we introduce localrqa, an open-source toolkit that enables researchers and developers to easily train, test, and deploy rqa systems using techniques from recent research. given a collection of documents, users can use pre-built pipelines in our framework to quickly assemble an rqa system using the best off-the-shelf models. alternatively, users can create their own training data, train open-source models using algorithms from latest research, and deploy a local rqa system that achieves similar performance compared to using paid services such as openai's models. to our knowledge, localrqa is the first toolkit that provides a wide range of training algorithms and automatic evaluation metrics curated from the latest research (see ## background rqa systems combine retrievers with powerful llms to provide answers that are more accurate and informative. given a user query, a retriever first selects k most relevant passages from a collection of documents. then, a generative model produces an answer conditioned on the user's query, selected passages, and a chat history. popular methods to achieve this include concatenating all inputs into a single string and generating with decoder-only models @xcite , or processing the k passages in parallel and generating with fusion-in-decoder techniques @xcite . ## localrqa we introduce localrqa, a python-based toolkit designed to help users flexibly train, test, and deploy rqa systems. as shown in figure 2 , our toolkits employ a modular design to allow users to: generate and prepare rqa data (section 3.1), train retrieval and generative models (section 3.2 and section 3.3), build an rqa system (section 3.4), evaluate the system (section 3.5), and finally deploy the system (section 3.6). ## applications to showcase our toolkit, we built two rqa systems using data scraped from databricks and faire's online documentations (under consent). databricks provides the world's first data intelligence platform powered by generative ai, providing products that facilitate building, sharing, and maintaining data at scale. faire is an online wholesale marketplace that connects independent retailers and brands around the world. since the documents we obtained include many company/product-specific details, we believe this is an ideal use case for rqa systems. first, we describe the documentation datasets we collected in section 4.1. then, we describe our model training, baselines, and evaluation procedures in section 4.2, section 4.3, and section 4.4. finally, we present our main results in section 4.5. ## conclusion we present localrqa, a python-based toolkit designed to help users develop novel retrievalaugemented qa systems. different from existing frameworks such as llamaindex and langchain, our toolkit features a wide collection of training algorithms, evaluation metrics, and deploy-ment methods to help users quickly develop costeffective rqa systems. strong results using models and training algorithms from recent research pave the way for future work to explore rqa methods in both practical and academic settings.