Optimizing LLM Based Retrieval Augmented Generation Pipelines in the Financial Domain (2024)

Abstract: Retrieval Augmented Generation (RAG) is a prominent approach in real-word applications for grounding large language model (LLM) generations in up to date and domain-specific knowledge. However, there is a lack of systematic investigations of the impact of each component (retrieval quality, prompts, generation models) on the generation quality of a RAG pipeline in real world scenarios. In this study, we benchmark 6 LLMs in 15 retrieval scenarios exploring 9 prompts over 2 real world financial domain datasets. We thoroughly discuss the impact of each component in RAG pipeline on answer generation quality and formulate specific recommendations for the design of RAG systems.

## introduction recent years have seen tremendous improvement in the ability of large language models (llm) such as @xcite and llama-2 @xcite to address users' questions/queries in diverse domains (medical questions, math problems, code assistants etc). despite llms acquiring immense parametric world knowledge during the pre-training, when adapting to real-world applications, their lack of customized domain-specific knowledge or knowledge of recent events @xcite , frequently results in outdated responses or baseless responses not grounded in the user's domain of interest, also termed hallucinations @xcite @xcite . hallucinations contribute to a lack of trust with users, and this unreliability is one of the biggest hindrances in the responsible deployment of llm based systems for critical business applications in the financial domain. retrieval augmented generation (rag) is the current go-to approach to connect llms to 1 equal contribution. live/updated information sources. existing works @xcite @xcite show rag can reduce hallucinations and improve answer quality, without the need for highly expensive and sometimes brittle domainspecific fine-tuning. given a user query, a typical rag system (figure 1 ) employs a retriever system to fetch a list of documents likely relevant to the query from an information source (retrieval). the documents are then fed into the context of the llm, with users' query / conversation history, and specific instructions / prompts on how to generate a response "grounded" in retrieved information @xcite . while there is growing number of proposals @xcite to improve rag systems (see the survey from @xcite ), very few studies @xcite systematically investigate the impact of each component (retriever, prompts, models) on answer generation quality and interactions among these various components. our goal of this paper is to evaluate the efficacy and limits of rag pipelines for question answering (q&a) systems in the highly specialized financial domain. in this study, we benchmark llms' answer generation quality and explore the following aspects: (i) comparing different generative llms as answer generation models against each other and baseline (purely extractive) models; (ii) examining how various llms handle differences in the quality of information retrieval; (iii) exploring the impact of varying prompts on answer quality of rag pipelines. in line with our objectives, we curated two datasets from the banking sector featuring real user queries. these datasets were used to design test scenarios that mimic the retrieval of information at varying levels of quality. additionally, we crafted prompts with distinct characteristics (e.g., level of detail in instructions, requirements for citations, user queries retriever retrievers at different quality levels (simulation) external data gold doc presence rate retrieve only (default) + gold docs (oracle) retrieved order shuffled order) retrieved order gold doc first gold doc last display order retrieved docs top 3, 5, 20 number of retrieval docs answer generation simple verbose citation quoting ... prompting gpt-4 gpt-35-turbo llama-2-13b llama-2-13b-chat ## experiment framework this section introduces the design of the evaluation framework (see figure 1 ). to summarize, we ran 1620 experiments to assess 6 llms in 15 retrieval conditions using 9 prompts over 2 datasets for 2 performance aspects. ## conclusion we conducted thorough investigation of the influence of several components of a rag pipeline on the overall generation quality. based on our findings, we find retrieval optimization is an important part of the rag pipeline design, even with high quality llms like gpt-4. firstly, we recommend prioritizing retrieval recall, while tuning retrieval systems, as llms exhibit pseudo-helpfulness when relevant gold document(s) are not retrieved. additionally, improving precision of retrieval, either by using re-rankers or fine-tuned retrievers, will likely improve performance as they improve the gold document(s) rank in the retrieved list. we also find rag systems to be sensitive to the presence of distractors in the context. we find a big delta between vendor llms (openai) and smaller scale open-source alternatives in our experiments, with respect to sensitivity to prompting, overall quality, and instruction following on a domain specific use cases. finally, for smaller-sized llama-2 models we recommend simple instructions as they often fail to follow longer or more complicated instructions.