Implementing Retrieval Augmented Generation Technique on Unstructured and Structured Data Sources in a Call Center of a Large Financial Institution (2025)

Abstract: The retrieval-augmented generation (RAG) technique enables generative AI models to extract accurate facts from external unstructureddata sources. For structured data, RAG is further augmented by function calls to query databases. This paper presents an industrialcase study that implements RAG in a large financial institution’s call center. The study showcases experiences and architecture for ascalable RAG deployment. It also introduces enhancements to RAG for retrieving facts from structured data sources using data embeddings, achieving low latency and high reliability. Our optimized production application demonstratesan average response time of only 7.33 seconds. Additionally, the paper compares various open-source and closed-source models for answer generation in an industrial context.

## introduction with the rapid development of generative ai @xcite , the retrieval-augmented generation (rag) @xcite technique has become popular in academia and industrial applications @xcite @xcite . rag involves two phases: ingestion, where document chunks are vectorized and stored in vector databases, and inference, where relevant chunks are retrieved to answer questions using a large language model (llm). although rag is effective with unstructured data, industrial applications often involve structured data. a common approach in the literature to retrieve structured data is to leverage llm to translate a text query into a database-specific query (such as sql), then call a database function to retrieve relevant facts @xcite . this approach increases the number of calls to llm (incurring cost and delay) and sometimes it doesn't translate queries correctly. in this paper, we present a case study on applying the rag technique to a call center of a business unit of a very large financial institution. the call center has been in business for many decades. its data span various structured and unstructured sources. when a customer calls, a customer service representative (csr) answers the questions by looking up information from unstructured policy documents or structured data sources. some of these sources can overlap and complicate the efforts of a csr to respond to queries promptly. our rag application converts structured and unstructured data into chunks and vectorizes them using embedding models during the ingestion phase. this optimization improves latency (fewer llm calls) and accuracy at inference time. we implement our approach by converting headers and rows of structured data (database tables) into json strings and grouping them by business concepts. these json chunks are transformed into embeddings and stored in a vector database index. similarly, we convert unstructured policy documents into chunks and store them in a separate index. during inference, we retrieve the top k relevant chunks from both indexes based on the input query, combine them into a prompt, and use gpt-3.5 to generate a grounded answer. an independent model (llama 3 or gpt-4) validates the answer's quality with a confidence rating. we monitor performance by capturing confidence ratings, human feedback and response times. our production application has consistently generated accurate, grounded answers without hallucinations since may 2024. we observed occasional errors due to missing data or ambiguous contexts. these were fixed through updates to data pipelines and prompt revisions. we also optimized response times from an initial launch of an average of 21.91s to an average of 7.33 seconds. we present a comparison study of popular llms in the rag application to facilitate model selection. finally, we also present our application architecture, which will help the community in developing industrial-scale rag applications. ## background and related work to develop a rag (retrieval-augmented generation) application, documents are first divided into smaller chunks @xcite . these chunks can be created using a sliding window approach with some overlaps of words between chunks @xcite , or through advance methods such as semantic chunking to keep semantically coherent text together in one chunk @xcite . later, each chunk is indexed with its corresponding vector representation using an embedding model. during inference, these chunks are retrieved based on their semantic similarity with a question and are passed as part of the prompt to an llm to generate an answer @xcite . if data is in a structured format like a relational database, then below are some of the methods to process the data for a rag application. raw sql query: sql is widely used for querying structured data due to its rapid query processing capabilities for real-time data analysis and simple syntax for sql queries @xcite . @xcite . sql queries can be used to retrieve structured data in the rag technique, and then llm can generate the answer using the prompt created from the retrieved data. however, the raw sql query approach could not be directly applied with a user's natural language query. the text-to-sql method is proposed to bridge this gap. text-to-sql: to bridge the gap between natural language queries and sql queries, the text-to-sql @xcite approach converts natural language queries into sql using encoder-decoder models, typically based on lstm @xcite or transformer architectures @xcite . the encoder transforms natural language into vectors, while the decoder generates sql queries, either through sketch-based methods (breaking down sql clauses) or end-to-end generation (producing entire sql queries). text-to-sql systems are user-friendly, eliminating the need for programming skills @xcite . modern large language models (llms) can convert text to sql. this means that we can use an llm to convert a query to sql in the rag technique and then make a function call to a database to retrieve data (langchain, 2024c). however, these models can be sensitive to input variations and may struggle with queries outside their training domain @xcite . this approach also increases the number of calls to an llm, resulting in increased latency. training table embedding model: other approaches such as tapas @xcite use transformer-based architectures to pretrain tabular embedding models by flattening tables into 1-d sequences and adding various positional embeddings to understand table structures. the pretraining employs a masked language model loss function @xcite , followed by fine-tuning with questions, tables, and answers. however, this method has limitations: it requires the full table input, which is impractical for large tables, and often only a subset of the table is relevant to the query, leading to noise and confusion. ## methodology the architecture of our application is shown in figure 1 , with four major components: ingestion and indexing, inference, monitoring, and user interface. ## conclusion and future work in this paper, we presented an industrial case study on the implementation of rag technique. we presented a novel enhancement to the rag technique by transforming structured data to json format and then embedding it in the same way as unstructured data for faster and accurate answer generation. we also showed a comparison of popular open-source and closed-source llms on answer generation in our business case. we conclude that lower response time and highly accurate answers can be retrieved using our approach combined with scalable infrastructure. we also conclude that ll-mops is important for industrial applications and helps in maintaining the high quality of answer generation. we also conclude that llama-3, gpt-4o and mistral small are as good as gpt-3.5 in answer generation. our proposed methodology is highly generalizable and could be easily applied to other business use cases, where both structured and unstructured data are queried to generate a grounded answer. in the future, we will expand the application to serve other business lines such as presale consulting services, where sales agents need access to both unstructured knowledge articles and product specifications stored in structured databases. in addition to serving financial institutions, our application can be readily adapted for other industries, such as healthcare institutions where a large amount of structured and unstructured medical data needs to be leveraged to answer a complex question. we hope that this work can provide insights into the use of both structured and unstructured data in an end-to-end manner in rag applications and inspire new advanced rag applications in industry.