Optimizing RAG : Classifying Queries for Dynamic Processing (2025)

Abstract: In Retrieval-Augmented Generation (RAG) systems efficient information retrieval is crucial for enhancing user experience and satisfaction, as response times and computational demands significantly impact performance. RAG can be unnecessarily resource-intensive for frequently asked questions (FAQs) and simple questions. In this paper we introduce an approach in which we categorize user questions into simple queries that do not require RAG processing. Evaluation results show that our proposal reduces latency and improves response efficiency compared to systems relying solely on RAG.

## introduction since the launch of chatgpt in november 2022, conversational systems powered by large language models (llms) have gained widespread adoption, allowing users to ask questions with the expectation of receiving accurate, factual answers @xcite @xcite . however, the responses of llms are not always accurate or even up-to-date. although llms are trained on vast datasets, they may lack access to domain specific information, such as data from a company's internal database. the knowledge of an llm is limited to the training data's cut-off date, resulting in potential obsolescence. moreover, unlike traditional retrieval systems, where knowledge is stored explicitly in structures such as knowledge graphs, llms encode knowledge implicitly within their model parameters, making information retrieval less transparent and potentially less reliable @xcite . retrieval-augmented generation (rag) has been developed as a method to address these limitations by combining the generative capabilities of llms with real-time information retrieval from external sources @xcite . in rag, external documents are embedded into vector representations and stored in a specialized vector database. when a user submits a query, it is similarly vectorized and used to retrieve relevant documents. these documents are then integrated with the query and sent to the llm for inference, ensuring that the generated response is based exclusively on the retrieved information. this hybrid approach enables rag systems to deliver accurate, up-to-date, and context-specific answers @xcite . the effectiveness of rag systems has been demonstrated across various domains. @xcite evaluated a rag-based questionanswering system for mental health support by embedding documents containing clinical practice guidelines. their results highlighted the system's ability to deliver answers that were coherent, accurate, and supported by scientific evidence. similarly, @xcite described a ragbased system designed to provide information about university fees, departments, facilities and other administrative details. their findings showed that the system retrieved relevant and accurate information more effectively than standalone llms. furthermore, the system enabled transparency and accountability by allowing users to trace each response back to its original source within the university dataset. one significant drawback of rag is that it is computationally expensive, particularly at the retrieval and inference stages. processing frequently asked questions through the entire rag workflow is both inefficient and costly. a more effective approach involves using a semantic cache, capable of handling variations and paraphrases of queries while returning consistent responses. on receiving a new request, the system ## methodology our proposal approach enhances rag chatbot capabilities through question classification and a routing mechanism, optimized to process queries of varying complexity. the main objective is to significantly improve computational efficiency and latency compared to traditional rag-based conversational agents that uniformly process all queries through the entire pipeline. at the core of this system lies a classifier that determines whether to bypass the retrieval stage for straightforward queries or engage the full retrieval pipeline for complex questions requiring additional factual support. as figure 1 shows, the proposed hybrid architecture comprises three distinct stages: classification, retrieval, and generation. in the classification stage, incoming queries are analysed to determine their complexity and information requirements. simple queries that can be addressed directly proceed immediately to the generation stage, while complex queries that necessitate additional context are routed through the retrieval pipeline. this selective engagement of the retrieval mechanism represents a key optimization in our design, substantially reducing the computational overhead associated with unnecessary document retrieval and processing. ## results the evaluation result shows the effectiveness of the rag-hybrid chatbot. figure 2 shows the plot of the latency comparison between rag-hybrid an rag-only. the plot illustrates a clear latency advantage of the rag_hybrid system over the rag_only system across 100 queries, consisting of 57 simple questions and 43 complex questions. rag_hybrid consistently demonstrates lower response times with minimal fluctuations, while rag_only exhibits significant spikes, exceeding 6 seconds for some queries. these results highlight the efficiency of the rag_hybrid system in leveraging predefined answers to maintain low latency and reduce computational overhead. also, in terms of processing needs, figure 3 demonstrates a notable difference in cpu usage between the rag_hybrid and rag_only systems across the 100 test queries. rag_hybrid consistently exhibits lower cpu utilization, maintaining efficiency and avoiding significant spikes, while rag_only shows pronounced peaks, with usage exceeding 3.5% for certain queries. these results highlight the computational efficiency of the rag_hybrid approach, which leverages predefined answers to reduce the processing burden, compared to the rag_only system that relies on resourceintensive retrieval and generation processes. the edge cases in rag_hybrid are situations where the chatbot had to respond to users' queries with rag. finally, ## conclusions and future work this study introduced a hybrid rag chatbot architecture that efficiently combines predefined question-answer pairs with retrieval-augmented generation, demonstrating notable improvements in latency, cpu usage, and overall accuracy compared to rag-only solutions. these results highlight the system's efficiency and scalability for real-time conversational ai. future efforts will focus on enhancing the classification model to adapt to evolving query patterns and integrating advanced language models to handle complex queries more effectively. we will also explore other methods of mitigating