Towards Reliable Agents: Benchmarking Customized LLM -Based Retrieval-Augmented Generation Frameworks with Deployment Validation (2025)

Abstract: The emergence of Large Language Models has created new opportunities for building agent applications across various domains. To address the lack of targeted open benchmarks for agent frameworks, we designed a benchmark that features domain-specific, small knowledge bases, and includes a diverse set of questions categorized by type, such as simple, multi-hop, aggregation, and reasoning questions. We evaluated OpenAI’s Assistants API versus a RAG assistant built with Langchain and deployed a RAG system based on benchmark insights as a course assistant over a two-year span in a computer science course. Our findings reveal how domain-specific retrieval impacts response accuracy and highlight key challenges in real-world deployment. Notably, in smaller agentic systems with constrained knowledge bases, the primary challenge shifts from retrieval accuracy to data availability in the knowledge bases. We present insights from both benchmark evaluation and real-world usage data to guide the development of more reliable and effective agentic applications.

## introduction intelligent agents and customized assistants are becoming increasingly vital across diverse domains, fundamentally changing how organizations interact with information and users. these agents understand their environment and leverage available tools. the applications span numerous sectors: customer support agents handling product inquiries, educational tutors providing personalized learning guidance, healthcare assistants supporting medical documentation, legal assistants analyzing case documents, and financial advisors processing market reports. these domain-specific agents offer end users more accurate, grounded, and tailored solutions compared to generic language models. to help build these applications, companies from big providers like openai's assistants api and ibm's watsonx to frameworks like langchain all provide services to build agents, combining retrieval/file search, web search, code interpreters, and other tools to build 'all-aware' agents. for many use cases, retrieving relevant information is critical. despite the growing popularity of agents, there is a lack of benchmarks specifically tailored to evaluate frameworks for adopters to compare commercial and custom systems. existing benchmarks for general-purpose rag systems, such as crag @xcite , rgb @xcite , multihop-rag @xcite , and crud-rag @xcite , often rely on large-scale, dynamically changing knowledge bases like search apis and news articles, limiting reproducibility. assistant rag systems typically query a much smaller knowledge base, which introduces distinct challenges in ensuring domain expertise and alignment with the content. a benchmark for these systems should evaluate how effectively they utilize the available documents to enhance their responses and maintain alignment with the provided content. in this paper, we address this gap by devising a comprehensive end-to-end benchmark that features domain-specific, small knowledge bases, and includes a diverse set of questions on the knowledge bases categorized by type, such as simple, multihop, aggregation, and reasoning. we evaluated the benchmark using openai's assistants api and a rag assistant built with langchain. we deployed the assistant rag system for course support in the form of an information retrieval chatbot to investigate practical challenges and considerations in deploying such applications. the user interface allows questions to be posed in a conversational way, and the llm is used to summarize top search results and display them in an integrated fashion for users. this deployment allows observing user interactions, gathering insights and creating recommendations for best practices. this work answers the research questions: the paper contributes by the introduction of a benchmark for evaluating frameworks to build customized rag systems and identifying optimization challenges for real world applications through a two year evaluation of a deployed rag system built with langchain. ## conclusions and future work we introduced an open benchmark for evaluation of agentic behavior in frameworks for customizing llms. our iterative deployments revealed several crucial areas for future development: implementing escalation mechanisms for unresolved queries, developing pipelines for dynamic database expansion based on query patterns, and enhancing agentic solutions through improved tool integration and adaptive retrieval strategies.