EKRAG : Benchmark RAG for Enterprise Knowledge Question Answering (2025)

Abstract: Retrieval-augmented generation (RAG) offers a robust solution for developing enterprise internal virtual assistants by leveraging domain-specific knowledge and utilizing information from frequently updated corporate document repositories. In this work, we introduce the Enterprise-Knowledge RAG (EKRAG) dataset to benchmark RAG for enterprise knowledge question-answering (QA) across a diverse range of corporate documents, such as product releases, technical blogs, and financial reports. Using EKRAG, we systematically evaluate various retrieval models and strategies tailored for corporate content. We propose novel embedding-model (EM)-as-judge and ranking-model (RM)-as-judge approaches to assess answer quality in the context of enterprise information. Combining these with the existing LLM-as-judge method, we then comprehensively evaluate the correctness, relevance, and faithfulness of generated answers to corporate queries. Our extensive experiments shed light on optimizing RAG pipelines for enterprise knowledge QA, providing valuable guidance for practitioners. This work contributes to enhancing information retrieval and question-answering capabilities in corporate environments that demand high degrees of factuality and context-awareness.

## introduction large language models (llms) have demonstrated remarkable zero-shot reasoning capabilities @xcite . however, their static knowledge base, acquired during pre-training, poses significant challenges in generating factual, timely, and salient responses to ambiguous and complex queries, particularly in corporate environments. this limitation is critical in enterprise settings, where accuracy and up-todate information are paramount for a functional * equal contribution. retrieval-augmented generation (rag) @xcite emerges as a promising solution, integrating external knowledge from domain-specific corpora to potentially generate upto-date, factually correct answers. it is particularly suited for corporate internal chatbots, facilitating employees and external investors access to current company policies, procedures, and knowledge across public facing corporate documents. despite its advantages, rag presents unique evaluation challenges compared to standalone llms. their answer quality depends not only on the llm's pre-trained knowledge and reasoning capabilities but also on the relevance of retrieved context and the model's ability to integrate and synthesize information. while recent works @xcite @xcite @xcite have evaluated rag systems in general domains, assessing both answer correctness and alignment with retrieved context, the evaluation of rag in enterprise knowledge applications remains largely unexplored. to address this gap, we introduce the enterprise-knowledge rag (ekrag) dataset, comprising 1,347 manually curated questions, designed to systematically evaluate the influence of each rag component on enterprise knowledge questionanswering over five core question types in multihop settings. using this benchmark, our extensive evaluations of various retrieval models and strategies reveal that well-known techniques like hyde and hybrid search do not significantly improve enterprise knowledge retrieval accuracy, while some straightforward mechanisms such as multiembedding vector achieve excellent performance. furthermore, we propose novel evaluation methodologies: embedding-model-as-judge and ranking-model-as-judge approaches, complementing the prevailing llm-as-judge technique. our comprehensive investigation provides insights into optimizing rag pipelines for enterprise knowledge systems, offering a nuanced understanding of answer quality that could lead to more robust and reliable rag implementations in real-world corporate applications. ## related work adlakha, behnamghader, lu, meade, and reddy (2023) utilizes the existing qa datasets and evaluates the answer quality of rag along the dimensions of correctness and faithfulness. to be specific, the correctness dimension reveals the relevance between the response and the ground-truth answer. on the other hand, faithfulness measures the relevance between the answer and the retrieved context to evaluate model's capability to ground the retrieved context. rgb @xcite evaluates rag from four aspects including noise robustness, negative rejection, information integration, and counterfactual robustness. alce @xcite proposes evaluation metrics of rag along three dimensions including fluency, correctness, and citation quality. ragas @xcite develops an automated evaluation pipeline by prompting llms and evaluate rag from the dimensions of faithfulness, answer relevance and context relevance. @xcite benchmarks different rag solutions in the field of medicine and ablates the influence of each component on the overall answer quality from a multi-dimensional perspective. crag @xcite proposes a comprehensive rag benchmark consisting of questions across five domains and eight categories. ## benchmark on retrieval metrics. there are multiple metrics to evaluate the retrieval such as map, precision@k, recall@k, ndcg. in rag scenarios, recall@k is the most cruial metric, which reveals the coverage of relevant information in the retrieved chunks. thus, by default, we use recall@k as the evaluation metric. ## conclusion our comprehensive study on retrieval-augmented generation (rag) for enterprise knowledge question-answering has yielded several significant insights. through the creation and utilization of the ekrag dataset, comprising 3200 manually curated questions, we have conducted an extensive evaluation of various components within the rag pipeline. the insights gained from this study have broad implications for the optimization of rag pipelines in enterprise knowledge management systems. by shedding light on the effective-ness of various retrieval strategies and introducing new evaluation paradigms, our work contributes to the ongoing refinement of question-answering systems tailored to enterprise needs.