A uto KB : Automated Creation of Structured Knowledge Bases for Domain-Specific Support (2025)

Abstract: Effective customer support requires domain-specific solutions tailored to users’ issues. However, LLMs like ChatGPT, while excelling in open-domain tasks, often face challenges such as hallucinations, lack of domain compliance, and imprecise solutions when applied to specialized contexts. RAG-based systems, designed to combine domain context from unstructured knowledge bases (KBs) with LLMs, often struggle with noisy retrievals, further limiting their effectiveness in addressing user issues. Consequently, a sanitized KB is essential to ensure solution accuracy, precision, and domain compliance. To address this, we propose AutoKB, an automated pipeline for building a domain-specific KB with a hierarchical tree structure that maps user issues to precise and domain-compliant solutions. This structure facilitates granular issue resolution by improving real-time retrieval of user-specific solutions. Experiments in troubleshooting and medical domains demonstrate that our approach significantly enhances solution correctness, preciseness, and domain compliance, outperforming LLMs and unstructured KB baselines. Moreover, AutoKB is 75 times more cost-effective than manual methods.

## introduction customer support agents (csas) are chatbots @xcite designed to resolve domain-specific user issues by providing customized, rule-compliant solutions 1 aligned with domain standards. the advent of llms like @xcite has revolutionized conversational ai, enabling it to handle diverse, open-domain queries with exceptional fluency. however, csas, such as product troubleshooting bots or medical assistants, face distinct challenges that demand precise, parent child seed child figure 1 : illustration of an issue subtree for the seed issue my headphones do not connect to phone with two child issues shown along with their respective metadata context-aware solutions for provided issues. while llms offer remarkable general-purpose capabilities, relying solely on them risks producing incorrect or generic solutions, limiting their effectiveness in these specialized roles. to address these shortcomings, rag @xcite has emerged as a promising framework for building csas, by grounding llm responses in retrieved domain-specific knowledge. however, the performance of rag applications is largely dependent on the quality of the backend kb being used. unstructured kbs, while covering a wide range of topics, are prone to noise and irrelevant information. in contrast, structured kbs resolve these issues by incorporating specific solutions and supporting domain constraints, enabling more precise and reliable knowledge grounding. rag using unstructured kbs face additional challenges like token length limitations in llms, and difficulties in dynamically enforcing domain rules (because of unverified content in kb). a significant limitation of existing approaches is their inability to provide solutions with the appropriate granularity. for instance, in troubleshooting scenarios, addressing a generic issue like bluetooth connectivity problem is insufficient when the user faces a specific problem, such as bluetooth not pairing with a device running an older android version. generic solutions that fail to address the user's exact issue often result in dissatisfaction. additionally, ensuring compliance with domainspecific policies-such as safeguarding rules for medical guidance or hardware safety instructions in product support-remains challenging in realtime solution generation. however, techniques like chain-of-thought @xcite and reflexion @xcite can enhance rule adherence but they may exacerbate input length constraints and increase latency, thereby complicating their practical application. to address issues with solution accuracy, specificity, and domain compliance in existing systems, this paper introduces autokb, an automated pipeline for constructing a structured kb for any domain. as shown in figure 1 , the proposed kb employs a hierarchical tree structure where nodes represent issues at varying levels of granularity. root nodes cover broad, generic issues, while child nodes capture more specific ones, each linked to solutions tailored to their level of detail. node relationships are defined by metadata differences, ensuring coverage of both generic and specific user issues. following are the contributions of our work: ## related work knowledge-based support systems aim to provide accurate, specific, and safe responses to user queries. the existing approaches can be broadly categorized into two main types: prompting tech-niques for llms and rag systems, which rely on underlying kbs. various prompting techniques have been developed to enhance llm performance, particularly in rule-following, reducing hallucinations, and providing specific solutions. chain-of-thought (cot) prompting @xcite and its variants like cot with in-context learning (cot-icl) @xcite have shown promise in improving reasoning and rule-following capabilities. however, these methods still rely heavily on the llm's pretrained knowledge and may not provide grounded, specific, and safeguarded responses @xcite . rag systems @xcite combine the power of pre-trained language models with additional information, typically using retrieval methods to fetch relevant content and augment llm responses. while rag systems can improve response grounding and quality, their effectiveness is highly dependent on the quality and structure of the underlying kb. kb construction approaches can be broadly categorized into two types: unstructured and structured. unstructured kbs, built using web crawlers @xcite on popular search engines @xcite and databases @xcite , cover a wide range of topics but often suffer from noise and irrelevant information. structured kbs @xcite excel at representing domain-specific factual information and relationships between entities. however, both types face challenges in addressing specific user issues. our kb framework develops a hierarchical tree-based structure capable of accommodating specific complex user issues and their solution knowledge, bridging the gap between issue representation and solution retrieval. ## results and analysis retrieval performance analysis: ## industry impact autokb demonstrated practical effectiveness and scalability in a large e-commerce context. (1) in self-serve troubleshooting, the kb offered curated solutions for 7k issue-solution pairs across 6 products, achieving a 95% acceptance rate from human annotators. (2) during a 4-week a/b test with a troubleshooting csa across 6 product categories, autokb helped reduce the return rate and improved chatbot adoption compared to an internal baseline using manually curated kb. cost comparisons revealed significant savings. creating a kb for 265 troubleshooting issues with claude-3-haiku cost @xmath03.75/hour and 0.5 hours per issue, would cost $496.87. this demonstrates that our approach is 75 times more costeffective, showcasing its potential to lower costs in knowledge-based support systems. ## conclusion in this paper, we propose autokb, an automated strategy for curating structured kbs to deliver cor-rect, domain-compliant, and issue-specific solutions. our approach introduces a hierarchical kb, organized into parent and child issues, effectively addressing varying levels of granularity in user concerns using metadata. by leveraging contextualized chunking and rag-based solution generation, we enhance the correctness of kb solutions. experimental evaluations in troubleshooting and medical domains demonstrate that our approach outperforms traditional methods in solution quality and retrieval performance within a csa.