KAPA : A Deliberative Agent Framework with Tree-Structured Knowledge Base for Multi-Domain User Intent Understanding (2025)

Abstract: Dialogue assistants have become ubiquitous in modern applications, fundamentally reshaping human daily communication patterns and information access behaviors. In real-world conversational interactions, however, user queries are often volatile, ambiguous, and diverse, making it difficult accurately and efficiently grasp the user’s underlying intentions. To address this challenge, we propose a simple yet effective deliberative agent framework that leverages human thought process to build high-level domain knowledge. To further achieve efficient knowledge accumulation and retrieval, we design a tree-structured knowledge base to store refined experience and data. Moreover, we construct a new benchmark, User-Intent-Understanding (UIU), which covers multi-domain, multi-tone, and sequential multi-turn personalized user queries. Extensive experiments demonstrate the effectiveness of our proposed method across multi-step evaluations.

## introduction with the rapid development of large language models (llms), the collaboration between llms and users for completing complex tasks is becoming more and more prevailing @xcite @xcite . a major challenge of such collaboration lies in the accurate understanding of user intents, which are usually quite volatile, ambiguous, and diverse in real-world applications. to better understand user intents, people have proposed many effective strategies. for example, procot @xcite uses a chainof-thought (cot) @xcite approach to design its prompting scheme, applying dynamic reasoning and planning to decide the next step (i.e., answering or asking clarification questions). @xcite propose to fine-tune the opensource llms using simulated dialogue data from user-assistant interactions, aiming to enhance their proactive interaction ability. while the above methods have achieved many promising results, there are still several inherent limitations: (1) to begin with, existing studies usually leverage simple prompting strategies like cot @xcite or construct labeled samples to train llms @xcite for estimating user intents, which are either too far from real human thought processes or requiring too much computational resources and unexplainable to the users. (2) secondly, some studies only focus on a single domain to build their conversation assistant. however, in practice, the user intents can be influenced by factors outside their domains. for example, given a user query like "i want advice on pursuing data science as a career", the assistant should not only have expertise in career planning but also a solid understanding of the technical field of data science. (3) in addition, existing works organize the external documentation into an unstructured list, which is less effective in retrieving related knowledge. to solve the above challenges, in this paper, we design a deliberative agent framework, which is inspired by the human thought process including four stages, that is, perception, analysis, reflection, and experience accumulation. in specific, through a process analysis of the specific reasons behind an agent's success or failure in reasoning and actions during simulated user-assistant dialogues, our framework iteratively explores and reflects, ultimately distilling high-level insights into a knowledge base. to better incorporate and retrieve multi-domain knowledge, we equip the above agent framework with a tree-structured knowledge base. specifically, we narrow the domain scope in a progressive and dynamic manner, transitioning from broad to refined levels to efficiently organize and re-trieve query-related knowledge. furthermore, considering that there are few public datasets available for multi-domain, multi-tone, sequential multi-turn user intent clarification tasks, we create a benchmark dataset for user-intent-understanding, called uiu, designed to evaluate our framework and support the advancement of research in this area. in a summary, the main contributions of this paper are as follows: (1) we design a deliberative agent framework with tree-structured knowledge base to better estimate user intents under multidomain settings. (2) we release a user-assistant interaction dataset, uiu, which can assess language model's capability on understanding user intents within complex multi-domain environments. (3) we conduct extensive experiments to evaluate the effectiveness of our proposed framework. ## related works this paper focuses on enhancing the proactive capabilities of llm-based agents in human collaboration and addressing their upstream task in dialog systems, namely clarifying ambiguous user queries. accordingly, we review related research on proactive agent and user intent understanding in the following sections. ## method previous works on intent understanding mainly follow either compute-intensive, non-transparent model fine-tuning or a simple pure prompt strategy that sacrifices accuracy, resulting an efficiencyeffectiveness trade-off that remains unresolved in open-domain intent clarification scenarios. furthermore, limiting intent estimation to single-domain settings is obviously inadequate for addressing the complex demands of real-world user requirements. how to prepare a self-introduction that is engaging enough? to this end, we propose a multi-domain knowledge-augmented proactive agent (kapa, for short). our core idea is to build a general multi-domain knowledge base that acts as an external extension for llm-based agents, enabling efficient and cost-effective skill enhancement. the key challenge lies in how to acquire and integrate this knowledge so that the assistant can ask fewer clarification questions and think more comprehensively during user interactions, ultimately boosting both efficiency and intelligence in the agentic dialogue system. to address the above issues, we propose a deliberative agent framework that progressively extracts multi-domain knowledge through human-like cognitive processes, while maintaining and accessing this knowledge via a tree-structured hierarchical base. ## user-intent-understanding benchmark in real-world applications, user queries usually exhibit various styles and contents. first, an intelligent assistant should be able to accurately understand multi-turn successive queries. when faced with unclear information in vague queries, it can infer some missing details from history dialogue instead of only turning to redundant clarification questions. second, in open-domain dialogue systems, user requirements may span diverse fields such as finance, technology, career, etc. third, user tones can range from succinct to enthusiastic, so the proactive agent need adopt robust conversational strategy to foster both system friendliness and user satisfaction. to create a benchmark that meets the above requirements for user intent understanding, we use a "task-dialogue" iterative generation method to produce sequential personalized tasks, reducing the complexity of simulating multi-task, multi-domain interactions. the illustration of the dataset generation process is shown in fig. 2 . in the following sections, we will elaborate on the task generation and dialogue generation, respectively. ## experiments through extensive experiments on uiu benchmark, we demonstrate that our proposed kapa effectively enhances both open-source and closedsource llms, achieving performance comparable to or even better than fine-tuned models.