Dialog Flow Induction for Constrainable LLM -Based Chatbots (2024)

Abstract: LLM-driven dialog systems are used in a diverse set of applications, ranging from healthcare to customer service. However, given their generalization capability, it is difficult to ensure that these chatbots stay within the boundaries of the specialized domains, potentially resulting in inaccurate information and irrelevant responses. This paper introduces an unsupervised approach for automatically inducing domain-specific dialog flows that can be used to constrain LLM-based chatbots. We introduce two variants of dialog flow based on the availability of in-domain conversation instances. Through human and automatic evaluation over 24 dialog domains, we demonstrate that our high-quality data-guided dialog flows achieve better domain coverage, thereby overcoming the need for extensive manual crafting of such flows.

## introduction the widespread use of large language models (llms) @xcite for chatbots, highlighted by their human-like conversational abilities across many topics, faces challenges in specialized domains due to their tendency to go off-topic. this generalization capability, while a strength, necessitates the development of more effective control mechanisms to ensure chatbots remain within the desired domain of conversation, especially in specialized fields such as healthcare or legal advice. controlling llm-based chatbots can be effectively managed through dialog flows or schemas foot_1 @xcite , which structure conversations along predefined paths of dialog actions, acting as directed graphs where nodes represent actions by the user or bot, and edges are the transitions between actions. this structure helps steer the conversation, keeping it within relevant topics, and also enables chatbots to adapt to new tasks or domains without prior training @xcite . however, the construction of precise dialog flows is challenging @xcite , given the diversity of dialog in different domains. the most prevalent approaches @xcite use schemas that are carefully handcrafted by the dialog system developers. the design of dialog schemas thus has significant manual overhead for developers, resulting in scalability and coverage limitations @xcite . this paper introduces an unsupervised method to generate domain-specific dialog flows, exploiting gpt-4's knowledge to systematically create detailed dialog flows reflecting conversational patterns in various domains. we begin by prompting gpt-4 to produce a structured representation of dialog interactions between users and bots, and then further refine this through self-reflective feedback based on a set of predefined criteria (see figure 2 ). further, when we have domain-specific conversations, our approach automatically identifies distinct user and bot dialog actions within these conversations (see figure 3 ). these dialog actions, along with selected conversations that exemplify each action, are used to condition the gpt-4 prompt to ensure the dialog flows are grounded using actual domain instances. this approach enables the automated creation of structured dialog flows, facilitating the development of effective domain-specific chatbots that adhere to their domain's conversational boundaries. our main contributions are: ## dialog flow induction a dialog flow is a flowchart comprising nodes which can be a user or bot dialog action, and edges that denote logical flow or transitions between these actions. dialog flows are tailored to different domains. figure 2 shows an excerpt of a dialog flow, with more detailed examples in the appendix. in this section, we detail our approach for automatically inducing the dialog flow for a given conversation domain. specifically, we induce two variants of dialog flows, namely intrinsic flows (in §2.1) or data-guided flows (in §2.2) depending on whether sample conversations in the domain are available. ## experiments we perform both human and automatic evaluations to assess the induced dialog flows. ## conclusion and future work we introduce a novel method for developing dialog flows that reflect the combined intrinsic knowledge of llms and existing domain-relevant dialogs. our data-driven dialog flow approach achieves better domain coverage than the intrinsic flow approach across human and automatic evaluations. our paper outlines a blueprint (in figure 1 ) for integrating the generated dialog flows into llm-based chatbots, with a primary focus on the methodologies for dialog flow generation. we believe these dialog flows can be a springboard for future interactive dialog systems that maintain a natural conversation flow within the domain.