C onversation L earner - A Machine Teaching Tool for Building Dialog Managers for Task-Oriented Dialog Systems (2020)

Abstract: Traditionally, industry solutions for building a task-oriented dialog system have relied on helping dialog authors define rule-based dialog managers, represented as dialog flows. While dialog flows are intuitively interpretable and good for simple scenarios, they fall short of performance in terms of the flexibility needed to handle complex dialogs. On the other hand, purely machine-learned models can handle complex dialogs, but they are considered to be black boxes and require large amounts of training data. In this demonstration, we showcase Conversation Learner, a machine teaching tool for building dialog managers. It combines the best of both approaches by enabling dialog authors to create a dialog flow using familiar tools, converting the dialog flow into a parametric model (e.g., neural networks), and allowing dialog authors to improve the dialog manager (i.e., the parametric model) over time by leveraging user-system dialog logs as training data through a machine teaching interface.

## introduction the proliferation of messaging applications and hardware devices with personal assistants has spurred the imagination of many in the technology industry to create task-oriented dialog systems that help users complete a wide range of tasks through natural language conversations. tasks include customer support, it helpdesk, information retrieval, appointment booking, etc. the wide variety of tasks has created the need for a flexible task-oriented dialog system development platform that can support many different use cases, while remaining simple for developers to use and maintain. a task-oriented dialog system is typically built as a combination of three discrete systems, performing language understanding (for identifying user intent and extracting associated information), dialog management (for guiding users towards task completion), and language generation (for converting agent actions to natural-language system responses). the dialog manager (dm) contains two sub-systems: the dialog state tracker (dst) for keeping track of the current dialog state, and the dialog policy (dp) for determining the next action to be taken in a given dialog instance. the dp relies on the internal state provided by dst to select an action, which can be a response to the user, or some operation on the back-end database (db). in this paper, we present a novel approach to building dialog managers (dms). in a typical industrial implementation of a taskoriented dialog system, the dm is expressed as a dialog flow, which is often a finite state machine, with nodes representing dialog activities (system actions) and edges representing conditions (dialog states that represent the previous user-system interactions). since a dialog flow can be viewed as a set of rules that specify the flow between dialog states, it may also be called a rule-based dm. there has been an increasing need for tools to help dialog authors 1 develop and maintain rule-based dms. these tools are often implemented as drag-and-drop wysiwyg tools that allow users to specify and visualize all the details of the dialog flow. they often have deep integration with popular integrated development environments (ides) as editing frontends. examples of rule-based or partially rule-based dms include microsoft's power virtual agents (pva) foot_1 and bot framework (bf) composer foot_2 , google's dialog flow foot_3 , ibm's watson assistant foot_4 , facebook's wit.ai foot_5 and amazon's lex foot_6 . it should be noted that most of these tools have some built-in machine-learned nlu capabilities, i.e. intent classification and entity detection, that can be leveraged to trigger different rule-based dialog flows, e.g. asking appropriate questions based on missing slots from the dialog state. however, a rule-based dm suffers from two major problems. first, these systems can have difficulty handling complex dialogs. second, updating a rule-based dm to handle unexpected user responses and off-track conversations is often difficult due to the rigid structure of the dialog flow, the long-tail (sparseness) of user-system dialogs, and the complexity in jumping to unrelated parts of the flow. in end-to-end approaches proposed recently @xcite , the dm is implemented as a neural network model that is trained directly on text transcripts of dialogs. @xcite presents a survey of recent approaches. one benefit provided by using a neural network model is that the network infers a latent representation of dialog state, eliminating the need for explicitly specifying dialog states. neural-based dms has been an area of active development for the research community as well as in industry; py-dial @xcite , parlai @xcite , plato @xcite , rasa @xcite , deeppavlov @xcite , and convlab @xcite are a few examples. however, these machine-learned neural dms are often viewed as black boxes from which dialog authors have difficulty interpreting why individual use cases succeed or fail. further, these approaches often lack a general mechanism for accepting task-specific knowledge and constraints, thus requiring a large number of validated dialog transcripts for training. collection and curation of this type of corpus is often infeasible. this paper presents conversation learner, a machine teaching tool for building dms, which combines the strengths of both rule-based and machinelearned approaches. conversation learner is based on hybrid code networks (hcns) @xcite and the machine teaching discipline @xcite . conversation learner allows dialog authors to (1) import a dialog flow developed using popular dialog composers, (2) convert the dialog flow to an hcn-based dm, (3) continuously improve the hcn-based dm by reviewing usersystem dialog logs and providing updates via a machine teaching ui, and (4) convert the (revised) hcn-based dm back into a dialog flow for further editing and verification. section 2 describes the architecture and main components of conversation learner. section 3 demonstrates conversation learner features. section 4 presents a case study of using conversation learner as the dm of a text-based customer support dialog system. ## conversation learner development of any dm follows an iterative process of generation, testing, and revision. conversation learner follows a three-stage dm development process: this development process is illustrated in figure 2 that allows dialog authors to revise the hcn-based dm; and (4) an evaluation module that allows sideby-side comparison of the dialogs generated by different dms. we describe each component in detail below. ## system demonstration the system demonstration consists of the following steps: ## case study to evaluate conversation learner's performance, we used dialog transcripts from microsoft's textbased customer support system foot_8 . this customer support system uses a sophisticated rule-based dm to handle various customer support scenarios. it was created and is regularly updated by domain experts. we chose to use transcripts of actual usersystem dialogs in order to measure the performance of the dms in a realistic setting. the customer support system consists of hundreds of "topics" out of which we selected three: "language settings", "installing or changing fonts", and "sign in problems". figure 3 shows the representation of a specific topic's dialog flow in the microsoft power virtual agents dialog composer. we exported the dialog flow graph of the support system from this system, then followed the process described in section 2.2 to train our machine-learned hcn-based dm. figure 4 shows an example of a generated train dialog. after generating the hcn-based dm, we ran the set of dialog transcripts against both rule-based and hcn-based dms. for the majority of conversations, users followed the expected flow, so the hcn-based dm produced the same results as the rule-based dm. for those that differed, we used human judges to do a blind qualitative evaluation of the conversations and choose the conversation that provided the best task-completion result. as shown in an example dialog comparison is shown in figures 5 and 6. as seen in figure 6 , the hcn-based dm handles cases where the user utterance does not match a known phrase, better than the rulesbased system. the hcn is also able to handle unexpected transitions between dialog nodes. in a rule-based system, updating the dm to handle unexpected user responses and off-track conversations is much harder due to the rigid structure of the dialog flow graph, the long-tail nature of user-system dialog transcripts (a large number of sparse examples), and complexity in transitioning between unrelated parts of the dialog flow. next, we demonstrate that the performance of the hcn dm can be substantially improved by adding just 3 to 5 teaching examples via the machine teaching ui presented in section 2.3. for this experiment, we chose dialog transcripts that contained common patterns of conversational problems, like users switching context, repeating themselves or asking follow-up questions, and corrected the dialog policy by creating or selecting the appropriate system action to resolve the problem. once these additional examples are added, as illustrated in figure 7 the hcn-based dm's performance improvement over the rule-based dm nearly tripled, from 4.53% to 13.8%. as shown in ## conclusion in this paper, we presented conversation learner, a machine teaching tool for building dialog policy managers. we have shown that the cl hcn-based dm can be bootstrapped from a rule-based dm preserving the same behavior expected from the rule-based system. using the cl machine teaching ui, the dialog author can provide corrections to the logged user-system dialogs and further improve the cl's dm performance. we demonstrated this through a case study based on dialog transcripts from microsoft's text-based customer support system where the gains were approximately 13%. we are planning to extend this work by looking into following problems: 1) investigating effectiveness of different ranking algorithms for log correction recommendation, 2) optimizing number of training samples and action masks generated from the rule-based dm, and 3) improving predictions of hcn-based dm by looking into alternative network architectures.