End-to-End Learning of Flowchart Grounded Task-Oriented Dialogs (2021)

Abstract: We propose a novel problem within end-to-end learning of task oriented dialogs (TOD), in which the dialog system mimics a troubleshooting agent who helps a user by diagnosing their problem (e.g., car not starting). Such dialogs are grounded in domain-specific flowcharts, which the agent is supposed to follow during the conversation. Our task exposes novel technical challenges for neural TOD, such as grounding an utterance to the flowchart without explicit annotation, referring to additional manual pages when user asks a clarification question, and ability to follow unseen flowcharts at test time. We release a dataset (FLODIAL) consisting of 2,738 dialogs grounded on 12 different troubleshooting flowcharts. We also design a neural model, FLONET, which uses a retrieval-augmented generation architecture to train the dialog agent. Our experiments find that FLONET can do zero-shot transfer to unseen flowcharts, and sets a strong baseline for future research.

## introduction task oriented dialog (tod) systems @xcite converse with users to help them with specific tasks such as calendar enquiry @xcite , restaurant reservation @xcite , and tourist package recommendation @xcite . these dialog systems (e.g., restaurant reservation system) are trained using past human-to-human dialogs and associated knowledge sources (e.g., a kb of restaurants). most existing tod systems are conversational recommender systems that gather user requirements in the form of attributes (such as cuisine, location), query a kb and generate recommendations based on the retrieved results (e.g, restaurant, its phone number). while there have been recent efforts @xcite to study non-recommendation tod, several important tasks such as troubleshooting are still unexplored. troubleshooting is a common task handled by customer support agents. it involves understanding a user's problem, narrowing down the root cause and providing a solution. figure 1 shows an example dialog between an agent and a user troubleshooting a car problem. support agents typically follow a flowchart (utterances a1, a4 in our example) to diagnose user problems, but may refer to supplementary knowledge sources like faqs (a3), if user asks a clarification question (u3). in this paper, we propose the novel task of endto-end learning of a tod system that troubleshoots user's problems by using a flowchart and a corpus of faqs. our task exposes novel research challenges for tod system design. first, the system must learn to ground each utterance in the flowchart without explicit supervision. second, when required, the agent must refer to additional knowledge in the corpus of faqs to issue clarifications and add details not present in the flowchart. third, it must learn the general skill of following a flowchart, which is tested in a zero-shot transfer setting with unseen flowcharts shown at test time. before collecting a dataset for the task, we first analyze a sample of 100 in-house troubleshooting dialogs with a human customer service agent. since this is a new task, existing end-to-end tod models are not directly applicable to it. we design a baseline network named flonet foot_1 -it follows the retrieval augmented generation framework @xcite and generates agent response in two steps. first, relevant information from the flowchart and faq corpus is retrieved based on the dialog history. then, this retrieved information and dialog history generate the agent response using an encoder-decoder. we evaluate flonet in two different settings: (1) seen flowcharts (s-flo) setting, tested on flowcharts seen at train time, and (2) unseen flowcharts (u-flo) setting, to evaluate flonet's zero-shot transfer ability in handling new flowcharts unseen at train time. to summarize, the main contributions of this paper are: ## related work dialog systems can be broadly divided into two types: task oriented (tod) @xcite and open domain dialog systems @xcite . task oriented dialogs systems can further be divided into end-to-end @xcite @xcite and traditional slot filling approaches @xcite . slot filling approaches require dialog state annotations in dialog transcripts. our work falls under end-to-end approaches, which do not require any such intermediate annotations. we first briefly discuss existing tod datasets and then review approaches for collecting dialog datasets. finally, we discuss dialog systems related to flonet. dialog datasets: exisiting tod datasets can be grouped based on the type of knowledge source on which the dialogs are grounded. most of the existing datasets are for the recommendation task and grounded on structured kbs. some notable kbgrounded datasets are multiwoz @xcite , stanford multi domain dataset @xcite , camrest @xcite , frames @xcite , schema guided dialogs @xcite and taskmaster-1 @xcite . @xcite augment multiwoz with utterances grounded on faqs. the dialogs in datasets such as sharc @xcite and doc2dial @xcite are grounded on snippets from unstructured text documents. to the best of our knowledge, flodial is the first tod dataset that is grounded on flowcharts and faqs. dialog data collection: crowd sourcing frameworks for creating dialog datasets can be broadly grouped into three types. (1) wizard-of-oz framework @xcite pairs up two crowd-workers who play the roles of user and agent while conversing. the user is provided with a goal and the agent is given the knowledge necessary to achieve the goal. (2) self-dialogs framework @xcite ) requires a single crowd-worker to write the entire dialog by playing both user and agent. (3) dialog paraphrasing framework @xcite systematically generates a dialog outline (user and agent utterance) and crowdsources paraphrases for each utterance to construct a dialog. we follow this framework for collecting flodial, as it gives us adequate control over dialog flow so that we can incorporate various utterance types in the task of zero-shot response generation requires a model to generalize to new domains with just domain descriptors and no training dialogs. existing approaches @xcite @xcite model slots and intents as domain descriptors. we model flowcharts as domain descriptors and expect the system to generalize to new flowcharts unseen during train. ## the flodial dataset flodial is a corpus of troubleshooting dialogs between a user and an agent collected using amazon mechanical turk (amt). the dataset is accompa-nied with two knowledge sources over which the dialogs are grounded: (1) a set of troubleshooting flowcharts and (2) a set of faqs which contains supplementary information about the domain not present in the flowchart -both are in english. the data collection process uses the dialog paraphrasing framework @xcite and is illustrated in figure 2 . at a high level, we first systematically construct an outline for each dialog, then decompose the outline into multiple amt paraphrasing tasks, and finally stitch the dialog using the collected paraphrases. our data collection process has the following advantages: (i) systematic outline construction guarantees coverage of all paths in the flowchart, and the desired balance of utterance types in dialogs, (ii) the process ensures the annotated labels foot_2 are always correct and (iii) it provides diversity in the paraphrases collected. ## task definition & baseline system in this section, we define the problem of learning flowchart grounded task oriented dialogs in an end-to-end manner without the use of intermediate labels. we then describe our proposed baseline model, flonet, which retrieves necessary knowledge from flowchart/faqs and generates the agent response using the retrieved knowledge. ## analysis & research challenges we now investigate flonet errors, with the goal of identifying new research challenges posed by flodial. we first manually inspect the output of the generator, given the retrieved document. we find that, by and large, the generator has learned its tasks well, which are deciding whether and how to use the retrieved document in generating a response. we attribute flonet's errors primarily to the retriever. this is also apparent from recall@1 in to further diagnose retriever errors, we split them into two categories based on whether the correct retrieval is a flowchart node or a faq (digression). for the former case, finally, the challenge of zero-shot generalization to unseen flowcharts gets to the core ability of following a conversation flow, leading to the key research question: how do we improve performance on unseen flowchart setting in flodial? ## conclusion we define the novel problem of end-to-end learning of flowchart grounded task oriented dialog (tod) for a troubleshooting scenario. we collect a new flowchart grounded tod dataset (flodial), which contains 2,738 dialogs grounded on 12 different flowcharts and 138 faqs. we propose the first baseline solution (flonet) for our novel task using retrieval-augmented generation. we outline novel technical challenges for tod research identified in our work. we release flodial foot_7 and all resources foot_8 for use by the research community. bloomberg and 1mg, a visvesvaraya faculty award by govt. of india, and the jai gupta chair fellowship by iit delhi. we thank morris rosenthal for providing us with permission to use the flowcharts from www.ifitjams.com . we also thank the iit delhi hpc facility for computational resources.