Beyond Domain API s: Task-oriented Conversational Modeling with Unstructured Knowledge Access (2020)

Abstract: Most prior work on task-oriented dialogue systems are restricted to a limited coverage of domain APIs, while users oftentimes have domain related requests that are not covered by the APIs. In this paper, we propose to expand coverage of task-oriented dialogue systems by incorporating external unstructured knowledge sources. We define three sub-tasks: knowledge-seeking turn detection, knowledge selection, and knowledge-grounded response generation, which can be modeled individually or jointly. We introduce an augmented version of MultiWOZ 2.1, which includes new out-of-API-coverage turns and responses grounded on external knowledge sources. We present baselines for each sub-task using both conventional and neural approaches. Our experimental results demonstrate the need for further research in this direction to enable more informative conversational systems.

## introduction traditionally, task-oriented dialogue systems have focused on providing information and performing actions that can be handled only by given databases or apis. however, in addition to task-focused requests, users also have needs that go beyond what is provided by the backend resources. for example, while most virtual assistants can help users book a hotel, a restaurant or movie tickets, they fall short of answering potential follow-up questions users may have, such as: where to park vehicles; whether they are allowed to bring pets or children to the reserved place; or what the cancellation policy is. no api/db entry is usually available to handle such requests. on the other hand, relevant domain knowledge is already available on web pages in the form of descriptions, faqs and customer reviews for many of these out-of-coverage scenarios. since current dialogue systems don't incorporate these external knowledge sources into task-oriented conversational modeling, users need to visit the websites by themselves to find out any additional information beyond api/db coverage, making conversational interactions inefficient. in this work, we propose a new conversational modeling task towards frictionless task-oriented scenarios, where the flow of the conversation does not break when users have requests that are out of the coverage of apis/db but potentially are already available in external knowledge sources. inspired by recent studies on knowledge-grounded conversational modeling @xcite @xcite , our proposed task aims to develop end-to-end dialogue systems to understand relevant domain knowledge, and generate system responses with the selected knowledge. different from previous work on social conversations @xcite @xcite , this task addresses task-oriented conversations grounded on fine-grained domain-level or entity-level knowledge sources related to given dialogue contexts. figure 1 shows an example conversation with unstructured knowledge access. the user utterances at turns @xmath0, 7} and @xmath1, 15} request the policy details about bringing pets and making payments, respectively, which are out of the coverage of the structured domain apis. on the other hand, the relevant knowledge contents can be found from the external sources as in the rightmost column which includes the qa snippets from the faq lists for each corresponding entity within domains such as train, hotel, or restaurant. with access to these unstructured external knowledge sources, the agent managed to continue the conversation with no friction by responding adequately at the turns @xmath2, 8, 12, 16}. task-oriented dialogue systems aim to enable users to complete tasks by interacting with an automated agent in natural language @xcite . these systems typically convert user utterances to a semantic representation (such as domain, intent, and slots @xcite ) based on what is used by the backend resources (such as apis) that accomplish the tasks. at each turn, the dialogue system decides the next action to take based on the estimated dialogue state as well as any results or responses from the backend resources @xcite @xcite . the next action, which is typically in the form of a semantic frame formed of dialogue acts, arguments and values, is converted to a natural language response to the user by natural language generation @xcite . on the other hand, social conversational systems typically follow an end-to-end approach, and aim to generate target responses based on the previous conversation context @xcite @xcite . @xcite proposed an extension to these models that grounds the responses on unstructured, textual knowledge, by using end-to-end memory networks where an attention over the knowledge relevant to the conversation context is estimated. along similar lines, @xcite used pattern matching, named entity recognition and linking to find facts relevant to the current dialogue and other related entities from a knowledge base. @xcite proposed both static and dynamic graph attention mechanisms for knowledge selection and response generation, respectively, using knowledge graphs. more recently, @xcite and @xcite both have publicly released large conversational data sets, where knowledge sentences related to each conversation turn are annotated. our proposed task, data, and baseline models in this work differ from these studies in the following aspects: we target task-oriented conversations with more clear goals and explicit dialogue states than social conversations; and we aim to incorporate task-specific domain knowledge instead of commonsense knowledge. the other line of related work is machine reading comprehension which aims to answer questions given unstructured text @xcite @xcite and has later been extended to conversational question answering @xcite . in our work, the document required to generate a response needs to be identified according to the conversation context. the responses are also different in that, rather than plain answers to factual questions, we aim to form factually accurate responses that seamlessly blend into the conversation. ## problem definition we define an unstructured knowledge-grounded task-oriented conversational modeling task based on a simple baseline architecture (figure 2 ) which decouples turns that could be handled by existing task-oriented conversational models with no extra knowledge and turns that require external knowledge resources. in this work, we assume that a conventional api-based system already exists and focus on the new knowledge access branch which takes a dialogue context u @xmath3, • • • , u t-1 , u t } and knowledge snippets @xmath4, • • • , k n }, where u i is the i-th utterance in a given dialogue, t is the time-step of the current user utterance to be processed, w is the dialogue context window size. our proposed task aims to generate a contextappropriate system response ũt+1 grounded on a set of relevant knowledge snippets k ⊂ k. the remainder of this section presents the detailed formulations of the following three sub-tasks: 'knowledge-seeking turn detection', 'knowledge selection', and 'knowledge-grounded response generation'. ## conclusions this paper proposed a new task-oriented conversational modeling problem grounded on unstructured domain knowledge, which aims to handle out-of-api coverage user requests. to support research on our proposed tasks, we introduced an augmented version of multiwoz 2.1 dialogues with additional knowledge-seeking turns collected given external knowledge sources. we presented baseline methods based both on non-machine learning approaches and neural model architectures. furthering this work, we plan to collect more dialogues including different domains, entities, and locales from the original ones for multiwoz 2.1. moreover, this new data set will include not only written conversations, but also spoken dialogues to evaluate the system performances for more realistic scenarios. then, all the data sets and the baselines will be released for establishing a new public benchmark in dialogue research. in addition, we will continue to iterate on the models with the following potential enhancements: end-to-end learning instead of the pipelined processing, joint modeling of both knowledge-seeking and api-driven branches, and few shot transfer learning for unseen domains or knowledge sources.