Can I Be of Further Assistance? Using Unstructured Knowledge Access to Improve Task-oriented Conversational Modeling (2021)

Abstract: Most prior work on task-oriented dialogue systems are restricted to limited coverage of domain APIs. However, users oftentimes have requests that are out of the scope of these APIs. This work focuses on responding to these beyond-API-coverage user turns by incorporating external, unstructured knowledge sources. Our approach works in a pipelined manner with knowledge-seeking turn detection, knowledge selection, and response generation in sequence. We introduce novel data augmentation methods for the first two steps and demonstrate that the use of information extracted from dialogue context improves the knowledge selection and end-to-end performances. Through experiments, we achieve state-of-the-art performance for both automatic and human evaluation metrics on the DSTC9 Track 1 benchmark dataset, validating the effectiveness of our contributions.

## introduction driven by the fast progress of natural language processing techniques, we are now witnessing a variety of task-orientated dialogue systems being used in daily life. these agents traditionally rely on pre-defined apis to complete the tasks that users request @xcite ; however, some user requests are related to the task domain but beyond these apis' coverage @xcite . for example, while task-oriented agents can help users book a hotel, they fall short of answering potential follow-up questions users may have, such as "whether they can bring their pets to the hotel". these beyond-api-coverage user requests frequently refer to the task or entities that were discussed in the prior conversation and can be addressed by interpreting them in context and retrieving relevant domain knowledge from web pages, for example, from textual descriptions and frequently asked questions (faqs). most taskoriented dialogue systems do not incorporate these external knowledge sources into dialogue modeling, making conversational interactions inefficient. to address this problem, @xcite recently introduced a new challenge on task-oriented conversational modeling with unstructured knowledge access, and provided datasets that are annotated for three related sub-tasks: (1) knowledgeseeking turn detection, (2) knowledge selection, and (3) knowledge-grounded response generation (one data sample is in section b.1 of supplementary material). this problem was intensively studied as the main focus of the dstc9 track 1 @xcite , where a total of 105 systems developed by 24 participating teams were benchmarked. in this work, we also follow a pipelined approach and present novel contributions for the three subtasks: (1) for knowledge related turn detection, we propose a data augmentation strategy that makes use of available knowledge snippets. (2) for knowledge selection, we propose an approach that makes use of information extracted from the dialogue context via domain classification and entity tracking before knowledge ranking. (3) for the final response generation, we leverage powerful pre-trained models for knowledge grounded response generation in order to obtain coherent and accurate responses. using the challenge test set as a benchmark, our pipelined approach achieves state-of-the art performance for all three sub-tasks, in both automated and manual evaluation. ## approach our approach to task-oriented conversation modeling with unstructured knowledge access @xcite includes three successive sub-tasks, as illustrated in figure 1 . first, knowledge-seeking turn detection aims to identify user requests that are beyond the coverage of the task api. then, for detected queries, knowledge selection aims to find the most appropriate knowledge that can address the user queries from a provided knowledge base. finally, knowledge-grounded response generation produces a response given the dialogue history and selected knowledge. dstc9 track 1 @xcite organizers provided a baseline system that adopted the finetuned gpt2-small @xcite for all three sub-tasks. the winning teams (team 19 and team 3) extensively utilized ensembling strategies to boost the performance of their submissions @xcite @xcite . we follow the pipelined architecture of the baseline system, but made innovations and improvements for each sub-task, outlined in detail below. ## experiments and results we used the same data split and evaluation metrics as the official dstc9 track 1 challenge. all model training and dataset details are summarized in the section b of the supplementary material. ## conclusions in this work, we propose a comprehensive system to enable the task-orientated dialogue models to answer user queries that are out of the scope of apis. we significantly improved the system's capability of finding the most relevant knowledge snippets, consequently providing excellent responses by introducing a novel data augmentation method, incorporating domain and entity identification modules for knowledge selection, and utilizing mixed negative sampling. to demonstrate the efficacy of our approach, we benchmark our system on the dstc9 track 1 challenge dataset and report the state-of-the-art performance.