Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access (2022)

Abstract: To alleviate the problem of structured databases’ limited coverage, recent task-oriented dialogue systems incorporate external unstructured knowledge to guide the generation of system responses. However, these usually use word or sentence level similarities to detect the relevant knowledge context, which only partially capture the topical level relevance. In this paper, we examine how to better integrate topical information in knowledge grounded task-oriented dialogue and propose “Topic-Aware Response Generation” (TARG), an end-to-end response generation model. TARG incorporates multiple topic-aware attention mechanisms to derive the importance weighting scheme over dialogue utterances and external knowledge sources towards a better understanding of the dialogue history. Experimental results indicate that TARG achieves state-of-the-art performance in knowledge selection and response generation, outperforming previous state-of-the-art by 3.2, 3.6, and 4.2 points in EM, F1 and BLEU-4 respectively on Doc2Dial, and performing comparably with previous work on DSTC9; both being knowledge-grounded task-oriented dialogue datasets.

## introduction task-oriented (or goal-oriented) dialogue systems aim to accomplish a particular task (e.g. book a table, provide information) through natural language conversation with a user. the system's available actions are often described by a pre-defined domainspecific schema while relevant knowledge is retrieved from stuctured databases or apis @xcite . as such, taskoriented dialogue systems are often limited on which actions can be taken and what information can be retrieved @xcite . to relax these restrictions, some dialogue systems (also referred how do i apply for disability benefits? you should apply for benefits as soon as you become disabled. are you interested in applying now? no, i don't think i am ready. do you have any documents or publications that i can read? we do, what are you interested in? applying online for disability benefits? i am wondering about if i don't have enough work credits to be eligible for disability. what should i do? we do have some information on page 5 of our supplemental security income ssi booklet. dialogue turns user system topic: how you apply you should apply for disability benefits as soon as you become disabled. if you are ready to apply now, you can : complete your application online, or call our toll. topic: publications apply online for disability benefits, disability starter kit, adult disability checklist, the appeals process, other disability publications. topic: application is denied people who don't have enough work credits may qualify for supplemental security income ssi. please read page 5 of our supplemental security income ssi booklet. knowledge candidates to as goal-oriented chatbots) adopt open-domain language that is by definition unconstrained by predefined actions @xcite , and dynamically extract any required knowledge from in-domain unstructured collections in the form of entity descriptions, faqs, and documents. access to external knowledge sources has also been shown to help dialogue systems generate more specific and informative responses, which helps with the "common response" problem @xcite @xcite @xcite . figure 1 shows an example of a task-oriented dialogue that exploits external unstructured knowledge sources. given a history of previous dialogue turns, with each turn consisting of one user and system utterance, and access to in-domain unstructured knowledge sources (either a document collection or a set of candidate facts), the dialogue system needs to generate an appropriate system response for the current turn. recent research @xcite tackles the task by decomposing it into two sub-tasks: to initially determine the relevant knowledge (if any) that needs to be extracted/selected from external resources, and to subsequently generate the response based on the selected knowledge and the dialogue history. when retrieving knowledge from unstructured sources, different sources may need to be accessed in different dialogue turns; this is to be expected in most conversation scenarios. in the example of figure 1 , the first turn is grounded on the first knowledge candidate, and subsequent turns are grounded on later candidates. if we consider that each knowledge source belongs to a different topic or domain (e.g. "how you apply", "publications", "application is denied" in our example), we can observe that as the knowledge selection shifts across sources during the course of the dialogue, a corresponding shift occurs between topics. previous work has not actively exploited this, but we posit that attending the topic shifts in the dialogue history can provide signals that help distinguish relevant from irrelevant sources for knowledge selection, and that such topical information can help the model derive an importance weighting scheme over the dialogue history for better response generation. in this paper, we model topic shifts in selected knowledge sources to improve topic-aware knowledge selection and response generation in taskoriented dialogue, and propose "topic-aware response generation" (targ), an end-to-end model for knowledge selection and response generation. our approach incorporates multiple topic-aware attention mechanisms to derive the importance weighting scheme over previous utterances and knowledge sources, aiming for a better understanding of the dialogue history. in addition, targ is built on top of recent breakthroughs in language representation learning by finetuning on the pretrained language model bart @xcite . we conduct extensive experiments with two taskoriented dialogue datasets, namely doc2dial @xcite and dstc9 @xcite . our results indicate that targ foot_0 is able to accurately select the appropriate knowledge source, and as a result generate more relevant and fluent responses, outperforming previous state-of-the-art by 3.2, 3.6, and 4.2 points in em, f1 and bleu-4 respectively on doc2dial, and performing comparably with previous work on dstc9. furthermore, we present an ablation study and a case study accompanied by analysis of the learned attention mechanisms. ## our approach as we mentioned in the introduction, our proposed approach (targ) exploits topic-aware mechanisms to derive an importance weighting scheme over different utterances in the dialogue history, with the goal to better inform knowledge selection and response generation. for a brief overview of targ, please consult figure 2 . the input in our task consists of the dialogue history of previous user and system utterances, and a set of external knowledge candidates (hereafter referred to as factoids for brevity). the goal is to generate the next system utterance in the dialogue, which may or may not be grounded in one of the factoids; some of the dialogue history utterances may also be grounded on factoids but not necessarily all of them are. briefly, to generate the next turn's system utterance, targ initially generates bart-based representations for every previous user and system utterance in the dialogue history, for every available factoid, and for both utterances' and factoids' corresponding topics. for each utterance / factoid pair, targ extracts matching features by calculating feature interaction over their encoded representations. targ subsequently weights the matching features by topic-aware attention mechanisms, and aggregates them in a tensor. finally, a knowledge selection layer outputs a relevance score over factoids, and the decoder generates the system utterance based on the most relevant factoid's encoding. ## conclusion in this paper, we proposed targ: "topic-aware response generation", a topic-aware model which incorporates multiple topic-aware attention mechanisms to derive the importance weighting scheme over both dialogue utterances and unstructured external knowledge, and through that facilitate better dialogue history understanding. our proposed method achieves state-of-the-art results in both knowledge selection and response generation, outperforming previous state-of-the-art by 3.2, 3.6, and 4.2 points in em, f1 and bleu-4 respectively on doc2dial, and performing comparably with previous work on dstc9. to provide further insights, we also presented an ablation study of our model that supported the importance of our method's various components, and discussed a case study accompanied by an analysis of the attention mechanisms.