D ialo KG : Knowledge-Structure Aware Task-Oriented Dialogue Generation (2022)

Abstract: Task-oriented dialogue generation is challenging since the underlying knowledge is often dynamic and effectively incorporating knowledge into the learning process is hard. It is particularly challenging to generate both human-like and informative responses in this setting. Recent research primarily focused on various knowledge distillation methods where the underlying relationship between the facts in a knowledge base is not effectively captured. In this paper, we go one step further and demonstrate how the structural information of a knowledge graph can improve the system’s inference capabilities. Specifically, we propose DialoKG, a novel task-oriented dialogue system that effectively incorporates knowledge into a language model. Our proposed system views relational knowledge as a knowledge graph and introduces (1) a structure-aware knowledge embedding technique, and (2) a knowledge graph-weighted attention masking strategy to facilitate the system selecting relevant information during the dialogue generation. An empirical evaluation demonstrates the effectiveness of DialoKG over state-of-the-art methods on several standard benchmark datasets.

## introduction traditional task-oriented dialogue systems are designed to achieve specific goals such a restaurant reservation, hotel booking and car navigation. these systems are often empowered by external domain-or task-specific knowledge that enables them to generate informative dialogues @xcite @xcite . the external knowledge in these systems is usually incorporated in the form of structured knowledge triples @xcite or unstructured documents @xcite . figure 1 depicts a knowledge-grounded dialogue about reserving a hotel. recent research primarily concentrated on various knowledge filtering methods for selecting relevant knowledge @xcite @xcite . these approaches treat the knowledge triples independently and leverage pointer networks and copy mechanisms to generate knowledge-grounded dialogues @xcite @xcite @xcite . typically, these systems generate a template or sketch-response during training and learn to fill in the slots with knowledge graph entries. such systems face two issues when they try to generate dialogues in a multi-domain setting. firstly, they are unable to capture the underlying semantics of a knowledge graph, such as the relationship between entity and relation. this leads frequently to in- in this paper, we propose a novel task-oriented dialogue system, named dialokg that employs structural information of the knowledge graph into a language model (lm) for generating informative dialogues (see figure 2a ). for this purpose, we exploit gpt-2 @xcite ) -a language model developed based on a stack of transformer decoders @xcite . specifically, we introduce a novel structure-aware multiple embedding layer-based knowledge embedding technique that constructively embeds the underlying relationship between the knowledge triples . dialokg interprets the knowledge as a knowledge graph; therefore, separate embedding layers for word token, entity, triple and token type enable the system to capture the graph features (e.g., subject, relation and object). this enables the system to generate correct and human-like dialogues and prevents generating erroneous responses such as "4 miles is located at 792 bedoin street starbucks away". furthermore, the ability to correctly capture the relationship in the knowledge graph eliminates the need for template-based or sketch-based response generation. in order to guide the decoder on relevant parts of the knowledge graph, we propose a new knowledge attention masking method. for constructing the knowledge attention mask, in each dialogue turn, a weighted graph is computed in two steps: 1) entity weights are computed using a pre-trained language model that estimates the importance of an entity for the given utterance, and 2) relation weights are computed based on the concept of graph convolution networks (gcn) @xcite . both steps take the user utterance into consideration, i.e., the obtained weighted graph is question specific. a set of triples is then selected based on the most relevant entities and relations of the weighted graph to construct a knowledge attention mask for the language model. this allows the masked language model to focus on relevant graph triples. we hypothesise that this leads to the generation of more accurate responses and enhance the model's capabilities of understanding the domain and task. to assess the performance of dialokg, we conduct experiments on three public benchmarks: smd @xcite , camrest @xcite and multi-woz 2.1 @xcite . we evaluate the system generated responses using both human and automatic metrics. furthermore, we analyse impact of the individual components on the overall performance to verify the effectiveness. our experimental results show that dialokg outperforms state-of-the-art models in knowledge-grounded dialogue generation and can generate human-like responses. we made our code publicly available foot_0 . ## conclusion we have presented dialokg, a novel knowledgegrounded task-oriented dialogue system improving the state-of-the-art across multiple benchmark datasets. dialokg focuses on capturing the underlying semantics of the knowledge graph and pays attention to the relevant graph triples to understand the task and generate correct and human-like responses. the key contributions of dialokg include 1) knowledge embedding technique, that embeds the structural information of a knowledge graph effectively, and 2) knowledge graphweighted attention masking, which guides the masked language model to attend to the relevant knowledge entries for generating correct and informative responses. finally, we showed dialokg's ability to generate accurate, diverse, and humanlike dialogues through quantitative and qualitative analysis. we performed an ablation study and studied the effect of dialogue history, knowledge embedding and knowledge attention masking.