Commonsense and Named Entity Aware Knowledge Grounded Dialogue Generation (2022)

Abstract: Grounding dialogue on external knowledge and interpreting linguistic patterns in dialogue history context, such as ellipsis, anaphora, and co-reference is critical for dialogue comprehension and generation. In this paper, we present a novel open-domain dialogue generation model which effectively utilizes the large-scale commonsense and named entity based knowledge in addition to the unstructured topic-specific knowledge associated with each utterance. We enhance the commonsense knowledge with named entity-aware structures using co-references. Our proposed model utilizes a multi-hop attention layer to preserve the most accurate and critical parts of the dialogue history and the associated knowledge. In addition, we employ a Commonsense and Named Entity Enhanced Attention Module, which starts with the extracted triples from various sources and gradually finds the relevant supporting set of triples using multi-hop attention with the query vector obtained from the interactive dialogue-knowledge module. Empirical results on two benchmark datasets demonstrate that our model significantly outperforms the state-of-the-art methods in terms of both automatic evaluation metrics and human judgment. Our code is publicly available at https://github.com/deekshaVarshney/CNTF ; https://www.iitp.ac.in/-ai-nlp-ml/resources/codes/CNTF.zip .

## introduction neural language models usually focus on fewer language components such as sentences, phrases, or words for text analysis. however, language acts on a much broader scale -there is frequently a central theme to a conversation, and the speakers share common information in order to comprehend one another. information is frequently reused, however to avoid overuse, same things and persons are * work done during an internship at iit patna referred in the dialogue multiple times by using relevant expressions. a dialogue becomes coherent and speakers can understand each other when all of this information is delivered in a structured, logical, and consistent manner. semantic understanding of dialogues can be aided by commonsense knowledge or world facts. additionally, as a key human language phenomena, co-reference simplifies human languages while being a significant barrier for machines to understand, particularly for pronouns, which are difficult to parse due to their weak semantic meanings @xcite . grounded response generation approaches @xcite can provide replication of facts in open-domain settings, whereas commonsense knowledge is critical for creating successful interactions since socially constructed commonsense knowledge is the collection of contextual details that humans are expected to understand and use during a conversation. despite demonstrating efficacy in empirical evaluation, past work has a few significant drawbacks. there is no explicit representation of entities, semantic relations, or conversation structures, in particular. to solve such restrictions, asking a conversation model to identify relevant structures in dialogue histories can be used to directly test the level of dialogue understanding. we focus on named entity level knowledge in this paper, and analyze references to entities in a dialogue history context. to ensure the generalizability of our model, we directly incorporate entities in the form of triplets, which is the most common format of modern knowledge graphs, instead of encoding it with features or rules as in conventional approaches. take, for example, figure 1 , where the dialogue consists of eight utterances. in the third utterance, to know if there exists any relation between the director "micheal mann" and the movie "the last of the mohicans", we need to resolve the co-reference agent 1: [the last of the mohicans] is a 1992 american epic historical drama [it] is also one of my favorite movies. agent 2: me too. i love [that movie] and the soundtrack in particular. agent 1: [it] was directed by michael mann, based on [james fenimore cooper's eponymous 1826] novel agent 2: i tried to read [the book] but gave up. i can't remember what other movies michael mann has directed. agent 1: i never got to reading [the book] but i am defenetly interested. agent 2: i couldn't get into the rhythm of the writing. a little bit different in 1826! agent 1: [the movie] was produced by [morgan creek pictures] and i think [they] did a great job. agent 2: is [morgan creek] still producing movies? topic-specific knowledge james fenimore cooper's read historical figure 1 : an example of named entity and concept based knowledge triples being used for grounding dialogues in addition to topic-specific knowledge sentences. in the conversation, various shades indicate the different coreference clusters obtained. blue nodes correspond to the concepts obtained from conceptnet, red nodes correspond to the named entities obtained from the utterances in the dialogue. named entities other than the ones present in co-reference chains are highlighted in green in the conversation. we create a conversational model called cntf, commonsense, named entity and topical knowledge fused neural network to generate successful responses by leveraging both topic-specific document information and using structured entity and commonsense knowledge. we first construct triples based on named entity after resolving co-references in the dialogues to enhance the already existing commonsense triples obtained from the concept-net @xcite . we use multi-hop attention to iterate over the multi-source information. we obtain a weighted query vector from the interactive dialogue-knowledge module, which is used to query over the dialogue, topical knowledge and the corresponding triples. in each round, cntf reasons on the dialogue history and knowledge sentences, using which we filter out relevant information from the dialogue context and topical knowledge. similarly, to reason over the triples, we again iterate in multiple rounds, masking out irrelevant triples. our work makes the following contributions: ## related work sequence-to-sequence models @xcite have long been used for natural language generation (nlg) tasks. stemming off the vanilla encoder-decoder architectureintroduced initially for neural machine translation @xcite , a variety of models have been developed to enhance the quality of the responses generated @xcite @xcite ; to effectively select the conversational context in multi-turn dialogues @xcite @xcite ; and to model persona while conversing @xcite . recent advances on dialogue systems aim at enhancing dialogue generation by making it more humanized by means of incorporating knowledge based on the dialogue context or from external sources, such as unstructured documents @xcite or knowledge graphs @xcite . numerous pre-trained language models @xcite have been utilized for dialogue generation @xcite . they have been extended to leverage the knowledge from the unstructured documents and other auxiliary sources via knowledge selection and various attention fusion techniques @xcite . the task was explored in low-resource setting @xcite using a disentangled response decoder, and the usability of language models itself as a knowledge base has also been investigated in @xcite . an issue with language models is the noise which these introduce during knowledge selection. in order to limit the noise by generative models, termlevel weighting @xcite for response generation after knowledge selection were studied. @xcite proposed a pre-training based multiple knowledge syncretic transformer that uses a single framework to integrate knowledge from multiple sources. knowledge based end-to-end memory networks have been developed for taskoriented dialogue generation @xcite @xcite @xcite using multi-level, working, and dynamic types of memory. in dual dynamic memory network (ddmn) @xcite , the flow of history information during conversations is dynamically tracked to retain the important parts from both dialogue and kb, using a memory manager for each. prior studies @xcite @xcite have demonstrated the feasibility of including commonsense knowledge into the dialogue systems. further, in conkadi @xcite , felicitous facts highly relevant to the context were selected and effectively integrated in the generated response by means of fusion mechanisms. recently, co-reference resolution has been utilized for obtaining coref-informed pre-trained models @xcite , task-oriented dialogue generation @xcite , and dialogue understanding @xcite . further, @xcite demonstrated the improvement upon explicitly incorporating co-reference information to enhance the attention mechanism for the reading comprehension task. in this paper, we show how both structured and unstructured knowledge can be used to improve the task of document-grounded dialogue generation. we propose an effective knowledge-grounded dialogue model named cntf, which is built with multi-source heterogeneous knowledge. experiments on knowledge-based dialogue generation benchmark datasets, viz. wizard of wikipedia and cmu_dog, have shown the efficacy of our proposed approach. our method employs a large-scale named entity enhanced commonsense knowledge network as well as a domain-specific factual knowledge base to aid in the comprehension of an utterance as well as the generation of a response using a novel mutli-hop attention based model. ## datasets and experimental setup in this section, we present the details of the datasets and the other experimental setups. implementation details can be found in appendix c. ## conclusion we present a commonsense, named entity, and topical knowledge fused neural network (cntf) to address reasoning over multiple knowledge bases in this paper. we propose, in particular, multihop attention over both structured and unstructured knowledge. unlike previous approaches in dialog, cntf can find relevant supporting named entities in dialogs at each step of multi-hop attention in addition to already present commonsense knowledge. we test cntf on woz and cmu_dog and achieve excellent results. furthermore, our analysis shows that cntf can generate consistent results. in the future, we hope to expand our work to build models which include emotions for knowledge grounded dialogues. also, to tackle repetition and incomplete response, we aim to introduce rewards functions for these factors. currently, our model does not consider the relation attribute in our proposed framework and hence the use of "re-latedto" relation is not really affecting the performance of the proposed approach. we aim to incorporate relation attributes for triple representations.