Diverse and Informative Dialogue Generation with Context-Specific Commonsense Knowledge Awareness (2020)

Abstract: Generative dialogue systems tend to produce generic responses, which often leads to boring conversations. For alleviating this issue, Recent studies proposed to retrieve and introduce knowledge facts from knowledge graphs. While this paradigm works to a certain extent, it usually retrieves knowledge facts only based on the entity word itself, without considering the specific dialogue context. Thus, the introduction of the context-irrelevant knowledge facts can impact the quality of generations. To this end, this paper proposes a novel commonsense knowledge-aware dialogue generation model, ConKADI. We design a Felicitous Fact mechanism to help the model focus on the knowledge facts that are highly relevant to the context; furthermore, two techniques, Context-Knowledge Fusion and Flexible Mode Fusion are proposed to facilitate the integration of the knowledge in the ConKADI. We collect and build a large-scale Chinese dataset aligned with the commonsense knowledge for dialogue generation. Extensive evaluations over both an open-released English dataset and our Chinese dataset demonstrate that our approach ConKADI outperforms the state-of-the-art approach CCM, in most experiments.

## introduction nowadays, open-domain dialogue response generation systems have shown impressive potential, to endow a machine with the ability to converse with a human, using natural language @xcite . although such models have achieved promising performance, they still suffer from generating generic and boring responses, such as "i don't know." such low-quality responses always reduce the attractiveness of generative dialogue systems to end-users. researchers have tried to tackle it from multiple aspects; for example, using the enhanced objective function @xcite ; introducing additional contents @xcite . however, these methods haven't solved the issue thoroughly. different from a human being, who is capable of associating the dialogue with the background knowledge in his/her mind, a machine can merely capture limited information from the surface text of the query message @xcite . consequently, it is difficult for a machine to understand the query fully, and then to generate diverse and informative responses @xcite . to bridge the gap of the knowledge between the human and the machine, researchers have begun to introduce large-scale knowledge graphs for enhancing the dialogue generation @xcite @xcite , and they have obtained lots of impressive results. generally, the retrieval of knowledge facts is based on the entity name; in detail, the first step is to recognize entity words in the given query message, and then facts that contain the mentioned entities can be retrieved as candidates 1 . subsequently, a knowledge-aware response can be generated based on the query message and previously retrieved facts. although such a straightforward paradigm works to a certain extent, some challenges in knowledge-aware dialogue generation still keep unsolved. 1) an entity word usually can refer to different concepts, i.e., an entity has multiple meanings, but only one specific concept is involved in a particular context. without considering this, some pre-fetched knowledge fact candidates can be irrelevant to the context. 2) even if we only consider a particular entity meaning, the related knowledge facts may cover various target topics. however, some of those topics do not contribute to the dialogue generation. figure 1 presents an illustrative example to demonstrate such two issues. here, a subgraph is retrieved based on the entity word "apple" in the query. in general, "apple" can be interpreted as either a type of fruit or a brand name. in this context, it is evident that "apple" refers to a brand name. however, some knowledge facts concerning a type of fruit are retrieved too. if a model makes an inappropriate choice of irrelevant facts, the generated response will make no sense to the query message. in our example, even for the entities in blue circle related to the brand name "apple", only some of them have a positive effect in the dialogue generation, e.g., "jobs" should not make any contribution to the "#1". 3) the integration of the knowledge and the dialogue generation in previous approaches is insufficient, including the way of integration, as well as the types of knowledge. to tackle such challenges, this paper proposes a context knowledge-aware diverse and informative conversation generation model, conkadi. first, we design a felicitous fact mechanism to help the model highlight the knowledge facts that are highly relevant to the context, that is, "felicitous facts". felicitous fact mechanism generates a felicitous fact probability distribution over the retrieved facts. for improving the selection of felicitous facts, human-generated answers (i.e., the ground-truth responses) are used as the posterior context knowledge to supervise the training of the prior felicitous fact probability distribution. next, context-knowledge fusion is proposed to lift the role of knowledge facts in the dialogue generation, by fusing the context and the felicitous knowledge before the decoding. last, conkadi can generate three types of words owing to the flexible mode fusion module, which aims at simultaneously fusing multiple types of knowledge. to summarize, felicitous fact mechanism can alleviate the first two issues, and the next two techniques solve the last issue. consequently, our approach can improve the utilization rate of knowledge graphs, as well as can promote the diversity and informativeness of the generated responses. in the experiments, a large-scale chinese weibo dataset is collected and aligned with the commonsense knowledge for dialogue generation. we perform extensive evaluations on two large-scale datasets: an open-released english reddit dataset and our proposed chinese weibo dataset. the experimental results demonstrate that our proposed conkadi model significantly outperforms representative methods in knowledge utilization, diversity, and informativeness. especially, conkadi exceeds the latest knowledge-aware dialogue generation model, ccm @xcite , in most experiments. ## related work seq2seq @xcite has been widely used in the open-domain dialogue generation. however, models tend to generate generic responses @xcite . to tackle this issue, researchers have proposed new objectives @xcite , enhanced decoding algorithms @xcite , latent-variable based methods @xcite @xcite . introducing additional contents into the dialogue generation is also helpful. @xcite uses meta-words; @xcite uses the retrieved existing dialogues. however, the leading cause of generating generic responses is that the model can not obtain enough background knowledge from the query message @xcite . recently, to alleviate the lack of background knowledge, researchers have begun to introduce the knowledge into the generation. the knowledge can be the unstructured knowledge texts @xcite , the structured knowledge graphs @xcite , or the hybrid of them @xcite . the structured knowledge has the best quality, because it is generally extracted and summarized by the human. the structured knowledge graph can be either domain-specific @xcite or open-domain @xcite . conceptnet @xcite is a multilingual open-domain commonsense knowledge graph, which is designed to represent the general knowledge and to improve understanding of the meanings behind the words people use. two previous studies @xcite have proved the feasibility of introducing commonsense knowledge into dialogue systems. the first work @xcite is designed for retrieval-based systems; therefore, only the current state-of-the-art ccm @xcite is our direct competitor. in comparison with ccm, 1) conkadi is aware of the context when using the knowledge. 2) conkadi uses human's responses as posterior knowledge in training. in addition, our felicitous fact mechanism is different from the word/knowledge selection mechanisms previously proposed in related tasks; for example, selecting a cue word @xcite or selecting a knowledge @xcite . first, conkadi can access more contextual information because our model is fully end-to-end, while previous works use independent and external modules. second, our felicitous fact outputs a probabilistic distribution instead of a hard singleton value, as did the previous works. ## conclusion and future work to bridge the gap of the knowledge between machines and human beings in the dialogue genera- although conkadi has achieved a notable performance, there is still much room to improve. 1) while ats2s m m i is behind our conkadi, we find mmi can effectively enhance the ats2s; hence, in the future, we plan to verify the feasibility of the re-ranking technique for knowledge-aware models. 2) we will continue to promote the integration of high-quality knowledge, including more types of knowledge and a more natural integration method.