Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs (2020)

Abstract: Human conversations naturally evolve around related concepts and hop to distant concepts. This paper presents a new conversation generation model, ConceptFlow, which leverages commonsense knowledge graphs to explicitly model conversation flows. By grounding conversations to the concept space, ConceptFlow represents the potential conversation flow as traverses in the concept space along commonsense relations. The traverse is guided by graph attentions in the concept graph, moving towards more meaningful directions in the concept space, in order to generate more semantic and informative responses. Experiments on Reddit conversations demonstrate ConceptFlow’s effectiveness over previous knowledge-aware conversation models and GPT-2 based models while using 70% fewer parameters, confirming the advantage of explicit modeling conversation structures. All source codes of this work are available at https://github.com/thunlp/ConceptFlow .

## introduction the rapid advancements of language modeling and natural language generation (nlg) techniques have enabled fully data-driven conversation models, which directly generate natural language responses for conversations @xcite @xcite . however, it is a common problem that the generation models may degenerate dull and repetitive contents @xcite , which, in conversation assistants, leads to off-topic and useless responses. @xcite @xcite . conversations often develop around knowledge. a promising way to address the degeneration problem is to ground conversations with external knowledge @xcite , such as open-domain knowledge graph @xcite , commonsense knowledge base @xcite , or background documents @xcite . recent research leverages such external knowledge by using them to ground conversations, integrating them as additional representations, and then generating responses conditioned on both the texts and the grounded semantics @xcite . integrating external knowledge as extra semantic representations and additional inputs to the conversation model effectively improves the quality of generated responses @xcite @xcite . never-theless, some research on discourse development suggests that human conversations are not "still": people chat around a number of related concepts, and shift their focus from one concept to others. @xcite models such concept shift by breaking discourse into several segments, and demonstrating different concepts, such as objects and properties, are needed to interpret different discourse segments. attentional state is then introduced to represent the concept shift corresponding to each discourse segment. @xcite shows that people may switch dialog topics entirely in a conversation. restricting the utilization of knowledge only to those directly appear in the conversation, effective as they are, does not reach the full potential of knowledge in modeling human conversations. to model the concept shift in human conversations, this work presents conceptflow (conversation generation with concept flow), which leverages commonsense knowledge graphs to model the conversation flow in the explicit concept space. for example, as shown in figure 1 , the concepts of a conversation from reddit evolves from "chat" and "future", to adjacent concept "talk", and also hops to distant concept "dream" along the commonsense relations-a typical involvement in natural conversations. to better capture this conversation structure, conceptflow explicitly models the conversations as traverses in commonsense knowledge graphs: it starts from the grounded concepts, e.g., "chat" and "future", and generates more meaningful conversations by hopping along the commonsense relations to related concepts, e.g., "talk" and "dream". the traverses in the concept graph are guided by graph attention mechanisms, which derives from graph neural networks to attend on more appropriate concepts. conceptflow learns to model the conversation development along more meaningful relations in the commonsense knowledge graph. as a result, the model is able to "grow" the grounded concepts by hopping from the conversation utterances, along the commonsense relations, to distant but meaningful concepts; this guides the model to generate more informative and on-topic responses. modeling commonsense knowledge as concept flows, is both a good practice on improving response diversity by scattering current conversation focuses to other concepts @xcite , and an implementation solution of the attentional state mentioned above @xcite . our experiments on a reddit conversation dataset with a commonsense knowledge graph, conceptnet @xcite , demonstrate the effectiveness of conceptflow. in both automatic and human evaluations, conceptflow significantly outperforms various seq2seq based generation models @xcite , as well as previous methods that also leverage commonsense knowledge graphs, but as static memories @xcite @xcite . notably, conceptflow also outperforms two finetuned gpt-2 systems @xcite , while using 70% fewer parameters. explicitly modeling conversation structure provides better parameter efficiency. we also provide extensive analyses and case studies to investigate the advantage of modeling conversation flow in the concept space. our analyses show that many reddit conversations are naturally aligned with the paths in the commonsense knowledge graph; incorporating distant concepts significantly improves the quality of generated responses with more on-topic semantic information added. our analyses further confirm the effectiveness of our graph attention mechanism in selecting useful concepts, and conceptflow's ability in leveraging them to generate more relevant, informative, and less repetitive responses. ## related work sequence-to-sequence models, e.g., @xcite , have been widely used for natural language generation (nlg), and to build conversation systems @xcite @xcite . recently, pretrained language models, such as elmo @xcite , unilm @xcite and gpt-2 @xcite , further boost the nlg performance with large scale pretraining. nevertheless, the degenerating of irrelevant, off-topic, and non-useful responses is still one of the main challenges in conversational generation @xcite @xcite . recent work focuses on improving conversation generation with external knowledge, for example, incorporating additional texts @xcite @xcite , or knowledge graphs @xcite . they have shown external knowledge effectively improves conversation response generation. the structured knowledge graphs include rich semantics represented via entities and relations @xcite . lots of previous studies focus on task-targeted dialog systems based on domain-specific knowledge bases @xcite @xcite . to generate responses with a large-scale knowledge base, @xcite and @xcite utilize graph attention and knowledge diffusion to select knowledge semantics for utterance understanding and response generation. @xcite focuses on the task of entity selection, and takes advantage of positive entities that appear in the golden response. different from previous research, concept-flow models the conversation flow explicitly with the commonsense knowledge graph and presents a novel attention mechanism on all concepts to guide the conversation flow in the latent concept space. ## methodology this section presents our conversation generation model with latent concept flow (conceptflow). our model grounds the conversation in the concept graph and traverses to distant concepts along commonsense relations to generate responses. ## experiment methodology this section describes the dataset, evaluation metrics, baselines, and implementation details of our experiments. ## evaluation five experiments are conducted to evaluate the generated responses from conceptflow and the effectiveness of the learned graph attention. ## conclusion and future work conceptflow models conversation structure explicitly as transitions in the latent concept space, in order to generate more informative and meaningful responses. our experiments on reddit conversations illustrate the advantages of conceptflow over previous conversational systems. our studies confirm that conceptflow's advantages come from the high coverage latent concept flow, as well as its graph attention mechanism that effectively guides the flow to highly related concepts. our human evaluation demonstrates that conceptflow generates more appropriate and informative responses while using much fewer parameters. in future, we plan to explore how to combine knowledge with pre-trained language models, e.g. gpt-2, and how to effectively and efficiently introduce more concepts in generation models. this part presents more evaluation results of the overall performance of conceptflow from two aspects: relevance and novelty. more results for model novelty evaluation are shown in these supplementary results further confirm the effectiveness of conceptflow. our model has the ability to generate the most relevant response and more informative response than other models.