CADGE : Context-Aware Dialogue Generation Enhanced with Graph-Structured Knowledge Aggregation (2024)

Abstract: Commonsense knowledge is crucial to many natural language processing tasks. Existing works usually incorporate graph knowledge with conventional graph neural networks (GNNs), resulting in a sequential pipeline that compartmentalizes the encoding processes for textual and graph-based knowledge. This compartmentalization does, however, not fully exploit the contextual interplay between these two types of input knowledge. In this paper, a novel context-aware graph-attention model (Context-aware GAT) is proposed, designed to effectively assimilate global features from relevant knowledge graphs through a context-enhanced knowledge aggregation mechanism. Specifically, the proposed framework employs an innovative approach to representation learning that harmonizes heterogeneous features by amalgamating flattened graph knowledge with text data. The hierarchical application of graph knowledge aggregation within connected subgraphs, complemented by contextual information, to bolster the generation of commonsense-driven dialogues is analyzed. Empirical results demonstrate that our framework outperforms conventional GNN-based language models in terms of performance. Both, automated and human evaluations affirm the significant performance enhancements achieved by our proposed model over the concept flow baseline.

## introduction open-domain dialogue generation has gained considerable traction in the field of natural language generation @xcite . this task aims to develop chatbots with the capacity to engage in conversations across a broad spectrum of topics, thereby enabling a multitude of practical applications, including virtual assistants and wellbeing support systems @xcite * corresponding author. figure 1 : illustration of the proposed model with an example. the retrieved facts are fed to the graph model, then the model learns the representations of concepts by aggregating the knowledge layer by layer. finally, responses are generated with these aggregated features. @xcite @xcite . in recent years, there has been a surge of interest in leveraging large language models for dialogue generation @xcite @xcite @xcite . these models, in general, exhibit an enhanced capacity to encapsulate knowledge within their networks as their model sizes increase. however, it is crucial to acknowledge a series of studies that have underscored the limitations of training on plain text corpora, where the knowledge structure is not explicitly represented during the learning process @xcite . consequently, a key research question concerns how to better exploit and use external knowledge to improve the quality of generated responses, which has received increasing attention in recent research @xcite @xcite . the knowledge incorporated into chatbots can be broadly divided into structured and unstructured forms. prior work @xcite @xcite has achieved successful integration of unstructured knowledge (such as free-text content from web pages and knowledge statements) into the generated responses of chatbots. this typically involves encoding the most appropriate retrieved facts together with the conversation context or encoding multiple pieces of facts into a uniform representation before passing it to the decoder alongside the conversation history. structured knowledge, on the other hand, usually takes the form of a graph. a range of neural models @xcite @xcite have been introduced to incorporate features from the retrieved graphstructured knowledge. for instance, the graph attention mechanism @xcite @xcite has been widely used to embed knowledge graph features, and has been successful in aggregating sparse features into rich representations. with regard to language models, the rise of pre-trained models @xcite @xcite has also substantially advanced the state-of-the-art (sota) in open-domain dialogue generation. however, existing dialogue systems still face a number of challenges to effectively exploit commonsense knowledge @xcite . since graph-structured knowledge and natural utterances have different representations, most prior work @xcite @xcite employed separate encoders to incorporate and leverage these heterogeneous features by concatenating their respective numeric vectors. however, since the separate encoders do not share lowdimensional representations, they may fail to fully account for the semantics of context contributed by given posts with additional external knowledge facts. in addition, existing frameworks directly conduct graph-attention-based encoding on retrieved facts from the knowledge base, which are isolated in separate sub-graphs. this strategy does not capture dependencies between sub-graphs nor between the graph knowledge and the context of the post, in turn making it hard for neural networks to fully capture the overall backgrounds from the inputs. to address the aforementioned challenges, this paper proposes a novel graph-based framework to leverage knowledge contained in concept-related facts. in contrast to employing separate encoders to encode knowledge in the form of disparate knowledge graphs and text, we first transform the graphstructured representations into plain text, and leverage a pre-trained language model, unilm @xcite , to generate unified features for all inputs. subsequently, to overcome inadequacies when capturing the context semantics provided by the given posts and retrieved knowledge facts, a novel, context-aware graph-based mechanism (context-aware gat) is proposed to incorporate the features from the post and the knowledge graph in the same learning process during hierarchical aggregation. the graph knowledge takes two steps (layers) before being aggregated into a condensed feature vector as the global features of given inputs. for each layer, the context embedding and the factual embedding are concatenated, and then graph attentions are computed for every sub-node. finally, all representations are aggregated into the root node and fed to the decoder for response generation. this whole process is illustrated in figure 1 . we also note that our model can be easily extended to incorporate multi-hop knowledge. experimental results show that our extended model can use multi-hop knowledge to further increase the informativeness of generated responses, and consequently yields considerable improvements over other dialogue systems that use multi-hop knowledge. the contributions of this work are summarised three-fold: • we propose a novel framework foot_0 , which is a successful exploration that leverages a unified language model for the heterogeneous inputs of graph knowledge and text, exploiting structured knowledge with context-aware subgraph aggregation to generate informative responses. • we conduct a range of experiments, and the extensive automatic and human evaluation results demonstrate our model significantly outperforms existing baselines to generate a more appropriate and informative response with external graph knowledge. • with extensive experiments, we investigate the advances and mechanisms of leveraging graph knowledge with our context-aware gat model. we also investigate the expansion of our model to accommodate multi-hop knowledge, and validate its effectiveness. large language models such as unilm @xcite , gpt-2 @xcite , and bart @xcite are widely used in open domain dialogue generation systems @xcite . dialogpt @xcite was pretrained on a dialogue dataset containing 147m conversations and is based on the autoregressive gpt-2 model, using a maximum mutual information (mmi) scoring function to address the low amount of information in the generated text. @xcite built a 2.6b-parameter evolved transformer architecture to model the relation between context-response pairs. to generate more informative responses, @xcite ) use latent variables to model one-to-many relationships in context-response pairs. finally, @xcite use a retrieval model to retrieve candidate responses and then concatenates them to represent the context before inputting them into the transformer to generate the model. please refer to appendix a for more details of related work. ## methodology we formulate our task as follows: the given inputs include a post @xmath0, x 2 , ..., x n } and a graph knowledge base @xmath1, τ 2 , ..., τ k }, in which a fact is represented in the form of a triplet {h, r, t} where h, r, and t denote the head node, the relation, through feature forwarding, the root nodes of each graph attentively read and aggregate all knowledge and become higher-level representations (from yellow to green, and then green to red). and the tail node, respectively. the goal is to generate a response @xmath2, y 2 , ..., y m } by modeling the conditional probability distribution p (y |x, g). figure 2 gives an overview of our framework. the knowledge retrieval process is fundamentally implemented by word matching (concepts in concept-net are formatted in one-word) and rule filtering to collect knowledge triples (for more details please refer to @xcite ). ## conclusion in this paper, we proposed a novel knowledge aggregation framework for the knowledge graph enhanced dialogue generation task. this framework implements a context-aware gat which applies representation learning of the heterogeneous features from graph knowledge text, and the neural networks effectively learn to incorporate globally aggregated features to enhance response generation with rich representations. extensive experiments are conducted to demonstrate that our framework outperforms sota baselines on both automatic and human evaluation, as the proposed context-aware gat largely improved the semantic understanding of both graph and text knowledge to enhance the appropriateness and informativeness of generated responses. the expansion of context-aware gat to two-hop knowledge also indicates the robustness and effectiveness of our framework in increasing the amount of grounded graph knowledge in responses. we hope that our proposed framework can benefit research in all text generation tasks where knowledge graphs are incorporated, and transferable research will be continued in further work.