Knowledge Graph Compression Enhances Diverse Commonsense Generation (2023)

Abstract: Generating commonsense explanations requires reasoning about commonsense knowledge beyond what is explicitly mentioned in the context. Existing models use commonsense knowledge graphs such as ConceptNet to extract a subgraph of relevant knowledge pertaining to concepts in the input. However, due to the large coverage and, consequently, vast scale of ConceptNet, the extracted subgraphs may contain loosely related, redundant and irrelevant information, which can introduce noise into the model. We propose to address this by applying a differentiable graph compression algorithm that focuses on the relevant knowledge for the task. The compressed subgraphs yield considerably more diverse outputs when incorporated into models for the tasks of generating commonsense and abductive explanations. Moreover, our model achieves better quality-diversity tradeoff than a large language model with 100 times the number of parameters. Our generic approach can be applied to additional NLP tasks that can benefit from incorporating external knowledge.

## introduction commonsense knowledge graphs (cskgs) have been used to improve the performance of downstream applications such as question answering @xcite and dialogue @xcite , as well as for enhancing neural models for commonsense reasoning tasks @xcite . typically, these methods extract keywords from the input and construct a subgraph around them using the kg knowledge, which is then incorporated into the model. recent popular cskgs such as conceptnet @xcite and atomic @xcite represent nodes in natural language, which allows flexibility but also adds redundancy and noise (wu 1 code is available at: https://github.com/eujhwang/kg-compression figure 1 : an example from comve @xcite . the subgraph obtained for the input sentence includes unimportant information (in red) that can lead to noisy outputs. @xcite . moreover, the retrieved subgraphs around a task's concepts potentially include information that is not relevant to the context. for example, in figure 1 , the goal is to generate a reason why the input sentence ("a shark interviews a fish") defies commonsense. the concepts tank and business are semantically irrelevant to either the input or the reference output sentences. including irrelevant information introduces noise that can deteriorate the model's performance. recent work has addressed this by pruning noisy paths based on low edge confidence scores in knowledge base embeddings @xcite or by using language models (lms) @xcite . yet, the relevance of paths is not determined in relation to the given task. in this paper, we propose to use differentiable graph compression that enables the model to learn how to select the crucial concepts that are actually related to the task. our method contains two main components: using self-attention scores to select relevant concept nodes in the retrieved subgraph, and employing optimal transport loss to ensure the chosen concepts preserve the most crucial information of the original graph. in this way, the irrelevant or redundant concepts can be automatically eliminated in the subgraph. we demonstrate the usefulness of our method on two commonsense generation tasks: commonsense explanation generation and abductive commonsense reasoning. our method outperforms a range of baselines that use kgs in terms of both diversity and quality of the generations. we further conduct a comprehensive analysis, exploring a different setup, such as the scenario of incorporating new knowledge into the subgraph. different from the baselines, our method enables the model to maintain performance, even in the presence of potentially increased noisy data. finally, we show that our approach demonstrates better quality-diversity tradeoff than the large language model vicuna-13b, which has 100 times more parameters. to reduce such noise, prior methods have proposed to score and prune the paths. @xcite used transe @xcite to score each edge in the path, while @xcite scores nodes based on the likelihood of a pre-trained lm to generate it after the input. in both methods, the scores are not trained to represent a node's importance in relation to the task. ## method our goal is to generate diverse sentences, {y 1 , y 2 , ..., y k } that explain a given instance x (see sec 2 for the specific task descriptions). the objective is to maximize the probability of generating each y i : p (y i |x), as well as to diversify them. previous kg-enhanced approaches usually add an external graph g x to make the generation also conditioned on the graph: p (y i |x, g x ). however, as we discussed in sec 1, g x often contains redundancy or noise. for example, given a target concept a, there is a semantically similar concept (e.g. a synonym) a ′ and a noisy concept b in the graph g x ). obviously, a ′ will negatively impact the diversity of generations because the model may select both a and a ′ for generation and the semantics of the generations are similar; concept b will hurt the generation quality since it is irrelevant to the context. so, a natural idea to solve the problem is to eliminate these concepts by compressing the graph. our method extends mokge @xcite ) by compressing the retrieved external knowledge graph. the framework is illustrated in figure 2 and described in detail subsequently. in a nutshell, it aims to identify the concepts within the kg that provide the most relevant knowledge for a particular instance. we first extract a subgraph from the kg based on the given input sentence, and encode it into a vector representation (sec 3.1). then, we learn a compressed graph that maintains only the most relevant concepts for the given instance (sec 3.2). we train the model with the corresponding losses (sec 3.3) and finally apply moe to generate diverse outputs (sec 3.4). ## results and discussion comparison to baselines, we additionally confirm that our optimal transport loss helps the model to keep the kg subgraph more coherently; see especially the α-nlg results. generation examples, figure 4 . observe that mokge tends to generate sentences with simpler structure and fewer concepts, whereas our model employs a broader range of kg concepts. this makes the generations effectively more similar to the human-written ones, where each of the three sentences addresses a different context. we show more examples in appendix b. testing robustness with potentially more redundancy and noise, comparison with large language model, ## analysis compression ratios, figure 3 . this hyperparameter determines the amount of concept nodes to be kept in the compressed subgraph. maintaining 65% of the nodes in the subgraph yields the optimal performance in terms of both diversity and quality on both datasets (see appendix c for comve dataset). interestingly, we do not observe a great negative impact on performance, even up to a ratio of 30%. this shows that conceptnet apparently contains much information that is not necessarily beneficial for diverse generations in the context of a particular task and hence justifies our proposal. unique concepts in the output, appendix d. the comparison of mokge and mokge+sag+ot shows that mokge tends to generate more sentences containing 0 human evaluation, ## conclusion we present a differentiable graph compression algorithm that enables the model to focus on crucial information. through experiments on two commonsense explanation generation tasks, we show that our approach not only improves the diversity but also maintains the quality of outputs. moreover, our graph compression helps the model regain performance when new and potentially noisy information is added to graphs. our work opens up future research in effectively selecting and incorporating symbolic knowledge into nlp models.