Commonsense Knowledge Graph Completion Via Contrastive Pretraining and Node Clustering (2023)

Abstract: The nodes in the commonsense knowledge graph (CSKG) are normally represented by free-form short text (e.g., word or phrase). Different nodes may represent the same concept. This leads to the problems of edge sparsity and node redundancy, which challenges CSKG representation and completion. On the one hand, edge sparsity limits the performance of graph representation learning; On the other hand, node redundancy makes different nodes corresponding to the same concept have inconsistent relations with other nodes. To address the two problems, we propose a new CSKG completion framework based on Contrastive Pretraining and Node Clustering (CPNC). Contrastive Pretraining constructs positive and negative head-tail node pairs on CSKG and utilizes contrastive learning to obtain better semantic node representation. Node Clustering aggregates nodes with the same concept into a latent concept, assisting the task of CSKG completion. We evaluate our CPNC approach on two CSKG completion benchmarks (CN-100K and ATOMIC), where CPNC outperforms the state-of-the-art methods. Extensive experiments demonstrate that both Contrastive Pretraining and Node Clustering can significantly improve the performance of CSKG completion. The source code of CPNC is publicly available on https://github.com/NUSTM/CPNC .

## introduction commonsense knowledge graphs (cskg) have been widely used to build commonsense-grounded ai applications, such as question answering @xcite , visual question answering @xcite , sentiment analysis @xcite , dialogue system @xcite , etc. commonsense knowledge graphs such as conceptnet @xcite and atomic @xcite provide a structured way of representing a commonsense concept, which consists of a head node, a tail node, and the relation edge. nodes in commonsense knowledge graphs are typically represented by free-form short text (word or phrase), resulting in many different nodes representing the same concept. figure 1 shows a subgraph of conceptnet, where nodes in the same color indicate the same concept. e.g., "have breakfast", "take breakfast" and "eat breakfast" all express the concept of "eat breakfast". this problem also results in a large number of missing edges between nodes, as illustrated by the dashed lines in figure 1 . e.g., "eat breakfast" and "prepare food" have a "hasprerequisite" relation, but such relation is missing between "eat breakfast" and "cook food". on the one hand, node redundancy in cskg makes different nodes of the same concept have inconsistent relations with other nodes. in cskg representation and completion, it would be beneficial to make use of the latent concept information behind different nodes to help learn more semantic-general representations. however, this intuition was ignored by most of the existing work in cskg completion. on the other hand, as analyzed by @xcite , edge sparsity in cskg limits information propagation in graph neural networks. upon graph neural networks, researchers further incorporated pre-trained language model such as bert to enhance the semantic representation of nodes @xcite @xcite . however, fine-tuning bert is still imperfect in representing the commonsense knowledge graph, which is made up of linked nodes represented in a free-form short text. to tackle the two issues, we propose a new cskg completion framework based on the encoder-decoder architecture, which contains two core modules contrastive pretraining and node clustering (cpnc). contrastive pretraining is to alleviate the difficulty of node representation learning induced by edge sparsity. through contrastive learning on positive and negative head-tail node pairs, the embedding distance between related nodes becomes closer, and that between unrelated nodes becomes farther so as to learn better node representations. node clustering aims to address the edge inconsistency issue caused by node redundancy. we cluster nodes with close semantic representations and take the mean vector as the latent concept representation for nodes in this cluster. to assist cskg completion, the latent concept representation is fused with the node representation. we evaluate our cpnc framework on two cskg completion benchmarks, i.e., cn-100k and atomic. the results show that our model outperforms the state-of-the-art models for this task significantly. ablation studies demonstrate that both contrastive pretraining and node clustering modules can significantly improve the performance of cskg completion. further experiments verify that our model can consistently improve as the sparsity of knowledge graphs increases: the higher the sparsity, the more improvement our model achieves. ## related work we briefly review traditional knowledge graph completion, and then pay more attention to commonsense knowledge graph completion. ## task definition given a cskg @xmath0, v, r) where n is the set of nodes, v is the set of edges and r is the set of relations. we regard each tuple v @xmath1, rel, t) in the cskg as a sample, which is composed of head node h, tail node t and relation rel, where h, @xmath2, v @xmath3, and @xmath4. given a query (h, rel) formed by a head node h and a relation rel, the target of the cskg completion is to maximize the score of the tail node t. following the previous work @xcite , for an edge (h, rel, t) existing in the cskg, we also add an inverse edge (t, rel -1 , h) to the graph, where rel -1 is the inverse relation of rel. ## approach in this paper, we propose a new framework, cpnc, for cskg completion using an encoder-decoder architecture (see figure 2 ). the encoder incorporates semantic, graph structure, and latent concept representation obtained from contrastive pretraining, graph convolutional network (gcn), and node clustering, respectively, to acquire node representations. given query (h, rel), the decoder ranks the nodes in the candidate set n and finds tail nodes for the query by the rank. ## conclusion in this work, we propose a new cskg completion framework cpnc to address issues arising from node redundancy and edge sparsity. cpnc obtains better semantic node information through contrastive pretraining, which alleviates the problems caused by edge sparsity. cpnc also utilizes the latent concept representation acquired through node clustering to alleviate the problem caused by node redundancy. on cn-100k and atomic, experimental results and extensive analysis demonstrate the effectiveness of contrastive pretraining and node clustering.