Structured Self-Supervised Pretraining for Commonsense Knowledge Graph Completion (2021)

Abstract: To develop commonsense-grounded NLP applications, a comprehensive and accurate commonsense knowledge graph (CKG) is needed. It is time-consuming to manually construct CKGs and many research efforts have been devoted to the automatic construction of CKGs. Previous approaches focus on generating concepts that have direct and obvious relationships with existing concepts and lack an capability to generate unobvious concepts. In this work, we aim to bridge this gap. We propose a general graph-to-paths pretraining framework that leverages high-order structures in CKGs to capture high-order relationships between concepts. We instantiate this general framework to four special cases: long path, path-to-path, router, and graph-node-path. Experiments on two datasets demonstrate the effectiveness of our methods. The code will be released via the public GitHub repository.

## introduction commonsense knowledge has been widely used to boost many nlp applications, such as dialog generation @xcite , question answering @xcite , story generation @xcite , and so forth. to ground an application with commonsense, one needs to access a commonsense knowledge graph (ckg) where nodes represent concepts such as ''maintain muscle strength'', ''exercise regularly'', and edges represent the relationships between concepts such as ''maintain muscle strength'' has a prerequisite of ''exercise regularly''. commonsense knowledge involves almost all concepts in a human's daily life. these concepts have very rich and diverse relationships. as a result, it is extremely challenging, if not impossible, for humans to list all commonsense concepts and relationships exhaustively. to address this is-sue, many efforts @xcite have been devoted to automatically constructing ckgs. a commonly used approach is: given a head concept and a relation, train a generative model to generate the tail concept. typically, the generative model consists of an encoder that encodes the concatenation of head concept and relation, and a decoder that takes the embedding generated by the encoder as input and decodes a tail concept. while simple, these approaches treat a ckg as a collection of individual concept-relation-concept triples without considering the rich structures in a ckg. as a result, although existing methods can generate new concepts that have direct and obvious relationships with existing concepts in a ckg, they are lacking in generating concepts that are indirectly and unobviously related to existing concepts. this problem can be potentially addressed by exploiting the rich structures in a ckg to perform high-order reasoning that helps to generate unobvious concepts. figure 1 shows an example. previous approaches are able to generate obvious concepts/relations such as generating ''hiking @xcite approaches are unlikely to generate a tail concept of ''sleep'' given a head concept ''hiking'' and a relation ''leads to''. one potential solution to address this problem is leveraging high-order structure: because fatigue is caused by hiking and fatigue makes people want to sleep, there should be a ''leads to'' relationship from hiking to sleep. to this end, we propose to leverage the rich structure in a ckg to pretrain encoder-decoder models for generating more accurate and diverse commonsense knowledge. we propose a general graph-to-paths structured pretraining framework. to construct a pretraining example, we randomly sample a sub-graph from a ckg and use it as input. figure 1 : illustration of leveraging high-order structure to generate unobvious concepts. because the relationship between ''hiking'' and ''sleep'' is not obvious, previous approaches are unlikely to generate a tail concept of ''sleep'' given a head concept ''hiking'' and a relation ''leads to''. our method addresses this problem by leveraging high-order structure: because fatigue is caused by hiking and fatigue makes people want to sleep, there should be a ''leads to'' relationship from hiking to sleep. then we randomly sample one or more paths from the ckg and use them as outputs. part of the nodes in output paths are overlapped with those in the input sub-graph. we pretrain an encoder and a decoder by mapping input sub-graphs to output paths. specifically, an input sub-graph is fed into the encoder, which yields an encoding. then the encoding is fed into the decoder which generates output paths. we instantiate the general graph-to-paths framework to four concrete cases, each capturing a special type of structured information. specifically, we consider four types of structures: (1) concepts have long-range relations; (2) multiple paths exist between a pair of source and target concept; the major contributions of this paper include: ## methods in this section, we first propose a general graph-topaths pretraining framework, then instantiate this general framework to four specific cases. given a commonsense knowledge graph, we automatically construct a pretraining dataset that captures high-order structured information of commonsense knowledge. each pretraining example consists of an input sub-graph and one or more paths. the input sub-graph can have arbitrary graph structures. an output path is a special type of subgraph whose structure is required to be a directed chain, that is, each node (except the first and last one) has one inbound edge and one outbound edge. the input sub-graph and output paths are required to have some overlapping nodes so that the input and output are related. given these training examples, we pretrain an encoder and a decoder by mapping input sub-graphs to output paths. specifically, given an input sub-graph, we use the encoder to encode it; then the encoding is fed into the decoder to generate output paths. afterwards, the pretrained encoder and decoder are finetuned for commonsense knowledge generation. for the outputs, we choose to use paths instead of arbitrarily structured sub-graphs because decoding a path is much easier than decoding a graph and can be readily done by many models such as gpt2, bart, etc. next, we discuss how to capture some specific types of structures by instantiating the general graph-to-paths pretraining framework to specific cases. ## experiments in this section, we present experimental results. ## conclusions and future work in this paper, we study the automatic construction of commonsense knowledge graphs (ckgs), for the sake of facilitating commonsense-grounded nlp applications. to address the limitation of previous approaches which cannot generate unobvious concepts, we leverage the rich structure in ckgs to pretrain commonsense generation models. we propose a general graph-to-paths pretraining framework which pretrains an encoder and a decoder by mapping input sub-graphs to output paths. we instantiate this general framework to four special cases, for capturing four types of structures: (1) individual long paths; (2) pairs of paths that share the same source and target concept; (3) multi-connectivity of each concept with other concepts; and (4) local graphs at each concept. the corresponding four cases are: (1) pretraining on individual long paths; for future work, we will develop a graph-tograph model which takes an existing commonsense subgraph as input and generates a larger graph containing novel concepts and relations. when generating the target graph, concepts and relations are generated simultaneously. in addition, we will incorporate external unstructured texts which contain implicit commonsense knowledge to generate ckgs.