Learning from Missing Relations: Contrastive Learning with Commonsense Knowledge Graphs for Commonsense Inference (2022)

Abstract: Commonsense inference poses a unique challenge to reason and generate the physical, social, and causal conditions of a given event. Existing approaches to commonsense inference utilize commonsense transformers, which are large-scale language models that learn commonsense knowledge graphs. However, they suffer from a lack of coverage and expressive diversity of the graphs, resulting in a degradation of the representation quality. In this paper, we focus on addressing missing relations in commonsense knowledge graphs, and propose a novel contrastive learning framework called SOLAR. Our framework contrasts sets of semantically similar and dissimilar events, learning richer inferential knowledge compared to existing approaches. Empirical results demonstrate the efficacy of SOLAR in commonsense inference of diverse commonsense knowledge graphs. Specifically, SOLAR outperforms the state-of-the-art commonsense transformer on commonsense inference with ConceptNet by 1.84% on average among 8 automatic evaluation metrics. In-depth analysis of SOLAR sheds light on the effects of the missing relations utilized in learning commonsense knowledge graphs.

## introduction commonsense inference, reasoning of unobserved conditions from an observed event, is an important but challenging task in natural language processing (nlp) @xcite @xcite . this is easy for humans, but still out of the reach of current artificial intelligence systems. commonsense inference aims to generate textual descriptions of the inference results, which is more in line with the process of humans reasoning based on their knowledge. for a given event "x walks into a hospital", the causal conditions (e.g., what to do before and after the event), physical conditions (e.g., capability and location of entities), and social conditions (e.g., the intention and reaction of x) of the event are to be inferred. recent studies on commonsense inference have adopted commonsense transformers @xcite , which are large-scale language models trained on commonsense knowledge graphs (kgs) like atomic @xcite and conceptnet @xcite . such models are grounded on the hypothesis that language models can memorize facts in their parameters during training @xcite . it is observed that training language models on commonsense kgs allows them to express commonsense knowledge more accurately @xcite . despite these efforts, commonsense trans-former models still suffer from two main obstacles inherent in commonsense kgs: (1) lack of coverage and (2) expressive diversity of the graphs. first, commonsense kgs lack the coverage required to be applicable for diverse situations in the real world @xcite . in atomic, even with the possibility of far more commonsense properties being relevant, any single node has only 2.2 commonsense properties directly related on average @xcite . second, with the noncanonical and free-form text representation for the nodes in commonsense kgs, semantically identical or similar expressions of events are represented as distinct nodes @xcite . for example, "personx is fond of dogs" and "personx likes dogs" are semantically identical, but represented as distinct nodes. the expressive diversity makes commonsense kgs substantially sparser than conventional kgs. owing to the lack of coverage and expressive diversity, a significant amount of valid relations between nodes are missing in commonsense kgs. in this study, we focus on learning from missing relations in commonsense kgs for commonsense inference. our key observation is that semantically identical or similar events can have the same relations as shown in figure 1 . for example, "personx likes dogs" and "personx loves animals" are semantically similar to "personx loves dogs", and the inference that "personx wants to adopt one" can be drawn from any of those events. modeling such missing relations helps the model learn richer representations from commonsense kgs. current approaches for alleviating the sparsity of commonsense kgs, such as automatic commonsense kg completion @xcite @xcite , do not effectively address missing relations, because the completion models only learn existing relations as valid. 2 therefore, this problem remains unexplored. we propose a novel learning framework of commonsense transformers, called self-supervised contrastive learning with missing relations (so-lar), to address the aforementioned problem. our framework trains large-scale language models to learn both existing and missing relations with selfsupervised contrastive learning, distinguishing between the missing and valid relations as positive and the invalid relations as negative. specifically, we construct sets of examples including semantically similar events and their relation-object pairs based on the similarity of language representations (e.g., person likes dogs and personx loves animals). we then contrast each set of examples with the sets including dissimilar events and their relationobject pairs. our contrastive learning framework allows the model to identify the interrelationship between semantically similar events and their relationobject pairs, leading to a better understanding of missing relations in commonsense kgs than a data augmentation approach. we evaluate our framework for commonsense inference on three commonsense kgs: conceptnet @xcite , atomic @xcite , and atomic 20 20 @xcite . empirical results show that solar outperforms the stateof-the-art commonsense transformers on commonsense inference. in particular, for conceptnet, so-lar with bart-large @xcite outperforms comet @xcite with bartlarge by 1.84% on average among 8 automatic evaluation metrics. in addition, we observe that so-lar with bart-base produces comparable results to comet with bart-large, which validates that our framework is superior to existing approaches in terms of both effectiveness and efficiency. our main contributions are as follows: ## methodology in this section, we describe the model architecture and training procedure of the proposed framework. ## experiments in this section, we demonstrate the efficacy of our framework by comparing the commonsense infer-ence performances of solar with those of the state-of-the-art commonsense transformers. ## conclusion we have presented a novel contrastive learning framework of commonsense transformers, called solar, to effectively learn from missing relations in commonsense kgs. moreover, we have developed a new construction scheme for positive and negative sets of examples based on similarities in language model representations. by utilizing our carefully designed methods, solar effectively learns both existing and missing relations of events, alleviating the difficulties in learning commonsense kgs. our empirical evaluations of diverse commonsense kgs demonstrate the efficacy of solar in commonsense inference. in particular, solar consistently outperforms the state-of-the-art commonsense transformers across all the evaluation metrics and commonsense kgs.