MICO : A Multi-alternative Contrastive Learning Framework for Commonsense Knowledge Representation (2022)

Abstract: Commonsense reasoning tasks such as commonsense knowledge graph completion and commonsense question answering require powerful representation learning. In this paper, we propose to learn commonsense knowledge representation by MICO, a Multi-alternative contrastIve learning framework on COmmonsense knowledge graphs (MICO). MICO generates the commonsense knowledge representation by contextual interaction between entity nodes and relations with multi-alternative contrastive learning. In MICO, the head and tail entities in an (h,r,t) knowledge triple are converted to two relation-aware sequence pairs (a premise and an alternative) in the form of natural language. Semantic representations generated by MICO can benefit the following two tasks by simply comparing the similarity score between the representations: 1) zero-shot commonsense question answering tasks; 2) inductive commonsense knowledge graph completion tasks. Extensive experiments show the effectiveness of our method.

## introduction commonsense reasoning is a fundamental problem in artificial intelligence. recently in the nlp field, much attention has been paid to commonsense reasoning in the following two aspects. first, more commonsense knowledge graphs (ckgs) @xcite were developed to support new types of reasoning tasks, such as commonsense knowledge graph completion (ckgc) @xcite . another way to evaluate machine learning models' commonsense reasoning capabilities is using commonsense question answering (cqa) tasks @xcite @xcite . existing approaches to deal with the above problems commonly involve fine-tuning large pre-trained language models, such as bert @xcite , roberta @xcite , and gpt2 @xcite , by either incorporating the entire knowledge base for ckgc @xcite or injecting the knowledge base to provide background knowledge for zero-shot cqa @xcite @xcite . in fact, both ckgc and zero-shot cqa can be formulated in a unified way, where a question can be constructed based on the head entity and relation in a knowledge graph, and then finding the tail entity, which is regarded as an answer, based on the constructed question. in this way, incorporating the entire knowledge base for ckgc and injecting the kg in pre-trained lms for zero-shot cqa can be unified as a semantic matching problem, where a powerful representation learning for the matching becomes the most important problem. this also means that, after we unify them for ckgc and cqa in the same way, we can perform zero-shot cqa by simply leveraging the model finetuned on the entire ckgs for ckgc. existing commonsense-related representation learning usually leverage a ckg embedding framework @xcite , or fine-tuning a generative language model @xcite . however, they were not aware of the challenges that a typical ckg brings. first, in a typical ckg, such as conceptnet @xcite and atomic @xcite , nodes are loosely structured free-from texts, which means that previous embedding based on negative sampling cannot substantially support sufficient training because of sparsity. on the other hand, a generative model can only take positive examples for training so the capability of determining the negative answers is limited. in this paper, we propose a new framework called mico, a multi-alternative contrastive learning framework for commonsense knowledge representation. the representations can benefit across tasks by easily calculating the semantic distances personx leaves personx's book personx leaves persony's book forgetful xattr xattr to find the book xwant to go back and get the book xwant easily distracted xattr personx ask persony if had seen to look for someone or something xneed xwant to keep looking xwant to find his dog context: jordan left their book in the library after studying all day. question: what will jordan want to do next? answer a: ask people if they've seen the book answer b: go home answer c: go the library and hide the book figure 1 : an example from a cqa task (siqa @xcite ) and related knowledge in a ckg (atomic). with unified vector representations. in this way, though many distinct nodes may have similar concepts @xcite , they are still close in the semantic space. figure 1 shows an example of this advantage on cqa. the entity node personx ask if persony had seen related to the right answer is not directly connected to the nodes personx leaves per-sonx's book or personx leaves persony's book but these nodes share similar tail entity nodes. therefore, the right answer can be found by semantic matching as it is close to the given context and question in the semantic space. to unify the form of ckgc and cqa, we follow the idea in copa @xcite where commonsense causal reasoning can be evaluated by selecting the most plausible alternatives given the premise. we first converts the knowledge triplets (h, r, t) into sequence pairs (p, a) (p for premise and a for alternative). mico then encodes the sequence pairs into embeddings and measures their distance by a similarity function as we assume the representations of related knowledge lie close in the embedding space. furthermore, we enhance the representation learning by a contrastive loss with sufficient sampling over the sparse ckg. the alternative from the same triplet is a positive sample to the premise under the contrastive learning framework. alternatives from other knowledge triples with different premises are negative samples. mico also takes consideration of the structure from ckgs, where one head node h may connect to several tail nodes t under the same relation r. mico dynamically selects a hard alternative from multi-alternatives for a premise during training. experiments on two typical commonsense knowledge graphs and two types of tasks, zero-shot cqa and inductive ckgc, demonstrate the effectiveness of our methodology. our code is openresourced. foot_0 2 related work ## methodology in this section, we introduce the terminologies and algorithms, and show the framework in figure 2 . ## experiments in this section, we first introduce the ckgs used as knowledge sources, and then two kinds of evaluation tasks (zero-shot cqa and inductive ckgc). finally, we introduce baseline methods for the two tasks separately. ## discussion this section shows that transformers from mico can construct commonsense representations for ckg and benefit commonsense knowledge retrieval given queries. one example from siqa and retrieved possible alternatives from atomic ckg sampling copa siqa csqa conceptnet random 64.0 38.3 41.3 mico 65.2 39.1 42.9 atomic random 71.1 49.2 40.3 mico 71.3 48.9 40.7 however, the representation generated from mico still has some drawbacks as shown in the results. "jordan look for it at library" would be a reasonable node instead of "jordan look for it at home". this shows that the representation still need future work to distinguish the detailed concepts. ## conclusion in this paper, we propose a mico, a multialternative contrastive learning framework over commonsense knowledge graphs to learn com-monsense knowledge representation. the framework converts the knowledge triplets into sequence pairs and learns superior knowledge representation through contrastive learning techniques. the generated representations perform well over zero-shot cqa tasks and inductive ckgc tasks. furthermore, for cqa tasks, the related knowledge can be provided by simply retrieving the commonsense knowledge representations of ckgs. relation rel template xattr personx is seen as xeffect as a result, personx will xwant as a result, personx wants xneed but before, personx needed xreact as a result, personx feels xintent because personx wanted oeffect as a result, persony or others will oreact as a result, persony or others feel owant as a result, persony or others want