Benchmarking Commonsense Knowledge Base Population with an Effective Evaluation Dataset (2021)

Abstract: Reasoning over commonsense knowledge bases (CSKB) whose elements are in the form of free-text is an important yet hard task in NLP. While CSKB completion only fills the missing links within the domain of the CSKB, CSKB population is alternatively proposed with the goal of reasoning unseen assertions from external resources. In this task, CSKBs are grounded to a large-scale eventuality (activity, state, and event) graph to discriminate whether novel triples from the eventuality graph are plausible or not. However, existing evaluations on the population task are either not accurate (automatic evaluation with randomly sampled negative examples) or of small scale (human annotation). In this paper, we benchmark the CSKB population task with a new large-scale dataset by first aligning four popular CSKBs, and then presenting a high-quality human-annotated evaluation set to probe neural models’ commonsense reasoning ability. We also propose a novel inductive commonsense reasoning model that reasons over graphs. Experimental results show that generalizing commonsense reasoning on unseen assertions is inherently a hard task. Models achieving high accuracy during training perform poorly on the evaluation set, with a large gap between human performance. We will make the data publicly available for future contributions. Codes and data are available at https://github.com/HKUST-KnowComp/CSKB-Population .

## introduction commonsense reasoning is one of the core problems in the field of artificial intelligence. throughout the development in computational commonsense, commonsense knowledge bases (cskb) @xcite are constructed to enhance models' reasoning ability. as humanannotated cskbs are far from complete due to * equal contribution the scale of crowd-sourcing, reasoning tasks such as cskb completion @xcite @xcite and population @xcite are proposed to enrich the missing facts. the cskb completion task is defined based on the setting of predicting missing links within the cskb. on the other hand, the population task grounds commonsense knowledge in cskbs to large-scale automatically extracted candidates, and requires models to determine whether a candidate triple, (head, relation, tail), is plausible or not, based on the information from both the cskb and the large number of candidates which essentially form a large-scale graph structure. an illustration of the difference between completion and population is shown in figure 1 . there are two advantages of the population task. first, the population can not only add links but also nodes to an existing cskb, while completion can only add links. the populated cskb can also help reduce the selection bias problem (heckman, 1979) from which most machine learning models would suffer, and will benefit a lot of downstream applications such as commonsense generation @xcite . second, commonsense knowledge is usually implicit knowledge that requires multiple-hop reasoning, while current cskbs are lacking such complex graph structures. for example, in atomic @xcite , a humanannotated if-then commonsense knowledge base among daily events and (mental) states, the average hops between matched heads and tails in aser, an automatically extracted knowledge base among activities, states, and events based on discourse relationships, is 2.4 @xcite . evidence in section 4.5 ( however, there are two major limitations for the evaluation of the cskb population task. first, automatic evaluation metrics, which are based on distinguishing ground truth annotations from automatically sampled negative examples (either a random head or a random tail), are not accurate enough. instead of directly treating the random samples as negative, solid human annotations are needed to provide hard labels for commonsense triples. second, the human evaluation in the original paper of cskb population @xcite cannot be generally used for benchmarking. they first populate the cskb and then asked human annotators to annotate a small subset to check whether the populated results are accurate or not. a better benchmark should be based on random samples from all candidates and the scale should be large enough to cover diverse events and states. to effectively and accurately evaluate cskb population, in this paper, we benchmark cskb population by firstly proposing a comprehensive dataset aligning four popular cskbs and a largescale automatically extracted knowledge graph, and then providing a large-scale human-annotated evaluation set. four event-centered cskbs that cover daily events, namly conceptnet @xcite (the event-related relations are selected), atomic @xcite , atomic 20 20 @xcite , and glucose @xcite , are used to constitute the commonsense relations. we align the cskbs together into the same format and ground them to a large-scale eventuality (including activity, state, and event) knowledge graph, aser @xcite . then, instead of annotating every possible node pair in the graph, which takes an infeasible o(|v | 2 ) amount of annotation, we sample a large subset of candidate edges grounded in aser to annotate. in total, 31.7k high-quality triples are annotated as the development set and test set. to evaluate the commonsense reasoning ability of machine learning models based on our benchmark data, we first propose some models that learn to perform cskb population inductively over the knowledge graph. then we conduct extensive evaluations and analysis of the results to demonstrate that cskb population is a hard task where models perform poorly on our evaluation set far below human performance. we summarize the contributions of the paper as follow: (1) we provide a novel benchmark for cskb population over new assertions that cover four human-annotated cskbs, with a large-scale human-annotated evaluation set. (2) we propose a novel inductive commonsense reasoning model that incorporates both semantics and graph structure. ## task definition denote the source cskb about events as @xmath0, r, t)|@xmath1, @xmath2, @xmath3, where h, r, and t are the set of the commonsense heads, relations, and tails. suppose we have another much larger eventuality (including activity, state, and event) knowledge graph extracted from texts, denoted as @xmath4, e), where v is the set of all vertices and e is the set of edges. g c is the graph acquired by aligning c and g into the same format. the goal of cskb population is to learn a scoring function given a candidate triple (h, r, t), where plausible commonsense triples should be scored higher. the training of cskb population can inherit the setting of triple classification, where ground truth examples are from the cskb c and negative triples are randomly sampled. in the evaluation phase, the model is required to score the triples from g that are not included in c and be compared with humanannotated labels. ## experiments in this section, we introduce the baselines and our proposed model kg-bertsage for the cskb population task, as well as the experimental setups. ## error analysis as defined in section 4.6, the evaluation set is composed of three parts, edges coming from the original test set (original test set), edges where heads come from cskbs and tails from aser (cskb head + aser tail), and edges from the whole aser graph (aser edges). the break-down auc scores of different groups given all models are shown in ## conclusion in this paper, we benchmark the cskb population task by proposing a dataset by aligning four popular cskbs and an eventuality graph aser, and provide a high-quality human-annotated evaluation set to test models' reasoning ability. we also propose kg-bertsage to both incorporate the semantic of knowledge triples and the subgraph structure to conduct reasoning, which achieves the best performance among other counterparts. experimental results also show that the task of reasoning unseen triples outside of the domain of cskb is a hard task where current models are far away from human performance, which brings challenges to the community for future research.