Materialized Knowledge Bases from Commonsense Transformers (2022)

Abstract: Starting from the COMET methodology by Bosselut et al. (2019), generating commonsense knowledge directly from pre-trained language models has recently received significant attention. Surprisingly, up to now no materialized resource of commonsense knowledge generated this way is publicly available. This paper fills this gap, and uses the materialized resources to perform a detailed analysis of the potential of this approach in terms of precision and recall. Furthermore, we identify common problem cases, and outline use cases enabled by materialized resources. We posit that the availability of these resources is important for the advancement of the field, as it enables an off-the-shelf-use of the resulting knowledge, as well as further analyses on its strengths and weaknesses.

## introduction compiling comprehensive collections of commonsense knowledge (csk) is an old dream of ai. besides attempts at manual compilation @xcite @xcite ) and text extraction @xcite @xcite @xcite , commonsense knowledge compilation from pretrained language models @xcite @xcite has recently emerged. in 2019, @xcite commonsense transformers (comet), an approach for fine-tuning language models on existing corpora of commonsense assertions. these models have shown promising performance in generating commonsense assertions after being trained on established human-authored commonsense resources such as atomic @xcite and atomic 20 20 @xcite . more recently, @xcite extracts commonsense assertions from a general language model, @xcite , using simple prompting techniques. surprisingly, using this machine-authored commonsense corpus to finetune comet helps it outperform gpt-3, which is 100x larger in size, in terms of commonsense capabilities. despite the prominence of this approach (the seminal comet paper @xcite receiving over 300 citations in just two years), to date, no resource containing commonsense knowledge compiled from any comet model is publicly available. as compilation of such a resource is a non-trivial endeavour, this is a major impediment to research that aims to understand the potentials of the approach, or intends to employ its outputs in downstream tasks. this resource paper fills this gap. we fine-tune the comet pipeline on two established resources of concept-centric csk assertions, conceptnet @xcite and ascent++ @xcite , and execute the pipeline for 10k prominent subjects. unlike the atomic resources, which were used to train comet in @xcite and have their main focus on events and social interactions, the two resources of choice are mostly about general concepts (e.g., lions can roar, or a car has four wheels). furthermore, as those two resources were constructed using two fundamentally different methods, crowdsourcing and web text extraction, it enables us to discover potentially different impacts they have on the comet models. by taking the top-10 inferences for each subjectpredicate pair, we obtain four resources, concept-net (gpt2-xl, bart) and ascent++ (gpt2-xl, bart), containing 900k to 1.4m ranked assertions of csk. we perform a detailed evaluation of the intrinsic quality, including fine-grained precision (typicality and saliency) and recall of each resource, derive qualitative insights into the strengths and weaknesses of the approach, and highlight extrinsic use cases enabled by the resources. ## related work early approaches at csk compilation relied on expert knowledge engineers (lenat, 1995) or crowdsourcing @xcite , and the latter approach has recently been revived @xcite . to overcome scalability limitations of manual compilation, text extraction is a second popular paradigm. following early attempts on linguistic corpora @xcite , increasingly approaches have targeted larger text corpora like wikipedia, book scans, or web documents @xcite @xcite , to build csk resources of wide coverage and quality. recently, both approaches have been complemented by knowledge extraction from pretrained language models: language models like bert @xcite or gpt @xcite have seen millions of documents, and latently store associations among terms. while @xcite used prompting to extract symbolic csk from gpt-3, @xcite proposed to tap this knowledge by supervised learning: the language models are finetuned on statements from existing knowledge resources, e.g., trained to predict the object africa when given the subject-predicate pair elephant, at-location, based on the conceptnet triple elephant, atlocation, africa . after training, they can be used to predict objects for unseen subject-predicate pairs, e.g., locations of wombats. the approach gained significant attention, and variants are employed in a range of downstream tasks, e.g., commonsense question answering @xcite , commonsense explanation @xcite , story generation @xcite , or video captioning @xcite . yet, to date, no materialized knowledge resource produced by any comet model is available (autotomic from @xcite being based on prompting gpt-3). the closest to this is a web interface hosted by the allenai institute at https://mosaickg.apps.allenai.org/ model_comet2020_entities . however, this visualizes only predictions for a single subject, making, e.g., aggregations or count impossible, and only shows top-5 predictions, and without scores. ## methodology we follow the implementations in the official code repository foot_0 of the comet-atomic 20 20 project @xcite to compute assertions, and decide on output thresholds. training cskbs. we use two established concept-centered commonsense knowledge bases (cskbs), conceptnet 5.7 @xcite and ascent++ @xcite as training resources, considering 13 csk predicates from each of them: atlocation, capableof, causes, desires, hasa, hasprerequisite, hasproperty, hassubevent, madeof, motivatedbygoal, partof, usedfor and receivesaction. language models. we consider two autoregressive language models (lms) that were also used in the original comet paper, gpt2-xl @xcite and bart @xcite . materialization process. we query the finetuned comet models for 10,926 subjects in con-ceptnet which have at least two assertions for the 13 csk predicates. for each subject-predicate pair, we use beam search to obtain completions, with different configurations (see output. the resulting resources, conceptnet (gpt2-xl, bart) and ascent++ (gpt2-xl, bart), contain a total of 976,296 and 1,420,380 and 1,271,295 and 1,420,380 assertions after deduplication, respectively, as well as their corresponding beam scores. all are available for browsing, as well as for download, at https://ascentpp. mpi-inf.mpg.de/comet (see screenshot of browsing interface in figure 2 ). ## analysis we perform three kind of analyses: (1) a quantitative evaluation of the intrinsic quality of the assertions, based on crowdsourcing, (2) a qualitative evaluation that outlines major strengths and weaknesses, and (3) an illustration of use cases enabled by both resources. ## conclusion we introduced four cskbs computed using two comet models (bart and gpt2-xl) trained on two existing csk resources (conceptnet and ascent++). our findings are: we also identified common problems of the comet generations, such as co-occurrence misreadings, subject copying, and redundancies, which may be subject of further research regarding postfiltering and clustering.