CANDLE : Iterative Conceptualization and Instantiation Distillation from Large Language Models for Commonsense Reasoning (2024)

Abstract: The sequential process of conceptualization and instantiation is essential to generalizable commonsense reasoning as it allows the application of existing knowledge to unfamiliar scenarios. However, existing works tend to undervalue the step of instantiation and heavilyrely on pre-built concept taxonomies and human annotations to collect both types of knowledge, resulting in a lack of instantiated knowledge to complete reasoning, high cost, and limited scalability. To tackle these challenges, we introduce CANDLE (ConceptuAlizationand INstantiation Distillation from Large Language ModEls), a distillation framework that iteratively performs contextualized conceptualization and instantiation over commonsense knowledge bases by instructing large language models to generate both types of knowledge with critic filtering. By applying CANDLE to ATOMIC (Sap et al., 2019a), we construct a comprehensive knowledge base comprising six million conceptualizations and instantiated commonsense knowledge triples. Both types of knowledge are firmly rooted in the original ATOMIC dataset, and intrinsic evaluations demonstrate their exceptional quality and diversity. Empirical results indicate that distilling CANDLE on student models provides benefits across three downstream tasks. Our data and models are publicly available at https://github.com/HKUST-KnowComp/CANDLE.

## introduction commonsense reasoning refers to the cognitive ability to make logical inferences and draw conclusions based on general knowledge and understanding of the world that is typically shared among individuals @xcite . however, a longstanding challenge is generalizability, as commonsense reasoning often necessitates applying knowledge to novel situations beyond simple pattern recognition or memorizing all special cases @xcite . personx enjoys exercising in the gym personx enjoys [sports] personx enjoys exercising in the [fitness facility] [health lifestyle] personx enjoys exercising in the [stadium] personx enjoys [surfing] personx enjoys [hiking] [personx stays hydrated] [personx avoid smoking] [personx goes on a balanced diet] stadium ↑ fitness facility healthy lifestyle ↓ goes on a balanced diet stay hydrated ↑ healthy lifestyle healthy lifestyle ↓ avoid smoking as a result, personx will: stay healthy as a result, personx will: maintain physical well-being one promising approach to address this is the chain of conceptualization @xcite and instantiation @xcite , which, akin to the process of conceptual induction and deduction in human reasoning @xcite , involves conceptualizing instances derived from known commonsense knowledge and subsequently instantiating these concepts in new situations to obtain the knowledge required for downstream reasoning. for example, in figure 1 , one can first conceptualize enjoys exercising in the gym as a healthy lifestyle, and then further instantiate it to go on a balanced diet. this process allows for the derivation of a novel event, personx goes on a balanced diet, which may entail new common-sense knowledge when connected with the original event's commonsense inferential tail. by possessing substantial knowledge to initiate the process of conceptualization and instantiation, one can extrapolate limited commonsense knowledge to a wide array of diverse scenarios. yet, replicating this fundamental ability on machines remains challenging due to the absence of both types of knowledge in widely used common-sense knowledge bases (cskbs; @xcite @xcite . various methods compensating the lack of conceptualization ability of language models have been proposed for entity-level @xcite @xcite @xcite and event-level @xcite @xcite conceptualizations by matching against concept taxonomies like probase @xcite and wordnet @xcite . however, several limitations still persist. firstly, despite the importance of both conceptualization and instantiation, most existing works underestimate the importance of the second step while focusing solely on conceptualization and using the resulting abstract knowledge directly. other studies that concentrate on instantiations either overlook the conceptualization step entirely or only retrieve instances from the original cskb, failing to introduce novel entities and events. secondly, most conceptualization methods heavily depend on matching instances with concepts in concept taxonomies, such as probase and wordnet, which have a limited scope and lack contextual information. consequently, the derived conceptualizations are constrained in scale by these taxonomies and are formulated without considering proper contextualization, necessitating further verification in the original context. lastly, the chain of conceptualization and instantiation can easily bring more than two orders of magnitude of data on top of the original cskb. however, current acquisition and verification methods for both steps heavily rely on human annotation, which can be extremely costly as the scale of the cskb increases. to address these gaps, we introduce candle, a conceptualization and instantiation distillation framework from large language models (llms) to aid commonsense reasoning. specifically, can-dle marks the first to complete the chain of conceptualization and instantiation by instructing powerful llms to sequentially generate both types of knowledge based on concrete commonsense triples while carefully considering the original context throughout the process. we further alleviate the human annotation cost by employing two critic filtering models to eliminate low-quality generations. the instantiated knowledge, representing concrete commonsense knowledge again, can be fed back into candle as input, iteratively augmenting the original cskb significantly. by applying candle to atomic @xcite , we construct a large-scale knowledge base comprising 6.18 million conceptualizations and instantiations from two powerful llms, chat-gpt (openai, 2022) and llama2 @xcite . we demonstrate the intrinsic efficacy of candle through automatic and human evaluations, highlighting the ability to generate highquality and diverse knowledge (section 5.1). we further show the extrinsic benefits of candle by leveraging the generated knowledge as complementary training data to distill student models that yield improvements across three downstream tasks, including cskb conceptualization, generative commonsense inference, and zero-shot commonsense question answering (section 5.2). ## definitions and datasets we follow the definitions proposed by @xcite and @xcite to formulate conceptualization and instantiation. denote the triples in the original cskb as d @xmath0, r, t)|h @xmath1, @xmath2, @xmath3, where h o , r, and t are the set of heads, relations, and tails in the original cskb. the objective of conceptualization is to form a conceptualized head event, denoted as h a , from the original head h o . this is achieved by linking a component i ⊆ h o to a concept c, forming h a by replacing i with c. consequently, abstract knowledge is formed by combining the conceptualized head event with the original relation and tail, represented by (h a , r, t). in the next step, the goal of instantiation is to associate the concept c ⊆ h a with a new instance i ′ . this process enables the formation of new commonsense knowledge in the format of (h i ′ , r, t), where h i ′ is obtained by replacing c ⊆ h a with i ′ . in this paper, we use atomic @xcite as the original cskb d o , which contains 310k (h o , r, t) triples after dropping those with wildcards and 18,839 unique h o head events. abstractatomic @xcite is used as the source of instances i for every head event h o . ## candle this section introduces our candle framework, illustrated in figure 2 . our framework can be outlined in three steps: (1) instruct chatgpt to generate contextualized conceptualizations based on the triples in the original cskb. (2) instruct llama2 to instantiate the conceptualizations obtained in ## evaluations and analysis in this section, we evaluate candle from both intrinsic and extrinsic perspectives. intrinsically, we demonstrate the high quality and diversity of conceptualizations and instantiations generated by candle (section 5.1). extrinsically, we explore the benefits by applying the distilled knowledge to downstream tasks (section 5.2). ## conclusion this paper introduces candle, a distillation framework that realizes the chain of conceptualization and instantiation over cskbs. we demonstrate the efficacy of candle through comprehensive evaluations of the distilled knowledge and its positive impact on downstream tasks. our research sheds light on distilling llms to enable more robust and generalizable commonsense reasoning.