CAT : A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoning (2023)

Abstract: Commonsense reasoning, aiming at endowing machines with a human-like ability to make situational presumptions, is extremely challenging to generalize. For someone who barely knows about “meditation,” while is knowledgeable about “singing,” he can still infer that “meditation makes people relaxed” from the existing knowledge that “singing makes people relaxed” by first conceptualizing “singing” as a “relaxing event” and then instantiating that event to “meditation.”This process, known as conceptual induction and deduction, is fundamental to commonsense reasoning while lacking both labeled data and methodologies to enhance commonsense modeling. To fill such a research gap, we propose CAT (Contextualized ConceptuAlization and InsTantiation),a semi-supervised learning framework that integrates event conceptualization and instantiation to conceptualize commonsense knowledge bases at scale. Extensive experiments show that our framework achieves state-of-the-art performances on two conceptualization tasks, and the acquired abstract commonsense knowledge can significantly improve commonsense inference modeling. Our code, data, and fine-tuned models are publicly available at [ https://github.com/HKUST-KnowComp/CAT ]( https://github.com/HKUST-KnowComp/CAT ).

## introduction instead, humans rely on conceptual induction and deduction @xcite to make inferences on novel situations without the need to memorize all special cases. as shown in figure 1 , humans can derive conceptualizations based on the assertion that "personx watches a football game, as a result, he feels relaxed" to infer that "relaxing events can make someone feel relaxed," where the acquired abstract commonsense knowledge can be further used as general knowledge to perform reasoning on similar or associated situations. a new commonsense knowledge "personx plays with his dog, as a result, he feels happy and relaxed" can be deduced by instantiating relaxing events to playing with his dog. as the cornerstone of generalizable commonsense reasoning, such a process is extremely challenging for machines to replicate due to the absence of contextualized conceptualizations and abstract commonsense knowledge in cskbs and a lack of relevant methodologies. yet, existing works address the process of induction and deduction separately via conceptualization and instantiation. several methods performing conceptualization are proposed with a specific focus on entity-level @xcite @xcite @xcite and event-level @xcite semantics. instantiation @xcite , as the process that simulates conceptual deduction, is tackled separately and not leveraged by these methods. though abstract commonsense knowledge can be derived by using existing conceptualization methods to abstract a certain instance from factual commonsense knowledge, several limitations still exist. first, the plausibility of abstract commonsense knowledge banks on both the correctness of conceptualization and proper contextualization under specific assertions. the latter one, which is an essential step for the deduction of abstract knowledge, is missing from current methodologies. take figure 1 as an example, the concept observe will not necessarily lead to the result of "feeling relaxed", as observe omits the entertaining property of the original instance as a cost of abstraction. second, instantiating abstract commonsense knowledge can yield much more and diverse concrete commonsense knowledge that can serve as an augmentation of the training dataset, while current methods undervalue such a process and only focus on conceptualization. finally, the complex contextualization and conceptualization of commonsense knowledge can easily bring more than two orders of magnitude of data on top of the original dataset. this makes current labeled data scarce and infeasible for practitioners to annotate all of them, leaving a large amount of unlabeled data. to fill in these research gaps, we propose cat (contextualized conceptualization and instantiation), a semi-supervised learning framework that unites event conceptualization and instantiation in cascade to conceptualize cskbs and acquire abstract commonsense knowledge to aid commonsense reasoning. inspired by how humans learn with concepts @xcite , we design a novel bootstrapping foot_0 method to enhance conceptualizations and abstract commonsense knowledge verification with the help of similar conceptualizations and instantiations as a reference. we demonstrate the effectiveness of cat by using the acquired abstract commonsense knowledge to train comet @xcite , a commonsense inference language model that generates if-then commonsense knowledge, and showing that our derived abstract commonsense knowledge can significantly improve commonsense inference modeling. our contributions are three-fold: (1) we introduce a semi-supervised learning framework, cat, to conceptualize cskbs with the assistance of progressively bootstrapping similar abstract concepts or instantiations in the conceptualization process. commonsense reasoning. endowing nlp systems with the ability to perform commonsense reasoning is an elusive goal of artificial intelligence @xcite . a diverse collection of commonsense reasoning tasks have been proposed as evaluation benchmarks @xcite @xcite . among them, @xcite proposed a generative model, comet, to learn to produce if-then commonsense knowledge as an effective approach toward modeling commonsense inference that can be applied in various commonsense reasoning tasks @xcite . semi-supervised learning. semi-supervised learning (ssl) aims at taking advantage of unlabeled data to equip models with stronger generalization ability (van engelen and hoos, 2020). the most common approach is using pseudo labels @xcite to expose more unseen data to the student model. it has been applied in various machine learning tasks such as image classification @xcite ), text classification @xcite @xcite , commonsense knowledge base population @xcite , and named entity recognition @xcite . ## problem definition definition. conceptualizing an event-centric cskb to derive abstract commonsense knowledge comprises two steps @xcite : event conceptualization and triple conceptualization. denote the triples in the original cskb as d @xmath0, r, t)|h @xmath1, @xmath2, @xmath3, where h o , r, and t are the set of heads, relations, and tails in the original cskb. the first step only operates on head events without considering the context in r and t. the goal of event conceptualization is to produce conceptualized head event h a from the original head h o to represent an abstraction of h o . in the second step, the task is to verify whether the conceptualized head h a still makes sense in the context of r and t, as r and t will further restrict the level of abstractness in h a . as shown in figure 1 , conceptualizing watch football game to observe is wrong within the context of having feel relaxed as a result. plausible (h a , r, t) triples will be considered as valid abstract commonsense knowledge. specifically, in the first step, there are two ways of conceptualizing head events alone: a retrievalbased discriminative way and a generative way. the retrieval-based discriminative paradigm identifies and links a component i in h o to a concept c in a concept taxonomy c to form a conceptualization h a by replacing i with c. the model needs to verify whether h a is a valid conceptualization of h o . the generative paradigm aims to generate a h a directly given h o and the designated component i in h o . formally, denote the annotated dataset in the first step, event conceptualization, as d l @xmath4, h a , y)|h @xmath5, h @xmath6, @xmath7, 1}}, where h o is an original head event without conceptualization, h a is a corresponding conceptualization of h o , and y is the human-annotated label indicating whether such a conceptualization is plausible or not. the labeled dataset in the second step, triple conceptualization, is denoted as d l @xmath8, r, t, y)|@xmath9, @xmath10, @xmath11, @xmath12, 1}}, where h is a conceptualized head event from the first step, r and t are a relation and a tail from the original cskb accompanied with the corresponding original head h o , and y is the human-annotated label indicating whether such abstract commonsense knowledge, in the form of a conceptualized triple, is plausible or not. besides labeled datasets, unlabeled datasets are defined similarly as d u h and d u t only with the difference that labels y are missing. thus, the task objective for discriminative event conceptualization is to determine whether a h o can be properly abstracted using h a , where h a is derived by replacing a component i ⊂ h o with its linked concept c from a concept taxonomy c. the task objective for generative event conceptualization is to generate h a directly from h o with text generation models. for the triple conceptualization task, the objective is to distinguish whether a conceptualized triple (h a , r, t), representing abstract commonsense knowledge, is plausible or not. ## cat framework this section introduces our proposed contextualized conceptualization and instantiation (cat) framework for conceptualizing commonsense knowledge bases and acquiring abstract commonsense knowledge. an overview is presented in figure 2 . our motivation is two-fold: first, adding instantiation after conceptualization to form a cycle can strongly benefit two conceptualization tasks simultaneously. on the one hand, instantiating conceptualized triple relies on the correctness of event conceptualization. on the other hand, properly conceptualized triples can benefit event conceptualization via instantiation by providing more context brought by (r, t). second, to address the lack of annotations, we resort to pseudo labeling, a typical semi-supervised learning approach to automatically assign pseudo labels to the vast majority of unlabeled data using a teacher model. following @xcite , we study the retrieval-based discriminative paradigm of event conceptualization and leave the generative paradigm as an intrinsic evaluation. in cat, we unify event conceptualization and triple conceptualization into one cycle and make them mutually benefit each other through instantiation and conceptualization. our framework can be summarized into four steps: (1) train teacher models for both event conceptualization and triple conceptualization on the labeled dataset d l h and d l t , respectively. use the two teachers to assign pseudo labels to unlabeled datasets. ## experiments we conduct conceptualization experiments using cat in section 5.1 and generative experiments as evaluations in section 5.2. these experiments demonstrate that cat has a strong capability in conceptualizing cskbs, and better conceptualization modeling can help populate more novel and diverse commonsense knowledge and thus help commonsense modeling (comet). ## conclusion in conclusion, this paper proposes cat, a semisupervised learning framework for commonsense reasoning, by leveraging the power of abstract commonsense knowledge. by achieving state-of-theart performances in cskb conceptualization tasks, we remarkably improve modeling commonsense inference, as an important cornerstone of many commonsense reasoning tasks. our analysis also demonstrates that high-quality abstract commonsense knowledge can benefit commonsense inference modeling by providing more generalizability on hard commonsense knowledge. we hope this work can draw insights toward commonsense reasoning from a conceptualization perspective.