C onstraint C hecker: A Plugin for Large Language Models to Reason on Commonsense Knowledge Bases (2024)

Abstract: Reasoning over Commonsense Knowledge Bases (CSKB), i.e. CSKB reasoning, has been explored as a way to acquire new commonsense knowledge based on reference knowledge in the original CSKBs and external prior knowledge.Despite the advancement of Large Language Models (LLM) and prompt engineering techniques in various reasoning tasks, they still struggle to deal with CSKB reasoning.One of the problems is that it is hard for them to acquire explicit relational constraints in CSKBs from only in-context exemplars, due to a lack of symbolic reasoning capabilities (CITATION).To this end, we proposed **ConstraintChecker**, a plugin over prompting techniques to provide and check explicit constraints.When considering a new knowledge instance, ConstraintChecker employs a rule-based module to produce a list of constraints, then it uses a zero-shot learning module to check whether this knowledge instance satisfies all constraints.The acquired constraint-checking result is then aggregated with the output of the main prompting technique to produce the final output.Experimental results on CSKB Reasoning benchmarks demonstrate the effectiveness of our method by bringing consistent improvements over all prompting methods.

## introduction commonsense knowledge bases (cskb) reasoning, as one of many commonsense reasoning tasks, has been well explored in natural language processing for the past few years. as humanannotated cskbs @xcite @xcite are usually incomplete and of a small coverage, reasoning over cskbs, i.e., cskb reasoning, is a way for expansion. cskb reasoning is defined as determining whether a new knowledge triple (head event, answer whether the following statement is plausible. answer with only yes or no: s: personx eat the sub, as a result, personx feels personx be full a: yes s: personx play game, as a result, personx feels personx forget to do homework a: no … s: personx prepare for the competition, as a result, personx feels personx win a: personx prepare for the competition personx's competitors feel insecure personx is ambitious personx heard about the competition personx wanted the award personx is confident personx win personx is skillful xwant oeffect xattr isafter xreact ? personx take the opportunity yes figure 1 : examples of cskb reasoning. solid arrows represent existing triples (i.e., instances) in cskbs, while the dashed arrows with question marks represent new knowledge triples which will be determined if they are commonsense. llms often fail to acquire the explicit relational constraints in cskbs, hence making wrong predictions for many new knowledge triples. recently, inspired by the emergence of large language models (llms) that can perform well in many commonsense reasoning tasks @xcite , @xcite attempted to use llms for a cskb reasoning bench-mark named cskb population (ckbp) @xcite . however, the result shows that llms still fall short in the benchmark, even with a large number of in-context examples. one of the problems is that llms find it hard to acquire the explicit relational constraints in cskbs, hence making wrong predictions. in the example in figure 1 , the xreact 1 relation in cskbs requires the tail event of the knowledge triple to express a mental state, such as "personx is confident", instead of an action, such as "personx win". meanwhile, llms fail to recognize the constraint from in-context exemplars, thus making the judgment mainly based on the semantics of the head and tail events. it leads to an incorrect prediction that the triple (per-sonx prepare for the competition, xreact, personx win) is plausible. in light of this, many advanced prompting techniques, such as chain-of-thouht (cot) @xcite , least-to-most @xcite , active-cot @xcite , etc., can be possible alternatives for improvements. nonetheless, they are task-agnostic and suffer from the inherent shortcoming of llms in inducing the rules in cskbs (which we refer as symbolic reasoning ability), as current deep learning still struggles to deal with symbolic and high-level concepts reasoning tasks @xcite @xcite . to this end, we propose constraintchecker, a plugin component for llms to handle the problem of explicit constraints in cskb reasoning. constraintchecker supports llms' reasoning as an independent component in addition to the maintask component that determines whether a knowledge triple is commonsense or not. there are two modules in this plugin. given a knowledge triple (head event, relation, tail event), we first employ a rule-based/symbolic module to produce a list of constraints based on the relation. the list is then passed to a zero-shot learning module, where we construct constraint-checking questions and use the same llm as in the main-task component in a zeroshot manner to check whether the instance satisfies all constraints. the acquired constraint-checking result is then aggregated with the prediction from the main-task component by logical conjunction to produce the final prediction. we implement constraintchecker and conduct extensive experiments on a cskb reasoning 1 by definition in @xcite , a tail event of the xreact relation expresses how the protagonist feels after the corresponding head event. to summarize, our contribution is two-fold: (1) we propose constraintchecker, an independent plugin that handles the problem of explicit constraints in cskb reasoning to improve over maintask prompt methods, and (2) we conduct extensive experiments on two cskb reasoning benchmarks ckbpv2 and sd-atomic 20 20 , to demonstrate our method's effectiveness and advantages over other advanced prompting techniques. ## constraintchecker an overview of our proposed constraintchecker is shown in figure 2 . the cskb reasoning task we focus on is inherently a binary classification task and the expected outputs are either plausible or implausible. constraintchecker consists of two modules, entitled module 1 and module 2. for each instance, module 1 queries preset rules to get all relational constraints corresponding to the instance's relation. module 2 then constructs questions accordingly to ask whether each constraint is satisfied, and passes these questions to the backbone llm to get predictions. together with the prediction from the main-task component, we use the logical conjunction (and operator) to aggregate the final prediction. in fact, constraintchecker only has the effect on instances that are predicted as commonsense (or "yes", corresponding to plausible) by the main-task component, and can only change the prediction from "yes" to "no", in view of the nature of logical conjunction. thus, it targets and corrects false-positive predictions. in this section, we elaborate on how we select the pool of constraints and the preset rules to map relations to constraints in module 1, as well as the constraint-checking prompt design in module 2 concerning the two benchmarks. ## conclusion in this paper, we proposed constraintchecker, a constraint-wise plugin, to help llms and prompting methods to cope with the problem of explicit relational constraint in cskb reasoning. experimental results show that constraintchecker consistently improves over any main-task prompting technique by a significant margin, achieving sota performance on the benchmark.