P a C o: Preconditions Attributed to Commonsense Knowledge (2022)

Abstract: Humans can seamlessly reason with circumstantial preconditions of commonsense knowledge. We understand that a glass is used for drinking water, unless the glass is broken or the water is toxic. Despite state-of-the-art (SOTA) language models’ (LMs) impressive performance on inferring commonsense knowledge, it is unclear whether they understand the circumstantial preconditions. To address this gap, we propose a novel challenge of reasoning with circumstantial preconditions. We collect a dataset, called PaCo, consisting of 12.4 thousand preconditions of commonsense statements expressed in natural language. Based on this dataset, we create three canonical evaluation tasks and use them to examine the capability of existing LMs to understand situational preconditions. Our results reveal a 10-30% gap between machine and human performance on our tasks, which shows that reasoning with preconditions is an open challenge.

## introduction improving a system's ability to reason with commonsense knowledge is at the frontier of natural language processing (nlp) research, as a critical component in many knowledge-driven tasks such as question answering @xcite , machine reading comprehension @xcite , narrative cloze @xcite , and dialogue systems @xcite . recently, dozens of systems @xcite @xcite and learning resources @xcite @xcite have been proposed, focusing on various aspects of commonsense knowledge such as naive physics and naive psychology. in cognitive studies, the theory of affordance @xcite suggests 1 code and data on https://github.com/luka-group/paco that understanding the circumstances in which an action or statement is possible or impossible is a key aspect of human intelligence. for example, a glass may be used for drinking water, under an implicit assumption that the water is at normal temperature, but may not if the glass is shattered. accordingly, we argue that for an nlp reasoner to understand common sense, it should comprehend the contextual preconditions associated with commonsense statements. such contextual preconditions can naturally be categorized into two classes: the ones that enable the statements, and the ones that disable them @xcite . causal preconditions may be partially inferred from text @xcite , however: 1) as is the case in many other aspects of common sense, we rarely write them explicitly in our text; 2) when mentioned in the text, it is difficult for models to distinguish whether they represent causation or correlation. similar to our work, @xcite collect the preconditions by crowdsourcing. here, the preconditions are seen as soft assumptions, namely: weakeners and strengtheners, which provides a model only with the relative correlation between statements, and is not explicitly testing the model on the underlying preconditions of the statement. instead, we propose to define the problem based on the crisp conditioning of disablers and enablers, which forces the lm to learn the decisive preconditions of a statement and facilitates explainability based on them. in comparison to a hard logical connection modeled by the crisp condition, although the notion of weakener is also helpful to the commonsense reasoner, it raises additional questions like "by how much?", or "is the statement still valid?". whereas in the notion of disablers, even though annotations are more difficult to collect, it can at least take the system one step forward by sorting out the clutter of the irrelevant statements. this paper presents a systematic study on the problem of situational preconditions expressed in natural language. as the first contribution, we define a new problem of reasoning with enabling and disabling preconditions associated with commonsense statements (section 2). given a statement, the task is to infer the preconditions that make the statement possible (enabling) or impossible (disabling). understanding such preconditions of commonsense knowledge would enable reasoning systems relying on a commonsense knowledge base to decide when to use a given commonsense statement. for example, given the statement "glass is used for drinking water" in conceptnet @xcite , a system should know that it is only possible if the "water is not too hot", and it is impossible when "the water is toxic". to foster research on preconditions of commonsense knowledge, we develop paco, a rich crowdsourced dataset with enabling and disabling preconditions of commonsense statements (section 3), as the second contribution of this paper. for paco, we start by extracting available commonsense statements. we then design and execute a crowdsourcing task to gather preconditions of the statements by asking participants: what makes the statement possible/impossible? for each of the statements. paco contains 12.4k labeled preconditions (6.6k enabling, 5.8k disabling), corresponding to 3 * 1k edges from three representative relations in con-ceptnet @xcite , covering knowledge on utility, causality, and motivation. example preconditions are illustrated in fig. 1 . these tasks for the first time allow analysis beyond what is done in prior work that cover enabling preconditions only. particularly, they realize a head-to-head compari-son of enabling and disabling statements which was not possible before. besides, they allow analysis of the impact of the knowledge types (e.g., utility) on the task difficulty for both humans and neural language models. our third contribution is an extensive nlp benchmarking based on paco. to this end, we transform paco into three tasks on preconditions: natural language inference (p-nli), multiple-choice question answering (p-mcqa), and generation (p-g). the three canonical tasks seek to provide a comprehensive evaluation of the ability of natural language reasoners to understand circumstantial preconditions (section 4). these three tasks examine the understanding of preconditions of a number of sota language models and reasoners, such as deberta @xcite , and unifiedqa @xcite . results show that sota methods largely fall behind human performance, therefore indicating the need for further research in order to improve the comprehension of contextual preconditions by commonsense reasoners (section 5). ## preconditions in commonsense reasoning problem definition. commonsense statements describe well-known information about concepts, and, as such, they are acceptable by people without need for debate @xcite . a commonsense statement can be formalized as @xmath0, r, t), where h and t are head and tail concepts, and r is the relation type. following the notion of "causal complex" @xcite , we define the precondition p f as a collection of eventualities (events or states) that results in s to happen. such preconditions contain eventualities that either allow (p + @xmath1. here, to prevent means to allow the negation of the statement @xcite . while enumerating a priori all such causal eventualities is impossible, people are still able to reason about them in a given situation @xcite . notably, preconditions are implicit, i.e., we usually omit them from conversation as they are considered obvious @xcite . @xcite and @xcite distinguish between two type of preconditions, based on causal connections (hard), or material implication (tends to cause; soft). here we focus on the more restrictive, hard preconditions; for soft preconditions, see @xcite . in this work, the problem of reasoning with preconditions is attempted in two ways: discriminative and generative (cf. ## paco this section introduces the procedure of developing the paco dataset. we start by selecting relevant commonsense facts (section 3.1), and crowdsourcing preconditions for each statement (section 3.2). finally, we present the paco data statistics (section 3.3). ## tasks given the data collected in section 3, we devise three complementary tasks to showcase the possible ways one could use the paco data to evaluate the current sota models' understanding of circumstantial preconditions. we select preconditions natural language inference (p-nli) and preconditions multiple-choice question answering (p-mcqa) as representative discriminative tasks, and preconditions generation (p-g) task as a generative task. in the rest of this section, we describe each task in detail and discuss the steps to prepare it from the raw precondition data. this preparation is fully automatic, and no human annotation or supervision signals have been used. p-nli task natural language inference (nli) refers to tasks where given a sentence pair composed of a hypothesis and a premise, the system has to decide whether the hypothesis is true (en-tailment), false (contradiction), or undetermined (neutral) given the premise @xcite . each of the preconditions (e.g., "water is clean" or "water is polluted") of a statement can directly serve as a premise in the sense of nli. enabling preconditions correspond to entailment cases (e.g., "water is clean" entails "water is used for drinking"), whereas disabling preconditions can be annotated as contradictions (e.g. "water is polluted" contradicts "water is used for drinking"). the p-nli task consists of 12.4k entries, with 6.6k entailment and 5.8k contradiction cases. p-mcqa task paco can also be directly converted to a multiple-choice question answering (mcqa) task in three steps. first, for each statement, each enabling (disabling) response is paired with three disabling (enabling) responses from the same statement. these three responses naturally act as negative samples (distractors), allowing us to have high-quality and fair questions. the question of the mcqa instance is then formed by appending "what makes this possible?" or "... impossible?" to the lexicalized statement. second, in order to have more distractors and increase the number of multiple-choice instances we applied the two negative sampling methods used by @xcite : cosine similarity filtering, and question/answer shuffling. finally, in order to remove the annotation artifacts from the data, hence trivial instances, and prevent the models to exploit these artifacts instead of answering the questions, we used the lite variation of the adversarial filtering method, which has been introduced in @xcite and formalized in @xcite . this resulted in a p-mcqa task with 47k multiple choice questions, each with 4 choices. p-g task despite our adversarial strategies, it remains possible that reasoning systems may identify annotation artifacts @xcite in the data and solve the discriminative tasks without correctly performing the logical inference, as a result of those artifacts @xcite . hence, we provide a third formulation as a generative commonsense reasoning task. in this task, we present the system with the exact question that has been presented to the human annotators, thereby mimicking the human annotation task of writing down the precondition as a natural language sentence. we then evaluate the model's response using the human responses as references. after removing the low-quality and invalid responses from paco, the p-g task consists of 5.2k instances, with an average of 2.4 reference sentences per instance. ## experiments this section pitches sota language models against the three tasks derived from paco (section 5.1), dives deep into the tuning process to pinpoint time of comprehension (section 5.2), investigates how lms react to different relation types (section 5.3), and finally revisits the distinction between soft and hard preconditions (section 5.4). ## related work resources of preconditions. a few resources have provided representations for preconditions of statements. conceptnet @xcite 's hasprerequisite relation, atomic @xcite )'s xneed relation, and causenet @xcite data can express concept dependencies, such as, e.g., before one bakes bread, they need to buy ingredients and go to a store. instead of adding new edges, our work annotates existing edges with contextual preconditions, which helps reasoners understand when to use an edge and when not to. aser @xcite and ascent @xcite extract edges from unstructured text together with their associated context. as such, their knowledge is restricted by information available in text, and they do not express disabling preconditions. it is also unclear to which extent their contextual edges express enabling preconditions, rather than coincidental information. glu-cose @xcite comes closer to our work, as they also extract enabling preconditions (e.g., possession state that enables x) via crowdsourcing. similarly, peko @xcite extract enabling preconditions between event pairs from available text and use it to propose precondition identification and generation tasks between pair of sentences. however focusing only on causal relations in available text hinders the extent of their tasks. both glucose and peko do not explore disabling preconditions. reasoning with preconditions. few efforts have been made on evaluating commonsense reasoning with preconditions. @xcite focus on modeling weakeners and strengtheners of commonsense statements. their work adds a utility sentence to the hypothesis-premise pair in nli-style tasks and ask whether it weakens or strengthens the relationship of the pair. similarly, hwang et al. (2020)'s hindered by and causes also focuses on similar relationship for events with focus on presenting a knowledge resource. our work differs as we focus on a crisp condition of enabling/disabling that can be particularly useful in logic-like reasoning tasks (as opposed to probabilistic inference). in addition, our task allows the reasoning to be processed as canonical nli and can benefit from existing nli architectures instead of modifying them. ## conclusions and future work we presented, paco, a dataset of 12.4k collected enabling and disabling preconditions of everyday commonsense statements from conceptnet. we utilize this resource to create three tasks for evaluating the ability of systems to reason over circumstantial preconditions, namely: p-nli, p-mcqa, and p-g. our evaluation shows that sota reasoners largely fall behind human performance, indicating the need for further investigation to develop precondition-aware systems. future work should cover the inclusion of preconditions in logical reasoning of the neurosymbolic reasoners. it should also expand to multimodal setup or investigate using weak-supervision to gather preconditions. alternatively, we can leverage the contributed resource to develop generative models for automated context-aware knowledge base construction @xcite .