COLA : Contextualized Commonsense Causal Reasoning from the Causal Inference Perspective (2023)

Abstract: Detecting commonsense causal relations (causation) between events has long been an essential yet challenging task. Given that events are complicated, an event may have different causes under various contexts. Thus, exploiting context plays an essential role in detecting causal relations. Meanwhile, previous works about commonsense causation only consider two events and ignore their context, simplifying the task formulation. This paper proposes a new task to detect commonsense causation between two events in an event sequence (i.e., context), called contextualized commonsense causal reasoning. We also design a zero-shot framework: COLA (Contextualized Commonsense Causality Reasoner) to solve the task from the causal inference perspective. This framework obtains rich incidental supervision from temporality and balances covariates from multiple timestamps to remove confounding effects. Our extensive experiments show that COLA can detect commonsense causality more accurately than baselines.

## introduction commonsense causal reasoning (ccr) aims at identifying plausible causes and effects of events in natural language that are typically reasonable by an average person @xcite . to solve the task, existing efforts devoted by the community mainly rely on language models wholeheartedly with supervised learning approaches @xcite @xcite @xcite . those ingenious engineering works have brought significant progress in recent years. however, recent studies @xcite found that pure engineering designs are inadequate to seize commonsense causation, as language mod-emma made a steak in the kitchen. 🥩 els tend to reach higher scores by exploiting superficial artifacts in data. recently, @xcite first studied grasping commonsense causation from the causal inference perspective, by drawing analogies between observational studies @xcite and natural languages @xcite . the proposed framework rock achieves good potential for the zero-shot ccr task (e.g., copa by @xcite ). however, @xcite only focuses on the commonsense causation between a pair of events without specifying context. given that events are complex @xcite , an event may have different causes under different contexts @xcite . thus, it is necessary to utilize context when detecting commonsense causation, such as other events related to given ones. missing a clear and specific context simplifies commonsense causal knowledge and hinders models from detecting commonsense causal relations more accurately. for example, as shown in figure 1 , the frequent cause of "emma made a steak in the kitchen." is "emma felt hungry." however, the cause also could be "emma was doing her job" if "emma is a chef at a steakhouse." without the context of emma's job, models cannot distinguish those two causes and may return to the frequent one. to involve context when detecting commonsense causation, we propose a new task to detect causes between two events in an event sequence, called contextualized commonsense causal reasoning (contextualized ccr). in this task, models are asked to detect commonsense causal relations between two given events enclosed in an event sequence. other events in the event sequence can provide a clear and specific definition of the current context, helping models to capture commonsense causation more accurately. in fact, we find that contextualized ccr is a non-trivial task. directly applying the framework rock @xcite cannot achieve competitive performance on the contextualized ccr since it cannot integrate context information. we propose the framework cola, which incorporates contextual information from an event sequence, to solve the contextualized ccr. our framework adopts the potential-outcomes framework @xcite @xcite to estimate the causal estimand ∆ defined as a type of "average treatment effect" (ate), which measures the change in the likelihood of e j 's occurrence when intervening e i (denoted by ¬e i ) as ## background and related works understanding events and relations between them have long been a challenging nlp task @xcite . the community has dedicated many works to studying various event-centric tasks, including event relation reasoning @xcite @xcite , event extraction @xcite @xcite , event-centric kg construction @xcite , and many others @xcite @xcite . among them, there are a few lines of work that are most related to our work: commonsense causal reasoning since our work is about contextualized ccr, we first discuss related works about commonsense causal reasoning. existing commonsense causal reasoning approaches are typically categorized under the general topic of commonsense reasoning @xcite . most previous works depend on language models. remarkable progress in ccr mainly comes from dataset augmentation, training procedure design, and external knowledge @xcite @xcite @xcite . studies @xcite show that language models exploit superficial artifacts to achieve suspicious high performance. causal event detection @xcite forms another line of work pertinent to ccr. the task aims to detect causal relations in documents, where various methods are proposed @xcite @xcite . however, those works consider verbal (e.g., "attack") or nominal predicates (e.g., "explosion") as events, oversimplifying the relation detection task. in this paper, we study events expressed in free text, facing a more challenging setup but being closer to real applications. narrative-related tasks since contextualized ccr is primarily about chains of events, our work is inspired by earlier research that deals with narratives and scripts @xcite @xcite @xcite . in contrast, our work aims to identify causal relations in a chain of events. @xcite provided the first study to solve the ccr task from the causal inference perspective. the human population studies have scrutinized extensively the causal inference, which identifies causal relations from ubiquitous associations, including biomedical research, agriculture, epidemiology, and economics @xcite @xcite @xcite , where researchers usually use the potential-outcomes framework @xcite @xcite , graphical and structural equation models @xcite @xcite , and granger causality @xcite . ## problem formulation notation we use sansserif fonts to represent an event, such as e i in figure 2 , where the subscript i means the i-th event in a sequence. simultaneously, uppercase serif letters denote indicators of whether the corresponding event occurs: e @xmath0, and a lowercase serif letter means the realizations of this indicator: e i,@xmath1. we introduce the point process to more clearly describe the order of a pair of events: e(t) with @xmath2, 1} (e.g., past versus present), so that e i (0) and e j (1) means that e i happens before e j . we also use task description we articulate the contextualized ccr as a tweaked form of the binary classification problem. specifically, we provide models with an event sequence of n events: e 1 , e 2 , . . . , e n . models need to find the top-k events in this sequence that more plausibly have commonsense causal relations with the last event e n , where k indicates the number of positive labels in the ground truth. then, models need to predict the commonsense causal relation between each event pair (e i , e n ) as a positive/negative one. the strength of causal relations between e i and e n can be expressed with average treatment effect as: as discussed in section 1, we articulate the contextualized ccr problem as the estimation of the causal estimand ∆, which we model as the change of temporal likelihood with contexts controlled. we adopt the potential-outcomes framework to design cola to eliminate potential confounding effects due to co-occurrences of events when estimating the causal estimand from data. in this section, we first clarify all essential concepts in the theoretical mechanism of cola one by one, including study unit, intervention, covariate, and propensity, by drawing analogies between the underlying causal mechanisms in natural languages with that in human population research. we then describe the implementation of each component in section 5. ## the cola framework after establishing the theoretical mechanism for our framework cola, we describe the implementation of each component of cola in this section. generally, since events are in free-text form in our task, pre-trained language models play a central role in our framework. given that lms are pre-trained on an enormous amount of textual data @xcite , it is sensible to suppose that those lms would emulate the responses of an average reasonable person. specifically, our framework cola takes two events e i and e n from a sequence e 1 , e 2 , . . . , e n as input. as shown in figure 2 , our framework cola contains four steps: (1) a multistamp covariate sampler samples a set x of covariates. (2) an intervention generator generates a set a of interventions. (3) a score estimator builds temporal propensity vectors and selects a matched subset a ′ out of a, by estimating the temporality with a temporal predictor. (4) eventually, the same score estimator computes ∆i according to equation ( 7 ). multistamp covariate sampler our multistamp covariate sampler is based on gpt-j (6b) @xcite . for an input event e, we add "before that," at the end to build "e before that," as the prompt template. for the i-th event in a sequence, we first sample a covariate set x i , which contains events before e i . to diversify covariates, we also sample events before e 1 , e 2 , . . . , e i-1 separately, forming x 1 , x 2 , . . . , x i-1 . those events are also before e i and can serve as covariates due to the transitivity of the temporal relation foot_1 . we evenly merge covariates before each timestamp to construct the final covariate set: union vs. intersection while the aforementioned method takes the union of events sampled according to the left-side context of e i , another intuitive approach is to take the intersection of events sampled according to multiple timestamps in the right-side context. in this way, we can collect covariates that happen before all of we discuss the experimental results of these two methods in section 7.3 and found taking union works better, which will be the default in the remaining sections. intervention generator this component generates a set a of interventions as discussed in section 4.2. there are a variety of related works about generating interventions (counterfactuals) of an event @xcite @xcite and we choose polyjuice @xcite in our framework owing to its non-task-specific training objective. polyjuice generates interventions by masking some phrases individually and filling in masks with a fine-tuned gpt2. then, we apply the semantic role labeling (srl) tool provided by al-lennlp @xcite to extract the verb v and two arguments arg0 and arg1 as phrases to be manipulated (see appendix a.2 for more details). temporal predictor we prompt a masked language model to estimate the temporal relation scores between two given events e i and e n . the prompt template "e @xmath3, e n ) and f a (e i , e n ) for the output tokens before and after. similarly, we can obtain a reversed estimation by inputting "e @xmath4." final temporal score f averages scores from both directions: f ( our temporal predictor needs to be fine-tuned on a temporal relation corpus. directly applying a pretrained lm encounter the problem of low coverage, where the tokens before and after cannot be found in the top-k prompted tokens (even @xmath5. thus, we fine-tuned masked language models to predict the masked connectives in a prompt learning setting. intuitively, temporal relations exist between each pair of adjacent events in a chronologically ordered event sequence. assuming an event sequence contains two adjacent events e i , e i+1 , we then can create an example e i before e i+1 and an symmetric example e i+1 after e i . we also construct negative samples by replacing e i or e i+1 with a randomly sampled event from other sequences. those negative examples can teach models when no temporal relation exists. while we mask before or after in a positive example for models to predict, a special token [none] should be prompted for negative examples. then, the cross-entropy loss is used to optimize the temporal predictor. we call this fine-tuning process temporal fine-tuning. more details about the temporal relation dataset and fine-tuning process are shown in appendix a.1. score estimator with the temporal predictor, we can estimate p(x ≺ a) for all covariates @xmath6: p(x(0), a(1)) = p(x ≺ a) = f (x, a). we also need to estimate p(x(0)) to compute conditional probabilities p(a(1) = 1|x(0)) in temporal propensity vectors q(x; a). as all covariates x are events preceding e i sampled by gpt-j, there is an implicit conditioning on e i . thus, we can approximately get p(x(0)) ≈ f (x, e i ) (see appendix b for more details). then, temporal propensity vectors are computed as finally, the score estimator computes ∆i in equation (7). we also test normalization methods in appendix e and observe that some of those normalization methods can benefit our framework. ## experiment we conduct extensive experiments and compare cola with a wide selection of baselines. ## main evaluation we provide results in our framework cola based on deberta-large and deberta-base (also bert-large) achieves the best performance on the validation and testing set, respectively. also, changing language models of the temporal predictor in cola only involves a small fluctuation in performance, showing that our framework is robust to underlying language models. another observation is that clm perplexity and clozepromptscore can achieve performance higher than the random result. this manifests that pre-trained language models with commonly used pre-training objectives can capture commonsense causal relations to some extent. ## conclusion in this paper, we design a new task to consider the context when detecting commonsense causal relations. we also crowd-sourced a dataset with strict quality control to benchmark the task. our cola framework is motivated by the principles of causal inference and attempts to leverage context information. the extensive evaluation demonstrates the effectiveness of cola for detecting commonsense causal relations.