Causal Inference of Script Knowledge (2020)

Abstract: When does a sequence of events define an everyday scenario and how can this knowledge be induced from text? Prior works in inducing such scripts have relied on, in one form or another, measures of correlation between instances of events in a corpus. We argue from both a conceptual and practical sense that a purely correlation-based approach is insufficient, and instead propose an approach to script induction based on the causal effect between events, formally defined via interventions. Through both human and automatic evaluations, we show that the output of our method based on causal effects better matches the intuition of what a script represents.

## introduction commonsense knowledge of everyday situations, as defined in terms of prototypical sequences of events, has long been held to play a major role in text comprehension and understanding @xcite @xcite @xcite . naturally, this has motivated a large body of work looking to learn such knowledge, such scripts, 1 from text corpora through data-driven approaches. a minimal and often implicit requirement for any such approach is to resolve for any pair of events e 1 and e 2 what quantitative measure should be used to determine whether e 2 should "follow" e 1 in script. that is, documents may serve as descriptions of events that occur in the same situation as other events: what function may we compute over the raw presence or absence of events in documents that is most useful for script induction? @xcite adopted point-wise mutual information (pmi) (church and 1 for simplicity we will refer to these 'prototypical event sequences' as scripts throughout the paper, though it should be noted scripts as originally proposed contain further structure not captured in this definition. figure 1 : the events of watching a sad movie, eating popcorn, and crying, may highly co-occur in a hypothetical corpus. what distinguishes valid event pair inferences (event pairs linked in a commensense scenario; noted by checkmarks above) versus invalid inferences (noted by a 'x')? @xcite ) between event mentions. others employed probabilities from a language model over event sequences @xcite @xcite @xcite , or other measures of event co-occurrence @xcite . in this work we ask: do measures rooted in cooccurrence best capture the notion of whether one event should follow another in a script? we posit that it does not, that while observed correlations between events indicate relatedness, relatedness is not the only factor in determining whether events form a meaningful script. consider the example of @xcite : hurricane events are prototypically connected with events of donations coming in. likewise, hurricane events are connected to evacuation events. however, while donation and evacuation events are not conceptually connected in the same sense, there will exist strong statistical associations between the two. figure 1 provides a second example: eating popcorn is not conceptually associated with crying, but they might co-occur in a hypothetical corpus describing situations of watching a sad movie. what do strict co-occurrence measures miss? in both examples the 'invalid' inferences arise from the same issue: an event such as eating popcorn may raise the probability of the event crying, but it does so only through a shared association with a movie watching context: the increase in probability is not due to the eating popcorn itself. in other words, what is lacking is a direct causal effect between these events, a quantity that can be formally defined using tools from the causal inference literature @xcite . in this work we demonstrate how a measure based on causal effects can be derived, computed, and employed for the extraction of script knowledge. using crowdsourced human evaluations and a variant of the automatic cloze evaluation, we show how this definition better captures the notion of scripts as compared to prior standard measures, pmi and event sequence language models. code and data available at github.com/weberna/causalchains. ## motivation does that fact that event e 2 is often observed after e 1 in the data (i.e. p(e 2 |e 1 ) is "high") mean that e 2 prototypically follows e 1 , in the sense of being part of a script? in this section we argue that observed associations are not sufficient for the purpose of extracting script knowledge from text. we argue from a conceptual standpoint that some notion of causal relevance is required. we then give examples showing the practical pitfalls that may arise from ignoring this component. finally, we propose our intervention based definition for script events, and show how it both explicitly defines a notion of 'causal relevance,' while simultaneously fixing the aforementioned practical pitfalls. ## method we would like to compute the effect of forcing an event of a certain type to occur in the text. the event types that get the largest increase in probability due to this are held to be 'script' events. computing these quantities falls within the domain of causal inference, and hence will require its tools be used. there are three fundamental steps in causal inference we will need to work through to accomplish this: (1) define a causal model: identify the variables of interest in the problem, and define causal assumptions regarding these variables, (2) establish identifiability: with the given model, determine whether the causal quantity can be computed as a function of observed data. if it can, derive this function and move to (3) estimation: estimate this function using observed data. we go through each step in the next three subsections. to best contrast with prior work, we use the event representation of @xcite and others @xcite . a description of this representation is provided in the supplemental. ## experiments and evaluation automatic evaluation of methods that extract scriptlike knowledge is an open problem that we do not attempt to tackle here, foot_7 relying foremost on crowdsourced human evaluations to validate our method. however, as we aim to provide a contrast to prior script-induction approaches, we perform an experiment looking at a variant of the popular automatic narrative cloze evaluation. ## related work our work looks at script like associations between events in a manner similar to @xcite , and works along similar lines @xcite . related lines of work exist, such as work using generative models to induce probabilistic schemas @xcite @xcite , work showing how script knowledge may be mined from user elicited event sequences @xcite , and approaches take advantage of hand coded schematic knowledge @xcite . the cognitive linguistics literature is rich with work studying the role of causal semantics in linguistic constructions and argument structure @xcite @xcite , as well as the causal semantics of lexical items themselves @xcite . work in the nlp literature on extracting causal relations has benefited from this line of work, utilizing the systematic way in which causation in expressed in language to mine relations @xcite @xcite @xcite . causal relations between events that are in some way explicitly expressed in the text (e.g. through the use of particular constructions).taking advantage of how causation is expressed in language may benefit our causal model, and is a potential path for future work. ## conclusions and future work in this work we argued for a causal basis in script learning. we showed how this causal definition could be formalized and used in practice utilizing the tools of causal inference, and verified our method with human and automatic evaluations. in the current work, we showed a method calculating the 'goodness' of a script in the simplest case: between pairwise events, which we showed still to be quite useful. a causal definition is in no way limited to this pairwise case, and future work may generalize it to the sequential case or to event representations that are compositional. having a causal model shines a light on the assumptions made here, and indeed, future work may further refine or overhaul them, a process which may further shine a light on the nature of the knowledge we are after. 1800 examples and a dev set of 200 examples. we train the model in a way similar to above, keeping the best validation model (at epoch 28).