Modeling Preconditions in Text with a Crowd-sourced Dataset (2020)

Abstract: Preconditions provide a form of logical connection between events that explains why some events occur together and information that is complementary to the more widely studied relations such as causation, temporal ordering, entailment, and discourse relations. Modeling preconditions in text has been hampered in part due to the lack of large scale labeled data grounded in text. This paper introduces PeKo, a crowd-sourced annotation of preconditions between event pairs in newswire, an order of magnitude larger than prior text annotations. To complement this new corpus, we also introduce two challenge tasks aimed at modeling preconditions: (i) Precondition Identification – a standard classification task defined over pairs of event mentions, and (ii) Precondition Generation – a generative task aimed at testing a more general ability to reason about a given event. Evaluation on both tasks shows that modeling preconditions is challenging even for today’s large language models (LM). This suggests that precondition knowledge is not easily accessible in LM-derived representations alone. Our generation results show that fine-tuning an LM on PeKo yields better conditional relations than when trained on raw text or temporally-ordered corpora.

## introduction recognizing logical connections between events in text is important for comprehensive document understanding and to improve global coherence in language generation systems. there is a rich body of work in identifying relations between textual events which covers causation @xcite , temporal relations @xcite , textual entailment @xcite , and discourse relations @xcite . in this work, we focus on the precondition relation, which offers a general view of why certain events occur together in the world. this is not easily deduced from other event-event relations. temporal ordering systems can sequence the order in which events occurred @xcite @xcite but can't explain why they occurred at all. which events in a sequence were by chance, and which were required? textual entailment identifies event paraphrases @xcite and some causation @xcite , but their view misses the broader look at enabling events like preconditions. let the following serve as an example: i heard a bird sing above as i turned the key in the door. it opened with a push. you can sequence these four events in order, but an ordering does not understand the why of the situation. one of these events (sing) is clearly not relevant to the door opening. how do we know that turning the key is a precondition to opened and not push? turning the key usually doesn't cause the door to open (perhaps on some doors, but here a push was needed). turning is simply a precondition. causation and entailment do not apply to turn either. preconditions thus provide a unique and still fine-grained understanding of this situation. how do we build models that can recognize (and learn from) this type of common-sense knowledge in text? do language models trained on vast amounts of data already capture it? since there are no large scale datasets that can effectively answer these questions, we introduce peko, the precondition knowledge dataset. we also introduce two tasks -one aimed at recognizing preconditions in text, and the other at generating precondition events for any given target event. the core contribution in this paper is this new publicly available crowd-sourced peko dataset. it consists of 28,948 event pairs annotated with precondition relations. we will first present our working definition of preconditions, and then discuss how to practically get crowd workers to identify them in text. we provide analysis of the new corpus and compare it against other existing corpora. in addition to the corpus, this paper proposes two new challenge tasks. the first is a traditional classification task on the corpus itself. we thus address critical questions of how to model precondition knowledge. for instance, do today's large language models (e.g., bert or xlnet) already capture precondition knowledge, and how do they perform on a precondition prediction task? second, does textual context assist in precondition prediction? we experiment with varying levels of context and show that identifying preconditions requires careful modeling of the context. the second proposed task is a precondition generation evaluation: models must generate necessary preconditions for a given target event. this is a test for how well models can reason about the necessary preconditions for a given situation, which is a useful capability for story generation and learning generalized scripts. we show how peko can be used to train (fine-tune) standard generative models, such as gpt-2, for this task. empirical results show that fine-tuning on the peko-derived training set generates at least twice as many preconditions as compared to training on general instances. all code and data are available at https:// stonybrooknlp.github.io/peko/. ## related work there has been a vast amount of research on extracting different types of relations between events including temporal @xcite , causal @xcite , and paraphrasal relationships @xcite , but relatively less research into precondition relationships. one of the early definitions and computational use of preconditions comes from the strips program @xcite . preconditions were defined as a set of conditions that must be met in order for the action (event) to be allowed to take place. later work focused on aggregating precondition knowledge for a small class of action words, leveraging framenet and a text corpus to generate candidate precondition words using a pmi-based heuristic @xcite . using small amounts of labeled data, they use hand-crafted pmi and wordnet based features to learn a svm-based classifier that scores preconditions for a given action. @xcite learned domain-specific preconditions from written instructions for the game of minecraft. the instructions are procedural and well suited for identification. these mostly target preconditions that are eventstate relations as opposed to our goals of textual event-event identification. atomic @xcite is a related crowdsourced dataset of event-event relations, where given a simple target event (verb phrase and its arguments), crowd workers provided various types of common-sense knowledge. this included 'need' events analogous to our precondition events for a target. the main difference is our work grounds both target and precondition events in news text, whereas atomic elicits general world knowledge, a complementary approach with different tradeoffs. interestingly, we find that the precondition relations learnt from textually grounded news events generalize to story events in atomic for our generation task. annotated text corpora three existing datasets capture some form of precondition knowledge in their annotations: the rich event description (red) dataset (o' @xcite , caters @xcite , and event storyline @xcite . these are generally too small for learning text classifiers as we briefly describe now. red is the most directly related, created to model a broad set of event-event relations in news. preconditions are not their sole focus, though, so this dataset only contains ~1000 precondition instances. caters shares a similar problem to red. it has an enables relation similar to precondition, but since the domain is 5-sentence short stories and preconditions aren't the main focus, it only has ~400 instances. the event storyline dataset is small in size too, but also doesn't have a precise precondition relation. the dataset instead has ris-ing_action that includes preconditions in its definition, but the same label captures other concepts like subevents and entailment. there are ~5000 instances, but only a fraction are preconditions and it is not possible to separate them out. this paper is thus unique to prior work by annotating grounded written text at a scale large enough to enable machine learning solutions. this enables our target tasks: text classification and generation. ## preconditions as relations our goal is to develop a resource that can help models reason about the necessary preconditions for events mentioned in text. this is useful for planning towards a goal, explaining how a certain situation came about, and predicting what future events are plausible. we make two important design choices in building such a resource: grounding -the resource is grounded in text, particularly over events in the news domain, and framing -we construct the resource with preconditions framed as event-to-event relation pairs in a specific context. grounding: we ground the resource to text so that we can leverage the full context of the events, and we choose the news domain due to its common use in other event-related tasks such as event extraction, schema generation, and temporal reasoning. framing: broadly speaking, preconditions specify what must exist/happen before something else can exist/happen @xcite . it is natural to think of a precondition as a state of the world that must be satisfied for an event to happen i.e. a state-event relation. however, the state of the world is hard to circumscribe for most real world events, and more importantly the precondition state is often left unsaid in a story. rather, the author will more often mention an event from which it follows that the precondition state is satisfied. thus, it makes sense to frame preconditions as relations between two events described in their specific textual context. we first present a formal definition based on this notion and then describe a crowdsourcing methodology for obtaining this knowledge at scale. definition: given a target event mention t and a candidate event mention p, we assert p is a precondition event for t if p is necessary for t to happen i.e., t likely would not have occurred without p, in the current text context. using the example of opening a door from the introduction, turning the key is a precondition event (for opening the door) because it results in a state where the door is unlocked. the opening event cannot occur without such a state. importantly, we do not define a precondition event as an absolute requirement for the target (the door opening) to occur in all scenarios. however, we do require that the target event likely would not have occurred in the current context. this allows another story with an alternate event, such as "i picked the lock". both picking-lock and turning-key are preconditions in their own story contexts. strict logicians might take issue, but language understanding requires a looser definition that uses likelihood of occurrence when interpreting real-world scenarios. ## preconditions dataset this section describes our methodology to annotate news articles with the previous section's definitions. one problem with annotating preconditions in text is the large number of event mentions in each article, which means annotation of all possible event pairs is infeasible. the temporal community has struggled with this same dilemma @xcite . we address the question of which pairs to annotate with two approaches. first, instead of attempting a dense annotation of few articles, we sub-sample candidate pairs of events across many articles. second, we use an automatic temporal relation classifier to filter pairs by identifying possible candidates. we then ask crowd-workers to annotate the resulting pairs for preconditions. ## peko tasks and evaluation having created the peko annotated corpus, we now propose two tasks that test for the ability to recognize and generate preconditions in textual contexts. here we describe evaluations to benchmark the performance of current models on these tasks and to better understand the challenges involved. ## conclusions knowing what conditions are necessary for an event to happen is critical for understanding and reasoning about events mentioned in text. in this work, we address the lack of a large scale resource for learning precondition knowledge about events. our crowdsourcing methodology yielded more than 10,000 precondition event relations (and 20,000 negative examples) from news domain texts. we showed in both classification and generation that these relations are not readily accessible in distributional knowledge encoded by large language models, highlighting the challenges in learning common-sense knowledge from text. we also proposed two new challenge tasks based on peko and hope it helps drive further research into rich event understanding that touches a variety of areas from schema learning, information extraction, and even story generation.