E v E nt S R ea LM : Event Reasoning of Entity States via Language Models (2022)

Abstract: This paper investigates models of event implications. Specifically, how well models predict entity state-changes, by targeting their understanding of physical attributes. Nominally, Large Language models (LLM) have been exposed to procedural knowledge about how objects interact, yet our benchmarking shows they fail to reason about the world. Conversely, we also demonstrate that existing approaches often misrepresent the surprising abilities of LLMs via improper task encodings and that proper model prompting can dramatically improve performance of reported baseline results across multiple tasks. In particular, our results indicate that our prompting technique is especially useful for unseen attributes (out-of-domain) or when only limited data is available.

## introduction modeling the effect of actions on entities (event implications) is a fundamental problem in ai spanning computer vision, cognitive science and natural language understanding. most commonly referred to as the frame problem @xcite , early solutions relied on a set of handcrafted rules and logical statements to model event implications. however, such methods require substantial manual effort and fail to generalize. more recently, modeling event implications has reemerged under the guise of common sense reasoning within nlp @xcite @xcite and action anticipation in computer vision @xcite . predicting event implications is a particularly difficult problem due to the complex nature of language and implicit knowledge required to answer such questions. for example, if we are given the sentence the mug fell on the floor and we want to determine whether the mug is broken, we need to know of several facts such as the material of the mug, the fragility of ceramics, the hardness of the floor, etc. and also how to combine these facts together to reason whether the mug will break or not. none of this knowledge is explicitly stated, instead being classified as common sense knowledge, and is traditionally acquired from observations or interactions with objects and the environment. core to this line of work is the assumption that events can be learned via language, not depending on other forms of perception. to explore the utility of other modalities and interaction, @xcite train a language model to predict physical changes in a virtual environment. while intuitively necessary @xcite , in this work we show that the purported limitations of languageonly models are not always well founded. key to their success (or failure) are (1) how we use the language models, and (2) the difficulty of the task domain and dataset. we find that the difficulty of the task is often a stand-in for whether reasoning is required. others have also noted that despite the tremendous gains in nlu made possible by large language models (llm), they still stumble when reasoning is required @xcite . if reasoning can be codified as patterns, we are presented with two new challenges: (1) can we test pattern acquisition via benchmarking generalization, and (2) how can new patterns or context be provided to the model? the nascent field of "prompting" @xcite @xcite hints at a possible approach for humans to encode novel reasoning patterns for models, however the best structure and the amount of information to convey via a prompt for a given task still remain open questions. this work makes three contributions to the literature of event implications. first, we show that language by itself provides enough information to predict event implications in current datasets, without the need of a physical interaction model. second, we establish the difficulty of the problem and limitations of current models by showing extreme differences in performance across different datasets: piglet @xcite , based on a virtual environment, and openpi @xcite , based on procedural text from wikihow. third, we explore how different prompting techniques affect model performance in terms of their information content and model nature. finally, we discuss the generalization properties of our models to unseen attributes (out-of-domain) and how this shows their ability to extract implicit reasoning mechanisms. ## related work related work in commonsense follows two directions: (1) predict event implications or track entity changes, and (2) use commonsense knowledge about events and their implications as necessary intermediate steps in reasoning. research that directly studies event implications mostly explores causality between social events and emotional states, based on social norm expectations @xcite @xcite @xcite . @xcite study specific linguistic phenomena such as contradiction and negation, while @xcite study the role of social biases and predicting implications of social events. although this line of research highlights the difficulty of predicting cause-effect relations, social scenarios are typically ambiguous and require knowledge of event chains. for example, in order to answer whether x gives a gift to y implies that x hugs y, we must be aware of the relation between x and y, their personalities, and the social context. on the other hand, event implications as physical changes of state of entities are, mostly, objective and depend on simple relations that a model could know a priori (e.g., the material of a mug), allowing us to isolate and study the reasoning abilities of a model. closer to our task is the prediction of physical implications of events. this problem often takes the form of entity changes in procedural text, such as in cooking recipes @xcite or wikihow articles @xcite . however, most datasets primarily focus on changes in location compared to other attributes, such as propara @xcite and babi @xcite . modeling approaches in both areas of commonsense explore the generation of explanations in a multi-task setting @xcite , the use of external knowledge graph @xcite , and automatic knowledge base construction to keep a representation of the state of the world and generate novel knowledge @xcite @xcite . the second type of commonsense reasoning includes question answering tasks that assume knowledge of commonsense relations and their implications on the context. this line of work includes short questions, such as openbookqa @xcite , commonsenseqa @xcite , swag @xcite and copa @xcite , or questions based on a provided document @xcite or knowledge base @xcite . ## task and datasets the problem of predicting event implications can be formulated in several ways, with varying levels of difficulty. for example, @xcite generate triplets of entity, attribute, post-state given some context, while @xcite are given an entity, attribute, pre-state, and context, to only predict the post-state of the entity. our task follows a similar formulation to piglet, where the model is given a context (i.e., a small paragraph followed by an action-sentence), an entity of interest and a list of attributes. then, the model needs to determine whether a changeof-state occurred for the entity with respect to the given list of attributes (see figure 1 ). however, unlike @xcite , we do not use the prestate encoding of the entity, instead we assume that the relevant information is better conveyed through the natural language description of the context. ## methodology next, we introduce our prompting techniques, which vary with respect to per-instance information content. each technique is tested with different llms and fine-tuning methods. the goal of each prompting mechanism is to show how model performance and generalization vary based on the infor-mation conveyed in our queries. our study focuses on four prompting methods depicted in figure 2 : zero-prompt, single-attribute, multi-attribute, and a variant of the latter, the k-attribute prompt. our approach builds on literature demonstrating benefits in using prompting to distinguish different tasks, when a model is trained in a multi-task setting @xcite . in our study, however, we explore how to use prompts as a medium to convey the task-specific information that a model must know in order to solve the task, similar to how one would ask a human. to the best of our knowledge, we are the first ones to demonstrate advantages and disadvantages of different ways to codify intermediate steps required for reasoning via prompting and use them to study llms' understanding of event implications. ## experiments & results our task is a multi-label classification where, given some context and an entity of interest, we need to identify which attributes change. due to the significant label imbalance, in our experiments we report micro-precision, recall, and f1 for the positive instances, across labels. in addition to these metrics, we measure per-attribute precision, recall and f1 for both datasets (details in subsection a.4). ## discussion we further study the models' behavior with respect to the type of attributes they see and their generalization properties. this analysis serves to uncover advantages and disadvantages of each technique and suggest promising methods for future work to enhance both model performance and robustness. for all our experiments we use open pi. due to its greater diversity of attributes and larger size, it is a better candidate than piglet to analyze the limitations of the models. ## conclusion predicting physical changes due to events is a challenging problem for current models, especially in out-of-domain or limited-data scenarios. we show that, by using proper task formulation, llms can learn physical event implications even without physical interactions. future work should explore the question of whether physical interactions are necessary in more complex and realistic settings, by (1) providing more challenging datasets that test the model limitations, and (2) ensure a fair comparison of the language-only baselines. furthermore, we show that the performance of a llm may significantly vary based on how we use it, and, overall, llms can benefit from: (1) verbalizing the attributes, (2) varying the prompt information content across instances, and (3) querying multiple attributes in the same instance. by following these guidelines, we show significant improvements in unseen attributes and attributes of low-frequency. last, our error analysis and discussion sections provide useful insights for future work, with respect to prompt content and shortcomings of the current datasets that study physical event implications.