PASTA : A Dataset for Modeling PA rticipant STA tes in Narratives (2023)

Abstract: The events in a narrative are understood as a coherent whole via the underlying states of their participants. Often, these participant states are not explicitly mentioned, instead left to be inferred by the reader. A model that understands narratives should likewise infer these implicit states, and even reason about the impact of changes to these states on the narrative. To facilitate this goal, we introduce a new crowdsourced English-language, Participant States dataset, PASTA. This dataset contains inferable participant states; a counterfactual perturbation to each state; and the changes to the story that would be necessary if the counterfactual were true. We introduce three state-based reasoning tasks that test for the ability to infer when a state is entailed by a story, to revise a story conditioned on a counterfactual state, and to explain the most likely state change given a revised story. Experiments show that today’s LLMs can reason about states to some degree, but there is large room for improvement, especially in problems requiring access and ability to reason with diverse types of knowledge (e.g., physical, numerical, factual).1

## introduction understanding narrative text requires forming a coherent representation of the scenario, including filling in details that are unstated in the text. one type of detail that is usually not mentioned is the state of its participants 2 (e.g., ''she unlocked the door'' implies the possession state that ''she has a key''). the reader easily infers these implicit states and their causal relationships with the narrative's explicit events, creating a detailed mental picture of the described world that is only partially observable from the text. many cognitive theories have been proposed to capture aspects of this in their representations, such as scripts @xcite , frames @xcite , and state/time formalisms @xcite . without committing to any one particular formal theory, this paper adds a theory-agnostic resource to test such theories by listing implicitly assumed participant states in simple narratives. consider the story in figure 1 from the roc-stories corpus @xcite . humans create a detailed mental representation of this spilled-soda scenario by inferring its commonsense states. in this story, using our commonsense knowledge about emotions and habituals, we can infer from the first two lines that kate's mother liked keeping her car clean (a state about kate's mother). similarly, based on our physical commonsense of a lid, i.e., that lids prevent spilling, we can also assert from the spill that the soda's lid was loose (a state about the soda). we can also reason about the likely change to the story due to a counterfactual state, i.e., if the soda's lid was tight, then most likely the soda wouldn't spill. to the best of our knowledge, no such resource exists that captures this kind of participant state knowledge. to capture this type of commonsense knowledge needed to understand and reason about participant states in narratives, we introduce pasta, a crowd-sourced dataset in english. as shown in figure 1 , for a given story s, pasta provides a participant state α that is likely to be inferred from s, a perturbation state α that is counterfactual to s along with the minimal changes to s that are required to make α likely to be inferred from the revised story s . pasta includes 10,743 instances of these story/state/ counterfactual/revision tuples. with this new dataset, we hope to enable models to make the kinds of state-based inferences that move beyond surface text understanding and lead to deeper reasoning. to this end, we describe three new state-based reasoning challenges with pasta which are illustrated in figure 2 . the first is story state inference: given a story and an inferred participant state, predict if the state is likely to be inferred from a given set of sentences in the context of the story. we formulate this as a binary classification task, and we create contrastive examples for training and evaluation purposes to guard against artifact-based reasoning. this can be seen as a form of textual entailment, a capability useful for applications such as question answering @xcite , claim verification @xcite , etc. the other two challenge tasks are generative. the second task, story revision for counterfactual states, measures the ability to reason about counterfactuals. given a story and a counterfactual state (i.e., a state that is not consistent with the story), the task is to revise the story such that the counterfactual state is now likely to be inferred from it. these types of counterfactual revisions serve as a test of reasoning @xcite and can support interactive story generation tasks @xcite . the third task, state change generation, requires the model to take a story and its perturbed version as input and then generate the two corresponding states (e.g., 'lid was loose' and 'lid was tight') that explain the differences in the way they unfold. from an application perspective, generating the underlying states that account for the differences between two narratives can assist with fake news detection using reliable sources @xcite @xcite and information fact checking @xcite . these three challenge tasks require a unique combination of commonsense abilities, thus helping to evaluate models on reasoning and knowledge capacity. these tasks require not only basic entailment ability, but also knowledge (numerical, factual, physical, etc.) and broader narrative understanding. having just one of these abilities will not suffice. to evaluate current models for these capabilities, we benchmark the llms t5 @xcite , bert @xcite , roberta @xcite , and gpt3 @xcite . for the generative tasks, we evaluate model performance through extensive human and automatic evaluations. the results show that, though these models can reason about states to some degree, there is substantial room for improvement on all tasks, suggesting avenues for future research. ## related work there are many formal theories on mental states and reasoning. the seminal work by @xcite introduced scripts as a way to structure knowledge about stereotypical event sequences with their participants. frames @xcite and theories of time @xcite provide related views. this paper does not commit to a formal theory, instead providing a challenge dataset to test aspects of them. statistical work on events @xcite @xcite @xcite curated event knowledge in an unsupervised manner from large text corpora. this paper augments their view with state-based knowledge about event participants. more recent work by @xcite , @xcite , and @xcite capture everyday inferential knowledge associated with an action performed by someone. this knowledge is organized through a fixed set of relationship classes between action and inferences. a subset of these classes is about participant mental states, but this commonsense knowledge is non-contextual in nature. in contrast, our work requires inferring commonsense knowledge about participant states in the context of coherent narratives. most similar to our work is the time-travel dataset by @xcite . it includes the counterfactual story rewriting task to edit a short story based on a counterfactual context. the authors insert an explicit counterfactual at a fixed position (2 nd sentence) in the story, and the revision task is then conditioned on this observed change. it is a language modeling generation task. in contrast, our work introduces unobserved counterfactual outside of the story's text, and the revised story must be generated with deeper state-based reasoning. this introduces additional complexity for the revision task. also, time-travel requires the revisions to be restricted to the story ending, which cannot be assumed in our setting. our states can be inferred from any part of the story. @xcite proposed tasks that predict a plausible hypothesis for two given observations, and curate a dataset for the same. their work mainly focuses on what happened in-between? type of inferences. @xcite introduced the glucose dataset, which focuses on several types of causal knowl-edge that are required to explain a causal event in narrative text. neither of these focuses entirely on implicit states (some glucose annotations are relevant, but not directly so), and neither addresses story revision in the face of counterfactual changes. recent work on understanding entity states has mostly focused on tracking entity state change in text. @xcite introduced propara, which captures physical state changes (creation, destruction, and movement), @xcite proposed the task of tracking ingredients in cooking recipes, and @xcite tracks the emotional reactions and motivations of characters in simple stories, for a fixed/small set of attributes. @xcite introduced the wiqa dataset for analyzing the effect of perturbing a process described by a procedural text on the elements (entities, events, etc.) of the text, as an influence graph of the process. however, the influence graph was assumed to have a fixed causal structure. it captured a very limited set of cause-effect relationships obtained as a result of analyzing perturbations that either accelerated or decelerated the main outcome of the process. @xcite introduced a dataset for tracking state changes in procedural text as a set of state change tuples of entity, attribute, beforestate, and after-state for each step of the process. the elements of the tuples were in free-form text instead of belonging to a set of pre-defined categories. our work differs from the above in several key ways: (i) participant states are unstated, (ii) participant state inferences do not depend on sentence ordering assumptions, (iii) state perturbations affect the entire discourse of the narrative, and (iv) captures how the participant states change between an original and a perturbed narrative. ## state-based reasoning tasks inferring each component of a pasta 4-tuple requires a different commonsense reasoning ability about a participant's state in a narrative, which enables us to use pasta to test models for these abilities. as illustrated in figure 2 , we introduce three pasta tasks, one classification and two generative, each of which can be used to evaluate current nlp models for the capabilities required to understand a participant's state in a narrative text. in the subsections below we provide the motivation and formal task definition for each task. ## experimental setup to establish modern baselines and measure their performance, we built benchmark models from gpt3, t5, bert, and roberta. this section describes how each was setup for the three tasks. ## results and analysis we now analyze the performance of these recent language models on the three pasta tasks. ## discussion here we discuss the main challenges and error analyses that highlight areas for future work. ## conclusion in this work, we introduced a new resource, pasta, that captures unstated commonsense knowledge required to understand and reason about participant states in a narrative. pasta opens the door to developing more complex reasoning abilities, especially those that require access to implicit information. we described three pasta reasoning tasks, one classification and two generation, that test for different aspects of state-based reasoning. this work shows that with careful crowdsourcing and contrastive design we can obtain a high-quality dataset that can be used to evaluate deeper reasoners. benchmarking results suggest that pasta tasks are not within the reach of current large sized models, as of yet, and encourages future research in modeling commonsense knowledge with states.