POQ ue: Asking Participant-specific Outcome Questions for a Deeper Understanding of Complex Events (2022)

Abstract: Knowledge about outcomes is critical for complex event understanding but is hard to acquire.We show that by pre-identifying a participant in a complex event, crowdworkers are ableto (1) infer the collective impact of salient events that make up the situation, (2) annotate the volitional engagement of participants in causing the situation, and (3) ground theoutcome of the situation in state changes of the participants. By creating a multi-step interface and a careful quality control strategy, we collect a high quality annotated dataset of8K short newswire narratives and ROCStories with high inter-annotator agreement (0.74-0.96weighted Fleiss Kappa). Our dataset, POQUe (Participant Outcome Questions), enables theexploration and development of models that address multiple aspects of semantic understanding. Experimentally, we show that current language models lag behind human performance in subtle ways through our task formulations that target abstract and specific comprehension of a complex event, its outcome, and a participant’s influence over the event culmination.

## introduction situations that people experience or describe can be complex, and developing a computational understanding of these situations is not straightforward. consider the short narrative from fig. 1 : after a decade as renters, [the brofmans] were finally able to buy a small house here four years ago. but if the argentine government yields to @xcite pressure to rescind emergency legislation meant to protect ordinary families like the brofmans, the couple stand to lose their home and the $32,000 they have paid for it so far. across multiple, interwoven events with multiple participants, this narrative describes part of the process of losing one's house. a possible ending (the loss of a home) is suggested, which is the result of a confluence of these events. this ending can be semantically grounded in various changes of state that the participants experience, though note how the use of counterfactual considerations, conditional statements ("if the argentine government..."), and varying levels of certainty over whether events have actually happened (e.g., realis vs. irrealis) contribute to the difficulty in understanding this complex event @xcite . knowledge about how event outcomes affect individual participants can help identify salient events in a narrative, fill in implicit missing information @xcite and chain events that lead to improved understanding of complex events @xcite . to infuse ai models with similar knowledge, narrative comprehension research has focused on learning event relationships @xcite @xcite , using temporal @xcite , causal @xcite and discourse @xcite relationships in text. however, as @xcite and @xcite argue, for a more useful, generalizable, and robust comprehension, we need to take a holistic view of complex events. in this paper, we tackle an understudied notion of this holistic view and examine knowledge of post-conditions based in states to support inferences of the form "who did what to whom and with what end result." a core insight we make is that viewing complex events through the lens of a single participant at a time, either from an agent (how the participant affects others) or a patient (how the participant is affected) view, can help mitigate the complexities we have discussed. in this we build on cross-disciplinary research that shows that humans mentally structure events along single participants @xcite , and that participant-based event and outcome analysis improves complex event understanding @xcite . we figure 1: our approach to understanding state change outcome of complex events. our four step annotation process involves describing abstractly what a story is about; writing an endpoint for the story; identifying and describing the changes that are a result of the endpoint, and identifying the salient sub-events that lead or support to those changes. to mitigate this complexity, we focus annotator's attention on one particular participant, and how that participant either causes or experiences the identified changes. this process has resulted in 4k annotated documents. we present poque, a dataset with postconditional knowledge about complex events. we identify cumulative outcome-oriented endpoints of the stories caused by related events and the consequences or post-conditions of those events as statebased changes in participants. seen in fig. 1 , in a storyline involving a participant ("the brofmans"), we identify an ending outcome for the complex event (the endpoint, "the brofmans might lose their home and payments"), with salient events that lead to this (factors leading to the endpoint). we relate a participant's involvement in the complex event (as a "patient" who "very likely" experienced the ending outcome) and the changes of state occurring as a result of the complex event (the changes in possession and location experienced by the brofmans and other families, and the change in possession by the argentine government and imf). to facilitate high quality annotations we designed a multi-stage crowd sourcing solution to acquire, monitor, assess and curate annotations at scale. we collected 7772 annotations across 4001 stories and assessed a random 1545 annotations (20%) in a multistage pipeline to obtain a highly curated test set. using poque, we test current lan-guage models on reasoning about complex events in narratives: we formulate challenge tasks to identify and generate post-conditions from a story, and evaluate how well trained models predict a participant's involvement in enabling a complex event. we summarize our contributions as follows: (1) we introduce a new annotation scheme focusing on complex events from the point of view of a single participant. (2) we create a new dataset of complex events from three collections of everyday stories, using free form text to obtain insight into implied outcomes. (3) we obtain high quality annotations from crowd workers without the use of requester generated qualification tests. (4) we formulate challenge tasks aimed at evaluating the ability of language models to perform richer complex event comprehension, specifically: a) generating a process summary of the complex event b) generating an endpoint of the complex event, c) generating the outcome of a complex event based on a participant's semantic role d) identifying a participant's involvement in a complex event, and e) generating post-conditions or changes caused by a complex event. our dataset and code are publicly available at https://github.com/saiumbc/poque . ## related work narrative texts communicate experiences and situations by connecting related events @xcite through events involv-ing participants @xcite @xcite . previous works, viewing narratives as sequences of events, annotated event pairs for event coreference, temporal, and causal relationships @xcite @xcite . newer works have studied event groups using predicate hierarchies @xcite and temporal graphs @xcite . however, these approaches focus on event-event relationships, without diving deeply into participant or entity analysis. unsupervised methods assume narratives are coherent and learn partially ordered event chains @xcite or sub event relationships @xcite but these are limited to what occurs in the text itself, which can lead to well-known issues of bias or evaluation limitations @xcite . @xcite obtain crowd annotations of causal relationships between events and assess their quality by relating them to expert annotations. peko @xcite uses crowd annotations of precondition relationships and fine tunes a language model for finding such relationships. however, both works limit their study to event pairs in short text snippets. the ester dataset @xcite consists of more comprehensive relationships in a story, though limited to within-text (i.e., stated) mentions of events only. glucose @xcite provides elaborate causal relationships for several event dimensions for each event in a story. however, in contrast to our effort, these works address direct causal relationships between within-text events and do not focus on participants. understanding complex events has long attracted cross-disciplinary attention. for example, theoretical linguistics and cognitive science work has shown that humans understand a narrative text using simulative inference @xcite @xcite . prior work has also shown how observing participants' events and the resulting consequences can lead to improved understanding of events @xcite . ## knowledge representation as an underlying motivation for our efforts, we posit that for language models to be able to reason about complex events from narratives, they should be able to identify a likely ending of that complex event, component events that lead to that ending, and the state changes that result from that ending. however, this type of knowledge is complex and has been computationally understudied, leading to a scarcity of sizable datasets. in our efforts to correct this, we appeal to classic, cognitively-and linguistically-backed results. first, inspired by the idea from @xcite ) that text comprehension involves reducing relevant details into an abstract coherent semantic text, our targeted annotations include an abstract high level summary and the minimal set of salient events that make up the complex event. second, we extend the idea of thematic roles for verbal arguments @xcite to a generalized semantic role for the complex event. specifically, dowty showed that easily verifiable characteristics and properties, such as volitional participation in an event or whether a participant underwent a change of state because of a particular event, can be used to define predicate-level prototypical semantic roles. inspired by this, we characterize the roles of complex events through the intentional engagement in changes of state of participants. as such, our targeted annotations account for both the intentional involvement of the participants in the complex event and the cumulative impact of all the events that make up the complex event. story and participant stories in our dataset are either a rocstory or heuristically salient portions of newswire (first 100-150 tokens). 1 for more information on story processing see §4.1. we define a participant as an entity that was mentioned several times in the story. see §4.1 for more on participant selection. multiple entities are considered a single participant if these entities are mentioned together and participate together in all the events. in fig. 1 , "the brofmans" are a participant. ## dataset we selected stories from three narrative english language datasets -the ester dataset @xcite , the roc stories dataset @xcite and passages from the annotated new york times newswire dataset @xcite . we selected these given the prominence the underlying documents have in the broader nlp community (the ester documents are a subset of the tem-peval3 (te3) workshop dataset @xcite . we included roc stories because they often contain a single situation with mostly salient information. we noticed these stories help crowd workers easily focus on salient events, providing cues for factors and state changes. meanwhile, es-ter and the selected newswire stories provide a variety of complex situations and discourse text. additionally, by selecting subsets of these wellknown datasets, we hope that future efforts may be able to aggregate our annotations with existing ones, enabling richer phenomena to be examined. ## tasks for state change knowledge based upon our collected dataset, we propose several tasks. these tasks are designed to test various aspects of comprehension involving complex events, their participants, and outcomes. ## state change benchmark experiments we benchmark the performance of current encoderdecoder transformer language models, t5 @xcite and bart @xcite , which are effective for both text generation and classification. we compare fine-tuned base and large foot_1 models with multiple automated metrics and crowd sourced human evaluation. we use bootstrap for calculating statistical significance via the mlxtend library @xcite . ## effect of discourse text on models we study the effect of discourse text on model generations of endpoint descriptions using the two story types we annotated: rocstories and newswrire stories. rocstories are simpler with short, concise and focused salient events, while newswire are more complex, containing more text not always salient to the complex action we annotated. we wish to answer the following questions: we fine-tuned bart and t5 base models separately on rocstories vs. newswire, and evaluated them on test sets for both story types. we calculated human scores from the endpoint evaluation hit. from ## conclusions we have argued that a deeper understanding of complex events can be achieved by examining their cumulative outcomes, grounded as changes of state. by focusing on a specific participant in a complex event, and a broad notion of its semantic role, we developed a crowdsourcing protocol to obtain 7.7k annotations about complex events and participant state change across 4k stories. we validated 20% of the annotations, with high inter-annotator agreement. we have formulated five challenge tasks that stress model's understanding of story outcomes, state changes and complex event understanding. our evaluations suggest that additional modeling advances are needed to achieve this understanding; we hope that our dataset spurs this future work.