SAGA : A Participant-specific Examination of Story Alternatives and Goal Applicability for a Deeper Understanding of Complex Events (2024)

Abstract: Interpreting and assessing goal driven actions is vital to understanding and reasoning over complex events. It is important to be able to acquire the knowledge needed for this understanding, though doing so is challenging. We argue that such knowledge can be elicited through a participant achievement lens. We analyze a complex event in a narrative according to the intended achievements of the participants in that narrative, the likely future actions of the participants, and the likelihood of goal success. We collect 6.3K high quality goal and action annotations reflecting our proposed participant achievement lens, with an average weighted Fleiss-Kappa IAA of 80%. Our collection contains annotated alternate versions of each narrative. These alternate versions vary minimally from the “original” story, but can license drastically different inferences. Our findings suggest that while modern large language models can reflect some of the goal-based knowledge we study, they find it challenging to fully capture the design and intent behind concerted actions, even when the model pretraining included the data from which we extracted the goal knowledge. We show that smaller models fine-tuned on our dataset can achieve performance surpassing larger models.

## introduction understanding goals is central to human understanding of text @xcite . however, this is challenging, as slight variations in actions reported in text can lead to vastly different goals, achievement outcomes and future actions. consider the stories in fig. 1 , where the actions of manny (the participant) in the "original" story (top left) indicate his goal of saving a life, which he is able to achieve at the end of the story. however, small changes in the narrative can lead to vastly different inferences. consider the three alternative stories shown: in alternative 1 (top right), a different action points to manny having a different figure 1 : a participant's goal inferred from the actual story when applied to 3 alternative stories, drawn from the pasta dataset @xcite ; slightly varying actions in the stories lead to different goal achievement outcomes. while knowledge of participants' intentions and how they are impacted by situations is strategic for communication and language understanding @xcite @xcite , acquiring the necessary goal and plan knowledge is not straightforward. first, situations in narrative text can be very complex! this is partly due to the implied nature of described situations @xcite , the subjective nature of someone's understanding of a participant's goal @xcite , and our ability to understand a goal at varying levels of abstraction and specificity. second, subjectivity and incomplete tasks 3b & 3c: next action & explanation transference to s c / / / / / / / / task 4: generate a future plan to achieve tasks 5a, 5b & 5c: achievement with s a or s c participant goals in stories future actions that affect goal outcome and achievement achievement after s a satisfaction within s a figure 2 : goal reasoning inferences from our dataset formulated as benchmarking tasks. these consist of both generating a participant's goal and future actions after the story aimed at goal achievement and identifying goal applicability and achievement. tasks 1, 3a and 4 examine the generative understanding of goals, explainable future actions and plans. tasks 2, 3b, 3c and 5 examine discriminative understanding of applicability and achievement. our formulation ultimately supports inferences of the form "who achieved what and under what conditions," i.e., inferences useful in application areas that rely on broader reasoning, like claim verification or question answering. we consider the rationale for "why" a participant carries out certain actions (goal knowledge) and their plans to achieve their goal by considering "what-next" type of future actions and plans (prospective knowledge). to aid our study and future work, we use our framework to analyze alternative stories and how small variations in the narrative affect what inferences can be made. via a multi-stage annotation pipeline, we collected 6225 goal annotation sets (consisting of 886 "actual" & 951 "alternative" stories). from this rich data, we formulate multiple inference tasks of increasing difficulty, seen in fig. 2 , including what can be inferred about the goals of participants (tasks 1 & 2), future actions that affect the goal outcome (tasks 3 & 4) and achievement (task 5). we examine llms on these tasks; since models trained with human feedback have shown impressive performance on understanding human intent @xcite , we compare gpt versions 4, 3.5-turbo, and various sizes of the t5 family of models flan-t5 and t5 models. we use prompting and fine-tune models on our collected dataset to examine the differences with both both options and the benefits of finetuning smaller models. our results and analysis show the strengths and weaknesses of these various models. we find that pre-trained larger models generally perform better than smaller models and contain less factual errors. few-shot prompting is especially useful for tasks requiring within-story details. some of the stories that are part of our dataset are part of the pretraining data for flan-t5 @xcite ; we examine how models handle subtle changes in these stories and find that few-shot prompting and fine-tuning help correct these errors. overall, we find that even the larger models struggle to hone-in on the human intent behind a set of actions. fine-tuning and few-shot prompting improve smaller models to be competitive with, or surpass, the larger models. we summarize our contributions as follows: natural language understanding explored goal oriented reasoning through activities in first person narratives @xcite , locationbased actions in news-text @xcite , chaining and ordering of action plans from procedural text @xcite and schema construction @xcite . leveraging participants' goals, recently, @xcite examined llms' sequential reasoning capabilities. these works explore the procedural and sequential relationship between activities, plans and explicit goals. another line of research, commonsense based implicit question answering @xcite @xcite collected goals as answers; the goals are implied by the context but their level of abstraction is question-dependent. we collect implied goals at the highest level of abstraction from the actions and intentions of a volitional participant and from varied alternative situations. inferences in alternative situations: proposing the common sense natural language inference task, swag @xcite and hel-laswag @xcite , explored sentence completion in alternative situations. while these tasks examined counterfactual reasoning in discriminative models, timetravel @xcite studied counterfactual knowledge possessed by generative models for alternative story rewriting. pasta @xcite introduced implied states that stories depend upon and examined the tasks of state inference from alternative stories and story rewriting for alternative states. we use the alternative stories from pasta to examine changes in goal inference in alternative situations. @xcite ; @xcite used alternative stories to examine situation plausibility through procedurally tracking natural physical laws or participants' physical states; we examine goal plausibility through participants' intentional actions. ## decomposing goal understanding reasoning over goals has natural subjectivity and complexity. we handle this in a few ways. first, we borrow from cross-disciplinary research @xcite to reduce the subjectivity in goals by collecting overarching super-ordinate goals: a super-ordinate goal is a complex higher order goal that is achieved through several actions. this super-ordinate goal licenses the rationale behind all of a participant's actions in a situation. second, we focus on intentions, which can be viewed as a representation of planned actions that lead to achieving a goal @xcite @xcite . third, we capitalize on prior research that people are primed to think of future details for a goal that is yet to be accomplished @xcite @xcite and predict future events that help in assessing goal achievement and obtain plans (revised as necessary and when possible) that lead to achievement. ## dataset collection & evaluation we annotated roc stories @xcite that have corresponding alternative stories in the pasta dataset @xcite . we selected up to 5 "story participants," defined as volitional entities that were mentioned relatively frequently in the story. this yielded participantspecific story instances for a random 886 actual, unrevised stories from the pasta dataset. in appendix a.1 we provide additional details on the processing. we annotated all of the participantspecific story instances for these selected stories and all the alternative pasta stories corresponding to over 50% of these stories. see tables 1 and 2 for high-level dataset and annotation statistics. ## inferences from goal knowledge annotations in our dataset mostly consist of implied information often unstated in the story. inferring a participant's goal, future goal achieving actions and plans across a variety of alternative story contexts requires a deeper understanding of text using commonsense and world knowledge. we present a variety of goal based inference tasks that benchmark gpt-3.5 turbo, gpt-4 and various sizes of flan-t5 and t5 using prompting and fine-tuning. refer to we explored in-context learning using 0 to 6 examples (balancing labels in classification tasks). performance improves slightly with more examples but a 3-shot setting allows for an even comparison across models; gpt models produce long multisentence generations in a zero-shot setting, unlike the short flan-t5 generations; token length of a base flan-t5 and t5 limits us to a max of 3 complete examples without any text cut-off. t5-11b model generates additional gibberish text along with task related generation making human evaluation cumbersome; we only report automated scores for this model. we fine-tuned several sizes of flan-t5 and the t5 base model on our training dataset consisting of both actual and alternative stories. 2 we compare all models and settings using automated evaluation metrics described in §5.2 and some models using our human evaluation metrics. we present human evaluation metrics in the main paper and present automated metrics in the appendix. the human evaluated generations across the various models are based on the same stories, but their reference goal annotations may differ (for these we report variance for the averaged scores). scores significantly 3 lower than reference scores with a p-@xmath0.05 are underlined. ## conclusions we showed examining goal achievement in alternative stories can lead to a deeper and nuanced understanding of complex events. focusing on a specific participant's actions we developed a multitiered crowd sourcing process to obtain 6.3k goal annotation sets for 1.3k alternative stories. we captured highly subjective story aspects with our annotations and validated 20% with high inter-annotator agreement. we formulated 5 inference tasks and several sub-tasks to evaluate current sota intentfollowing llms. our evaluations show that each model differs in specific aspects of goal reasoning providing multiple future avenues to study. we think modeling advances can achieve a broader and deeper narrative understanding and hope that our work can help further this research.