A Theoretical Framework for Evaluating Narrative Surprise in Large Language Models (2025)

Abstract: Narrative surprise is a core element of storytelling for engaging audiences, and yet it remains underexplored in the context of large language models (LLMs) and narrative generation. While surprise arises from events that deviate from expectations while maintaining retrospective coherence, current computational approaches lack comprehensive frameworks to evaluate this phenomenon. This paper presents a novel framework for assessing narrative surprise, drawing on psychological theories of narrative comprehension and surprise intensity. We operationalize six criteria—initiatoriness, immutability violation, predictability, post-dictability, importance, and valence—to measure narrative surprise in story endings. Our study evaluates 120 story endings, generated by both human authors and LLMs, across 30 mystery narratives. Through a ranked-choice voting methodology, we identify significant correlations between reader preferences and four of the six criteria. Results underscore the continuing advantage of human-authored endings in achieving compelling narrative surprise, while also revealing significant progress in LLM-generated narratives.

## introduction narrative surprise represents a fundamental mechanism through which stories engage and captivate audiences, yet our understanding of how to systematically measure this phenomenon in large language models (llms) remains limited. while traditional narratology has long recognized surprise as one of three key components of narrative tension alongside suspense and curiosity @xcite @xcite , the emergence of llms as storytelling agents presents novel challenges in quantifying their ability to generate genuine narrative surprise. 1 these authors contributed equally to the paper. recent work in computational story generation has focused on two key challenges relevant to this area that have nevertheless remained distinct from one another. narrative coherence is essential for establishing narrative meaning by ensuring continuity among multiple narrative elements such as setting, characters, and events @xcite . narrative surprise, on the other hand, depends on the introduction of novel information while also maintaining narrative coherence. as @xcite argues, for surprise to be effective, the unexpected turn of events must be retrospectively coherent. from this perspective, recent approaches to evaluating narrative surprise in computational storytelling have important limitations. while researchers have made progress in developing wordlevel surprise metrics @xcite and tracking narrative turning points through sentiment analysis @xcite @xcite , these methods do not capture the complex temporal relationships that make stories coherent and meaningful. specifically, they do not address how surprising events must deviate from expectations while remaining logically consistent within the broader narrative framework. this disconnect between the evaluation of local surprise and global coherence represents a significant gap in the field, underscoring the need for a more comprehensive theoretical framework that can assess both the unexpectedness of generated story elements and the success of their narrative integration. in this paper, we present a novel theoretical framework for evaluating narrative surprise, grounded in psychological research on narrative comprehension and evaluation. our framework introduces six key metrics that capture different dimensions of cognitive surprise in narrative understanding. to validate this framework, we conduct an analysis of 120 story endings, generated by both human authors and llms, focusing on 30 mystery stories sourced from the reedsy fiction platform. these stories are manually truncated before their pivotal revelations to enable controlled testing of ending generation. using a ranked-choice voting methodology, we assess the relative quality of different endings and examine how our proposed metrics correlate with reader preferences. our analysis reveals that four of our six variables demonstrate significant associations with reader preferences, providing initial validation of our theoretical framework. we compare llm and human performance using both voting data and our six-metric framework. we conclude by discussing future directions for enhancing narrative surprise evaluation in computational storytelling and share our underlying data. foot_0 2 prior work ## a theoretical framework for measuring narrative surprise we evaluate six criteria for narrative surprise, drawing from foundational work on story comprehension and narrative affect discussed above (see our framework integrates bae and young (2013)'s work on computational models for generating surprising narratives and ortony and partridge (1987)'s framework for surprise intensity. the framework relies on narratives segmented into two structural components: the 'stem,' encompassing the beginning and middle of the narrative, and the 'ending,' which resolves earlier narrative events, often in the form of a 'big reveal'. note that we assume the surprising event with unknown causes (se) is presented in the 'stem,' while its initiating events (ies), i.e. causes, are presented in the 'ending.' category description initiatory ending describes a novel event that temporally precedes and causes the se. immutability violation ending contradicts an immutable fact of the story world. predictable a typical reader could have predicted the ending given the stem. post-dictable looking backwards at the whole story, the events are explainable, i.e. there are neither loose ends nor contradictions. important events of the ending meaningfully impact the protagonist. valence events of the ending are positive for the protagonist. the first criterion, initiatoriness, which operationalizes brewer and lichtenstein (1980)'s surprise generation hypothesis, examines whether initiatory events are presented in the ending which offer a causal explanation for the se that occurred in the stem. a highly initiatory narrative ending will provide key initiating events that explain how the surprising event(s) of the narrative stem occurred. the second criterion, immutability violation, builds on @xcite theoretical framework concerning proposition violation. this dimension assesses the degree to which narrative events challenge established axioms within the story world's logical framework. immutability violations occur when narratives contradict fundamental beliefs about the world (such as the absence of flying pigs). narratives contradicting more flexible beliefs, such as the belief that employers always hire by merit, are less immutability violating and easier to accept as plausible. the third criterion, predictability, builds on the observation of @xcite that an expectation-reality discrepancy is required to elicit surprise. our framework posits outcome predictability to be inversely related to surprise magnitude, while acknowledging that to ensure reader satisfaction, narrative surprises must not be totally impossible to predict. this suggests an optimal zone of moderate predictability. the fourth dimension, post-dictability, is drawn from @xcite . it measures the degree to which the narrative maintains internal consistency and fully explains plot events in order to leave readers with the feeling that the story makes sense in retrospect. this aligns with @xcite 's argument that surprise necessitates events to be retrospectively coherent. the final two criteria, valence and importance, are taken directly from the framework of @xcite , where negativity and importance are hypothesized to be positively correlated with surprise. ## conclusion this paper presents a novel theoretical framework for evaluating narrative surprise in stories generated by large language models (llms) and human authors. by integrating theoretical insights from narrative comprehension and cognitive surprise, we develop six key metrics to assess narrative surprise. our analysis of mystery story endings highlights the value of these metrics in understanding reader preferences, with initiatoriness and post-dictability emerging as particularly significant factors in driving narrative surprise. while our findings underscore the potential of llms to produce engaging narrative surprises, they also reveal limitations in their current ability to match the complexity and nuance of humanauthored endings. the preference for humanauthored stories suggests that llms need further advancements in generating unexpected yet coherent twists. in particular, enhancing the ability to generate causal relationships (initiatoriness) and logically coherent endings (post-dictability) and avoiding overly positive endings that are highly predictable offer promising avenues for improving the quality of machine-generated narratives. future research should go beyond the mystery genre to explore how narrative surprise varies across different storytelling traditions and audience expectations. incorporating multilingual datasets will also be essential for understanding how cultural and linguistic factors shape perceptions of surprise, coherence, and narrative quality. additionally, employing more diverse evaluation methodologies, such as real-time audience engagement tracking or large-scale reader surveys will help capture the multifaceted nature of narrative surprise. these efforts will not only refine our understanding of narrative dynamics but also advance the development of computational storytelling systems that are better equipped to create more nuanced and interesting stories.