Computational Narrative Understanding: A Big Picture Analysis (2023)

Abstract: This paper provides an overview of outstanding major research goals for the field of computational narrative understanding. Storytelling is an essential human practice, one that provides a sense of personal meaning, shared sense of community, and individual enjoyment. A number of research domains have increasingly focused on storytelling as a key mechanism for explaining human behavior. Now is an opportune moment to provide a vision of the contributions that computational narrative understanding can make towards this collective endeavor and the challenges facing the field. In addition to providing an overview of the elements of narrative, this paper outlines three major lines of inquiry: understanding the multi-modality of narrative; the temporal patterning of narrative (narrative “shape”); and socio-cultural narrative schemas, i.e. collective narratives. The paper concludes with a call for more inter-disciplinary working groups and deeper investment in building cross-cultural and multi-modal narrative datasets.

## introduction the native-american writer, gerald vizenor, once remarked: "there isn't any center to the world but a story" @xcite . storytelling is a ubiquitous human practice, exhibited in all human cultures, languages, and recorded historical time periods. many of the world's most enduring and widespread belief systems are encoded through stories, and research suggests that human reasoning @xcite and selfhood @xcite are fundamentally grounded in narrative. today, a growing body of research is developing across a variety of domains that focus on storytelling as a key mechanism for explaining human beliefs and behavior, from mental health @xcite , to political stance taking @xcite , to consumer persuasion @xcite , to financial decision making @xcite . given this widespread interest in, and awareness of, narrative as a crucial driver of human behavior, the field of "computational narrative understanding" has a great opportunity to contribute to a range of research fields. computational narrative understanding has crystallized over the past 5-10 years as a vibrant subset of natural language processing @xcite . its aim is to develop computational systems for the detection and understanding of narrative communication across different media and different cultural domains. while we may typically think of stories as encoded in written documents, the practice of narrative can be represented through a diverse array of media, including oral speech, song, still or moving images, social media, playable media like video games, or some combination of the above. the aim of this paper is to provide a big picture view of some of the key higher-level goals for computational narrative understanding. a great deal of on-going and inspiring work continues to make progress in the detection and analysis of different components of narrative communication (for a review see @xcite ). it thus seems timely to provide a vision of where we are going as a community to help motivate and organize future work in the field. in section two, i provide a brief minimal definition of narrative communication highlighting its constituent parts building on prior work @xcite . before moving to the big picture, it is important to ground our understanding of this core concept. in section three, i describe a research framework that aims to develop a more multi-modal understanding of narrative. with its grounding in nlp, computational narrative understanding has understandably focused on narrative as a linguistic phenomenon. however, as narratologists have long pointed out @xcite , storytelling can transpire in numerous different media. being able to integrate observations across media, from speech to text to images to playable media should become a central goal of computational narrative understanding. in section four, i describe a research framework aimed at understanding narrative "shape" (also called "form" or "structure" @xcite ), which can be understood as the temporal patterning of narrative elements. one of the fundamental aspects of storytelling is the encoding of events in time @xcite @xcite . narrative meaning is thus contingent on the temporal organization of information. seeing narratives as temporal artifacts, made in time and composed of time, then leads to the highest-level form of integration described in section five, that of narrative "schemas." as berns (2022) has argued, narratives are forms of information compression, reducing the vast scope of experienced data down to a much more limited set of communicated data. such compression necessarily follows archetypes or patterns that can be biologically or culturally conditioned (or some mixture of the two). while the idea of "scripts" has been applied to understand the local schematic encoding of events @xcite , prior work in folklore studies has offered promising frameworks for expanding the idea of schema to include whole stories within various typologies @xcite . essential to this framework is an attention to larger narrative ecologies, the ways in which such schemas play a generative and/or organizing role within broader, and potentially interactive, communicative domains @xcite . it is common to think of narrative as located within an individual document or artifact (this book or blog post tells a story), but narratologists have also highlighted the way story structures emerge from the complex social interactions of numerous agents (known as the "small stories" paradigm @xcite ). such "small stories" can then coalesce into larger socially circulatable schemas, variously referred to as "ontological narratives" @xcite , "deep stories" (hochschild, 2018), or "collective narratives" @xcite . such schemas can then guide the processing and circulation of new information to "fit the narrative," potentially creating informational feedback loops that are durable over shorter or longer stretches of time. in sum, we want to have a research framework capable of scaling the ladder from local elements (section 2), different media (section 3), formal structure (section 4), all the way up to schemas and social dynamics (section 5). figure 1 provides a schematic overview of this big picture. i conclude in section six with a reflection on the need for greater inter-disciplinary collaboration and deeper investment in building cross-cultural and multi-modal datasets. as we develop more sophisticated systems for detecting narrative communication, we will want to invest more deeply in the infrastructure for large-scale narrative understanding. this will necessarily entail collaborations across disciplines to better understand socially relevant applications as well as the ability to develop appropriate data. it will also require developing an awareness around the limitations or risks of narrative communication @xcite . stories not only inspire and move audiences, they can also deform reality and misinform, a point that should remain at the forefront of our thinking about how stories stand at the centre of so much human behavior, for better and for worse. ## the elements of narrative at its most elementary level, a story can be said to occur when all of the following criteria are met: a someone b tells c someone d somewhere that e someone f did something(s) g [to someone] h somewhere i at some time j for some reason. for there to be a story, we need (a) a teller, (b) a mode of telling (i.e. medium), (c) a recipient, (d) a social situation, (e) an agent, (f) at least one action or event, (g) a possible object, (h) a location, (i) a time-frame, and (j) a motivation or cause of the actions involved. narratologists make a distinction between the frame of the storyworld (i.e. all of the elements that come after the double lines above) known as "diegetic" elements, and the frame of telling (i.e. all of the elements that come before the double lines) known as "heterodiegetic" elements, where diegesis refers to a narrative "frame" or "world." importantly, not all of these elements need to be explicit. for example, in one of the most famous short stories ever proposed by ernest hemingway, very little from the above list is specified: for sale: baby shoes. never worn. we don't know where and when this happened, nor do we know who is telling the story. all we know is what happened (on two levels): a baby died and a family needs money. but no matter how much is implicit in this story all of the parts are there. something happens to someone somewhere at some time for some reason and someone tells someone this story. such a definition can be useful because it highlights the array of narrative elements that require computational solutions to "understand" the cultural meaning of a story. such applications have included: character detection @xcite @xcite , object detection @xcite , character relation detection @xcite , event detection @xcite , geographic and spatial understanding @xcite @xcite , temporal understanding @xcite @xcite , and causality mining @xcite . a full review can be found in @xcite and @xcite . a second, higher-level way that a story can be broken down into constituent parts is through discourse elements. as we will see, this problem is associated with challenges of text segmentation, though importantly differs from prior work focused on sequential and/or paratextual (i.e. chapter) segmentation @xcite . narratives not only contain event-frames (i.e. scenes), but are also composed of heterogeneous linguistic styles in which the act of narration is but one component. this is one reason recent narrative theory has emphasized the idea of "narrativity" @xcite @xcite , which captures the degree of narration intrinsic to a narrative. an ostensibly narrative document like a short story will engage in moments of non-narrative statements, just as putatatively non-narrative documents like scientific articles may engage in occasionaly moments of narration. narration is in this sense not a universal property of documents, but a local linguistic phenomenon. as @xcite write, "we believe that narrative as genre and activity can be fruitfully examined in terms of a set of dimensions that a narrative displays to different degrees and in different ways." narratologists typically break down narratives into at least four basic discourse components: discourse contents 1. narration agents and events 2. description setting, modification, context 3. dialogue reported speech 4. evaluation meta-level discourse "narration," also known as "diegesis," refers to the linguistic structures described above that occur after the double horizontal line (e-j). this is the classic understanding of narrative, where events pertaining to an agent are recounted (this can also fall under the heading of "eventfulness" @xcite ). finally, many stories contain what we might call meta-textual statements (called "evaluation" by @xcite ), where the narrator provides some higher-level assessment with regards to the story, either a reflection on the story contents, their meaning, or some didactic lesson that should be imparted, making a latent feature of storytelling (it's meaning or purpose) manifest. while it may come at the end of a story, it can also be interspersed throughout. here are a few examples of such statements: while there are many more ways one can parse a story (see @xcite ; genette (1983)), the frameworks above provide practical heuristics for the ways that stories can be broken down into more elementary parts to ground computational models. ## the modality of narrative grounded in nlp, computational narrative understanding has largely prioritized written narratives for understandable reasons. however, such textdriven approaches leave out large portions of storytelling behavior, including movies and television @xcite , usergenerated streaming content, illustrated content in comic strips @xcite , graphic novels, or children's books @xcite , and finally video games, which might have stronger or weaker narrative structures. while textual narratives are largely unimodal in nature (though the physical and visual dimensions of books has been a vibrant area of study for a long time @xcite ), these other narrative forms are all crucially multi-modal in nature. sound, image, and language can interact in ways that are complex and dynamic 2023 )). nevertheless, investigations into multimodal narrative understanding, such as the relationship between text and illustrations in children's books or graphic novels is in need of more attention (see @xcite for an example exploring the visual qualities of children's book illustrations with respect to race). understanding the kinds of gestural or pictorial preferences that are foregrounded given certain textual cues could give us insights into the way humans translate language into image (and vice versa) across different cultural domains. similarly, we still lack major comparative studies of narrative behavior across media, i.e. comparisons of narrative elements and archetypes in film, television, user-generated content, oral performances and books. for example, evidence suggests that written and oral narratives have similar "establishing shot" structures similar to movies and television @xcite . more precise comparisons can highlight the modalspecificity of different narrative elements along with the transmodal practices that are independent of a given modality. understanding the ways in which storytellers marshal images, sounds, and words to create immersive experiences for audiences will greatly contribute to the project of computational narrative understanding. ## the shape of narrative the writer and critic italo calvino was fond of quoting a sicilian expression that "time takes no time in a story" @xcite . a narrator can tell a story that traverses centuries in a few sentences or can slow time down to the point where a few seconds takes minutes to describe. narratologists refer to this as the difference between narrated time (the time transpiring in the storyworld) and narrative time (how long a story takes to tell). no matter how much stories may compress time, they cannot be told all at once. contrary to calvino's favored sicilian expression, all stories, even the shortest, take time to tell. this temporal dimension of narrative -that stories take time to tell and tell of things happening in time -has long been one of the privileged topics of narrative theory @xcite @xcite . as the theorist david herman has argued, "narrative is a basic human strategy for coming to terms with time, process, and change" @xcite . a number of approaches have been proposed for the computational modeling of temporal patterns in narrative (for a review of modeling narrative structure see @xcite . schmidt (2015) used topic modeling to identify thematic arcs in television screenplays, while @xcite used topic models to study thematic progression in philosophical texts and social media. @xcite used sentiment analysis to model the concept of narrative fortune @xcite , for which @xcite provides a more in-depth study of the validity of sentiment arcs as models of narrative structure. boyd et al. (2020) used particular word types to capture three primary narrative stages, and @xcite used the predictability of next sentences to capture the concept of narrative "flow." @xcite used word embeddings to model narrative non-linearity using the traveling salesman problem, while @xcite offer two further ways of thinking about narrative shape called "speed" and "volume." researchers have also used information theoretic frameworks to model the concept of narrative revelation using time series methods @xcite ) and stylistic novelty over narrative time using a bloom filter @xcite . @xcite and @xcite devised methods for predicting narrative "turning points" as larger structural qualites, drawing on aristotelian and augustinian theories of narrative respectively. common to all of these models is the assumption that the dissemination of information over narrative time assumes observable patterns (called "form" or "structure") and that these patterns encode cultural meaning. the most common framework to date has been that of the narrative "arc," drawn from french neo-classical tragedy @xcite . according to this model, narratives encode a central conflict that results in some form of resolution or change, which can be approximated by an arc of rising and falling fortune or conflict. much future work remains to better understand relevant ways of capturing narrative time in terms of its formal patterns. the first area of consideration should be further work into the choice of feature distributions that are used to capture narrative time. where prior work has focused to date on topic models, sentiment vocabulary, word embeddings, lexemes, and letters, higher-level narrative features (see section 1) should continue to be developed and studied. we assume that the distribution of characters, event types, locations, or narrative modes may also contribute to the overall structural qualities of stories. second, modeling narrative change itself remains a key area of further research. prior empirical work has shown that long narratives may employ multiple "arcs" rather than single turning points @xcite , while other work has emphasized the significance of single turning points @xcite . additionally, the identity or meaning of such moments of change, regardless of how many, are also not well understood. the dramatic model of narrative denouement suggests that turning points are best understood as forms of "conflict/resolution," while other narrative theories suggest that "surprisingness" is the optimal way of understanding narrative change @xcite . @xcite have proposed two further affective states of suspense and curiosity in addition to surprise to capture the discrepancy between storyworld information and nar-rative information (i.e. when key information is withheld or forms of temporal anachrony are used such as flashbacks and flashwords known as analepsis and prolepsis respectively). in addition to these temporal issues, the role that causality plays in describing narrative change has been relatively underexplored. as the writer and essayist george saunders has argued, causality is the "wind in the kite" of narrative @xcite . as @xcite have demonstrated, readers are much more moved by "why" questions than "what" questions when it comes to narrative comprehension and recall. future work will want to explore more fully both different constructs of "change" as well as draw on methodologies such as markov models, time series analysis and systems dynamics to develop increasingly sophisticated models of change over narrative time. finally, most prior work is guided by a single spatial metaphor for narrative time, that of the arc. future work will want to explore other possible structures or forms @xcite that might capture the temporal patterns of narrative. the translation of time into spatial form represents an exciting and novel space of research for computational narrative understanding. ## narrative schemas narratives are forms of information compression @xcite . they select certain experiential data and structure this data into prescribed grammatical slots (as described in section 1). this basic insight serves as the foundation of the theory of narrative "scripts" @xcite , where narrative is understood as a probabilistic sequence of actions (i.e. given the event of being in a restaurant certain subsequent actions are more or less likely). such compression is what allows stories to be both memorable as well as easily shareable (i.e. tellable @xcite ). the discussion of narrative form or shape in the prior section is one such example of the schematic nature of narrative, i.e. that narratives have structure and this structure is essential to their meaning. but schemas can also be represented as a variety of conceptual metaphors (that often have spatial associations). for example, in the field of clinical psychology researchers refer to two self-narrative schemas, called narratives of redemption (when bad things turn good) and narratives of contamination (when good things turn bad) @xcite . patients who structure life experience into the former schema are far more likely to be associated with positive mental health outcomes than those who engage in telling their life stories according to the latter. the first extensive (and later controversial) study of narrative schemas emerged in the field of folklore studies @xcite . faced with large collections of documents with a high degree of repetitiveness, folklorists began developing systems for classifying stories according to different typologies. the most famous undertakings were stith thompson's motif-index of folk-literature @xcite , the aarne-thompson-uther (atu) tale index, and vladimir propp's emphasis on character "function" @xcite . fundamental to this research was the insight that certain larger narrative patterns are maintained while local units can be changed. as @xcite highlighted, whether it is an eagle or a horse or a ring that is the gift that carries away its recipient, the point of each of these stories is the event of being transported, or even more generally, the danger or affordance of gift giving. while it is beyond the scope of this paper to rehearse debates around narrative classification (for a review see @xcite , there remains a fundamental value in developing narrative taxonomies for different domains. narratives are indeed reducible to schemas and those schemas serve particular social and psychological functions. and yet we currently lack agreed-upon or widely used frameworks for discussing schemas, either at the individual or socio-cultural level. folklorist and computational narratologist timothy tangherlini has begun using the idea of schemas to study conspiracy theories circulating through social media @xcite @xcite , which function much like folklore in that various narrative units (bill gates, 5g) can be utilized for larger functional purposes (a global cabal of elites is controlling us). related research by @xcite looks at "dogwhistle" detection, which can be understood as phrases with latent, toxic meanings and that likely have a narrative element to them. understanding schemas requires two challenging research questions. the first we can refer to as motif tracking, which requires the ability to model variability and repetition at both the level of local units (agents, actions, objects) and more general schemas (when certain units are deployed to tell certain kinds of stories). while systems currently exist to identify the narrative units described in section 1 (including agents, actions, and objects), we still need ways of aggregating these units into story "types." when is bill gates being used to tell a story about global elites and when is he being used to tell a story about the power of philanthropy? more importantly, we want to model the causes as well as social effects of these different story types. do we see certain narrative schemas deployed in response to major social events (for example what are the prevalent narrative responses to financial or political or climatic shocks?). or can certain narrative schemas predict future behavior? similar to the clinical psychology example mentioned above but moving into the social realm, do we see the persistent invocation of narratives of national decline associated with shifts in electoral behavior? if we assume narrative is a key predictor of human behavior, we need more reliable and sophisticated ways of classifying narratives to better understand their causes and effects. the second key dimension in studying narrative schemas is the aspect of social dynamics. as folklore studies first highlighted, narrative types are aggregates of numerous local instances of storytelling behavior. each unit (whether an oral tale or social media post) may contribute to a larger narrative schema but may itself only loosely embody this schema. narratologists refer to these local dynamics as "small stories" @xcite , i.e. when a larger story is told through the participation of numerous actors. the quintessential example of this behavior is the "family dinner table," where family lore is the product of multiple actors engaging in the process of narrative recounting, potentially over long spans of time. at the macro-level narratologists refer to these larger narrative schemas -the aggregate of small stories -as "collective narratives" @xcite , "ontological narratives" @xcite , or "deep stories" @xcite . social media and online news (broadly understood) greatly expand the complexity of collective narrative construction and small-story dynamics. one can imagine "top-down" approaches that start with known schemas and then classify individual stories or collections of stories within these taxonomies or "bottom-up" approaches that cluster individual stories into larger schemas that emerge from the collective behavior among the data. modeling this complex, large-scale narrative behavior represents one of the major challenges for the field but one that has the most explanatory pay-offs in terms of understanding social behavior. ## narrative infrastructures as computational narrative understanding comes into its own as a distinct field within the nlp community, now is a good time to begin coordinating more of this research effort. these initiatives can take the form of shared tasks, dataset curation, and collective efforts to develop systems for narrative classification. shared tasks have a long history within nlp, though to date only three have been proposed for narrative understanding. the first is the narrative cloze test @xcite @xcite , where systems predict the next agent-event in an event chain. @xcite have proposed a task for detecting narrative scenes, while @xcite have proposed a task for detecting narrative levels (when diegetic worlds are imbedded within one another, either in the form of stories within stories or temporal anachronisms such as flashbacks). @xcite and hatzel and biemann (2023) have proposed annotation frameworks for narrative detection, i.e. identifying the degree to which a stretch of discourse can be identified as containing narration. future work will want to refine these existing initiatives as well as develop systems for the further detection of the remaining discursive units described in section 2 (i.e. description, dialogue, evaluation). the automated identification of narrative communication in particular will prove extremely valuable for broader social and cultural analysis. given the value of narrative for understanding human behavior it is somewhat surprising how few datasets are available for the study of human storytelling. much of this is due to intersecting problems of intellectual property restrictions, large library collections with low-levels of metadata, and the dynamic and ever-changing nature of online storytelling. @xcite provide a large-scale annotation of ca. 200,000 fictional narratives in english in the hathi trust digital library that has been refined and updated by @xcite to include a comparison corpus of non-fiction prose across 1.5 million sampled pages published since 1800. @xcite extends this work to include multilingual fiction annotation across 521 different languages. @xcite provide metadata on translations of fiction into english from 120 different languages also located in the hathi trust. outside of the hathitrust, piper (2022b) provides derived data on a collection of 2,700 works of professionally published english prose drawn from 12 different genres including goodreads' user ratings. @xcite developed an artificial corpus of very short stories (4-5 sentences) generated by crowdsourced workers. @xcite curated a collection of ca. 5,000 askreddit stories told by users in response to particular prompts (e.g. what is your scariest real-life story?). researchers in the field should be aware that while project gutenberg offers a large collection of potentially narrative texts, problems of sample selection and poor metadata can lead to downstream problems that result in erroneous claims @xcite . for addressing cultural and historical questions, researchers are strongly encouraged to use the collections described above. incumbent on all of these initiatives is a greater investment in inter-disciplinary collaboration. computational narrative understanding will benefit as an endeavor with deeper collaborations between humanists and social scientists and the nlp community. as detailed in @xcite , narratology is a field with a long and robust theoretical tradition. those in the nlp field working on computational systems will benefit from expert collaborations with researchers who have deep backgrounds in studying narratives. similarly, narratologists and their research frameworks stand to benefit from exposure to computational models @xcite . it is time to invest more heavily in these larger cross-disciplinary collaborations, especially if we aim to address the larger socio-cultural goals outlined in this paper. ## conclusion as vizenor envisioned, narratives are things we live by. they provide meaning and hold communities together. they play a role in financial, political, and psychological decision-making. the production of imaginary narratives in particular represent a massive cultural industry, spanning book publishing, movie-making, and gaming. the field of computational narrative understanding has made impressive strides in developing systems to study the causes and effects of narrative behavior across a diverse array of languages and cultural domains. we are in the process of establishing key workshops, tasks, and datasets. by way of conclusion, i provide a sliding scale of calls to action, located from particular to general (