Are Large Language Models Capable of Generating Human-Level Narratives? (2024)

Abstract: As daily reliance on large language models (LLMs) grows, assessing their generation quality is crucial to understanding how they might impact on our communications. This paper investigates the capability of LLMs in storytelling, focusing on narrative development and plot progression. We introduce a novel computational framework to analyze narratives through three discourse-level aspects: i) story arcs, ii) turning points, and iii) affective dimensions, including arousal and valence. By leveraging expert and automatic annotations, we uncover significant discrepancies between the LLM- and human- written stories. While human-written stories are suspenseful, arousing, and diverse in narrative structures, LLM stories are homogeneously positive and lack tension. Next, we measure narrative reasoning skills as a precursor to generative capacities, concluding that most LLMs fall short of human abilities in discourse understanding. Finally, we show that explicit integration of aforementioned discourse features can enhance storytelling, as is demonstrated by over 40% improvement in neural storytelling in terms of diversity, suspense, and arousal. Such advances promise to facilitate greater and more natural roles LLMs in human communication.

## introduction storytelling serves as an integral part in shaping our understandings of ourselves, our society and our world @xcite . as large language models (llms) grow in capabilities @xcite and are integrated into our daily communicative routines @xcite , assessing the narrative structures of the stories they tell is crucial to understanding the ways they will shape our society. humans incorporate discourse structures that span local and global levels to captivate audiences, evoke emotions, convey complex messages, and share unique perspectives @xcite . a recent hci study has pointed to gaps in machine storytelling ability at the globallevel: despite being able to craft fluent narratives, * the two authors contributed equally. compared with human storytellers, llms tend to (1) adopt homogeneously happier, less complex story arcs, (2) introduce plot turning points earlier in the timeline, and (3) have less suspense or fewer setbacks in their storylines. the impact of these differences grow as llms gain greater prominence in communicative patterns. llms such as gpt-4 and claude exhibit plot holes or produce repetitive themes that are less preferred by human critics @xcite . however, a computational framework remains to be established for quantitative assessments of narratives at the global or discourse level. we take a step towards the much-desired analytical framework and attempt to answer pertinent questions such as: do stories generated by llms exhibit the same narrative complexity and diversity as human storytelling? do llms have the capacity to comprehend narrative structures? concretely, we measure narrative discourse structures at three distinct levels: 1) story arcs (i.e., macro-level narrative development), 2) turning points (i.e., meso-level shifts) and 3) arousal & valence (i.e., micro-level dynamics). we collect a dataset of movie synopses, on which we conduct a wide range of human and automated annotations for each of these levels, with the goals of (1) contrasting llm and human storytelling, and (2) probing llm narrative structure comprehension. first, we explore llm storytelling abilities. as rags to riches riches to rags man in a hole double man in a hole icarus cinderella oedipus starts low and gradually rises, ending in a high state. starts high and gradually falls, ending in a low state. starts high, has a dilemma or crisis and finally finds a way out. two cycles of fall and rise. a rise followed by a sharp fall. a rise, followed by a fall, ending with a significant rise. a fall, followed by a rise, ending with a significant fall. motivated by this finding, we explore whether we can improve machine story-telling by leveraging the aforementioned discourse features, and we find promising results. incorporating discourse reasoning in prompts can serve as important guidance towards better story generation (section 6). in two parallel experiments, we demonstrate that integrating awareness of story arcs enhances model diversity (outperforming vanilla prompting by 45%), whereas incorporating turning point information significantly improves narrative suspense and engagement (outperforming vanilla by 40%). in summary, our contributions are threefold: 1. we unify three levels of discourse in narrative analysis: story arc, turning point, and affective dimension. based on this, we present the first quantitative analysis framework to benchmark narrative development, and demonstrate that it can be operationalized by humans on benchmark dataset which we release ( §2 and §3). ## background: discourse in narratives we identify three aspects of plot progression in story-telling: story-arcs (macro-level), turningpoints (meso-level) and arousal/valence (microlevel), each representing a different level on which storytellers develop their narratives @xcite . we describe each of them before describing how we collect data to measure them in stories. ## data collection and annotation films are our culture's gesamtkunstwerk (or "total work of art") according to @xcite , and our mass market vehicle for telling narratives @xcite . thus, we take films as a basis for exploring the stories our culture tells itself. the narratives we examine are condensed versions of some of the most intricate storylines humans create-those found in movies. while these synopses focus on key plot developments, they should not be considered simple or straightforward. in this section, we will describe first how we built our dataset of film plots, and then we will describe our annotation approach to study the plots' discourse structures. ## benchmarking narrative comprehension we hypothesize that poor narrative comprehension of llms lead to their poor generative outcomes, as much evidence exists for these skills being tied @xcite . therefore, we designed and conducted two benchmark tests to measure narrative reasoning. we start by outlining our tasks ( § 5.1) and the methodologies employed to evaluate performance ( §5.2). we ## towards better machine storytelling finally, we investigate whether incorporating the discourse aspects into the generation stage enhances machine's storytelling ability. reasoning about tps improves overall narrative construction, including reduced plot holes and enhanced suspense and emotion provocation. motivated by our observations that a major flaw in vanilla llm story-telling is narrative pacing (in §4) we hypothesize that integrating discourse features can improve pacing and significantly improve narratives. we test three variations of a planning-first @xcite approach (i.e. generating first the outline and then the narrative). each variation incorporates different degrees of explicit structure. we annotate comparatively, ranking three randomly shuffled narratives in terms of suspense, emotion provocation, and overall preference. incorporating explicit directives about story arcs helps improve narrative diversity motivated by our observations, in figure 4 , that llm generations lack arc-level diversity, we explore whether explicit instruction can induce a more human-like story arcs. we design another variant, arc enhanced, that explicitly instructs the model to generate story with a specified story arc, specifies the number of major rises and falls and details the initial and ending state of the protagonists. first, we evaluate how well llms are able to follow the requested story-arcs. the results, in next, we compare the narrative diversity between sets of arc enhanced stories and outline-only stories 3 . as seen in ## qualitative study: understanding human preferences after completing all annotation tasks, we conducted two follow-up interviews to gather qualitative feedback from our annotators. these interviews focused on annotators who worked on the task in § 6 -reading pairs of machine generated narratives (outline-only vs arc-enhanced) and examining diversities plus overall preference. they were encouraged to freely provide justifications or comments on any of the readings. after the interviews, we reconstructed their feedback and presented representative comments in figure 7 and figure 9 . overall, the human annotators prefer concrete narratives with twists in plot development that are logical and well-motivated. they dislike straightforward, positive plots or those with 'miracle turns' that are not adequately justified. ## related work discourse-aware evaluation. in contrast to conventional story evaluation frameworks, which primarily focus on fluency and coherence, discourseaware evaluation focuses on critiquing the structural and creative quality of machine-generated content @xcite @xcite . @xcite introduced a model that assesses stories by embedding conventional narrative structures within the evaluation process. complementing this, @xcite explore narrative differences between humans and ai through a qualitative study and @xcite reveal that rlhf-aligned language models are less diverse than the base lms.. begus (2023) delves into the creative outputs of llms, questioning their true creativity versus their capacity to merely replicate observed patterns, which encourages further exploration of the creative limits of these models. additionally, @xcite introduces the positional discourse coherence metric to quantitatively assess logical narrative progression. however, prior works have been limited by the vague definitions of creativity and discourse structure. we take inspiration from the literature theories of discourse analysis of drama and fiction @xcite . for annotator general comment (pointing to multiple gpt-4 generated stories) … the authors are unwilling to put characters in real risk in the sense that characters make a plan and everything goes according to plan. moreover, oftentimes conflicts are just resolved by "talking it out" or some "community effort" which i dislike because it doesn't allow the characters to grow stronger or improve themselves. excerpt (arc enhanced) … as the fire raged around him, tom stumbled through the manor, the woman in white appearing amidst the flames, guiding him deeper into the inferno … in his final moments, tom realized the woman in white was not guiding him to safety, but delivering him to his fate. annotator comment this was a neat little twist which was especially unexpected in a ai story since others tend to be very straightforward and extremely positive. excerpt (arc enhanced) … she turns to alex, her voice steady with conviction, "let's go shoot the sunset" a daily ritual that has become their shared passion. together, they head towards the horizon, their steps light and sure, the camera slung over alex's shoulder, ready to capture the golden hues painting a new chapter in their lives… annotator comment … this was a nice example of the kind of specificity i mean wrt dialogue --this is a very specific reference to photography which other stories didn't include. annotator general comment for the stories [pointing to multiple set id], i ultimately couldn't choose one that was better because both had very positive endings, which was annoying to me and this was probably my least favorite set for that reason. figure 7 : human annotators' feedback on machinegenerated stories when comparing the outline-only vs arc-enhanced strategy. they were blind to the prompting strategies and all presented stories were randomly shuffled. we reconstruct their comments and color-code with green for favorable ones and red for unfavorable ones. continued in figure 9 . instance, freytag (1894) proposed a a five-part dramatic structure, now commonly understood as exposition, rising action, climax, falling action, and dénouement. @xcite combined such literature theories and annotated story macrostructures. in non-fiction storytelling like news writing, choubey et al. (2020); @xcite demonstrated that language models can classify similar elements, with artificial news differing from human-generated content. likewise, we define specific story arcs and turning points in creative stories and examine how stories generated by lms structurally differ from human ones. 2022 ) that focus on generating coherent, logical, and interesting stories. @xcite incorporates affective dimensions to foster the creation of more captivating stories. @xcite examines the efficacy of "bookends" as a structural enhancement in narrative quality. further studies have also ventured into embedding elements of suspense to forge more engaging narratives @xcite . different from previous endeavors, our study enhances narrative construction through the systematic incorporation of discourse elements, similar to @xcite who focus on news structures. our approach seeks to bridge the gap between humanlike storytelling and the capabilities of current ai systems through three levels of discourse elements. ## conclusion this work aims to advance the understanding and generation of narratives through the lens of three discourse elements: story arcs at macro-level, turning points at meso-level, and affective dimensions at micro-level. we contribute an expert-annotated dataset, based on which we conducted quantitative comparison between human and ai in terms of narrative generation and comprehension: llms fall short especially in story writing. we find models lack narrative diversity, and struggle at develop crucial turning points, such as major setback and climax, leading to less engaging stories. we also show promising results that discourse-aware generation improves ai's story-telling ability in terms of suspense, emotion engagement, and narrative diversity. we view our effort as a useful starting point towards a systematic analysis of narrative discourse. we hope the collected dataset and experimental results, along with our proposed perspective, will attract wider academic interest in discourse studies and provide insights into better narrative generation, comprehension, and evaluation.