Creating Suspenseful Stories: Iterative Planning with Large Language Models (2024)

Abstract: Automated story generation has been one of the long-standing challenges in NLP. Among all dimensions of stories, *suspense* is very common in human-written stories but relatively under-explored in AI-generated stories. While recent advances in large language models (LLMs) have greatly promoted language generation in general, state-of-the-art LLMs are still unreliable when it comes to suspenseful story generation. We propose a novel iterative-prompting-based planning method that is grounded in two theoretical foundations of story suspense from cognitive psychology and narratology. This theory-grounded method works in a fully zero-shot manner and does not rely on any supervised story corpora. To the best of our knowledge, this paper is the first attempt at suspenseful story generation with LLMs. Extensive human evaluations of the generated suspenseful stories demonstrate the effectiveness of our method.

## introduction among numerous nlp tasks, automated story generation is a representative one that requires creativity. by learning from human-written stories, an automated storyteller mimics humans and becomes competent in producing stories useful for various application scenarios, such as entertainment, education, and social bonding @xcite . the notable achievements in the field of deep learning have led to the subsequent emergence of data-driven methodologies for automated story generation @xcite @xcite @xcite . with the rapid development of large language models @xcite @xcite , generated stories have further increased greatly in length, complexity, and fluency. these enhancements are primarily realized through the application of methodologies that are built based the reader probably expects max to infiltrate the enemy headquarters and discover the leader's identity. max sterling must track down the leader of an international spy organization about to kidnap the president. little does max know, but a mole has tipped off the enemy, who have set a trap for max. oh no! will max be okay? tell me a spy thriller story. next the reader probably expects max to track down and interrogate former members of the enemy organization. max barely escapes the enemy compound but time is running out and max needs a new plan. the defector has been threatened with their life and refuses to cooperate with max. max's plans are getting desperate and time is running out! process continues until story generator allows the protagonist to succeed. figure 1 : our suspenseful story generation technique is predicated on a theory of suspense in which the quantity and quality (likelihood) of ways in which a protagonist can avoid an undesirable outcome are decreased. our method iteratively produces possible actions for the protagonist to take and adversarially creates the conditions under which the protagonist will fail. the dialogue boxes indicate the story snippets generated by our method. the thought bubbles of the reader represent the mental process. the thought bubbles of the system depict the planning process in our method. on either fine-tuning pre-trained models on supervised story datasets or prompting large language models (llms) of strong zero-shot capability. existing work on language model based computational storytelling mostly focuses on optimizing automated story generation systems from different angles, including both generation settings (e.g., long-form generation @xcite , controllable generation @xcite @xcite ), and story characteristics (e.g., com-monsense reasoning @xcite @xcite , temporal and causal relationships @xcite @xcite ). in this paper, aiming to expand the capability of automated storytellers, we direct our focus towards an under-explored dimension of stories: suspense, which is one of the strong affective responses humans often feel when being told a story. suspense is important as expert storytellers frequently use it to maintain reader engagement. due to the intricate nature of suspense as a complex cognitive phenomenon that only emerges under the right conditions, the generation of narratives with suspense, herein referred to as suspenseful story generation, remains an open research challenge. prior research pertaining to narrative suspense @xcite @xcite @xcite predominantly focuses on the computational modeling of this cognitive phenomenon, with a very limited exploration of story suspense from the language generation perspective @xcite @xcite . general-purpose generative models such as chatgpt (openai, 2022) exhibit a poor understanding of story suspense. model fine-tuning on suspenseful story corpora does not result in an llm that is able to capture the latent conditions necessary for the reader to feel suspense. given the current widespread popularity of llms, how can one employ them to effectively generate suspenseful stories? a psychological theory on suspense (gerrig and bernardo, 1994) suggests that readers feel suspense when they believe that the protagonist is facing a negative outcome and that the quantity or quality of feasible means of escaping the impending negative outcome has become diminished. following this theory, we propose an iterative-prompting-based planning technique that can effectively guide llms in suspenseful story generation. specifically, we establish a character with a goal and a negative outcome if they fail. we prompt the llm to consider the different ways the protagonist might achieve their goal. we then adversarially attack these plans by establishing conditions in the story world in which the protagonist's plans will fail. repeating this several times produces an outline for the story, which we then elaborate into longer text sequences (chapters). we depict and illustrate the overall intuition of our method in figure 1 . in summary, the main contributions of our work are as follows: ## iterative-prompting-based planning in this section, we introduce our method for generating suspenseful stories with an llm: iterative-prompting-based planning. one of the reasons that llms cannot be easily prompted to generate suspenseful stories is because suspense is an affective response to a cognitive state that only comes about under certain circumstances. there must be a protagonist that is empathetic to the reader. that protagonist must face a high possibility of an outcome undesired by the protagonist and the reader. the quantity and quality (roughly in terms of expected probability) of ways of avoiding that undesired outcome are reduced. this suggests an adversarial planning process wherein the generator establishes the conditions for suspense in three stages of generation. background setup §3.1 establishes the necessary story background, describing the protagonist, their goal, and a dire situation the protagonist will be put in if the goal is not successfully achieved. outline planning §3.2 plans out the story outline by providing a series of possible actions for the protagonist to take to achieve their goal, and then generating potential reasons why actions might turn out to be ineffective in achieving the goal. in essence, we craft a plan for the pro-tagonist to escape the undesired outcome, and then adversarially establish the conditions that block the protagonist from that escape. detail elaboration §3.3 composes event sequences that manifest each of the planned actions and failures, then elaborates on the details of the events. our method does not rely on any external story corpora. once the problem is decomposed in this way, the llm is capable of generating all intermediate information required for generating a full suspenseful story. ## evaluation human participant evaluation has been generally held as the gold-standard evaluation technique for creative language generation tasks such as story generation @xcite @xcite . to extensively evaluate our method, we design three human participant studies. in the first, participants are asked to judge the goodness of the stories we generate and compare stories generated by our technique against those generated by the baseline method. the comparisons are based on multiple evaluation criteria. in the second, we perform ablations on our system and compare our full system against the ablations. ablation studies help us understand whether different parts of our system are significant to the solution. in the third, participants are requested to examine the story outline to help determine if the internal workflow is running properly as we expected. all human participant studies are performed using the prolific crowdsourcing platform foot_0 . these studies have been approved by our institution's institutional review board (irb). we qualify participants by first asking them a screening question at the beginning of the survey, and then verifying answers manually to disregard answers provided by those who fail the screening. we require participants to be physically located in the u.s. and to speak english as a first language. for each human participant study, we source a distinct set of participants (no overlap) to avoid potential bias in annotations that could occur from participating in related studies in the past. for each study, we measure the average inter-annotator agreement using fleiss's kappa @xcite . we choose chatgpt (openai, 2022) (gpt-3.5-turbo-0613) as the llm we use in the implementation. as mentioned in section 3, our method does not rely on any external story corpora. suspenseful stories can be generated with our method in a fully zero-shot way from chatgpt. we start every chatgpt dialogue session with "you are a creative storyteller.". when generating anything other than stories (e.g., interim information such as the protagonist's goal and actions), to discourage chatgpt from generating excessively long responses that are usually not quite useful when serving as inputs to subsequent generation stages, we append "use no more than [n] sentences in your answer." (n can be three, four, or five) to the prompt when necessary. ## factors affecting suspense perception given the success of our method in creating suspenseful stories, we recognized a unique opportunity to contribute to a more fine-grained understanding of suspense. prior work from psychology used hand-crafted stories from literature and thus did not have the means to run controlled studies with a large number of stories with fine-grained interventions. we conducted a study to investigate the timing and ordering of knowledge revealed to the reader, as well as the relationship between empathy for the protagonist and suspense. we specifically investigate how the following affect suspense perception: (1) clues that set up the failure; (2) information revelation timing (before vs after); (3) reader's empathy. clue setup: we recruited 90 participants to ask for their preferences regarding the suspense level of 30 randomly selected pairs of stories. for each pair of stories, which is generated with and without clue setup (detailed in subsection 3.3) respectively, we ask participants which one of the two stories makes them feel a higher level of suspense. 57.9% of participants reported higher suspense when reading stories with clues, whereas 10.9% of participants reported higher suspense when reading stories without clues (31.2% were a tie). these results are significant at @xmath0.05 confidence level using a wilcoxon sign test. thus we conclude that clues of impending failure are a factor that increases reader perceptions of suspense. information revelation timing: we recruited 90 participants to ask for their preferences regarding the suspense level of 30 randomly selected pairs of stories. for each pair of stories, which is generated by revealing the information beforehand and after the fact (detailed in subsection 3.3) respectively, we ask participants which one of the two stories makes them feel a higher level of suspense. 42.1% of participants reported higher suspense when read- reader empathy: we recruited 30 participants to ask for both the level of empathy for the protagonist and the level of suspense they feel (@xmath1. results are presented in figure 2 as a heatmap which shows that a high level of perceived suspense is usually associated with a high level of reader's empathy. however, there does exist a noticeable portion of participants who exhibit a very low level of perceived suspense even when they are reading suspenseful stories, associated with a very low level of empathy. one possible explanation for this phenomenon is methodological. not a lot of character setup is given in the experiments and empathy may not be triggered strongly in some participants, who then also do not perceive suspense. ## conclusions in this paper, we propose a theory-grounded iterative-prompting-based planning approach to generating suspenseful stories. to the best of our knowledge, this is the first attempt at suspenseful story generation with llms. through extensive human evaluations, we prove that our method is more capable of producing suspenseful stories than the baseline method. our in-depth analysis of factors that affect suspense perception is also informative for story generation researchers and practitioners.