The LSG Challenge Workshop at INLG 2024: Prompting Techniques for Crafting Extended Narratives with LLM s (2024)

Abstract: The task of generating long narratives using Large Language Models (LLMs) is a largely unexplored area within natural language processing (NLP). Although modern LLMs can handle up to 1 million tokens, ensuring coherence and control over long story generation is still a significant challenge. This paper investigates the use of summarization techniques to create extended narratives, specifically targeting long stories. We propose a special prompting scheme that segments the narrative into several parts and chapters, each generated iteratively with contextual information. Our approach is evaluated with GAPELMAPER, a sophisticated text coherence metric, for automatic evaluation to maintain the structural integrity of the generated stories. We also rely on human evaluation to assess the quality of the generated text. This research advances the development of tools for long story generation in NLP, highlighting both the potential and current limitations of LLMs in this field.

## introduction long story generation with llms is an underexplored topic in nlp. most recent llms with wide context windows intuitively seem to be an appropriate tool for this task. however, in practice researchers often struggle to control the generation and keep it consistent @xcite . we propose to use summarization and for llms to generate long stories of 40,000 words in length. our approach does not require any fine-tuning and utilizes llama 3 with 70b parameters with special prompting scheme. we score relevance, consistency, fluency and coherence of the text in human evaluation and gapelmaper in automatic evaluation on the lsg challenge task. we make the code publicly available. 1 our pipeline can be summarized as follows: ## related work long story generation is a dynamic field of nlp with new approaches emerging quickly after the introduction of llms. co-authoring with llms has been suggested in @xcite . storyverse, a system for human-driven story generation, leverages llms for character simulations. the plot is written by humans, while a language model is responsible for detailed story development. we propose to automate the process of plot creation as well, with specifically crafted role prompts for an instructiontuned model. another interesting approach of using language models as co-writers is explored in @xcite . interleaved (generated with human help) stories are found to be less preferred by human readers than non-interleaved stories. we use these findings to construct a long narrative in a fully automatic way. unlike the authors of this work, who only test their approach on commercial models, we deploy an open-source llm, which allows for more flexibility in tuning generation parameters. iterative story planning with llms is presented in @xcite . this study utilizes prompting techniques and relies on findings from psychology to construct the system. in attempts to automate the story generation further, the authors of @xcite develop a multi-llm prompt-based approach where different models are responsible for various story components generation. we draw upon these and other works to introduce our approach that leverages prompting and summarization techniques to generate long fiction stories with an open-source llm. ## method in the competition baseline, the number of book parts (6) and the number of chapters in each part (12) are hardcoded. then, in a loop, mixtral model generates an entire part of the book, providing context in the form of the book's plot (main characters, storyline, etc.). we decided to continue to develop the idea of the baseline based on generation of the book components. the full pipeline consists of 2 partssummary generation and generation of chapters in a loop with the transmission of context about previous events in the book via the system prompt. here are our sampling parameters for both parts in ## conclusion our exploration into the use of summarization techniques for long story generation with large language models has revealed promising avenues and notable challenges. the iterative generation process, combined with an evaluation metric like gapelmaper, shows potential in producing coherent and structured extended narratives. however, the difficulty in maintaining narrative consistency and control over extensive text generation underscores the need for further refinement of these techniques. future work should focus on enhancing the control mechanisms and coherence metrics to better harness the capabilities of llms for long-form storytelling. this study lays the groundwork for more advanced narrative generation frameworks, pushing the boundaries of what llms can achieve in story telling task. 122 kaige xie and mark riedl. 2024. creating suspenseful stories: iterative planning with large language models. arxiv preprint arxiv:2402.17119. zoie zhao, sophie song, bridget duah, jamie macbeth, scott carter, monica p van, nayeli suseth bravo, matthew klenk, kate sick, and alexandre ls filipowicz. 2023. more human than human: llmgenerated narratives outperform human-llm interleaved narratives. in proceedings of the 15th conference on creativity and cognition, pages 368-370. a appendix a.1 prompts a.1.1 prompt 1 -summary you are a popular fanfiction creator tasked with writing an extended piece for a harry potter fanfiction. you have to come up with an interesting plot. follow the classical composition and include beginning, climax and outcome in this book. write a detailed plan for each composition chapter. in the format of chapter 1: what happens in this chapter chapter 2: what happens in this chapter ... it is important to display the following aspects in the plot: action and conflict: introduce conflicts and challenges that the characters must face. whether it's a battle with dark forces, a personal dilemma, or a complex mystery, ensure there is plenty of action and tension to keep readers hooked. magical elements: highlight the magical aspects of the harry potter universe. describe new spells, potions, magical creatures, and enchanted locations. make magic an integral part of the plot and the characters' lives. world-building: expand on the existing lore of the harry potter universe. introduce new locations, traditions, and histories. make the world feel alive and full of possibilities. do not use p.s., p.p.s. and exc. the following text is the beginning of the first chapter of this book. generate the summary according to this text.