Overview of Long Story Generation Challenge ( LSGC ) at INLG 2024 (2024)

Abstract: This report describes the setup and results of the shared task of human-like long story generation, the LSG Challenge, which asks to generate a consistent, human-like long story (a Harry Potter fanfic in English for a general audience) given a prompt of about 1,000 tokens. We evaluated the submissions using both automated metrics and human evaluation protocols. The automated metrics, including the GAPELMAPER score, assessed the structuredness of the generated texts, while human annotators rated stories on dimensions such as relevance, consistency, fluency, and coherence. Additionally, annotators evaluated the models’ understanding of abstract concepts, causality, the logical order of events, and the avoidance of repeated plot elements. The results highlight the current strengths and limitations of state-of-the-art models in long-form story generation, with key challenges emerging in maintaining coherence over extended narratives and handling complex story dynamics. Our analysis provides insights into future directions for improving long story generation systems.

## introduction this report presents an analysis of the results of the long story generation challenge (lsgc), where participants showcased their systems for creating extended stories. with this shared task, we aimed to advance the generation of long-form literary texts. our evaluation was based on two main criteria: statistical metrics and a human evaluation protocol. the lsgc was originally proposed by @xcite ; this report follows the cited work closely. over 110 years ago, mathematician andrei markov demonstrated how to study effectively the text using mathematical methods @xcite . in his work, he examined the relationship between vowels and consonants in the early chapters of eugene onegin. he later gave his name to processes known as markov chains. markov chains formed the basis of early text generation algorithms that generated basically nonsense based on the probabilistic distribution of words in a text. today, text generation has advanced tremendously. autoregressive probabilistic large language models (llms) have become a cornerstone for solving every task in computational linguistics through few-shot learning @xcite or prompt engineering @xcite . many users now interact with advanced commercial models such as gpt, claude, or google bard in chat setting regularly. however, these models still have many deficiencies. despite the targeted effort, they can generate false information, propagate social stereotypes, and produce toxic language @xcite . specifically, current autoregressive language models fail to catch long-range dependencies in the text consistency. while the autoregressive window for commercial models reaches tens or even hundreds of thousands of tokens at the time of writing, which is a lot, it, however, does not allow them to generate long coherent texts. while relevance, consistency, fluency and coherence are relatively easily achieved by the latest autoregressive generative models on short texts (under 10k tokens), all the current models fail when one tries to generate a long story in a single pass. modeling long stories requires many additional abilities compared to short texts @xcite , including (1) commonsense reasoning regarding characters' reaction and intention, and knowledge about physical objects (e.g., ''river'') and abstract concepts (e.g., ''irony''); (2) modeling discourse-level features such as inter-sentence relations (e.g., causality) and global discourse structures (e.g., the order of events); and (3) the generation coherence and controllability, which require both maintaining a coherent plot and adhering to controllable attributes (e.g., topics). several authors have shown theoretically and empirically @xcite @xcite that the power law autocorrelations decay is closely connected to the hierarchical structure of texts. indeed, the hierarchical structure of, for example, leo tolstoy's war and peace consists of at least 7 levels: the whole novel, books, parts, chapters, paragraphs, words, and letters. there are strong reasons to think that this structure reflects an important aspect of human thinking: people do not generate texts autoregressively. writing a long text requires some thinking ahead, and going back to edit previous parts for consistency. this going back and forth can be reflected by navigating a tree-like structure. the autoregressive nature of the current state-of-the-art models does not reflect this; for example, s4 model @xcite exhibits clear exponential autocorrelations decay @xcite . ## task description the lsg challenge task required participants to provide a system that could output a coherent, human-like long story (a harry potter fanfiction for a general audience of at least 40,000 words) given a prompt of about 1,000 tokens. the organizers provided a set of story starters for developers. systems were evaluated based on text generated from these starters, written by volunteers and imitating the stylistic features of harry potter fan fiction. the starters were designed from scratch specifically for this task. it is important to note that no copyrighted texts were used in the creation of our dataset. the evaluation protocol below also does not require the usage of any of the original harry potter texts. it is based on the assumption that the assessors have a general knowledge of the harry potter universe, and this is enough to rate the texts using the provided questionnaries. we employ both automatic and human evaluation to evaluate the quality of the texts. in particular, we used gapelmaper @xcite as an unreferenced automatic, statistical metric of the text structuredness. we adopt multiple human evaluation metrics to better measure model performance. similarly to @xcite , we ask annotators to rate the texts across four dimensions: extending @xcite we ask annotators to rate repeating similar texts. finally, we asked the annotators to evaluate the creative dimensions of the resulting texts: ## dataset description story starters were created by undergraduate students majoring in linguistics as a part of their coursework with a proper credit. for testing and development purposes, we presented participants with five distinct story starters. ## shared task timeline the lsgc was planned throughout the recent academic year. the key dates of the shared task were: ## baseline we developed a baseline, published at https://lsgc.vercel.app/baseline , that generates a fan fiction text complying to the shared task requirements to make sure that the shared task is feasible. in light of the shared task's objective to create a lengthy, coherent fanfiction, we incorporated a hierarchical prompting system into the baseline to ensure the narrative's "completeness". the baseline implements a process that begins with a "story starter". by establishing a clear narrative structure, we create a framework for generating additional content, with the aim of remaining faithful to the original story in the fanfiction we produce. after setting up the narrative framework, we then focus on fleshing out the details of each section, creating chapter outlines that outline the events to be included (see figure 1 ). the number of chapters produced will depend on the capabilities of the generative model to generate believable text. this includes:  introduction: establishing the protagonist's world and introducing key themes. ## participants two teams participated in the challenge. each team submitted one story generated using their systems. all texts were anonymized prior human evaluation to ensure objective evaluation. each text was assessed using the gapelmaper metric and the human evaluation described below. team 1 (decision stump, boriskin, galimzianova, 2024) -the approach does not include any fine-tuning and utilizes llama 3 with 70b parameters with special prompting scheme for the text generation. team 1 developed the baseline in the direction of generating of the book components. the full pipeline consists of 2 partssummary generation and generation of chapters in a loop with the transmission of context about previous events in the book via the system prompt. the team presented a text consisting of 14 chapters, each chapter spanning 10 pages. this design aims to have only 14 potential points of discontinuity (at the junctions between chapters) where plot inconsistencies might arise, such as repeated scenes. for instance, at the end of chapter 1, the main character theo encounters the heroine pansy, and at the beginning of chapter 2, the model again describes their meeting. however, even such minor flaws blend reasonably harmoniously into the overall context. throughout the fanfic, the narrative thread is maintained, making it challenging to distinguish the text from that of a real author. team 2 (neurowling, seredina, 2024) -approach is based on fine-tuning the mistral-7b-instruct-v0.2-gptq model with supervised learning (sl). the final text of a fanfiction was generated with the fine-tuned model and the prompts following the baseline. the team also delivered commendable results: their text comprises numerous short chapters with rapidly unfolding action, unlike the first team's story. this format makes for easier reading, but due to the brevity of each chapter (approximately one page), inconsistencies and contextual discrepancies can be noticed at the chapter boundaries. for example, a character's gender might be female in one chapter and male in another. nevertheless, the provided structure, which involved preliminary generation of all chapters according to a unified concept (outline), ensured reasonable consistency of the narrative. ## conclusions both teams have demonstrated their capacity to generate long-form narratives with structured coherence, as evidenced by their gapelmaper scores. however, based on the combined quantitative and qualitative evaluations, team 2 ("neurowling") emerges as the stronger contender. both teams not very significantly departed from the baselines in terms of the system architecture. the results of both teams also only sparsely improved on the baseline. the gapelmaper score of 0.44 for team 2 indicates a significantly more cohesive narrative structure compared to team 1's score of 0.91. although both texts exhibited certain narrative inconsistencies, team 2's shorter chapter format and faster pacing made the text more accessible to readers, even if this format occasionally led to contextual discrepancies. moreover, the manual evaluation highlighted that team 2's text maintained a better alignment between chapter titles and content, as well as a clearer thematic structure. while team 1 ("decision stump") produced a more extensive narrative, the manual assessment revealed that this length led to redundancy and a lack of emotional engagement, as well as difficulties for readers unfamiliar with the "harry potter" universe. in contrast, team 2's text, despite its flaws, was more favorably received in terms of readability and structure. the evaluators easily detect the generated texts. the generated texts are still behind even nonprofessionally writing humans in terms of text quality and creativity. nevertheless, we can say that our expectations for this challenge were reasonably justified. the results of this study show the difference between using fine-tuning and prompt engineering approaches in text generation and demonstrate the advantages and disadvantages of each. in future, we would like to continue this research with a larger data set, and see more diverse text generation approaches from participants. this would allow us to get closer to understanding the linguistic nature