Long Story Generation Challenge (2023)

Abstract: We propose a shared task of human-like long story generation, LSG Challenge, that asks models to output a consistent human-like long story (a Harry Potter generic audience fanfic in English), given a prompt of about 1K tokens. We suggest a novel statistical metric of the text structuredness, GloVe Autocorrelations Power/ Exponential Law Mean Absolute Percentage Error Ratio (GAPELMAPER) and the use of previously-known UNION metric and a human evaluation protocol. We hope that LSG can open new avenues for researchers to investigate sampling approaches, prompting strategies, autoregressive and non-autoregressive text generation architectures and break the barrier to generate consistent long (40K+ word) texts.

## task overview the human-like long story generation (lsg) task asks models to output a consistent human-like long story (a harry potter generic audience fanfic in english), given a prompt of about 1k tokens. the text will be evaluated by automated metrics described in section 3.1, and a human evaluation protocol described in section 3.2. ## motivation autoregressive probabilistic large language models (llms) have become a cornerstone for solving every task in computational linguistics through few-shot learning @xcite or prompt engineering @xcite . many users now interact with such models as chatgpt, claude, or google bard in chat setting regularly. however, these models still have many deficiencies. despite the targeted effort, they can generate false information, propagate social stereotypes, and produce toxic language @xcite . the llm deficiency we particularly want to attack is their inability to produce a human-grade long text. current autoregressive language models fail to catch long-range dependencies in the text consistently. large language models such as @xcite , llama @xcite , alpaca @xcite push the boundary of "short text" rather far, but do not remove the problem. commercial instructionfollowing language models such as chatgpt, gpt-4, claude and google bard are targeted at the use in a dialogue (and probably that is not for nothing). they generate a limited number of tokens per user input, and only generate further text after additional prompting. while the autoregressive window for commercial models at the time of writing reaches 32k tokens for openai and even 100k tokens for anthropic, which is a lot, it does not allow them to generate long coherent texts. while relevance, consistency, fluency and coherence are easily achieved by the latest autoregressive generative models on short texts (under 10k tokens), all the current models fail when one tries to generate a long story in a single pass. modeling long stories requires many additional abilities compared to short texts @xcite , including (1) commonsense reasoning regarding characters' reaction and intention, and knowledge about physical objects (e.g., ''river'') and abstract concepts (e.g., ''irony''); (2) modeling discourse-level features such as inter-sentence relations (e.g., causality) and global discourse structures (e.g., the order of events); and (3) the generation coherence and controllability, which require both maintaining a coherent plot and adhering to controllable attributes (e.g., topics). @xcite have recently studied autocorrelations in long texts using pretrained word vectors. that allowed to study a wide range of autocorrelation distances in human-written and model-generated texts and show that the autocorrelations in human-written literary texts decay according to power laws on distances from 10 to 10k words independently from the language. on the other hand, the behavior of autocorrelations decay in generated texts is quantitatively and often qualitatively different from the literary texts. large language models often exhibit markovian @xcite behavior with exponential autocorrelations decay. several authors have shown theoretically and empirically @xcite that the power law autocorrelations decay is closely connected to the hierarchical structure of texts. indeed, the hierarchical structure of, for example, leo tolstoy's war and pease consists of at least 7 levels: the whole novel, books, parts, chapters, paragraphs, words, and letters. there are strong reasons to think that this structure reflects an important aspect of human thinking: people do not generate texts autoregressively. writing a long text requires some thinking ahead, and going back to edit previous parts for consistency. this going back and forth can be reflected by navigating a tree-like structure. the autoregressive nature of the current state-of-the-art models does not reflect this, for example, s4 model @xcite exhibits clear exponential autocorrelations decay @xcite . we hope that this challenge can gain interest from the nlg community and advance sampling approaches, prompting strategies, autoregressive and non-autoregressive text generation architectures and other subfields of text generation. ## task description formally, the task of lsg challenge asks participants to provide a system that can output a consistent human-like long story (a harry potter generic audience fanfic at least 40k words long), given a prompt of about 1k tokens. a set of at least three dev prompts will be provided by organizers. the systems will be evaluated on a withheld test prompt. the prompts similar to the beginnings of human-written fan fiction will be developed from scratch specifically for the task. it is important to note that no copyright-eligible texts will be used in the shared task. the evaluation protocol below does not require using the original harry potter texts, and subjective evaluation relies on the fact that judges have read harry potter books/seen the films, but no factual knowledge of harry potter books is also required for the evaluation criteria below. given the open-ended and cutting-edge nature of the generation task and ongoing discussion on the best corpora and approaches to training llms, we feel that constraining the training set can be harmful to the task performance and participants are open to train their models on any dataset, as long as it is described in the system report. we employ both automatic and human evaluation, described below to evaluate the quality of the texts. ## conclusion we propose the lsg challenge to address the task of long text generation, with the hope that it can open new avenues for researchers to investigate sampling approaches, prompting strategies, autoregressive and non-autoregressive text generation architectures and break the barrier to generate consistent long (40k+ token) texts, and the frontier of text generation can be pushed further.