The Next Chapter: A Study of Large Language Models in Storytelling (2023)

Abstract: To enhance the quality of generated stories, recent story generation models have been investigating the utilization of higher-level attributes like plots or commonsense knowledge. The application of prompt-based learning with large language models (LLMs), exemplified by GPT-3, has exhibited remarkable performance in diverse natural language processing (NLP) tasks. This paper conducts a comprehensive investigation, utilizing both automatic and human evaluation, to compare the story generation capacity of LLMs with recent models across three datasets with variations in style, register, and length of stories. The results demonstrate that LLMs generate stories of significantly higher quality compared to other story generation models. Moreover, they exhibit a level of performance that competes with human authors, albeit with the preliminary observation that they tend to replicate real stories in situations involving world knowledge, resembling a form of plagiarism.

## introduction automatic story generation poses a significant challenge as it requires more than just individually coherent sentences. a good story should exhibit a natural flow, adhere to commonsense logic, and be captivating to the reader. in recent times, the prevailing approach in story generation involves fine-tuning pre-trained language models (plms) like gpt-2 @xcite or bart @xcite ) on specific datasets. these models generally excel in generating fluent sentences, devoid of glaring grammar issues. however, they often struggle to construct a coherent story that adheres to commonsense and fails to create an engaging narrative @xcite . to overcome these challenges, state-of-the-art (sota) story generation models integrate higher-level features, such as plots and commonsense knowledge. prompt-based learning @xcite is a recent paradigm specifically tailored for large language models (llms) that possess in-context learning capabilities @xcite . in contrast to the conventional "pre-train and fine-tune" approach, which necessitates a substantial amount of data for fine-tuning, promptbased learning enables llms to learn a task by providing them with multiple examples as a "prompt", eliminating the need for gradient-based fine-tuning @xcite . recently, llms have demonstrated remarkable performance across various language generation tasks, with notable attention on models such as chatgpt and gpt-4 @xcite @xcite . for instance, a comparative analysis by @xcite highlighted the superior performance of fine-tuned llms over smaller pre-trained models in zero-shot scenarios for tasks like dialogue and summarization. it is worthwhile to note though, that story generation was not specifically examined in their experiments. this paper aims to address this research gap by conducting a comprehensive evaluation of automatic story generation. specifically, we compare the performance of llms, with a particular focus on gpt-3, using prompt-based learning, against sota models. we compare generated stories in terms of various automatic evaluation metrics from lexical and semantic matching ones to recently proposed model-based ones. we follow the best practice in literature to conduct rigorous human evaluations including both crowdworkers from amazon mechanical turk and in-house judges, and assess story quality at a fine-grained level, such as coherence and logicality. to summarise, our contributions are: there are also studies which explore the use of gpt-3 for story generation. for example, @xcite conducts a turing test between gpt-3 generated and human-written stories @xcite probe for gender and representation bias in gpt-3 generated stories. these studies, however, do not provide a systematic evaluation that assesses gpt-3 against the sota story generation models. story evaluation automatic story evaluation is admittedly a challenging task, and the lack of standardized evaluation metrics has somewhat impeded progress of story generation @xcite . human evaluation is usually considered as the gold standard for story quality evaluation, but it is expensive and time-consuming @xcite and it can not capture diversity @xcite . subsequently, several automated evaluation metrics are introduced as alternative measures to evaluate the quality (the degree of readability) and diversity (the extent of variation) of the generated stories. for quality, most metrics measure lexical overlap between strings @xcite @xcite or semantic similarity by comparing embedding of models @xcite between generated stories and their human references. recently, learning @xcite and generation @xcite based methods are explored and they are based on pre-trained language models such as bert @xcite and bart. nevertheless, these evaluation metrics are limited in that they provide a single score to indicate the overall quality of the story, and few metrics are specifically designed to assess specific aspects such as logicality (the adherence to commonsense) or interestingness (the level of reader engagement) @xcite . ## human evaluation to obtain a comprehensive assessment of the generated stories, we recruit human annotators to evaluate their quality. in order to gain insights into consistency, we employed both crowdsourced workers and in-house annotators. this approach allows us to gather diverse perspectives and obtain a more nuanced understanding of the story quality. ## plagiarism considering the strong performance of gpt-3 on story generation, we next provide a preliminary investigation to understand the extent to which gpt-3 copies from its training data. ithenticate we use ithenticate foot_4 -a professional plagiarism detection software that has comprehensive coverage over online articles -to assess how much gpt-3 plagiarises. we include only the generated content (without the condition) when checking for plagiarism. results show that there is no strong plagiarism: similarity score for roc, wp and cnn is 4%, 3% and 14% respectively. this is consistent with findings from @xcite that the language models do not simply memorize, but combine familiar parts into novel way. however, ithenticate looks for lexical overlap to identify plagiarism. a more subtle form of plagiarism is one that copies the ideas without parroting the words @xcite , motivating us to perform manual check. manual check we randomly sample 10 gpt-3 generated stories for wp and cnn respectively and manually check whether they copy from web articles by searching for keywords, named entities and long phrases. for wp, in most instances we do not find web articles that resemble our stories, except for one instance where it copies a story from wikipedia (noting that it is not a direct word-toword copy). for cnn, we find 7 out of 10 instances are based on real stories from news articles, where most details (e.g., names, incident origins, sentence penalty) are faithful to the original stories. these ## discussion in this paper, we only experiment with a simple prompt-based learning approach to adapt gpt-3 to the different story datasets and one can argue more prompt engineering can be explored @xcite @xcite . we contend, however, that our goal here is not to propose a better prompt-based method to do story generation with gpt-3, and ultimately even with our simple approach we find that gpt-3 generates high quality stories, suggesting that with additional prompt engineering it can do better. we next discuss qualitatively some of the strengths and drawbacks of gpt-3 in story generation. ## conclusion we present an extensive comparison of gpt-3 with sota models for story generation, and found that stories generated by gpt-3 are substantially better than sota models on multiple aspects and even rival human authors. the findings of this study indicate that we have entered a new chapter in story generation with llms. future research is likely to concentrate on prompt-engineering llms to achieve enhanced customization, such as varying their style and length, further advancing the capabilities of story generation models. in terms of evaluation metrics, our work: 1) reveals a weak correlation between automatic lexical-based evaluation metrics and human evaluation, and that recently proposed model-based metrics appear to more reliable; and 2) contributes a new test bed for metric development, through the release of a dataset that contains story quality annotations by two groups of judges. in spite of the positive results of gpt-3 in story generation, we discuss some of its issues, the chief one being that it has a tendency to reproduce details or plots from its memories, raising foundational questions about its generation creativity.