Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluation (2024)

Abstract: Storytelling is an integral part of human experience and plays a crucial role in social interactions. Thus, Automatic Story Evaluation (ASE) and Generation (ASG) could benefit society in multiple ways, but they are challenging tasks which require high-level human abilities such as creativity, reasoning, and deep understanding. Meanwhile, Large Language Models (LLMs) now achieve state-of-the-art performance on many NLP tasks. In this paper, we study whether LLMs can be used as substitutes for human annotators for ASE. We perform an extensive analysis of the correlations between LLM ratings, other automatic measures, and human annotations, and we explore the influence of prompting on the results and the explainability of LLM behaviour. Most notably, we find that LLMs outperform current automatic measures for system-level evaluation but still struggle at providing satisfactory explanations for their answers.

## introduction the task of automatic story generation (asg) @xcite consists in the creation of a narrative from a short sentence. previous research showed that storytelling enables a narrator to communicate honestly with their audience @xcite and to provide listeners with an engaging and instructive experience @xcite . indeed, the process of story creation is a salient testimony of human creativity, requiring both the discovery of interesting ideas and their adept expression through a carefully-built narrative. strong automatic story generating systems could therefore be useful for a variety of applications, such as gaming @xcite , education @xcite , mental health @xcite , and marketing @xcite . meanwhile, over the last few years, advances in natural language processing (nlp) have been spearheaded by the development of large language models (llm) such as @xcite , lamda @xcite , palm @xcite , and llama @xcite . upon release, these models have been setting new state-of-the-art performance standards for a wide array of nlp tasks, e.g., question answering, summarization, and translation. in particular, for asg, llms are now able to produce convincing stories, so much so that they can be hard to distinguish from human stories @xcite . as their performance improves, they may become valuable assistants to our creative process; already, writing contests have been shown to encourage their use @xcite . the increased availability of llms to the general public underlines the need for reliable story evaluation methods that can be used to improve both the performance of asg models and our understanding of their strengths and weaknesses. since the human annotation of stories is costly and time-consuming @xcite , automatic story evaluation (ase) systems could provide an efficient and scalable replacement for human evaluation. however, current automatic measures have been shown to be poorly correlated with human judgment for asg @xcite . in this paper, we investigate whether llms themselves can be used as substitutes for human annotators for story evaluation. to that end, we perform several annotation experiments where we ask different llms to rate stories according to different criteria and to explain their rating. we show an example in figure 1 and a schema of our experiments in figure 2 . our contributions are the following: ## analysis of the results our work aims at answering five important questions for ase and asg: ## discussion on llm performance our work is part of the ongoing research on the general ability of llms for understanding and thinking. @xcite distinguish formal (the statistical features of language) and functional linguistic competence (the ability to use language in the world) and show that llms are very successful on formal linguistic tasks but struggle at functional linguistic tasks. @xcite argue that llms do display impressive performance at a wide variety of tasks but lack ''slow thinking'' capabilities, referring to the system 1-system 2 dichotomy introduced by @xcite . thus, the high performance of llms at ase should be interpreted with caution: we hypothesize that the ''rating'' part of our story evaluation experiments could be linked to formal linguistic competence and the fast, automatic system 1, while the ''explanation'' part would correspond to functional linguistic competence and the slow, conscious system 2. this analogy would explain the good correlations of llm ratings with human ratings: the internal criterion of llms for story evaluation may be formal quality (vocabulary, syntax, grammar), regardless of the criterion mentioned in the eval-prompt. indeed, the six criteria from @xcite are mostly orthogonal but not completely independent: their correlation with one another may be related to the general ''system 1'' tendency of human raters to favor stories that display better formal qualities. in that sense, llms may reflect a human bias towards easy, intuitive thinking. by contrast, the less convincing performance of llms at explaining their ratings may highlight their weaker system 2 capabilities as argued by @xcite and @xcite . 6 conclusions 6.1 practical takeaways 1. used with prompts based on specific criteria, llms are currently the best proxy for human evaluation of story generation (section 4.1.2). in particular, llms display very high system-level correlations with human judgment.