A Confederacy of Models: a Comprehensive Evaluation of LLM s on Creative Writing (2023)

Abstract: We evaluate a range of recent LLMs on English creative writing, a challenging and complex task that requires imagination, coherence, and style. We use a difficult, open-ended scenario chosen to avoid training data reuse: an epic narration of a single combat between Ignatius J. Reilly, the protagonist of the Pulitzer Prize-winning novel A Confederacy of Dunces (1980), and a pterodactyl, a prehistoric flying reptile. We ask several LLMs and humans to write such a story and conduct a human evalution involving various criteria such as fluency, coherence, originality, humor, and style. Our results show that some state-of-the-art commercial LLMs match or slightly outperform our writers in most dimensions; whereas open-source LLMs lag behind. Humans retain an edge in creativity, while humor shows a binary divide between LLMs that can handle it comparably to humans and those that fail at it. We discuss the implications and limitations of our study and suggest directions for future research.

## introduction in recent years, large language models (llms) have achieved remarkable progress in a wide range of language processing and generation tasks, such as question answering, machine translation, or text summarization, among many others @xcite . this has motivated research on evaluating and comparing the performance of llms in various tasks, both between each other and with respect to human performance; including both task-specific evaluations (see e.g. @xcite ) and overarching benchmark suites that seek to provide comprehensive evaluation throughout many dimensions @xcite @xcite . creative writing is also one application where llms have been observed to produce good results. according to @xcite , their generated outputs in poetry or storytelling are "often of astonishing quality", and @xcite showed that humans cannot reliably distinguish human-from llm-authored stories. however, and despite the amount of papers experimenting with llms for this purpose, an evaluation comparing the abilities of current llms as standalone systems for creative writing seems to be lacking. here, we provide such an evaluation, comparing the storytelling capability of 12 recent, instructionaligned language models between each other and with human writers. we do so using a rubric based on established creative writing evaluation proposals @xcite , but specifically adapted to the task. our comparison is performed on a purely zero-shot setting, with a natural human prompt (based on a combat between ignatius j. reilly, protagonist of a confederacy of dunces, and a pterodactyl) that has been specifically chosen to be challenging and meaningful while preventing as much as possible the option for llms to resort to regurgitating or adapting material from their training set. here our goal is not to produce a specialized system, but to evaluate the performance of llms by themselves as creative writers. thus, we focus on the purely zero-shot setting, where a generalistic llm is asked to write a story with no extra fine-tuning, in-context learning @xcite , prompt engineering or additional components. this has only become viable with the extra coherence and consistency in long texts provided by newer llms, especially those that are aligned to follow instructions with instruction tuning @xcite or reinforcement learning with human feedback @xcite . to our knowledge, there was no previous work in this line. in fact, evaluation in creative writing is a conspicuous gap in llm evaluation benchmarks: the huge big-bench suite @xcite currently has over 200 tasks, but does not include any creative writing, and helm @xcite cites it as an "aspirational scenario" for future work. this likely owes to benchmarks focusing on easilyautomatable metrics, whereas the gold standard for creative writing is human evaluation @xcite , which is much costlier. the closest previous work to our proposal is the recent preprint by @xcite , where gpt-3 is compared to previous storytelling systems via human evaluation. however, there are several impor-tant differences with respect to our work: (1) they use prompt-based learning, providing examples to adapt the model to the task, rather than a purely zero-shot conversational prompt, (2) they evaluate a single llm while our goal is to compare llms, and (3) they use pre-existing story datasets, which increases the risk of models benefitting from similar stories present in their training set, something that we have tried to avoid as described below. in another recent preprint, @xcite generate lovecraftian horror literature. however, they also focus on a single llm (gpt-4), using careful prompt engineering to optimize its performance rather than a pure zero-shot setting, and evaluation is only on whether humans can distinguish ai-generated from real stories (concluding that, in those circumstances, they cannot). @xcite apply a similar evaluation (but automated) to whitmanian poems generated by three versions of gpt, also with a negative result. finally, concurrently with our study, a preprint by @xcite , released a few months after our submission, evaluates three llms for creative writing in a more similar way to ours: they apply human evaluation to compare stories by humans and llms in a zero-shot setting. however, there are important differences in methodology and scope between both studies. a comprehensive comparison will be made in section 5, following the exposition of our methods and results. creative writing evaluation creative writing is a challenging and complex performative language act that requires a number of skills, such as an expertise in craft, cultural and literary competency, linguistic fluency, coherence, complex connotative and metaphorical levels of understanding, innovation, originality and imagination, to name a few. the craft of writing involves innovation with style and voice, needs a fundamental understanding and use of structural elements (grammar, spelling, punctuation), craft elements (plot, character, setting, point of view and imaginative capacity, such skills defined by bloom as 'putting elements together to form a coherent or functional whole; reorganizing elements into a new pattern or structure through generating, planning, or producing' @xcite . evaluation of creative writing therefore must take into account all these factors, and assessment in university creative writing courses is usually based on a rubric that attempts to measure the basic elements of narrative craft, as well as the specific requirements on the assignment @xcite @xcite @xcite . ## discussion we have evaluated recent llms on a creative writing task in english, using a carefully-designed scenario to provide a demanding challenge and avoid confounding factors like training data memorization @xcite . to our knowledge, this is the most thorough evaluation of llms on creative writing conducted so far, both in terms of scope (12 llms considered, plus comparison to human writers) and detail (using human evaluation with a 10-item rubric based on established creative writing evaluation practices). simultaneously to our work, the recent preprint by @xcite provides an evaluation of three of the top-performing commercial llms (chatgpt, gpt-4 and claude) for creative writing. this approach is close to ours, as it uses the models in a zero-shot setting and evaluation is performed by humans using a specific rubric. however, there are important methodological differences between both studies, which we summarize here: writers were unconstrained when they created them, while the llms have to adapt to write an alternative story with the same plot. in ours, humans and llms are given the exact same prompt to work with. the outcome of this study is substantially different from ours, with llm-generated stories rated clearly behind human-authored ones. this is not surprising considering the methodological differences: in particular, differences 1 and 2 in the list above clearly set a higher bar for llms, as they are compared to highly successful human stories by top authors that wrote freely and the llms are asked to adapt to their plots. we hypothesize that these are the main reasons for the difference in outcome. on the other hand, item 5 in the list above could in principle benefit llms, and there are other factors that could benefit humans or llms in nonobvious ways (including items 3, 4 and 6, as well as different story genres and target lengths). this underscores the need of more studies in this area. ## conclusion the results show that state-of-the-art llms can perform a creative writing task at a very competent level, with the top two (chatgpt with gpt-4 and claude) achieving high scores that outperform human writers in most rubric categories. while we must be careful not to take this as evidence of "superhuman storytelling" (both because our sample size is not enough to draw such categorical conclusions, and because our 5 human writers are not necessarily representative of human writing ability as a whole); it does at least strongly suggest that these models' stories are not distinguishably worse than those by reasonably-trained humans. this is even more remarkable given that we did not use any in-context learning or other techniques to optimize the llms for the task, but just a straightforward prompt from a fresh state, so it is possible that even better results are achievable with careful prompting. our analysis also shows that the best results are achieved by commercial llms, with open-source models clearly lagging behind at the moment. looking at individual characteristics, humans retain the lead in originality, while llms tend to excel in more technical aspects like readability or structure. humor is an especially challenging aspects where most llms utterly fail, but the best three models do succeed at achieving human-like ratings, contrasting with results on older llms that showed their lack of grasp of human humor @xcite . interesting avenues for future work include evaluation of different literary genres, languages other than english, and studying whether the quality of the generated stories can be improved with prompt engineering or fine-tuning. selected stories from our corpus (available at https://doi.org/10.5281/zenodo.8435671 , together with all rating data) are in appendix e.