Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing? (2024)

Abstract: Are LLMs ready to compete in creative writing skills with a top (rather than average) novelist? To provide an initial answer for this question, we have carried out a contest between Patricio Pron (an awarded novelist, considered one of the best of his generation) and GPT-4 (one of the top performing LLMs), in the spirit of AI-human duels such as DeepBlue vs Kasparov and AlphaGo vs Lee Sidol. We asked Pron and GPT-4 to provide thirty titles each, and then to write short stories for both their titles and their opponent’s. Then, we prepared an evaluation rubric inspired by Boden’s definition of creativity, and we collected several detailed expert assessments of the texts, provided by literature critics and scholars. The results of our experimentation indicate that LLMs are still far from challenging a top human creative writer. We also observed that GPT-4 writes more creatively using Pron’s titles than its own titles (which is an indication of the potential for human-machine co-creation). Additionally, we found that GPT-4 has a more creative writing style in English than in Spanish.

## introduction large language models (llms) have recently showed strong competences generating human-like text, and in particular in creative writing tasks @xcite , which is the focus of this paper. llms are increasingly influencing creative industries, impacting both the economy and the labor market, as highlighted by significant events such as the hollywood screenwriters' strike @xcite @xcite . experimentation shows that, under different settings, llms can perform better than average humans at short creative writing tasks @xcite . llms seem to be ready, then, for the next level of experimental inquiry: can they already compete with the best human creative writers? note that, in the history of artificial intelligence (ai), symbolic landmarks involve competition between the best ai systems with the best humans at the task, as in deepblue vs kasparov @xcite and alphago vs @xcite . however, despite extensive research into humanmachine collaboration @xcite , there is still little experimentation on how the best llms compare with the best fiction writers in autonomous creative text writing. in this work, we make the first attempt (known to us) to conduct a formal contest of autonomous creative writing between two top writers: gpt-4 turbo foot_0 (gpt-4-0125-preview, the best llm at the time of conducting this research, together with claude-3 opus @xcite and gemini ultra @xcite ) and patricio pron foot_1 , a distinguished spanish-speaking writer recognized among the 22 best writers of his generation by granta magazine foot_2 . our experimentation, and this paper, are structured along the following research questions: rq1: can the current state of generative ai match the skills of the best human authors in creative writing tasks? as in previous ai duels, we do not try to compare a top ai machine with average humans: we focus on a one-on-one comparison between two (ai and human) top performers. comparing with one top writer certainly limits the scope of our results, but also lets us put all experimental efforts in providing a comprehensive side by side evaluation, which involves designing 60 text writing assignments, under the same conditions for both contenders, and collecting the manual evaluation of literary critics on those 180 pieces of text (60 by pron, 60 by gpt-4 in english and 60 by gpt-4 in spanish), with a carefully crafted rubric composed of 10 questions each (see section 3 on methodology). rq2: what is the role of the prompt in the creativity of the generated text? different studies indicate that through prompting the model can be guided to write more creative texts @xcite . in our experiment, each text assignment is just a proposed title, and titles are provided by gpt-4 and patricio pron (30 titles each). in a second stage, they have to write 60 synopsis of imaginary movies with the proposed title, both for their own titles and their contender's. this lets us explore how the source of the title influences the quality dimensions of the texts produced, for each of the authors. by "prompt" we do not mean how things are asked to the llm; in our case the prompt variations consist in what is asked -the requested topic of the text as conveyed by the title. we believe that, in a creative writing setting, the request is part of the creative process, and we want to model how it influences the result. rq3: are llms less skilled for creative writing in languages other than english? the largest llms are often trained on unbalanced training data across languages where english is predominant. for example, llama-3 (meta, 2024) is only trained on 5% multilingual data. in our experimentation, we compare the performance of gpt-4 in english with its performance in spanish, the third most spoken language in the world, to verify if there is a performance gap even with respect to other resource-rich languages. rq4: does gpt-4 have a recognizable style for a literature expert when generating creative text without constraints? in our experimentation, we do not constraint gpt-4 writing style via prompting, and we request assessors to (blindly) identify if each text has been written by a machine or a human. we then explore not only if the assessors are able to recognize machine-produce text, but whether this recognition improves along the evaluation process. rq5: can we effectively measure creativity using boden's framework in the context of aigenerated texts? margaret boden's definition of creativity @xcite requires novelty, surprise, and value in creative outputs. these minimal requisites are ubiquitous in most definitions of creativity, and we used them as a guide to prepare an evaluation rubric for our expert assessors. the rubric provides a framework for an objective analysis of boden's dimensions, where we can measure if originality (novelty and surprise in the context of literary writing) and attractiveness (the value of the literary text) do correlate with creativity assessments. the main contributions of our work are: 1. we conduct the first comprehensive symmetrical empirical study that compares a state of the art llm (gpt-4) with an award-winning novelist, quoted as one of the best of its generation. with this comparison we approach the question of whether llms can already be better than any human at creative writing, rather than better than average humans. our methodology includes a carefully crafted rubric to evaluate creative writing texts according to boden's dimensions of creativity. ## related work since the rise of llm technology, creative text writing has gained renewed interest within the nlp research community. @xcite survey machine learning and creativity, discussing computational creativity theories, generative techniques, and evaluation methods. evaluating creativity remains challenging @xcite , but progress is being made; for an extensive explanation of the challenges of evaluating computational creativity see @xcite . regarding machine-assisted human writing, @xcite introduced story centaur, a tool for creative writers to prototype few-shot learning models. and @xcite presented copoet, a system for poetry writing that enhances user-generated content. in both cases, evaluators prefer texts generated in co-authorship with ia systems. however, @xcite highlighted limitations in current llm tools, such as issues with narrative consistency and plot development. our focus is rather on autonomous llms creative writing. @xcite examined the stylistic quality of ai-generated texts, finding them generally rated lower than human-written texts despite being indistinguishable. @xcite found that a fine-tuned bart model outperformed average human writers in a creative writing task, obtaining higher scores in grammaticality, coherence and attractiveness, and almost matching their creativity. unlike our study, they used casual readers to assess the texts, and the human texts were not produced by top writers. the study by @xcite examines the capability of several large language models (llms) in autonomous english creative writing, focusing on a single imaginative task where models and humans compose a story about a combat between ignatius j. reilly and a pterodactyl. they reveal that llms performed well in fluency and coherence but lagged in creativity and humor. their study's single-task focus contrasts with our broad evaluation of 60 titles. lastly, @xcite proposed the torrance test of creative writing (ttcw) to evaluate ai-generated stories. their findings reveal that while llms perform well in terms of fluency and structure, they lag significantly behind human writers in originality and emotional depth. a limitation of the study is that the tasks given to humans and machines are asymmetrical: human stories are selected from already published material. then, gpt-4 summarizes the stories, and llms are asked to generate a full story starting from each of the summaries, which is only a part of the creative writing process. another difference in methodology is that they adapt the ttcw test for their rubric, while we design our rubric following boden's notion of creativity. overall, our study complements previous work being the only one that simultaneously (i) uses the best possible writer and llm for the experimentation; (ii) gives the same tasks to both contenders in equal conditions (iii) explores 60 different writing assignments (proposed by the contenders) and collects assessments for 180 texts using a rubric that adapts boden's notion of creativity to the task, and (iv) includes a study on the effect of the prompt and also measures the gap between texts written in english and spanish. ## experimental design contenders. the llm chosen for the experiment is gpt-4 turbo (in gpt-4-0125-preview version), which was the strongest llm when we initiated the experiment. after some initial experimentation with the system, we fixed temperature at 1: going beyond this value occasionally impacted on grammaticality (particularly with spanish texts), so we chose the highest value that produced always formally correct texts. once the experiment was initiated, other llms that seemed to rival the performance of gpt-4 were launched: most notably claude 3 opus, gemini ultra and llama 3. experimenting with these models, we did not notice any clear advantages with respect to gpt-4, so we proceeded with our initial setup. finding a top novelist that would engage in this experiment was easier than we initially thought. we contacted patricio pron because, besides being awarded with some of the most prestigious distinctions in spanish literature (the alfaguara award for novel writing, the juan rulfo narrative award, among others) and besides being translated into dozens of languages, he also has a strong curiosity towards artificial intelligence and autonomous machine writing. task design. in the first stage, each contender proposed 30 movie titles. in the second stage, both contenders wrote synopses (approximately 600 words) for each of the 60 titles. the prompt for gpt-4 was as follows: "we are conducting an experiment to compare your creative writing skills with those of the renowned novelist patricio pron. your task is to generate synopses for imaginary movie titles. these synopses should be creative, appealing to critics and audiences, and possess inherent literary value. here is some information about patricio pron: he is a celebrated writer, recognized as one of the top young writers in spanish by granta in 2010, and the winner of the alfaguara prize in 2019 for his work mañana tendremos otros nombres. the proposed title is: {title}. please write a 600-word synopsis that meets these criteria." more information on the prompts can be found in appendix c. languages. titles were originally proposed in spanish. then, we manually translated them into english. pron wrote a text in spanish for each of the titles, and gpt-4 wrote a text in spanish and a different text in english for each of the titles, so that we can measure how its writing skills depend on the language. rubric design. the rubric, designed by three experts in pedagogy, psychometrics, literature, and nlp, focuses on creativity-related dimensions, as previous work has shown that llms already excel at grammaticality, coherence and fluency @xcite . the point of departure is margaret boden's definition @xcite : creativity is the ability to come up with ideas that are new, surprising and valuable.; it is a simple, operative definition compatible with most studies on the subject, both from philosophers and psychologists, with a long tradition @xcite . it is a conceptualization of creativity in three specific dimensions: novelty, surprise and value. our experts in the process of creating the rubric agreed that, in fiction writing, novelty and surprise can be conflated into one single feature, originality. they rely on bartel's definition: a work is original if it is the first to display some unique or different attribute that is then adopted by other works @xcite . value, on the other hand, is a catch-all, which involves both economic and historical dimensions of art. the approach the experts take is intrinsically product-based: they evaluate the creativity of the text in itself; regardless of historical or social considerations that would make the evaluation noisy. in the context of fiction writing, they mapped value to attractiveness: a synopsis is valuable if it engages the reader and provides a satisfying reading experience. the experts' rubric encompasses the following quality dimensions (see appendix a for details) rated from 0 to 3: attractiveness: literary appeal of the title, the style of the text, and its content (theme/plot). criteria include the title's captivation, style's enjoyment, and the engagement of story and characters. originality: novelty and uniqueness of the title, the text style and the text theme/plot. criteria include the title's uniqueness, the style's distinctiveness, and the plot's innovation. creativity: this assessment evaluates the creativity of both the title and synopsis without distinguishing between style and theme. a unique aspect of this evaluation is the inclusion of the term "creativity" in the definition of each level. this approach aims to determine if evaluators' mental models of creativity correlate with the defined aspects of attractiveness and originality. the primary purpose of assessing creativity in this manner is to examine its correlation with attractiveness and originality. according to boden, creativity is defined as something that is new, surprising, and valuable. originality involves novelty and surprise regardless of the value of the text. this distinction is reflected in how experts develop the evaluation rubric, where each degree of originality is clearly defined. the creativity rubric is designed to reveal how evaluators-such as critics and literature scholars-perceive creativity. specifically, we aim to measure its correlation with originality and attractiveness to validate boden's definition. ## results and discussion in this section, we present the main findings of our study. each subsection provides a detailed analysis of the expert assessment annotations in order to answer our research questions. ## conclusions our results indicate that gpt-4 turbo, despite its impressive writing capabilities, still falls short of matching the skills of a world-class novelist. texts generated by gpt-4 are consistently rated lower in all quality dimensions in our study: attractiveness and originality of both style and theme, and overall creativity, among others. comparing with previous results, this indicates that it is much easier to match the average performance of human writers than to actually match the best ones: llms still lack the nuanced depth, originality and intent characteristic of a top novelist such as patricio pron. also, our study highlights the significant role of prompts in creative text writing: titles provided by pron resulted in gpt-4 texts which are significantly more creative and original than the ones written for its own titles. even the simplest prompting (short titles in our case) should be considered coauthorship, as it has a profound influence on the results. we also found that gpt-4's performance in generating creative texts was significantly better in english than in spanish, in spite of being also a resource-rich language. this discrepancy is likely due to the model being trained on a larger corpus of english text, reflecting a bias towards english in the available training data. the results underscore the need for more balanced and comprehensive training datasets to enhance the multilingual creative writing capabilities of ai systems. our expert evaluators were able to identify aigenerated texts with increasing accuracy over time, suggesting that gpt-4 has a recognizable style that becomes more apparent as evaluators gain experience with its outputs. this indicates that despite its ability to mimic human writing, gpt-4's generated text retains a certain uniformity that can be detected by expert readers. finally, our study successfully applied boden's approach to creativity (as a combination of novelty, surprise, and value) to create a rubric that serves to evaluate creative writing texts, either human or machine-generated. a statistical analysis of the 7,200 manual assessments collected shows that both attractiveness (value) and originality (novelty and surprise) significantly contribute to the perceived creativity of texts. this validates the use of boden's dimensions in evaluating the creative outputs of ai systems. it is reasonable to conclude that there are inherent limitations in current llms. llms rely on pattern extraction from large corpora of text, which allows them to generate text that is contextually appropriate and often mimics the stylistic nuances of human writing. however, this approach can also lead to the generation of content that tends to conform to common patterns and clichés, which may be enough when compared to average professional writers, but lacks the originality and intent found in the best human writers. probably, a key limitation of llms is their tendency to approximate meaning through probability. while human writers can produce low-probability text that carries deep meaning and creativity, llms tend to generate content that aligns with the most likely patterns observed in their training data. this probabilistic approach can result in outputs that are high in coherence and fluency but low in innovative thinking and originality. as llms are refined and improved, they are likely to become more adept at solving objective tasks. however, their creative output may remain constrained by a tendency to replicate familiar patterns, leading to a literature filled with clichés. overall, our study suggests that while llms can be valuable tools for generating text and assisting with various writing tasks, they are not yet capable of fully replicating the creative process of top human writers, who often produce work that is not only meaningful but also surpass conventional expectations. for now, we will have to wait until a duel between top human and machine writers is actually disputed. all experimental data and texts are available for reproducibility purposes at https://github.com/ grmarco/pron-vs-prompt .