Automated Creativity Evaluation for Large Language Models: A Reference-Based Approach (2025)

Abstract: Creative writing is a key capability of Large Language Models (LLMs), with potential applications in literature, storytelling, and various creative domains. However, evaluating the creativity of machine-generated texts remains a significant challenge, as existing methods either rely on costly manual annotations or fail to align closely with human assessments. In this paper, we propose an effective automated evaluation method based on the Torrance Test of Creative Writing (TTCW), which evaluates creativity as product. Our method employs a reference-based Likert-style approach, scoring generated creative texts relative to high-quality reference texts across various tests. Experimental results demonstrate that our method significantly improves the alignment between LLM evaluations and human assessments, achieving a pairwise accuracy of 0.75 (+15%).

## introduction creative writing is a key capability of large language models (llms), with applications in literature, storytelling, and other creative domains @xcite . however, studies have revealed a significant gap between the creative writing capabilities of llms and those of human experts @xcite . bridging this gap requires further exploration and innovation, which in turn necessitates an effective and practical approach to evaluating the creativity of language models. although some studies @xcite @xcite have adapted creativity evaluation methods from traditional educational and psychological research-such as the alternate uses task (aut) @xcite and the torrance test of creative thinking (ttct) @xcite -to assess llms, these approaches rely heavily on manual * corresponding author annotations. furthermore, these methods typically evaluate creativity as a process by analyzing responses to open-ended questions designed to elicit creative thinking @xcite , which are inherently difficult to assess automatically. additionally, the limited number of predefined test questions introduces randomness and increases the likelihood of accidental outcomes @xcite , potentially resulting in unreliable evaluations of llm performance. to address these challenges, evaluating creativity as a product rather than a process offers a promising alternative. for instance, @xcite introduced the torrance test of creative writing (ttcw), which assesses creativity based on candidates' textual outputs. this approach enhances scalability by allowing the number of test cases to increase continuously while adding the generated texts, thereby reducing randomness through averaging over larger samples. moreover, automated evaluation of generated texts is more practical compared to subjective judgments of open-ended tasks. however, when applied with llms as evaluators, ttcw has not achieved satisfactory results, as reported by @xcite . in this paper, we aim to develop an effective automated evaluation method for assessing the creativity of llms using ttcw. we draw inspiration from reference-based evaluation methods commonly used in human assessments and automatic evaluations in other fields @xcite , and propose an approach which assigns a relative score to the generated texts compared to high-quality reference texts. additionally, we adopt likert-style scoring, a widely used method in psychological assessments, to rate subjective qualities like creativity @xcite . our main contributions are: • proposing a novel reference-based, likert-scale comparative evaluation method tailored for creative text assessment; ## experiment 4.1 datasets and setup we evaluate our proposed framework on two datasets to assess both its effectiveness and generalizability. ## analysis and ablation studies 5.1 cutoff score to determine the optimal cutoff score, we conducted a hyperparameter search on the ttcw dataset, as detailed in appendix a.8. the results indicate that setting the cutoff at -2 yields the best ranking similarity, which aligns with our expectations. a cutoff of -2 corresponds to cases where the average performance across two trials is slightly worse than the reference. given that reference texts generally exceed the minimum passing standard by a significant margin, we consider candidates who perform only slightly worse than the reference to have still met the test's criteria. ## conclusion we proposed an automated evaluation framework for assessing the creativity of large language models (llms) using the torrance test of creative writing (ttcw). by adopting a reference-based likert-style evaluation and an analyze-rate prompting strategy, our method improves alignment with human assessments, achieving a pairwise accuracy of 0.75 (+15%). ablation studies highlight the complementary roles of the reference-based approach and analyze-rate prompting, while additional grw dataset demonstrate its robustness and generalizability. these results establish a new benchmark result for automated creativity evaluation, offering a scalable alternative to manual annotation.