Evaluating the Creativity of LLM s in P ersian Literary Text Generation (2025)

Abstract: Large language models (LLMs) have demonstrated notable creative abilities in generating literary texts, including poetry and short stories. However, prior research has primarily centered on English, with limited exploration of non-English literary traditions and without standardized methods for assessing creativity. In this paper, we evaluate the capacity of LLMs to generate Persian literary text enriched with culturally relevant expressions. We build a dataset of user-generated Persian literary spanning 20 diverse topics and assess model outputs along four creativity dimensions—originality, fluency, flexibility, and elaboration—by adapting the Torrance Tests of Creative Thinking. To reduce evaluation costs, we adopt an LLM as a judge for automated scoring and validate its reliability against human judgments using intraclass correlation coefficients, observing strong agreement. In addition, we analyze the models’ ability to understand and employ four core literary devices: simile, metaphor, hyperbole, and antithesis. Our results highlight both the strengths and limitations of LLMs in Persian literary text generation, underscoring the need for further refinement.

## introduction as llms continue to evolve and gain widespread use, there has been growing interest in their potential to perform tasks that require creativity. a prominent application is creative writing, where llms are increasingly employed to generate stories, poetry, and other literary forms. yet debate persists over whether these models can genuinely emulate or replace human writers in producing creative text @xcite . a common criticism is that llms struggle with creativity, particularly in generating original, high-quality, and culturally nuanced outputs @xcite @xcite . most existing research on creative text generation has focused on english and english-speaking contexts. consequently, the creative capabilities of llms in other languages-especially low-resource ones such as persian-remain largely underexplored. despite the significance of literary text generation as a distinct form of creative expression, to our knowledge no study has systematically evaluated llm-generated literary texts in persian. moreover, prior work-whether in persian or english-rarely considers the challenge of evaluating culturally grounded literary texts produced by native speakers, beyond limited domains such as story writing and poetry. this gap underscores the need for evaluating how well models align with the cultural and literary practices of human communities. evaluating creativity in llms presents unique challenges due to their distinct reasoning processes, the subjective nature of creativity, and the limitations of manual evaluation. a widely used framework for assessing human creativity is the torrance tests of creative thinking (ttct) @xcite , which evaluate four core dimensions: originality, fluency, flexibility, and elaboration. current benchmarks such as the alternative uses task (aut) @xcite capture divergent thinking but fall short in addressing the cultural depth and stylistic richness required for literary creativity. building on the work of @xcite , who adapted the ttct for evaluating general-purpose creativity in llms, we extend this approach to persian literary text generation-a domain that poses unique linguistic and cultural challenges. unlike prior studies that focus on open-ended prompts, our framework emphasizes the generation of stylistically rich and culturally grounded persian sentences. to this end, we introduce a culturally adapted evaluation framework based on the four ttct dimensions. moreover, in the absence of suitable resources, we compile and release cpers (creativity in persian)-the first dataset designed for this purpose-which provides a foundation for systematic creativity evaluation in low-and midresource languages such as persian. evaluating six state-of-the-art llms-including gpt-3.5, gpt-4.1, deepseek-v3, deepseek-r1, qwen2.5, and gemma-we conduct one of the first systematic analyses of persian literary creativity in llm outputs. to ensure reliable scoring, we combine human annotations with an llm-as-ajudge framework using claude 3.7 sonnet, which demonstrates strong alignment with human evaluators. beyond creativity evaluation, we conduct two complementary studies: first, we examine word usage across topics to assess each model's cultural alignment with native persian speakers and values; second, we analyze the presence of four literary devices frequently used in persian literature-simile, metaphor, antithesis, and hyperbole-to explore how stylistic elements relate to creativity. our work makes the following key contributions: ## related work creative writing is a cognitively complex and performative language task that requires linguistic fluency, cultural and literary competence, narrative coherence, and the capacity for originality and imagination. recent work has increasingly explored the use of llms in creative domains, including humor generation @xcite , comedy creation @xcite , and psychological creativity assessments @xcite . studies show that llms can produce poetic and narrative content of high quality @xcite , and that human judges often struggle to distinguish between human-written and model-generated stories @xcite . creativity, however, remains difficult to evaluate due to its subjective nature. common assessment methods include the divergent association task (dat) @xcite , the remote associates test (rat) @xcite , and the widely used torrance tests of creative thinking (ttct) @xcite in psychometric studies. @xcite show that human creative outputs outperform those of gpt-3 on the alternative uses task. @xcite further demonstrate that while gpt-3 could generate original ideas, it often failed to filter out impractical ones. @xcite introduce the onlyconnect wall dataset to simulate rat-like tasks for evaluating creative problem solving in llms. their findings reveal that red herrings reduce model performance, though their analysis does not incorporate advanced prompting or retrieval-augmented methods, leaving room for further exploration. similarly, @xcite propose the cs4 benchmark to assess creativity under varying prompt specificity, promoting originality over memorization. unlike cs4's focus on general storytelling, our work specifically targets literary creativity, with emphasis on style, emotion, and cultural depth. recent narrative-level analyses @xcite show that llms systematically generate criteria questions originality -is the sentence creative and far from common clichés? -is the sentence similar to famous sentences, poems, or persian proverbs? -does the sentence contain at least one of the literary devices of simile, metaphor, antithesis, or hyperbole? fluency -is the sentence grammatically correct and understandable? -does the sentence seem fluent and natural to a persian reader? -can the sentence be used in a literary text or everyday conversation? flexibility -does the sentence use multiple ideas to express the intended topic? -does the sentence look at the topic from a new perspective? -does the sentence use different styles (e.g., ironic, humorous, philosophical)? elaboration -does the sentence go into detail and use a variety of vocabulary? -does the sentence create a clear mental image in the reader? -does the sentence convey a specific feeling (e.g., love, sadness, hope) well? other studies indicate that, although llms perform strongly in linguistic fluency, they still lag behind humans in novelty, diversity, and surprise @xcite . collaborative generation with multiple models can enhance diversity and creativity, but often at the cost of coherence @xcite . @xcite present a scalable benchmark for evaluating llm creativity using a modified version of the ttct and automated gpt-4 scoring on general creative tasks. while their work demonstrates the feasibility of large-scale creativity testing in english, it does not account for cultural or linguistic differences that are central to literary creativity. in contrast, our study is the first to evaluate llmgenerated literary creativity in persian, a culturally rich yet underrepresented language. we adapt the ttct to reflect stylistic, emotional, and metaphorical aspects characteristic of persian literature. this not only fills an important gap in cross-lingual creativity evaluation but also offers a framework for studying literary creativity in diverse cultural traditions, with potential extensions to other languages. ## methodology we propose a framework for evaluating the creativity of llms in persian literary text generation. our evaluation builds on the torrance tests of creative thinking (ttct) by focusing on four dimensions: originality, fluency, flexibility, and elaboration. we construct a dataset of persian literary texts and employ both human annotators and llm-based reviewers. ## experiments here, we evaluate the creativity of persian literary texts generated by various llms. we compare model performance, analyze the use of key rhetori- ## conclusion in this work, we introduce a novel framework for evaluating creativity in llms within the context of persian literary text generation. building on the ttct, we propose a culturally adapted evaluation scheme that captures four core dimensions-originality, fluency, flexibility, and elaboration-which can be applied to any language. our analysis shows that no single model performs well across all dimensions of creativity. instead, different models exhibit varying strengths in different aspects: for example, deepseek-r1 and gpt-4.1 demonstrate high expressive richness, whereas others, such as qwen2.5, generate more concise yet culturally resonant outputs. this highlights that creative ability in llms is distributed unevenly across dimensions rather than concentrated in a single model. a key outcome of our analysis is that models tend to follow learned patterns rather than demonstrating genuinely diverse creativity. their use of rhetorical devices skews heavily toward simile and metaphor, with limited balance across other devices such as antithesis and hyperbole. human texts, in contrast, often integrate multiple figures of speech within a single sentence, resulting in richer and more creative expression. moreover, when we attempt to use llms as judges for labeling rhetorical devices, they struggle-particularly in distinguishing between metaphor and simile-highlighting current limitations in nuanced literary understanding. we also curate a unique dataset, cpers, comprising 4,371 single-sentence literary texts spanning diverse topics and emotions. this dataset is the first of its kind in persian, and, to the best of our knowledge, no comparable dataset exists in english. we also release our human-annotated subset, including 100 human-written and 600 model-generated texts, with annotations covering both creativity scores and the figures of speech employed in each text. taken together, our findings indicate that llms serve as useful tools for persian literary text generation, but expectations remain modest: their creativ-ity does not yet parallel human-level diversity and literary nuance. these results emphasize the need for culturally grounded evaluation in multilingual nlp, particularly for low-resource, high-context languages. future work explores increasing the number of annotators, improving the judge model, examining more diverse topics, testing diverse prompting strategies, and conducting cross-lingual comparisons to assess the adaptability of llm creativity across cultures.