Testing Language Creativity of Large Language Models and Humans (2025)

Abstract: Since the advent of Large Language Models (LLMs), the interest and need for a better understanding of artificial creativity has increased.This paper aims to design and administer an integrated language creativity test, including multiple tasks and criteria, targeting both LLMs and humans, for a direct comparison. Language creativity refers to how one uses natural language in novel and unusual ways, by bending lexico-grammatical and semantic norms by using literary devices or by creating new words. The results show a slightly better performance of LLMs compared to humans. We analyzed the responses dataset with computational methods like sentiment analysis, clusterization, and binary classification, for a more in-depth understanding. Also, we manually inspected a part of the answers, which revealed that the LLMs mastered figurative speech, while humans responded more pragmatically.

## introduction while the last years witnessed a boom of research on the task-solving impressive performance of large language models (llms) @xcite , the study of their creativity potential is just at the beginning. computational creativity started in the late 90s @xcite , but nowadays, the possibilities seem more exciting than ever @xcite @xcite , in spite of challenges such as safety, ethics, or evaluating standards. these machines, trained on huge amounts of data containing information on human society, culture, and language, proved to be worthy opponents to humans, in textual @xcite , musical @xcite , or graphical creativity @xcite . creativity represents a human's innate ability to create, based on preexisting knowledge and experience, something innovative and viable @xcite . it started as a research direction in psychology, with guilford's plead for creativity @xcite , which caused an explosion of research in the field. @xcite stated that there are two main types of thinking in the creative process: divergent thinking, which refers to the plethora of ideas that occur when faced with a creative task, and convergent thinking, which limits these ideas to only the most suitable ones. since then, creativity has spread from psychology to numerous other domains, such as philology, history, philosophy, arts, mathematics, sciences, it, and more. indeed, creativity became a pervasive, multifaceted, and interdisciplinary topic @xcite . that fact is reflected in a multitude of creativity types: ideational creativity, linguistic creativity, figural (imagistic) creativity, personality creativity, and others. linguistic creativity, which is the central focus of this study, is less formally studied than other types of creativity. previous work on computational linguistic creativity focused only on particular aspects of computational linguistic creativity, such as metaphors, similes, idioms, hyperbolas, novel compounds, morphological productivity, or neologisms @xcite . in an effort to integrate most of these aspects into a single language creativity test that reflects the overall linguistic creativity of an individual, we designed a test that inherently incorporates divergent and convergent thinking and includes various aspects of linguistic creativity, such as figures of speech, stylistic aspects of language, or word formation, suited for both humans and llms. we administered it to both humans and machines to explore their general capacity to innovate language. we also performed an in-depth analysis of the dataset that contains answers from both humans and machines to this language creativity test, by means of computational methods such as clusterization, automatic classification, and sentiment analysis, and by selective manual inspection. ## related work humanity has recently experienced the shock of generalized mass access to artificial intelligence through direct natural language communication, with the advent of large language models such as chat gpt foot_0 . currently, llms have an impressive capacity to assist humans in a significant number of tasks such as writing, planning, informing, teaching, and so on. for obvious security and ethical reasons, mainstream research on llms focuses on how to constrain or filter their output, to keep their hallucination and toxicity to a minimum. in contrast, much less attention was paid to encouraging them to be creative and to investigate their creative abilities @xcite . studies focus mostly on llms' capacity to assist humans in creative writing, like story writing, slogan writing, prewriting, or ideation @xcite , and less on their ability to produce autonomous creative texts @xcite . @xcite conducted a comprehensive review of llms creativity testing, summarizing the research on llms' creativity which, so far, dealt only with: ideational creativity expressed verbally and figuratively (in images), personality creativity, or image generation. moreover, some research focused on just one creative task @xcite , on just one llm @xcite , or on just a specific creativity type of test @xcite ). an integrated evaluation of verbal creative thinking of humans and llms @xcite included several verbal creativity tasks and ten llms. the results showed that llms were slightly better than humans, based on automated scoring. another thorough survey focusing on general ai creativity @xcite points to recent studies on specific language creativity aspects, such as humor, like puns generation @xcite , noun compound interpretation @xcite , and figurative language, like metaphor @xcite , simile @xcite , or idiom @xcite . @xcite propose automatic systems that creatively modify linguistic expressions, with pragmatic aims, like attracting the reader's attention or helping people remember concepts. in a recent study, @xcite ) test only human language creativity (not llms' or ai's), targeting just word formation creativity. to the best of our knowledge, no integrated test was proposed in the literature for testing llms's ability to use language creatively, an area where there is a great need for theoretical frameworks, data, standards, and evaluation methods. ## methodology in this section, we describe the creativity test we have designed and the evaluation criteria and methods used. we also specify the conditions and guidelines for testing humans and machines. ## dataset we give here the general statistics of the dataset we collected, comprising the human and llm answers to the language creativity test. the data is slightly unbalanced in favor of humans in terms of the number of answers and the number of words. the dataset contains a total of 4095 responses, with 17384 words: we preprocessed the data as follows. stop words such as "the" and "a" were manually removed from some responses, both of the llms and of humans, because they were irrelevant to the creativity assessment and did not meet the word limit. we also eliminated human and machine formatting errors such as additional punctuation. ## results llms obtained overall scores between 0.45 and 0.57, while humans scored between 0.42 and 0.52, as shown in tables 1 and 2, respectively. the llms' overall mean is 0.49, while humans' overall mean is 0.45. as illustrated in figure 1 , llms outperformed humans with some decimals, for all criteria. also, llms slightly outperformed humans in most of the tasks, except for rename and euphemism/dysphemisms tasks, as one can see in figure 2 . we also performed a t-test on the overall creativity scores of llms and humans. the t-statistic value is 3.6762 and the p-value is 0.0008, much less than the usual threshold of 0.05, showing that the mean difference between humans and machines of 0.04 is statistically significant. also, we notice that the standard deviation for llms, of 0.033 is higher than the standard deviation for humans, of 0.025, showing more consistency among humans and more variability among llms. also, since the performance varies in both groups, some humans can still perform better or comparable to some llms. in general, llms scored slightly higher than humans on the test, with few exceptions: two tasks out of twelve. this contrasts with @xcite , who found that pre-trained llms perform worse than humans on idiom and simile continuation tasks, meant to test the understanding of these literary devices. this difference in results might be due to the distinct nature of the proposed tasks. in this work, we tested the capacity to generate figura-tive speech, while they tested llm's understanding of idiom and simile. another possible explanation for the contrastive results might be the increased performance of the latest models we used (such as chatgpt 4), as compared to the models used in their work @xcite ). ## computational analysis of human and machine answers 6.1 computational setting for experiments the computational analysis was performed by using python in google colab, with coding assistance from llm claude 3.5 opus, using zero-shot prompt engineering. this was a process of trial and error, until receiving optimal results. the following python libraries were used for data visualization and analysis: spacy 14 , scikit-learn 15 , matplotlib 16 , numpy 17 , pandas 18 , scipy 19 , and vader 20 , (valence aware dictionary and sentiment reasoner), a sentiment analysis tool, fine-tuned for sentiment scoring on social media. ## conclusions in this study, we proposed an extensive benchmark for assessing the language creativity abilities of both llms and humans. we gathered a dataset of language creativity answers in english from hu-mans and machines, and we automatically evaluated it, using ocsai tool. the creativity scores were very similar between humans and machines, with a slight advantage for llms. also, the performance of llms varies across different individuals a bit more than human performance, as shown by the higher standard deviation of the llms compared to humans. the computational and manual analysis of this dataset revealed that llms have remarkable creative abilities, displaying human-like creativity that covers the whole continuum from f-creativity to e-creativity. while it is conceivable that some of the llms' answers were present in their training data, the fact remains that the llms at least behave humanlike in this respect. there is no principled way of telling if what they display is "genuine" language creativity or merely a collage of human creativity. the automatic clusterization and binary classification methods showed that the answers of the llms and of the humans to the language creativity test differ significantly, but also present similarities, their nature needing further research.