Evaluation of Text-to-Image Generation from a Creativity Perspective (2025)

Abstract: In recent years, driven by advancements in the diffusion process, Text-to-Image (T2I) models have rapidly developed. However, evaluating T2I models remains a significant challenge. While previous research has thoroughly assessed the quality of generated images and image-text alignment, there has been little study on the creativity of these models. In this work, we defined the creativity of T2I models, inspired by previous definitions of machine creativity. We also proposed corresponding metrics and designed a method to test the reliability of the metric. Additionally, we developed a fully automated pipeline capable of transforming existing image-text datasets into benchmarks tailored for evaluating creativity, specifically through text vector retrieval and the text generation capabilities of large language models (LLMs). Finally, we conducted a series of tests and analyses on the evaluation methods for T2I model creativity and the factors influencing the creativity of the models, revealing that current T2I models demonstrate a lack of creativity. The code and benchmark will be released.

## introduction inspired by the diffusion process, researchers have designed a series of text-to-image (t2i) models, which exhibit outstanding performance and have significantly contributed to the development of image generation, such as stable diffusion @xcite @xcite , flux (labs, 2024) and dall-e3 @xcite , demonstrating powerful capabilities in generating relevant visual images from textual input. despite the rapid advancement of image generation, a significant challenge remains: automated image evaluation @xcite , where the primary focus is typically on image quality and corresponding author. in image quality evaluation, inception score @xcite has measured diversity with a pre-trained inception network, while fid @xcite has compared the distribution of generated and real images. for text-image consistency, approaches typically have involved comparing generated captions with human-annotated ones @xcite , or utilizing the clip score @xcite @xcite with clip model @xcite , which quantifies the cosine similarity between image and text embeddings. t2i models have been capable of generating high-quality, stylistically distinct images, achieving high scores on existing evaluation metrics; however, the evaluation perspectives discussed above give limited attention to the creativity of the models. evaluating creativity is crucial for measuring a model's ability to generate interesting content. this is especially important in assisting professionals in fields such as art, design, and innovation. at the same time, model creativity extends the practical value of the models, enabling it to contribute to the development of industries such as advertising, fashion, and entertainment. @xcite has demonstrated that the creativity exhibited in text artifacts can be predicted using appropriate formulations of computational creativity metrics. @xcite have defined the creativity of images as their uniqueness in advertisement image generation and have exhibited that current t2i models faced challenges when it comes to generating creative outputs and there was previously a lack of relevant evaluation metrics. however, current work on evaluating creativity has not defined the creativity of t2i models or designed corresponding metrics based on psychological or philosophical definitions of creativity. building upon the broader definitions of machine creativity @xcite in previous works, we extended this concept to t2i models, providing a specific definition that comprises three components: value, novelty, and surprise. value refers to whether the images align to human's instruction. novelty refers to the uniqueness of the image in relation to other images generated by the same model. the uniqueness of image refers to the content that is exclusively present in one image within a set generated by the same t2i model using the same prompt. this unique content may include aspects such as color, perspective, object composition and so on. surprise refers to whether the images contain unexpected or surprising content. based on the definitions we proposed, we established corresponding metrics, benchmarks, and a pipeline capable of automatically generating benchmarks based on existing image-text datasets. the pipeline has the capability to create a benchmark where one prompt corresponds to multiple images by clustering and merging similar texts from textimage pairs. through multiple experiments, we tested the proposed metrics and demonstrated their feasibility, which can accurately differentiate the creativity of various models based on the three dimensions mentioned above. additionally, we explored various factors that influence the evaluation of model creativity. on the generated benchmark, we tested the creativity of different versions of stable diffusion and observed that while value consistently increased with each version, surprisingly, both novelty and surprise did not follow the same upward trend and, in fact, showed a decline (up to -0.081, -0.019, respectively). this means that as the model upgrades, although it can produce high-quality images, the likelihood of generating imaginative, novel, and inspiring content has decreased, which is something we previously overlooked. this finding underscores the importance of evaluating model creativity. in summary, the key contributions of our study are threefold: ## conclusion in this paper, we explore the definition of creativity and its application in t2i models. for evaluation, we propose creativity metrics, consisting of value, novelty and surprise, and a fully automatic benchmark generation pipeline. experimental results across the generated benchmark validate creativity is a new, valuable perspective for t2i model evalu-ation. furthermore, we conducted detailed analysis experiments on the influences of hyper-parameters on the evaluation of creativity.