Creative Text-to-Image Generation: Suggestions for a Benchmark (2022)

Abstract: Language models for text-to-image generation can output good quality images when referential aspects of pictures are evaluated. The generation of creative images is not under scrutiny at the moment, but it poses interesting challenges: should we expect more creative images using more creative prompts? What is the relationship between prompts and images in the global process of human evaluation? In this paper, we want to highlight several criteria that should be taken into account for building a creative text-to-image generation benchmark, collecting insights from multiple disciplines (e.g., linguistics, cognitive psychology, philosophy, psychology of art).

## introduction creativity is generally defined as the ability to produce new work that departs from existing practices and is appropriate, e.g., "normally fitted or adapted to the resolution of problems or difficulties existing within defined constraints" @xcite . as a peculiar human feature, creativity has been investigated by multiple disciplines (psychology, aesthetics, linguistics) to find recurring patterns and regularities in the motivated and intentional breaking or bending of rules that every creative act implies. creativity varies in time and space as every culturally determined concept. the evaluation of creativity during the renaissance differed from today's practices because cultural expectations have changed. in eastern cultures, the focus is on the creative act per se instead of on the final result @xcite . the analysis of creativity is also dependent on the media: linguistic and visual creativity are both the result of complex psychological processes in the creator's mind but give rise to radically different perceptual experiences for the receiver. indeed, linguistic and visual outcomes require different interpretation processes. to make the understanding of creativity more complicated, the artificial generation of creative instances such as texts, music, images -under the name of computational creativity -poses further challenges to creativity's definition. sometimes the artist cooperates with the automatic system (as in generative art), while in other cases the result is independent of human agency, such as in text-toimage generation systems. today, computational models for text-to-image generation based on unsupervised deep learning methods can output realistic images, translating human written textual descriptions of variable length into images. text-to-image (t2i, henceforth) generation model aims to generate photorealistic images semantically consistent with the text descriptions. starting with the generation of images from single labels or keywords, these models can handle more complex linguistic descriptions. recent works mainly try to understand how much these images are referentially coherent and complete @xcite . the evaluation of the referential aptness is more straightforward than the evaluation of creativity, which depends more on subjective variability in judgments and needs well-posed questions to be adequately isolated from other cooccurring variables that influence the aesthetic experience. the generation of creative images is not under scrutiny at the moment, but it raises interesting questions: should we expect more creative images when using more creative prompts? what is the relationship between captions and images in the holistic process of evaluation? can we have creativity without agency and authorial responsibility? in this paper, we want to highlight several criteria that should be taken into account for building a text-to-image generation benchmark that addresses those questions, collecting and discussing insights from multiple disciplines (e.g., linguistics, cognitive psychology, philosophy, psychology of art). the remainder of this article is structured as follows: in section 2, we describe relevant works from generative art and psychology of art. section 3 focuses on the definition of creativity in language. we discuss how it is realized at different levels, together with examples that can be included in the benchmark. in section 4, visual creativity is presented under the lens of the findings from psychology of art. section 5 introduces the t2i available systems usable at the moment for generating images using textual prompts. finally, in section 6, we delineate the key features of human evaluation of creativity for automatically generated images that would help to answer important research questions before concluding in section 7. ## related work the definition of a benchmark for creative t2i generation is a practical effort deeply influenced by theoretical questions previously addressed by other disciplines. in this section, we briefly report their relevant findings. ## creativity in language linguistic creativity is a multi-dimensional construct, a distinctive trait of human beings not necessarily limited to literary texts but also retrievable in daily conversations @xcite . thanks to corpus linguistics, creativity in language uses is an investigable topic: corpora represent the average level against which to measure novelty. since we know the regularities of language, creative linguistic usages seem something that can be measured and organized along a cline. they gradually outdistance themselves from the norms. however, very creative usages could not be attested even if the corpus is reputed representative for the language investigated. moreover, the same corpus could not contain with significative frequency widely used idiomatic expressions that would be wrongly recognized as creative. for this reason, corpus linguistic methodologies should be used with care in the study of linguistic creativity. the creative exploitation of linguistic means is retrievable at various levels that differ by granularity: if creativity at the morphological level concerns single words or pairs of words, when the focus is on the metaphors, syntagmatic units, or whole sentences are the objects of the analysis. in principle, multiple instances of creative language can be found in a single example: a creative word resulting from blending can be inserted in a metaphorical pattern that the reader recognizes as a flash fiction story (e.g., a very short story, limited in length, see 3.3). in the following sub-paragraphs, short descriptions of creativity at various levels are reported, with examples and theoretical stances from works on these topics. as an illustrative purpose, we include just metaphors and analogies because they are among the most frequent figures of speeches exploited creatively to illustrate how they can be realized at different linguistic levels. for each level, we distinguish when possible between denotational and connotational creative exploitations. the first kind aims to introduce new words or collocations that identify a new referent in the world. we produce connotational creative exploitations when we want to communicate about an abstract property, a state of mind, a moral disposition, etc. while with denotational creativity the generated image should keep some of its referential properties (e.g., objects mentioned in the prompt should be identifiable in the image) this is not necessary for connotational creativity that, on the contrary, should produce more abstract images. ## text-to-image generation systems in this section, we briefly introduce t2i systems, linking models based on them if available. glide (guided language to image diffusion for generation and editing) @xcite is a system based on a diffusion model and two guidance strategies, e.g., clip and classifier-free guidance. the diffusion model is trained with 3.5 billion parameters on the same dataset as dall-e @xcite , using a text encoder to condition natural language descriptions. when compared with other systems, it is preferred by human evaluators for photorealism and caption similarity. glide is able to produce artistic renderings of novel concepts (see figure 3 ). however, when the text prompt defies world knowledge (e.g., "a mouse hunting a lion"), glide fails (see figure 4 ). the system is not able to handle complex textual prompts; for this reason, the model has editing capabilities that allow users to improve model samples until they match the complex prompt. the authors released a smaller diffusion model on a filtered dataset based on clip without editing capabilities foot_0 . in january 2021, openai introduced dall•e, a neural network (12-billion parameter version of gpt-3) that creates images from text, using as training set text-image pairs @xcite . one year later, dall•e 2 @xcite was released. it generates more realistic and accurate images with greater resolution using diffusion. unfortunately, none of those systems is open source, but a smaller model of dall•e is available online through an online interface foot_1 . imagen is a text-to-image diffusion model that does not use only image-text data for training but is based on large transformer generic language models @xcite that produce realistic images with a good image-text alignment. it consists of a text encoder that maps text to a sequence of embeddings and a set of conditional diffusion models that map the embeddings to images of increasing resolutions. a pytorch implementation is freely available foot_2 . for the evaluation, the authors introduce draw-bench 4 , a benchmark of 200 text prompts designed to probe different semantic properties of the models, such as compositionality, spatial relations, rare words, and more creative prompts that, according to the authors, "push the limits of models' ability to generate highly implausible scenes well beyond the scope of the training data." however, these prompts do not explicitly contain instances of creative language. parti (pathways autoregressive text-to-image model) @xcite is an autoregressive text-toimage generation model that outputs photorealistic image generation coherent with world knowledge. it is complementary to imagen because it explores different families of generative models (autoregressive vs. diffusion). for parti, text-to-image generation is a sequence-to-sequence modeling problem analogous to machine translation: it outputs sequences of image tokens instead of text tokens in another language. parti uses a powerful image tokenizer, vit-vqgan, to encode images as sequences of discrete tokens, and takes advantage of its ability to reconstruct such image token sequences as high-quality, visually diverse images. a pytorch implementation is freely available foot_4 . as part of this project, the authors released par-tiprompts (p2) foot_5 , a rich set of over 1600 prompts in english that constitute a holistic benchmark. p2 can be used to measure model capabilities across various categories and challenging aspects. it contains 52 examples such as a high resolution photo of a chicken working out in a gym labeled as imagination to test if t2i systems are able to reproduce a not realistic state of affairs. ## how to structure the evaluation process the images generated by t2i systems, thanks to prompts included in the benchmark -both paintings and pictures, when it is possible to specify the type of output -should be evaluated by human subjects to answer research questions about the relationship between artificially generated outputs and perceived creativity. the evaluation of artificially generated images presents both similarities and differences with respect to the evaluation of images -pictures, drawings, paintings -created by humans. in the following paragraph, valuable criteria for collecting judgments are listed and discussed, reporting, if necessary, how they would impact the selection of prompts included in the benchmark. ## conclusions and future work creativity is contextually and historically framed and depends on the medium. nowadays, an unprecedented occasion for investigating the topic is represented by t2i generation models that combine linguistic inputs with visual outputs. however, while the evaluation of the referential quality and coherence of the automatically generated images has been investigated, there are no papers extensively discussing the role and the evaluation of creativity in t2i generation systems. is the output of t2i generation systems perceived as creative by humans? is creativity a property that could be computationally mimicked and empirically increased in models that generate artificial instances? should we expect more creative images using more creative prompts? what is the relationship between prompts and images in the process of human evaluation? these are several of the questions that could be addressed with a proper benchmark. in this paper, we critically revised multidisciplinary works that could help to design a good benchmark for the evaluation of creativity in t2i generation systems and highlight several criteria that could shape the evaluation process.