Comparison of Image Generation Models for Abstract and Concrete Event Descriptions (2024)

Abstract: With the advent of diffusion-based image generation models such as DALL-E, Stable Diffusion and Midjourney, high quality images can be easily generated using textual inputs. It is unclear, however, to what extent the generated images resemble human mental representations, especially regarding abstract event knowledge. We analyse the capability of four state-of-the-art models in generating images of verb-object event pairs when we systematically manipulate the degrees of abstractness of both the verbs and the object nouns. Human judgements assess the generated images and demonstrate that DALL-E is strongest for event pairs with concrete nouns (e.g., “pour water”; “believe person”), while Midjourney is preferred for event pairs with abstract nouns (e.g., “raise awareness”; “remain mystery”), irrespective of the concreteness of the verb. Across models, humans were most unsatisfied with images of events pairs that combined concrete verbs with abstract direct-object nouns (e.g., “speak truth”), and an additional ad-hoc annotation contributes this to its potential for figurative language.

## introduction nowadays tools for automatic image generation are accessible to laypeople as much as to experts. but do the generated images capture human mental representations? and which images are generated for abstract concepts and events that are not easily depictable, such as the concept patience and the event speak the truth, given that what we really see in the images depicting abstract knowledge are concrete objects? the current study assesses four image generation models on how well they depict abstract vs. concrete event descriptions: we compare dall-e 2 @xcite , stable diffusion @xcite , stable diffusion xl @xcite and midjourney foot_0 , as well as images retrieved by the search engine bing foot_1 . following frassinelli and schulte im @xcite , the prompts for the models are represented by 40 phrase-level events consisting of a verb and a direct object noun, where we systematically vary the words' degrees of abstractness by relying on the ratings in @xcite , cf. build a perspective vs. carry a box. we evaluate the generated images through human ratings (i) in a standard large-scale crowd-sourcing task, and (ii) in a two-step small-scale setup where we prime our participants on their expectations by asking them to first describe what they would expect to see in an image of a specific event, before asking them to judge the quality of the automatically generated images. our hypothesis is that humans will be less satisfied with the depiction of abstract in comparison to concrete event knowledge, while it is unclear how and to what extent the abstractness of verbs vs. nouns influences the human judgements with regard to the four-way combinations of abstract/concrete verb-noun events. we thus propose an exploration of the capabilities of image generation models regarding abstract vs. concrete event descriptions, while previous work primarily focused on concrete events such as scenes with concrete objects and relations @xcite , person appearance and shape @xcite , and transformer-based text-toimage generation across different styles @xcite , or on investigating prompts variants for optimising the generation of abstract and figurative concepts @xcite . examples of research that not only targeted concrete but also abstract knowledge in images, are studies by @xcite who performed priming experiments for abstract words in images, @xcite who proposed standard vision detection and retrieval tasks to distinguish between concrete and abstract concepts in visual metaphors, and @xcite who trained image generation models to illustrate any kind of textual input, including figurative language. ## target and data collections as the basis for our experiments we create verbnoun event pairs of varying degrees of concreteness (section 2.1). these event pairs are used as prompts for the image generation models (section 2.2). ## model evaluation we evaluate the generated images through human ratings in two studies. the images, the full annotation instructions and all collections are publicly available at https://www.ims.uni-stuttgart . de/data/image-generation. serve food reduce noise steal idea remain mystery a waiter bringing a platter filled with food to a table at a dimly lit diner, three people sitting at the table; a man stands behind a counter and dishes up a variety of foods to a customer a slider with a speaker symbol next to it and an arrow over the slider pointing away from the speaker symbol; a grainy picture followed by an arrow and a very soft looking version of the picture a person in a lab coat leafing through a notebook, the body language shows unease; person with thought bubble above their head, the thought bubble is being snatched away by another person a woman burns a letter from an ex without reading it; an archaeologist tries to decipher a text from an unknown language study 1: crowdsourcing ratings we gather ratings of the generated images for our verb-noun events from amazon mechanical turk (amt) 4 workers based in either usa or uk, and with more than 10,000 prior submissions and @xmath0. the workers are asked to rate on a scale from 1 to 6 how well each of the 800 generated images depicts the associated verb-noun pair event. we also add 80 images as sanity checks; these include an obviously wrong image for additional verb-noun pairs, e.g., an image of a car for play football. study 2: expectation-based ratings this evaluation is conducted in two consecutive tasks. in task 1 we aim at collecting precise descriptions of what our participants expect to see in an image of a particular verb-noun event, by asking them to provide one or more phrases describing the mental image they created of the given event. in this way, participants can reflect on the given event and the mental representations they are generating. the annotators are university students highly proficient in english (b2 level or higher). we collect 19 responses from our annotators describing their image expectations for the verb-noun event in task 1 (see examples in ## results study 1: crowdsourcing ratings we collected a total of 7,200 ratings for our 800 images, with nine unique annotators rating each image on a scale from 1 to 6. after removing all ratings by annotators that failed the sanity check, and using only those images that @xmath1, our final set contains 4,212 ratings. these ratings distribute over our event categories as follows. figure 3 presents the proportions of how often a model received an extremely low (bad) rating of 1 or 2 (left plot) or an extremely high (good) rating of 5 or 6 (right plot), out of the total number of ratings for that model and a specific verb-noun event category. for example, sd received a very low rating for 62 (48%) of the generated images in the abstv+abstn category and a very high rating for only 13 (10%) generated images in this category. overall, we can clearly see that sd (orange bars) received most low ratings and fewest high ratings across event categories; bing (blue bars) serves as an upper bound (i.e., receiving few low and many high ratings across most event categories); and dall-e, sdxl and mj show more variable results across event categories. more specifically, the right plot in figure 3 displays closer competitions across the image generation models: our best performing model for abstv+concn and concv+concn is dall-e, while mj is best regarding the other two categories. therefore, dall-e performs best when the direct-object noun is concrete, while mj performs best when the directobject noun is abstract, irrespective of the concreteness of the verb. mj also exhibits a rather uniform success rate across categories. sdxl (green bars) is the second best generation model in three out of four categories. study 2: expectation-based ratings figure 4 shows how many images from each model were selected by the annotators across verb-noun categories in task 2, after they had previously described their expectations (see examples in overall, our human expectations evaluation confirms the general trends from the crowdsourced evaluation regarding (dis)preferences that annotators perceived when judging the generated images. in fact, figure 4 presents a similar yet sharper picture of the human evaluation preferences in comparison to figure 3 . abstract events and figurative language our initial hypothesis was that humans would be less satisfied with the depiction of abstract in comparison to concrete event knowledge. looking into our best model results, this hypothesis has been confirmed but in an unexpected way. we found that dall-e performs best when the direct-object noun is concrete (however with a rather large standard deviation), while mj performs best when the direct-object noun is abstract, irrespective of the concreteness of the verb. in particular, annotators were much less satisfied across models with images for the concv+abstn event category. so overall it seems as if the abstractness of the noun plays a core role in how well the generated images depict verb-noun events. we suspected that this is the case because concv+abstn events predominantly express figurative language usage, as suggested by @xcite , which is inherently difficult to depict. in order to look into this follow-up hypothesis, we ran an additional annotation study by asking 12 annotators for their binary judgements on figurative vs. literal language of our 40 event pairs. foot_3 figure 5 shows that indeed concv+abstn (and to a lesser degree also the most abstract combination abstv+abstn) are strongly perceived as figurative language. ## conclusion this paper systematically assessed image generation models on their capacity to generate images for abstract vs. concrete event descriptions. we demonstrated through human evaluations that dall-e is strongest for event pairs with concrete nouns, while mj is strongest for event pairs with abstract nouns. regarding images for events with a concrete verb and an abstract direct-object noun, humans were generally not satisfied with any model, which an additional annotation attributed to a strong tendency for representing figurative language. we cannot conclusively say why some models perform better than others, but we suspect that this is due to reasons such as mj's tendency to produce more creative images in contrast to dall-e producing simplistic and to-the-point images (which humans seem to like for concrete nouns). overall, all models were outperformed by bing images, which we attribute to less artifacting, randomness and consistency issues in those images.