I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors (2023)

Abstract: Visual metaphors are powerful rhetorical devices used to persuade or communicate creative ideas through images. Similar to linguistic metaphors, they convey meaning implicitly through symbolism and juxtaposition of the symbols. We propose a new task of generating visual metaphors from linguistic metaphors. This is a challenging task for diffusion-based text-to-image models, such as DALL \cdot E 2, since it requires the ability to model implicit meaning and compositionality. We propose to solve the task through the collaboration between Large Language Models (LLMs) and Diffusion Models: Instruct GPT-3 (davinci-002) with Chain-of-Thought prompting generates text that represents a visual elaboration of the linguistic metaphor containing the implicit meaning and relevant objects, which is then used as input to the diffusion-based text-to-image models. Using a human-AI collaboration framework, where humans interact both with the LLM and the top-performing diffusion model, we create a high-quality dataset containing 6,476 visual metaphors for 1,540 linguistic metaphors and their associated visual elaborations. Evaluation by professional illustrators shows the promise of LLM-Diffusion Model collaboration for this task.To evaluate the utility of our Human-AI collaboration framework and the quality of our dataset, we perform both an intrinsic human-based evaluation and an extrinsic evaluation using visual entailment as a downstream task.

## introduction visual metaphors are rhetorical devices that serve to communicate a message through an image. they are often used as a means of persuasion in advertising @xcite , as their use leads to more favorable attitude toward the ad @xcite . similarly to linguistic metaphors @xcite , a visual metaphor takes a concept from a source domain and applies it to a target domain. in the case of visual metaphors, these domains need to be in some way visually grounded. large diffusion-based text-to-image models, such as dall•e 2 @xcite , parti @xcite , stable diffusion @xcite , or imagen @xcite , can generate visually compelling images conditioned on input texts. however, in order to generate visual metaphors from linguistic metaphors, models are required to first identify the implicit meaning as well as the objects, properties, and relations involved, and then find a way to combine them in the generated image. for instance, given the linguistic metaphor "my bedroom is a pig sty", as shown in figure 1 , a model would ideally need to extract the implicit meaning of the bedroom being "messy", and then compose the concepts "bedroom", "messy" & "pig". however, as shown in the left two images, when presented just with the linguistic metaphor, dall•e 2 generates images of a bedroom where pink is the prevalent color (perhaps due to pig's skin color), sometimes with the presence of a pig as a toy in a corner, and with little indication of a mess in the room. the visual metaphor generation task is greatly impacted by two common challenges in text-toimage models, namely under-specification and attribute-object binding @xcite @xcite . underspecification refers to the fact that finite and reasonable-length linguistic descriptions of realworld scenes by necessity omit a great deal of visual information @xcite . attribute binding is the task of binding the attributes to the correct objects, and is a fundamental problem for a more complex and reliable compositional generalization. our proposed contributions address these challenges: for our linguistic metaphor "my bedroom is a pig sty", the visual elaboration generated by instruct gpt-3 (davinci-002) with cot prompting is "a bedroom with clothes and garbage everywhere with a pig in the center rooting around." (see we release our dataset, code, prompts, and illustrator annotations at https://github.com/ tuhinjubcse/visualmetaphors . ## related work generative art. there has recently been a huge surge of ai-generated artwork and imagery with the new diffusion-based models being substantially better than previous variational autoencoders (vae) and generative adversarial networks (gans). some of the most popular current models are dall•e 2 @xcite , mid-journey, foot_0 craiyon, foot_1 and stable diffusion @xcite . these image generation models are able to handle a wide variety of prompts, though recent work has shown that there are still aspects of accurate depiction that these models fail to capture @xcite . recently, @xcite showed that diffusion models can handle language that is content-based and aimed at a neutral description of the scene, and fail to capture the underlying abstraction of figurative language. recent work has also explored cutting-edge systems showcasing the power of large language models and text-to-image models in aiding creative processes across various applications. @xcite present popblends, a system that leverages traditional knowledge extraction methods and large language models to automatically generate conceptual blends for pop culture references, significantly increasing the number of blend suggestions while reducing mental demand for users. similarly, @xcite introduce generative disco, an ai system that generates music visualizations using large language models and text-to-image models, offering an enjoyable, expressive, and easy-to-use tool for professionals in the creative field. @xcite present reelframer, a system where gpt4 and dalle2 collaborate in order to assist journalists in transforming written news stories into engaging short video narratives, by generating scripts, character boards, and storyboards. the proposed user study shows reelframer's effectiveness in easing the process and making framing exploration rewarding for journalism students. ## human-ai collaboration for visual metaphor dataset creation we propose a three-step human-ai collaboration approach for generating visual metaphors from linguistic metaphors. this process involves 1) selecting linguistic metaphors that are visually grounded; 2) using large language models to generate visual elaborations of linguistic metaphors that capture relevant objects and implicit meaning, with expert edits when required; 3) using diffusion-based models to generate visual metaphors from visual elaborations, with filtering of low quality samples by experts. a detailed pipeline diagram for our dataset creation is shown in figure 2 . we source our linguistic metaphors from six resources, removing any duplicates: flute @xcite , advertisements @xcite , copoet @xcite , figqa @xcite , figure -of-speech, 3 crossling metaphors @xcite and metaphor paraphrase @xcite . your task will be to elaborate a metaphor with rich visual details along with the provided objects to be included and implicit meaning. make sure to include the implicit meaning and the objects to be included in the explanation 1. metaphor: my lawyer is a shark. objects to be included: lawyer, shark ## evaluation our goal is to assess the impact of the llm-diffusion model collaboration (section 4.1), and of the human-ai collaboration on building a highquality dataset. ## compositionality in visual metaphors in prior work, @xcite showed that metaphorical meaning is not only a property of individual words but arises through cross-domain composition. @xcite further argues that a metaphor is a visual material rather than conceptual. it is a mechanism of syntactic structure, forms, and material composition, which goes along with the perception of structures and compositions. many images from our haivmet data showcase the compositional nature of visual metaphors, as can be seen in ## conclusion we show that using chain-of-thought prompting for generating visual elaborations of linguistic metaphors leads to significant improvements he froze with fear when he saw it. beauty is a fading flower. love is a crocodile in the river of desire. in the quality of visual metaphors generated by diffusion-based text-to-image models. these models excel at depicting literal objects and actions, but cannot make the leap from figurative phrases to visual depiction without a detailed explanation of the implicit meaning. though there are still particular aspects of visual composition and figurative imagery that current models fail to capture, the breadth of information collected in this dataset not only allows us to understand the current limitations of image generation but also provides the data necessary to improve visual metaphor generation in the future. we plan to further examine the effect of prompt phrasing on the quality of the generated visual metaphors,and how that effect differs across different models.