GOME : Grounding-based Metaphor Binding With Conceptual Elaboration For Figurative Language Illustration (2024)

Abstract: The illustration or visualization of figurative language, such as linguistic metaphors, is an emerging challenge for existing Large Language Models (LLMs) and multimodal models. Due to their comparison of seemingly unrelated concepts in metaphors, existing LLMs have a tendency of over-literalization, which illustrates figurative language solely based on literal objects, ignoring the underlying groundings and associations across disparate metaphorical domains. Furthermore, prior approaches have ignored the binding process between visual objects and metaphorical attributes, which further intensifies the infidelity of visual metaphors. To address the issues above, we propose GOME (Grounding-based Metaphor Binding), which illustrates linguistic metaphors from the grounding perspective elaborated through LLMs. GOME consists of two steps for metaphor illustration, including grounding-based elaboration and scenario visualization. In the elaboration step, metaphorical knowledge is integrated into systematic instructions for LLMs, which employs a CoT prompting method rooted in rhetoric. This approach specifies metaphorical devices such as vehicles and groundings, to ensure accurate and faithful descriptions consumed by text-to-image models. In the visualization step, an inference-time metaphor binding method is realized based on elaboration outputs, which register attentional control during the diffusion process, and captures the underlying attributes from the abstract metaphorical domain. Comprehensive evaluations using multiple downstream tasks confirm that, GOME is superior to isolated LLMs, diffusion models, or their direct collaboration.

## introduction figurative language, such as metaphors, is a rhetorical device that describes an object or concepts in a *these authors contributed equally to this work †corresponding author figure 1 : for the illustration of 'a blanket of snow covered the streets', we are expecting some metaphorical attributes, such as pervasive or encompassing, to be adapted from 'blanket' (source domain) to 'snow' (target domain), instead of a real blanket to be presented (over-literalization). due to the non-literal juxtaposition in figurative expressions @xcite , metaphors can not be visualized directly through large diffusion-based text-to-image models, which can only work conditioned on descriptive texts with literal captions @xcite . recent works primarily deal with this issue through objectbased visual elaboration @xcite , which is a query rewrit-ing method with large language models (llms) focusing on the objects to be represented. for instance, the metaphorical statement 'a blanket of snow coverd the streets', can be elaborated into a descriptive caption, like 'an illustration of a blanket with snowflakes falling on it and the streets below', which identifies the objects of 'blanket', 'snowflake', and then consumed by diffusion-based models for illustration, as shown in figure 1 (a) . despite their inspiring exploration, we've found two main problems in the entire process, including over-literalization and metaphorical attributeobject binding. (1) over-literalization means that, when depicting a linguistic metaphor as an image with llms, objects within the metaphor are excessively detailed, especially for the objects in the source domain for evoking abstract concepts, leading to a cluttered or diverted representation from the metaphor's original intent @xcite . still take figure 1 as the example, 'blanket' in the statement is used for reflecting the pervasive or encompassing nature of 'snow', rather than a referential object to be depicted. such excessive concretization may diminish the metaphor's original grounding, becoming overly straightforward and singular @xcite . (2) attribute binding is the task of binding the attributes to the correct objects @xcite , which is particularly challenging for figures of speech because, the attributes is metaphorically entailed across different metaphorical domains (source domain and target domain), which impulses extra burden to diffusion models. to address the issues above, we propose gome (grounding-based metaphor binding), which illustrates linguistic metaphors from the grounding perspective to avoid over-literalization in llm elaborations. the core idea of gome is to unfold the non-literal expressions through a textual description from a rhetorical perspective, including tenor, vehicle, and pragmatic groundings, which are further leveraged for metaphor binding to preserve provoking attributes, instead of referential objects. gome involves three main stages, firstly, following @xcite , we compile a collection of linguistic metaphors from six sets as a rich source of figurative language, which is post-filtered by llm for visualizable metaphors. secondly, we construct groundingbased visual elaboration with a cot prompting method from a rhetoric perspective, which generates fine-grained metaphorical elements, as well as visual elaborations for subsequent depiction. finally, an inference-time binding method is conducted through cross-attention controlling, which realizes compelling and faithful metaphor illustration by integrating objects and figurative attributes. overall, our contributions are the following: (1) the problem of over-literalization is firstly noticed in llm elaborations for metaphors, which is then analyzed by a grounding-based depiction method to avoid excessive concretization. (2) a publicly available dataset foot_0 is introduced with 1351 visual elaborations of metaphors, together with the fine-grained metaphorical elements, including tenor, vehicle, and groundings for comprehensive metaphor illustration. (3) we propose a metaphorical attribute-object binding approach at an inference-time speed, which realizes attentional registration in the text-to-image process. (4) comprehensive experiments verify the high robustness and fidelity of our method, which paves the way for figurative language visualization, as well as other downstream applications. ## methodology we present gome, a collaboration of large language models and text-to-image models designed to generate visual elaborations and pictures from metaphorical text inputs. the development of gome comprises three main stages, including data collection and the other two depiction steps illustrated in figure 3 . firstly, we perform data collection by preprocessing a collection of metaphors sourced from previous researches. secondly, we utilize a large language model (llm) to generate visual elaborations for the metaphors by appropriate cot prompt design with rhetoric knowledge in the system role. finally, the paired data of metaphors and generated visual elaborations are fed into a diffusion model to realize metaphor depiction. although previous research used dall•e @xcite to generate images, we utilize stable diffusion for a transparent and reproducible approach, and more importantly, a novel method to explore metaphorical attribute-object binding through attentional control. concretely, the diffusion process is enriched with metaphorical object-attribute binding, using an inference-time optimization with a loss over cross-attention maps. the primary goal of gome is to generate detailed textual descriptions of visual scenes (visual elaborations) to convey the intended meaning of the rich figurative phrases in metaphors. ## evaluation evaluating the visualization of figurative language presents a significant challenge due to its inherently subjective nature. additionally, current evaluation methodologies vary widely, encompassing image recognition @xcite , visual entailment @xcite , as well as retrieval and localization @xcite . consequently, to thoroughly assess the robustness of gome, we advocate for an evaluation complemented by diverse automated metrics, together with human evaluations applied at multiple levels of granularity. ## conclusion in this paper, we introduced gome, the first model with linguistic binding for visualizing metaphors from the grounding perspective. our research notices the problem of over-literalization for the first time, and solves this issue through conceptual elaborations for binding implicit metaphorical attributes, rather than their presentation. overall, our contributions are the following: firstly, a groundingbased depiction method is proposed for accurately binding metaphorical attributes. secondly, a dataset with conceptual elaborations of metaphors is introduced, encompassing fine-grained metaphorical elements such as tenor, vehicle, and groundings. finally, extensive experiments validate the fidelity of our method in capturing the underlying meaning of metaphors. in future work, we plan to employ gome with knowledge from other related fields, such as cognitive science.