Looking Beyond the Pixels: Evaluating Visual Metaphor Understanding in VLM s (2025)

Abstract: Visual metaphors are a complex vision–language phenomenon that requires both perceptual and conceptual reasoning to understand. They provide a valuable test of a model’s ability to interpret visual input and reason about it with creativity and coherence. We introduce ImageMet, a visual metaphor dataset, featuring 2177 synthetic and 350 human-annotated images. We benchmark several SOTA VLMs on two tasks: Visual Metaphor Captioning (VMC) and Visual Metaphor VQA (VM-VQA). We establish strong baselines by fine-tuning on ImageMet, which yields substantial performance gains in VMC (+4.67% SBERT-Similarity, +4.84% task-specific metric) and VM-VQA (+9.3% Accuracy on average). Additionally, we introduce a task-specific CoT prompting strategy that outperforms standard few-shot baselines (+1.99% in VMC, +5.21% in VM-VQA). We observe that despite strong performance on the VMC task, VLMs still significantly lag behind humans in understanding visual metaphors, indicating that their success often relies on learned associations rather than genuine analytical reasoning. We note that this gap is often obscured in metaphor captioning tasks where the automatic metrics correlate only moderately at best with human judgment (Pearson r < 0.6), highlighting the need for careful, holistic evaluation of the visual metaphor understanding of the models.

## introduction visual metaphors are figurative pictorial comparisons in which one entity (primary concept or target domain) is represented in terms of another (secondary concept or source domain). they are monomodal in nature, i.e. both the target and source domains rely entirely on visual cues to convey metaphorical meaning. due to their visual appeal and cognitive impact, visual metaphors are widespread in advertising @xcite , political cartoons, and contemporary art, making them a vital component of nonverbal communication and cultural expression. in this work, we focus exclusively on visual metaphors, leaving out multimodal metaphors (where textual and visual clues jointly construct the metaphor's meaning). this restriction allows us to isolate and focus on the challenges posed by visual metaphors, stemming from their monomodal nature and the difficulty of interpreting meaning solely from visual cues. for more on metaphors, see appendix a. visual metaphor understanding involves two key challenges @xcite . first, visual metaphors lack the syntactic structure of language, making it harder to identify source and target elements. viewers must rely solely on visual cues like composition, emphasis, and familiarity to interpret the mapping. second, to analyze visual metaphors conceptually, the visual relation must be translated into a verbal "a is b" format. this translation is interpretive and often ambiguous, as different verbalizations can emphasize distinct aspects of the concepts, which may alter the metaphor's overall meaning. visual metaphor understanding requires perceptual reasoning to infer target-source mappings and conceptual reasoning to translate them into a verbal "a is b" format. thus, visual metaphor understanding offers an interesting probe into the multimodal reasoning abilities of vision-language models (vlms). while prior work in visual metaphors @xcite has addressed visual metaphor detection, retrieval, and generation, metaphor understanding remains largely unexplored. we introduce two tasks to evaluate this: (i) visual metaphor captioning (vmc), where models must generate coherent captions that capture the intended meaning of a given visual metaphor, and (ii) a visual metaphor vqa task (vm-vqa) that assesses a model's ability to infer target-source mappings in a multiple-choice setting. our contributions are: we release our code and data to support further research in this area 1 . ## dataset: imagemet we introduce imagemet, a visual metaphor dataset developed with two key objectives: (1) to provide a challenging, manually-annotated test set containing clearly intended visual metaphors, thereby minimising ambiguity and subjectivity; and (2) to supply synthetic training data for fine-tuning models to improve their visual metaphor understanding. for the captions, we adopt a strict simile-based template-"<primary @xmath0. this format explicitly decomposes metaphorical comparisons into three components: the primary (target domain), the secondary (source domain), and the shared attribute. during finetuning, this structured format allows for more efficient learning than open-ended metaphorical expressions (appendix b). during evaluation, it facilitates reliable automatic assessment and emphasises the importance of verbalising the correct concepts, an essential aspect of visual metaphor understanding @xcite . this explicit verbalisation challenges models to go beyond surface-level pattern recognition and demonstrate true perceptual reasoning and conceptual grounding. the train and the validation sets were synthetically generated using the following automated pipeline: to ensure metaphorical quality, a generation is retained only if the primary and secondary concepts are semantically dissimilar (cosine @xmath1.5). to ensure consistency and reduce reliance on a single llm, we decompose simile generation into smaller controlled steps, incorporating external signals where needed. our train and validation sets consist of 2000 and 177 instances, respectively, each pairing a synthetic image with a synthetic simile caption. synthetic data validation: we sampled 10% of similes from imagemet and an equal number from vflute and asked human annotators to rate the coherence of the similes on a 5-point scale. im-agemet achieved an average score of 4.19, while vflute scored 4.25. we also evaluated 10% of image-text pairs from both datasets for alignment quality using the same 5-point scale. in image-text alignment, imagemet scored 4.17 and vflute 4.53. these results demonstrate that, despite being synthetically generated, imagemet instances are comparable in coherence and alignment to vflute's human-written content, validating both our simile and image generation pipelines. test set: it contains 350 manually curated images featuring clearly intended visual metaphors, each paired with a simile-style caption. images were scraped using targeted queries such as "metaphor" and "symbolism". two master's students, fluent in english and experienced in linguistic annotation, independently wrote concise captions in our preferred simile format. in round 1, each annotator worked independently. in round 2, they met to resolve disagreements. if consensus was reached, the agreed caption was retained; otherwise, the instance was discarded. this framework was inspired by the two-round protocol of @xcite . annotation guidelines are detailed in appendix j. ## methodology 4.1 tasks to evaluate models' understanding of visual metaphors, we define two tasks: (i) visual metaphor captioning (vmc) and (ii) visual metaphor vqa (vm-vqa). the food is as costly as jewellery. objects: person, jewellery, plate. ## qualitative error analysis while models show strong perceptual abilities, their main limitation lies in composing visual observations into coherent metaphorical interpretations. we identify three recurring error patterns: overall, these errors reinforce the observation that visual metaphor understanding requires not only perceptual grounding but also flexible reasoning over abstract conceptual mappings, where current models remain limited. ## conclusion we present imagemet, a benchmark for visual metaphor understanding with 2177 synthetic train/val and 350 human-annotated test instances. we evaluate 4 open and 2 closed models, and a human baseline on two visual metaphor understanding tasks (vmc and vm-vqa). finetuning on imagemet improves performance, narrowing the gap with closed models. our cot strategy sets the strongest baseline (gpt4o ms+cot) overall. despite gains, models still struggle with metaphor domain mapping, as evidenced in vm-vqa, highlighting a gap in abstract reasoning. imagemet serves as a valuable tool for studying both the conceptual and perceptual reasoning abilities of vlms, and it further contributes to the development of models capable of metaphor-aware generation and reasoning.