Understanding Figurative Meaning through Explainable Visual Entailment (2025)

Abstract: Large Vision-Language Models (VLMs) have demonstrated strong capabilities in tasks requiring a fine-grained understanding of literal meaning in images and text, such as visual question-answering or visual entailment. However, there has been little exploration of the capabilities of these models when presented with images and captions containing figurative meaning, such as metaphors or humor. To close this gap, we propose a new task framing the figurative meaning understanding problem as an explainable visual entailment task, where the model has to predict whether the image (premise) entails a caption (hypothesis) and justify the predicted label with a textual explanation. The figurative phenomena can be present in the image, in the caption, or both. Using a human-AI collaboration approach, we build the accompanying expert-verified dataset V-FLUTE, containing 6,027 image, caption, label, explanation instances spanning five diverse figurative phenomena: metaphors, similes, idioms, sarcasm, and humor. Through automatic evaluation, we find that VLMs struggle to generalize from literal to figurative meaning, particularly when it is present in images. Further, we identify common types of errors in VLM reasoning (hallucination and incomplete or unsound reasoning) across classes of models via human evaluation.

## introduction figurative language is integral to human communication, enabling a variety of communicative goals @xcite , including affective communication @xcite . figurative language presents a significant challenge to computational approaches as it requires understanding of implicit meaning behind an expression @xcite @xcite . recently, chakrabarty et al. 1 code and data: github.com/asaakyan/v-flute label: entailment ## related work textual entailment @xcite and visual entailment @xcite tasks have been proposed to measure language and multimodal understanding. however, models trained to simply improve label accuracy on these data can be brittle and suffer from spurious correlations @xcite @xcite . datasets such as e-snli @xcite ) and e-snli-ve @xcite augment existing entailment datasets with natural language explanations and train models to not only predict the label, but also generate a textual explanation for the reason behind the prediction. however, they only focus on literal meaning in text and images. recently, explainable entailment has been utilized to assess llms' capabilities on understanding figurative language through the flute dataset @xcite . flute frames figurative language understanding as an explainable textual entailment task. recent progress in multimodal models @xcite @xcite @xcite prompts us to asses understanding of figurative meaning present in the multimodal setting, contained in both images and text beyond intent and sentiment @xcite . to this end, we present an equivalent of the flute dataset for the visual modality: v-flute. ## v-flute task and dataset following prior work on figurative language understanding in text defined as explainable textual entailment, flute @xcite , we define understanding figurative meaning as an explainable visual entailment task: given an image (premise) p and a caption (hypothesis) h, output a textual explanation ê justifying whether the premise entails or contradicts the hypothesis and assign a label ŷ ∈ {entailment, contradiction}. we focus on the binary classification task, since for neutral labels, the explanations would be trivial (simply describing the image). to build v-flute, we start with existing multimodal figurative datasets which cover phenomena such as metaphors, similes, idioms, sarcasm or humor. we utilize human-ai collaboration frameworks with expert annotators @xcite @xcite to augment them with expert-verified textual explanations and entailing/contradicting captions. each instance then includes an image and a caption, and the figurative phenomenon can be either in the image, the caption or in both. an overview of the v-flute dataset and our contributions w.r.t to the source datasets can be found in ## experiments we empirically study how several baseline models perform on the task of explainable visual entailment. we investigate both off-the-shelf and fine-tuned model performance. we provide human baseline performance in appendix 5.4. hyperparameters are provided in appendix d. ## human evaluation and error analysis we conduct human evaluation of generated explanations to assess their quality and identify key errors in reasoning about multimodal figurative meaning. error type image caption label and explanation hallucination (describes sharp pencil as having a blunt tip) the tip is as sharp as a knife. predicted label: contradiction predicted explanation: the image depicts a pencil with a blunt tip. [...] incomplete (does not address metaphorical meaning of iceberg imagery) they looked at each other in an almost hostile way. predicted label: entailment predicted explanation: the image depicts two characters looking at each other with a tense and confrontational expression, which is consistent with the claim [...] unsound (concludes that large amount of money + upward trending chart implies rough waters) the world recession of 2001 was one of the worst in history. predicted label: entailment predicted explanation: the image depicts a man in a boat, holding a large amount of money and a chart with an arrow pointing upwards. this suggests that the man is navigating through rough waters [...] we recruit two expert annotators with background in linguistics for the task and sample 95 random instances from the test set. for each instance, we first provide the annotators with the image, caption and reference explanation and ask the annotators to choose the right label. if the annotator succeeds, they can view the rest of the task, which consists of 3 explanations from our top models by f1@0 in each category: llava-evil-vf, llava-34b-sg, gpt-4-5shot. the explanations are taken for both correct and incorrect model predictions. for each explanation, we ask whether the explanation is adequate (accurate, correct, complete and concise). if not, we ask them to identify one of the errors based on the following taxonomy: ## conclusion we introduce a novel dataset for understanding figurative meaning in multimodal input, v-flute, via an explainable visual entailment task. our dataset consists of 6,027 {image, caption, label, explanation} instances covering diverse phenomena. we find that vlms struggle to generalize from literal to figurative meaning, particularly in images. when figurative meaning is present in the image rather than text, models perform worse. vlms benefit from the visual information during training to understand visual figurative meaning. finally, humans still outperform even powerful vlms overall. we identify three common error types in vlm reasoning about multimodal figurative phenomena: hallucination and incomplete or unsound reasoning.