Probing Vision-Language Understanding through the Visual Entailment Task: promises and pitfalls (2025)

Abstract: This study investigates the extent to which the Visual Entailment (VE) task serves as a reliable probe of vision-language understanding in multimodal language models, using the LLaMA 3.2 11B Vision model as a test case. Beyond reporting performance metrics, we aim to interpret what these results reveal about the underlying possibilities and limitations of the VE task. We conduct a series of experiments across zero-shot, few-shot, and fine-tuning settings, exploring how factors such as prompt design, the number and order of in-context examples and access to visual information might affect VE performance. To further probe the reasoning processes of the model, we used explanation-based evaluations. Results indicate that three-shot inference outperforms the zero-shot baselines. However, additional examples introduce more noise than they provide benefits. Additionally, the order of the labels in the prompt is a critical factor that influences the predictions. In the absence of visual information, the model has a strong tendency to hallucinate and imagine content, raising questions about the model’s over-reliance on linguistic priors. Fine-tuning yields strong results, achieving an accuracy of 83.3% on the e-SNLI-VE dataset and outperforming the state-of-the-art OFA-X model. Additionally, the explanation evaluation demonstrates that the fine-tuned model provides semantically meaningful explanations similar to those of humans, with a BERTScore F1-score of 89.2%. We do, however, find comparable BERTScore results in experiments with limited vision, questioning the visual grounding of this task. Overall, our results highlight both the utility and limitations of VE as a diagnostic task for vision-language understanding and point to directions for refining multimodal evaluation methods.

## introduction in recent years, breakthroughs in artificial intelligence have driven substantial improvements in both natural language processing and computer vision. while these domains were traditionally separate, the emergence of multimodal learning has unified them, allowing systems to interpret, reason, and produce meaning from combined textual and visual input. in this paper, we investigate whether a vision-language model can meaningfully combine information from visual and textual modalities in a visual entailment task. visual entailment is a multimodal task @xcite that extends the traditional textual entailment (te) task @xcite . in the te task, given a text premise p and a text hypothesis h, the goal is to determine whether a premise implies some hypothesis. as such, the model tested outputs a label among three possible classes: entailment, contradiction, and neutral, based on the relation derived from the text pair (p, h) @xcite . explanation: "the image shows a woman holding a stick, which is consistent with the description of a person carrying a stick. this suggests that the image supports or implies the truth of the hypothesis." when there is sufficient evidence in p to conclude that h is true, then entailment holds. wherever h contradicts p, a contradiction is identified. if not, the relation is neutral, suggesting that there is not enough data in p to infer anything from h. the difference between the te and ve task is the replacement of the text premise with an image. the ve task is therefore multimodal as a model must predict by combining a visual premise with a textual hypothesis (figure 1 ) this paper aims to understand the capabilities and limitations of multimodal language models, using llama 3.2 vision as a test case, when performing the ve task and to investigate the factors that affect its performance. through a series of experiments employing zeroshot, few-shot, and fine-tuning settings, we explore the promises and pitfalls of using the ve task to probe vision-language understanding. specifically, we ask: comparing the performance of llama 3.2 vision across these different settings, we critically reflect on how well ve truly probes vision-language understanding. human intelligence is inherently multimodal, and learning often involves processing and integrating data from multiple senses. the central promise of multimodal language models is that they bring ai systems closer to human-like perception and understanding by combining the strengths of different modalities and providing a degree of language grounding @xcite . however, there are still a number of difficulties and challenges in this domain. a general challenge in multimodal learning involves the difficulty that accompanies learning how to represent and summarize multimodal data in a way that takes advantage of the complementarity and redundancy of multiple modalities @xcite . furthermore, transference @xcite , also called colearning @xcite , referring to the ability to transfer knowledge between different modalities to aid the target modality, is still a core challenge. in models that combine vision and language, more specific limitations have been identified. for example, early work in visual question answering (vqa) identified a heavy reliance on language priors where models often ignored visual input @xcite . similarly, it was demonstrated that vqa models may perform well by exploiting dataset shortcuts rather than truly grounding answers in the visual content @xcite . when evaluated on basic spatial relations (e.g., distinguishing "left of" vs "right of"), pre-trained models perform barely above chance, demonstrating their inability to represent spatial language robustly @xcite . in addition, vision-language models often struggle to correctly interpret interactions among objects and their attributes, and fail to visually distinguish pairs like "a red ball on a blue cube" and "a blue ball on a red cube" @xcite , showing their poor visio-compositional reasoning abilities. recent investigations, moreover, find conflicting evidence regarding the presence of human-like cross-modal associations in vision-language models @xcite @xcite . even when models appear to perform well on complex multimodal tasks, this does not necessarily mean that they are reasoning in a human-like way. fair evaluation should therefore not dismiss mechanistic strategies of ai models like llms or vision-and-language models that differ from those present in humans @xcite as they may rely on shortcut learning-the exploitation of spurious correlations or cues that happen to be present in a given dataset @xcite . in computer vision, models may latch onto dataset artifacts in images during training, making it seem like a classifier was successfully trained, for example, to distinguish between horse and non-horse images. however, further analysis may reveal the model does not focus on horses in the images, but uses cues like copyright watermarks that only appeared in horse-labeled images @xcite . in natural language inference, models may perform well on benchmarks by exploiting syntactic heuristics, rather than actually understanding sentence meaning or logic @xcite . prior work has demonstrated that the textual entailment (te) task, which acts as the precursor to ve, can largely be solved through the use of simple rules, such as assuming entailment if all the words in the hypothesis appear in the premise, rather than performing true semantic inference. models that performed well on the standard te dataset failed dramatically on a carefully constructed novel dataset with examples that can not be solved through sole reliance on these heuristics @xcite . these limitations highlight a critical disconnect between task success and genuine understanding, and demonstrate the need for evaluations that go beyond standard performance metrics. here, we comprehensively compare llama 3.2 vision performance across multiple experiments to assess whether the ve task, as evaluated on the e-snli-ve @xcite dataset, effectively measures multimodal understanding. ## related work the ve task was introduced by xie et al. @xcite , who proposed a model called explainable visual entailment model (eve). this model uses attention mechanisms to learn the inner relationships in both image and text feature spaces, and achieves better performance compared to other vqa-based models. a major advancement in this field came with the ofa model (one for all) @xcite . ofa is a sequence-to-sequence learning framework and unifies various unimodal and cross-modal tasks, including the ve task. ofa achieves the state-of-the-art performance for the ve task on the snli-ve dataset (described in more detail in section 4.2) with an accuracy of 91.2% on the test set. extending this, ofa-x @xcite is a proposed multitask framework that predicts not only the labels but also explanations. ofa-x is a fine-tuned version of the ofa model and achieved the state-of-the-art performance for the ve task on the larger e-snli-ve dataset (also described in more detail in section 4.2) with an accuracy of 80.9% on the test set. perhaps the boldest perspective comes from an approach in which the proposed model close (cross modal transfer on semantic embeddings) can achieve a comparable performance, without images, using only textual input @xcite . for the ve task, close uses the snli dataset for training (it uses a text premise instead of an image), while for evaluation, the snli-ve dataset was used, which combines vision and language. despite not using images, close achieves similar performance to the image model. this suggests that the snli dataset may contain sufficient evidence to conclude the relationship without relying heavily on visual information. this raises questions about whether a visual grounding is required and hints at the previously mentioned concept of shortcut learning @xcite . the knowledge from these previous works directly influenced the design of our experiments. inspired by ofa, we adopted a promptbased few-shot setup to investigate how effective a model performs without direct supervision. in addition, the idea of explanation generation in ofa-x led us to design an experiment to analyze the explanations from the model, helping assess its interpretability and reasoning. finally, the innovative approach of the close model and its findings led us to test different experiments with limited vision to explore the extent to which our model depends on visual input. comparing the performance of one state-of-the-art multimodal language model, llama 3.2 vision, across all these different settings allows us to analyze the suitability of the ve task to probe vision-language understanding. ## methodology in this study, we evaluate the llama 3.2 vision 11b model on the e-snli-ve dataset using three approaches: zero-shot inference, fewshot inference, and fine-tuning. ## discussion this study investigates the capabilities of the llama 3.2 vision model on the ve task using the e-snli-ve dataset. the experiments yielded various findings, revealing to what extent ve is a suitable task to probe vision-language understanding. first, the baseline results demonstrated modest performance, indicating the limited capabilities of the model in zero-shot inference. this is somewhat surprising given the enormous number of images and textual captions the model has seen in training and the impressive performance reported for other vision-language tasks such as visual question answering. three-shot inference improves the performance of the model; however, we also observe that additional in-context examples are not always beneficial. the most significant finding is the major improvement after fine-tuning, where the model achieved an accuracy of 83.3%, outperforming the sota performance achieved by the ofa-x model. moreover, the fine-tuned model has strong interpretability since it achieved an f1-score of 89.16% using bertscore, an evaluation metric that utilizes contextual embeddings for the explanation evaluation. this indicates high semantic similarity between the human produced reference and model generated text. while these results are promising, the overall findings also reveal some pitfalls in using the ve task and e-snli-ve dataset to effectively measures multimodal understanding. in the zero-shot inference experiments with limited or absent vision, we saw that the model was highly prone to hallucination and imagined visual evidence in order to support the hypotheses. the experiments with prompt variations and three-shot inference reveal that factors such as the order of the class labels in the prompt and the order of the in-context examples significantly affect the model's predictions, revealing highly inconsistent reasoning, which does not align with the assumption that the model shows vision-language understanding in a human-like way. also, the bertscore results for the zero-shot inference experiments with black images were on par with those of the fine-tuned model, showing that model generated explanations with semantic similarity to human explanations do not necessarily reveal the model is effectively using the visual input to solve the task. finally, the observation of individual errors in the zero-shot experiment exposed problems with the e-snli-ve dataset, which still contains examples with wrong labels or examples that can be interpreted in multiple ways, technically making more than one class label correct. before including ve in broader benchmarks used for training and testing in the area of general multi-modal reasoning (as already the case in @xcite for example) we recommend further investigation into these issues. these findings additionally offer several lessons for the broader field of multimodal learning and understanding. in particular, the study underscores that, while general pre-training is powerful, even advanced multimodal language models such as llama 3.2 vision may not be suitable for complex reasoning tasks like ve without special adaptation. the few-shot results underline that a deeper understanding of how models utilize context is needed, for example, by interpreting their attention patterns using grad-cam @xcite . additionally, the study highlights that the effectiveness of in-context learning depends on the number and ordering of examples. this bears much resemblance to known consistency effects in llms, which heavily depend on prompt ordering @xcite . the dramatic increase in performance after fine-tuning exposes that the model's visual and linguistic embeddings are highly adaptable and are, in principle, rich enough for visual entailment. the findings provide helpful insights into the ve capabilities of llama 3.2 vision, but there are some limitations that should be noted. first, our methodology relies on generated answers for both the class label and the explanation. the former of which is somewhat debatable since generated multiple-choice answers are often inconsistent with actual model beliefs @xcite . while this may affect the observed results, and could be alleviated by prefilling class options and selecting the most likely class @xcite , the modus operandi of commercially available and deployed models is to use generation, i.e., without prefilling. as such, our results should be seen through this lens, and we see extended analyses using log probabilities as future work. second, every experiment was evaluated once because of time and computational constraints. the metrics are not averaged over multiple runs. this affects the few-shot experiments where a different random selection of in-context example could yield a different performance. another limitation lies in the restricted experiments for the few-shot inference. a small number of configurations were tested, particularly for the six-shot inference, which included just one permutation. several possible combinations are left out. however, given the issues found with biases, sensitivity to order effects, and hallucinations, strong improvements for the right reasons are unlikely. in addition, the fine-tuning was conducted using only the first prompt. however, we expect that predictions will not be greatly affected by the order of the classes in the prompt, given the significant performance gain observed by the fine-tuning. a worthwhile direction for future work would be to further investigate few-shot inference. for example, exploring different sets of examples for each strategy in three shots, examining different orderings of classes for six shots, and testing a larger number of examples within a context, such as fifteen shots, could still be valuable, not primarily to focus on performance, but to gain deeper insights, such as understanding the threshold beyond which providing more examples becomes disadvantageous. given its ability to improve many reasoning tasks, another promising direction is to integrate chain-of-thought prompting @xcite into few-shot and zero-shot inference. this perhaps extends the models' already observed tendency to produce coherent explanations and better use these in predictions. a broader direction for future work includes systematic prompt engineering. this involves improving the wording and structure of the prompts. since this study demonstrates that the design of the prompts significantly affects the predictions, optimizing the prompts could perhaps lead to better generalization and fewer hallucinations. finally, our results need to be corroborated by investigating other, perhaps larger, models. doing so enables careful comparison between, for example, architectural, data, and optimization design decisions, informing which ingredients improve visual entailment. in a similar vein, earlier work investigating whether model representations align with human representations suggests that dataset diversity and scale are the primary drivers of alignment @xcite . ## conclusion in conclusion, we used the llama 3.2 vision model to explore the possibilities and limitations of using the visual entailment task to probe vision-language understanding. a comparison of results in zero-shot, few-shot, and fine-tuning settings as well as experiments involving limited vision and prompt sensitivity analyses together revealed several problems. these included inconsistent reasoning, a limited reliance on visual information and a strong tendency to hallucinate. these findings underscore the importance of critical investigations into benchmark and dataset quality to make sure the predictions of the model actually reflect vision-language reasoning instead of an exploitation of spurious correlations. future work is necessary to further explore what causes the substantial difference in performance between zero-shot and fine-tuned settings and what kind of heuristics the model may be learning from the dataset during finetuning. this would help to further develop the ve task into a suitable method for probing vision-language understanding in multi-modal language models.