Can VLM s Recall Factual Associations From Visual References? (2025)

Abstract: Through a controlled study, we identify a systematic deficiency in the multimodal grounding of Vision Language Models (VLMs). While VLMs can recall factual associations when provided a textual reference to an entity, their ability to do so is significantly diminished when the reference is visual instead. Forcing VLMs to rely on image representations of an entity halves their ability to recall factual knowledge, suggesting that VLMs struggle to link their internal knowledge of an entity with its image representation. We show that such linking failures are correlated with the expression of distinct patterns in model internal states, and that probes on these internal states achieve over 92% accuracy at flagging cases where the VLM response is unreliable. These probes can be applied, without retraining, to identify when a VLM will fail to correctly answer a question that requires an understanding of multimodal input. When used to facilitate selective prediction on a visual question answering task, the probes increase coverage by 7.87% (absolute) while also reducing the risk of error by 0.9% (absolute). Addressing the systematic, detectable deficiency is an important avenue in language grounding, and we provide informed recommendations for future directions.

## introduction systems that incorporate multiple modalities, such as image and text, address the 'symbol grounding problem,' i.e., the problem of connecting symbolic concepts to sensory properties of objects in the world @xcite . it has been argued that addressing this problem is a prerequisite to truly 'understanding meaning' (bender and koller, 2020), making multimodal grounding a vital frontier in the pursuit of capable ai systems @xcite . an important instance of this problem is that of vision-language grounding @xcite textual reference to entity in what year was stonehenge legally protected? query in what year was the landmark in the image legally protected? query vlm 1882 accurate factual recall vlm 1970 visual reference to entity ## background and related work efforts to jointly model textual and visual modalities have made significant progress in recent years @xcite , growing from approaches that were limited to representing isolated word meanings @xcite to powerful systems that can handle arbitrary image and textual inputs on a wide variety of tasks @xcite . these methods take advantage of powerful pretrained language models (lms) @xcite , training visual encoders that convert images to representations that can be passed into the lms @xcite , with minimal changes to the lm itself @xcite . since pretraining lms on internet-scale corpora imbues them with factual knowledge @xcite , the resulting system can recall facts about entities present in the images they are provided @xcite . this approach to vision-language grounding does not jointly learn representations, raising concerns of a grounding gap between the visual repre-sentations and the lm component @xcite . qualitative studies on the semantic representations of such vlms reveal notable differences between multimodal and text-only representations of words @xcite . however, whether such differences affect downstream performance, such as the capability of a vlm to access the internal knowledge that it contains, remains an open question. prior work evaluates vlms on their ability to recall factual knowledge by asking knowledgeintensive visual questions @xcite @xcite @xcite . however, these approaches are oriented towards benchmarking the overall capability of vlms, and are not able to isolate failures in multimodal grounding specifically. a failure to answer a question from these benchmarks could be due to a failure to link the visual representation with the vlms internal knowledge, however, it may also be due to a failure in identifying the image, a gap in the knowledge contained in the vlm, a reasoning error, or some combination of these. we create an experiment that controls for such confounding factors, isolating the ability of a vlm to access its internal knowledge using only visual representations of an entity. while contemporary work explores this problem @xcite from the lens of mechanistic interpretability, they focus only on the popvqa dataset. in this work, we construct a testbed from multiple datasets that span a range of entities, showing that vlms struggle to recall factual associations from visual references over a wide variety of entity types. ## creating the benchmark we start by collecting images of various entities from image classification datasets, with each class label serving as a distinct entity. we source from: cifar100 @xcite : a generalpurpose dataset with entities ranging from insects such as 'beetle' to vehicles such as 'train'. food101 @xcite : a food dataset, with entities including 'baklava' and 'ceviche'. landmarks @xcite : a dataset of famous landmarks, with entities such as 'niagara falls' and 'stonehenge'. for each entity, we compile general knowledge question-answer (qa) pairs. unlike most existing benchmarks @xcite , our questions are well-formed in text alone, i.e., the answer can be determined without referring to any specific, contextualized image of the entity. we use the wikipedia api @xcite ) to automatically gather text containing common knowledge about each entity, and use llama3.1-8b @xcite as a parser that extracts question-answer (qa) pairs from the text. we additionally directly prompt llama3.1-8b to generate qa pairs for each entity. the collected pairs are filtered through multiple steps to ensure they are valid questions with a single correct answer (appendix a). finally, three authors of the paper conduct a round of human annotation and establish, with substantial inter-annotator agreement (κ > 0.65), that the remaining qa pairs are both relevant to the entity and ones where the answer to the question is correct. for each question, we create an analogous version that does not state the entity but instead refers to the image. ultimately, each datapoint in our testbed (figure 2 ) consists of an entity's image, a textual reference question that explicitly mentions the entity and a visual reference question that does not name the entity. separate from this pipeline, we also design a task based on the mnist @xcite dataset. for each digit in the dataset, we randomly generate addition and multiplication questions with a single other operand of no more than 2 digits (e.g. for the digit '5', we may generate the textual reference question '5 + 17 =' and visual reference question 'the digit in the image + 17 ='). however, when given the image and visual reference question, there are multiple reasons why a vlm may answer incorrectly. the vlm might have misidentified the entity, or may not contain the relevant information, which are scenarios where even a perfectly grounded system will fail. to control for these factors, we conduct an additional round of filtering. for each datapoint, we ask the vlm under evaluation to identify the object in the entity image. we also have it answer the question when given both the entity image and the text reference version of the question. we retain the question only if the identification is accurate and the answer is correct, ensuring that the vlm recognizes the entity and contains the relevant internal knowledge. we also ask the vlm to answer the question when provided a trivial image with no information about the entity (e.g., random pixel values) and the visual reference version of the question. these instances should be unanswerable as they do not contain the information required to identify the entity, and a correct response indicates that the question's answer can be guessed using language-based priors. we remove such instances to ensure that our testbed contains questions which require knowledge of the entity to answer correctly. in practice, we aggregate the vlms output over multiple different trivial images, for details see appendix b. the final benchmark allows us to isolate the vlms ability to perform factual recall from different modalities in a way that is not possible with existing complex vqa benchmarks. consider the visual reasoning question answering benchmark, gqa @xcite . this benchmark includes highly compositional images and may contain, for instance, an image of an apple and an orange on a table. we may create the textual reference question 'what colour is an apple?', which has the analogous visual reference question 'what color is the object in the image?'. however, since the image contains multiple objects, the correct answer to the textual reference question (red) is no longer the only correct answer to the visual question (red or orange). by sourcing our images from simple, single-image datasets, we can precisely isolate the entity and ensure that the textual and visual questions have the same answers. the final number of datapoints varies by dataset and model, with an average of 792 distinct imagequestion pairs remaining per dataset-model combination. for dataset-model specific counts, see appendix a ## evaluating vlms factual recall using different modalities we evaluate the vlms under two different settings, isolating the capability of the lm component to use the textual and visual representations to recall the relevant fact regarding an entity. in both settings, the vlm is prompted to directly state the answer. text only: we provide a trivial image and the textual reference question that states the entity. this setting measures the vlm's ability to recall factual associations from the textual representation of the entity. visual: we provide the image of the entity, and the visual reference question that does not name the entity in text. this setting forces the vlm to rely on its representation of the image to identify and access the facts it has stored about the entity. we evaluate vlms from the following families: instructblip: an approach that leaves the lm and vision encoder frozen, learning a q-former @xcite to convert image representations to visual tokens that the lm can use. llava: a method that keeps the vision encoder frozen, but learns a linear projection from the image representation to the input space of the lm and also learns updates to the lm weights. ## results a consistently grounded vlm that can access its internal knowledge using references from either modality should achieve comparable performance in both test settings. the results (figure 3 ) show that answer correctness under the text only setting, as judged by the two-way inclusion metric, is consistently high at 84.70% on average. this performance strikes a stark contrast with the average correctness of the visual setting (42.05%), which is less than half of the performance of the text only setting. this divide is also seen when using the exact match (52.87% vs 23.85%) and bleu (55.93% vs 21.92%) metrics (appendix c). when recalling factual associations regarding an entity, vlm performance declines by over half if it is forced to rely on visual representations of the entity instead of textual representations. the grounding gap, which occurs consistently across all models and datasets, is a systematic failure in the ability of vlms to recall factual associations using image representations of entities. this gap is most chronic in the mnist dataset, where the text only setting achieves near perfect performance (98.66%), but the visual setting achieves only 31.28% correctness. concerningly, the gap continues to be wide even for the most powerful vlm, gpt4v, suggesting that it is a failure that will not be resolved naturally through a scaling of model parameters and dataset sizes. within the same model family, models of higher scales occasionally have less of a gap; figure 3 : performance (two-way inclusion) on vlms when forced to rely on visual (visual) or textual (text only) representations of an entity. empty bars for instructblip are datasets where fewer than 100 datapoints passed filters. while the text only setting shows performance decreases (15.3%), the performance in the visual declines by 58.95% (nearly 4x the decline). this decline is evidence that vlms, across pretraining paradigms, architectures and scales, struggle to link their internal knowledge of an entity with visual representations of it. ## identifying linking failures our controlled experiments have shown significant declines in the ability of a vlm to access its internal knowledge of an entity using visual references, a grounding gap which makes these systems more vulnerable to false utterances. such responses can be particularly harmful in settings where users depend on the model @xcite , trust in its output @xcite or use its output to make consequential decisions @xcite . in this section, we develop methods that identify when a vlm will fail to link its visual representation of an entity with its internal knowledge, allowing model providers to flag instances where the response should not be trusted, and minimize the harms of linking failures. we focus on llava-vicuna7b and investigate the hidden representations of the last input token in the lm component. during the forward pass on datapoint i, specifically after the computation at layer l of a transformer lm layer, the last input token is represented by a ## addressing the grounding gap the controlled experiments above have demonstrated that vlms struggle to link their internal knowledge of an entity with a visual representation of it. however, one might argue that the vlm can always verbalize the image's contents first, allowing it to rely on the textual representation for factual recall. we advance several arguments to motivate the claim that the grounding gap shown in this work will limit the performance of vlms in practical settings. inference cost: the most direct consequence of relying on the vlm to verbalize the contents of an image before answering is an increase in the number of tokens generated before the final answer can be provided. with the increasing scale of lms @xcite , this threatens to become prohibitively expensive for both users @xcite and the environment @xcite . addressing this grounding gap would enable the visual representation of the entity to recall the information directly and enable more efficient and accessible systems. conflicts between text and image modalities: in practice, the information in the text used to describe an associated image may conflict with the information in the image's contents. recent work shows @xcite that answer generation is primarily driven by the text component and that in such cases where the modalities conflict, text information is given more weight. a superior system would make contextual decisions based on the strength of the evidence in each modality. however, vlms that rely disproportionately on the textual modality for factual recall will consistently favour the text modality and lack this capability. verbalization failures: vlms that rely on verbalization to recall factual associations face a failure mode-inability to access factual, internally stored information about the objects in the image, because they were not explicitly verbalized when describing the image. this concern is most pressing for images that have many objects or entities that the query could involve, leading the vlm to only state a subset of them, or hallucinate objects that are not in the image @xcite . certain vlm architectures, such as the llava approach @xcite , create visual representations that are agnostic to the text of the input prompt. these approaches may be more vulnerable to such failures, because the image encoding used to drive the verbalization cannot allow for a greater focus on the regions of the image most relevant to the text query. lack of visual knowledge in text training data: lms are trained on internet-scaled text corpora (brown, 2020) and hence store factual associations from the texts they have encountered @xcite . however, the concepts that are well learned from text data often do not fully include visual concepts such as spatial understanding @xcite , leading to lms that struggle on questions requiring spatial knowledge @xcite . if vlms remain reliant on their lm subcomponents for factual recall, they risk being limited in their ability to answer questions requiring a deep understanding of such visual concepts. a novel data augmentation approach: we suggest a novel training paradigm, a supplement to current pretraining procedures that may help address the grounding gap shown in this work. while our results (figure 3 ) show that vlms struggle to recall factual associations from visual references, most models still retain some capacity to do this. this suggests that the various architectures of these models do not necessarily prevent them from being consistently grounded across modalities. we suggest a variation of the feature-alignment pretraining procedure used to construct the llava models @xcite . the suggested procedure assumes only that the vision component and language component are linked with a trainable bridging module. the current practice is to use an image captioning dataset, where the inputs to the vlms are images and the target output is the caption. the bridging module is tuned to maximize the likelihood of the caption given the image, hence serving as an interface between the visual and textual modalities. we suggest that this paradigm be expanded to include factual recall. this would include collecting qa pairs for the entities in the captions of the images, a procedure which, as shown in this paper, can be done automatically at moderate scales. instead of only using the captioning task for feature alignment, the pretraining phase can include tuning the bridging module to maximize the likelihood of the answer given the question. this paradigm may allow the bridging module to learn how to transform the visual representation such that it can draw out relevant facts from the layers of the lm. ## conclusion in this work, we show that vision language models of various pretraining paradigms, architectures and scales suffer from a multimodal grounding gap. specifically, we use a controlled experiment to demonstrate that vlms struggle to recall factual associations regarding an entity from a visual reference, as opposed to a textual reference. this failure to link an entity's visual representation with its internal knowledge correlates with the expres-sion of distinct patterns in the activations of the hidden states, and probes trained on these hidden states show promise in identifying these cases as well as providing a signal for selective prediction. we argue that addressing this gap is an important avenue for future research and suggest novel future directions to do so.