Vision-Language Models Struggle to Align Entities across Modalities (2025)

Abstract: Cross-modal entity linking refers to the ability to align entities and their attributes across different modalities. While cross-modal entity linking is a fundamental skill needed for real-world applications such as multimodal code generation, fake news detection, or scene understanding, it has not been thoroughly studied in the literature. In this paper, we introduce a new task and benchmark to address this gap. Our benchmark, MATE, consists of 5.5k evaluation instances featuring visual scenes aligned with their textual representations. To evaluate cross-modal entity linking performance, we design a question-answering task that involves retrieving one attribute of an object in one modality based on a unique attribute of that object in another modality. We evaluate state-of-the-art Vision-Language Models (VLMs) and humans on this task, and find that VLMs struggle significantly compared to humans, particularly as the number of objects in the scene increases. Our analysis also shows that, while chain-of-thought prompting can improve VLM performance, models remain far from achieving human-level proficiency. These findings highlight the need for further research in cross-modal entity linking and show that MATE is a strong benchmark to support that progress.

## introduction several real-world applications demand the ability to perform cross-modal entity linking, i.e., being able to align entities and attributes across modalities. in autonomous driving, for example, a single image of a scene may contain multiple entities, such as pedestrians and other vehicles. additionally, textual or structured data about these entities, provided by smart devices or other cars, can include information like speed or future trajectory. while some attributes, such as vehicle color or shape, are 1 our dataset, evaluation results, and code are publicly available at https://github.com/hitz-zentroa/mate { "name": "object_0", "shape": "cylinder", "size": 0.7, "material": "metal", "3d_coords": [0.371, 0.445, 0.699] }, { "name": "object_1", "shape": "sphere", "size": 0.7, "material": "rubber", "3d_coords": [1.875, -1.769, 0.681] }, { "name": "object_2", "shape": "cube", "size": 0.351, "material": "rubber", "3d_coords": [3.605, -0.679, 0.349] }, { "name": "object_3", "shape": "cube", "size": 0.7, "material": "metal", "3d_coords": [-0.904, -2.605, 0.699] } ## related work several multimodal tasks in the literature are related to the one we propose, often grouped under the generic term of visual grounding. for example, referring expression comprehension (rec) @xcite requires identifying the image region described by a textual mention, typically referring to objects or physical entities along with their attributes and relations to other objects. similarly, the situated and interactive multimodal conversations dataset (simmc) @xcite introduces a multimodal dialogue task where a system assists users in a virtual shopping scenario. to complete the task, the system must link visual objects to their textual metadata and search for relevant information. while both tasks share the challenge of aligning visual and textual content, our task extends this further by requiring explicit cross-referencing of object attributes across modalities. in particular, simmc avoids this challenge in its shared task by providing gold object ids, eliminating the need for linking from raw multimodal inputs. multimodal entity linking (mel) @xcite @xcite is a related task where mentions in multiple modalities are disambiguated by linking them to the corresponding named entities in a knowledge base such as wikipedia. while previous research @xcite has primarily focused on scenarios where the image provides supporting visual information for a single entity, our cross-modal entity linking setup involves multiple entities present in both the image and the knowledge base. similarly to mel, our task also requires linking textual mentions with visual regions, but we go further by considering pointer attributes of entities (e.g., color and shape) and ensuring that both visual and textual descriptions of the objects are linked to accurately solve the task. this requires the model to perform both visual and textual searches to establish a coherent link across modalities. all these tasks and many other related ones rely on the core ability of visual search, the process of efficiently recognizing and localizing key objects within given scenes, a long-studied topic in cognitive sciences @xcite @xcite . several computational models have been proposed for visual search, showing the difficulties of matching human performance @xcite @xcite . recent work @xcite proposes that those difficulties are closely related to the binding problem @xcite , i.e., the ability to associate one feature of an object (e.g., its color) with the other features of that object (e.g., its shape and location). our task is also based on visual search, but it adds the homologous textual search and poses the challenge of linking the textual description of an object (with a given set of attributes) with its visual description using a unique pointer attribute (section 3). to the best of our knowledge, no similar task has been studied previously. ## cross-modal entity linking cross-modal entity linking refers to the ability to understand that an entity described in two different modalities is actually the same entity. for example, in figure 1 , the red cylinder is represented in two ways: visually, with attributes such as color, shape, and size, and textually, as a set of attribute-value pairs. in order to evaluate whether vlms possess this ability, we propose a question-answering task. we create 3d scenes containing multiple objects and provide a textual collection of attribute-value pairs for all objects in the scene (see figure 1 ). importantly, while some attributes are shared across modalities, others are exclusively available in one modality. in figure 1 , the color attribute is not included in the textual modality, while the object's name appears only in text and other attributes (shape, size, and material) are common to both. using this setup, we design questions that ask for the value of a particular object's attribute. in figure 1 , the first question asks for the name of the red object. in this example, the color red acts as the pointer attribute, which identifies the object in the scene. to avoid ambiguity, each pointer attribute is unique to a single object. the "name" attribute, only available in the textual modality, serves as the target attribute. answering correctly requires linking the object's visual and textual representations. question 1 in figure 1 is an image-to-text task because the pointer attribute is only represented in the visual modality and the target attribute in the text. question 2, instead, is text-to-image, where the pointer attribute "name" is only represented in text, and the target attribute is the color, represented only in the image. this question-answering task is a suitable proxy for testing the ability of cross-modal entity linking, since to achieve high performance, models must link two different representations of the same entity. ## experiments we conducted our experiments using a variety of open and closed models (see appendix d for a full list). for open models, we report results for the topperforming model from each family. the families we considered are llava-1.5 @xcite , llava-1.6 @xcite , molmo @xcite , llama 3.2 (dubey et al., 2024), and qwen2.5-vl @xcite ). 2 for closed models, we included gpt-4o, claude 3.5 and gemini in our evaluation. 3 all open-weight models where evaluated in a computing cluster using four nvidia a100 80gb gpus. the entire set of experiments was estimated to have required approximately 300 gpu hours. for the closed-weight commercial models, we used their respective official apis. we tested all models using zero-, one-, and twoshot prompting. as most evaluated models restrict visual input to a single image, we used one image along with two question-answer examples for twoshot prompting. when the target attribute resided in the visual modality, we included a set of possible options to help the models align their responses with the expected terminology (see appendix h.2). we represented the scene in the textual modality using json format. in an initial comparative study, we explored how different textual representation formats (json, yaml, xml, or verbalized descriptions) affected performance. the results showed no statistically significant differences between formats, though json and verbalized representations slightly outperformed the others. thus, all experiments in this work were conducted using json as the scenes' representation format in the textual modality. ## results we report the exact match accuracy across all experiments, as they involved either extractive questionanswering (for textual target attributes) or multiplechoice question-answering (for visual target attributes). we also report results for the two-shot settings due to their better performance and stability across inferences. we consider this setting a better representation of the capabilities of these models in this task (see appendix f). nevertheless, the results reveal a clear progression in vlm capabilities. there is a noticeable trend between model performance, parameter count, and release date, with more recent and larger models consistently outperforming their counterparts (see complete results in appendix d). it is also interesting to note that the text-to-image configuration is easier for all vlms, foot_1 indicating that it is easier to identify the pointer attribute in the text and link it with the visual attribute of the image. this is not the case of humans, who have the same performance for both task configurations. in figure 2 we further analyze the performance of vlms and humans as the number of objects in the scene increases (3-10). the performance is calculated as the average between image-to-text and text-to-image accuracies. when correlating attributes and linking entities correctly, the number of objects in a scene does increase the difficulty of the task, but humans show no degradation in their performance. on the contrary, for vlms, performance degradation was significant, even for top-performing models like claude, which is almost 30 absolute points behind humans for scenes of 10 obejcts. this behavior can be explained by feature interference, which tends to increase with the number of objects. according to @xcite , humans are more robust to feature interference than current vlms, as observed again in our experiments. finally, ## analysis our human evaluations revealed that solving the cross-modal entity linking task typically involves a series of sequential steps. we use the image-to-text scenario in figure 1 (question 1) as an example: for the text-to-image setting, the same steps are followed, but in reverse order. in question 2 of figure 1 , for instance, step 2 would involve combining at least two attributes, such as shape and material, or shape and size. in our analysis, we break down the cross-modal entity linking task into these three subtasks to analyze vlm performance and identify the most challenging aspects for these models. ## conclusions in this work we demonstrate that vlms are unable to consistently match the same entity across modalities and retrieve one of its attributes, even guiding their inference with chain-of-thought prompting. to support the evaluation of this fundamental skill, we present mate, a benchmark designed to assess the proficiency of both current and future models in this area. future work could focus on evaluating tasks with greater complexity regarding the pointer and target attributes. one of the main advantages of this benchmark is that it can be easily extended to scenarios involving multiple pointer and target attributes (e.g., "identify the name and rotation of the blue sphere") and/or a higher number of objects. the last alternative may be especially interesting to evaluate test-time scaling techniques, trying to better align with humans' capability of using more time for more complex scenarios. finally, it would be interesting to test models on cross-modal linking in a real-world scenario. one could choose a set of real-world entities (people, cars, etc.) and discretise a set of attributes that uniquely identify the entities (clothing, colour, size, etc). however, the main challenge for such an undertaking is to design the accompanying knowledge base such that you maintain a well-defined scope for the controlled evaluation of cross-modal entity linking.