VP - MEL : Visual Prompts Guided Multimodal Entity Linking (2025)

Abstract: Multimodal entity linking (MEL), a task aimed at linking mentions within multimodal contexts to their corresponding entities in a knowledge base (KB), has attracted much attention due to its wide applications in recent years. However, existing MEL methods often rely on mention words as retrieval cues, which limits their ability to effectively utilize information from both images and text. This reliance causes MEL to struggle with accurately retrieving entities in certain scenarios, especially when the focus is on image objects or mention words are missing from the text. To solve these issues, we introduce a Visual Prompts guided Multimodal Entity Linking (VP-MEL) task. Given a text-image pair, VP-MEL aims to link a marked region (i.e., visual prompt) in an image to its corresponding entities in the knowledge base. To facilitate this task, we present a new dataset, VPWiki, specifically designed for VP-MEL. Furthermore, we propose a framework named IIER, which enhances visual feature extraction using visual prompts and leverages the pre-trained Detective-VLM model to capture latent information. Experimental results on the VPWiki dataset demonstrate that IIER outperforms baseline methods across multiple benchmarks for the VP-MEL task.

## introduction linking ambiguous mentions with multimodal contexts to the referent unambiguous entities in a knowledge base (kb), known as multimodal entity linking (mel) @xcite , is an essential task for various multimodal applications. most mel works @xcite @xcite @xcite @xcite mainly focus on improving the interaction of multimodal information and achieve promising * corresponding author. figure 1 : comparison between mel and vp-mel tasks. mel is typically limited to selecting mentions from text. in contrast, vp-mel addresses this limitation by using visual prompts to link specific regions in the images to the correct entities in the knowledge base. text information dependency: mel primarily relies on mention words for entity linking, as these words frequently exhibit significant overlap with entity names in real-world applications. such overlap serves as a strong cue for identifying entities within the knowledge base(kb). however, mel performs poorly when mention words are absent or unannotated. as shown in figure 1 , without annotated mention words, mel computes similarity based on the entire text, which can lead to erroneous entity linking. for instance, mel may incorrectly associate the data with the entity olympic games due to high textual similarity. since lee c w and lin dan are not explicitly mentioned, mel fails to establish correct links to these entities. this issue underscores the difficulty of mel in accurately linking data when the mention words related to entities are missing from the data. image modality impurity: compared to textual data, images often exhibit higher levels of noise. misinterpreting or misusing image information can substantially impact mel performance. most existing coarse-grained methods @xcite @xcite @xcite directly encode the entire image, making it difficult to eliminate noise interference. @xcite enhances mel performance by extracting fine-grained image information through object detection. however, this approach still relies on sufficient textual information for accurate object localization and is prone to interference from visually similar objects. therefore, a potentially effective strategy to mitigate image noise is enhancing object localization precision while reducing the dependence on textual data. these limitations hinder the ability of mel to fully exploit multimodal data effectively. despite being fundamental to multimodal data, images contribute minimally to mel. furthermore, the strong reliance on textual data significantly limits mel performance, especially when the text is scarce or incomplete. so we ask: is it possible to link specific objects in multimodal images to the kb even with insufficient textual information? investigating this possibility could unlock the full potential of multimodal data for mel. in this paper, we introduce visual prompts guided multimodal entity linking (vp-mel), a new task designed for entity linking in imagetext pairs, as shown in figure 1 . vp-mel annotates mentions directly on images using visual prompts, eliminating reliance on textual mention words. this approach broadens its applicability, enabling effective linking of multimodal data to the kb even when textual information is limited or image-based information is prioritized. to support this task, we develop the vpwiki dataset by extending existing public mel datasets, where visual prompts are assigned to each mention within the corresponding images. to tackle the challenges of vp-mel, we propose the implicit information-enhanced reasoning (iier) framework. iier leverages visual prompts as guiding texture cues to focus on specific local image regions. to reduce reliance on textual data, it employs an external implicit knowledge base to heuristically generate auxiliary information for the reasoning process. specifically, a clip visual encoder is employed to extract both global image features and local features guided by visual prompts. additionally, a vision-language model(vlm) incorporating a clip visual encoder is pre-trained to generate textual information from visual prompts, supplementing existing text data. iier integrates both supplementary visual and textual information, enhancing the linking of objects in images to the kb. main contributions are summarized as follows: ## dataset as there is no existing mel dataset that incorporates visual prompts, constructing a high-quality dataset is essential for establishing a strong benchmark for the vp-mel task. data collection. our dataset is built based on two benchmark mel datasets, i.e., wikidiverse @xcite and wikimel @xcite . appendix a.7 provides detailed information. annotation design. given an image-text pair with corresponding mention words, annotators are required to: 1) identify and annotate relevant visual prompts in the image based on the mention words; 2) discard samples where the image and mention words are unrelated; 3) refine annotations for samples with inaccurate automated labels; 4) assign an entity type to each instance (i.e., person, organization, location, country, event, works, misc.). ## methodology in this section, we describe the proposed iier framework for the vp-mel task. as illustrated in figure 4 , iier utilizes visual encoder to extract both deep semantic features and shallow texture features, which are enhanced by visual prompts ( §5.1). to avoid excessive reliance on visual features, the detective-vlm module is designed to generate supplementary textual information guided by visual prompts( §5.2), which is then combined with the original text and processed by the text encoder ( §5.3). finally, a similarity score is computed after integrating the visual and textual features( §5.4). ## conclusion in this paper, we propose vp-mel, a novel task designed to link visual regions in image-text pairs to their corresponding entities in a knowledge base, guided by visual prompts. to support this task, we develop vpwiki, a high-quality dataset constructed using an automated annotation pipeline to improve annotation efficiency. to tackle vp-mel, we propose iier, a framework that effec-tively leverages visual prompts to extract enriched local visual features and generate supplementary textual information. iier maintains a balance between visual and textual features, preventing excessive reliance on a single modality. extensive experimental results demonstrate that iier surpasses state-of-the-art methods. furthermore, vp-mel significantly alleviates the constraints of mention words and expands the applicability of mel to real-world scenarios.