VCRMNER : Visual Cue Refinement in Multimodal NER using CLIP Prompts (2025)

Abstract: With the continuous growth of multi-modal data on social media platforms, traditional Named Entity Recognition has rendered insufficient for handling contemporary data formats. Consequently, researchers proposed Multi-modal Named Entity Recognition (MNER). Existing studies focus on capturing the visual regions corresponding to entities to assist in entity recognition. However, these approaches still struggle to mitigate interference from visual regions that are irrelevant to the entities. To address this issue, we propose an innovative framework, Visual Cue Refinement in MNER(VCRMNER) using CLIP Prompts, to accurately capture visual cues (object-level visual regions) associated with entities. We leverage prompts to represent the semantic information of entity categories, which helps us assess visual cues and minimize interference from those irrelevant to the entities. Furthermore, we designed an interaction transformer that operates in two stages—first within each modality and then between modalities—to refine visual cues by learning from a frozen image encoder, thereby reducing differences between text and visual modalities. Comprehensive experiments were conducted on two public datasets, Twitter15 and Twitter17. The results and detailed analyses demonstrate that our method exhibits robust and competitive performance.

## introduction named entity recognition (ner) primarily identifies key entities (e.g., person, locations, organizations) within unstructured textual data sources @xcite . in the context of social media applications, ner technology is primarily used to analyze and track the dynamics of public opinion, major events, and other related information trends. as social networks evolve, the volume of multi-modal data on social media continues to grow, rendering traditional text-based ner methods insufficient for this new form of data. consequently, researchers have developed multi-modal named entity recognition (mner) @xcite . mner integrates image and text data to identify named entities, effectively resolving the ambiguities present in traditional ner tasks @xcite . it has now become an important research direction in the field of information extraction. in mner tasks, textual content and images often exhibit low relevance @xcite , with entities usually concentrating on specific visual regions (visual cues). other regions might interfere with the accurate identification of named entities @xcite . early studies @xcite @xcite have explored the inherent correlations between images and text using attention mechanisms. however, this approach does not address the low correlation between images and text, and it is challenging to assess the effectiveness of implicit alignments. subsequently, research @xcite focused on reducing the influence of irrelevant images on entity recognition by evaluation mechanisms to assess the correlation between entire images and their corresponding textual sentences. for instance, they utilized contrastive learning methods to assess image-text similarity, or they employed pre-trained models for this assessment. additionally, this approach diminishes the significance of object-level visual regions. recently, studies @xcite @xcite have attempted to exploit object-level visual regions using visual tools such as mask r-cnn @xcite . the object-level visual information typically corresponds directly to visual objects with less noise, these visual regions can better assist in entity recognition. however, these visual tools are typically trained solely on visual datasets, which hampers their ability to accurately capture the visual regions pertinent to the entities. consequently, irrelevant visual regions can mislead the model's judgments, resulting in error propagation. as shown in fig. 1 , the similarity score between the input image and the sentence, as evaluated by similarity assessment, is merely 0.36. despite the person in the image being "taylor swift" as mentioned in the text, the relevance of the visual region where this person is located has decreased. in the method that utilizes mask r-cnn to detect object-level visual regions within the image, three pertinent object regions were identified. however, the "car" in the image does not contribute to the identification of the entity "taylor swift". if the significance of this visual region is not diminished, it could potentially mislead the identification process of the entity. employing semantic information of entity category words to evaluate the relevance of visual regions helps minimize the interference caused by visual regions that are unrelated to the entities. clip @xcite as a pre-trained multi-modal model, bridges the modalities of images and text. when used as an evaluator, clip is capable of assessing the relevance between images and textual content. however, the training corpus for clip lacks category words for entity classifications, including person, location, organization, and miscellaneous. consequently, it is crucial to guide visual region assessments with category words. in this paper, we propose a new framework visual cue refinement for mner (vcrmner) that uses prompts instead of category words to guide the evaluation of visual regions. specifically, we characterize the types of entities using a series of prompts and compute the center of these prompts' vector representations to represent the entity category words. the similarity between these category word representations and the representations of vi-sual cues is assessed with clip to determine the relevance of these visual regions. furthermore, we developed a two-stage modal fusion interactive transformer. first, attention is calculated within each modality separately. then, the model fuses the modalities together. this balances the differences between text and visuals, avoiding excessive interference from visual representations. by refining visual cues from the frozen image encoder, our model reduces modal discrepancies effectively. comprehensive experiments were conducted on two public datasets twitter15 @xcite and twitter17 @xcite . the results and detailed analyses confirm that our method provides robust and competitive performance. our main contributions are summarized as follows: ## conclusion in this paper, we proposed a new framework to implement the mner task in two stages. it guides mask r-cnn to mine visual objects through prompts and obtains visual objects closely related to the entity category words. through the interactive transformer, we refined the visual cues and narrowed the semantic gap between modalities. we have constructed a variety of experiments to prove that our method is effective and achieves sota effects.