Enhancing Multimodal Entity Linking with J accard Distance-based Conditional Contrastive Learning and Contextual Visual Augmentation (2025)

Abstract: Previous research on multimodal entity linking (MEL) has primarily employed contrastive learning as the primary objective. However, using the rest of the batch as negative samples without careful consideration, these studies risk leveraging easy features and potentially overlook essential details that make entities unique. In this work, we propose JD-CCL (Jaccard Distance-based Conditional Contrastive Learning), a novel approach designed to enhance the ability to match multimodal entity linking models. JD-CCL leverages meta-information to select negative samples with similar attributes, making the linking task more challenging and robust. Additionally, to address the limitations caused by the variations within the visual modality among mentions and entities, we introduce a novel method, CVaCPT (Contextual Visual-aid Controllable Patch Transform). It enhances visual representations by incorporating multi-view synthetic images and contextual textual representations to scale and shift patch representations. Experimental results on benchmark MEL datasets demonstrate the strong effectiveness of our approach.

## introduction multimodal entity linking (mel) @xcite @xcite aims to connect visual and textual references to the entities in a multimodal knowledge graph @xcite @xcite . as exemplified in figure 3 , mel connects the spider-man referecnes in the image and input sentence to the spider-man entity in the multimodal knowledge graph. this alleviates textual ambiguities by incorporating visual cues such as character portraits, which benefits various downstream applications, like information extraction @xcite , question answering @xcite , and search engines @xcite . however, current mel methods are limited by their random negative sampling and understudy the potential issue of varied visual representation. first, these methods tend to employ contrastive learning to align input sentences and entities, but limited by its random negative sampling @xcite . they generally randomly draw negative samples from the datasets. this may lead to unfair predictions due to numerous similar entities in the knowledge base, for example, different but highly similar "supreme court" entities in 1. using them as negative samples hinders the connection between input sentences and correct entities. second, existing mel approaches overlooked the dissimilar visual modality between mention and target entity. for instance, 2 shows that the input image refers to an actor "doohan", but the input sentence includes four distinct entities. as a result, this greatly disturbs linking the input image to the entities in the knowledge base. for further specificity, the "doohan" image on the left illustrates the sentence, offering significant utility to the model for user queries related to "doohan" within the knowledge base due to image-text similarity. however, it could potentially introduce interference if the user intends to query "wikipedia". hence, prelinking control of the visual modality, contingent upon the mentioned target, emerges as a crucial consideration in the multimodal entity linking (mel) task, particularly when other modalities coexist alongside textual input. to address the limitation of random negative sampling, we present a jaccard distance-based conditional contrastive learning (jd-ccl) approach for mel. it leverages meta-information to mitigate the influence of prominent attributes during contrastive pre-training. jd-ccl automatically utilizes the information provided by each entity in the knowledge base to sample hard negative entities with nearly identical attributes. this strategy makes it more challenging for the model to link the mention with the correct entity, as it cannot rely on simple attributes (such as human, building, or animal) but must distinguish positive pairs from negative ones based on more complex attributes. to identify similar entities, we define a pre-processing stage that sorts the jaccard similarity scores between the meta-attributes between entities in knowledge base. this process brings more difficult and nuanced comparisons during training, which improves the linking ability by diminishing the impact of "easy" attributes in contrastive training. moreover, to address the discrepancy challenge of visual modality in image-text pairs, we introduce the contextual visual-aid controllable patch transform (cva-cpt) module. by leveraging synthetic images from a text-to-image diffusion, the model can better control essential features of the input image based on specific mention. consequently, the model can selectively control useful patch representations from the input image rather than using all image patches with the same contribution. additionally, for sentences containing multiple entities, each entity will have a unique input image, enhancing diverse visual representation according to the entity being queried. our contributions can be summarized as follows: multimodal entity linking in the landscape of social media and news, where content frequently cosplayers dressed up like marvel comic's loki and spider-man; depicting loki's wish to rule the earth spider-man: secret identity is peter benjamin parker. initially, peter was... spider-woman: her origin story relates that she was a brainwashed spy working for... venom: is an alien symbiote with an amorphous, liquid-like form, who survives by bonding with a host... input knowledge base mention image mention text entity image entity text encompasses both text and visuals, integrating textual and visual information for entity linking is essential and pragmatic. in an early pioneering effort, @xcite utilized images to enhance entity linking in response to ambiguous and incomplete references in social media posts. advancing this, @xcite investigated the inter-modal interactions through a cross-attention mechanism between text and vision, integrating a gated hierarchical architecture. to mitigate the impact of noisy and irrelevant images, @xcite assessed the relevance of images by correlating their category with the semantic content of text mentions, filtering images using a predefined threshold. @xcite compiled a dataset featuring extensive movie reviews, associated entities, and images. more recently, research @xcite has included the use of scene graphs from images to achieve object-level encoding, thus enhancing the granularity of visual semantic cues. mimic @xcite presents a multi-grained multimodal interaction network designed for the multimodal entity linking task. gmel @xcite offers a generative multimodal entity linking framework employing llms that directly produces names of target entities. mmell @xcite introduces a module that jointly extracts features to learn representations from both visual and textual inputs for context and entity candidates. dwe @xcite describes a dual-way enhancement framework where the query benefits from refined multimodal information, and wikipedia is used to augment the semantic representation of entities. contrastive learning contrastive learning @xcite @xcite @xcite has become increasingly prominent in various fields @xcite @xcite @xcite @xcite . established training methods such as n-pair loss @xcite , triplet margin loss @xcite , and arccon @xcite are foundational in metric learning. in supervised learning scenarios, key methodologies include center loss @xcite , sphereface @xcite , cosface @xcite , and arcface @xcite , which are extensively applied in computer vision and natural language processing. recent developments in contrastive learning incorporate additional conditional variables to enhance representation quality, involving auxiliary attributes @xcite , information pertinent to the downstream task @xcite , downstream labels @xcite , or data embeddings @xcite . these advancements allow for the extension of contrastive self-supervised learning into weakly supervised @xcite , semi-supervised @xcite , or fully supervised frameworks @xcite . the work by @xcite specifically seeks to enhance fairness, rather than solely focusing on representation quality, by employing sensitive attributes from the dataset as the conditional variables in a self-supervised context. ## methodology in this section, we first introduce the overall framework and then go through the details of our approaches. we visualize the overall architecture in figure 4 . ## analysis 7.1 regularization to validate the significance of each of our objectives, we conducted additional experiments where one of our proposed methods was removed. as illustrated in