Differentiated Vision: Unveiling Entity-Specific Visual Modality Requirements for Multimodal Knowledge Graph (2025)

Abstract: Multimodal Knowledge Graphs (MMKGs) enhance knowledge representations by integrating structural and multimodal information of entities. Recently, MMKGs have proven effective in tasks such as information retrieval, knowledge discovery, and question answering. Current methods typically utilize pre-trained visual encoders to extract features from images associated with each entity, emphasizing complex cross-modal interactions. However, these approaches often overlook the varying relevance of visual information across entities. Specifically, not all entities benefit from visual data, and not all associated images are pertinent, with irrelevant images introducing noise and potentially degrading model performance. To address these issues, we propose the Differentiated Vision for Multimodal Knowledge Graphs (DVMKG) model. DVMKG evaluates the necessity of visual modality for each entity based on its intrinsic attributes and assesses image quality through representativeness and diversity. Leveraging these metrics, DVMKG dynamically adjusts the influence of visual data during feature integration, tailoring it to the specific needs of different entity types. Extensive experiments on multiple benchmark datasets confirm the effectiveness of DVMKG, demonstrating significant improvements over existing methods.

## introduction multimodal knowledge graphs (mmkgs) extend traditional knowledge graphs by integrating multimodal data, such as images and text, to overcome the limitations of incomplete or ambiguous entity representations that arise from relying solely on textual information @xcite . by incorporating visual data , mmkgs generate richer entity embeddings, improving performance in downstream tasks, including knowledge graph completion @xcite @xcite , entity alignment @xcite and information retrieval @xcite . existing mmkg approaches focus on the encoding and fusion of different modalities @xcite : late fusion @xcite @xcite combine modality features just before output, and early fusion @xcite integrate features earlier to foster deeper cross-modal interactions. recent studies @xcite address the challenges of integrating visual information by reducing noise through link-specific image relevance and improving image-text matching for long-tail entities. however, these methods assume that the visual modality uniformly enhances performance, neglecting to evaluate its necessity for entities, thus lacking a systematic approach to assess its entity-specific benefits. to effectively harness visual knowledge in mmkgs, two critical aspects must be considered: entity-level necessity and image-level selection. as shown in figure 1(a) , experiments reveal that randomly removing visual data-either by excluding all images for 10% of entities or 10% of images across all entities-can unexpectedly improve model performance. this indicates that not all entities benefit from visual features. for example, as illustrated in figure 1(b) , entities with distinct visual characteristics, such as fc_schalke_04 gain from visual data, which provides additional context for predictive accuracy. conversely, abstract entities like norwegian_ language often lack relevant visual cues, and including such data may fail to capture their conceptual depth, potentially degrading performance. at the image level, when multiple relevant images are available, selecting those that meaningfully enhance the entity representation is essential. as illustrated with fc_schalke_04, im- fc_schalke_04 simon_cziommer norway shackleton_ (tv_serial) jules_dassin norwegians norwegian_ language norway shackleton_ (tv_serial) jules_dassin norwegians not all entities have the same need for visual information. norwegian_ language germany veltins_arena orlando fc_schalke_04 simon_cziommer snag imf adamf randomly delete all images of 10% entities. randomly delete 10% images of all entities. δhit@1 vs. baseline (%) (a) rsme @xcite @xcite figure ages are evaluated based on representativeness and diversity. the team logo, with its high representativeness score, captures the club's identity, while diverse images like team photos and stadium views provide broader perspectives, supporting a wellrounded understanding. balancing representativeness and diversity prevents overfitting to specific visual cues and enhances the richness of entity representations in the knowledge graph. to address the above issues, we introduce the differentiated vision for multimodal knowledge graphs (dvmkg) model, which classifies entities as abstract or concrete based on inherent properties and assigns a visualizability score to determine the necessity of visual data for each entity. at the entity level, a large language model (llm) assigns a visualizability score to each entity, determining its need for visual data. for image selection, we leverage object detection and pretrained vision-language model (pvlm) to evaluate semantic alignment, ensuring visual accuracy while promoting diversity to reflect various entity facets. a multi-head attention mechanism dynamically adjusts visual embeddings according to entity-specific requirements, and intermodal contrastive learning aligns representations across modalities. finally, a weighted fusion layer integrates these embeddings to optimize the multimodal representation. our main contributions are summarized as follows: ## assessing visual modality in mmkg in mmkg, visual modality can enhance knowledge graph representation, yet its necessity varies across entities. even for entities that benefit from visual information, the most suitable images differ. this section explores how to identify entities that require visual information and select images to represent them, addressing this challenge at two levels: entity level and image level. at the entity level, we assess the relevance of visual data by scoring entity visualizability. at the image level, we focus on selecting images for entities to capture essential characteristics while maintaining diversity. ## methodology to effectively leverage the visual modality needs of different entities, we propose the dvmkg framework. the evs and rds serve as key inputs to our framework, integrating entity-specific visual demands into the multimodal knowledge graph completion (mkgc) task. as shown in figure 2 , we first introduce the pre-trained encoders for different modalities. we then detail the entity-specific visual weight adjustment (es-vwa) module, inter-modal contrastive learning, and the weighted modality fusion mechanism. ## conclusion in this paper, we propose the dvmkg model to address the critical challenge of effectively integrating visual information in mmkgs, that not all entities benefit equally from visual data and not all images are pertinent. at the entity level, dvmkg evaluates the degree of visualizability for each entity. at the image level, dvmkg assesses the quality of associated images through comprehensive representativeness and diversity scores. based on these scores, dvmkg dynamically adjusts the influence of visual information during the feature integration process, employing techniques such as multi-head attention for refined visual embeddings and contrastive learning for cross-modal alignment. our extensive experiments conducted on multiple benchmark datasets robustly demonstrate dvmkg's superiority over existing methods. the results confirm that an entity-specific approach to visual modality integration can significantly enhance the representational capabilities of multimodal knowledge graphs.