Progressively Modality Freezing for Multi-Modal Entity Alignment (2024)

Abstract: Multi-Modal Entity Alignment aims to discover identical entities across heterogeneous knowledge graphs. While recent studies have delved into fusion paradigms to represent entities holistically, the elimination of features irrelevant to alignment and modal inconsistencies is overlooked, which are caused by inherent differences in multi-modal features. To address these challenges, we propose a novel strategy of progressive modality freezing, called PMF, that focuses on alignment-relevant features and enhances multi-modal feature fusion. Notably, our approach introduces a pioneering cross-modal association loss to foster modal consistency.Empirical evaluations across nine datasets confirm PMF’s superiority, demonstrating state-of-the-art performance and the rationale for freezing modalities. Our code is available at https://github.com/ninibymilk/PMF-MMEA.

## introduction multi-modal knowledge graphs (mmkgs) integrate various modalities, including text, vision, and structural data, to provide comprehensive insights into knowledge. this integration underpins a range of applications, from question answering @xcite and information retrieval @xcite , to recommendation systems @xcite . multi-modal entity alignment (mmea) aims to identify identical entities across heterogeneous mmkgs, which is essential for the integrity of the knowledge represented within these kgs. the essence of mmea lies in identifying feature commonalities across entities from varied modalities to determine their alignment. the diversity in kg construction introduces potential mismatches in multi-modal features of entities meant to be aligned. for example, figure 1 depicts how the entity "interstellar (film)" might be represented in one knowledge graph g by a poster image of a spaceship, whereas in another knowledge graph g ′ , by a portrait of anne hathaway, the starring actress. although both images are related to "interstellar (film)", their disparate perspectives can weaken the semantic relationship, challenging the alignment task. this scenario underscores the problem of alignment-irrelevant features, distinctly different features that complicate accurate entity alignment. while recent studies @xcite have employed attention mechanisms to calculate crossmodal weights at both the kg and entity levels, they overlook the critical step of evaluating modality relevance, thereby neglecting to exclude these irrelevant features. another obstacle in mmea involves ensuring semantic consistency across different modalities. achieving consistent representations across modalities is crucial for effective entity alignment. existing methods @xcite utilize contrastive learning to minimize the intra-modal inconsistencies among aligned entities, yet overlook the cross-modal inconsistencies. furthermore, the presence of alignmentirrelevant features exacerbates these inconsisten-cies, complicating the task of achieving consistent representations across modalities. to address the aforementioned challenges, we propose a novel method for multi-modal entity alignment, named progressive modality freezing (pmf), designed to effectively filter out irrelevant features. pmf is structured around three key parts: multi-modal entity encoders, progressive multimodality feature integration, and a unified training objective for mmea. initially, pmf separately encodes entity features from each modality, allowing for flexible adjustment of input modalities. following this, the method employs a progressive approach to selectively freeze features deemed less relevant for alignment, simultaneously integrating valuable multi-modal features based on their alignment relevance. the culmination of our strategy is a unified training objective, aimed at minimizing the discrepancies both within individual kgs and across different modalities. to encapsulate, our research contributes in the following three-folds: ## model we propose a multi-modal entity alignment framework based on progressively freezing entities with irrelevant features and contrastive learning. as shown in figure 2 , the framework consists of three parts: multi-modal entity encoder, progressive multi-modality feature integration, and crossgraph contrastive learning. first, a multi-modal encoder is used to transform the raw input of each modality graph into entity embeddings. then, we integrate modality features progressively during training, which gradually freezes alignmentirrelevant features and fuse the modality graphs. finally, the model is optimized through contrastive loss between different knowledge graphs and different modality graphs. ## conclusion in this study, we presented the progressive modality freezing (pmf) model to advance multi-modal entity alignment. by measuring and evaluating the relevance of various modalities, pmf progressively freezes features deemed less critical, thereby facilitating the integration and consistency of multi-modal features. furthermore, we introduced a unified training objective tailored to foster a harmonious contrast between kgs and modalities. empirical evaluations on nine sub-datasets affirm the superiority of pmf and validate the rationale behind the selective freezing of modalities.