Multi-modal Contrastive Representation Learning for Entity Alignment (2022)

Abstract: Multi-modal entity alignment aims to identify equivalent entities between two different multi-modal knowledge graphs, which consist of structural triples and images associated with entities. Most previous works focus on how to utilize and encode information from different modalities, while it is not trivial to leverage multi-modal knowledge in entity alignment because of the modality heterogeneity. In this paper, we propose MCLEA, a Multi-modal Contrastive Learning based Entity Alignment model, to obtain effective joint representations for multi-modal entity alignment. Different from previous works, MCLEA considers task-oriented modality and models the inter-modal relationships for each entity representation. In particular, MCLEA firstly learns multiple individual representations from multiple modalities, and then performs contrastive learning to jointly model intra-modal and inter-modal interactions. Extensive experimental results show that MCLEA outperforms state-of-the-art baselines on public datasets under both supervised and unsupervised settings.

## introduction knowledge graphs (kgs) such as dbpedia @xcite and yago @xcite employ the graph structure to represent real-world knowledge, where the concepts are represented as nodes and the relationships among concepts are represented as edges. kgs have been widely applied to knowledge-driven applications to boost their performance, like recommendation system @xcite , information extraction @xcite and question answering @xcite . in recent years, an increasing amount of knowledge has been represented in multi-modal formats, such as mmkg @xcite and richpedia @xcite . these multi-modal kgs usually contain images as the visual modality, like profile photos, thumbnails, or posters. the augmented visual modality has shown the significant capability to improve kg-based applications @xcite . it was also proven that the incorporation of visual modality can enhance the contextual semantics of entities and also achieve improved kg embeddings @xcite . due to the large scope of real-world knowledge, most kgs are often incomplete, and multiple different kgs are usually complementary to one another. as a result, integrating multiple kgs into a unified one can enlarge the knowledge coverage and can also assist in refining kg by discovering the potential flaws @xcite . to integrate heterogeneous multi-modal knowledge graphs, the task of multi-modal entity alignment (mmea) is therefore proposed, which aims to discover equivalent entities referring to the same real-world object. several previous mmea works have shown that the inclusion of visual modality in modeling helps to improve the performance of entity alignment. for instance, mmea @xcite and eva @xcite proposed distinct multi-modal fusion modules to integrate entity representations from multiple modalities into joint embeddings and they achieved state-of-the-art performance. 2 however, these methods mainly utilize existing representations from different modalities, and customized representation learning for ea is not fully explored. in addition, existing methods only explore the use of diverse multi-modal representations to enhance the contextual embedding of entities, the inter-modal interactions are often neglected in modeling. to address aforementioned problems, we propose mclea, a multi-modal contrastive learning based entity alignment model, which effectively integrates multi-modal information into joint representations for ea. mclea firstly utilizes multiple individual encoders to obtain modality-specific representations for each entity. the individually encoded information includes neighborhood structures, relations, attributes, surface forms (i.e., entity names), and images. furthermore, we introduce contrastive learning into ea with intra-modal contrastive loss and inter-modal alignment loss. specifically, intra-modal contrastive loss aims at distinguishing the embeddings of equivalent entities from the ones of other entities for each modality, thus generating more appropriate representations for ea. inter-modal alignment loss, on the other hand, aims at modelling inter-modal interactions and reducing the gaps between modalities for each entity. with these two losses, mclea can learn discriminative cross-modal latent embeddings and ensure potentially equivalent entities close in the joint embedding space, regardless of the modality. mclea is also generic as it can support a wide variety of modalities. moreover, it combines multiple losses and simultaneously learns multiple objectives using task-dependent uncertainty. the contributions of this paper are three-fold: (i) we propose a novel method, called mclea, to embed information from different modalities into a unified vector space and then obtain discriminative entity representations based on contrastive learning for entity alignment. (ii) we propose two novel losses to explore intra-modal relationships and inter-modal interactions, ensuring that to-bealigned entities between different kgs are semantically close with minimum gaps between modalities. (iii) we experimentally validate the effectiveness and superiority of mclea as it achieves state-ofthe-art performance on several public datasets in both supervised and unsupervised settings. the overall results also suggest that our mclea is capable of learning more discriminative embedding space for multi-modal entity alignment. ## proposed method we start with the problem formulation and the notations. a multi-modal kg is denoted as @xmath0, r, a, v, t ), where e, r, a, v, t are the sets of entities, relations, attributes, images, and triples, respectively. as two kgs to be aligned, the aim of ea is to find aligned entity pairs ## conclusion this paper presented a novel method termed mclea to address the multi-modal entity alignment. mclea utilizes multi-modal information to obtain the joint entity representations and it is composed of two losses, intra-modal contrastive loss, and inter-modal alignment loss, to explore the intra-modal relationships and cross-modal interactions, respectively. we experimentally validated the state-of-the-art performance of mclea in several public datasets and its capability of learning more discriminative embedding space for entity alignment. for future work, we plan to explore more side information such as entity descriptions to boost alignment performance. therefore, similar to icl, we can automatically learn the relative weight of ial for each modality through task-dependent uncertainty. as mentioned above, the total loss in eq. ( 10 ) can be rewritten as: