Capturing Latent Modal Association For Multimodal Entity Alignment (2025)

Abstract: Multimodal entity alignment aims to identify equivalent entities in heterogeneous knowledge graphs by leveraging complementary information from multiple modalities. However, existing methods often overlook the quality of input modality embeddings during modality interaction – such as missing modality generation, modal information transfer, modality fusion – which may inadvertently amplify noise propagation while suppressing discriminative feature representations. To address these issues, we propose a novel model – CLAMEA for capturing latent modal association for multimodal entity alignment. Specifically, we use a self- attention mechanism to enhance salient information while attenuating noise within individual modality embeddings. We design a dynamic modal attention flow fusion module to capture and balance latent intra- and inter-modal associations and generate fused modality embeddings. Based on both fused and available modalities, we adopt variational autoencoder (VAE) to generate high quality embeddings for the missing modality. We use a cross-modal association extraction module to extract latent modal associations from the completed modality embeddings, further enhancing embedding quality. Experimental results on two real-world datasets demonstrate the effectiveness of our approach, which achieves an absolute 3.1% higher Hits@ 1 score than the sota method.

## introduction multimodal knowledge graphs (mmkgs) @xcite are an extension of traditional knowledge graphs that integrate information from multiple modalities, such as text, images, and triples. mmkgs have demonstrated wide applicability in tasks like question answering @xcite and recommendation systems @xcite . multimodal entity alignment (mmea) aims to identify equivalent entities in mmkgs by leveraging their multimodal information. the process of mmea can be broadly divided into two stages: modality embedding and modality interaction. current mmea methods primarily focus on optimizing modality embeddings or improving modality interaction mechanisms to enhance overall model performance. prominent examples for modality embedding encoders include transe @xcite , bert @xcite , and resnet @xcite , which have become well-established for deriving relational, attribute, and visual embeddings, respectively. modality interaction refers to the integration of multimodal information, such as missing modality generation, modal information transfer and modality fusion. specifically, eva @xcite employs weighted concatenation to fuse modality embeddings. msnea @xcite uses the visual modality to guide other modalities. meaformer @xcite adopts an attentionbased dynamic weighting and concatenation strat-egy to fuse modality embeddings. ack-mmea @xcite constructs attribute-consistent mmkgs by fusing and generating modality embeddings. geea @xcite employs a variational autoencoder (vae) to generate more expressive modality embeddings. pmmea @xcite enhances modality embeddings by incorporating positional information. although existing methods have achieved promising results, they often overlook the quality of input modality embeddings during modality interactions-such as missing modality generation @xcite , modal information transfer @xcite , and modality fusion @xcite -which may lead to noise amplification and the suppression of key information, ultimately affecting the entity alignment performance. as shown in the figure 1 , the quality of the input modality embeddings significantly affects the quality of the modality embeddings obtained after interaction, which in turn impacts the final alignment performance. moreover, studies such as ibmea @xcite , simdiff @xcite similarly indicate that highlighting alignment information and suppressing irrelevant information are crucial, and the quality of embeddings plays a key role in the outcomes of subsequent interactions. to address the above issues, we propose a novel method-clamea, which aims to improve the quality of modality embeddings by capturing latent modal associations. firstly, we introduce a self-attention mechanism to highlight the key information in modal embeddings and suppress noise, thereby enhancing modal embeddings. secondly, we design a dynamic modal attention flow fusion module to capture and balance latent intra-and inter-modal associations, generating fused modality embeddings. thirdly, we leverage both the fused and avaiable modalities to generate embeddings for missing modalities, thereby mitigating the negative impact of missing modalities. finally, we further extract the latent modal associations between modalities to further optimize the modal embedding. the main contributions of this paper are summarized as follows: ## experiment in this section, we conduct extensive experiments on two real-world datasets. we first briefly introduce the experimental settings, and then present and analyze the results to verify the effectiveness of the clamea model. for more detailed experimental setup, please refer to appendix f. ## conclusion this paper proposes a novel mmea method called clamea, which aims to address the issue of modality embedding quality in input modality interactions by capturing latent modal associations. specifically, we design a dynamic modal attention flow fusion module to capture and balance the associations between modalities, a cross-modal association extraction module to further extract latent cross-modal associations, and a modal feature enhancement module and a dynamic missing modality generation module to enhance and generate modality embeddings, respectively. extensive experiments on two datasets demonstrate the effectiveness of clamea.