Multi-Modal Knowledge Graph Transformer Framework for Multi-Modal Entity Alignment (2023)

Abstract: Multi-Modal Entity Alignment (MMEA) is a critical task that aims to identify equivalent entity pairs across multi-modal knowledge graphs (MMKGs). However, this task faces challenges due to the presence of different types of information, including neighboring entities, multi-modal attributes, and entity types. Directly incorporating the above information (e.g., concatenation or attention) can lead to an unaligned information space. To address these challenges, we propose a novel MMEA transformer, called Meaformer, that hierarchically introduces neighbor features, multi-modal attributes, and entity types to enhance the alignment task. Taking advantage of the transformer’s ability to better integrate multiple information, we design a hierarchical modifiable self-attention block in a transformer encoder to preserve the unique semantics of different information. Furthermore, we design two entity-type prefix injection methods to redintegrate entity-type information using type prefixes, which help to restrict the global information of entities not present in the MMKGs.

## introduction multi-modal entity alignment (mmea) is a challenging task that aims to identify equivalent entity pairs across multiple knowledge graphs that feature different modalities of attributes, such as text and images. to accomplish this task, sophisticated models are required to effectively leverage information from different modalities and accurately align entities. this task is essential for various applications, such as cross-lingual information retrieval, question answering @xcite , and recommendation systems @xcite . mmea @xcite @xcite is challenging due to the heterogeneity of mmkgs (e.g., different neighbors, multi-modal attributes, distinct types), which makes it difficult to learn rich knowledge representations. previous approaches such as poe @xcite concatenated all modality features to create composite entity representations but failed to capture interactions among heterogeneous modalities. more recent works @xcite designed multi-modal fusion modules to better integrate attributes and entities, but still did not fully exploit the potential interactions among modalities. these methods also ignored inter-modality dependencies between entity pairs, which could lead to incorrect alignment. generally speaking, although mmkgs offer rich attributes and neighboring entities that could be useful for multi-mdoal entity alignment, current methods have limitations in (i) ignoring the differentiation and personalization of the aggregation of heterogeneous neighbors and modalities leading to the misalignment of cross-modal semantics, and (ii) lacking the use of entity heterogeneity resulting in the non-discriminative representations of different meaning/types of entities. therefore, the major challenge of mmea task is how to perform differentiated and personalized aggregation of heterogeneous information of the neighbors, modalities, and types. although such information is beneficial to entity alignment, directly fusing will lead to misalignment of the information space, as illustrated in figure 1 . firstly, notable disparities between different modalities make direct alignment a challenging task. for example, both the visual attribute of entity ruby in mmkg1 and the neighbor information of the entity ruby in mmkg2 contain similar semantics of programming, but data heterogeneity may impede effective utilization of this information. secondly, complex relationships between entities require a thorough understanding and modeling of contextual information and semantic associations. entities such as the ruby, the perl, and the entity larry wall possess unique attributes, and their inter-relationships are non-trivial, necessitating accurate modeling based on contextual information and semantic associations. furthermore, the existence of multiple meanings for entities further exacerbates the challenge of distinguishing between two entities, such as in the case of the ruby, which has different meanings in the mmkg1 and mmkg3 where it may be categorized as a jewelry entity or a programming language entity, respectively. to overcome the aforementioned challenges, we propose a novel multi-modal entity alignment transformer named moalign foot_0 . our framework hierarchically introduces neighbor, multimodal attribute, and entity types to enhance the alignment task. we leverage the transformer architecture, which is known for its ability to process heterogeneous data, to handle this complex task. moreover, to enable targeted learning on different modalities, we design a hierarchical modifiable self-attention block in the transformer encoder, which builds associations of task-related intra-modal features through the layered introduction. additionally, we introduce positional encoding to model entity representation from both structure and semantics simultaneously. furthermore, we integrate entitytype information using an entity-type prefix, which helps to restrict the global information of entities that are not present in the multi-modal knowledge graphs. this prefix enables better filtering out of unsuitable candidates and further enriches entity representations. to comprehensively evaluate the effectiveness of our proposed approach, we design training objectives for both entity and context evaluation. our extensive experiments on benchmark datasets demonstrate that our approach outperforms strong competitors and achieves excellent entity alignment performance. our contributions can be summarized as follows. multi-modal entity alignment task. multimodal entity alignment @xcite @xcite aims to determine if two entities from different multimodal knowledge graphs refer to the same realworld entity. this involves calculating the similarity between pairs of entities, known as alignment seeds. the goal is to learn entity representations from two multi-modal knowledge graphs ## framework this section introduces our proposed framework moalign. as shown in figure 2 , we introduce positional encoding to simultaneously model entity representation from both modality and structure. to hierarchically introduce neighbor and multi-modal attributes, we design a hierarchical modifiable selfattention block. this block builds associations of task-related intra-modal features through the layered introduction. furthermore, for integrating entity-type information, we design a prefix-injected self-attention mechanism, which helps to restrict the global information of entities not present in the mmkgs. additionally, moalign also design training objectives for both entity and context evaluation to comprehensively assess the effectiveness. ## conclusion this paper proposes a novel mmea framework. it incorporates cross-modal alignment knowledge using a two-stage transformer encoder to better capture complex inter-modality dependencies and semantic relationships. it includes a mmkg transformer encoder that uses self-attention mechanisms to establish associations between intra-modal features relevant to the task. our experiments show that our approach outperforms competitors.