MM - C hat A lign: A Novel Multimodal Reasoning Framework based on Large Language Models for Entity Alignment (2024)

Abstract: Multimodal entity alignment (MMEA) integrates multi-source and cross-modal knowledge graphs, a crucial yet challenging task for data-centric applications.Traditional MMEA methods derive the visual embeddings of entities and combine them with other modal data for alignment by embedding similarity comparison.However, these methods are hampered by the limited comprehension of visual attributes and deficiencies in realizing and bridging the semantics of multimodal data. To address these challenges, we propose MM-ChatAlign, a novel framework that utilizes the visual reasoning abilities of MLLMs for MMEA.The framework features an embedding-based candidate collection module that adapts to various knowledge representation strategies, effectively filtering out irrelevant reasoning candidates. Additionally, a reasoning and rethinking module, powered by MLLMs, enhances alignment by efficiently utilizing multimodal information.Extensive experiments on four MMEA datasets demonstrate MM-ChatAlign’s superiority and underscore the significant potential of MLLMs in MMEA tasks.The source code is available at https://github.com/jxh4945777/MMEA/.

## introduction multimodal entity alignment (mmea) aligns equivalent entities across diverse multimodal knowledge graphs (mmkgs) @xcite , playing a key role in synthesizing heterogeneous data for data-centric applications. unlike traditional entity alignment, mmea necessitates the integration of information across various modalities and mmkgs, thereby imposing higher demands on the visual reasoning ability of mmea methods. current representative mmea methods @xcite @xcite @xcite mainly adopt knowledge representation learning (krl) and measure the similarity of entity embeddings for mmea. as shown in figure 1 , these methods face critical challenges. firstly, their reliance on the representation learning approaches, which convert various attributes of entities into embeddings respectively, weakens the insight into underlying connections in visual attributes. as shown in figure 1 , the left image in icews shows a scene of president biden talking to leaders from other countries, and the right image in wiki is a portrait of young biden. although both images are related to biden, they have different embeddings for their contrasting representation. existing embedding-based mmea methods lack the visual reasoning capability to explicitly recognize that two images represent visual attributes of biden at different times. secondly, high-quality mmea demands the learning of associations with visual, structural, literal, and other information in mmkgs to achieve complementary information integration. however, discrepancies in modal information within mmkgs lead to cross-modal misalignment @xcite , across different modalities may limit the ef-fectiveness of cross-modal information utilization achieved through feature fusion @xcite @xcite . multimodal large language models (mllms) have emerged as front-runners in comprehending visual information and integrating diverse modalities, especially in natural language generation and visual reasoning @xcite . these models excel in deciphering the deep semantic information beyond the visual information, bringing a significant advantage to multimodal knowledge reasoning tasks @xcite . crucially, the extensive background knowledge and advanced reasoning ability of mllm open avenues for enriching the entity information and bridging the semantic gap of various crossmodal attributes, showcasing their potential in adequately understanding and utilizing the breadth of multimodal data. in this paper, we propose mm-chatalign. different from the representation learning-based mmea paradigm of previous methods, this novel framework is designed to maximize the potential of mllms oriented to the mmea task. mm-chatalign utilizes the entity set derived from embedding-based methods as candidates, enhancing alignment accuracy through the visual reasoning capabilities of mllms. the framework initially implements the mmkg-code translation module to effectively represent mmkg in a code format @xcite that is highly compatible with mllms, thus facilitating a better understanding of multi-modal information in the mmkg. furthermore, mm-chatalign capitalizes on the background knowledge and visual reasoning abilities of mllms by generating comprehensive descriptions for entities based on their images, names, and relational data, and reasons for alignment. the rethinking phase evaluates the probabilities of entity pair alignment, revisits results, and potentially expands the search scope via an iterative candidate collection process to ensure precise alignments. extensive experiment results over four representative mmea datasets demonstrate the effectiveness of mm-chatalign and also highlight the feasibility of using mllms for the mmea task. in general, our main contributions are as follows: ## methodology in this paper, we propose the mm-chatalign, a versatile plug-and-play mmea framework that capitalizes on the advanced reasoning abilities and background knowledge of mllms, while optimizing both efficiency and accuracy. the overall architecture of mm-chatalign is shown in figure 2 . the framework integrates an embeddingbased candidate collection module, configurable across various krl methods, designed to exclude non-relevant candidates. moreover, it features a reasoning and rethinking module, powered by mllms, that enhances alignment by effectively leveraging multimodal information. ## conclusion this study introduces mm-chatalign, an innovative framework for mmea that leverages the advanced capabilities of mllms. by incorporating a code-type transformation module for mmkgs and a two-stage multimodal reasoning process, the method realizes the efficient and effective mmea. our experimental results not only validate mm-chatalign's superior performance on newly developed mmea datasets and classical datasets but also highlight the tremendous potential of mllms in challenging mmea tasks. future work will continue to focus on optimizing efficiency, further unleashing the potential of mllm in mmea.