MELOV : Multimodal Entity Linking with Optimized Visual Features in Latent Space (2024)

Abstract: Multimodal entity linking (MEL), which aligns ambiguous mentions within multimodal contexts to referent entities from multimodal knowledge bases, is essential for many natural language processing applications. Previous MEL methods mainly focus on exploring complex multimodal interaction mechanisms to better capture coherence evidence between mentions and entities by mining complementary information. However, in real-world social media scenarios, vision modality often exhibits low quality, low value, or low relevance to the mention. Integrating such information directly will backfire, leading to a weakened consistency between mentions and their corresponding entities. In this paper, we propose a novel latent space vision feature optimization framework MELOV, which combines inter-modality and intra-modality optimizations to address these challenges. For the inter-modality optimization, we exploit the variational autoencoder to mine shared information and generate text-based visual features. For the intra-modality optimization, we consider the relationships between mentions and build graph convolutional network to aggregate the visual features of semantic similar neighbors. Extensive experiments on three benchmark datasets demonstrate the superiority of our proposed framework.

## introduction entity linking (el) is the task of assigning ambiguous mentions in text to their corresponding entities in a knowledge base. as a bridge between unstructured content and structured knowledge bases, el plays a vital role in many downstream natural language processing applications, such as content analysis @xcite , semantic search @xcite , question answering @xcite and so on. traditional el methods @xcite mainly focus on addressing the text-based el by resolving text context. however, with the rapid development of social media, images along with text have become the most common form of web information, which brings new challenges for el models to effectively integrate visual information and understand multimodal content. thus, the multimodal entity linking (mel) task @xcite has been proposed, which extends the scope from textual el to heterogeneous formats, i.e. linking mentions with textual context and visual context to referent entities in multimodal knowledge bases. multiple previous mel works @xcite @xcite @xcite @xcite @xcite mainly focus on exploring complex multimodal interaction mechanisms to better capture coherence evidence between mentions and entities by mining complementary information. although these works have achieved promising performance in the past few years, they usually assume that each mention is associated with high-quality, high-value, and high-relevance visual images, which is unrealistic in real-world social media scenarios. figure 1 shows several social media examples and mel still faces the pervasive challenges presented by the following poor visual images: low-quality. user-posted images may appear blurry, unclear, or pixelated due to factors such as image compression, low original image quality and network transmission issues. low-value. as shown in figure 1 (b), this image is widely used in social media, which only expresses the happy mood of users in most cases. however, this type of images does not enhance the overall comprehension of the text and may even mislead the mel model into linking "michael" to a person rather than the brand "michael kors". ## methodology figure 2 shows the overall architecture of our melov. we first extract both global and local features of textual and visual inputs to obtain global descriptive semantics and preserve fine-grained details of words or image patches. afterwards, we optimize visual features at two perspectives: intermodality generation based on the shared information of heterogeneous textual modality and intramodality aggregation based on the correlation of mentions. we obtain the optimized visual features by adaptively fusing the two perspective features and original features. finally, we employ multigrained interactions using optimized visual features as input to derive the similarity matching score for each mention-entity pair to make the ultimate multimodal entity linking decisions. ## conclusion in this paper, we focus on poor images in multimodal entity linking. to avoid the negative effects of poor images while preserving the implicit visual cues of original images, we propose melov, a novel joint optimization framework to combine inter-modality generation and intra-modality aggregation to optimize visual features in latent space. this effectively leverages the shared information from heterogeneous textual features and relevant visual details of semantic similar neighbors, allowing for simultaneously handling all types of poor images. experimental results on three public multimodal entity linking datasets demonstrate the effectiveness of optimizing visual features in latent space and our proposed melov achieves new state-of-the-art performance.