Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking (2024)

Abstract: Multimodal entity linking (MEL) aims to link ambiguous mentions in multimodal contexts to entities in a multimodal knowledge graph. A pivotal challenge is to fully leverage multi-element correlations between mentions and entities to bridge modality gap and enable fine-grained semantic matching. Existing methods attempt several local correlative mechanisms, relying heavily on the automatically learned attention weights, which may over-concentrate on partial correlations. To mitigate this issue, we formulate the correlation assignment problem as an optimal transport (OT) problem, and propose a novel MEL framework, namely OT-MEL, with OT-guided correlation assignment. Thereby, we exploit the correlation between multimodal features to enhance multimodal fusion, and the correlation between mentions and entities to enhance fine-grained matching. To accelerate model prediction, we further leverage knowledge distillation to transfer OT assignment knowledge to attention mechanism. Experimental results show that our model significantly outperforms previous state-of-the-art baselines and confirm the effectiveness of the OT-guided correlation assignment.

## introduction entity linking (el) is an important yet challenging task in knowledge acquisition, which can facilitate applications such as information retrieval @xcite , question answering @xcite and dialogue systems @xcite . in recent years, with the rapid increase of multimodal content, multimodal entity linking (mel) has been proposed, which aims to link ambiguous mentions in multimodal contexts to entities in the multimodal knowledge graph (kg). as shown in figure 1 , given a mention robert in texts with related images, the mel model requires the mention to be corresponded to the entity robert_downey in the multimodal kg. since the mention can be ambiguous with limited textual information, it is crucial to enrich the text with its related visual images. to utilize multimodal information for mel, a pivotal challenge is to leverage multi-element correlations within entity and mention contexts: 1) the multi-element correlations between modalities, where the textual tokens and visual patches may have correlative correspondence. as shown in figure 1 , the token robert and holland correspond to different local areas of human faces in the image. these correlations serve as message-passing pivots between the text and image, which can be valuable to bridge the semantic gap of modalities @xcite . 2) the multi-element correlations between mentions and entities, where the context of mentions and entities can have multiple correlative cues for semantic matching. as shown in figure 1 , the mentions and entity both involve token robert, and both the mention and the entity involve the word actor. these correlations depict similarities between the mention and entity, providing explicit evidence for el. to facilitate these multi-element correlations, existing studies propose sophisticated correlative mechanisms, such as feature concatenation @xcite , attention-based modules @xcite , hierarchical gates @xcite and graph-based aggregation @xcite . however, these models learn the correlations of elements locally, and rely heavily on the automatically learned correlation assignments. this design may over-concentrate on part of elements, dominating the correlations without global consideration. for example, all entities in figure 1 have surface name robert the same to mention robert, which may attract the correlation assignment and conceal crucial correlations of token actor in texts and the visual patch of human face in images. to mitigate the above issues, we formulate the multi-element correlation assignment as an optimal transport (ot) problem. taking the multimodal correlation as an example, given the cost of each pair of textual tokens and visual patches, the ot problem aims to find an optimal transport plan to transfer the token to a patch with assigned weights, such that the total cost is minimal. in contrast to the automatically learned attention map, the ot plan globally considers the correlation between source and target elements, surpassing the degraded dominant correspondence @xcite . following the above idea, we propose a novel mel framework with ot guided correlation assignment, termed ot-mel. specifically, we first adopt multimodal feature encoders to represent textual tokens and visual patches for both mentions and entities. afterward, we propose an ot-based correlation assignment method between textual tokens and visual patches, with which we further integrate multimodal features and bridge the semantic gap of modalities. additionally, we also build unimodal correlations between mentions and entities with ot-based correlation assignment. by measuring the similarity of the multimodal and unimodal features between mentions and entities, the model derives the overall matching score. to enhance efficiency, we further propose distillation between ot assignment to attention map, such that we can use attention to approximate ot plan. the contributions of this paper are three-fold: ## method in this section, we introduce our ot-mel model, which consists of three modules: 1) multimodal feature encoder, 2) multimodal feature interaction, and 3) multimodal entity linking. the architecture of ot-mel is shown in figure 2 . ## experiments in this section, we carried out comprehensive experiments on three public mel benchmark datasets to evaluate the effectiveness of ot-mel model. ## conclusion this paper introduces a mel model with otguided correlation assignment, termed ot-mel. previous studies rely heavily on the automatically learned attention weights, which may overconcentrate on partial correlations. to this end, ot-mel proposes capturing these multi-element correlations across modalities with ot. to improve the efficiency, we further transfer knowledge from ot to attention mechanism. experiments on 3 datasets illustrate that our model outperforms current stateof-the-art models. in the future, we will explore other efficient method to deploy ot in mel.