Multimodal Knowledge Learning for Named Entity Disambiguation (2022)

Abstract: With the popularity of online social media, massive-scale multimodal information has brought new challenges to traditional Named Entity Disambiguation (NED) tasks. Recently, Multimodal Named Entity Disambiguation (MNED) has been proposed to link ambiguous mentions with the textual and visual contexts to a predefined knowledge graph. Existing attempts usually perform MNED by annotating multimodal mentions and adding multimodal features to traditional NED models. However, these studies may suffer from 1) failing to model multimodal information at the knowledge level, and 2) lacking multimodal annotation data against the large-scale unlabeled corpus. In this paper, we explore a pioneer study on leveraging multimodal knowledge learning to address the MNED task. Specifically, we first harvest multimodal knowledge in the Meta-Learning way, which is much easier than collecting ambiguous mention corpus. Then we design a knowledge-guided transfer learning strategy to extract unified representation from different modalities. Finally, we propose an Interactive Multimodal Learning Network (IMN) to fully utilize the multimodal information on both the mention and knowledge sides. Extensive experiments conducted on two public MNED datasets demonstrate that the proposed method achieves improvements over the state-of-the-art multimodal methods.

## introduction nowadays, online social media have become more and more important in our daily life. the massivescale blogs posted on these social media hide valuable information that can be used to understand users and distill user preferences. however, how to extract valuable information is extremely challenging because the posts are always free-form, especially the text. named entity disambiguation (ned) is such a critical task for extracting structured information, aiming to map ambiguous mentions from free-form texts to specific entities in a predefined knowledge graph. ned can benefit many downstream applications, such as information retrieval @xcite ), question answering @xcite , relation extraction @xcite , etc. existing research on ned mainly focuses on texts and has been proven to be successful for well-formed texts. however, with the popularity of incorporating a mix of text and images on social media platforms (e.g. twitter foot_0 , instargram foot_1 , snapchat foot_2 , etc.), more ambiguous mentions appear in the short and noisy text. due to the enormous number of mentions arising from incomplete and inconsistent expressions, the traditional text-only ned methods are limited in dealing with crossmodal ambiguity, making it difficult to link these mentions accurately. for example, on one hand, it is difficult to distinguish the mention swift refers to taylor swift or ben swift from the textual context in fig 1 . on the other hand, due to the obstruction of eyes, hats, and other objects, the target person cannot be directly recognized from the image alone through face recognition techniques. when multimodal contexts in the post, as well as the historical knowledge, are combined, the correct entity ben swift can be predicted from the candidates. that is, the textual and visual features can complement each other. although some recent methods have achieved promising performance for the mned task @xcite , challenges still exist. first, sufficient annotated corpus with texts and images is required to train a multimodal model, which is costly in practice @xcite . and lacking sufficient training data would limit the performance of neural models. second, previous works mainly learn from the multimodal mention contexts and do not exploit available information at the knowledge level, without harvesting useful descriptions and historical data with visual features. in this paper, we focus on solving mned tasks at the knowledge level. to reduce the dependence on annotated data and fully use the unsupervised multimodal corpus, we firstly train a multimodal feature extractor by implementing a knowledgeguided transfer learning strategy. then we enrich multimodal information at the knowledge level using a meta-learning aggregation strategy, aiming to obtain multimodal entities and mentions using a small number of knowledge annotations. finally, we design an interactive multimodal learning network (imn) to flexibly utilize the multimodal information from both mention contexts and knowledge graph and integrate them. the main contributions of this paper are summarized as follows: the rest of the paper is organized as follows: in section 2, we summarize the related work. in section 3, we formulate the mned task and introduce the proposed method in detail. in section 4, we conduct extensive experiments and analyses. finally, we conclude this work in section 5. ## related work multimodal learning as an efficient mechanism of leveraging contextual information from multiple modalities in parallel, multimodal learning has been applied in a wide range of tasks in recent years @xcite . in previous works, representation of different modalities was mostly obtained separately. for visual representation, cnn-based models such as vgg (simonyan and zisserman, 2014) , google inception @xcite , resnet @xcite are widely adopted in many multimodal tasks. textual features are mostly represented by language models such as glove @xcite , gpt @xcite , xlnet @xcite etc. recently, with the success of pre-training and selfsupervised learning @xcite , several multimodal transfer learning methods and architectures @xcite @xcite have been proposed, and have achieved state-of-the-art results on various vision language tasks, including visual question answering, visual commonsense reasoning, region-to-phrase grounding, image-text retrieval, etc. videobert @xcite learns joint distributions over sequences of visual and linguistic tokens as multimodal features. visionand-language berts @xcite @xcite extend bert architecture to adapt multimodal input by extracting rois from images and regards as image tokens. although these pretraining models can learn unsupervised features in unsupervised corpus, they still need further improvement in tasks that require additional knowledge. and we argue that the self-supervised models still requires guidance of knowledge. named entity disambiguation traditional ned methods mainly focus on text-only corpus which can be divided into two categories, local methods and global methods @xcite . for local methods, each mention is disambiguated separately via handcrafted features @xcite and contextual representations learned by neural networks @xcite . global methods @xcite jointly disambiguate mentions by taking into account the topical coherence among the referred entities in the same document @xcite . for the mned task, the work from @xcite is the first to utilize multimodal mention contexts via weighting the embeddings of images and words based on attention mechanism. the previous multimodal works primarily depend on sufficient training data with fully annotations on all mention modalities which is costly in practice @xcite . although @xcite involve a zero-shot layer in their model to allow for disambiguation of unseen entities during training, the performance is limited if the multimodal information is incomplete in the training data. inspired by recent success on multimodal knowledge graph @xcite @xcite ,we aim at handle mned tasks at the knowledge level, which is much easier than collecting and annotating multimodal corpus. ## conclusion we propose to solve mned task at the knowledge level through multimodal transfer learning and meta learning. with large-scale unsupervised data and a small amount of annotated knowledge, our model significantly outperforms the state-of-theart mned methods. experimental results show that enrich multimodal features at the knowledge level is more conducive to improving the effect of mned models compared with mention contexts annotation. there are still many points worth continuing to explore. in particular, the structural information in the knowledge graph which can be learned by knowledge representation models such as transe may also be useful. besides, the prototype aggregation method still needs further exploration with graph learning models such as gcn etc.