W iki D iverse: A Multimodal Entity Linking Dataset with Diversified Contextual Topics and Entity Types (2022)

Abstract: Multimodal Entity Linking (MEL) which aims at linking mentions with multimodal contexts to the referent entities from a knowledge base (e.g., Wikipedia), is an essential task for many multimodal applications. Although much attention has been paid to MEL, the shortcomings of existing MEL datasets including limited contextual topics and entity types, simplified mention ambiguity, and restricted availability, have caused great obstacles to the research and application of MEL. In this paper, we present WikiDiverse, a high-quality human-annotated MEL dataset with diversified contextual topics and entity types from Wikinews, which uses Wikipedia as the corresponding knowledge base. A well-tailored annotation procedure is adopted to ensure the quality of the dataset. Based on WikiDiverse, a sequence of well-designed MEL models with intra-modality and inter-modality attentions are implemented, which utilize the visual information of images more adequately than existing MEL models do. Extensive experimental analyses are conducted to investigate the contributions of different modalities in terms of MEL, facilitating the future research on this task.

## introduction entity linking (el) has attracted increasing attention in the natural language processing community, which aims at linking ambiguous mentions to the referent unambiguous entities in a given knowledge base (kb) @xcite . it has been applied to a lot of downstream tasks such as information extraction @xcite , question ## related work textual el there is vast prior research on textual entity linking. multiple datasets have been proposed over the years including the manuallyannotated high-quality datasets like aida @xcite , automatically-annotated large-scale datasets like cweb (guo and barbosa, 2018) and zero-shot datasets like zeshel (logeswaran et al., 2 https://www.wikinews.org . it is a free-content news wiki. multimodal el in recent years, the growing trend towards multimodality requires to extend the research of el from monomodality to multimodality. moon et al. (2018) first address the mel task and build a zero-shot framework, which extracts textual, visual and lexical information for el in social media posts. however, its proposed dataset is unavailable due to gdpr rules. adjali et al. (2020a,b) propose a framework of automatically building the mel dataset from twitter. the dataset has limited entity types and ambiguity of mentions, thus it is not challenging enough. zhang et al. (2021) study on a chinese mel dataset collected from the chinese social media platform weibo, which mainly focuses on the person entities. gan et al. (2021) release a mel dataset collected from movie reviews and propose to disambiguate both visual and textual mentions. this dataset mainly focuses on characters and persons of the movie domain. peng (2021) propose three mel datasets, which are built from weibo, wikipedia, and richpedia information and use cndbpedia, wikidata and richpedia as the corresponding kbs. however, using wikipedia as the target dataset may lead to the data leakage problem as many language models are pretrained on it. our mel dataset is also related to other named entity-related multimodal datasets, including entityaware image caption datasets @xcite @xcite , multimodal ner datasets @xcite , etc. however, the entities in these datasets are not linked to a unified kb. so our research of mel can enhance the understanding of named entities, thereby enhancing the research in these areas. ## problem formulation multimodal entity linking is defined as mapping a mention with multimodal contexts to its referent entity in a pre-defined multimodal kb. since the boundary and granularity of mentions may be con-troversial, the mention span is usually pre-specified. here we assume each mention has a corresponding entity in the kb, which is the in-kb evaluation problem. formally, let e represent the entity set of the kb, which usually contains millions of entities. each mention m or entity e @xmath0, v e i and textual context t m , t e i . here t m and t e i represent the textual spans around m and e i respectively. v m is the image correlated with m and v e i is the image of e i in the kb. in real life, entities in kbs may contain more than one image. to simplify it, we select the first image of e i as v e i and leave mel with multiple images per entity as the future work. so the referent entity of mention m is predicted through: ## dataset construction in this section, we present the dataset construction procedure. many factors including annotation quality, coverage of topics, diversity of entity types, coverage of ambiguity are taken into consideration to ensure the research value of wikidiverse. ## methods it is challenging to directly predict the entity from a large-scale kb because it consumes large amounts of time and space resources. therefore, following previous work @xcite @xcite , we split mel into two steps: 1) candidate retrieval (cr) is first used to guarantee the recall and obtain a candidate entity set consisting of the topk entities that are most similar to the mention; 2) entity disambiguation (ed) is then conducted to guarantee the precision and predict the entity with the highest matching score. ## conclusion and future work we propose wikidiverse, a manually-annotated wikipedia-based mel dataset collected from wikinews. to overcome the weaknesses of existing datasets, wikidiverse covers a wide range of topics, entity types and ambiguity. we implement a series of baselines and carry out multiple experiments over the dataset. according to the experimental results, wikidiverse is a challenging dataset worth further exploration. besides multimodal entity linking, wikidiverse can also be applied to evaluate the pre-trained language model, multimodal named entity typing/recognition, multimodal topic classification, etc. in the future, we plan to 1) utilize more than one images of each entity 2) adopt finer-grained multimodal interaction models for this task and 3) transfer the model to more general scenarios such as el in articles.