Visual Named Entity Linking: A New Dataset and A Baseline (2022)

Abstract: Visual Entity Linking (VEL) is a task to link regions of images with their corresponding entities in Knowledge Bases (KBs), which is beneficial for many computer vision tasks such as image retrieval, image caption, and visual question answering. While existing tasks in VEL either rely on textual data to complement a multi-modal linking or only link objects with general entities, which fails to perform named entity linking on large amounts of image data. In this paper, we consider a purely Visual-based Named Entity Linking (VNEL) task, where the input only consists of an image. The task is to identify objects of interest (i.e., visual entity mentions) in images and link them to corresponding named entities in KBs. Since each entity often contains rich visual and textual information in KBs, we thus propose three different sub-tasks, i.e., visual to visual entity linking (V2VEL), visual to textual entity linking (V2TEL), and visual to visual-textual entity linking (V2VTEL). In addition, we present a high-quality human-annotated visual person linking dataset, named WIKIPerson. Based on WIKIPerson, we establish a series of baseline algorithms for the solution of each sub-task, and conduct experiments to verify the quality of the proposed datasets and the effectiveness of baseline methods. We envision this work to be helpful for soliciting more works regarding VNEL in the future. The codes and datasets are publicly available at https: //github.com/ict-bigdatalab/VNEL.

## introduction an in-depth understanding of visual content in an image is fundamental for many computer vision tasks. vel @xcite is a task to put the image understanding to the entity-level. for example, given an image of the debate between trump and hillary, the goal of vel is not only to recognize the region of trump and hillary, but also to link them to the correct entity in kbs (e.g., wikidata @xcite , dbpedia @xcite , or yago @xcite ). just as the significance of textual entity linking for many nlp tasks such as information extraction and information retrieval @xcite , visual tasks, such as image retrieval @xcite and image caption @xcite , would also benefit from entitylevel fine-grained comprehension of images. in recent years, vel has been given increasing attention. early works @xcite try to link objects in images with general entities, e.g., 'person' and 'suit', in kbs as is described in figure 1 @xcite . apparently, these works are restricted to the coarse-level entity linking and fail to distinguish objects within the same class. besides, there are also some works that make use of deep image understanding to link objects with named entities in kbs @xcite @xcite . however, they generally require detailed entity mention information in text, which plays a vital role via multi-modal entity linking as shown dataset multi-modal entity-aware entity-labeled modality kb source lang size aida @xcite ) ✓ ✓ t @xmath0, v e wikipedia news en 8k captions in figure 1(c ). we argue that all the above tasks fail to process the named entity linking well for images without any text annotations, which is often the case in social media platforms. in this work, we consider a purely visual-based named entity linking (vnel) task, which is described in figure 1(d ). given an image without textual description, the goal is to link the visual mention in the image with the whole image as the context to the corresponding named entity in kbs. considering the format of entity in kbs, such as textual descriptions, images, and other structured attributes, we further introduce three sub-tasks according to the type of entity context, i.e., the visual to visual entity linking (v2vel), visual to textual entity linking (v2tel), and visual to visual-textual entity linking (v2vtel). we believe these tasks could put forward higher requirements and more detailed granularity for image understanding, crossmodal alignment, and multi-modal fusion. following the definition of vnel, currently public available el datasets may not fit for our research, as they either only focus on textual modality or lack of detailed annotations for entity information in each image. as a result, we release a new dataset called wikiperson. the wikiperson is a high-quality human-annotated visual person linking dataset based on wikipedia. unlike previously commonly-used datasets in el, the mention in wikiperson is only an image containing the person entity with its bounding box. the corresponding label identifies an unique entity in wikipedia. for each entity in wikipedia, we provide textual descriptions as well as images to satisfy the need of three sub-tasks. in the experiments, we benchmark a series of baseline models on wikiperson under both zeroshot and fine-tuned settings. in detail, we adopt a universal contrastive learning framework to learn a robust and effective representation for both mentions and entities. experimental results show that visual mention detection visual context …… greg merson q4806 …… q2406 q318149 q4806 an american professional poker player.. ## the visual named entity linking task this section first presents a formal definition of the task. then we introduce the complete building procedure of the human-annotated dataset, which covers a wide variety of wikipedia person entities for further research. finally, an in-depth data analysis will be elaborated on in detail. ## baseline methods generally, the vnel task is to link mentions in the input image with the corresponding entities from a large-scale kb. typically, the existing vnel system is often implemented as a two-stage process, i.e., the candidate retrieval stage and the entity disambiguation stage, to balance the efficiency and the effectiveness. in this work, we implement a fast end-to-end linking directly from a large-scale collection by employing an efficient model. we take a widely-used bi-encoder contrastive learning framework to learn robust and effective representations of both visual mentions and entities. given a visual mention v m and a candidate entity e i , which is accompanied by visual description v e i and/or textual description t e i , the framework aims to produce a relevance score between the mention and the entity. the overall structure of the framework is shown in figure 6 , which consists of two major components, namely the mention encoder and the entity encoder. these two encoders aim to extract features as embeddings f m for the input image and f e for the entity. for each encoder, we directly take existing pre-trained models as the implementation. inspired by existing works @xcite in applying pretrained model, we add a feed-forward layer to transform the vector generated from the encoder to the task-oriented embedding space. after that, a residual connection @xcite is added to obtain f m and f e i , followed by using l2 norm and dotproduction to calculate the similarity score. where w m 1 and w m 2 are learnable parameters for mention representation learning, and w e 1 and w e 2 are learnable parameters for entity representation learning. since each sub-task of vnel have different types of inputs, we thus implement each baseline with different encoders: ## experiments during experiments, we split images in wikiperson into train, dev, and test set with the ratio of 6:2:2. besides, to avoid the bias of popular entities affecting the evaluation, each named entity appears at most once in test set. for evaluation, we report two widely-used metrics of top-k retrieve: recall@k (@xmath1, 3, 5, 10) and mean reciprocal 4 the detailed analysis of this strategy is displayed in the appendix due to the page limitation. ## related work entity linking. there is extensive research on el, which serves as a classic nlp task. with the help of large-scale pre-train language models @xcite , several recent deep learning methods @xcite @xcite achieve 90%+ accuracy on aida @xcite , which is a commonly used high-quality robust el dataset. however, as mentioned in @xcite , it seems that the current methods have already torched the task ceiling. as a result, many more challenging el-related tasks are formulated. for example, zeroshot entity linking @xcite , engaging other features like global coherence across all entities in a document, nil prediction, joining md and ed steps together, or providing completely end-to-end solutions to address emerging entities is rapidly evolving @xcite . multi-modal entity linking. recently, multimodal entity linking(mel) @xcite task has also been proposed for consideration. given a text with images attached, mel uses both textual and visual information to map an ambiguous mention in the text to an entity in the kbs. @xcite proves that image information helps identify the mention in social media for the fuzzy and short text. furthermore, @xcite transfer the scene to twitter and perform mel on twitter users. @xcite proposes an attention-based structure to eliminate dis-tracting information from irrelevant images and builds a multi-source social media multi-modal dataset. @xcite builds a multi-modal entity linking dataset with diversified contextual topics and entity types. however, for all those works, the text input plays a vital part, and the visual input only serves as a complementary role to the text. multi-modal dataset. at the same time, our work is also related to the multi-modal imagetext datasets, which is also a hot issue in recent years. flicker30k @xcite annotates 30k image-caption pairs from flicker with five descriptive sentences per image, such as "a man is wearing a tie." in addition, mscoco caption @xcite scale up the size with over one and a half million captions describing over 330000 images. however, the caption in all these datasets is descriptive sentences and non-entity aware. as a result, some work has started to build a news-related dataset for entity-aware image caption tasks. for example, @xcite focus on the news website and have crawled 100k image-caption pairs. @xcite expand the size of the dataset. nevertheless, the detailed entity information is neither annotated nor linked to the kbs. ## conclusion and future work to tackle the limitation that previous visual entity linking either rely on textual data to complement a multi-modal linking or only link objects with general entities, we introduce a purely visual-based named entity linking task, where the input only contains the image. the goal of this task is to identify objects of interest in images and link them to corresponding named entities in kbs. considering the rich multi-modal contexts of each entity in kbs, we propose three different sub-tasks, i.e. the v2vel sub-task, the v2tel sub-task, and the v2vtel sub-task. moreover, we build a high-quality human-annotated visual person linking dataset, named wikiperson, which aims at recognizing persons in images and linking them to wikipedia. based on wikiperson, we introduce several baseline algorithms for each sub-task. according to the experimental results, the wikiperson is a challenging dataset worth further explorations. in the future, we intend to build a larger scale vnel dataset with diverse types and adopt more advanced models to achieve higher accuracy.