Incorporating Object-Level Visual Context for Multimodal Fine-Grained Entity Typing (2023)

Abstract: Fine-grained entity typing (FGET) aims to assign appropriate fine-grained types to entity mentions within their context, which is an important foundational task in natural language processing. Previous approaches for FGET only utilized textual context information. However, in the form of short text, the contextual semantic information is often insufficient for FGET. In many real-world scenarios, text is often accompanied by images, and the visual context is valuable for FGET. To this end, we firstly propose a new task called multimodal fine-grained entity typing (MFGET). Then we construct a large-scale dataset for multimodal fine-grained entity typing called MFIGER based on FIGER. To fully leverage both textual and visual information, we propose a novel Multimodal Object-Level Visual Context Network (MOVCNet). MOVCNet can capture fine-grained semantic information by detecting objects in images, and effectively merge both textual and visual context. Experimental results demonstrate that our approach achieves superior classification performance compared to previous text-based approaches.

## introduction fine-grained entity typing (fget) aims to classify an entity mention with its context into one or more fine-grained types. for example, given a sentence "lionel messi won the championship of 2022 fifa world cup", the mention "lionel messi" should be classified as person as its coarsegrained type and athlete as its fine-grained type. fget serves many down-stream nlp applications, such as relation extraction @xcite and entity linking @xcite , thus is the foundation for building knowledge graphs. one major challenge of fget lies in its rich and fine-grained labels with some kind of hierarchical structure @xcite @xcite . without taking into account labels' interdependencies, it's hard to classify entity mention in isolation due to the large label set. some recent works attempt to address this issue by leveraging label structures or statistics @xcite @xcite . most of the previous methods only focus on textual content. however, in some certain situations, text can not provide sufficient contextual information to serve as the basis for classification, making it difficult to directly determine the ground truth label of the entity mention. in many real-world scenarios, texts are often accompanied by images and the images also contain rich semantic information, which provides additional help to the fget task. the advantages of incorporating visual contexts of images into textual contexts for fget are summarized as follows: therefore, considering that visual context is helpful for fget, we try to introduce images into fget, proposing a new task called multimodal fine-grained entity typing (mfget). in the meanwhile, we construct an mfget dataset with a corresponding image for each sentence. the images are derived from wikidata. 1 to incorporate visual information, we propose a multimodal object-level visual context network movc-net. movcnet can effectively extract local object features in the image that are relevant to the text. through the object-based attention mechanism, movcnet can better fuse the text and image context and further improve the performance of fine-grained classification. experimental results demonstrate the effectiveness of movcnet compared with previous approaches. the main contributions of this work can be summarized as follows: ## methodolgy in this section, we first introduce the definition of fget and mfget tasks. next, we provide a detailed explanation of how to construct the mfiger dataset. then, we elaborate on the implementation details of our proposed model movcnet in four parts. figure 2 shows the comprehensive architecture of movcnet. ## experiments in this section, we compare our proposed method with previous state-of-the-art approaches to validate the effectiveness of our model. we first introduce the dataset and the statistical information that we use. then, we provide a brief overview of the baseline models for comparison and our implementation details. finally, we present the overall results and analysis on the performance comparison between our model and others. ## conclusion in this paper, we propose a new task called multimodal fine-grained entity typing (mfget). based on figer, we construct a multimodal dataset mfiger with both text and accompanying images for mfget. we propose a novel multimodal model movcnet to incorporate object-level visual context for fget. specifically, movcnet can capture relevant objects in images and merge visual and textual contexts effectively. experimental results on mfiger demonstrate that our proposed model achieves the best performance compared with competitive existing models.