ESPVR : Entity Spans Position Visual Regions for Multimodal Named Entity Recognition (2023)

Abstract: Multimodal Named Entity Recognition (MNER) uses visual information to improve the performance of text-only Named Entity Recognition (NER). However, existing methods for acquiring local visual information suffer from certain limitations: (1) using an attention-based method to extract visual regions related to the text from visual regions obtained through convolutional architectures (e.g., ResNet), attention is distracted by the entire image, rather than being fully focused on the visual regions most relevant to the text; (2) using an object detection-based (e.g., Mask R-CNN) method to detect visual object regions related to the text, object detection has a limited range of recognition categories. Moreover, the visual regions obtained by object detection may not correspond to the entities in the text. In summary, the goal of these methods is not to extract the most relevant visual regions for the entities in the text. The visual regions obtained by these methods may be redundant or insufficient for the entities in the text. In this paper, we propose an Entity Spans Position Visual Regions (ESPVR) module to obtain the most relevant visual regions corresponding to the entities in the text. Experiments show that our proposed approach can achieve the SOTA on Twitter-2017 and competitive results on Twitter-2015.

## introduction named entity recognition (ner) is a fundamental task in the field of information extraction, which can automatically recognize named entities in text and classify them into predefined categories. ner has been widely used for many downstream tasks, such as entity linking and relationship extraction. with the rapid development of social media, multimodal deep learning is widely used to perform structured extraction from massive multimedia news and web product information. among them, multimodal named entity recognition (mner) aims to identify and classify named entities from text using images as auxiliary information. mner can disambiguate multi-sense words by augmenting linguistic representations with visual information, resulting in superior performance compared to traditional named entity recognition (ner). while previous efforts have yielded promising results, they still fall short in effectively selecting visual information. for existing methods of utilizing visual information, we classify it into two types: global visual information and local visual information. some previous works @xcite @xcite @xcite @xcite consider that if the whole image information is input to the multimodal interaction module, then such image information is global visual information. however, the multimodal interaction module relies on attention to select the visual regions associated with the text for interaction. therefore, the visual information that eventually interacts with the text is mainly the local visual information related to the text. in other words, even if the input to the multimodal interaction module is the whole image information, finally the local visual information is selected using an attention-based method and then interacts with the text, so we consider it as a process of using attention to select the local visual information. when using attention to extract visual regions, attention is distracted by the entire image, rather than being fully focused on the visual regions most relevant to the text. therefore, using an attention-based method to select the local visual information not only obtains valuable visual information but also introduces irrelevant visual information. besides, most of the previous approaches @xcite @xcite @xcite use object detection (e.g., mask r-cnn) to detect visual ob-ject regions, and treat visual objects as local visual information to interact with the text. however, object detection has a limited range of recognition categories, so it may not detect all objects within the categories defined by the dataset. moreover, the visual regions obtained by object detection may not correspond to the entities in the text. in summary, the goal of these methods is not to extract the most relevant visual regions for the entities in the text. the visual regions obtained by the above two methods may be redundant or insufficient for entities contained in the text, leading to identifying a non-entity as an entity or incorrectly predicting an entity category. therefore, to obtain the most relevant visual regions for the entities in the text, we propose an entity spans position visual regions (espvr) module. specifically, the espvr module consists of two modules: entity spans identifying (esi) module, visual regions positioning (vrp) module. first, the esi module identifies all entity spans in the text. then the vrp module uses these entity spans to extract entity features and uses the entity features to locate the visual regions that are most relevant to the entities in the text. to summarize, the major contributions of our paper are as follows: ## related work in general, studies about mner are similar in terms of text feature extraction. however, there are differences in research methods when using image information and fusing modal information. the existing work can be classified into the following two categories based on the use of visual information: in fact, not all visual regions within an image are beneficial to improve the accuracy of the model prediction. to address this problem, some researchers proposed a method of first dividing the whole image into multiple visual regions equally, and then extracting the most relevant visual regions to the text in order to filter out irrelevant visual information. the visual information that ends up interacting with the text is actually the local visual information after filtering, even if the input is global visual information. @xcite used the pretraining resnet model to extract visual regions and then added them to the text embedding by a visual attention model. to make full use of text and visual information, @xcite used adaptive co-attentive networks to fuse text embedding and visual regions representation. @xcite proposed a multimodal interaction module to obtain both image-aware word representations and word-aware visual representations, and used textonly entity spans detection as an auxiliary module to mitigate visual bias. @xcite used an external knowledge database to obtain the final multimodal representation by attention-guided visual layer. irrelevant text-image pairs account for a large proportion of the dataset. therefore, @xcite and @xcite used a modified bert encoder to obtain information for inter-modal fusion and then introduced text-image relationship classification as a subtask to determine whether image features were useful. in addition, @xcite proposed a novel uncertainty-aware framework for social media mner. in addition, for the problem of fine-grained semantic correspondence between objects in images and words in the text. @xcite performed an enhanced representation of each word in the text by semantic enhancement and performed crossmodal semantic interaction between text and vision at different visual granularities. @xcite proposed a scene graph driven multimodal multigranularity multitask learning framework. visual objects are considered fine-grained image representations, and for the text with multiple entity types, the visual object regions of the related images can be used to capture different entity information. wu et al. (2020) used mask rcnn for object detection, they embedded top-k objects into vectors to interact with text features through the dense co-attentive module. in addition to this, wang et al. (2020) proposed a multimodal alignment framework that used a contrastive objective, to guide alignment between visual and textual representations. zheng et al. (2020) used an adversarial learning technique that aimed to fuse textual and visual features into a common feature space. for the same problem, zhang et al. (2021) first extracted noun phrases in the text and visual object regions in images and then used gnn to model the relations between them. although many multimodal neural techniques have been proposed to incorporate images into the mner task, the ability of models to exploit multimodal interactions remains unclear. therefore, @xcite provided an in-depth analysis of existing multimodal fusion techniques from different perspectives and described a situation where the use of image information did not always im-prove performance. based on this work, @xcite aligned image features into the text space by using image-text alignment to better utilize the attention mechanism in transformer-based pre-trained textual embeddings. ## method in this section, we first introduce the task definition, and then describe the proposed model in detail. ## experiments in the following section, we conduct experiments on two mner datasets, comparing our entity visual regions positioning (espvr) approach with several unimodal and multimodal approaches. ## conclusion in this paper, we present an entity spans position visual regions (espvr), which obtains the most relevant visual regions for the entities in the text as fine-grained local visual information. the experimental results reveal the superiority of finegrained local visual information acquired through this method. this information proves more advantageous in enhancing the performance of multimodal named entity recognition (mner) compared to using attention-based and object detection-based methods.