Can images help recognize entities? A study of the role of images for Multimodal NER (2021)

Abstract: Multimodal named entity recognition (MNER) requires to bridge the gap between language understanding and visual context. While many multimodal neural techniques have been proposed to incorporate images into the MNER task, the model’s ability to leverage multimodal interactions remains poorly understood. In this work, we conduct in-depth analyses of existing multimodal fusion techniques from different perspectives and describe the scenarios where adding information from the image does not always boost performance. We also study the use of captions as a way to enrich the context for MNER. Experiments on three datasets from popular social platforms expose the bottleneck of existing multimodal models and the situations where using captions is beneficial.

## introduction traditional named entity recognition(ner) on social platforms has been studied mainly using text @xcite @xcite . with the increase in popularity of platforms like twitter, instagram, and snapchat, where users can create multimedia posts, images and text are frequently used together. this trend enables the use of images to improve current ner systems. given a pair of text and image, the multimodal ner(mner) task's goal is to identify and classify named entities in the text. current work in mner mainly focuses on the alignment between words and image regions and the fusion of textual information and visual context. recent successful architectures for mner mainly rely on attention mechanisms combined with different fusion techniques @xcite @xcite @xcite . however, we observe that incorporating images in mner is not trivial. there are two critical but often neglected aspects in the prior art: the first is the assumption that images and text are aligned. as discussed in @xcite , mner models are sensitive to errors when images are added to convey irony or abstract concepts instead of illustrating what is in the text. the second aspect is the semantic density @xcite ) of the visual context information. named entities are too specific since they are instances of general entities. on the contrary, the information provided by automatic image processing models is too general. for example, mner models frequently use the output of convolutional neural networks (cnns) trained on image classification tasks as the image's representation. linking such specific entities with more general information from images demands a giant leap for the models to fill in the semantic gap. thus, we speculate that mner would benefit more from semantically dense learning signals like image captions instead of image classification. in this work, we take a step further towards understanding the behavior of multimodal models for the mner task. we first analyze the existing multimodal fusion techniques with both unimodal and multimodal transformer-based sequence labeling models. we investigate three different ways to represent images: global image features from image classification algorithms, regional image features from object detection algorithms, and image captions. by conducting experiments that emphasize the contribution of different image representations, we demonstrate the effectiveness of using captions to represent images in the mner task. to summarize, we make the following contributions: ## related work as more and more social media data contains text and images, the multimodal ner task has attracted increased research interest. most of the recent methods for this task have been introduced using attention-guided models to extract visual information related to the named entities @xcite , @xcite , @xcite ). despite showing promising results, these methods have two main limitations: ignoring mapping relations between visual objects and named entities, and the distribution differences between images and text. this can lead to incorrect visual context clues being extracted when the images and texts are not relevant, which is not uncommon in social media data. @xcite addresses these problems by proposing a neural network to better exploit visual and textual information. however, their results revealed that their model remains to extract entities incorrectly when visual objects cannot reveal the label semantics of entities. to address the problem of semantic disparity between different modalities, @xcite chooses to transform object labels into word embeddings. their dense co-attention module introduced can take the interand intra-connections between visual objects and textual entities into account, which helps extract entities more precisely. sun et al. (2020) introduced a pre-trained multimodal language model based on relationship inference and visual attention (riva) for tweets, and a gated visual context based on text-image relation. sun et al. (2021) proposed a text-image relation propagation-based multimodal bert model (rpbert) to reduce the interference from irrelevant images, which effectively resolved failed cases mentioned in previous work (lu et al. (2018); arshad et al. (2019); yu et al. (2020)). recent advances in large pre-trained models @xcite have led to significant performance gains in many multimodal tasks (e.g, visual question answering). however, most of these techniques are pre-trained on image captioning or visual question answering datasets where multimodal interactions are required. applying these techniques to the mner task may not result in a good performance. there some significant differences between the those task and the mner task. we argue that in the mner we can have three possible scenarios: 1) images and text are not related and thus we don't expect the image information to help; 2) images and text are related and adding image information is helpful and 3) images and text are related but adding the image information does not help the mner task. therefore, before our work, it was not clear whether adding image captions would contribute to the performance. in contrast, in the visual question answering or visual commonsense reasoning task, the images are relevant and needed in order to answer the questions, and therefore, using captions is more likely to yield good results. these differences in the scenarios warrant further investigation. ## methodology in this work, we study how to represent images and how to fuse different modalities in mner. we explore three different ways to represent images in different levels of semantic density. then we adapt a unimodal transformer bert @xcite and a multimodal transformer visualbert @xcite as base models. to fuse textual information with visual information, we investigate four different approaches for multimodal fusion. ## datasets in this work, we use three datasets for evaluation in our experiments: twitter-2015 from @xcite , twitter-2017 from @xcite , and a snapchat dataset collected from snapchat posts for this study. due to the weak relatedness between named entities and images in two twitter datasets, multimodal architectures are not superior to unimodal ones. therefore, we create a new multimodal dataset for further study based on snapchat posts since text and images from snapchat are highly related. the snapchat dataset is composed of text and image pairs exclusively extracted from snaps submitted to the public and crowd-sourced stories (aka "our stories") and is labeled by expert human annotators (entity types: per, loc, org, misc). examples of such public crowd-sourced stories are the "airpods everywhere story" or the "thanksgiving story", which comprise snaps that are aggregated for various public themes and events. all snaps were posted between january and march 2020. ## experiment and analysis this section empirically diagnoses the multimodal interactions from the relatedness between different modalities and the semantic density from images. then we explore the role of images in the mner task in both low-context and low-resource scenarios to expose the situations where incorporating images is beneficial. we also diagnose the model's ability to disambiguate and generalize. ## conclusion in this work, we systematically analyze the behavior of existing multimodal models for the mner task. with extensive experiments and analysis based on bert and visualbert, we show that existing fusion techniques can only bring marginal, if any, gains to strong text-only models. we believe this is due to the weak relatedness between named entities and images and the sparse semantic density of visual signals. we then examine the use of image captions as a more semantically dense signal and find promising results, concluding it is always worth adopting this approach as it usually outper-forms existing alternatives and is never detrimental to performance. finally, we investigate multiple scenarios and share our findings of when images can help the mner task.