Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer (2020)

Abstract: In this paper, we study Multimodal Named Entity Recognition (MNER) for social media posts. Existing approaches for MNER mainly suffer from two drawbacks: (1) despite generating word-aware visual representations, their word representations are insensitive to the visual context; (2) most of them ignore the bias brought by the visual context. To tackle the first issue, we propose a multimodal interaction module to obtain both image-aware word representations and word-aware visual representations. To alleviate the visual bias, we further propose to leverage purely text-based entity span detection as an auxiliary module, and design a Unified Multimodal Transformer to guide the final predictions with the entity span predictions. Experiments show that our unified approach achieves the new state-of-the-art performance on two benchmark datasets.

## introduction recent years have witnessed the explosive growth of user-generated contents on social media platforms such as twitter. while empowering users with rich information, the flourish of social media also solicits the emerging need of automatically extracting important information from these massive unstructured contents. as a crucial component of many information extraction tasks, named entity recognition (ner) aims to discover named entities in free text and classify them into pre-defined types, such as person (per), location (loc) and organization (org). given its importance, ner has attracted much attention in the research community @xcite . although many methods coupled with either discrete shallow features @xcite @xcite or continuous deep features @xcite hovy, * corresponding author. 2016) have shown success in identifying entities in formal newswire text, most of them perform poorly on informal social media text (e.g., tweets) due to its short length and noisiness. to adapt existing ner models to social media, various methods have been proposed to incorporate many tweet-specific features @xcite @xcite . more recently, as social media posts become increasingly multimodal, several studies proposed to exploit useful visual information to improve the performance of ner @xcite @xcite . in this work, following the recent trend, we focus on multimodal named entity recognition (mner) for social media posts, where the goal is to detect named entities and identify their entity types given a {sentence, image} pair. for example, in fig. 1 .a, it is expected to recognize that kevin durant, oracle arena, and jordan belong to the category of person names (i.e., per), place names (i.e., loc), and other names (i.e., misc), respectively. while previous work has shown success of fusing visual information into ner @xcite @xcite , they still suffer from several limitations: (1) the first obstacle lies in the non-contextualized word representations, where each word is represented by the same vector, regardless of the context it occurs in. however, the meanings of many polysemous entities in social media posts often rely on its context words. take fig. 1 .a as an example, without the context words wearing off, it is hard to figure out whether jordan refers to a shoe brand or a person. (2) although most existing methods focus on modeling inter-modal interactions to obtain word-aware visual representations, the word representations in their final hidden layer are still based on the textual context, which are insensitive to the visual context. intuitively, the associated image often provides more context to resolve polysemous entities, and should contribute to the final word representations (e.g., in fig. 1 .b, the image can supervise the final word representations of kian and david to be closer to persons than animals). (3) most previous approaches largely ignore the bias of incorporating visual information. actually, in most social media posts, the associated image tends to highlight only one or two entities in the sentence, without mentioning the other entities. in these cases, directly integrating visual information will inevitably lead the model to better recognize entities highlighted by images, but fail to identify the other entities (e.g., oracle arena and king of the jungle in fig. 1 ). to address these limitations, we resort to existing pre-trained contextualized word representations, and propose a unified multimodal architecture based on transformer @xcite , which can effectively capture inter-modality interactions and alleviate the visual bias. specifically, we first adopt a recently pre-trained contextualized representation model @xcite as our sentence encoder, whose multi-head self-attention mechanism can guide each word to capture the semantic and syntactic dependency upon its context. second, to better capture the implicit alignments between words and images, we propose a multimodal interaction (mmi) module, which essentially couples the standard transformer layer with cross-modal attention mechanism to produce an image-aware word representation and a wordaware visual representation for each input word, respectively. finally, to largely eliminate the bias of the visual context, we propose to leverage textbased entity span detection as an auxiliary task, and design a unified neural architecture based on transformer. in particular, a conversion matrix is designed to construct the correspondence between the auxiliary and the main tasks, so that the entity span information can be fully utilized to guide the final mner predictions. experimental results show that our unified multimodal transformer (umt) brings consistent performance gains over several highly competitive unimodal and multimodal methods, and outperforms the state-of-the-art by a relative improvement of 3.7% and 3.8% on two benchmarks, respectively. the main contributions of this paper can be summarized as follows: ## methodology in this section, we first formulate the mner task, and give an overview of our method. we then delve into the details of each component in our model. task formulation: given a sentence s and its associated image v as input, the goal of mner is to extract a set of entities from s, and classify each extracted entity into one of the pre-defined types. as with most existing work on mner, we formulate the task as a sequence labeling problem. let @xmath0, s 2 , . . . , s n ) denote a sequence of input words, and @xmath1, y 2 , . . . , y n ) be the corresponding label sequence, where y @xmath2, 1999). ## experiments we conduct experiments on two multimodal ner datasets, comparing our unified multimodal transformer (umt) with a number of unimodal and multimodal approaches. ## related work as a crucial component of many information extraction tasks including entity linking @xcite , opinion mining @xcite , and event detection @xcite , named entity recognition (ner) has attracted much attention in the research community in the past two decades @xcite . methods for ner: in the literature, various supervised learning approaches have been proposed for ner. traditional approaches typically focus on designing various effective ner features, followed by feeding them to different linear classifiers such as maximum entropy, conditional random fields (crfs), and support vector machines @xcite @xcite @xcite @xcite . to reduce the feature engineering efforts, a number of recent studies proposed to couple different neural network architectures with a crf layer @xcite for word-level predictions, including convolutional neural networks @xcite , recurrent neural networks @xcite , and their hierarchical combinations @xcite . these neural approaches have been shown to achieve the state-of-the-art performance on different benchmark datasets based on formal text @xcite . however, when applying these approaches to social media text, most of them fail to achieve satisfactory results. to address this issue, many studies proposed to exploit external resources (e.g., shallow parser, freebase dictionary, and orthographic characteristics) to incorporate a set of tweet-specific features into both traditional approaches @xcite @xcite and recent neural approaches @xcite , which can obtain much better performance on social media text. methods for multimodal ner (mner): as multimodal data become increasingly popular on social media platforms, several recent studies focus on the mner task, where the goal is to leverage the associate images to better identify the named entities contained in the text. specifically, @xcite proposed a multimodal ner network with modality attention to fuse the textual and visual information. to model the inter-modal interactions and filter out the noise in the visual context, @xcite and @xcite respectively proposed an adaptive co-attention network and a gated visual attention mechanism for mner. in this work, we follow this line of work. but different from them, we aim to propose an effective multimodal method based on the recent transformer architecture @xcite . to the best of our knowledge, this is the first work to apply transformer to the task of mner. ## conclusion in this paper, we first presented a multimodal transformer architecture for the task of mner, which captures the inter-modal interactions with a multimodal interaction module. moreover, to alleviate the bias of the visual context, we further proposed a unified multimodal transformer (umt), which incorporates an entity span detection module to guide the final predictions for mner. experimental results show that our umt approach can consistently achieve the best performance on two benchmark datasets. there are several future directions for this work. on the one hand, despite bringing performance improvements over existing mner methods, our umt approach still fails to perform well on social media posts with unmatched text and images, as analyzed in section 3.5. therefore, our next step is to enhance umt so as to dynamically filter out the potential noise from images. on the other hand, since the size of existing mner datasets is relatively small, we plan to leverage the large amount of unlabeled social media posts in different platforms, and propose an effective framework to combine them with the small amount of annotated data to obtain a more robust mner model.