Grounded Multimodal Named Entity Recognition on Social Media (2023)

Abstract: In recent years, Multimodal Named Entity Recognition (MNER) on social media has attracted considerable attention. However, existing MNER studies only extract entity-type pairs in text, which is useless for multimodal knowledge graph construction and insufficient for entity disambiguation. To solve these issues, in this work, we introduce a Grounded Multimodal Named Entity Recognition (GMNER) task. Given a text-image social post, GMNER aims to identify the named entities in text, their entity types, and their bounding box groundings in image (i.e. visual regions). To tackle the GMNER task, we construct a Twitter dataset based on two existing MNER datasets. Moreover, we extend four well-known MNER methods to establish a number of baseline systems and further propose a Hierarchical Index generation framework named H-Index, which generates the entity-type-region triples in a hierarchical manner with a sequence-to-sequence model. Experiment results on our annotated dataset demonstrate the superiority of our H-Index framework over baseline systems on the GMNER task.

## introduction fueled by the rise of phones and tablets with camera functions, user posts on social media platforms such as twitter are increasingly multimodal, e.g., containing images in addition to text. the explosive growth of multimodal posts is far beyond humans' capability to digest them. hence, it presents a pressing need for automatically extracting important information such as entities and relations from the large amount of multimodal posts, which is crucial for structured knowledge graph construction to help people efficiently understand massive ## task formulation given a multimodal input containing a piece of text with n words @xmath0, . . . , s n ) and an accompanying image v, the goal of our grounded multimodal named entity recognition (gmner) task is to extract a set of multimodal entity triples: ## dataset since there was no available corpus for the gm-ner task, we construct a twitter dataset as follows. data collection. our dataset is built on two benchmark mner datasets, i.e., twitter-15 @xcite and twitter-17 @xcite , which have already annotated all the entities and their types for each multimodal tweet. to alleviate the annotation difficulty, we filter samples with missing images or with more than 3 entities belonging to the same type, and then merge the remaining 12k+ samples as our raw dataset for annotation. ## methodology in this section, we present the details of our proposed hierarchical index generation (h-index) framework. overview. as illustrated in fig. 3 , h-index formulates the gmner task as an index generation problem, which resorts to a pre-trained sequenceto-sequence model bart @xcite to encode the textual and visual inputs, followed by decoding the indexes of entities, types, and groundable or ungroundable indicators. for entities with groundable indicators, another output layer is added to predict the distribution over visual regions for entity grounding. ## conclusion in this paper, we introduced a new task named grounded multimodal named entity recognition (gmner), aiming to identify the named entities, their entity types, and their grounded bounding boxes in a text-image social post. moreover, we constructed a new twitter dataset for the task, and then extended four previous mner methods to benchmark the task. we further proposed a hierarchical index generation framework (h-index), which generates the entity-type-region triples in a hierarchical manner. experimental results demonstrate the effectiveness of our h-index framework.