Entity-level Interaction via Heterogeneous Graph for Multimodal Named Entity Recognition (2022)

Abstract: Multimodal Named Entity Recognition (MNER) faces two specific challenges: 1) How to capture useful entity-related visual information. 2) How to alleviate the interference of visual noise. Previous works have gained progress by improving interacting mechanisms or seeking for better visual features. However, existing methods neglect the integrity of entity semantics and conduct cross-modal interaction at token-level, which cuts apart the semantics of entities and makes non-entity tokens easily interfered with by irrelevant visual noise. Thus in this paper, we propose an end-to-end heterogeneous Graph-based Entity-level Interacting model (GEI) for MNER. GEI first utilizes a span detection subtask to obtain entity representations, which serve as the bridge between two modalities. Then, the heterogeneous graph interacting network interacts entity with object nodes to capture entity-related visual information, and fuses it into only entity-associated tokens to rid non-entity tokens of the visual noise. Experiments on two widely used datasets demonstrate the effectiveness of our method. Our code will be available at https://github.com/GangZhao98/GEI.

## introduction multimodal named entity recognition (mner) aims at combining both textual and visual contents to detect and classify named entities from multimodal social media posts (e.g., tweets). different from traditional ner @xcite @xcite that focuses on formal single-modal texts, mner confronts two specific challenges: 1) how to capture useful entityrelated visual information; 2) how to alleviate the interference of visual noise. as shown in figure 1 , the misc entity "oscars" appearing in the tweet may be wrongly recognized as per, since it can refer to both a person name and the movie award. but the trophies in image can help figure out that the "oscars" actually indicates the latter. effectively capturing entity-related information from the image is essential and challenging. though helpful, incorporating images may also interfere the nonentity tokens that have no corresponding visual information, and makes them easily misidentified as entities. effectively alleviating the interference brought by images is also a critical challenge. recent works on mner have gained progress by either improving cross-modal interacting mechanisms @xcite @xcite , or seeking for better visual features @xcite @xcite . however, existing methods neglect the integrity of entity semantics and directly interact all textual tokens with visual features, which we regard as token-level interaction. though straightforward, token-level interaction fails to use integral entity semantics to capture related visual information, and makes non-entity tokens easily interfered with by visual noise. thus in this paper, we propose an end-to-end heterogeneous graph-based entity-level interacting model (gei) for mner. as shown in figure 1 , the key insight of entity-level interaction is obtaining entity representations to query related visual information from object features, which has several benefits: 1) entity representations carry integral entity semantics, which can capture entity-related information effectively. 2) interacting visual features with only entity representations instead of all token representations can protect non-entity tokens from the interference brought by images. in detail, gei first introduces a span detection subtask to obtain entity representations, which serve as the bridge between two modalities. then, a multimodal heterogeneous graph is constructed with token, entity and object nodes, whose semantic relationships are modeled by four kinds of edges. after that, gei interacts entity nodes with object nodes to capture related visual information, and fuses it to token nodes that are connected with entity nodes. finally, a crf layer is employed to decode named entities from object-aware token representations. overall, our contributions are as follows: ## methodology figure 2 shows the architecture of gei, which contains the following components: entity representation extractor (ere), object feature encoder (ofe), heterogeneous graph interacting network (hgin), crf decoding modules. ## conclusion in this paper, we propose an heterogeneous graphbased entity-level interacting model (gei) for mner. gei interacts entity representations with visual objects to capture useful entity-related visual information, and excludes non-entity tokens from the interaction to rid them of the visual noise. experiments on two public mner datasets demonstrate the effectiveness of our method.