Flat Multi-modal Interaction Transformer for Named Entity Recognition (2022)

Abstract: Multi-modal named entity recognition (MNER) aims at identifying entity spans and recognizing their categories in social media posts with the aid of images. However, in dominant MNER approaches, the interaction of different modalities is usually carried out through the alternation of self-attention and cross-attention or over-reliance on the gating machine, which results in imprecise and biased correspondence between fine-grained semantic units of text and image. To address this issue, we propose a Flat Multi-modal Interaction Transformer (FMIT) for MNER. Specifically, we first utilize noun phrases in sentences and general domain words to obtain visual cues. Then, we transform the fine-grained semantic representation of the vision and text into a unified lattice structure and design a novel relative position encoding to match different modalities in Transformer. Meanwhile, we propose to leverage entity boundary detection as an auxiliary task to alleviate visual bias. Experiments show that our methods achieve the new state-of-the-art performance on two benchmark datasets.

## introduction named entity recognition (ner) is a fundamental task in the field of information extraction, which involves determining entity boundaries from free text and classifying them into pre-defined categories, such as person (per), location (loc), and organization (org) @xcite . along with the rapid development of social media, multi-modal deep learning is widely applied in the structured extraction from massive multimedia news and web product information @xcite . as an important research branch of ner, multi-modal named entity recognition (mner) significantly extends the text-based ner by taking the {sentence, image} pair as inputs @xcite @xcite ; lu @xcite . since the visual context associated with text content has been confirmed to help resolve the recognition of ambiguous multisense words and out-of-vocabulary words, mner plays an important role in extracting entities from user-generated content on social media platforms such as twitter @xcite . it has been the core issue in mner to fully exploit the effective visual information and suppress the interference information, which directly affects the model performance. to this end, there are three lines of methods to integrate visual information into ner. (1) the first line is to encode the whole image into a global feature vector (figure 1(a) ) for augmenting each word representation @xcite . (2) the second line is to divide the feature map extracted from the whole image into multiple regions averagely (figure 1(b) ), which is the most dominant method currently. the method guides the word to learn a vision-aware representation through co-attention and gating mechanism @xcite , or uses a transformer framework based on the combination of self-modal and cross-modal attention to interact textual and visual information @xcite . (3) the last line is to use noun phrases to detect the image bounding boxes (figure 1(c) ) and fuse finegrained words and visual objects by graph neural networks (gnn) @xcite . despite the success, the above works may not precisely exploit the fine-grained semantic correspondence between semantic units in an input sentence-image pair. specifically, as shown in figure 1 , we believe that the crux of the issue lies in two aspects. (1) the global clue provided by image (a) and regional feature maps provided by images (b) are both implicit and vague, which are difficult to fit into fine-grained words. previous practice tends to map the visual and textual representations into different spaces and then fuse them adaptively. however, the indirectness of information interaction through cross-modal attention and gating mechanism will lead to asymmetry in information acquisition @xcite . (2) specific visual objects derived from noun phrases are overly targeted, which makes some non-entities embodied in the images identified as entities incorrectly. generally speaking, this kind of explicit information is helpful to identify some words as the correct entity type, such as "the taj mahal". however, the visual objects prominent in the image may easily misidentified as entities, such as "sunset". to handle the aforementioned issues, we propose a novel flat multi-model interaction transformer for mner. the key insight comes from the lattice structure in chinese ner @xcite , where word sequence is used as additional information to enhance the character representation. to fully exploit the available visual information, we use a combination of two visual objects, extracted from the whole image and derived from the noun phrases. we first represent the input sentence and image with a unified flat lattice structure consisting of fine-grained semantic units. each unit corresponds to a word or visual object and its position. meanwhile, inspired by the strategies of position representation, we design an ingenious position encoding for our flat lattice structure @xcite . in detail, we assign two positional indices for a unit: head position and tail position, by which we can correspond the visual object to the associated words span. based on the flat lattice structure, we then resort to the fully-connected self-attention structure and long-distance dependencies modeling capability in transformer @xcite to build bridges in the interaction between self-modal and cross-modal units. finally, we utilize the crf decoder to obtain the predicted labels. to largely eliminate the bias of visual context, we further introduce the entity boundary detection (ebd) as an auxiliary task. we conduct extensive experiments on twitter 2015 and twitter 2017 benchmark datasets. the state-of-the-art performance and efficiency demonstrate the effectiveness of our methods. ## related work multi-modal ner. as an important role in many downstream nlp tasks, including information retrieval @xcite , relation extraction @xcite ) and question answering system @xcite , the text-based ner task has attracted much attention in the journalistic and social fields. deep learning approaches such as cnn, lstm, attention mechanism and pretrained models have achieved significant success in ner, by which we can effectively uncover and combine the character, word and sentence information in text sequence @xcite @xcite . influenced by the extensive applications of multi-modal learning in neural machine translation, visual question answering and emotion detection @xcite @xcite , many researches have focused on constructing multi-modal ner datasets and exploring the methods to guide entity recognition using images. the main idea of these early methods is encoding the text through lstm and the image through pre-trained cnn, then implicitly interacting the information of two modalities @xcite @xcite . recently, @xcite ) leverage bert to model text sequence and creatively design a multi-modal interaction module based on transformer and gating mechanism to perform self-modal and crossmodal information interaction alternately. @xcite further represent the input sentenceimage pair as a unified graph to capture the various semantic relationships and introduce an extended gnn to conduct graph encoding via multi-modal semantic interactions. different from above studies, our approach aims at representing the fine-grained semantic units of the text and image as a unified flat lattice structure. we further design a novel relative position encoding strategy to directly capture the interaction between different modalities in transformer. lattice structure. since word information is potentially useful for character-based chinese ner task, the lattice structure is proposed for injecting the word information into the associated characters. specifically, @xcite first proposes lattice-lstm to explicitly exploit word boundary information, in which the matched lexical words are integrated into characters via a directed acyclic graph. later, to overcome the limitations of the lattice structure so that it can be flexibly exploited, @xcite converts the lattice into graph and designs a collaborative graph network for encoding. @xcite proposes a dynamic crossand self-lattice attention network to model dense interactions over word-character pairs. ## method figure 2 illustrates the overall architecture of our fmit, which contains three main components: (1) unified flat lattice structure for representing the input sentence-image pairs. (2) transformer encoder with relative position encoding method for interacting multi-modal information. (3) training with entity boundary detection as an auxiliary task. task formulation. given a sentence s and its associated image o as input, the goal of mner is to extract a set of entities from s and classify each extracted entity into one of the pre-defined categories. as with most existing work in mner, we formulate the task as a sequence labelling problem. let @xmath0, w 2 , ..., w n ) denote a sequence of input words, where w i with @xmath1, 2, ...n denotes the ith word in the sentence and n represents the length of the sentence, and @xmath2, y 2 , ..., y n ) be the corresponding entity labels for all words, where y 1 ∈ y and y is the pre-defined label set with standard bio schema @xcite . we also use @xmath3, o 2 , ..., o m ) to denote a set of input visual objects of number m. ## conclusion in this paper, we propose a novel flat multi-modal interaction transformer for mner, which exploits flat lattice structure and relative position encoding to directly interact fine-grained semantic units between different modalities. moreover, we put forward entity boundary detection as an auxiliary task to alleviate visual bias. we conduct extensive experiments on two mner datasets, and experimental results demonstrate that our approach outperforms other text-based and multi-modal approaches.