ITA : Image-Text Alignments for Multi-Modal Named Entity Recognition (2022)

Abstract: Recently, Multi-modal Named Entity Recognition (MNER) has attracted a lot of attention. Most of the work utilizes image information through region-level visual representations obtained from a pretrained object detector and relies on an attention mechanism to model the interactions between image and text representations. However, it is difficult to model such interactions as image and text representations are trained separately on the data of their respective modality and are not aligned in the same space. As text representations take the most important role in MNER, in this paper, we propose I mage- t ext A lignments (ITA) to align image features into the textual space, so that the attention mechanism in transformer-based pretrained textual embeddings can be better utilized. ITA first aligns the image into regional object tags, image-level captions and optical characters as visual contexts, concatenates them with the input texts as a new cross-modal input, and then feeds it into a pretrained textual embedding model. This makes it easier for the attention module of a pretrained textual embedding model to model the interaction between the two modalities since they are both represented in the textual space. ITA further aligns the output distributions predicted from the cross-modal input and textual input views so that the MNER model can be more practical in dealing with text-only inputs and robust to noises from images. In our experiments, we show that ITA models can achieve state-of-the-art accuracy on multi-modal Named Entity Recognition datasets, even without image information.

## approaches we consider the ner task as a sequence labeling problem. given a sentence @xmath0, • • • , w n } with n tokens and its corresponding image i, an sequence labeling model aims to predict a label sequence @xmath1, • • • , y n } at each position. in our framework, we focus on incorporating visual information to improve the representations of the input tokens by aligning visual and textual information effectively. we use a visual context generator to convert the image i into texts forming visual contexts w ′ = {w ′ 1 , • • • , w ′ m } with m tokens. we then concatenate the input text and visual contexts as a cross-modal text+image (i+t) input view instead of the text (t) input view. we feed the i+t input into a pretrained textual embeddings model to get stronger token representations of the input sentence. then the token representations are fed into a linear-chain crf layer to get the label sequence y. to further improve the model accuracy of both input views, we use the cross-view alignment module to align the output distributions of i+t and t transformer-based embeddings y n ... y 3 y 2 y 1 input texts visual contexts : rt @whitesox : chris sale has his eyes on the prize . : smiling man white jersey ... [x] a baseball player is standing on a field ... [x] "but | don't need the biggest contract ... cross-view alignment � � (�|� ) � � (�|� , � ′) y n ... y 3 y 2 y 1 crf layer � � ′ -a baseball player wearing a white uniform and a hat . -a baseball player is standing on a field . ... image captions smiling man white jersey white writing ... object tags ga la "but | don't need the biggest contract. t want that big shiny [world series] ... ## experiments we conduct experiments on two mner datasets. to show the effectiveness of our approaches, we use two embedding settings and compare our approaches with previous multi-modal approaches.