RIVA : A Pre-trained Tweet Multimodal Model Based on Text-image Relation for Multimodal NER (2020)

Abstract: Multimodal named entity recognition (MNER) for tweets has received increasing attention recently. Most of the multimodal methods used attention mechanisms to capture the text-related visual information. However, unrelated or weakly related text-image pairs account for a large proportion in tweets. Visual clues unrelated to the text would incur uncertain or even negative effects for multimodal model learning. In this paper, we propose a novel pre-trained multimodal model based on Relationship Inference and Visual Attention (RIVA) for tweets. The RIVA model controls the attention-based visual clues with a gate regarding the role of image to the semantics of text. We use a teacher-student semi-supervised paradigm to leverage a large unlabeled multimodal tweet corpus with a labeled data set for text-image relation classification. In the multimodal NER task, the experimental results show the significance of text-related visual features for the visual-linguistic model and our approach achieves SOTA performance on the MNER datasets.

## introduction social media such as twitter has become a part of many people's everyday lives. it is an important source for various applications such as open event extraction @xcite , social knowledge graph (hosseini, 2019). named entity recognition (ner) for tweets is the first key task of these applications. ner has achieved excellent performance on news articles; however, the recognition results in tweets are still not satisfactory @xcite . one of the reasons is that tweets are short messages and the context for inference is insufficient. recent works on tweets based on multimodal learning have been increasing @xcite @xcite . the researchers attempted to improve the performance of ner in tweets with the aid of visual clues. most of the multimodal ner (mner) methods used attention weights to extract visual clues related to the nes @xcite @xcite . the visual attention-based models always assume that the images in tweets are related to the texts, such as words in the text are represented in the image, e.g., figure 1 (a) shows a successful visual attention example from @xcite . in fact, texts and images in tweets have diverse relations. @xcite ) categorized text-image relationship according to whether "image adds to the tweet meaning". "image adds to the tweet meaning" represents the role of image to the semantics of text in tweets. the type of "image does not add to the tweet meaning" account for approximately 56% in the vempala's bloomberg dataset. in addition, we test a classifier regarding whether "image adds to the tweet meaning" on a large randomly collected corpus, twitter100k @xcite , the proportion of the classified negatives is approximately 60%. figure 1( @xcite shows a failure visual attention example of "image does not add to the tweet meaning". therefore, visual features represented by attention weights do not always have positive effects on training visual-linguistic models. moreover, the unrelated visual clues may increase the possibility of wrong connection to the text inference. in this paper, we consider inferring the text-image relationship to address the problem of inappropriate visual clues fused in the multimodal model. the text-image relationship is defined as whether the image's content can contribute additional information beyond the text. the contributions of this paper can be summarized as follows: ## related work multimodal ner for social media posts has been investigated in recent years. @xcite proposed a modality-attention module at the input of the ner network. the module computed a weighted modal combination of word embeddings, character embeddings, and visual features. @xcite presented a visual attention model to find the image regions that were related to the content of the text. the attention weights of the image regions were computed by a linear projection of the sum of the text query vector and regional visual representations. the extracted visual context features were incorporated to the word-level outputs of the bilstm model. @xcite designed an adaptive co-attention network (acn) layer, which was between the lstm and crf layers. the acn contained a gated multimodal fusion module to learn a fusion vector of the visual and linguistic features. the author designed a filtration gate to decide whether the fusion feature was helpful to improve the tagging accuracy of each token. the output score of the filtration gate was computed by a sigmoid activation function. @xcite also presented a gated multimodal fusion representation for each token. the gated fusion is a weighted sum of visual attention feature and token alignment feature. the visual attention feature was calculated by the weighted sum of vgg-19 @xcite visual features and the weights were the additive attention scores between a word query and image features. overall, the problem of attention-guided visual feature is that the incorrect visual context clues could be extracted when the images and texts are not relevant. they pointed out that the unrelated images caused the wrong attention and prediction errors, and showed the failed examples. although @xcite attempted to use a sigmoid-based gate to filter the unrelated visual feature, the improvement was marginal when we tested the available source code by the authors. the pre-trained models such as elmo @xcite , bert @xcite have achieved great success in natural language processing (nlp). these models are trained on large unlabeled corpus by self-supervised learning tasks. the latest pre-trained multimodal models, visual-linguistic bert, were presented in @xcite @xcite . the authors extended the popular bert architecture to multimodal models for vision and language. li et al. (2019) pre-trained visual-bert on the coco image caption dataset @xcite . the visualbert consisted of a stack of transformer layers that implicitly aligned elements of the text and regions in the image with selfattention. @xcite and @xcite trained the models on a larger caption dataset, conceptual captions @xcite ) consisting of 3.3 million images annotated with captions. @xcite presented co-attentional transformer layers in vilbert to learn interactively between visual and linguistic features. two pre-training tasks: masked learning and alignment prediction, were applied. the alignment task was a binary classification which defined the pairs in the caption dataset as positives and the pairs generated by replacing the image or text with each other as negatives. @xcite designed four types of embeddings in vl-bert and input the sum to the bert architecture. the attention module took both visual and linguistic features as input. the input element was either word embeddings or fast r-cnn @xcite features for regions of interest (rois) in an image. @xcite performed the text-image relation prediction and the data setting was the same as the alignment task in @xcite . the ablation study of the training tasks in @xcite showed that the task of text-image relation would decrease the accuracy of all downstream tasks, and the authors guessed the reason was that the unmatched image and caption pairs were introduced. ## conclusion this paper concerns the problem of visual attention features in multimodal learning when images are unrelated to texts, typically in tweets. the text-unrelated visual features would incur negative rewards in multimodal ner. in the paper, we propose a pre-trained multimodal model based on text-image relationship inference. the relation of whether "image adds to the tweet meaning" is employed and the classification achieves excellent performance in our model. the riva model is trained on a large multimodal corpus of tweets under a multitask framework of text-image relation classification and next word prediction. in the experiments, we show the quantitative results of the impact of text-unrelated visual attention features on ner task in the ablation study, -1.2% on the fudan univ. dataset and -1.5% on the snap res. dataset. we illustrate the failed visual attention examples that can be resolved by the riva model. the performance of the riva model is better than other visual-linguistic models and sota performance of mner is achieved in this paper.