A Span-based Multimodal Variational Autoencoder for Semi-supervised Multimodal Named Entity Recognition (2022)

Abstract: Multimodal named entity recognition (MNER) on social media is a challenging task which aims to extract named entities in free text and incorporate images to classify them into user-defined types. However, the annotation for named entities on social media demands a mount of human efforts. The existing semi-supervised named entity recognition methods focus on the text modal and are utilized to reduce labeling costs in traditional NER. However, the previous methods are not efficient for semi-supervised MNER. Because the MNER task is defined to combine the text information with image one and needs to consider the mismatch between the posted text and image. To fuse the text and image features for MNER effectively under semi-supervised setting, we propose a novel span-based multimodal variational autoencoder (SMVAE) model for semi-supervised MNER. The proposed method exploits modal-specific VAEs to model text and image latent features, and utilizes product-of-experts to acquire multimodal features. In our approach, the implicit relations between labels and multimodal features are modeled by multimodal VAE. Thus, the useful information of unlabeled data can be exploited in our method under semi-supervised setting. Experimental results on two benchmark datasets demonstrate that our approach not only outperforms baselines under supervised setting, but also improves MNER performance with less labeled data than existing semi-supervised methods.

## introduction multimodal named entity recognition (mner) has become a fundamental task to extract named entities from unstructured texts and images on social media @xcite . compared with traditional named entity recognition (ner), mner on social media poses the unique challenge that bridging the semantic gap between the posted texts and images is critical to extracting named entities. therefore, the existing mner models utilized cross-modal attention module to fuse the text and image features @xcite . besides, @xcite proposed crossmodal matching and alignment modules to make the representations of the texts and images more consistent. and to retain the useful image information for mner, @xcite exploited the two-stage model to refine uncertain labels by fusing the features from the texts and images. to reduce labeling costs in mner, semisupervised learning is widely utilized to exploit the useful information of unlabeled data in text modal. unlike the supervised setting with adequate labeled data, there are small amount of labeled data and large amount of unlabeled one in semi-supervised setting as shown in figure 1 . intuitive semi-supervised learning methods including: self-training (st) @xcite and entropy minimization (em) @xcite use the pseudo labels generated by ner models for unlabeled data to train models. the ner task can be modeled as the sequence labeling problem, and seqvat @xcite ) was proposed to combine virtual adversarial training (vat) @xcite with conditional random field (crf) @xcite for semisupervised sequence labeling. however, the existing semi-supervised ner methods are not efficient for mner under semi-supervised setting. because the previous methods are only focused on the text modal and mner needs considering the semantic correlation between the texts and images of both labeled and unlabeled data. to overcome the above disadvantages of the existing methods, we propose the span-based multimodal variational autoencoder (smvae) foot_0 for semi-supervised multimodal named entity recognition. the previous mner models fused the sentence-level features and image ones for predicting sequence labels and had the difficulty to model mulitmodal features of unlabeled data under semi-supervised setting. because the semantic correlation between sentences and images should be focused on the specific tokens. therefore, the proposed method splits the texts into span-level tokens, and combines the span-level features of texts with image features for predicting labels of all spans in each text. smvae utilizes modal-specific vaes to model latent representations of images and span-level texts respectively, and acquires the multimodal features by applying product-of-experts (poe) @xcite on the latent representations of two modals. the prediction probabilities and multimodal features are exploited to reconstruct the input features for implicitly modeling the correlation between span label and multimodal features. therefore, the useful information of unlabeled multimodal data can be exploited to improve the performance on mner. the contributions of this manuscript can be summarized as follows: the above studies are under the supervised setting, and we focus on the semi-supervised mner to reduce the labeling costs. unlike the supervised learning with adequate labeled data, the semisupervised learning is focused on utilizing the useful information of unlabeled data. ## conclusion in this manuscript, we propose the semi-supervised multimodal named entity recognition (mner) task and pose the critical challenge of it compared with traditional semi-supervised named entity recognition (ner). further more, we analyze the disadvantage of the existing semi-supervised ner methods that are not sufficient to multimodal data. therefore, we propose the span-based multimodal variational autoencoder to tackle semi-supervised mner. the proposed model exploits multimodal vae including two modal-specific vaes to learn the multimodal latent representations and jointly model the implicit correlation between labels and multimodal features to make use of unlabeled multimodal data effectively. the experimental results verify that our approach not only outperforms supervised learning baselines, but also gains superior the proposed model is limited to the length of input sentence because it needs to predict the type of all candidate spans during inference time. and the number of spans is proportional to the length of the sentence. therefore, the inference time is increased with the length of sentence. besides, our model has poor scalability to process more than one image, and the posted twitter message may contain more than one image. therefore, the future mner model should be able to process the text with more images.