P - MNER : Cross Modal Correction Fusion Network with Prompt Learning for Multimodal Named Entity Recognitiong (2023)

Abstract: “Multimodal Named Entity Recognition (MNER) is a challenging task in social mediadue to the combination of text and image features. Previous MNER work has focused onpredicting entity information after fusing visual and text features. However, pre-traininglanguage models have already acquired vast amounts of knowledge during their pre-training process. To leverage this knowledge, we propose a prompt network for MNERtasks (P-MNER).To minimize the noise generated by irrelevant areas in the image, wedesign a visual feature extraction model (FRR) based on FasterRCNN and ResNet, whichuses fine-grained visual features to assist MNER tasks. Moreover, we introduce a textcorrection fusion module (TCFM) into the model to address visual bias during modalfusion. We employ the idea of a residual network to modify the fused features using theoriginal text features. Our experiments on two benchmark datasets demonstrate that ourproposed model outperforms existing MNER methods. P-MNER’s ability to leveragepre-training knowledge from language models, incorporate fine-grained visual features,and correct for visual bias, makes it a promising approach for multimodal named entityrecognition in social media posts.”

## introduction with the rapid development of the internet, social media platforms have experienced an exponential growth of content. these platforms offer a wealth of user-generated posts that provide valuable insights into the events, opinions, and preferences of both individuals and groups. named entity recognition (ner) is a crucial task in which entities contained in textual data are detected and mapped to predefined entity types, such as location (loc), person (per), organization (org), and miscellaneous (misc). incorporating visual information from posts has been shown to significantly enhance the accuracy of entity prediction from social media content. for instance, as illustrated in fig. 1 , the sentence "alban got rikard a snowball in the snow" can be easily resolved by leveraging the visual cues in the accompanying image, allowing us to identify "rikard" as an animal. however, relying solely on textual data to predict entities may lead to erroneous predictions, such as identifying "rikard" as a name. with the continuous evolution of deep learning models, several multi-modal named entity recognition (ner) models have been proposed to enhance the prediction performance of entities by incorporating visual information. these models employ techniques such as crossattention @xcite , adversarial learning @xcite , and graph fusion @xcite . however, previous methods fused text features with visual features and directly fed them into a neural network model for prediction. this approach overlooks the wealth of information embedded in the pre-training language model itself. to overcome this limitation, we propose the use of prompt learning @xcite to process the fused features, followed by final training. the presence of irrelevant content in an image may negatively impact the performance of named entity recognition (ner) models. as illustrated in fig. 1 , regions a and b in an image may aid in identifying entities in a sentence, while regions c and d may not contribute to model prediction. in previous multi-modal ner (mner) tasks, however, all visual regions were involved in cross-modal fusion. to address this issue, we propose a novel model(frr), which utilizes visual objects in images for modal fusion. this approach effectively eliminates extraneous image features that are irrelevant to the corresponding text. in this paper, we present a new transformer-based @xcite text correction fusion module (tcfm) to address the issue of cross-modal visual bias in the named entity recognition (ner) task. inspired by the residual network, the tcfm continuously integrates the original text features with the fusion features to iteratively correct the fusion features. this approach effectively alleviates the problem of visual bias and enhances the performance of the ner task. in order to showcase the effectiveness of our proposed approach, we conducted a comprehensive set of experiments on two publicly available datasets: twitter-2015 and twitter-2017. the obtained experimental results unequivocally demonstrate that our method outperforms the existing mner algorithm in terms of performance. the significant contributions of our work can be summarized as follows: ## related work named entity recognition (ner) has emerged as a crucial component in a plethora of downstream natural language processing (nlp) applications, including but not limited to affective c c l 2 0 2 3 analysis @xcite , relationship extraction @xcite , and knowledge graph @xcite construction. with the advent of neural network models, the use of bi-lstm @xcite or convolutional neural network (cnn) @xcite as encoders, and softmax @xcite , rnn @xcite , or conditional random field (crf) @xcite as decoders has gained popularity in the ner community. for instance, huang et al. @xcite utilized bilstm and crf as the encoder and decoder, respectively, for ner tasks. similarly, chiu and nichols et al. @xcite proposed a cnn-based encoder with crf as the decoder to accomplish the final prediction. social media posts are characterized by their brevity and high levels of noise, which often lead to suboptimal performance of conventional ner methods when applied to such data. to address this challenge, several recent studies have proposed novel approaches for cross-modal fusion in the context of ner. for instance, sun et al. @xcite introduced a novel image-text fusion approach for ner tasks, while zhang et al. @xcite employed bilstm to combine visual and textual features in multimodal social posts. similarly, zheng et al. @xcite proposed an adversarial learning approach to tackle the issue of semantic gap in multimodal ner tasks. lu et al. @xcite integrated an attention mechanism into the modal fusion process and introduced a visual gate to filter out noise in the image. lastly, yu et al. @xcite designed a multimodal transformer model for the mner task and introduced an entity span module to facilitate the final prediction. in order to utilize the vast amount of knowledge encoded in pre-trained language models, wang et al. @xcite proposed a prompt-based method, namely promptmner, which extracts visual features and subsequently fuses them with input text for enhanced performance. ## methods the objective of this study is to predict a tag sequence @xmath0, l 2 , ..., l n ). given a sentence y and an associated picture v as input. here, belongs to a predefined set of tags for the bio2 tagging pattern. fig. 2 depicts the overall architecture of our proposed model, which comprises four modules: visual feature extraction, text feature extraction, modal fusion, and prompt learning. the visual feature extraction module takes the objects in the picture as input. in the text feature extraction module, we leverage bert to extract features by processing the wrapped text. we also introduce a text correction fusion module to obtain more precise fusion features. in the prompt learning module, we utilize the bert masked language model for prompt learning. ## conclution in this paper, we have introduced the p-mner architecture, which has been specifically designed to tackle named entity recognition (mner) tasks. our proposed architecture leverages the power of prompt learning to process modal fusion features, thereby enabling the model to fully exploit the wealth of knowledge that pre-trained language models have to offer during training. we also proposed a fine-grained visual object feature extraction module (frr) to address the issue of noise caused by irrelevant visual areas. this module aids in the mner task by extracting only the relevant visual information, thus improving the accuracy of the model. to further address the issue of visual bias across modes, we proposed a new text correction fusion module. this module aligns the fusion features with text features to reduce visual bias and improve the model's performance. experimental results on benchmark datasets demonstrate that our p-mner model outperforms state-of-the-art approaches. our model's superior performance is attributed to its ability to effectively utilize pre-trained language models and its innovative feature extraction and fusion modules. overall, our proposed p-mner architecture offers a promising solution for named entity recognition tasks, and we believe that our approach can be extended to other natural language processing tasks to improve their performance.