Different Data, Different Modalities! Reinforced Data Splitting for Effective Multimodal Information Extraction from Social Media Posts (2022)

Abstract: Recently, multimodal information extraction from social media posts has gained increasing attention in the natural language processing community. Despite their success, current approaches overestimate the significance of images. In this paper, we argue that different social media posts should consider different modalities for multimodal information extraction. Multimodal models cannot always outperform unimodal models. Some posts are more suitable for the multimodal model, while others are more suitable for the unimodal model. Therefore, we propose a general data splitting strategy to divide the social media posts into two sets so that these two sets can achieve better performance under the information extraction models of the corresponding modalities. Specifically, for an information extraction task, we first propose a data discriminator that divides social media posts into a multimodal and a unimodal set. Then we feed these sets into the corresponding models. Finally, we combine the results of these two models to obtain the final extraction results. Due to the lack of explicit knowledge, we use reinforcement learning to train the data discriminator. Experiments on two different multimodal information extraction tasks demonstrate the effectiveness of our method. The source code of this paper can be found in https://github.com/xubodhu/RDS .

## introduction social media, with its wealth of user-generated posts, provides a rich platform for understanding events, opinions and preferences of groups and individuals @xcite . information extraction, such as named entity recognition @xcite , relation extraction @xcite and sentiment detection @xcite , is a critical step in uncovering these hidden insights in social media posts. in social media scenarios, information is expressed not only through textual modality, but through multiple modalities (e.g., text, image, etc.). considering only text modality may lead to inaccurate information extraction. for example in therefore, many multimodal models for information extraction have been proposed which are using visual modality to complement text modality. they mainly focus on using a multimodal interaction mechanism to combine text representation with image representation. for example, @xcite propose an adaptive co-attention network to control the combination of text representation and image representation dynamically for multimodal named entity recognition. @xcite propose a multimodal relation extraction neural network with an effective alignment strategy for textual and visual graphs to find the correlations between visual objects and textual entities. @xcite propose multi-channel graph neural networks for multimodal sentiment detection. despite their success, current approaches overestimate the significance of images. in fact, images are not always needed to understand information on social media posts. it is possible to get comparable performance using only text modality, while using mismatched visual modality can hinder performance. the mismatched phenomenon is very common in social media posts. as reported in @xcite , about 33.8% of tweets had textual content that was not reflected in the images, and the images did not add additional content. for example, in fig. 1(b) and fig. 1(d), with the mismatched images, the multimodal named entity recognition model may mistakenly associate the person in the image with nasa and make a wrong prediction, and the multimodal relation extraction model may incorrectly classify the relation between angel and jesenia rodriguez as colleague. based on the above observations, we argue that different social media posts should consider different modalities for multimodal information extraction. multimodal models cannot always outperform unimodal models 1 . some posts are more suitable for the multimodal model, while others are more suitable for the unimodal model. this is consistent with human behavior in accomplishing multimodal information extraction tasks. when someone reads a post on social media, he first determines whether the image will help him complete the task; if not, he goes directly to the text, and if so, he views both the image and the text. according to a research on the multimodal named entity recognition (ner) benchmark dataset twitter-2017 by @xcite , about 22% of entities were incorrectly predicted by the state-of-the-art text-based ner model, but correctly predicted by the state-of-the-art multimodal ner model; and about 12% of entities were correctly predicted by the text-based ner model, but incorrectly predicted by the multimodal ner model. in this paper, we propose a general data splitting strategy to divide the social media posts into two sets so that these two sets can achieve better performance under the information extraction models of 1 in this paper, unimodal model refers to text-based model. the corresponding modalities. specifically, for an information extraction task, we first propose a data discriminator that divides social media posts into a multimodal and a unimodal set, which is used to determine whether the posts are more suitable for the multimodal or unimodal model. then we feed these sets into the corresponding models. finally, we combine the results of these two models to obtain the final extraction results. the core component is the data discriminator. due to the lack of explicit knowledge about which data are more suitable for multimodal models and which data are more suitable for unimodal models, we use reinforcement learning to train the data discriminator. the reward based on the performances of the multimodal model and the unimodal model on both the multimodal set and the unimodal set will be used as a reinforcement signal to update the parameters of the data discriminator. our main contributions are summarized as follows: ## overview in this section, we first formulate our problem, and then introduce our framework. ## method in this section, we first introduce our data discriminator, then we show how to train it using reinforcement learning, which consists of a reward function and a training algorithm. ## experiment to validate the effectiveness of our data splitting strategy, we conducted experiments on two different multimodal information extraction tasks, namely multimodal named entity recognition (mner) and multimodal relation extraction (mre). ## related work in this section, we review and summarize three multimodal ie tasks, namely multimodal named entity recognition, multimodal sentiment detection and multimodal relation extraction. for the multimodal named entity recognition task, at the earliest, @xcite inputs the whole image into a convolutional neural network (e.g. resnet) to obtain a representation of the whole image to establish the relationship between the text and the image. since only some regions in the image are useful for recognizing entities, @xcite @xcite @xcite divide the image into multiple regions, obtained a representation of the image regions and establish a relationship between the image regions and each word in the text. next, since the image and text representations come from different encoders, there is a semantic gap that affects the establishment of image and text relationships. to solve the above problem, @xcite proposes an alignment and matching framework to make the text and image representations more consistent by contrastive learning. more directly, @xcite extract the semantic information of the image directly to represent the image: @xcite extracts the objects in the image by the object detection model and uses the labels of the objects (e.g., apple, trophy) to represent the image, while @xcite uses the image captioning model and the ocr model in addition to the object detection model to obtain the overall semantic information of the image and the textual information in the image, respectively. in addition, @xcite noticed that the mismatched image can impair the prediction of multimodal models and both propose a method to calculate the image and text similarity scores to filter the image information. for the multimodal relation extraction task, @xcite first propose this task and demonstrate that previous text-based relation extraction models perform poorly in social media texts, and that incorporating visual information can help improve relation extraction model performance. subsequently, in order to be able to fully exploit the relationships between objects in the image and to establish the alignment of the image with the text, @xcite use graph structure information to align the relations between entities in text and images and then uses image information to supplement the missing semantic information. for the multimodal sentiment detection task, @xcite obtain the information more important for sentiment by capturing the correlation between text and images. @xcite use an adversarial learning model to learn a joint multimodal representation to combine text and image representations. @xcite use a novel graph neural network based on the global characteristics that encode different modalities to capture hidden representations and learn multimodal representations. however, current approaches overestimate the significance of images. although several works @xcite have proposed methods to filter image information, they all use the similarity scores of images and text to filter the image information as a whole, more or less retaining some image information and not accurately filtering out the noise in the images. therefore, we propose a general data splitting technique to process different data using different models (i.e., multimodal model and unimodal model). ## conclusion in this paper, we propose a general data splitting strategy to divide the social media posts into two sets so that these two sets can achieve better perfor-mance under the information extraction models of the corresponding modalities. the core component is the data discriminator. due to the lack of explicit knowledge, we use reinforcement learning to train the data discriminator. experiments conducted on two widely used multimodal named entity recognition datasets and a multimodal relation extraction dataset show that our data discriminator can effectively split the data and our proposed data splitting strategy for multimodal information extraction achieves the best performance.