Co-attention based Multimodal Factorized Bilinear Pooling for I nternet Memes Analysis (2021)

Abstract: Social media platforms like Facebook, Twitter, and Instagram have a significant impact on several aspects of society. Memes are a new type of social media communication found on social platforms. Even though memes are primarily used to distribute humorous content, certain memes propagate hate speech through dark humor. It is critical to properly analyze and filter out these toxic memes from social media. But the presence of sarcasm and humor in an implicit way analyzes memes more challenging. This paper proposes an end-to-end neural network architecture that learns the complex association between text and image of a meme. For this purpose, we use a recent SemEval-2020 Task-8 multimodal dataset. We proposed an end-to-end CNN-based deep neural network architecture with two sub-modules viz. (i)Co-attention based sub-module and (ii) Multimodal Factorized Bilinear Pooling(MFB) sub-module to represent the textual and visual features of a meme in a more fine-grained way. We demonstrated the effectiveness of our proposed work through extensive experiments. The experimental results show that our proposed model achieves a 36.81% macro F1-score, outperforming all the baseline models.

## introduction social media such as facebook, twitter, instagram, etc., are interactive platforms that accelerate the idea of creating and sharing information. this information made an enormous impact in different fields of society more powerfully and effectively. but on the other hand, we observe a significantly large amount of offensive content in the various social networking sites, which spread hatred, rumors, etc., between the different communities, groups, or individuals. meme @xcite is the form of multimodal media that has been initially created to spread humorous content, but due to its multimodal nature, some memes help users to spread hate speech in the form of dark humor. on social media, posting such memes to troll, cyberbully, or targeting someone is increasing rapidly. unlike other multimodal tasks (e.g., visual question answering, image captioning, etc.), in sentiment analysis for memes, textual and visual information are very weakly semantically aligned. in such a situation, we cannot uncover the complex meaning of hateful content until we get to know both the modalities and their contributions in any content hateful. analysis of such memes can bring valuable insights that are not explored yet. for example, if there is a meme with text containing "look how many people love you." the sentiment of this meme can, itself, be positive, negative, or neutral. the sentiment can only be found if and only if we add an image to it (c.f. figure 1 ). some memes are purely humorous, while others spread offensive content in the form of dark humor, sarcasm, mockery, etc. sentiment analysis of memes in a more effective way will facilitate combating such social media issues. figure 1 : a meme where only after focusing on both text and image, negative sentiment can be identified. with the phenomenal growth of social media networks, sentiment analysis plays a significant role in handling various aspects of political and religious views of society. sentiment analysis research has been a progressive area in natural language processing (nlp). it is ranging from document-level classification @xcite to learning the word and phrase polarity @xcite . several supervised machine learning and feature-based techniques have been used to tackle this problem @xcite . however, deep learning-based techniques have gained a lot of popularity in recent years @xcite . a significant amount of works have been done which includes the analysis of opinions about hotel reviews @xcite ), product reviews ( @xcite @xcite ) etc. initially, sentiment analysis has been carried out mostly using text @xcite @xcite . recently due to the growth of multimedia contents in social media, we also need to develop robust models that would deal with multimodal content. there have been very few attempts towards analyzing the sentiment of memes by researchers. however, research shows that there is a far way to go if we compare the system-generated output to the human evaluation. there can be several reasons for this, such as hateful meaning hidden behind humor, sarcasm, the use of very twisted words, or image to spread hate @xcite . lack of annotated datasets can also be one of the reasons for this kind of failure. the key attributes of our current work can be summarized as follows: (i) we develop a deep neural network-based architecture to explore the idea of co-attention to identify the impact of text and image simultaneously for predicting the correct sentiment of a given meme.(ii) we also explore the concept of multimodal factorized bilinear pooling to represent the textual and visual features in a internet meme analysis dataset. we test the significance of our proposed method on semeval2020 @xcite dataset. evaluation results the accuracy and macro-f1 of 54% and 36.81%, respectively, which are higher than the baseline model for the given task. ## related work this section briefly discusses the review related to two aspects: a) sentiment analysis for unimodal data, b) sentiment analysis for multimodal data. ## methodology our current task aims at determining the sentiment of a given meme in a multimodal dataset. the problem can be defined as follows: given every meme m i in the dataset which is a combination of text t @xmath0, t i2 , ...., t ik ) and image i i with the shape (224,224,3) in rgb pattern, our task is to create one classifier that should predict one correct label y ⊆{neg,neu,pos} for m i i.e. predict the correct sentiment whether a given meme is negative, neutral or positive. the respective optimizing goal is then to learn the parameter θ and get the optimum loss function l(y |m, θ). at first, we develop a unimodal baseline system for text and image each. finally, different multimodal approaches for the fusion of both modalities, i.e., textual and visual, have been described in the following sections: ## conclusion in this work, we have proposed an end-to-end cnn-based deep neural network that consists of co-attention that jointly reasons about textual and visual representation. additionally, we incorporated one common portrayal of a meme by utilizing the multimodal factorized bilinear pooling of textual and visual features. by introducing these joint representations, we obtain more effective multimodal features to identify the sentiment of a given meme. from the quantitative and qualitative error analysis on the recently released semeval-2020 task-8 @xcite ) dataset, we observed that our proposed method produces promising results with respect to the baseline models. in the future, we will investigate more fusion strategies to combine both the modalities effectively; and investigate methods to extract essential objects from the meme.