Only text? only image? or both? Predicting sentiment of internet memes (2020)

Abstract: Nowadays, the spread of Internet memes on online social media platforms such as Instagram, Facebook, Reddit, and Twitter is very fast. Analyzing the sentiment of memes can provide various useful insights. Meme sentiment classification is a new area of research that is not explored yet. Recently SemEval provides a dataset for meme sentiment classification. As this dataset is highly imbalanced, we extend this dataset by annotating new instances and use a sampling strategy to build a meme sentiment classifier. We propose a multi-modal framework for meme sentiment classification by utilizing textual and visual features of the meme. We found that for meme sentiment classification, only textual or only visual features are not sufficient. Our proposed framework utilizes textual as well as visual features together. We propose to use the attention mechanism to improve meme classification performance. Our proposed framework achieves macro F1 and accuracy of 34.23 and 50.02, respectively. It increases the accuracy by 6.77 and 7.86 compared to only textual and visual features, respectively.

## introduction the rapid growth of users on social media platforms leads to new ways of spreading information. meme nowadays has become one of the most popular words for social media. a meme is an idea, the way in which a person behaves in response to a particular situation or a manner that spreads from one person to another within a culture. spreading of memes on social media platforms such as facebook, instagram, reddit, and twitter is very fast. sentiment analysis is a growing field of natural language processing (nlp), aiming to identify the polarity of opinion. sentiment can be positive, negative or neutral @xcite . sentiment analysis has a vast number of applications in real life, including the product's recommendation to a user based on opinions provided by other users @xcite , in political uses @xcite , etc. memes play an important role in handling various political battles or public relations on social media platforms. the most common practice in sentiment analysis is finding the sentiment of textual content crawled from twitter, product reviews, hotel reviews, etc. existing literature has mostly addressed the problem of sentiment analysis primarily using textual contents @xcite @xcite . but with the growing social media, users are expressing their opinions through text and the image. hence, researchers nowadays are also giving attention to sentiment analysis in multi-modal content @xcite @xcite . spreading of memes is also very fast, but meme analysis is yet to be explored. recently, semeval-2020 proposed a task to detect the meme's polarity, which can fall into three predefined classes: positive, negative, or neutral @xcite . this is the very first attempt towards the meme sentiment analysis. to analyze the sentiment of memes, the text-only approaches may not be sufficient. for example, consider the meme given in figure 1 , if the only textual content is considered, then the sentence 'fi-nally gets job interview' seems to have a neutral sentiment (no explicit positive words are used). however, if we also consider the visual information of meme, as shown in figure 1 , then we can say overall sentiment is positive. hence, to analyze memes, both text and visual features have their own importance. in this paper, we work on the semeval-2020 task-8 dataset to detect the sentiment of memes. but this dataset is imbalanced. hence we extend this dataset by adding more training instances for figure 1 : meme example balancing purposes and then propose a multi-modal framework based on deep neural networks to classify the sentiment of the meme into one of the predefined classes, namely positive, negative, and neutral. we use a multi-modal framework with attention applied to both image and text to find out important regions and important words. thereafter, to combine the image and textual modality, we use a fully connected layer that tries to find the relation between textual and visual features and finally produces a combined feature vector. we evaluate the proposed approach using accuracy and macro f1 score on the test set of the semeval-2020 dataset. we get the macro f1 of 34.23%, and accuracy of 50.02%, respectively, which is higher than the se-meval baseline, i.e., macro f1 of 0.21%. the rest of the paper is organized as follows. section 2 describes the related work. section 3 describes the methodology for classification. section 4 describes the data collection, annotation, and experimental setup. section 5 describes the results and detailed error analysis. section 6 concludes the paper and describes future research plans. ## related work this section describes the works on sentiment analysis for text as well as for multi-modal content. (murty and allu) proposed an approach for finding the sentiment analysis on text reviews by using long short term memory. @xcite proposed the framework to classify the sentiment of tweets into positive, negative and neutral class using prior polarity scoring, which is based on the prior polarity of words. @xcite proposed a sentiment-feature-enhanced deep neural network (sdnn) to detect the sentiment of text by deep neural network integrated with sentiment linguistic knowledge via attention mechanism. @xcite proposed a framework for textual sentiment analysis using lexicon based and machine learning based approach. sentiment is predicted from the set of sentiment-bearing words identified in the text using lexicons. @xcite proposed a set of domain transferable twitter lexicons, obtained from tweets for the task of sentiment analysis. @xcite proposed a model to detect the sentiment of images using convolutional neural network. they used flicker images dataset to train their model and twitter images dataset for testing. @xcite proposed a stacked ensemble model for predicting the degree of intensity for sentiment and emotion. they used multi-layer perceptron network to combine outputs of feature based models and deep learning models. @xcite explored different deep-learning based architectures for multi-modal sentiment classification. they used deep convolutional neural network (cnn) to extract features from the visual and text modalities. @xcite proposed a fusion-extraction network model for multi-modal sentiment analysis. their proposed model learned two types of representations, visualspecific textual representations and textual-specific visual representations using interactive information fusion mechanism. above mentioned works are either for text or multi-modal content. meme classification has not been explored much in detail. so we proposed a framework for meme classification by utilizing text written on it and image features. ## methodology this section represents our proposed methodology in detail. we develop a multi-modal neural network that learns from the two modalities, viz. textual and visual. for text modality, our model takes as input the embedding representation of each word present in the ocr extracted text. further, we use convolutional neural network (cnn) to learn textual features, and then we apply attention to the output of cnn to extract the most relevant features for classification. we use the pretrained model vggnet to extract the visual features for image modality, and then we apply attention to the extracted features to detect important visual features for classification. finally, both the features are fused with the help of a fully connected layer. the overall architecture of our proposed model is shown in figure 2. ## dataset and experiment in this section, we discuss about the dataset used for the experiment, data collection, data annotation, and experimental details. ## conclusion in this paper, we have proposed a multi-modal framework for meme sentiment classification by utilizing textual and visual information of memes. we use the semeval-2020 task data and also annotated our own dataset to make this dataset balanced. we found that only textual information or only visual information is not sufficient to analyze a meme's sentiment. our proposed framework utilizes textual and visual features and finally fuses both the information through a fully connected layer. our proposed framework achieved the macro f1 and accuracy of 34.23% and 50.02%, respectively. our proposed framework increases the accuracy by 6.77% and 7.86% compared to only textual and visual features, respectively. in the future, we are planning to explore other fusion methods to incorporate textual and visual features. we would also explore contextual embeddings for the text part of meme classifications.