IITK at S em E val-2020 Task 8: Unimodal and Bimodal Sentiment Analysis of I nternet Memes (2020)

Abstract: Social media is abundant in visual and textual information presented together or in isolation. Memes are the most popular form, belonging to the former class. In this paper, we present our approaches for the Memotion Analysis problem as posed in SemEval-2020 Task 8. The goal of this task is to classify memes based on their emotional content and sentiment. We leverage techniques from Natural Language Processing (NLP) and Computer Vision (CV) towards the sentiment classification of internet memes (Subtask A). We consider Bimodal (text and image) as well as Unimodal (text-only) techniques in our study ranging from the Na ̈ıve Bayes classifier to Transformer-based approaches. Our results show that a text-only approach, a simple Feed Forward Neural Network (FFNN) with Word2vec embeddings as input, performs superior to all the others. We stand first in the Sentiment analysis task with a relative improvement of 63% over the baseline macro-F1 score. Our work is relevant to any task concerned with the combination of different modalities.

## introduction an internet meme conveys an idea or phenomenon that is replicated, transformed and spread through the internet. memes are often grounded in personal experiences and are a means of showing appeal, resentment, fandom, along with socio-cultural expression. nowadays, the popularity of the internet memes culture is on a high. this creates an opportunity to draw meaningful insights from memes to understand the opinions of communal sections of society. the inherent sentiment in memes has political, social and psychological relevance. the sarcasm and humor content of memes is an indicator of many cognitive aspects of users. social media is full of hate speech against communities, organizations as well as governments. analysis of meme content would help to combat such societal problems. the abundance and easy availability of multimodal data have opened many research avenues in the fields of nlp and cv. researchers have been working on analyzing the personality and behavioral traits of humans based on their social media activities, mainly posts they share on facebook, twitter, etc. @xcite . in this regard, we attempt to solve the sentiment classification problem under semeval-2020 task 8: "memotion analysis" @xcite . memotion analysis stands for analysis of emotional content of memes. we dissociate the visual and textual components of memes and combine information from both modalities to perform sentiment-based classification. the literature is rich in sentiment classification for tweets @xcite and other text-only tasks @xcite . the multimodal approaches are relatively recent and yet under exploration @xcite @xcite . we first describe the problem formally in section 2, followed by a brief literature review of the work already done in this domain. in section 3, we describe the methods proposed by us. section 4 contains the description of the dataset provided by the organizers, along with the challenges accompanying the data. it further takes a deeper dive into the method yielding the best results. section 5 summarizes the results with a brief error analysis. towards the end, section 6 concludes the paper along with future directions. the implementation for our system is made available via github foot_0 2 background subtask a of memotion analysis is about sentiment analysis of memes. it is concerned with the classification of an internet meme as a positive, negative, or neutral sentiment. a major portion of the research is devoted to separate handling of text and image modalities. among the text-only methods, the transformer @xcite is an encoder-decoder architecture that uses only the attention mechanism instead of rnn. it has wide applications in different nlp tasks. bert @xcite is the state-of-the-art transformer model for text only classification. it introduced the concept of bidirectionality to the attention mechanism to allow better use of contextual information. reformer @xcite is the most recent transformer, but more memory efficient and faster, hence works better for long sequences. for image-based tasks, imagenet @xcite is a visual dataset with hand annotated images for visual object recognition task. resnet-152 @xcite is a deep learning model trained using the imagenet dataset and used for object classification. the joint handling of image and text modalities has gained attention relatively recently. @xcite proposed a text-image sentiment analysis model that linearly combines image and text features. mmbt @xcite or the multimodal bitransformer fuses information from text and image encoders (bert and resnet) by mapping the image embeddings into the text space. ## methods a wide variety of methods, ranging from a simple linear classifier to transformers, were employed. we broadly classify the set of techniques into bi-modal and uni-modal. ## experimental setup in this section, we quantitatively describe the dataset provided by the organizers and challenges accompanying it. we then mention the preprocessing steps briefly. finally, we discuss the architecture and parameters of the ffnn with word2vec approach (section 3.2.2) in detail. this method made it to the final submission as it performed better than all the other approaches. ## results the official evaluation metric for memotion analysis is the macro-f1. we present the macro-f1 scores of our five main approaches for subtask a in the dominance of the majority class over the others played a crucial role in sub-par performance for the other classes. for the 'negative' class, there are not enough data-points for the system to train. to some extent, this issue was resolved using simple upsampling. the transformer-based approaches overfit heavily to the majority class. the repetition of meme templates is another issue for the bimodal approaches. sarcasm also introduced ambiguity in the sentiment classification task. neutral class dominated relatively due to the absence of polar words in some sarcastic texts. our results may be surprising as the state-of-the-art models, bert and mmbt, are expected to do better. some of the underlying reason for such behavior can be fewer data points, sarcasm, noise, and large variance in caption lengths. moreover, bert has been pre-trained on wikipedia and book corpus, a dataset containing +10,000 books of different genres. these contained well-defined sentences, but our dataset is noisy, lacks punctuation marks and comprises of sarcasm. simpler approaches did a better job since they involve no pre-training on any other (large) corpus. ## conclusion we attempt to perform a complex task of classifying memes constrained by the data size and quality. while the results were sound for the populous classes, it could only be par after re-sampling for the skewed classes. the best results were obtained for ffnn with word2vec embeddings ( a better research problem would be to define some rules to obtain meme data and then perform the above task so domain knowledge could be used to improve performance. in future, a study could be designed to observe the diffusion of memes among different communities by analysing which meme is most liked or hated by a particular community (e.g. a facebook group). finding the political inclination of memes is also a possible path. memes have become an increasingly popular mode of expressing opinions by party supporters, party critics and the affected people. they may be considered as a means of propaganda. this makes the problem of detecting the inclination of memes in a political context an important research exercise.