DSC IIT - ISM at S em E val-2020 Task 8: Bi-Fusion Techniques for Deep Meme Emotion Analysis (2020)

Abstract: Memes have become an ubiquitous social media entity and the processing and analysis of such multimodal data is currently an active area of research. This paper presents our work on the Memotion Analysis shared task of SemEval 2020, which involves the sentiment and humor analysis of memes. We propose a system which uses different bimodal fusion techniques to leverage the inter-modal dependency for sentiment and humor classification tasks. Out of all our experiments, the best system improved the baseline with macro F1 scores of 0.357 on Sentiment Classification (Task A), 0.510 on Humor Classification (Task B) and 0.312 on Scales of Semantic Classes (Task C).

## introduction social media is becoming increasingly abundant in multimedia content with internet memes being one of the major types of such content circulated online. internet memes, or simply, memes are mostly derived from news, popular tv shows and movies making them capable of conveying ideas that the masses can understand readily. today memes are shared in numerous online communities and are not only limited to one language. it is therefore imperative to study this emerging form of mass communication. although they are mostly for amusement and satire, memes are also being used to propagate controversial, hateful and offensive ideologies. since it is not feasible to manually inspect such content it is important to build systems that can process memes and segregate them into appropriate categories. the semeval 2020 shared task on memotion analysis @xcite aims to bring attention to the study of sentiment and humor conveyed through memes. the challenge involves 3 tasks: here, we consider task b as a set of 4 independent binary classification subtasks while task c comprises 3 subtasks each being a multiclass classification problem. this paper describes our work on all three tasks. here, a meme has two modalities -textual and visual cues. we use a combination of state-of-the-art model architectures and adapt them for processing multimodal memes through transfer learning while also training models from scratch for comparison. we explore various modality fusion techniques and also the inter-task dependency. our systems † score higher than the baseline results released on all the 3 tasks. we investigate our model performances from relative and absolute perspectives and also explore some possible improvements. we make our code publicly available. ## related work discussing the sentiments of memes has been a relatively new research area. although interest in the research related to meme generation, circulation and perception has been increasing in the last couple of years where studies of vyalla and udandarao (2020), wang and wen (2015) have shown pioneering work, studies exclusively analysing meme emotion are relatively scarce. with the explosion of social media in recent years and human generated content on them like tweets, memes etc., deep learning techniques have captured the attention of researchers due to their ability to significantly outperform traditional methods in sentiment analysis tasks. studies based on discussing textual segments @xcite and based on other channels like images and videos @xcite have been in progress. visual sentiment and emotion analysis, although having received less attention than text based analysis, has involved studies of @xcite , @xcite and @xcite where features extracted from images using deep learning have been used for predictions. apart from the unimodal methods we have seen so far, the multimodal methods involving simultaneous processing of textual and visual information are of special interest, as studies by @xcite and @xcite have reported improvement in performance by combining modalities. multimodal processing of features using hierarchical methods @xcite and attention based methods @xcite have shown to be effective fusion techniques and have been used by teams in previous iterations of semeval as well. multitask learning for processing multimodal information has also been studied by works like @xcite , where the authors have tried to capture inter-dependence between multimodal tasks like emotion recognition and sentiment analysis. in addition to above, processing of multimodal information specifically for memes have also been studied by some papers which have greatly inspired our work, including papers by @xcite , which discusses correlation between the implied semantic meaning of image-based memes and the textual content of discussions in social media and also @xcite , which have developed models to generate rich face emotion captions from memes. ## data the dataset @xcite consists of almost 7k meme images along with their extracted text. each sample has a sentiment annotation into one of 3 classes-negative (0), neutral (1) or positive (2). there are another 3 semantic classes namely humor, sarcasm and offence with their fine grained labels as not (0), slightly (1), mildly (2) and very (3). also there is a binary column denoting whether a meme is motivational. another set of around 2k samples form the test set. we split the dataset into train and validation sets of around 6k and 1k samples respectively. from the data distribution in ## methodology a meme typically consists of an image paired with a catchphrase on it. the image is generally derived from some popular source and is reused as a template to represent an underlying theme. the text on the other hand is usually original and specific, written by the creator, and it is what makes the meme unique. together they convey some sentiment and emotion which we study here. in this task we focus on three main approaches: a more detailed explanation of our approaches can be found in the following subsections. ## experiments using the techniques mentioned in the previous section we train several models for our submissions. in this section, we provide the training parameters for reproducibility 1 and also their results on the test data. ## discussion in order to justify the results and to discuss some observations we have about the dataset and memes in general, we present a detailed analysis in this section. we investigate why the state-of-the-art pretrained models could not give the superior results that is expected from them and highlight the areas of potential improvement. text arrangement: apart from deep learning models we also try the tf-idf model with classical machine learning classifiers like logistic regression. although the deep learning models perform better, the validation set score difference is not large. on examining the data we find examples in which the text is scattered around the canvas and since an ocr is used for preparing the dataset, it ended up giving jumbled phrases in an order which is different from what is intended by the maker of the meme. in one such example foot_0 , the image had an overlay phrase "challenge accepted!" and under that was the dialogue "friend: you can't honestly watch how i met your mother again for like the 4th time... me:". in this case, the phrase "challenge accepted" will appear at the beginning of the text when read by an ocr whereas in reality it is intended to be perceived at the end of the dialogue when read by a human. such samples undermine the advantage that sequential models have over frequency based models. language use: for the unimodal bilstm model, apart from training the embedding layer parameters from scratch we also use the pretrained glove embeddings @xcite . although the glove embeddings are trained on a much larger corpus than the dataset here, we found that it did not improve the performance of the text model. this can be attributed to the difference in the vocabulary usage of memes over standard english and the intentional misspellings used for common words. this can be seen in a meme foot_1 in the dataset which had the russian president vladimir putin with a smirk on his face and the text reading "not ukraine ... mykraine" to imply "my ukraine". moreover, the incorrect grammar used in a lot of memes may also be the reason why roberta, despite being pretrained on a huge english corpus, failed to generalize well over the meme language and scored below bilstm for all the three tasks. using character level embedding approaches to the language model can be helpful in dealing with such cases. image manipulation: for the image models also, in spite of having an edge over alexnet, the pretrained deep models, inception @xcite and resnet, which we try, didn't outperform it by a large margin. we believe this is because there is very little similarity between the imagenet dataset and the meme images used here. memes are usually images that have been edited and also have some text overlay which can act as noise for the network thereby hindering representation learning. a lot of memes in the dataset were found to have considerable editing or using multiple images combined into one along with text annotations on top 4 . to deal with the text overlay in memes, some techniques in image masking can be used as a preprocessing step to replace the text with its background. overall, the winning macro-averaged f1 scores in the competition for task a, task b and task c were less than 0.36, 0.52 and 0.33 respectively which shows there is ample room for improvement. out of our three fusion strategies, simple concatenation (early fusion) turned out to be the best. but when we see a meme both text and image rely on each other to jointly convey an idea. we strongly feel that there is a need for a better fusion strategy or learning technique that can generate a unified and coherent representation, which is analogous to how a human brain processes memes. for the image captioning task, there has been some work done regarding inter-modal correspondences between language and visual data @xcite to align the two modalities. but in image captioning, the image and the text essentially have the same semantic information which is not the case for memes. ## conclusion in this paper, we explore the challenges in meme sentiment and humor analysis using deep learning. we compare the performance of large pretrained models with the simpler models that we train from scratch. through our detailed analysis we show that the unconventional presence of both the modalities in memes reduces the effectiveness of domain adaptation or transfer learning techniques and discuss some ways to improve our models as future work.