Memebusters at S em E val-2020 Task 8: Feature Fusion Model for Sentiment Analysis on Memes Using Transfer Learning (2020)

Abstract: In this paper, we describe our deep learning system used for SemEval 2020 Task 8: Memotion analysis. We participated in all the subtasks i.e Subtask A: Sentiment classification, Subtask B: Humor classification, and Subtask C: Scales of semantic classes. Similar multimodal architecture was used for each subtask. The proposed architecture makes use of transfer learning for images and text feature extraction. The extracted features are then fused together using stacked bidirectional Long Short Term Memory (LSTM) and Gated Recurrent Unit (GRU) model with attention mechanism for final predictions. We also propose a single model for predicting semantic classes (Subtask B) as well as their scales (Subtask C) by branching the final output of the post LSTM dense layers. Our model was ranked 5 in Subtask B and ranked 8 in Subtask C and performed nicely in Subtask A on the leader board. Our system makes use of transfer learning for feature extraction and fusion of image and text features for predictions.

## introduction a meme is an approach, concept, idea or style that spreads through social media within a society often with the goal of expressing a trend, topic, or significance represented by the meme @xcite . recent developments in social media analytics have shown a widespread increase in the usage of memes in expressing sentiment. social media networks like facebook, instagram and twitter usually get flooded with memes upon occurrence of any popular event. memes are present in almost every form of media and they are constantly evolving considering the events happening around the world. the spread of memes on social media can be considered analogous to human genetics. slightly altered versions of the same memes spread in social media in a similar way as copies of human genes pass through human lineage @xcite . they work as a medium for sharing humour on the pretext of cultural themes. they represent the sentiment of a community or culture with respect to the event involved. this is beneficial in gauging the sentiment of people. memes concerning to political events may represent the political outlook of a community. simultaneously, they can also be engineered to further political ideals, amplify echo chambers and alienate minorities. they are altogether a different means of communication and have become fundamental part of the current generation. memes are uniquely multimodal, they contain images in conjugation with textual comments and conversations to add emphasis and provide additional meaning. rarely, memes contain only the text or visual component. thus, they pose strong challenge in identifying their semantics which may involve use of text as well as image features. significant research has been carried out on sentiment classification of twitter data @xcite , but they mostly confine to only the text. in comparison, multimodal analysis of social media content has very little research on it. meme text generation @xcite on image templates provides some useful insights on use of deep learning on memes. the semeval 2020 task 8: memotion analysis @xcite draws attention towards analyzing sentiment of memes on social media to extract the conveyed message and identify the semantics of the memes. in this paper we describe our deep learning model that competed in the semeval 2020 task 8: memotion analysis @xcite . our proposed system makes use of transfer learning as a key component for extracting features from meme images and their optical character recognition (ocr) extracted text. we use stacked bi-directional long short term memory (bi-lstm), gru and attention mechanism for fusing the features of the two modalities into a single feature space. we use these combined features for classifying the humour class as well as quantify them on the given semantic scale. our model performed well for subtask b and subtask c. we were ranked 5 th and 8 th on the official leader board. we propose a single multitask learning model for joint feature extraction which can then be used for multiple tasks dependent on similar feature space. this removes the need for training different systems for subtask b and subtask c. this brings down computational cost and complexity. moreover, it advances the idea of transfer learning for tasks dependent on similar semantics. our model was ranked 23 th in subtask a. one possible reason for comparatively low performance could be the inability to differentiate between neutral and positive classes as they maybe remarkably similar. our code is available online foot_1 for method replicability. ## background understanding semantics of memes requires multimodal analysis, since they contain an image with some text. occasionally it is an image without text that is symbolic of an emotion, such as surprise or joy @xcite , it poses an important challenge in recognizing their semantics. we need to take both components of a meme to analyse them. the study done in @xcite focused on classifying the relationship between meme text and its corresponding image from social media. sentiment classification using images combined with their label embeddings @xcite outperformed both only image and only text models. the text only model's performance was comparable with combined (image and text) model, which implies that textual data plays a crucial role while analysing multimodal data. combining of the text and image modalities also poses a major challenge while performing analysis on meme data. another challenge in performing analysis on such data is combining the two different modalities i.e text and image. work done by @xcite involves different fusion techniques for effectively combining the data and discusses their performance. @xcite applied deep learning techniques to investigate the structure of emotions in multimodal data. they used transfer learning for extracting the features of each modality and concatenated them for further classification. another important study on memes involved generating captions for meme templates @xcite . it made use of image captioning model proposed by @xcite which uses an encoder-decoder network. work presented at @xcite , @xcite provides useful insights for understanding the sentiment represented in text data. memes have become an inherent part of modern-day world. meagre research has been carried on memes and using combined text and images data for analysis. semeval 2020 task 8: memotion analysis is an effort to bring research attention to this topic, it has three subtasks which we describe as follows: subtask a (sentiment classification): given a labelled dataset d of internet memes and their ocr extracted text, the objective of the task is to learn a classification function that can predict label l for a given meme, where @xmath0, neutral, positive}. subtask b (humour classification): given a labelled dataset d of internet memes and their ocr extracted text, the objective of the task is to learn a classification function that can identify the type of humour h for a given meme, where @xmath1, humorous, offensive, motivational}. subtask c (scales of semantic classes): given a labelled dataset d of internet memes and their ocr extracted text, the objective of the task is to learn a classification function that can quantify the extent e to which a particular humour is expressed, where @xmath2, slightly, mildly, very}. analysis of training set: training set provided consisted of 6992 meme images and their ocr extracted text. the images were annotated as per the requirements of each subtask. 3 system overview ## results and analysis our model performed considerably well for subtask b and subtask c. our team rank was 5 and 8. for subtask a our rank was 23. the final f1 scores for our test set are given in the our model performed considerably well for subtask b and subtask c. for subtask a the lower performance can be due to difficulty in identifying the neutral class from the positive class. these two classes may have similar features which make it difficult for the model in identifying the correct classes. the difference in macro and micro f1 scores for subtask a justifies the above premise. another important aspect which we did not cover was the interdependence of humour classes h and sentiment l on each other. we trained independent models for each class for classification and quantification using similar architectures. we used transfer learning using only inception and glove models. testing our architecture with separate models may lead to different results. one of the main problems which we came across was identifying sarcasm in images which is a key component of expressing sentiment and humour using memes. a lot of work has been done in identifying sentiment from text. identifying sentiment in images that represent sarcasm is a tough task. combining the two modalities improves the understanding of the memes. but better understanding of sentiments from images may further improve the results in future works. ## conclusion this paper describes our deep learning system that competed in all the subtasks of the semeval 2020 task 8: memotion analysis. transfer learning was used to extract features from image and text separately, employing the state-of-the-art models. we used lstm with attention to combine the features of our model. a single model was used for classification as well as quantification for each h, where @xmath3, humorous, offensive, motivational}. we performed nicely in subtask b and subtask c. one important aspect which we would like to work on in the future is the interdependence of humour classes h on each other so that we can create a single model which utilises the interdependence of humour classes and perform classification as well as quantification for each h using a single combined model.