IIITG - ADBU at S em E val-2020 Task 8: A Multimodal Approach to Detect Offensive, Sarcastic and Humorous Memes (2020)

Abstract: In this paper, we present a multimodal architecture to determine the emotion expressed in a meme. This architecture utilizes both textual and visual information present in a meme. To extract image features we experimented with pre-trained VGG-16 and Inception-V3 classifiers and to extract text features we used LSTM and BERT classifiers. Both FastText and GloVe embeddings were experimented with for the LSTM classifier. The best F1 scores our classifier obtained on the official analysis results are 0.3309, 0.4752, and 0.2897 for Task A, B, and C respectively in the Memotion Analysis task (Task 8) organized as part of International Workshop on Semantic Evaluation 2020 (SemEval 2020). In our study, we found that combining both textual and visual information expressed in a meme improves the performance of the classifier as opposed to using standalone classifiers that use only text or visual data.

## introduction the word meme was first coined by richard dawkins to refer "an idea, behavior, or style that spreads from person to person within a culture" @xcite . internet memes are the result of deliberate modification of an original idea using one's own creativity 1 . it is a type of meme that spreads via the internet. memes have become a new way of communication on the internet. people mostly use it as a way to share jokes. but there are also other classes of memes that are offensive in nature. they spread hatred and racism. this second class of memes is harmful to society. they need to be detected and removed from the internet. but the scale involved makes manual monitoring difficult. automated systems can help detect offensive memes. however, the automatic classification of memes has a lot of challenges. it is not textual data or images in isolation. the information provided by both the image and the text needs to utilized to correctly understand the emotion expressed by the meme. the text itself may be very short having only a few words. but the image provides a context and the expressions exhibited by the images supplement the textual information and it is the combinations of both text and image that enable one to understand the message conveyed by the meme. memotion analysis (task 8) organized as part of the international workshop on semantic evaluation 2020 (semeval 2020) required detecting the emotion expressed by a given meme @xcite . this task consisted of three subtasks: subtask a -detect the sentiment of a given meme. it was a three-way classification problem with the labels being positive, negative, or neutral, subtask b -this was a multi-label classification problem where it was required to determine if a given meme is humorous, motivational, offensive or sarcastic. a meme can belong to multiple class, subtask c: this was a multiclass multi-label classification problem where each class mentioned in subtask b is further sub-divided into four levels. for example, the humorous class is sub-divided as not funny, funny, very funny, and hilarious. we participated in all the three subtasks. we used the pre-trained convolutional neural networks vgg-16 @xcite and inception-v3 @xcite as our image classifiers. for processing text, we used lstm @xcite and the pre-trained bert classifier @xcite . the rest of the paper is structured as follows: section 2 discusses the related works that have used multimodal approaches for detecting emotions expressed by memes, section 3 describes the dataset used in this shared task, section 3 discusses the methodology and the architecture of the classifier used by us, section 5 describes the different experiments we have performed in this task including the questions we are trying to answer through the experiments, and section 6 discusses the results our classifier obtained on the development set and the test set. ## related work extracting the text from the meme is an important step in the multimodal processing of meme. @xcite describes facebook's ocr system for extracting text from images in realtime. @xcite talks about integrating both text and image based distributional information to create multimodal distributional semantic vectors. @xcite used a multimodal approach to determine the sentiment of a meme. google lens was used to separate the text from the meme. the sentiment expressed by the text was determined using the combination of a convolutional neural network and the sentiment scores of words from the vader sentiment lexicon. when determining the sentiment scores of the words, the context in which the word appears was taken into consideration. the image was processed using an svm classifier trained with bag of visual words features. a boolean decision system was then used to combine the text and image scores to make the final classification. @xcite also used a multimodal approach to determine the emotion expressed in the tumblr posts. inception and lstm were used to process the image and text respectively. sabat et al. (2019 ) used a multimodal approach to detect hate memes. bert was used to process the text and vgg-16 was used to process the image. both the information was concatenated and the final classification was done using an mlp classifier. ## data set the train and test data set provided as part of this task consisted of 6,992 and 1,878 memes respectively. the memes in the train data set were annotated for the following categories: sentiment, humour, motivational, offensive, and sarcasm. ## methodology in this multimodal approach of detecting the emotion expressed by a meme, we combined both the visual and textual information to perform the classification. the pre-trained image classifiers vgg-16 @xcite and inception-v3 @xcite were used to process the image. the textual data was processed using lstm @xcite and the pre-trained bert @xcite classifier. vgg-16 is a convolutional neural network (cnn) based architecture for image classification. it was ranked second in the imagenet image classification task in 2014 (ilsvrc 2014). it has 13 convolutional layers, 5 max pooling layers, and 3 dense layers. out of these 21 layers, only 16 are weight layers. inception-v3 is the third version of google's inception network. it too is a cnn based architecture. it was the 1 st runner-up in the imagenet image classification task in 2015. it consists of 42 layers. l1 3,507 (50%) l2 1,544 (22%) l3 1,547 (22%) l4 394 (6%) total 6,992 lstm is a type of recurrent neural network (rnn). it handles the vanishing and the exploding gradient problem through the use of an input gate, output gate, and forget gate. it is thus able to handle long range dependencies. bert is a bi-directional model based on the transformer architecture. the transformer architecture is an architecture based solely on attention mechanism @xcite . ## experiments we performed the following nine types of experiments for this task: (1) only vgg-16, (2) only inception-v3, (3) only lstm with fasttext embeddings, (4) only bert, (5) vgg-16 for processing image and lstm with glove embeddings for processing text, (6) vgg-16 for processing image and lstm with fasttext embeddings for processing text, (7) inception-v3 for processing image and lstm with glove embeddings for processing text, (8) inception-v3 for processing image and lstm with fasttext embeddings for processing text, and (9) inception-v3 for processing image and bert for processing text. the first four experiments used standalone classifiers (meaning classifiers that used only textual or visual data). the next five experiments used multimodal classifiers (classifiers that combined both textual and visual data). the reason for conducting these experiments was to find an answer to the following questions: ## results as can be seen from the table, except for task b offensive subtask, all the other subtasks have benefited from combining both the image and text data. in the case of task b offensive subtask, the best f1 score 7 conclusion memes are becoming a very popular means of communication in social media. some of these memes are offensive in nature and they are used to spread hatred. thus, it is very important to develop systems to automatically determine the emotion expressed by memes. developing such systems requires combining both textual and visual information present in the meme. in this paper, we presented a multimodal architecture that combines both textual and visual data to determine the emotion expressed by the meme. this classifier obtained f1 scores of 0.3309, 0.4752, and 0.2897 for task a, b, and c respectively. in our study, we found that combining both textual and visual data improves the performance of the classifiers than using standalone classifiers.