Infotec + C entro GEO at S em E val-2020 Task 8: Deep Learning and Text Categorization approach for Memes classification (2020)

Abstract: The information shared on social media is increasingly important; both images and text, and maybe the most popular combination of these two kinds of data are the memes. This manuscript describes our participation in Memotion task at SemEval 2020. This task is about to classify the memes in several categories related to the emotional content of them. For the proposed system construction, we used different strategies, and the best ones were based on deep neural networks and a text categorization algorithm. We obtained results analyzing the text and images separately, and also in combination. Our better performance was achieved in Task A, related to polarity classification.

## introduction the kind of information usually shared on social media can be in various modalities such as text, images, audio, or video. in spite that this kind of information is shared in a recreational way, many institutions have paid attention to what the users or citizens shared on the internet. an aspect that has been analyzed in the past is the polarity of the text (for instance tweets, reviews, news). the polarity reflects if the text has negative, positive, or neutral content. more recently, other kinds of data have gain attention, for instance, images. the most popular and descriptive mix of image and text are the memes. richard dawkins proposed the term meme referring to mimeme in greek, that is associated with imitation @xcite . he expressed that a meme is a form of social engendering or cultural propagation. memes are important because they express emotion, joke, humor, or something that only using words could not be possible. the automatic classification of memes has very important challenges. the meaning of a meme could be interpreted in different ways by different people. furthermore, for analyzing and classifying the meaning of a meme, image and text processing must be tackled and combining them is the best way to reach a good classification result @xcite , which is not precisely easy to achieve. some works have tackled the meme classification problem. @xcite proposed a method for detecting only sarcasm tone in memes. for the sarcastic detection, a set of typo-graphic only memes where used. the authors extracted the text from the images and used standard tf-idf approach over the words. finally, they reported a result of an accuracy of 88% with a multilayer perceptron classifier from a dataset called memebank obtained from the instagram social network using the hashtags #sarcasm, #sarcastic and #irony as positive examples and #motivational and #inspirational as negative. @xcite ; peirsoon and tolunay, (2018), and kanai, (2016) dealt with meme classification. psychological study and commercial marketing are some of the primary motivations for sentiment analysis from an image with text; that is, the classification of memes can be useful in marketing, advertising, trend analysis, so on @xcite . also, performing this task using automatic techniques allows the optimization of response times in previous analyzes. in this paper, we present the proposed system to tackle for memotion analysis task from semeval 2020. in the following sections is described in more detail our proposal. the rest of the paper is organized as follows. section 2 describes the data and the task description. section 3 describes our approach to solve the problem and our implemented system. section 4 details our experimental results, and finally, conclusions are given in section 5. ## dataset and task description the memotion analysis task was divided into three sub-tasks: a, b, and c. the overall task description can be consulted in the task overview @xcite . task a is for sentiment classification, that is, given a meme, the system must classify its content as positive, negative, or neutral. task b is for humor classification where the system has to identify the kind of humor expressed by the meme. the types of humor considered were sarcastic, humorous, offensive and motivational, but a meme can have more than one of these categories. here, a meme can have one, more or none categories. task c is for scales of semantic classes where the goal is to quantify the degree of emotion expressed in the meme, these quantifications are divided into not (0), slightly (1), mildly (2), and very (4), all of them for the sarcastic, humorous, and offensive emotions. the sizes of the datasets provided by the competition organizers were, for the training dataset, a total of 7000 memes, 1000 memes for the trail, and finally, 2000 meme images for the test. the datasets had an associated file, with the contained text in the meme. for a profound description of the datasets see @xcite . the text used to feed all our models was obtained from the concatenation of the fields text ocr, text corrected and image name. this concatenation was done looking for a more amount of text no matters if was duplicated or not. ## experimental results we have two types of experimental results, one without the test and one evaluated on it. the evaluation without the test set was made using the predicted labels from the stacked generalization. with that, we obtained unbiased predictions of the whole training set. we show the results of the xgboost and our final method over the training set on the figure 5 shows the confusion matrix for the task a on the test set, here, we can see that the model has problems classifying the negative and neutral sentiment. on task b, the confusion matrices on figure 6 show that our solution struggles to identify the type of humor. for task c, the prediction is more complicated. figure 7a shows that the model fails to detect very humorous memes. something similar 7b with problems also on not sarcastic memes. figure 7c follows the same trend for offensive samples. the motivational detection on figure 7d shows that most of the errors occur assigning the negative label to positive examples. p o s n e g predicted label pos neg true label 41 404 142 1291 task b -humour (a) p o s n e g predicted label pos neg true label 83 338 288 1169 task b -sarcasm (b) p o s n e g predicted label pos neg true label 83 624 152 1019 task b -offensive (c) p o s n e g predicted label pos neg true label 537 651 315 375 task b -motivational (d) figure 6: confusion matrices for task b n o t s li g th ly m il d ly v e ry predicted label not sligthly mildly very true label 71 148 226 0 82 216 356 0 83 202 319 1 23 52 99 0 task c -humour finally, the official results for our participation are in ## conclusions this paper describes our system submitted to memotion analysis semeval 2020 task. the proposed system was based on deep learning, specifically using the inception-resnetv2 as a base model. unfortunately, we reached the best results using in a separate way the text and the images instead of using both. we design our system based on task a in which we obtained six general position (over 35 results) which is a good result for our team, nevertheless, in task b and c our results were in 24 and 25 positions.