M emo SYS at S em E val-2020 Task 8: Multimodal Emotion Analysis in Memes (2020)

Abstract: Internet memes are one of the most viral types of content in social media and are equally used in promoting hate speech. Towards a more broad understanding of memes, this paper describes the MemoSys system submitted in Task 8 of SemEval 2020, which aims to classify the sentiment of Internet memes and provide a minimum description of the type of humor it depicts (sarcastic, humorous, offensive, motivational) and its semantic scale. The solution presented covers four deep model architectures which are based on a joint fusion between the VGG16 pre-trained model for extracting visual information and the canonical BERT model or TF-IDF for text understanding. The system placed 5th of 36 participating systems in the task A, offering promising prospects to the use of transfer learning to approach Internet memes understanding.

## introduction among all social networks, internet memes are one of the most widespread types of content and have a huge virality factor. these memes are usually thought of as countless derivatives of template images, that become ingrained in our culture and very easily convey a message. while their most common purpose is to serve in a humorous or ironic way, they can unintentionally exhibit racism or promote hate speech that goes undetected because we lack a form of meme understanding at the moment. the attributes that illustrate the anatomy of a meme are the textual information, which consists of a couple of highly informal words, along with one (or more) images based on either tv shows or "meme characters". these characters are symbolic entities that set expectations on what kind of expression it conveys @xcite , an example being "bad luck brian", which usually presents an unlucky, yet funny situation. the trouble in understanding memes is caused by their humor -they cannot be taken individually, as standalone jokes. the best analog to internet memes is "joke cycles", which are large clusters of mutually related texts @xcite , sometimes relying on implicit references that can impose trouble in understanding even in human cognition. secondly, the incredible amount of variations that can emerge from a meme, such as gross occlusion by textual elements, mutation of the meme characters, or even collages, makes it very challenging to discover features that can contribute to the classification. an approach to tackle memes understanding has been proposed in the memotion analysis task @xcite , which consists of the following sub-tasks: task a -sentiment classification given an internet meme, the first task is to classify it as a positive, negative or neutral meme. task b -humor classification given an internet meme, the system has to identify the type of humor expressed. the categories are sarcastic, humorous, offensive and motivation meme. a meme can have more than one category. ## dataset the dataset proposed @xcite for the task consisted of 7000 memes, including their caption extracted and manually corrected. these memes include a great level of diversity, do not exhibit a similar structure, and do not represent a subset of meme templates or characters. additionally to having no limitation regarding the content of the memes, there is also extra noise added by being only a part of the actual image, as in a screenshot of the phone screen or from social media platforms. ## preprocessing the images that were chosen for the dataset present a big variety of quality and scales. we resized all images to address this issue to the size of 320x320, which seemed to preserve most of the features of the sub-images of the memes. given the multiple font styles, sizes and placement of the caption, usually occluding a big part of the image, we aimed to ease the noise in the visual information by using the pre-trained east text detector @xcite to replace the textual elements with white boxes. this allowed the neural network to learn to ignore such areas and to easier distinguish the features that contribute to the classification. we have noticed that data augmentation, such as mirroring and random cropping of the images, helps the fine-tuning of the model. to train the models further, we split the dataset 70%-30% into training and validation subsets. ## models for tasks b and c we trained four models for each attribute (humor, motivation, sarcasm, offense). transforming the task b from multi-label classification to multiple binary classifications led to better results in all experiments, showing that our models were not able to learn and use the potential interdependencies between the attributes in a meaningful manner. to address the highly imbalanced data in all three sub-tasks and avoid biasing the prediction towards the majority classes, we applied a weighting factor in the loss computation that is inversely proportional to the number of samples of that class. we used adam (kingma and ba, 2014) optimization for training, a learning rate starting with 0.0001 and a mini-batch size of 32. the stopping condition was observing whether any improvement has been encountered in the last three epochs. for implementation the tensorflow 2.0 libray was used with the built-in keras api. ## analysis of results the results presented in ## interpretability it is nontrivial to understand the predictions made by these models, as memes understanding requires a high level of abstraction, cultural knowledge, and subjectivity that are hard to interpret even in human reasoning. based on prediction results, we observed that less complex memes based on templates, such as "success kid" meme, tend to be classified correctly. we considered this is probably linked to image understanding and we used the grad-cam approach @xcite to gather some visual explanations of the high level features the vgg16 network learns after training. it was surprising to see that many of the highlighted regions did not reveal any important information. however, on a certain subset, it seems that it highlights the face of the characters present), showing the strong tie meme understanding has with the character it is based on. ## challenges as the dataset contains a very diverse collection of memes, it was difficult for all participants to score way above chance (33% for task a, 50% for task b, 25% for task c). although deep neural networks are powerful, it is of great importance to have enough training data to generalize learning. extending the current dataset requires a certain level of consistency in the labeling process. people may have, for example, a different understanding of the humor depicted in the same meme and an even inaccurate rating of the sarcasm, the humor of offense. since the images are weakly labeled, it may lead to poor generalizability of the network. we believe that even more cleaning has to be made aside from removing caption, but it is a big challenge to automatically reduce the noise of the images, splitting them into components or removing the outer border from screenshots. the high-class imbalance also raised difficulties, even with balancing techniques as oversampling and class weights. for example, the dataset contains 1544 memes (out of 7000) labeled as being not sarcastic. this brings issues in classifying the input as sarcastic or not (task b) because the model does not have a broad range of samples to learn from what the lack of sarcasm looks like, but rather naturally over-fit features of the small selection. having in mind the goal of identifying hate speech, it is crucial to address the interpretability of the model and the multimodal approaches add an increasing level of complexity. ## future work the experiments we worked on have shown that pre-trained deep neural networks, along with a certain level of pre-processing of the raw images give promising results in the classification of memes. a very big leap towards meme understanding would be acquiring a bigger and cleaner dataset to analyze to what extent the neural networks can generalize and learn useful features, through accurate labeling methods that are less sensitive to subjectivity. one way would be to focus on viral memes that already have a broad audience such as social media posts and derive their sentiment from facebook reactions or reddit comments. furthermore, by focusing more on reducing the noise and emphasizing the features that contribute most to a person's understanding of a meme, it is more likely to improve the accuracy. an example would be the knowledge associated with the most prevalent meme characters already documented by websites like knowyourmeme. ## conclusions our results show that fusions of pre-trained models have the potential to overcome some of the challenges in memes understanding. this approach can distinguish in certain cases the humor, sarcasm, and most important, offensiveness, being a tool that we can work upon and use further to act against hate speech on social media networks, to asses the quality of the existing content and to advance the automatic generation of memes.