NUAA - QMUL at S em E val-2020 Task 8: Utilizing BERT and D ense N et for I nternet Meme Emotion Analysis (2020)

Abstract: This paper describes our contribution to SemEval 2020 Task 8: Memotion Analysis. Our system learns multi-modal embeddings from text and images in order to classify Internet memes by sentiment. Our model learns text embeddings using BERT and extracts features from images with DenseNet, subsequently combining both features through concatenation. We also compare our results with those produced by DenseNet, ResNet, BERT, and BERT-ResNet. Our results show that image classification models have the potential to help classifying memes, with DenseNet outperforming ResNet. Adding text features is however not always helpful for Memotion Analysis.

## introduction the growing ubiquity of internet memes on social media platforms such as facebook, instagram, and twitter reinforces the importance of processing multimodal content. despite memes being so deeply ingrained in the internet culture, it is still hard to leverage ai methods to understand the underlying meanings expressed by memes. this, in turn, makes it challenging to identify certain kinds of content in these memes, such as malicious information posted on social media. to tackle this problem, semeval 2020 task 8 held a meme emotion analysis task @xcite . it consisted of three sub-tasks: (1) predicting the sentiment of internet memes as positive, neutral or negative, (2) identifying the type of humor expressed as sarcastic, humorous, offensive or motivation, and (3) quantifying the extent to which a particular emotion is being expressed. internet memes can comprise a variety of modalities such as textual, visual, and audio. adhering to the task guidelines at hand, we approach the problem by considering only the image-with-caption class of meme, which is commonly found on social media. this paper describes a system that combines bert (bidirectional encoder representation from transformers) and densenet (dense convolutional network) to classify internet memes. we find that the performance of multi-modal deep neural network is better than one model deep neural network and the performance of resnet (residual network) is worse than densenet. this paper is organized as follows. in section background, we describe memotion analysis task, the characteristics of the internet memes, prior work on multi-modal classification, prior research exploring data analysis approaches to study memes, and our contributions in this paper. then in section 3, we propose bert-densenet, a deep learning model to classify memes. we then use the models to study memes in task a, task b, and task c in section 4. in section 5, we display the results of our experiments, including comparative experiments. finally, we conclude the findings of this research and suggest directions for future work. ## background task a of this competition consists of classifying an internet meme as positive, negative, or neutral. task b consists of identifying whether an internet meme is sarcastic, humorous, offensive, or motivation. a meme can have more than one category in this case. task c consists of quantifying the scales of every type of humor. the system should classify it as positive for task a, identify it as sarcastic, humorous, not offensive and not motivational for task b, and quantify it as general (the scale of sarcasm), hilarious (the scale of humor) and not offensive (the scale of offensive) for task c. a dataset with 6,992 annotated memes is used in this task @xcite . the dataset contains a set of meme images, the corresponding textual content ocr'd and manually corrected, and the corresponding class. a meme is not only about the funny picture, the internet culture, or the emotion that passes along, but also about the richness and uniqueness of its language: it is often highly structured with a special writing style and forms interesting and subtle connotations that resonate among the readers @xcite . for example, the boromir memes with the caption of "one does not simply. . . " are derived from the movie lord of the rings. it shows that the same example can lead to the production of several memes making use of the same background image but different texts, which gives it different meanings and contexts. although the language of internet memes is a relatively new research topic, our work is broadly related to studies on multi-modal classification @xcite @xcite and multi-modal sentiment analysis @xcite @xcite . most recently, @xcite learned text embeddings using bert, the current state-of-the-art in text encoders. however, none of the above studies have applied multi-modal approaches for memes sentiment analysis. previous research on memes mainly focuses on the automatic generation of memes @xcite @xcite , but not on the analysis as we do in this task. in a typical automatic generation work, wang & wen (2015) combined textual and visual information for predicting and generating popular meme descriptions. @xcite detected and characterized internet memes with multi-modal deep learning. however, fewer studies focus on sentiment and emotion analysis of memes. one of the few works who did focus on sentiment analysis is that by @xcite , who developed a maximum entropy classifier for memes sentiment classification, which was however in an early stage of development. based on previous work, we take a multi-modal approach to do memes sentiment analysis. we propose to use keras functional api foot_0 to model the multimodal dependencies among images, text, and corresponding labels. we then introduce bert @xcite and densenet @xcite to the model. in empirical experiments, we show that our model outperforms strong discriminative baselines by large margins except for the second sub-task (b). our contributions are three-fold: ## system overview we approach the task of memotion analysis as eight classification sub-tasks (one for task a, four for task b and three for task c). four classification sub-tasks from task b are binary but others are multiclass. our model is depicted in figure 1 , with inputs at the top and outputs at the bottom. in a nutshell, our model combines bert that processes the caption text and densenet that extracts features from images. data pre-processing: firstly, we delete quotation marks because this would raise an error when reading csv files. secondly, we copy text from ocr extracted text to corrected text when the value of corrected text is null and the value of ocr extracted text is not null. we check the corresponding image and add the text to corrected text if the value of ocr extracted text and corrected text are both null. thirdly, we use wordpiece tokenisation, which can break a word into more than one sub-word. this kind of tokenisation is beneficial for memes because they contain several out of vocabulary words. ## experimental setup we tried three different groups of classifiers: (1) unimodal classification using only text (2) unimodal classification using only machine vision, and (3) multimodal classification using text and vision. data: we use the publicly released dataset for experiments. as there are eight classification tasks in total, different tasks have different proportions of labels. the details of the splitting train/dev/test are shown in the tables below ( ## results among all participating systems in this semeval task, our model achieved the 14th score both on the evaluation of task a and task c. our score on task b is however below the baseline, which we look to improve in future work. ## conclusion in this paper, we describe our entry for semeval 2020 task 8: memotion analysis. we present a method for using deep learning to classify memes by emotion. we find that image classification models help classifying memes, with densenet outperforming resnet. adding text-based features is not always helpful for memotion analysis. our proposed model outperforms baselines except for task b. we are looking to improve performance scores for task b by further tweaking our method and by testing additional parameters. we should stress that the classifier we trained is still in an early stage of development. we aim to improve it by: (i) enriching the meme datasets from social media and trying to keep the balance among classes; (ii) considering additional features (e.g. adult slang); (iii) trying other methods for combining features. our present work is limited to one type of meme made of image and text, and we aim to take other types of memes (e.g. video and audio memes) into consideration in the future.