Urszula Walińska at S em E val-2020 Task 8: Fusion of Text and Image Features Using LSTM and VGG 16 for Memotion Analysis (2020)

Abstract: In this paper, we describe the entry to the task of Memotion Analysis. The sentiment analysis of memes task, is motivated by a pervasive problem of offensive content spread in social media, up to the present time. In fact, memes are an important medium of expressing opinion and emotions, therefore they can be hateful at many times. In order to identify emotions expressed by memes we construct a tool based on neural networks and deep learning methods. It takes an advantage of a multi-modal nature of the task and performs fusion of image and text features extracted by models dedicated to this task. Moreover, we show that visual information might be more significant in the sentiment analysis of memes than textual one. Our solution achieved 0.346 macro F1-score in Task A – Sentiment Classification, which brought us to the 7th place in the official rank of the competition.

## introduction nowadays, spreading of abusive content in social media is becoming an increasingly serious problem. due to the anonymity provided by the internet, users posting offensive content feel that they can ignore the law with impunity. a remedy to this problem is automatic detection and elimination of such content, and a tremendous progress was made in the past years in the area of sentiment analysis of text @xcite . however, little research was done when it comes to image sentiment analysis, which is a grave omission, as people frequently express emotions in a multi-modal way, using both text and image. a particular example of such multi-modal and potentially harmful content are internet memes. in this paper we tackle a problem of memotion analysis, which is the 8-th task of semeval-2020 competition. dataset with memes in english, provided by @xcite defines the following sub-tasks: task a -sentiment classification given an internet meme, classify it as a positive, negative or neutral meme. task b -humor classification given an internet meme, identify expressed humor type: sarcastic, humorous, offensive, motivation. a meme can express more than one humor type at the same time. task c -scales of semantic classes quantify the extent to which humor types are being expressed by the given meme, using the following 4-point scale scale: not (0), slightly (1), mildly (2) and very (3). we participated only in the task a and the consecutive analysis is based only on this task. the dataset provided for this task contains 6946 unique, complete examples. each example consists of the following features: an image name, an image url, text extracted using ocr from the image, corrected text and overall sentiment. the sentiment is expressed on the following 5-point scale: very positive, positive, neutral, negative, negative, very negative, but in the task a defining whether given example is positive, neutral or negative is sufficient, i.e., very positive and positive were considered as a single class positive, while negative and very negative as a single class negative. figure 1 presents the distribution of the classes across the whole dataset and we observe that they are heavily imbalanced. even if we collapse categories into a single negative, a single positive and a neutral category, the problem still remains. while developing our solution we split provided dataset into training, validation and test set in the proportion 8:1:1. organizers of the competition provided us also with the official test set for the final evaluation. ## related work sentiment analysis of text is an established field within the computer science, employing numerous approaches such as dictionaries @xcite , ontologies @xcite , statistical approaches (turney and littman, 2003), machine learning @xcite or a combination thereof. recently, deep learning became a very popular tool, e.g., @xcite . the area of image sentiment analysis is much younger, and rooted mostly in machine learning and deep learning techniques, e.g., @xcite , however, images alone are rarely considered and instead the problem of multi-modal classification is considered @xcite . a notable example is work by @xcite , concerned with using visual and textual information to automatically detect hate speech in internet memes. the authors built a dataset of 5,020 memes to train and evaluate a model capable of identifying hateful memes. to extract text from images, they performed ocr first and later used a pre-trained bert model @xcite to extract relevant features from the text. for image feature extraction, a pretrained vgg-16 network @xcite was used. the features for both modalities were then combined using a multi-layer perceptron (mlp). moreover, their experiments indicated how the visual modality can be much more informative for hate speech detection than the linguistic one in memes. ## experiment in order to decide whether using feature fusion has any advantages over using any single modality, we performed the following experiment. the evaluation is based on the results obtained by each method using a test set according to macro f1-score. the results on the test set are presented in the text only approach is the result of the complete network (with the dense part) as described in subsection 3.3. the image only approach consisted of features extracted using the method described in subsection 3.4 and an additional dense feed-forward network consisting of 3 dense layers with 512, 100 and 3 neurons, respectively. layers were separated with dropout with the rate of 0.1. network was trained under the same regime as described in subsection 3.3. we observe that feature fusion was the best of the considered methods (f1 score of 0.39) and we hypothesise that this was due to the richer representation obtained by feature fusion, increasing the level of flexibility to fit to the data. seeing that incorporating visual information increased f1 score by 0.07 (0.32 to 0.39), we hypothesise that the visual information is of utmost importance for the problem of sentiment analysis of memes, and that the knowledge derived from text and from images is, to some extent, complementary. this is also in line with the results of @xcite . ## competition results we took part in task a of memotion analysis competition. taking into consideration the results of experiment we obtained, our final solution was based on the model with feature fusion of lstm (textual modality) and vgg-16 (visual modality). model was trained on training set provided by organizers of memotion analysis 2020 and described in the introduction. our result on the official test set including obtained f1-scores and place in the official rank is presented in ## conclusions taking into account number of cases of public discrimination and humiliation of people on the grounds of race, ethnic origin or religion, we can undoubtedly say that dealing with these pervasive problems in social media is of utmost importance. what is more, we need to focus not only on the hate visible in text posted by people but also on memes which more and more popular way of expressing emotions. automatic sentiment analysis methods, particularly the method presented in this paper, can help to reduce harmful impact of hate in social media. as the results show, fusion of features of different modalities using neural networks can be a very powerful tool for multi-modal sentiment analysis. textual information might enrich visual information and vice versa, therefore the topic of multi-modal classification is very interesting and is undoubtedly worth further analysis also in other tasks.