PRHLT - UPV at S em E val-2020 Task 8: Study of Multimodal Techniques for Memes Analysis (2020)

Abstract: This paper describes the system submitted by the PRHLT-UPV team for the task 8 of SemEval-2020: Memotion Analysis. We propose a multimodal model that combines pretrained models of the BERT and VGG architectures. The BERT model is used to process the textual information and VGG the images. The multimodal model is used to classify memes according to the presence of offensive, sarcastic, humorous and motivating content. Also, a sentiment analysis of memes is carried out with the proposed model. In the experiments, the model is compared with other approaches to analyze the relevance of the multimodal model. The results show encouraging performances on the final leaderboard of the competition, reaching good positions in the ranking of systems.

## introduction nowadays, social networks have become one of the main means of messages spread. shared information usually consists of different types of content. in this sense, we can find publications of videos, images, texts or audios. memes are examples of posts that usually combine image and text. daily the number of memes on popular sites like twitter and facebook increases considerably. this makes automatic memes analysis a very important task at present. often, memes contain subjective information, and can have ironic or sarcastic content. irony and sarcasm are forms of figurative speech in which authors write the opposite of what they mean @xcite . irony is an umbrella term whereas sarcasm is considered by many researchers a more direct and sometimes somewhat aggressive form of irony. therefore, the analysis of memes requires an extensive study that combines several tasks. on the one hand, it is necessary to design multimodal systems that can handle both textual and visual information. on the other hand, different studies should be carried out to obtain all the semantic information that the texts may contain. in this way, the study can involve sentiment analysis, irony or sarcasm detection, and so on. offensive language detection is an important task among them. it has gained great interest with the growth of the spread of offenses on internet and its implication in society. the memotion analysis shared task has been organized in order to bring the attention towards internet memes processing. the three subtasks defined in the task are sentiment classification (subtask a), humour classification (subtask b) and scales of semantic classes (subtask c). the main goal of the subtask a is to classify a meme as positive, negative or neutral according to sentiment content. the subtask b aims to identify the type of humour expressed among the categories: sarcasm, humour, offense and motivation. it should be noted that many researchers do not consider sarcasm as a humour category since irony/sarcasm partially overlap with humour @xcite . finally, the subtask c focuses on quantifying the level to which a particular category is expressed. we propose a multimodal model that combines the analysis of textual and visual information. the pretrained bert base model is used for the text processing, and a pretrained vgg model for images. features obtained from both text and image analysis are combined to feed a simple classifier that obtains the final categories. we perform an ablation study in course of the proposal design to analyze the importance of the multimodal model. furthermore, we analyze different models rather than bert and vgg, including traditional machine learning models such as support vector machines and logistic regression. the rest of the paper is organized as follows. next section briefly describes the principal approaches used in multimodal models. the details of the proposed methodology for each subtask and the dataset are described in section 3. experimental results are presented in section 4, and finally the study is concluded in section 5. ## related work recently, different proposals for some tasks have focused on multimodal analysis. in general, strategies can be divided into two approaches @xcite , regardless of whether they are based on deep learning or not. on the one hand, some models use a feature level fusion approach, where each input source, for instance text or image, is processed to extract a set of features. then, the features sets are put together for the final decision @xcite . on the other hand, other models carry out a complete processing of each input source and the fusion is made at the decision level @xcite . our proposal is based on the first category. ## methodology and dataset the dataset provided for the task consists of a set of 7000 memes @xcite . some information is supplied for each meme with the image, such as the url and the text extracted from the image. for the latter one, the text extracted by an optical character recognition (ocr) system and the corrected text are given. each meme contains labels for the five categories analyzed in the task. the categories and their labels are listed below. the sentiment analysis category corresponds to the subtask a, where a meme must be classified as positive, neutral or negative, considering very positive as positive, and very negative as negative. the rest of the categories are used in the subtasks b and c. in the subtask b, the category present in each meme must be identified. the memes can contain none, one or more categories. in the subtask c, each category is analyzed independently to identify its level according to the labels in the dataset. ## experiments and results to measure the performance of the approaches we use the metrics proposed in the memotion analysis task. for the subtask a: macro f1, and for the subtasks b and c: macro f1 for each of the subtasks, and then the average. ## ablation analysis ## conclusion in this work, we studied the problem of internet memes analysis as part of a shared task of semeval 2020. we proposed a multimodal approach for each subtask that combines both textual and visual modalities. the architecture contains the bert model for text processing and vgg19 for images. we obtained a good ranking position regarding the second subtask. however, our experimental results do not allow a thorough analysis of the proposed approach, obtaining less than 0.65 of macro f1 in the subtask b and a lower performance for subtasks a and c.