SESAM at S em E val-2020 Task 8: Investigating the Relationship between Image and Text in Sentiment Analysis of Memes (2020)

Abstract: This paper presents our submission to task 8 (memotion analysis) of the SemEval 2020 competition. We explain the algorithms that were used to learn our models along with the process of tuning the algorithms and selecting the best model. Since meme analysis is a challenging task with two distinct modalities, we studied the impact of different multimodal representation strategies. The results of several approaches to dealing with multimodal data are therefore discussed in the paper. We found that alignment-based strategies did not perform well on memes. Our quantitative results also showed that images and text were uncorrelated. Fusion-based strategies did not show significant improvements and using one modality only (text or image) tends to lead to better results when applied with the predictive models that we used in our research.

## introduction semeval 2020 task 8 @xcite is a sentiment analysis task targeted at memes foot_0 divided into three sub-tasks of increasing complexity: sub-task a is predicting the sentiment polarity of a meme, sub-task b is a multi-label binary classification task which aims to predict whether a meme is humorous, offensive, sarcastic and/or motivational (it can also have neither of these attributes), sub-task c is a multi-output ordinal classification task which aims to predict the degree of humour, offence, sarcasm and motivation of a meme. the dataset used for this task contains memes images whose text has been extracted by optical character recognition (ocr) and manually corrected when needed. each meme is annotated on different aspects: sentiment polarity for sub-task a and the degree of humour, sarcasm, offence and motivation for sub-tasks b and c. memes sentiment analysis is a challenging task as memes are multi-modal, rely heavily on implicit knowledge, and often use humour and sarcasm. while this topic is of growing interest for nlp community, the way image and text interact in memes has barely been explored, leading to sub-optimal representation learning. in an attempt to shed some light on the role of both modalities, we investigate their correlation and their impact on each sub-task prediction. our code is available at https://github.com/bonheml/sesam . ## related work sentiment analysis of text is a very active research area which still faces multiple challenges such as irony and humour detection @xcite and low inter-annotator agreement caused by the high subjectivity of the content @xcite . research has been extended to multimodal sentiment analysis during the last years @xcite , but the focus was mostly on video and text or speech and text. the specific multi-modality of memes in sentiment analysis has only been addressed recently by @xcite , who investigated their correlation with other comments in online discussions. the growing usage of memes as an alternative medium of communication on social media has also recently drawn the attention of the online abuse research community. @xcite studied the propagation of memes posted by fringe web communities foot_1 , and their influence and transmission between different social media. @xcite performed hate speech detection on memes and showed that images were more important than text for the prediction. however, as pointed out by @xcite , memes completely make sense only if one takes both text and image content into account. these modalities can also lead to totally different perceived sentiment when recombined. for example, a meme whose image is a grumpy cat and the text is "happy birthday" will have a very different sentiment from a meme with the same text but with an image of a happy puppy. we argue that having a better understanding of both modalities interaction will contribute to more informed joint representations and is a crucial topic to explore. thus, we investigate the impact of multiple embeddings applied to both modalities on several models with different types of decision boundaries and verify the consistency of our findings assessing the embeddings across the three different sub-tasks. as our main focus is to study the impact of representations used, we chose simple classification models from scikit-learn @xcite such as k nearest neighbours or gaussian naïve bayes over more complex ones such as the deep learning architecture composed of three bidirectional gated recurrent units networks with contextual intermodal attention proposed by @xcite . we did not perform any hyper-parameter tuning. in multi-view representation learning, different techniques can be used to represent the views, depending on the nature of the relationship between them @xcite . when they share latent traits, one can use alignment to project their embeddings into a common space given a constraint (e.g., distance, correlation) @xcite . on the other hand, if they are complementary, fusion techniques will be more useful as they will group the meaningful latent variables of each view into a compact representation @xcite . in section 3.1 we assess the usefulness of aligned representation for memes by investigating the possible correlations between images and memes. then, in section 3.2, we study the added value of voter-based fusion techniques such as the one proposed by @xcite in their work on multimodal sentiment analysis. ## experimental setup data cleaning and preprocessing we manually added the text values of seven memes which had neither ocr nor corrected text values in the training dataset and removed urls corresponding to meme sources from transcribed texts. a number of websites used for meme generation add their url to the final meme, and this was often caught and transcribed by the ocr extraction. following @xcite study on text preprocessing, we did not perform any lemmatisation or lowercasing. to obtain one text embedding per meme, the text of each meme was vectorised using a pre-trained universal sentence encoder (use) @xcite retrieved from tensorflow hub @xcite . the images were processed with xception @xcite , pre-trained on imagenet @xcite , and the penultimate layer was used as embedding. ## results this section provides an analysis of our results at each step of our experiment. first, we investigate the results on the dev dataset which guided our model selection for the evaluation phase. the results retrieved from codalab during the evaluation phase for the selected models are then analysed, and we finally conclude with the analysis of the scores provided during the final ranking. ## conclusion we have provided an overview of the impact of different representations on meme sentiment analysis. we tested alignment-based and fusion-based techniques on a range of models on each sub-task. while none of them seemed to be beneficial for the different sub-tasks, we found that 1) alignment-based techniques were not suitable for meme analysis as image and text of memes are not correlated 2) using only one modality (text or image) tends to perform better than a combination of both when we use standard (i.e. non-deep learning) predictive models. however, these conclusions should be taken with caution as the scores obtained on the dev and testing datasets vary greatly and other factors, such as the label distribution of the dataset can also have influenced these results. finally, we argue that a more adapted image representation, possibly enriched with contextual knowledge, as well as more complex fusion techniques, may be promising to explore.