LT 3 at S em E val-2020 Task 8: Multi-Modal Multi-Task Learning for Memotion Analysis (2020)

Abstract: Internet memes have become a very popular mode of expression on social media networks today. Their multi-modal nature, caused by a mixture of text and image, makes them a very challenging research object for automatic analysis. In this paper, we describe our contribution to the SemEval-2020 Memotion Analysis Task. We propose a Multi-Modal Multi-Task learning system, which incorporates “memebeddings”, viz. joint text and vision features, to learn and optimize for all three Memotion subtasks simultaneously. The experimental results show that the proposed system constantly outperforms the competition’s baseline, and the system setup with continual learning (where tasks are trained sequentially) obtains the best classification F1-scores.

## introduction while internet memes initially seemed to be restricted to users of specific communities online, memes have rapidly spread among the wider user base, having now claimed its status among emojis and reaction gifs as a common mode of expression on a variety of online platforms. their user base is now so varied, it ranges from youngsters, boomers to marketers. despite the lack of consensus on the exact meaning of the original term "meme", as coined by @xcite , we follow the growing consensus in communications research in employing the specified definition of internet memes, as "amateur media artifacts, extensively remixed and recirculated by different participants on social media networks" @xcite . internet memes can take on many different roles in the online communicative sphere. they may be used for entertainment purposes. they have also been described as a form of visual rhetoric @xcite , functioning as persuasive devices, all the while disguising their message under a layer of humour @xcite . memes also play a significant role in the practice of online trolling, in which they are often the preferred mode of expression because of their potential for spreading provocative and attention-grabbing humour @xcite . they have been described both as speech acts @xcite and performative acts, involving a conscious decision to either support or reject an ongoing social discourse @xcite . in their capacity as speech acts, the range of functions the memes can perform, extend to the range of functions of any other type of speech act, such as the use of memes to question something @xcite . in terms of their form, up to 13 different types of memes have been identified: ranging from simple text-based memes to quotes, rage comics and drawings @xcite . in their most recognizable form, memes often follow specific image macros @xcite , or templates with a recognizable image and a text-overlay which changes in individual occurrences of the meme. this multimodal aspect of memes causes meaning to be created on various levels: within each mode and in the interaction between them @xcite . recent research focusing on automatic genre classification of (mostly political) memes @xcite will help facilitate data collection, an important aim given the current lack of gold standard datasets of memes. the multimodal aspect has caused research on the automatic detection of internet memes to be divided in two separate fields: nlp research on memes tends to focus only on the textual aspect, whereas the field of computer vision primarily takes into account the visual aspect. the semeval 2020 memotion task @xcite aims to introduce the novel task of emotion detection of internet memes to the research fields of nlp and computer vision. the task aims to answer the need for broadening the focus of the study of social media data to a more hybrid perspective @xcite and to bridge the gap between this separation of modalities. in this paper, we describe our contribution to the semeval-2020 memotion analysis task. we propose a multi-modal multi-task learning system, which incorporates "memebeddings", viz. joint text and vision features, to learn and optimize for all memotion subtasks simultaneously. ## related research the wide variety of communicative functions has made memes a topic of interest for various research fields. the cohesive effect of internet memes on communities has been studied in sociolinguistics @xcite as well as their potential to generate awareness for social and political issues @xcite . the impact of certain types of humour on the potential of a meme going viral is of particular interest within communication studies @xcite . internet memes have also been a subject of study in the field of computational creativity with attempts to automate the meme generation process @xcite and in the field of information retrieval which is interested in the possibility of personalized searches for memes @xcite . the aspect of multimodality makes the internet meme an especially challenging research object for automatic detection. nevertheless, internet memes play a significant role in users' online expression and thus constitute a wealth of possible information on uncovering user sentiment. the automatic detection of sentiment expressed in memes may help advance research into the viral spread of internet phenomena, specifically regarding the principle of emotional contagion @xcite . despite the current lack of research in automatic sentiment detection of internet memes, an exception being french (2017), who researched the extraction of its inherent sentiment by linking the memes to the surrounding user comments, the focus of the emerging sentiment detection research promisingly lies in multimodal classification @xcite . while great efforts have been made for the sentiment classification on textual data @xcite , recent attempts to incorporate image-based features from the field of computer vision like ocr and face recognition have uncovered the need for improvements in the text analysis of memes due to their short, complex nature @xcite . sentiment classification has been applied to image data for the purposes of automatic tag predictions for images uploaded on social media @xcite , which in turn will help optimize image search algorithms by providing a large collection of tagged image data. the importance of joining together the insights gained in the study of the two modalities of memes becomes evident when we consider the complexity such multimodal communication modes add to tasks such as the automatic detection of offensive discourse online @xcite when compared to offensive language detection in textual data @xcite . the semeval memotion task not only aims to contribute to the field of automatic detection of sentiment in internet memes, but also to the extraction of more fine-grained information such as sarcasm, humour and offensiveness. distinguishing these three types of humour is not an easy task, since even the type of humour typically found in non-offensive memes, tends to skirt the boundaries of what is acceptable. it is in this often politically incorrect capacity that their cohesive potential lies @xcite . due to the ambiguous nature of the categories to be identified, the task extends to the detection of the degree (not, slightly, mildly, very) to which the humour type identified is present in the meme. as the meme generation experiments of @xcite showed, the perceived humorous nature of memes is often based on the macros used and the meaning these already carry. since these macros are multimodal, the automatic detection of the meaning they carry must rely on a combination of text-and image-based features. the characteristics of sarcasm forming the basis of the features used in automatic detection methods, such as incongruity, shared knowledge, plausibility and ridicule @xcite , originate from traditional theories of humour, such as raskin's (1989) three sources of humour: incongruity, arousal-safety and disparagement. sarcasm detection on data that is not purely textual, has only been performed on typographic memes (memes consisting only of text, but often formatted with a variety of possible fonts) using a multi layer perceptron, resulting in an accuracy score of 88% @xcite . while the detection of textual sarcasm has achieved some important advances, among which the use of semi-supervised pattern extraction, lexico-semantic knowledge bases and data-driven methods to identify implicit sentiment @xcite , the detection of sarcasm in multimodal environments will need to take into account new features for sarcasm as produced by the image and the interaction between text and image. the goal of the memotion analysis task is to investigate methods suited to the detection of finegrained information in memes, such as the type of humor or offense present. the purposes of the automatic detection of sentiment and humour types may aid the general aim of detection tasks on social media data, namely gaining a better understanding of online communities, but may also help gain important insights on the link between popular ("viral") memes and types of humour they represent to help inform communication strategies online. ## system architecture this section describes the general system architecture we designed for the semeval 2020 memotion analysis task, which comprises the following subtasks: we propose a unified multi-modal multi-task system that learns meaningful representations of memes. independent networks for each task, while being better at encoding information about the particular task, do not capture a lot of context about the memes outside of the task. a transformer, for example, which is independently trained to predict sarcasm in memes, while excelling at the task, will not encode anything meaningful about memes in the context of other values like humour and motivation. to address this issue, we propose a joint multi-task network for memes which learns embeddings and optimizes for all three tasks simultaneously. for training the system, these three tasks were further divided into the following five sub-tasks: first, the memes are encoded with the help of bert @xcite and visual features oriented at understanding the contents of the image, other than the text (see section 4). after obtaining these "memebedding" encodings, viz. the joint embeddings from text and image, this information is passed along to the 5 heads for classification for each sub-task. cross-entropy loss is computed for each head independently and the weighted average of the combined loss is jointly optimized with the adam optimizer. the following section describes the encoding and featurization in more detail. 4 experimental setup the encoding step derives information from 2 different sources, a pre-trained bert masked lm, and a visual feature extractor. bert, the bidirectional encoder representations for transformers @xcite , leverages the bidirectional transformer for masked language modelling. it demonstrates that a language model which is bidirectionally trained has a deeper, more salient understanding of language than the previous uni-directional language models. fine-tuning bert has been leveraged in many downstream tasks and is the state-of-the-art for 11 such tasks like glue @xcite , squad @xcite and multinli @xcite . we use the standard base-uncased bert pre-trained model available from the hugging face transformers package foot_0 , to encode the meme text into a 768-dimensional vector. ## results and analysis we conducted various experiments to understand the working of the multi-task learning (mtl) model better. the "multi-modal mtl average loss" flavor was our official submission for the semeval task. "multi-modal mtl weighted average loss" and "multi-modal mtl continual learning" were evaluated afterwards on the held-out test data. ## conclusion in this research, we have focused on constructing a unified network for meme analysis that incorporates both textual and visual information. the system has shown to perform well on multiple tasks at the same time. in future research, it would be worth exploring the joint "memebeddings" it creates by combining the visual and textual features, to understand what the network is able to encode. the visual features are tuned on memes and thus tailored for this domain. the bert text embeddings, however, are generic and could be better tailored for this context as well. another interesting direction for future research would be to add further sub-tasks to the multi-task training. a task to predict the "premise" (the underlying argument or concept) of a meme would be highly interesting as it would help the network further understand the general concept behind a meme.