S em E val-2020 Task 8: Memotion Analysis- the Visuo-Lingual Metaphor! (2020)

Abstract: Information on social media comprises of various modalities such as textual, visual and audio. NLP and Computer Vision communities often leverage only one prominent modality in isolation to study social media. However, computational processing of Internet memes needs a hybrid approach. The growing ubiquity of Internet memes on social media platforms such as Facebook, Instagram, and Twitter further suggests that we can not ignore such multimodal content anymore. To the best of our knowledge, there is not much attention towards meme emotion analysis. The objective of this proposal is to bring the attention of the research community towards the automatic processing of Internet memes. The task Memotion analysis released approx 10K annotated memes- with human annotated labels namely sentiment(positive, negative, neutral), type of emotion(sarcastic,funny,offensive, motivation) and their corresponding intensity. The challenge consisted of three subtasks: sentiment (positive, negative, and neutral) analysis of memes,overall emotion (humor, sarcasm, offensive, and motivational) classification of memes, and classifying intensity of meme emotion. The best performances achieved were F1 (macro average) scores of 0.35, 0.51 and 0.32, respectively for each of the three subtasks.

## introduction in the last few years, the growing ubiquity of internet memes on social media platforms such as facebook, instagram, and twitter has become a topic of immense interest. memes are one of the most typed english words @xcite in recent times which are often derived from our prior social and cultural experiences such as tv series or a popular cartoon character (think: "one does not simply" -a now immensely popular meme taken from the movie lord of the rings). these digital constructs are so deeply ingrained in our internet culture that to understand the opinion of a community, we need to understand the type of memes it shares. @xcite aptly describes them as performative acts, which involve a conscious decision to either support or reject an ongoing social discourse. the prevalence of hate speech in online social media is a nightmare and a great societal responsiblity for many social media companies. however, the latest entrant "internet memes" @xcite has doubled the challenge. when malicious users upload something offensive to torment or disturb people, it traditionally has to be seen and flagged by at least one human, either a user or a paid worker. even today, companies like facebook and twitter rely extensively on outside human contractors from different companies. but with the growing volume of multimodal social media it is becoming impossible to scale. the detection of offensive content on online social media is an ongoing struggle. offenseeval @xcite is a shared task which is being organized since the last two years at semeval. but, detecting an offensive meme is more complex than detecting an offensive text -as it involves visual cues and language understanding. this is one of the motivating aspects which encouraged us to propose this task. analogous to textual content on social media, memes also need to be analysed and processed to extract the conveyed message. a few researchers have tried to automate the meme generation @xcite process, while a few others tried to extract its inherent sentiment @xcite in the recent past. nevertheless, a lot more needs to be done to distinguish their finer aspects such as type of humor or offense. the paper is organised as follows: related work is described in section 2. the proposed task is described in section 3. data collection and data distribution is explained in section 4 while section 5 demonstrates the baseline model. section 6 shows the reason for considering macro f1 as evaluation metric. in section 7, participants and the top performing models are discussed in detail. section 8 shows the results, analysis and the takeaway points from memotion 1.0. finally, we summarise our work by highlighting the insights derived along-with the further scope and open ended pointers in section 9. ## related work identifying the text in the image is as important as the context of the image, so we present the related work in two parts, one involving the models used to extract text from the meme and the other on the analysis of memes. @xcite proposed one of the first cnn based approach for text recognition to classify words into fixed set of character texts. (2019) uses an n-gram model to correct the ocr text extracted. @xcite performed a comprehensive literature review on handwriting character recognition. a survey analysed in @xcite shows an overview of different aspects of ocr and discuss different methods at resolving issues related to ocr. template matching using contours is used in @xcite to recognize visual characters from real-world scenarios. while there are not many works involving direct classification of emotions on memes. early works on the detection of offensive content on online social media is offenseeval @xcite shared task, organized at semeval since 2019. the latest entrant "internet memes" @xcite has doubled the challenge. detecting an offensive meme is more complex than detecting an offensive text as it involves visual cues and language understanding. automate of meme generation process are explored in @xcite while others have sought to extract the memes' inherent sentiment @xcite . nevertheless, this challenge proves the necessity of more research in the field of multimodal approaches to detect and classify the sentiments of memes. ## the memotion analysis task memes typically induce humor and strive to be relatable. many of them aim to express solidarity during certain life phases and thus, to connect with their audience. some memes are directly humorous whereas others go for sarcastic dig at daily life events. inspired by the various humorous effects of memes, we propose three task as follows: we have released 10k human annotated internet memes labelled with semantic dimensions namely sentiment, and type of humor that is, sarcastic, humorous, or offensive and motivation with their corresponding intensity. ## dataset to understand the complexity of memes, as discussed in the prior sections, it is essential to collect memes from different categories, with varying emotion classes. details of preparing the data-set is presented below: emotion about memes highly depends upon an individual's perception of an array of aspects within society, and could thus vary from one person to another. this phenomenon is called as "subjective perception problem" as noted in @xcite . to address this challenge, the annotation process is performed multiple times ie. each sample is provided to 5 annotators, and the final annotations are adjudicated based upon majority voting scheme. the filtered and the annotated data-set comprises to the size of 9871 data samples. in addition to these aspects, textual content plays a pivotal role in ascertaining the emotion of a meme. to understand the textual content from the memes, text has been extracted using google vision ocr apis. the extracted text was not completely accurate, therefore amt workers were asked to provide the rectified text against the given ocr extracted text, for the inaccurately extracted ocr text. data distribution: the statistical summaries are provided in for the challenge, 1k samples were provided as trial data, 6992 samples as training data while 1879 samples as test data. ## baseline model memes are the types of multi-modal content, wherein the graphical content and textual messages are self-sufficient to convey some meaning. but within the context of their dissemination, it is a specially tailored idea, that is designed to be propagated. such complex ideas cannot be conveyed as effectively by any of the constituent data modality, as by their combination. in order to fully address the system modeling tasks that use such data, it is imperative to study the efficacy of individual content modality ie. ## evaluation metric the challenge comprises of classifying the sentiment and emotion associated with a meme. the task a is a multi-class problem involved in identifying the sentiment (positive, negative, neutral) associated with a meme while the other 2 tasks b and c are multi-label classification problem associated with emotion detection. there are various evaluation metrics for multi-class and multi-label classification problem such as hamming loss, exact match ratio, macro/micro f1 score etc. the most used metric for this kind of problem is hamming loss which is evaluated as the fraction of wrongly classified label to the total number of labels. as our problem deals with different emotions associated with a meme, we have used macro f1 score that will help us to evaluate and analyse the individual class performance. ## participation and top performing systems the challenge was a great success, involving total of 583 participants, with varying submissions in different tasks comprising of 31, 26 and 23 submissions in task a, task b and task c respectively where in evaluation phase, a user is allowed for 5 submissions per day. 27 teams submitted the system description paper. a brief description of the task wise top performing models is shown below. ## memotion analysis -the next horizon! the submissions that we received also came along with their respective analytical reasoning, towards ascertaining whether an image or a text or their combination contributes towards modeling the associated emotions from memes. most of the analyses provided present conflicting views, regarding the importance of a particular content modality. this essentially reinstates the requirement of further investigations into better approaches towards modeling the affect related information from the multi-modal content like memes. the complexity of understanding the emotions from a meme arises primarily due to the interaction of both image and embedded text. although, few results reported are better when evaluating over either image or text, a human always attempts to take cognizance of both image and text to understand the meaning intended. the challenge is highlighted more by memes which are domain specific, i.e. based on a popular movie or tv show. while there are several state-of-the-art deep learning based systems that leverage data intensive training approaches, that perform various tasks at par with humans on both image and especially for text, still there is a lot more space for applications involving multi-modality like memes, to drive the required progress. recently facebook proposed a challenge @xcite to classify the meme as hateful and not hateful content. at present, memes have become one of the most prominent ways of expressing an individual's opinion towards societal issues. further on classifying the emotion of memes, this work can be extended as follows: