Overview of Shared Task on Multitask Meme Classification - Unraveling Misogynistic and Trolls in Online Memes (2024)

Abstract: This paper offers a detailed overview of the first shared task on “Multitask Meme Classification - Unraveling Misogynistic and Trolls in Online Memes,” organized as part of the LT-EDI@EACL 2024 conference. The task was set to classify misogynistic content and troll memes within online platforms, focusing specifically on memes in Tamil and Malayalam languages. A total of 52 teams registered for the competition, with four submitting systems for the Tamil meme classification task and three for the Malayalam task. The outcomes of this shared task are significant, providing insights into the current state of misogynistic content in digital memes and highlighting the effectiveness of various computational approaches in identifying such detrimental content. The top-performing model got a macro F1 score of 0.73 in Tamil and 0.87 in Malayalam.

## introduction in the ever-changing landscape of online communication @xcite , memes have emerged as a remarkable phenomenon, transcending linguistic, cultural, and geographical boundaries @xcite . their ability to succinctly and often humorously convey complex ideas and emotions has made memes an integral part of digital discourse @xcite . however, this rise in meme culture has also revealed the obscene side of online content, which features misogynistic stories and trolling @xcite . we initiated the "multitask meme classification -unraveling misogynistic and trolls in online memes" competition to understand and address these critical issues through memes. this pioneering endeavor leverages goldstandard datasets to illuminate the intricate world of online memes. our competition aims to inspire the development of cutting-edge models for meme classification, primarily focusing on detecting misogyny and trolling in various languages. further, it is centered around carefully selected high-quality datasets. these datasets have been precisely annotated to establish a standard for classifying memes. these datasets represent various languages and meme categories, offering a comprehensive view of the meme landscape. let us delve into the two core tasks our competition addresses: task 1: detecting misogynistic memes: this task revolves around identifying misogynistic memes, which perpetuate harmful stereotypes and attitudes towards women. the gold standard dataset for this task spans languages like tamil and malayalam, reflecting the global reach of this issue. participants must develop models capable of analyzing textual and visual elements within memes to distinguish between misogynistic and non-misogynistic content. task 2: troll meme classification: this task broadens our perspective to encompass the classification of troll memes characterized by provocative and disruptive behavior. the gold standard dataset for this task includes languages such as kannada and telugu. participants face the challenge of categorizing memes into 'troll' and 'non-troll' categories, navigating the intricate interplay of humor, satire, and harmful intent. our competition is structured to encourage innovation and collaboration across the global research and practitioner communities. participants receive comprehensive training and development datasets, offering various memes for practical model training. evaluation of these models relies on the macro-f1 score, a robust metric commonly used in natural language processing. in total, 52 teams participated in our shared task: four teams in tamil and three teams in malayalam submitted a system to task 1 and achieved the top score of 0.73 in tamil and 0.87 in malayalam. due to the null participation in task 2, we stopped running the task further. beyond the competition, our overarching goal is to contribute to a safer and more inclusive digital ecosystem. by dissecting and understanding the dynamics of meme content, we aim to pave the way for more effective content moderation strategies. we envision this initiative as a catalyst for fostering responsible online behavior and promoting gender equality. ## related works in recent research, @xcite did a comprehensive data collection of 22,435 instances of fact-checked content from social media to scrutinize the proliferation of fake news across india between 2013 and 2020. this dataset is distinguished by its coverage across 13 languages, encapsulating 14 distinct attributes. it highlights the diversity and complexity of fake news dissemination within the multilingual and multicultural indian context, offering insights into the dynamics of misinformation across various domains and media types. @xcite presented "spotfake," a novel framework that surpasses existing systems by avoiding dependency on sub-tasks like event discrimination, focusing instead on directly leveraging textual and visual content through advanced language and image processing models . this approach demonstrates superior performance on twitter and weibo datasets, improving detection accuracy significantly. @xcite introduced a pioneering approach to meme analysis, providing goldstandard data for sentiment analysis, emotion classification, and intensity of emotion. the study presented baseline models, including a text-only model using lstm and a multimodal model combining resnet-50 and bert, demonstrating the potential of incorporating text and images for improved performance. @xcite proposed a comprehensive framework for analyzing image-withtext (iwt) memes, or "troll memes," introducing a three-level taxonomy to understand trolling's impact on domain-specific opinion manipulation. they enriched the memotion dataset to create the trollswithopinion dataset, containing 8,881 iwt memes in english, revealing challenges in classifying memes on the third level of the taxonomy. @xcite introduced the multimodal dataset "memosen" for the bengali language, comprising 4,368 memes annotated with sentiment labels. experiments on the memosen dataset showed a significant enhancement in meme sentiment classification with multimodal information integration. gasparini et al. ( 2022 ) created a benchmark meme dataset for automatic misogyny detection using 800 memes collected from various online sources. the dataset, analyzed by experts and crowdsourcing, included categories such as misogynistic, hostile, and ironic, with 100% agreement on 800 memes from three experts. suryawanshi et al. (2020) developed a system employing an early fusion technique to combine text and image modalities, contrasting its efficacy with baseline models focusing solely on either text or image. potential research gaps include the need for a unified evaluation metric and benchmark dataset for consistent comparison, the exploration of crosscultural meme classification, the investigation of interpretability in model decision-making, and the development of more robust techniques to address biases and fairness concerns in meme classification models. ## task description the competition, "shared task on multitask meme classification -unraveling misogynistic and trolls in online memes," includes a challenge that focuses specifically on spotting harmful content in memes. while the overall competition has two parts, this paper discusses the part about finding misogynistic memes in tamil and malayalam languages. organized as a part of the lt-edi@eacl 2024 event 1 , this task is designed to encourage experts to come up with ways to identify when a meme is offensive towards women. in task 1, the participants are tasked with creating a tool that can look at memes (combining pictures with text) and determine if the meme is disrespectful or harmful to women. the task deals with memes in tamil and malayalam, and the participants are given training and development sets. they use these sets to teach their tools to distinguish between misogynistic memes and those that are not. then, we will provide them with the test set without the labels. with this set, the participants will make the model to predict whether the meme is misogyny and submit it as a submission. finally, we will judge the participant's model with their prediction with the true labels of the test set. the tools are judged by how accurately they can make these distinctions, with the macro f1 score. the goal here is to push forward the development of tools that can spot and reduce the sharing of memes that can be hurtful to women in these two languages. this challenge is meant to attract attention from people worldwide who work in language and technology-related fields, hoping to spark more research and solutions in this important area. this shared task is conducted via the codalab competition foot_1 . ## dataset description the dataset underneath the misogyny meme classification competition offers a comprehensive look at the manifestation of misogynistic content within the digital landscape, particularly within tamil and malayalam languages. misogynistic memes target women or girls, often by leveraging stereotypes, displaying bias, or promoting discrimination. they might generalize women's capabilities with statements implying inferiority, demean their achievements, or mock female-specific issues to reinforce negative stereotypes and biases. our dataset encompasses both monolingual and bilingual memes, with some featuring a mix of tamil-english or malayalam-english content, presenting an open challenge for research due to the code-mixed nature of these texts. this dataset focuses primarily on monolingual content in tamil and malayalam, both of which are part of the dravidian language family. through this dataset, we aim to understand the extent and nature of misogynistic memes in these languages, exploring the linguistic and cultural factors contributing to their creation and spread. this is vital for researchers and practitioners dedicated to combating digital misogyny, especially in the context of dravidian languages. the dataset consists of 1,776 tamil memes, with 1,135 employed in the training set, 285 in development, and 356 in the test set. the data statistics for malayalam data are shown in ## participants methodology a total number of 52 participants were enrolled in this competition. in task 1, we got a total of 4 submissions for the tamil language and 3 submissions for the malayalam language. the methodologies and results of these tasks have been discussed. to get more crucial material, please consult their papers, which are listed below: quartet (h et al., 2024) team participated in task 1. they employed two different approaches to obtain the classification probabilities from the image and text data. with the textual data, every word of the text was translated into english. subsequently, the translated sentences were preprocessed by eliminating emojis, punctuations, and stopwords. then, the tf-idf vectorizer is employed to obtain the embeddings from the preprocessed texts. the probability of the text being misogynistic was determined using the multinomial naive bayes classifier. with the pictorial data, they employed the resnet50 model @xcite for performing transfer learning to obtain the probability of images being misogynistic. using those probabilities, the employed fusion technique calculates the resultant probability. dlrg team participated in task 1. they worked on only textual data to classify the memes as misogyny or not. they employed multilingual bert( bidirectional encoder representations from transformers) (kenton and toutanova, 2019), a transformer-based multilingual pretrained model. they performed a transfer learning approach with the transcriptions and the labels. word wizards team participated in task 1. they worked on only textual data to classify the memes as misogyny or not. they performed tokenization and extracted word embeddings using mucs @xcite team also participated in task 1. they work on both meme images and transcriptions. their methodology comprises a dual-encoder approach incorporating three distinct textual feature encoders alongside a shared image feature encoder: i) bert-base-uncased + resnet-50, ii) muril-base-cased + resnet-50, and iii) bertbasemultilingual-cased + resnet-50. ## conclusion in conclusion, the first shared task on "multitask meme classification -unraveling misogynistic and trolls in online memes" has been a groundbreaking effort to address the pressing issue of online misogyny and trolling within the context of dravidian languages. the participation of dedicated teams in the tamil and malayalam classification tasks demonstrates a collective commitment to understanding and combating such harmful online content. the datasets, precisely compiled and annotated, provided a robust foundation for the teams to deploy and test a variety of machine learning and deep learning models, which were assessed based on their macro f1 scores. furthermore, this paper stands as a testament to the collaborative efforts required to address the multifaceted challenges presented by online misogynistic and troll memes, and it is hoped that it will inspire continued research and action in this vital area.