MUCS @ LT - EDI -2024: Exploring Joint Representation for Memes Classification (2024)

Abstract: Misogynistic memes are a category of memes which contain disrespectful language targeting women on social media platforms. Hence, detecting such memes is necessary in order to maintain a healthy social media environment. To address the challenges of detecting misogynistic memes, “Multitask Meme classification - Unraveling Misogynistic and Trolls in Online Memes: LT-EDI@EACL 2024” shared task organized at European Chapter of the Association for Computational Linguistics (EACL) 2024, invites researchers to develop models to detect misogynistic memes in Tamil and Malayalam. The shared task has two subtasks, and in this paper, we - team MUCS, describe the learning models submitted to Task 1 - Identification of Misogynistic Memes in Tamil and Malayalam. As memes represent multi-modal data of image and text, three models: i) Bidirectional Encoder Representations from Transformers (BERT)+Residual Network (ResNet)-50, ii) Multilingual Representations for Indian Languages (MuRIL)+ResNet-50, and iii) multilingual BERT (mBERT)+ResNet50, are proposed based on joint representation of text and image, for detecting misogynistic memes in Tamil and Malayalam. Among the proposed models, mBERT+ResNet-50 and MuRIL+ ResNet-50 models obtained macro F1 scores of 0.73 and 0.87 for Tamil and Malayalam datasets respectively securing 1st rank for both the datasets in the shared task.

## introduction memes, in the digital age, have become a common form of cultural expression, often shared widely across social media platforms and internet communities. these memes typically comprising of images/videos and text embedded on them, started with the idea of sharing humor @xcite . but these days, memes are often being misused to spread hateful, troll, and misogynistic content. misogynistic memes are a category of memes that propagate negative attitude towards women. these memes often promote dangerous or harmful pranks, challenges, or behaviors which leads to physical harm, injury, or legal consequences @xcite . hence, it is necessary to detect such content to protect users from getting harmed and also to maintain a safe and inclusive online environment. detecting misogynistic memes on social media is challenging due to the combination of text, image/video, and sometimes audio also, which exhibits a multi-modal nature. this problem becomes more challenging when the embedded text belongs to low-resource languages like tamil, malayalam etc., where lack of digital resources and computational tools is the common issue. "multitask meme classification -unraveling misogynistic and trolls in online memes: lt-edi@eacl 2024" @xcite shared task encourages the researchers to develop models to detect misogynistic and trolling content in tamil and malayalam memes. the shared task has two subtasks and in this paper, we -team mucs, describe the learning models submitted to task 1 -identification of misogynistic memes in tamil and malayalam. as memes are made up of textual and visual components, they can be represented as multimodal data of textual and visual features integrated into a single representation known as joint representation. three models: i) bert+resnet-50, ii) muril+resnet-50, and iii) mbert+resnet-50, are proposed based on joint representation, for detecting misogynistic memes in tamil and malayalam. the rest of the paper is arranged as follows: a review of related work is included in section 2 and the methodology is discussed in section 3. experiments and results are described in section 4 followed by concluding the paper with future work in section 5. ## related work researchers have explored several models for detecting memes by representing the visual and textual components of memes as two uni-modal data as well as integrating visual and textual components into a single joint representation. few of such relevant research works are described below: raha et al. (2022) among their proposed models, the avg ensemble model outperformed other models with a macro f1 score of 0.761. @xcite employed an ensemble of ml models (multinomial nb (mnb) and gradient boosting classifiers trained with tf-idf of word bigrams and unigrams respectively, and random forest (rf) classifier trained with various image features (hu moment invariants, haralick textures, and image histograms), for the identification of misogynous memes. in addition, the ml models of the ensemble are also trained independently with the respective features as mentioned. among all their models, rf classifier trained with various image features outperformed other models by achieving a macro f1 score of 0.665. the above literature reveals that the joint representation of image and text exhibits promising performances for meme detection tasks. however, most of the meme detection tasks focus on english language giving less importance for low-resource languages like tamil and malayalam. ## methodology the objective of this work is to identify misogynistic memes in the given tamil and malayalam datasets. this is achieved by proposing learning models based on the joint representation of image and text components in the given memes. the steps ## conclusion and future work this paper describes, three models: i) bert+resnet-50, ii) muril+resnet-50, and iii) mbert+resnet-50, based on joint representation of text and image features, for detecting misogynistic memes in tamil and malayalam datasets, submitted by our team -mucs to "multitask meme classification -unraveling misogynistic and trolls in online memes: lt-edi@eacl 2024" shared task. among the proposed models, mbert+resnet-50 and muril+resnet-50 models obtained macro f1 scores of 0.73 and 0.87 for tamil and malayalam datasets respectively, securing 1 st rank for both the datasets in the shared task. more efficient joint representations that improve the performance of the learning models will be explored further.