L ast R esort at S em E val-2022 Task 5: Towards Misogyny Identification using Visual Linguistic Model Ensembles And Task-Specific Pretraining (2022)

Abstract: In current times, memes have become one of the most popular mediums to share jokes and information with the masses over the internet. Memes can also be used as tools to spread hatred and target women through degrading content disguised as humour. The task, Multimedia Automatic Misogyny Identification (MAMI), is to detect misogyny in these memes. This task is further divided into two sub-tasks: (A) Misogynous meme identification, where a meme should be categorized either as misogynous or not misogynous and (B) Categorizing these misogynous memes into potential overlapping subcategories. In this paper, we propose models leveraging task-specific pretraining with transfer learning on Visual Linguistic models. Our best performing models scored 0.686 and 0.691 on sub-tasks A and B respectively.

## introduction the term "misogyny" means hatred towards women. misogyny can be interpreted through multiple forms such as male privilege, sexual harassment, violence against women, objectification. memes that targeted women focus on appearance, intellect, their traditional gender roles and capabilities of women @xcite . for this, semeval 2022 task 5 @xcite focuses on identifying such behaviour in a multimodal setting (text + image). the textual cues to this task are given in the english language. the task is divided into two sub-tasks. the first subtask is modelled as a binary classification problem. the second sub-task focuses on identifying type of misogyny from a set of overlapping categories, making it a multi-label classification problem. a meme contains text superimposed on an image. the image's aim in a meme is generally to reinforce a technique in the text, thus making its classification a multimodal problem. both the modes of information are crucial to establishing the message conveyed by the meme, which can be very different from when the two modalities are evaluated separately. we experiment with visual linguistic (vl) models like oscar @xcite and uniter @xcite to understand the memes through both modalities. we employ transfer learning to use a model trained on another similar dataset and then finetune it on our dataset. as task-specific pretraining has shown to improve results on several nlp tasks ( gururangan et al. (2020) ), we experiment with task-specific pretraining our vl models before finetuning it and also finetune it on models task-specifically pretrained for other similar task like hateful memes detection @xcite ). we also train bert @xcite based models on only the textual data, thus comparing the performances of multimodal setting vs unimodal settings. this comparison helps us understand how vital each modality is and how much using both together makes a difference. we discover that even though detecting misogyny in memes can be modelled as a multi-modal task, it can, to a very good extent be done through working with just the textual cues but when it comes to detecting more subtle forms of misogyny, the visual cues play an important role as well. our system ranked 38 th and 19 th for sub-taska and sub-taskb respectively. the paper is structured as follows: section 2 describes the dataset along with related work. section 3 describes our system and model architecture. section 4 has information regarding the dataset size and splits with libraries used to implement our system. section 5 has the discussion about the findings from our experiments and section 6 concludes our paper. ## background nowadays, the internet and various social media platforms have become an intrinsic part of more and more people's lives. with its growth, the problems associated with it have also increased exceedingly, like the increase in hate speech against certain groups including women. detecting misogyny and sexist slurs in general over social media can be challenging as its overall meaning can depend on its context and the user it is shown. @xcite . for this, look at the few examples in figure 1 to exhibit the importance of visual and textual cues. memes can be defined as an image, video, or text, typically humorous in nature, that is copied and spread rapidly by internet users, often with slight variations. memes in online culture have been seen to push potential instances of misogyny as a form of "joke" and "irony" while disguising itself as a harmless form of humour. @xcite . there has been previous work done to detect hate speech and misogyny. @xcite employed support vector machine(svm) based architectures with a novel lexicon of abusive words to detect misogyny in english and spanish tweets. @xcite compared unimodal textual classifiers to multimodal classifiers trained with both visual and textual features using early fusion on a dataset of advertisements consisting of image and text marked for being sexist. the meme classification task is primarily a visual-linguistic(vl) task where we are trying to classify data where the image can be seman-tically correlated with the text. traditional vl approaches are based on primary fusion techniques like early or late fusion, where each modality is learned separately. however, a multimodal pretrained model might perform better at memes classification (afridi et al., 2020). ## system overview we use transformer based models for both the tasks with task specific modifications. ## results and discussion the detailed results from all our experiments conducted can be seen in we here use the f1 macro scores to judge our models. for subtaska, we see that oscar ensemble models, task-specifically pre-trained on hateful memes dataset perform the best. another interesting thing to notice is the textual only roberta large model performs almost as good as our best performing vl model and better than all other vl models and is significantly better than bert large. we also see that simple average ensemble models for oscar perform better than each of its constituent models, and using transfer learning methods with model fine-tuned on hateful memes dataset performed unexpectedly worse. it means that even though hateful memes detection and detecting misogyny in memes are closely related in their idea, they are still not necessarily similar to predict. in sub-taskb, we see that the ensemble of models with task-specific pretraining on our dataset worked the best and slightly better than the ensemble with task-specific pretraining on the hateful memes dataset. we also see that our oscar vl models worked significantly better here than textonly models like bert and roberta, which is unexpected since the text-only models worked very well compared to vl models in sub-taska. as we observe that bert based models give comparable, and in the case of roberta, better performance than almost all the vl models, it indicates that detecting misogyny might not be an utterly multimodal problem, and just the textual cues are enough in identifying the misogyny. we also observe that even though text-only models performed very well on misogyny detection, they performed poorly on more fine grained classification tasks, showcasing that the visual cues mattered as well to figure out the subtleties in the classification of the type of misogyny. we also trained both textual, and vl models on just the data points marked for misogyny as those are the only ones where at least one of the subcategories of misogyny will be marked positively. however, in this case, the models performed much more poorly. it is because they are not trained on examples that are not misogynous in nature and thus perform poorly on them in the test dataset. the scores according to the official metrics for our best performing unimodal and multimodal models were as follows: sub-taska: roberta large : 68.3; oscar hm_pretrained_ens : 68.6; sub-taskb: roberta large : 63.6; oscar hm_pretrained_ens : 69.1 ## conclusion in this paper, our experiments indicate that although misogyny detection in memes is designed as a multi-modal setting, the textual cues also perform very well and, in some instances, better than visual linguistic models. we also found out that when it comes to detecting more subtle forms of misogyny, visual cues seem to help in the classification task and perform better than transformer models with just textual cues. more work can be done to improve the results. future work like experimenting with more upcoming vl models, employing better techniques to address the class imbalance, and using more advanced ensembling techniques like rank averaging, power averaging & simplex optimization can improve results.