IIITH at S em E val-2022 Task 5: A comparative study of deep learning models for identifying misogynous memes (2022)

Abstract: This paper provides a comparison of different deep learning methods for identifying misogynous memes for SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification. In this task, we experiment with architectures in the identification of misogynous content in memes by making use of text and image-based information. The different deep learning methods compared in this paper are: (i) unimodal image or text models (ii) fusion of unimodal models (iii) multimodal transformers models and (iv) transformers further pretrained on a multimodal task. From our experiments, we found pretrained multimodal transformer architectures to strongly outperform the models involving the fusion of representation from both the modalities.

## introduction with the social media turning out to be a medium for propagation of hate speech and other perils, misogyny and sexism is incident upon women in explicit and implicit ways. although memes have turned out to be a potent mechanism for exchanging humorous messages, they have been turning out to also be bearers of such malicious content. with this motivation, the task of multimedia automatic misogyny identification (mami) @xcite was proposed with two subtasks: (1) determining whether a meme is misogynous as a binary classification problem (2) finegrained misogyny classification into categories of stereotype, shaming, objectification and violence as a multilabel problem. in our work, we have compared different deep learning approaches for identifying misogyny in memes and also further classifying them into different kinds of misogyny. we base our experiments on unimodal architectures making use of only either the textual or the image content in memes. the unimodal architectures were naturally superseded by their multimodal counterparts, since they made use of both the modalities in misogyny identification. among the multimodal architectures, we initially experimented with simple fusion-based approaches which involved combining the image and text representations. these experiments were followed by trying out multimodal transformer architectures in which we made use of mmbt @xcite , vil-bert @xcite and @xcite . we used these architectures pretrained on unimodal as well multimodal objectives. we found visualbert and vilbert trained using multimodal objectives to perform competitively on the task. in order to further improve the capability of the models for misogynous content identification, we tried out further pretraining the models on a dataset for classifying hateful memes. this strategy involving a task-adaptive further pretraining stage turned out to further boost the performance of the models showing the benefit obtained from larger datasets for adapting models to a finegrained downstream task. figure 1 shows the training stages of such an architecture. our best model achieved a macro-f1 score of 0.712 for subtask 1, while the best performing model for subtask 2 gave a weighted f1 of 0.706. meme classification. the ubiquity of memes on internet, presence of malicious / hateful content in memes and the challenges involved in meme understanding were discussed in @xcite . the work presented a new dataset and a challenge for understanding emotions in memes which involved subtasks for identifying the sentiment, humour category and scale (or intensity) of the detected class. @xcite introduced a dataset for detection of offensive content in memes. a larger, challenging dataset was introduced in @xcite ) by involving 'benign confounders' to force the multimodal architectures to learn robust representations using both the modalities. the work also introduces formidable baselines with multimodally pretrained transformer encoders. among the top performing models on this dataset, (velioglu and rose, 2020) perform an ensemble of multiple trained visualbert models, while ensembling was done in @xcite on a set of five predictions from different trained models, with the predictions for each model averaged from 3-5 different runs. ## task and dataset overview the task consists of two subtasks: for this task, given a meme, we have to predict if a meme is misogynous in nature or not. given that a meme is misogynous, this task further identifies the kinds of misogyny among potential overlapping categories such as stereotype, shaming, objectification, and violence. the dataset for the task was provided by the workshop organizers. the training set consisted of 10000 memes, whereas the hidden test consisted of 1000 memes. each row in the dataset contained a unique identifier, the path to the image file for a corresponding meme, the text in the meme, and the label values of misogyny, stereotype, shaming, objectification, and violence. ## methodology in the following section, we discuss our approaches for misogyny detection. we discuss our models in detail and provide a comparison between the models. we have explored unimodal models which use just the text or image as the input. the unimodal fusion models take the representation of the image part of the textual part separately and combine them to give the output. we have also exploited different pretrained multimodal transformers models. we have also experimented with how to further pretraining of these multimodal transformers models affect the quality of the predictions. ## results it can be noted that we mostly used the default hyperparameters from the mmf framework and did not perform rigorous hyperparameter tuning for our experiments, so the model performances still be improved with the search for optimal hyperparameters using cross-validation. we analyze the results of the approaches tried for each subtask. among the baselines, we saw the unimodal-text models perform better for both the subtasks than the unimodalimage models. among the unimodal-text models, bert performed better than roberta. the fusion models performed a bit better than the unimodal models for both the subtasks, with bert concatenation performing significantly better than late fusion in the first subtask. the multimodal transformers models give a performance increase over the fusion models, with vilbert performing the best for subtask 1 and visualbert giving the best performance for the second subtask. among the multimodal transformer architectures, the ones pretrained with multimodal objectives turned out to be better in performance than those trained using unimodal objectives. given that visualbert and vilbert performed the best, we further pretrained them in a multimodal task-adaptive setting. vilbert pretrained on the hatefulmemes dataset gave the best results for the first subtask, whereas visualbert pretrained on the hateful-memes dataset was our best model for the second subtask. ## conclusion in our worked, we have provided a comparative analysis of architectures for solving the task of misogyny identification. although our bestperforming model did achieve a substantial improvement over the baselines, the scores still in-dicate scope for further improvement. this also brings forth the challenging nature of the task in itself. furthermore, using ensemble models like vilio @xcite can result in betterperforming models which can be tried out in future scope.