Exploring Contrastive Learning for Multimodal Detection of Misogynistic Memes (2022)

Abstract: Misogynistic memes are rampant on social media, and often convey their messages using multimodal signals (e.g., images paired with derogatory text or captions). However, to date very few multimodal systems have been leveraged for the detection of misogynistic memes. Recently, researchers have turned to contrastive learning, and most notably OpenAI’s CLIP model, is an innovative solution to a variety of multimodal tasks. In this work, we experiment with contrastive learning to address the detection of misogynistic memes within the context of SemEval 2022 Task 5. Although our model does not achieve top results, these experiments provide important exploratory findings for this task. We conduct a detailed error analysis, revealing promising clues and offering a foundation for follow-up work.

## introduction hateful expressions on the internet are widespread and mostly based on religion, gender, race, or physical attributes @xcite . such language exacerbates damaging societal problems such as racism, sexism, and other types of discrimination. in particular, misogynistic abuse has become very prevalent and poses a serious problem in cyberspace @xcite . although extensive research on hate speech and misogyny has been conducted @xcite , it has thus far centered on the analysis of text or images alone. memes, or social media images that communicate messages through the creative use of imagery understood to carry specific rhetorical value among members of a community, are a common platform for misogyny and hateful expressions. approximately 78% of women use image-based social media multiple times per day (compared to 65% of men) @xcite , making exposure to this harmful content alarmingly common. classifying memes is challenging because of their multimodal interplay between image and text, as well as their regionspecific interpretation, and existing multimodal approaches do not perform very well in the classification of hateful memes @xcite . the underlying goal of semeval-2022 task 5 was to address this research gap, tackling the challenge of identifying misogynistic memes using multimodal data by inviting researchers to experiment with a variety of approaches. our team, uic-nlp, investigated an adapted version of the contrastive language-image pretraining (clip) technique recently established and applied with success to numerous other tasks @xcite @xcite . our model, recorded on the leaderboard in the 6th leaderboard cluster under the openreview username "charicfc," ranked 71 st out of 83 participants and obtained an average f1 score of 0.62. we analyze our model's predictions and offer insights and recommendations for improving upon it in the future. ## system overview our system architecture is a variation of openai's clip model @xcite . we refer readers to the original paper for a detailed overview and illustrations of the architecture, and summarize it here. inspired by the idea of usability and generality, clip uses a contrastive learning objective to build a joint visual-linguistic space for learning visual concepts from natural language supervision. we describe the various components of our architecture below. ## results during development we used the training data provided by the mami task. we divided it into training (90%) and test (10%) sets, and measured performance using f1, accuracy, precision, and recall on the 10% test set. we provide our results on the development data in ## conclusion and future work in this paper, we describe our system implementation for semeval-2022 task 5. our model ranked 71 st out of 83 participants' teams on subtask 1. we comprehensively investigate the use of a state-of-the-art multimodal contrastive learning approach for the classification of misogynistic memes. more experiments and tests should be done to improve the model's performance on this task. in particular, upsampling the dataset and addressing the possible biases caused by identity terms should be investigated. finally, at an architectural level, our current system encodes images as one form of input and encodes paired text content and labels as another form of input, similarly to text encoding strategies used for unimodal sequence prediction tasks. exploring joint encodings of image and paired text content as a single form of input, with only labels as the other form of input, may be an additional design avenue worth pursuing. overall, it is our hope that this work motivates additional interest in contrastive learning solutions for multimodal misogynistic meme detection. we make our source code 1 available to other researchers to facilitate follow-up work by others.