Mitra Behzadi at S em E val-2022 Task 5 : Multimedia Automatic Misogyny Identification method based on CLIP (2022)

Abstract: Everyday more users are using memes on social media platforms to convey a message with text and image combined. Although there are many fun and harmless memes being created and posted, there are also ones that are hateful and offensive to particular groups of people. In this article present a novel approach based on the CLIP network to detect misogynous memes and find out the types of misogyny in that meme. We participated in Task A and Task B of the Multimedia Automatic Misogyny Identification (MaMi) challenge and our best scores are 0.694 and 0.681 respectively.

## introduction in the past few years more and more people have been using memes on social media platforms to express their thoughts and sometimes their beliefs. although there are countless memes that are humorous and fun without expressing hate towards any certain group of people, there are also memes that aim to attack people. misogynistic memes use a combination visual and textual content to put down women and some of them are so violent that can be triggering to previous sexual abuse victims. that is why it would be so useful if these instances could be identified automatically. one of the main challenges of this problem is that the images of these memes come in various forms. also, the text on the memes adds occlusion to the objects in the images which makes the image understanding part of the problem even more challenging. to address these challenges, we present a novel multi-modal classification approach based on clip @xcite to identify misogynistic memes and also determine which 4 subcategories of shaming, stereotype, objectification or violence they belong to. clip is a multi-modal network trained for object detection. we use a multilabel classification method and detect misogynistic memes and all the subcategories using one single pipeline. ## related work there have been numerous researches conducted to solve hateful speech detection online in the past decade. many of these approaches such as @xcite and @xcite focus on only one modality which is text. more recently, researchers such as @xcite gathered multi-modal data-sets to be able to detect those instances of hate that go undetected by using only textual context. memes are also a form of multi-modal data that are widely used to indirectly communicate some meaning online. for the past few years, researchers have tried to solve the problem of hateful meme detection in various ways. in @xcite ) a small data-set was gathered and an approach based on vgg16 @xcite neural network was proposed. in 2020 facebook gathered a the hateful meme data-set with 10,000 instances which was part of a hateful meme detection challenge @xcite . the winner of that challenge (zhu, 2020) used an ensemble of visual-linguistic models such as @xcite and ernie-vil @xcite and fine-tuned them to solve the problem. ## conclusion in this paper we demonstrated how a successful model such as clip can be used to detect misogynous memes. we presented a novel architecture based on that multi-modal model and used multilabel training. we were able to achieve good results using this approach.