TIB - VA at S em E val-2022 Task 5: A Multimodal Architecture for the Detection and Classification of Misogynous Memes (2022)

Abstract: The detection of offensive, hateful content on social media is a challenging problem that affects many online users on a daily basis. Hateful content is often used to target a group of people based on ethnicity, gender, religion and other factors. The hate or contempt toward women has been increasing on social platforms. Misogynous content detection is especially challenging when textual and visual modalities are combined to form a single context, e.g., an overlay text embedded on top of an image, also known as meme . In this paper, we present a multimodal architecture that combines textual and visual features to detect misogynous memes. The proposed architecture is evaluated in the SemEval-2022 Task 5: MAMI - Multimedia Automatic Misogyny Identification challenge under the team name TIB-VA. We obtained the best result in the Task-B where the challenge is to classify whether a given document is misogynous and further identify the following sub-classes: shaming, stereotype, objectification, and violence.

## introduction detection of hate speech has become a fundamental problem for many social media platforms such as twitter, facebook, and instagram. there have been many efforts by the research community and companies to identify the applicability of advanced solutions. in general, hate speech is defined as a hateful language targeted at a group or individuals based on specific characteristics such as religion, ethnicity, origin, sexual orientation, gender, physical appearance, disability or disease. the hatred or contempt expressed towards women has been drastically increasing, as reported by plan international (2020) and @xcite . detection of such misogynous content requires large-scale automatic solutions @xcite @xcite and comprehensive annotation processes @xcite . the detection of hateful content has been mainly studied from the textual perspective based on the computational linguistics and natural language processing (nlp) fields. however, hateful content on social media can be found in other forms, such as videos, a combination of text and images, or emoticons. misogynous content detection is especially challenging when textual and visual modalities are combined in a single context, e.g., an overlay text embedded on top of an image, also known as meme. recent efforts in multimodal representation learning @xcite pushed the boundaries of solving such problems by combining visual and textual representations of the given content. several datasets have been proposed using multimodal data @xcite @xcite @xcite for various tasks related to hate speech. each dataset includes an image and corresponding text, which is either an overlay text embedded on an image or a separate accompanying text such as tweet text. in contrast to existing datasets based on memes @xcite @xcite , the addressed task in this paper aims to identify misogyny in memes specifically. among the previously mentioned work, only the dataset from @xcite is intended for misogyny detection, in which the text is in the italian language. in terms of dataset size, the dataset by @xcite contains approximately 150,000 image-text pairs, while other datasets have moderate sizes that range between two and ten thousand image-text pairs. moreover, existing model architectures that are evaluated on such benchmark datasets use a combination of various textual and visual features extracted from pre-trained visual and textual models @xcite . the semeval-2022 task 5: mami -multimedia automatic misogyny identification @xcite , 2022) 1 is a new challenge dataset that focuses on identifying misogynous memes. the memes in this dataset are composed of an image with an overlay text. some samples with their corresponding class labels are shown in figure 1 . the dataset includes two sub-tasks as described below. in this paper, we present our model architecture for which we submitted results under the team name tib-va. the model architecture is based on a neural model that uses pre-trained multimodal features to encode visual and textual content and combines them with an lstm (long-short term memory) layer. our proposed solution obtained the best result (together with two other teams) on the task-b. the remainder of the paper is structured as follows. in section 2, we describe the proposed model architecture. in section 3, the experimental setup, dataset details, as well as evaluations of the model architecture are described in detail. finally, section 4 concludes the paper and outlines areas for future work. 2 multimodal architecture our model architecture is a neural model that uses a clip @xcite model to extract textual and visual feature representations, which is pre-trained on over 400 million image-text pairs. the goal is to investigate whether this model is applicable to identify misogynous content in memes where both visual and textual content are considered. we used recently available vit-l/14 variant of clip. the tokens in the overlay text and the image are fed into clip text encoder and clip image encoder respectively. the text encoder outputs a sequence of 768-dimensional vectors for each input token. these token vectors are then fed into an lstm layer with a size of 256. this layer is another essential part of the proposed architecture. it learns the contextual relatedness among tokens in the text by combining all token representations extracted from the clip text encoder branch. the output from the image encoder is fed into a fullyconnected layer with a size of 256. the output from an lstm layer for text and output from the fully-connected layer for the image are fed into separate dropout layers (dropout rate of 0.2), the outputs are concatenated, and then fed into another fully connected layer with a size of 256. the final vector representation is then fed into separate sigmoid functions for each task. for task-a, the sigmoid outputs a single value that indicates the probability of misogyny. for task-b, each sub-class of misogyny (stereotype, shaming, violence, objectification) has a separate sigmoid function that outputs a probability value for the corresponding class. the model architecture is shown in figure 2 . the source code of the described model is shared publicly with the community foot_0 . ## conclusion in this paper, we have presented a multimodal model architecture that uses image and text fea-tures to detect misogynous memes. the proposed solution is built on the pre-trained clip model to extract features for encoding textual and visual content. while the presented solution does not yield top results on task-a, it achieves the best performance in task-b for identifying sub-classes of misogyny such as stereotype, shaming, objectification, and violence. in future work, we will explore the combination of multiple multimodal features that measure different aspects of visual content such as violence, nudity or specific objects and scene-specific content.