Transformers at S em E val-2022 Task 5: A Feature Extraction based Approach for Misogynous Meme Detection (2022)

Abstract: Social media is an idea created to make theworld smaller and more connected. Recently,it has become a hub of fake news and sexistmemes that target women. Social Media shouldensure proper women’s safety and equality. Filteringsuch information from social media is ofparamount importance to achieving this goal. In this paper, we describe the system developedby our team for SemEval-2022 Task 5: MultimediaAutomatic Misogyny Identification. Wepropose a multimodal training methodologythat achieves good performance on both thesubtasks, ranking 4th for Subtask A (0.718macro F1-score) and 9th for Subtask B (0.695macro F1-score) while exceeding the baselineresults by good margins.

## introduction with the rising usage of social media and the internet, it is tougher to establish an inclusive and welcoming community among users. offensive speech, hate speech, and targeted insult have been increasing among users, disturbing everyone. with the rising utilization of the internet in a pandemic, hate speech prevalence on the internet is also increased. online hate speech with targeted discrimination also creates threats and crimes offline. among these, online misogyny or sexist comments have been increasing among women @xcite , which includes name-calling, sexual threats, shaming. this emphasizes the need for specialized automatic misogyny detection in online platforms. platforms such as twitter foot_1 and facebook already have policies for banning hateful content. however, these systems are primarily through manual methods and might not scale well for large users and multimodal content. moreover, hate speech is also prevalent in multimodal form since most social media platforms support images, text, audio and video content. these memes have been popular among users to express their opinions since people express information through memes, gifs, and videos. but, unfortunately, this also causes the rise of multimodal hate, and offensive content, which is disturbing to users @xcite . this includes misogynistic posts, which are targeted towards women. previous works on misogyny detection have been primarily focusing on one modality, which is text @xcite . misogyny detection in text falls under an area of text classification. text classification methods such as bert, lstm, naive bayes have been used to detect misogynistic comments. in this work, we have used the provided data, which contains both images of the memes and the extracted ocr text from the memes. this paper describes our submission for the task; we have used multiple concatenation-based fusion models and ensembled them for the final submission. ## related work the past works have concentrated on collecting the dataset from popular social media sites such as facebook, reddit, and twitter. recent statistics do not simply focus on hate but also on the kind of hate the meme attempts to spread. work has also been done in detecting offensive memes using various pre-trained models. @xcite presents a novel dataset for the sexist meme classification task. sexism could be of several forms that could be categorized based on the context of the caption and the objects on the meme. the main types of sexism against women addressed in the dataset are shaming, stereotypes, objectification and violence. the research paper largely focused on a comparison between the unimodal and multimodal classifiers. the article has attempted to answer various research concerns such as whether unimodal architectures can predict the target correctly, will merging the features of both text and picture capture the inherent complexity of the sexist memes, and which one of the two modalities dominates the other. the research discovered that unimodal classifiers have shown that textual information is an excellent indicator, whereas visual information is a poor indicator to identify sexist memes. this study between unimodal and multimodal showed that unimodal architectures performed better than multimodel architectures. in the paper, @xcite the analysis is done on the dataset that focuses beyond hateful or not hateful by annotating the hate meme dataset further by the kind of hate the meme is actually spreading. this would help in understanding the meme and the intention of the person who created the meme better rather than just labelling it as hateful or not. the paper focused on two tasks. the first task was to identify the kind of hate the meme intended to spread. the second task was to detect the type of attack the meme did on a particular group such as slurs, inferiority, and mocking. models such as clip (contrastive language image pre-training) and laser (language agnostic sentence representations), labse (language agnostic bert sentence embedding) were used to extract features from the image and text. the paper concluded that multimodel architectures outperformed unimodal architecture. the multimodal, concatenated textual features (clip, laser, and labse) and visual features (clip) was the best performing model with auroc of 0.96 and 0.97 for task a and task b, respectively. @xcite introduced a taxonomical dataset of 6,383 samples from reddit. the dataset has a three-level taxonomy which makes this dataset very different from what already exists. the first level is a binary classification between misogynistic content and non-misogynistic content. the second level corresponds to the subtypes of misogynistic and non-misogynistic content. misogynistic content categories include misogynistic pejoratives, misogynistic treatment, misogynistic derogation and gendered personal attacks against women. non-misogynistic content categories include counter speech against misogyny, non-misogynistic personal attacks and none of the categories. in the third level, additional flags for some of the second-level categories have been defined. bert based models were trained on the dataset to achieve a test accuracy of 0.93. ## dataset the dataset provided for the competition @xcite consisted of images of memes and ocr extracted text and labels for both subtask a and b. for subtask a, the binary label of misogynous is given; for subtask b, four labels were given: they are shaming, stereotype, objectification and violence, each of them containing binary values 0 or 1. the provided dataset contains 10,000 training images, 100 validation images and 1000 images for test set submission. the training dataset was randomly shuffled and split into five-folds, with each fold containing 2000 images each. the first four folds were used to train the model, and the last fold, along with the eval dataset provided, was chosen as the validation dataset to improve the model performance. the final number of samples in training, validation and test set are reported in ## preprocessing since the text was extracted from the memes using ocr tools, much noise in the text had to be cleaned manually. first, all internet links, stopwords and twitter user handles were removed from the text. then, the text was lemmatized using a word-net based lemmatizer. the text truncation length was set to 256. an interesting observation in the dataset was that memes that were not misogynistic in nature did belong to the other four classes. so, it was evident that a meme might not be misogynistic yet belong to any of the other subcategories. ## results and discussion the test set results are reported in ## conclusion thus we illustrate the system developed by us for semeval-2022 task 5: multimedia automatic misogyny identification. we compare multimodal finetuning vs classification of pre-trained network feature extraction. we have also discussed various methods adopted to train such models and also the data preprocessing done. we show the potential of employing such a model in real-world use cases.