Q i N i A n at S em E val-2022 Task 5: Multi-Modal Misogyny Detection and Classification (2022)

Abstract: In this paper, we describe our submission to the misogyny classification challenge at SemEval-2022. We propose two models for the two subtasks of the challenge: The first uses joint image and text classification to classify memes as either misogynistic or not. This model uses a majority voting ensemble structure built on traditional classifiers and additional image information such as age, gender and nudity estimations. The second model uses a RoBERTa classifier on the text transcriptions to additionally identify the type of problematic ideas the memes perpetuate. Our submissions perform above all organizer submitted baselines. For binary misogyny classification, our system achieved the fifth place on the leaderboard, with a macro F1-score of 0.665. For multi-label classification identifying the type of misogyny, our model achieved place 19 on the leaderboard, with a weighted F1-score of 0.637.

## introduction even though women are as much present online as men, some online spaces, such as microblogging websites, are still male dominated. misogynistic jokes and memes are inevitably somewhat common in certain parts of the internet and have the potential to perpetuate harmful ideas about gender or instill false ideas and expectations. notably, the way they are spread is often through various modalities, most commonly visual and textual. therefore, it would be useful to have a system that could automatically detect if certain combinations of texts and images are misogynistic or not. this is not a trivial task however, since misogyny can manifest in many different forms such as stereotyping, objectifying or threatening violence against women. a crucial difficulty in this multi-modal classification task is also the interplay between text and image. some memes may appear harmless if only either the image or text are viewed separately. the multimedia automatic misogyny identification (mami) challenge at semeval-2022 @xcite seeks to find solutions to solve this task. we propose two models that automatically detect english misogynistic memes and classify the type of problematic ideas they perpetuate. 1 for simple binary misogyny detection, we created an ensemble model that makes predictions based on majority voting on two text-based and two imagebased classifiers using partly hand-crafted features such as age, gender and nudity classification. for task b we relied only on text information and used a transformer based approach, creating a roberta model @xcite that classifies the type of misogyny that is perpetuated in the memes. ## background automatically identifying misogynistic texts has been explored in the past. in 2020, this task was proposed as an evalita shared task, using italian tweets @xcite . so far, the research in the area of misogyny detection has mostly focused on pure text data from social media, specifically twitter @xcite . in the current multi-modal task however, competitors were given the opportunity to explore classification using both visual and textual data given. @xcite used a multi-modal multi-task learning system with bert @xcite features to extract textual information and resnet features to handle image classification. a similar approach proved to be useful in another related shared task, in which tamil memes should be identified as trolling or not @xcite . the provided data set includes both images and text transcriptions of memes. the two subtasks we participated in were structured as follows: in subtask a, the memes should simply be identified as misogynous or not misogynous. subtask b posed a more advanced challenge, as memes should additionally be classified as being part of four overlapping categories. these categories specify the type of misogyny expressed in the meme: stereotypes, shaming, objectification and violence. shaming memes will insult women's appearance, stereotypes perpetuate harmful ideas about (groups of) women, objectification reduces women to their sexuality or body and violence downplays or advocates for violence against women. examples of text transcriptions of these categories can be found in ## subtask b: multi-label classification of misogyny types although the ensemble model performed reasonably well for subtask a, performance was significantly worse when using it for the extended misogynistic labels introduced in subtask b. because of this, we built and trained a different model infrastructure using a deep learning approach (which performed worse than our ensemble model, when applied to subtask a). ## conclusion in our experiments, we explored and compared different models for multi-modal analysis of misogynistic memes. surprisingly, ngram-models on both word and character levels did not perform as well as expected for this task. we discovered that ensemble models using text and image information can work well even if the text classifier uses simple features such as tf-idf vectors. bert based text analysis performs better than the baselines, even if image features are not included. we found that an ensemble model can be improved through gradient boosting and adding information about nudity, age, gender of depicted humans, and text sentiment. in the future, it would be worth exploring how well a similar ensemble model with an (additional) bert-classifier or other more powerful text classifier performs. for subtask b, a roberta-based multi-label classification model showed its power with purely text information. it would be interesting to train it with a class of weights and different threshold values. however, we were not able to create a well-performing multi-modal model that uses image information, which might be another interesting direction for further studies.