I 2 C at S em E val-2022 Task 5: Identification of misogyny in internet memes (2022)

Abstract: In this paper we present our approach and system description on Task 5 A in MAMI: Multimedia Automatic Misogyny Identification. In our experiments we compared several architectures based on deep learning algorithms with various other approaches to binary classification using Transformers, combined with a nudity image detection algorithm to provide better results. With this approach, we achieved an F1-score of 0.665 in the evaluation process

## introduction misogyny is hatred or contempt for women. it is a form of sexism used to keep women at a lower social status than men, thus maintaining the societal roles of patriarchy. misogyny has been widely practiced for thousands of years. it is reflected in art, literature, human societal structures, historical events, mythology, philosophy, and religion worldwide @xcite . the internet represents for many an extension of our offline interactions, and seemingly mundane everyday practices (e.g. participating in social media) form a significant part of our everyday experiences. unfortunately, it is too common to see examples of harassment towards women and marginalized groups online within these experiences and practices @xcite . women have a solid presence on the web, especially in picture-based web media like twitter and instagram: 78% of females utilize online media on numerous occasions each day, in contrast with 65% of men. a popular way of communicating via social media platforms are memes. a meme is an image portrayed through pictorial content with overlaid text which is written a posteriori, with the fundamental objective of being entertaining and/or ironic. even though most of memes are created with the goal of making amusing jokes, shortly after their standardization individuals began to use them to disseminate hate against women, leading to sexist and aggressive messages in internet environments that allow people to freely express sexism without the fear of retaliation. the detection of this disrespectful content is essential to eliminate it as soon as possible and stop spreading misogyny as a "joke". in mami: task 5, track a (meme binary classification) @xcite , participants must determine whether a meme (text + image) is misogynist or not. meme sentiment-related tasks analysis is challenging, as memes are used created for various purposes, they are always evolving and often use sarcasm and humour. while misogyny and hate speech detection in text has been widely explored by the nlp community @xcite . its detection in images and text and how they correlate has not been explored in depth. in this field we used bert @xcite , roberta @xcite for training and classifying text, and nudenet tool for image classification ( https://github.com/notaitech/nudenet ). our results show that combining text and image classification results are slightly better than using only one of the two methods. the rest of the paper is organized as follows. section 2 contains a briefly description of the dataset and its structure, section 3 features the analysis of some of the previous works related to our task. in section 4 we describe the different models and algorithm used, and their configurations. section 5 provides details about data and models setup, while section 6 reports experimental results and the paper is concluded in section 7. ## background this paper is focused on subtask a: binary classification. the corpus provided is composed of 5000 1-value rows (misogyny) and 5000 0-value rows (not misogyny), so it is already well-balanced. each row has the following format: where "file_name" is the .jpg link to the image, and "text transcription" is the text extracted from the image. for our binary classification task, we only need the fields "file_name", "misogynous" and "text transcription" to train the models presented. for models that also used nudity recognition, a column "unsafe" was added later. the csv file used follows this structure: -file_name: "10.jpg" -misogynous: "1" -text transcription ## related work sentiment analysis of text is a very active research area that still faces multiple challenges such as irony and humour detection @xcite . in this area, the focus of the nlp community has increased towards detection of offensive language, aggression, hate-speech detection @xcite and specifically misogyny, taking into account it can be expressed in a direct, explicit manner or an indirect, sarcastic manner, and even if this message is genered or not-gendered @xcite . the analysis of misogyny in memes has already been done in a psychologic point of view @xcite , but never in computational models. multimodal analysis research has been extended during the last years, but the focus was mostly on video and text or speech and text @xcite . the specific multi-modality of memes in sentiment analysis has only been addressed recently by investigating their correlation with other comments in online discussions @xcite . the growing usage of memes as an alternative medium of communication on social media has also recently drawn the attention of the online abuse research community. however, memes completely make sense only if one takes both text and image content into account. these modalities can also lead to totally different perceived sentiment when recombined. for example, a meme whose image is a scary clown and the text is "happy birthday" will have a very different sentiment from a meme with the same text but with an image of a funny clown. sabat et al. ( 2019 ) performed hate speech detection on memes and showed that images were more important than text for the prediction. in the other hand, bonheme and grzes (2020), investigated the relationship of text and image in sentiment analysis of memes, and found that images and text were uncorrelated. fusion-based strategies did not show significant improvements and using one modality only (text or image) tends to lead to better results. ## system overview we focus on exploring different training techniques for text using bert and roberta, given their superior performance on a wide range of nlp tasks, while for image we used the python module nudenet. each text encoder, image classifier and training method used in our model are detailed below. ## experimental setup for each text transcription row in the corpus, a small preprocessing was applied. every word was undercased, web pages' links, hashtags, usernames, emojis, punctuation symbols, numbers, and words with length less than 2 characters were removed, as well as english stop words using nltk.corpus module @xcite . for all the experiments, we split the training dataset in two parts: 80% for training and 20% for validation using a stratify approach. the parameters used in the training phase were: batch size of 32 and 5 epochs. ## results according to the official metrics (f1-score for the positive class), our results are all around 0.60 -0.65, being the best model bilstm neural network with bert as embedding + nude detection, with a 0.665 f1-score. with this result, we obtained the 43 place in the ranking. as we expected, the model that obtained the best results during the training phase also obtained the best result in the evaluation phase. ## conclusions in this paper, several approaches and systems descriptions on task 5 (subtask a) in semeval 2022: multimedia automatic misogyny identification are detailed. the main aim was to develop various deep learning models and check how multi-modality of text and image could help achieve better classification results. six different models were developed and bilstm neural network with bert as embedding + nude detection (0.45 threshold) was the definitive one. after training and analyzing each model, we achieved an f1-score of 0.665 in the evaluation phase for class "1". we can conclude that merging text and image classifiers improves the results in the task of misogyny detection in memes. in future works, we intend to improve our image classifiers models. also, we want to use other pretrained models based on transformers.