YMAI at S em E val-2022 Task 5: Detecting Misogyny in Memes using V isual BERT and MMBT M ulti M odal Pre-trained Models (2022)

Abstract: This paper presents a deep learning system that contends at SemEval-2022 Task 5. The goal is to detect the existence of misogynous memes in sub-task A. At the same time, the advanced multi-label sub-task B categorizes the misogyny of misogynous memes into one of four types: stereotype, shaming, objectification, and violence. The Ensemble technique has been used for three multi-modal deep learning models: two MMBT models and VisualBERT. Our proposed system ranked 17 place out of 83 participant teams with an F1-score of 0.722 in sub-task A, which shows a significant performance improvement over the baseline model’s F1-score of 0.65.

## introduction participating in online social networks (osn) has become more common nowadays, especially for women, as 78 percent of them use social media several times per day compared to 65 percent of the men. anyhow, hateful speech against women did not stop until offline methods but also found its way on the web using popular communication tools such as memes. a meme is an online spread of captioned pictures or gifs meant to be funny or critical to people or society. it has the most humor characteristics on social network platforms. however, in a short period, some minorities started to develop memes aiming at gender discrimination, inequality, and spreading hate against women. due to the growth of artificial intelligence (ai), natural language processing (nlp), and computer vision (cv), the machine started to recognize and understand languages and images more than ever. therefore, preventing hate speech and misogyny became more achievable by applying these methodologies. this paperwork presents our participation in semeval-2022 task 5 @xcite which aims to detect misogyny in memes. in this study, multimodal bitransformers (mmbt) @xcite and @xcite have been used to build the proposed model by employing the ensemble technique. the proposed model reaches an f1-score of 0.722 in sub-task a depending on the provided 10k memes dataset and without external datasets. this paperwork is constructed as follows: section 2 presents the related work, followed by section 3, which clarifies the task description. section 4 shows dataset details and preprocessing procedures. section 5 shows off the architecture of the solution system. finally, sections 6, 7, and 8 provide the experiments, results, and conclusion. ## related work several researchers are attracted to detecting hate, harm, sarcasm, and satire on social media @xcite @xcite . internet memes are considered one of the most popular ways to communicate on all topics on social media. since it is a global issue and a world trend, many researchers have started to compete in preventing harmful memes from pervasion. this paperwork @xcite aimed to detect offensive analogy memes to prevent this from spreading out. the focus was here on images as it attracts more people and has richer information than the text alone. one competition from facebook @xcite aimed to detect hate speech in memes as it is hard to be tackled by having humans checking out every meme. from their vision, hate speech was understood as "any communication that disparages a target group of people based on some characteristic such as race, color, ethnicity, gender, sexual orientation, nationality, religion, or other characteristics" @xcite . the dataset consisted of 12540 memes, split between 7834 for non-hate and 4706 for harmful. on the other hand, deep learning models have been successfully applied to learning features for single modalities: text @xcite and images @xcite . moreover, learning-based models that combine elements from multiple modalities are more robust in visual-linguistic tasks such as detecting hateful memes @xcite . ## task description the main objective of this task @xcite is the identification of misogynous memes. memes can be identified using text, images, or both. using both images and text surely produces better results. the task is divided into two sub-tasks: although we did not use any external datasets, it is allowed for participants to do so. ## experiments regarding sub-task a, we have experimented with roberta as it is one of the state-of-the-art language models. we have used huggingface @xcite and pytorch @xcite to implement roberta-base. we chose a maximum sequence length of 64. following this, we have experimented with vgg-16, which is one of the most popular pre-trained cnn models, and implemented it using keras @xcite . we set the model to expect images of size 300x300 and added a dropout layer with a drop rate of 0.3, followed by a dense layer. for roberta and vgg-16, we split the provided dataset into training and validation datasets with a validation split value of 0.1. both roberta and vgg-16 performed poorly, as memes classification tasks require models that understand images and text. subsequently, we have experimented with two vision+language models: mmbt and visualbert. both of them were implemented using huggingface and pytorch. mmbt was experimented with using resnet-152 as image encoder and bert as text encoder. two bert variants were used: bert-base-uncased and bert-base-large. visual embeddings for visualbert were generated using maskrcnn+resnet-101+fpn model checkpoint from detectron2. for mmbt and visualbert, we used k-fold cross-validation with a k value of 5. in other words, the training and validation datasets consist of 8000 and 2000 memes, respectively, and after each epoch, the validation dataset switches to a completely different 2000 memes. this allows the models to train on the entire dataset eventually. in our experiments, mmbt outperformed visual-bert. adamw optimizer was used for all models. figure 1 : the architecture of the final system for sub-task a. ŷ represents the final prediction after applying voting technique between models. the pipeline also illustrates some pre-processing steps before modelling. ## results the final results for sub-task a are produced using the ensemble technique between two mmbt models trained using different random seeds and visualbert. roberta and vgg-16 were not included in the final system as they performed poorly compared to mmbt and visualbert. as shown in looking at figure 2 , which shows the confusion matrix for the final results in sub-task a, we can see that the final system succeeded in predicting 359 non-misogynous memes and 363 misogynous memes. the system also failed to identify 137 misogynous memes and mistakenly identified 141 non-misogynous memes as misogynous. 8 conclusion this paper presented our deep learning system that contended at semeval-2022 task 5. we experimented with different deep learning models, starting with the roberta language model and the vgg-16 cnn model. subsequently, we experimented with vision and language models, such as mmbt and visualbert, which significantly outperformed vgg-16 and roberta. using ensemble technique between two mmbt models and visualbert produced an f1-score of 0.7222 which led to ranking 17 th place in sub-task a.