INF - UFRGS at S em E val-2022 Task 5: analyzing the performance of multimodal models (2022)

Abstract: This paper describes INF-UFRGS submission for SemEval-2022 Task 5 Multimodal Automatic Misogyny Identification (MAMI). Unprecedented levels of harassment came with the ever-growing internet usage as a mean of worldwide communication. The goal of the task is to improve the quality of existing methods for misogyny identification, many of which require dedicated personnel, hence the need for automation. We experimented with five existing models, including ViLBERT and Visual BERT - both uni and multimodally pretrained - and MMBT. The datasets consist of memes with captions in English. The results show that all models achieved Macro-F1 scores above 0.64. ViLBERT was the best performer with a score of 0.698.

## introduction social media and anonymity enable the spread of hateful speech, which explains why misogyny is prevalent and abundant on the internet. not only is it present, but also increasingly so, as confirmed by @xcite . the platforms that contribute to the sharing of hateful content dedicate a considerable amount of human effort in detecting, analyzing, and eventually removing these contents. the task is demanding due to the nature of the posts, which are frequently not straightforward -they often contain irony and slang. additionally, the textual information needed to automatically classify a post as misogynistic might be part of an image, in the form of a meme. that prevents sexist posts from being immediately detected by algorithms that rely solely on textual input. in this paper, we describe the training and usage of five different multimodal models applied to detecting misogynistic memes in the scope of semeval-2022 task 5 multimodal automatic misogyny identification (mami) @xcite . we explain the distinction between the models and compare their performances in light of differences in pretraining (unimodal or multimodal). among the five models, the one which achieved the highest score was vilbert, reaching the 32 nd position on the leaderboard (out of 83 participants), with a score of 0.698. the one which performed the worst was mmbt-grid, with a score of 0.649. the remainder of this paper is organized as follows: section 2 covers background and related work. section 3 presents an overview of our system. the experimental setup is described in section 4. section 5 presents our results. then, section 6 concludes the paper. ## background and related work one crucial aspect of this task is the multimodality of inputs. most of the time, a meme requires both textual and visual information to be correctly understood. not only because the punch line usually comes in written form, but also because texts and images often contradict each other for humouristic purposes. take for example figure 1 . the text alone indicates a positive feeling towards an object that makes sandwiches. the image, if one would remove the caption, would show a woman. but when taken into consideration simultaneously, it is a sexist meme implying that women exist to make men sandwiches. we took part only in subtask a, in which the goal of the model is to take a meme such as the one in figure 1 as input and indicate whether it is misogynistic or not. the hateful memes challenge @xcite is similar to mami since both address hateful multimodal contents. participants in the hateful memes challenge received a dataset of memes with visual as well as textual inputs and had to predict whether the memes were hateful. mmf (singh et al., 2020 ## system overview we used mmf @xcite to train five models on the mami dataset. these models can be briefly described as follows. ## experimental setup the dataset used to train all models was the one provided by the organization team. the data has not been augmented or modified in any way. training data consists of 10,000 memes, trial data has 100 memes, and validation data has 1,000 memes. the mmf framework comes with implementations of state-of-the-art models, preconfigured with hyperparameters. in our experiments, the default configurations of the models were used. all models share the same values for the main configurations, such as 1e-5 for learning rate, 22,000 for maximum number of steps, 128 tokens at most for text processing and, due to memory limitations, one hyperparameter was set to a fixed value, that is, models used batch sizes of 16 for training. hyperparameter optimization could, therefore, be applied to the models to obtain better results. the configuration files are open to inspection and change, given that they are publicly available at facebook research's github repository foot_0 . the specific commit that was used for this work is available here foot_1 . the training process of all models, excluding mmbt-grid, uses image features, which were not included with the dataset supplied by the organization team. we relied on a script included in mmf to extract these features, using resnet-152. the main evaluation metric used in the task and during training is macro-f1. here we also report true positive (tp), true negative (tn), false positive (fp), and false negative (fn) rates. ## results in this section, we report on our experimental results organized around four questions. ## conclusion in this paper, we described our submission to semeval-2022 task 5. using hateful memes and mmf as inspiration, we wanted to replicate their methods in a similar context. although hateful and misogynistic memes share some overlap, there are important distinctions between them, regarding different vocabulary, context, and targets (i.e., hate can be directed towards anyone, while misogyny cannot). in our experiments, we trained five models and confirmed that they reach similar performances in this dataset as they do in hateful memes. our best model, vilbert, reached a f1 score of 0.698 and ranked 32 nd out of 83 on the leaderboard. we showed that using a majority voting system with all models would not be beneficial. the models could be further improved by hyper-parameter