U ni BO at S em E val-2022 Task 5: A Multimodal bi-Transformer Approach to the Binary and Fine-grained Identification of Misogyny in Memes (2022)

Abstract: We present our submission to SemEval 2022 Task 5 on Multimedia Automatic Misogyny Identification. We address the two tasks: Task A consists of identifying whether a meme is misogynous. If so, Task B attempts to identify its kind among shaming, stereotyping, objectification, and violence. Our approach combines a BERT Transformer with CLIP for the textual and visual representations. Both textual and visual encoders are fused in an early-fusion fashion through a Multimodal Bidirectional Transformer with unimodally pretrained components. Our official submissions obtain macro-averaged F _1 =0.727 in Task A (4th position out of 69 participants)and weighted F _1 =0.710 in Task B (4th position out of 42 participants).

## introduction evolving from the two previous editions of the automatic misogyny identification initiatives @xcite , the multimedia automatic misogyny identification shared task at semeval 2022 (mami) targets multimodal data for the first time. within mami, @xcite propose two classification tasks: task a a basic task about misogynous meme identification, where a meme should be categorized either as misogynous or not. task b an advanced task, where the type of misogyny should be recognized among potentially overlapping categories: stereotyping, shaming, objectification and violence. the increasing volume of meme posts on social media renders the development of models able to identify malicious instances timely. the task is more challenging than when dealing with text alone because, in general, both the textual and the visual channels play an indivisible role in conveying the desired message. 1 we build upon our previous experience in identifying misogyny and aggressiveness in text @xcite ) and approach both multimodal tasks with a supervised multi-modal bitransformer model (mmbt) @xcite . we use bert-base-uncased-hatexplain @xcite and bert-base-uncased @xcite for the textual embeddings, and clip @xcite for the visual ones. we also build two unimodal baselines to compare against. 2 our experiments aim at understanding if and to what extent our multimodal model outperforms the two unimodal ones that address the problem separately. since meme classification is a challenging task due to its multimodal nature, we shed some light on which component should weigh more in the decision process -text or image-by observing the impact of both modalities in the predictions. our official submission for task a achieved a macro-averaged f 1 score of 0.727, whereas that for task b obtained a weighted f 1 score of 0.710, positioning our team in the 4th position in the task, for both tasks. in addition, we identify the linguistic and visual elements which are potentially responsible for the misclassification. we perform an error analysis to check whether misclassified memes rely heavily on external world knowledge and/or are subtle and implicit, as we believe that those two aspects cause struggle to the models. the rest of the paper is structured as follows. section 2 provides essential definitions for this task, such as misogyny and memes. we then move on to dataset description as well as a summary of related work. section 3 describes our models for both tasks. section 4 describes the experimental setup. our results are presented and discussed in section 5. section 6 presents our error analysis. section 7 concludes with a summary of our findings. ## system overview our approach is based on the multimodal bitransformer model (mmbt) @xcite . mmbt fuses image and text embeddings in an early fashion. mmbt jointly finetunes unimodally pretrained text and image encoders by projecting image embeddings to the text token space. figure 1 represents the model architecture. mmbt combines two segments: segment 0 corresponds to the text, whereas segment 1 corresponds to the picture. they are fed together to use attention over both modalities at the same time. each token is indexed according to its position from 0 to the maximum text length, which we set to 80. each image representation is indexed from 0 to 640. the original mmbt combines bert @xcite and resnet @xcite . we consider other models. for the textual embedding we tried with bert-base-uncased-hatexplain @xcite , a version of bert trained on identifying hate speech. for the image embedding we used clip, since it has outperformed all alternative models in a large variety of multimodal tasks, including ocr, action recognition in videos, geolocalization, and various types of fine-grained object classification @xcite . clip is pre-trained on the task of predicting which caption should be tied together with a given image. in this way it learns state-of-the-art image representations from scratch, enabling zero-shot transfer of the model to downstream tasks. the two embeddings are fused through mmbt. for task a we use the sigmoid activation function for the output layer and threshold at 0.5 to discriminate between misogynous or not. for task b we adopt a binary relevance approach @xcite , combining four binary classification models. the output for each classifier is a sigmoid function too. we opt for this approach after observing that treating the classes separately increased the performance in a multi-class model predicting misogynous, misogynous-aggressive or none @xcite . this approach allows us to predict multiple mutually non-exclusive classes. we apply a heuristic to refine the multi-label decisions in task b. all four decisions are turned off if an instance had not been predicted as misogynous by our task-a model. pre-processing since clip requires square images, following neskorozhenyi (2021) we produce 288×288 pixel versions of all memes. the memes come in different sizes and orientations, hence we rescale them until the largest side reaches 288 pixels respecting the aspect ratio and pad to fill the empty pixels in the square. then, we slice the resized images into three equal parts horizontally if the image orientation is landscape and vertically if it is portrait to obtain both global and local image features. we extracted four vectors for each image: a vector for each part encoding spatial information and one for the whole image. we used the pillow library @xcite to perform these operations. no preprocessing is applied to the text, other than applying the berttokenizer @xcite . ## experimental setup we shuffled the training set and take 10% of the data for development preserving the class distribution through stratified random sampling @xcite . we trained three alternative models to identify the best possible configuration. uni txt is a bertbased unimodal system that considers the text alone. uni img is a clip-based unimodal system that considers the image alone. hyperparameters for multi, we tried learning with different numbers of learning epochs, in range @xcite . the best validation performance was obtained after 5 epochs in task a in the development set. we trained over 5 epochs in task b. for both uni txt we train over 4 epochs. for the uni img we train over 5 epochs. in all cases we saved the model only when an increase in the performance was obtained. since we also aimed at assessing how clip performs in making zero-shot predictions in this task, we used clip without fine-tuning on the training set, to check whether it could be effectively used to detect misogyny without prior annotation. we considered batch sizes of 16 and 32, with the former consistently performing better. we used a learning rate of 2e-4, the madgrad optimizer @xcite , and a binary cross-entropy loss function. the results reported in section 5 are obtained with a model trained during 5 epochs for task b and 6 epochs for task a with 16 as the batch size. ## results in this section we present the results obtained by our submissions to both task a and task b. ## qualitative analysis in this section we present a qualitative analysis of the results to further examine the strengths and weaknesses of our approach. ## conclusions we presented our participation to the multimedia automatic misogyny identification shared task. we addressed two problems: spotting whether a meme is misogynous and, if it is, what kind of misogyny it expresses. we compared unimodal models (text only and image only) with a multimodal model based on multimodal bi-transformers. our imageonly model performs better than the text-only one, suggesting that the visual information might be easier to capture than the textual one. our multimodal approach performs the best in both tasks. the errors come from more false positives than false negatives. from our error analysis we observed that stereotyping and shaming are the most misclassified categories. this proves that more focus on subtle and implicit forms of misogyny and sexism is needed.