UAEM - ITAM at S em E val-2022 Task 5: Vision-Language Approach to Recognize Misogynous Content in Memes (2022)

Abstract: In the context of the Multimedia Automatic Misogyny Identification (MAMI) competition 2022, we developed a framework for extracting lexical-semantic features from text and combine them with semantic descriptions of images, together with image content representation. We enriched the text modality description by incorporating word representations for each object present within the images. Images and text are then described at two levels of detail, globally and locally, using standard dimensionality reduction techniques for images in order to obtain 4 embeddings for each meme. These embeddings are finally concatenated and passed to a classifier. Our results overcome the baseline by 4%, falling behind the best performance by 12% for Sub-task B.

## introduction the multimedia automatic misogyny identification (mami) competition @xcite consists in the identification of misogynous memes, taking advantage of both text and images available as source of information. the task was organized around two main sub-tasks. sub-task a: a basic task about misogynous meme identification, where a meme should be categorized either as misogynous or not misogynous; sub-task b: an advanced task, where the type of misogyny should be recognized among potential overlapping categories such as stereotype, shaming, objectification, and violence. in this paper, we present a proposed solution for sub-task b only, which consists of a framework for extracting lexical-semantic features from text and combine them with semantic descriptions of images, together with image content representation. we propose to use a pre-trained bert model @xcite as lexical feature enhancer, and to use a vision-language model @xcite as visual descriptor. the rest of this paper is organized as follows. we introduce our multimodal framework in section 2. in section 3 we present and discuss our results. we conclude the paper in section 4. ## methods fig. 1 shows a diagram of the method that we propose to describe memes using both visual and text inputs. it is known that local descriptors can provide rich representations, as they can extract fine details from local areas within documents @xcite . therefore, we compute two types of description for each modality -visual or text-. namely, global and local descriptors, and then concatenate the resulting four descriptors into a single vector that we use for classification. more specifically, we relied on transfer learning coupled with fine tuning procedures, in which one pre-trained model was further adjusted for each of the four input modalities, and for the five classes problem presented by the mami competition. ## results and discussion fig. 3 shows the confusion matrices produced by our model on the instances of the mami challenge, computed on the joined training and validation sets. we show five confusion matrices because the categories are not mutually exclusive. each of the matrices contains true negatives [0,0], false positives [0,1], false negatives [1,0], and true positives @xcite . in all cases, the accuracy is greater than 0.90, with the "misogynous" class having the highest score (0.97) and the stereotyped class the lowest (0.90). using the f1 measure, the "shaming" and "violence" (the most unbalanced) classes have the worst performance with 0.68 and 0.71 respectively. this situation is due to the small number of true positive instances in these two classes, which might bias the model towards the prediction of the negative label. meanwhile, the "misogynous", "stereotype" and "objectification" (the most balanced) classes have a similar performance with an f1 score above 0.82. moreover, the classes with the lowest performance, have a proportionally much lower number of training examples. this fact limits the fine tuning of the model parameters in the overall training process. fig. 4 visualizes the roc curve, which represents the rate of true positives versus the rate of false positives. as in the confusion matrices, it can be observed that the best performance is obtained in the class "misogynous", while the worst occurs with the classes "shaming" and "violence". ## conclusions in this paper, we proposed a framework for extracting lexical-semantic features from text and combine them with semantic descriptions of images in the context of the multimedia automatic misogyny identification (mami) competition 2022. our results overcame the baseline by 4%, but fell behind the best performance by 12% for sub-task b. our model's performance could be explained by the un- as future work we propose three alternatives: 1) to fine tune the classification threshold of the sigmoid output layer of the model, independently for each class, i.e., not all classes need to have 0.5 as clas-sification threshold; 2) optimize the loss function directly on the f1 score; and, 3) weight the contribution of each class in the overall loss function during backpropagation.