AIDA - UPM at S em E val-2022 Task 5: Exploring Multimodal Late Information Fusion for Multimedia Automatic Misogyny Identification (2022)

Abstract: This paper describes the multimodal late fusion model proposed in the SemEval-2022 Multimedia Automatic Misogyny Identification (MAMI) task. The main contribution of this paper is the exploration of different late fusion methods to boost the performance of the combination based on the Transformer-based model and Convolutional Neural Networks (CNN) for text and image, respectively. Additionally, our findings contribute to a better understanding of the effects of different image preprocessing methods for meme classification. We achieve 0.636 F1-macro average score for the binary subtask A, and 0.632 F1-macro average score for the multi-label subtask B. The present findings might help solve the inequality and discrimination women suffer on social media platforms.

## introduction the proposed task, semeval-2022 task 5 multimedia automatic misogyny identification (mami) @xcite consists in the identification of misogynous memes in english language, taking advantage of both text and images available as a source of information. overall, our proposed method consists of a multimodal approach combining different features (e.g., logits, probabilities, embeddings) of a text transformer-based model and an image cnn model in a late fusion approach. this late fusion step implies that both models are trained and finetuned separately to the task. then, the features from each model are concatenated and jointly used as input for a final classifier that combines their knowledge to obtain a final prediction (see figure 1 ). different preprocessing steps, text and image models, concatenated feature combinations, and classifiers are explored to obtain the final multimodal architecture. our presented method has been developed for sub-task a and b from the mami competition independently. sub-task a consists of misogynous meme binary classification, where a meme should be categorized either as misogynous or not misogynous. on the other hand, sub-task b requires a more detailed multi-label classification where misogynous content should be recognized among potential overlapping categories such as stereotype, shaming, objectification, and violence. it is noteworthy that our multimodal late fusion method outperforms single models in both subtasks, being more remarkable in complex sub-task b. similarly, considering that both sub-tasks share the same data, the results of model evaluation on both tasks show how the model trained on complex sub-task b can achieve the same results as a model trained only on binary sub-task a. therefore, future studies should investigate the complexity and pruning of the required models. this paper provides new insights into information fusion for tackling multimodal tasks, presenting an in-depth exploration of different late fusion approaches and image processing steps. the presented work might help identify malicious behaviours towards women on social media. ## background misogyny comprises every hateful and prejudicial action against women, ranging from discrimination, objectification, violence and disdain affecting women to all the types of manifested male superiority like patriarchy, androcentrism and privilege @xcite . misogynist posts represent one type of hate speech on online social networks, but it is complex to distinguish them from other sorts of offensive discourses in an automated way @xcite . whereas research shows a survey of methods for misogyny recognition in text including traditional methods and their ensembles and neural networks @xcite , studies reviewing mysogyny in images are more scarce, and more work can may be found under the generalization of sexist content @xcite than under the particular topic of misogyny. these sexist images can be in the form of memes, multimedia content with a humorist goal composed of photos and/or illustrations with some text on it @xcite . furthermore, this initially shows that sexist and also misogynist content on social networks do not need just the isolated use of image and text processing but the combination of both. in fact, having images or texts on their own may not lead to hateful speech and only their mix is offensive @xcite , underlining the major necessity of architectures that encode the global meaning of memes. on the one hand, the latest advances in nlp for text classification include the use of transformers, which has demonstrated to be successful in the detection of misogyny @xcite . these distributional models encode the meaning of texts in vectors that also capture the context @xcite . however, they may not be so optimal for the detection of subclasses, for instance between the absence and the implicit or sarcastic presence of aggressiveness (samghabadi et al., 2020), or when the female subject that is attacked is not present in the text @xcite . on the other hand, when images are also taken into account, the use of convolutional neural networks (cnns) for images is still present @xcite through image-based state-of-the-art models such as vgg16, resnet, densenet and inception. specifically, vgg16 works by itself for the prediction of offensive memes and succeeds when combined with different models for the text inside them (lstms, bilstms and cnns), which were later compared @xcite , but when the comparison is among vision models, depending on the dataset and the multimodal approach followed, resnet50 can surpass vgg16 and resnet152 for vgcn-bert combined pipelines for hateful images detection @xcite , but inception outperforms the f1scores from resnet50 for multimodal approaches with bilstms for 'troll' (sarcastic or offensive) meme detection @xcite , and just using densenet alone for meme emotion recognition can be even better than resnet alone and than either densenet or resnet in image joining forces with bert for the textual features @xcite . alternative implementations for vision are transformers such as visualbert, which represents an architecture with image and text mod-els @xcite or clip, which predicts the text that better describes the images @xcite . recent research has shown that joining the probabilities from the cnn classification and the output statistical variables obtained after training a successive classifier (huertas-tato et al., 2022) improves the task. these results invite to use this advance in information fusion for practical applications such as hateful images and, specifically, misogynist memes. in line with this theoretical framework, our work will concatenate the different outputs (e.g, logits, probabilities, last hidden embeddings) from cnn-based image classification and from transformers-based text classification towards a better performance in mami. ## system overview as previously mentioned, our proposed approach is depicted in figure 1 . this section details the preprocessing steps, the text and image models employed, and the methodology followed to train the final multimodal late fusion classifier that combines different features. ## experimental setup as previously mentioned, both sub-tasks share the same data and the official metric for system evaluation, f1-macro averaged. the dataset consists of 10.000 memes and its corresponding text transcriptions. to develop the proposed approach, balanced data for sub-task a is split into 64% train, 16% validation and 20% test in a stratified way using scikit-learn package @xcite with 42 as random state. regarding multi-label sub-task b where the data is unbalanced, the data is split into 64% train, 16% validation and 20% test using the "iterative train test split" method from scikitmultilearn package @xcite to equally represent the different combina-tion of overlapping labels in the splits. to obtain the best results and avoid overfitting, we optimized several hyperparameters. ## conclusion as a conclusion of the results obtained in the exploration and evaluation of models for the development of the multimodal late information fusion architecture, it is evident that the contribution between image and text is different, being the text much more informative in both sub-tasks. in the case of the image model, it is interesting to note that the different proposed pre-processing techniques do not seem to have a beneficial effect on model training. although further future analysis is needed, one possible justification could be that the information supplied by the text present in the images provides valuable information rather than noise in the cnn models. finally, it is also important to point out that in sub-task a, the multimodal strategy is not as relevant as expected as the baseline strategy that combines image and text information and our approach has the lowest scores in in general, our results from the presented multimodal late fusion approach are encouraging to counteract malicious misogynistic behavior against women on social media.