UMUT eam at S em E val-2022 Task 5: Combining image and textual embeddings for multi-modal automatic misogyny identification (2022)

Abstract: In this manuscript we describe the participation of the UMUTeam on the MAMI shared task proposed at SemEval 2022. This task is concerning the identification of misogynous content from a multi-modal perspective. Our participation is grounded on the combination of different feature sets within the same neural network. Specifically, we combine linguistic features with contextual transformers based on text (BERT) and images (BEiT). Besides, we also evaluate other ensemble learning strategies and the usage of non-contextual pretrained embeddings. Although our results are limited, we outperform all the baselines proposed, achieving position 36 in the binary classification task with a macro F1-score of 0.687, and position 28 in the multi-label task of misogynous categorisation, with an macro F1-score of 0.663.

## introduction this manuscript describes the participation of the umuteam in the multimedia automatic misogyny identification (mami) shared task @xcite , proposed at semeval 2022. this sharedtask consists in the identification and categorisation of misogynous content from a dataset composed of memes @xcite . a meme is essentially a pictorial content with an overlaying text that pretends to be funny. however, some of these memes are being used as a form of hate against women, with sexist messages in online social networks that amplify misogynous traits such as sexual stereotyping or gender inequality. mami shared task proposes two challenges. a binary classification task, in which each meme should be labelled as misogynous or not misogynous, and a multi-label classification task, to categorise different misogynous traits, namely shaming, stereotype, objectification, and violence. ## background information from the last years, the number of shared tasks in workshops regarding hate-speech and misogyny detection are increasing. to name just but a few, in italian there is the evalita 2018 dataset @xcite ; in spanish, the ami 2018 dataset @xcite and the exist dataset @xcite . in german, the germeval 2021 @xcite dataset. the common approaches for misogyny detection and categorisation consists in the training of an automatic machine learning classifier. for example, the authors of @xcite compiled and labelled a corpus from twitter focused on misogynous content, and evaluate several feature sets and machine-learning models. in spanish, a similar approach was conducted in @xcite , in which the authors released the spanish misocorpus 2020. this dataset is organised into three splits: (1) varw (violence against relevant women), focused on aggressive messages on twitter to women who have gained social relevance; (2) sela (european spanish vs that of latin america), focused on distinguish between misogynistic messages from spain and latin america; and (3) ddss (discredit, dominance, sexual harassment and stereotype), focused on general traits related to misogyny. the spanish misocorpus 2020 is balanced and contains 3841 misogynous documents, annotated by three human annotators. it is worth noting that our research group evaluated of a set of hand-crafted linguistic and negation features along with spanish pre-trained contextual and non-contextual embeddings for detecting hatespeech @xcite . these work included two datasets concerning misogyny and sexist behaviour. ## dataset ## methodology for solving the challenges proposed in mami, we build a system which architecture is depicted in figure 1 . it a nutshell, our system works as follows. first, we select some of the documents of the training mami dataset to create a custom validation dataset. next, we extract a subset of language-independent linguistic features (lf), noncontextual sentence (se) and word embeddings (we) from fasttext, the contextual word embeddings from bert (bf), and the image embeddings from beit (bi). second, we train several neural network models by performing an hyperparameter tuning process. the models evaluated included one model per feature set, and models based on several feature sets together. besides, we evaluate two ensembles based on soft voting (mode) and averaging all the probabilities (mean) of the neural networks trained with each feature set. to handle the multi-label challenge, we repeat this process per trait. that is, we evaluate the problem as a binary classification problem per trait. next, some insights of the feature sets involved are given. as the memes are images with overlaying text, this shared-task has a multi-modal perspective. our proposal uses several feature sets based on texts, and one for the images. first, we use the umutextstats tool to obtain a set of relevant psycho-linguistic features (lf). this tool has already been used in studies related to misogyny, such as @xcite . the lf included low-level linguistic categories concerning phonetics and syntax's, and high-level features related to semantics and pragmatics, including features proper from figurative language (del pilar salas-zárate et al., 2020). moreover, these kinds of features have proven to be effective for performing other automatic classification tasks such as irony and satire identification (garcía-díaz and valencia-garcía, 2022). as some of the dictionaries of umutextstats are not translated to english, we select a subset of language-independent linguistic features, based on linguistic metrics, part-of-speech features and the usage of social media jargon. second, we extract sentence and word embeddings using the pre-trained fasttext model @xcite . third, we use contextual sentence embeddings from bert @xcite , which sentence embeddings are obtained in a similar manner as described at s-bert @xcite . forth, we use visual embeddings from beit @xcite , which is a self-supervised model trained with imagenet-21k with more than 21000 labels. beit learns the embeddings images as a sequence of fixed-size elements using relative position. this allows us to perform the classification using a mean-pooling strategy from the final hidden states of the patches instead of placing a linear layer on top of the final classification token. however, the suggested way to fine-tune the model for performing downstream tasks is to attach a new linear layer that uses the last hidden state of the training (60%) oficial testing split custom splitter validation (20%) bert embeddings (bf) word embeddings (we) linguistic features (lf) sentence embeddings (se) dataset preprocessing ray-tune hyper-parameter tunning knowledge integration ensemble learning mami dataset (memes and their transcription) image transformers beit (bi) results task 1 results task 2 classification token as a representation of the whole image. to obtain the embeddings from bert (for text) and beit (for images) we conduct an hyperparameter optimisation stage using raytune @xcite with a tree of parzen estimators (tpe) to select the best combination of the hyperparameters over 10 trials. the hyperparameters evaluated and their interval range are: (1) weight decay (between 0 and .3), (2) training batch size ([8, 16]), (3) warm-up steps ([0, 250, 500, 1000]), (4) number of training epochs ([1-5]), and (5) learning rate (between 1e-5 and 5e-5). once all features are obtained, we train a neural network per feature set. the training of each neural network is performed with hyperparameter optimisation. each training involved: (1) 20 shallow neural networks, that are multi-layer perceptrons (mlp) composed by one or two hidden layers with the same number of neurons per layer connected with one activation function (linear, relu, sigmoid, and tanh); (2) 5 deep-learning networks, that are mlp between 3 and 8 hidden layers, in which the neurons per layer are disposed for each layer in different shapes, namely brick, triangle, diamond, rhombus, and funnel, and connected with an activation function (sigmoid, tanh, selu and elu). the learning rate of the deep-learning models is 10e-03 or 10e-04. besides, on the neural networks with the pre-trained word embeddings from fasttext (we) we also evaluate 10 convolutional neural networks (cnn) and 10 bidirectional recurrent neural network layers (bilstm). in all experiments, we evaluate two batch sizes: 16 and 32. these small values were selected because the training split was balanced, and a dropout mechanism ([false, .1, .2, .3]) for regularisation. apart of the neural networks trained with the feature sets separately, we evaluate different forms for combining the strengths of each feature set in the same system. the combination of the feature sets is performed using two strategies: (1) knowledge integration, in which each feature set is used as input of the same neural network. for this, we train another neural network repeating the hyperparameter optimisation stage; and (2) ensemble learning, in which the output of each neural network model trained with a feature set is combined by averaging the predictions or calculating the mode of the predictions. ## results and discussion each system was evaluated using the custom validation split (see section 3). the results for the first challenge are depicted in as it can be observed in for the second challenge, the results with the custom validation split are depicted in the best result for the first challenge is achieved by the srcb_roc team, with an f1-score of 83.4. it is worth mentioning that, although there was a restriction of the number of accounts available in codalab per team and user, the organisers of the task are not able to control it. nevertheless, in the official leader board the second best result was also achieved by the team srcb_roc, with an f1-score of 81.1. however, as the team name is the same, we have removed this result from the table. therefore, for the official results we ask to the reader to check the official results published in the overview of the task. the best result was a f1-score of 73.1, with a triple tier between teams srcb_roc, tibva, and pafc. our best proposal, however, achieved position 28 in the official leader board, outperforming all baselines. ## conclusions in this paper the participation of the umuteam in the mami shared task, concerning the identification and categorisation of misogynous content in memes, is described. our approach for solving the binary and multi-label classification tasks consisted in the combination of a set of language-independent linguistic features with contextual images and textual features obtained from the documents. our best result was achieved in the misogynous categorisation task, with an macro f1-score of 66.3, reaching position 28 in the ranking. we consider that the weakest point of our proposal is that we have not handle class imbalance in the testing dataset. however, we have evaluated some strategies that have improve our results, as reducing the number of instances and the application of class weight. as further work, we will evaluate data augmentation techniques, both for images and for text in order to deal class imbalance.