R ub CSG at S em E val-2022 Task 5: Ensemble learning for identifying misogynous MEME s (2022)

Abstract: This work presents an ensemble system based on various uni-modal and bi-modal model architectures developed for the SemEval 2022 Task 5: MAMI-Multimedia Automatic Misogyny Identification. The challenge organizers provide an English meme dataset to develop and train systems for identifying and classifying misogynous memes. More precisely, the competition is separated into two sub-tasks: sub-task A asks for a binary decision as to whether a meme expresses misogyny, while sub-task B is to classify misogynous memes into the potentially overlapping sub-categories of stereotype, shaming, objectification, and violence. For our submission, we implement a new model fusion network and employ an ensemble learning approach for better performance. With this structure, we achieve a 0.755 macro-average F1-score (11th) in sub-task A and a 0.709 weighted-average F1-score (10th) in sub-task B.

## introduction hate speech against women remains rampant despite many efforts at education, prevention and blocking. misogyny takes place online and offline. especially on social media platforms, misogyny appears in different forms and has serious implications @xcite . currently, automated detection and filtering seem to be the most effective way to prevent hate speech online. however, over the past few years, the rising popularity of memes brought misogyny to a new multi-modal form, which may be more likely to go viral due to their often surprising combinations of text and image that may strike viewers as funny and hence, as eminently shareable. the multi-modality of memes also makes automatic detection more challenging. some memes express their hatred implicitly or through juxtaposition, so they may even appear harmless when considering the text or the image in isolation. semeval-5 2022 multimedia automatic misogyny identification (mami) @xcite aims to identify and classify english misogynous memes. in recent years, the transformer model @xcite has been widely used in natural language processing (nlp) and image processing. transfer learning @xcite with a pre-trained transformer model can save training resources and increase efficiency with less training data @xcite . therefore, in this work, we consider transfer learning to customize three uni-modal models based on the transformer model: i) fine-tuning a pre-trained roberta model for classification (bertc) @xcite ; ii) training a graph convolutional attention network (gcan) using the pre-trained roberta model for word embedding; iii) fine-tuning a pre-trained image model, the vision transformer (vit) @xcite . based on these three uni-modal models, four bi-modal models are trained through our proposed model fusion network, namely bertc-vit, gcan-vit, bertc-gcan, and bertc-gcan-vit. all models are evaluated with 10-fold cross-validation. the macro-average and weightedaverage f1-scores are employed as the metrics for the sub-tasks. ultimately, the ensemble strategy is applied on both the dataset-and the model-level (detailed in section 3.3) for better performance. the remainder of the paper is structured as follows: section 2 introduces the mami challenge and related solutions to the task. our ensemble model is described in section 3, followed by the experimental setup in section 4. finally, our results are shown and conclusions are drawn in sections 5 and 6. ## background the mami dataset contains 10,000 memes as the training and 1,000 memes as the test set; all of these are given together with the text transcription as obtained through optical character recognition (ocr). the reference labels are obtained by manual annotation via a crowdsourcing platform. the challenge is composed of two sub-tasks: sub-task a represents a binary classification task and focuses on the identification of misogynous memes, so each meme should be classified as not misogynous (nomis) or misogynous (mis). subtask b, in contrast, presents a multi-label classification task, where the misogynous memes should be grouped further, into four potentially overlapping categories. the dataset class distribution is illustrated in some researchers have already worked on meme datasets. for example, (sabat et al., 2019) created a hateful memes database, using the bert model to extract a contextual text representation and the vgg-16 convolutional neural network @xcite for image features. then, text and image representations are concatenated to obtain a multi-modal representation. facebook also organized a challenge for the identification of hateful memes in 2020 @xcite . the winner of this challenge adopted an ensemble system with four different visual-linguistic transformer architectures @xcite . the transformer model has shown excellent performance in many tasks, and it also shows promising results in the above studies, based on its use of the attention mechanism to extract the contextual information within a text. however, its ability to capture global information about the vocabulary of a language remains limited @xcite , and we hypothesize that this is even more of an issue in the task at hand, due to the very short texts in the given challenge. for this reason, we combine a transformer model with a graph convolutional network (gcn) @xcite , which may help to address this issue. gcns can be understood as a generalization of cnns, where the data has graph structure and locality is defined by the connectivity of the graph. as input, a gcn receives features that connect to a set of nodes. from layer to layer, the features of a node are updated as weighted combinations of its neighbors' features. in our case, the graph is defined as follows: there is a node for every word in the vocabulary and for every document. the collection of nodes is a fixed-size sliding window with step size 1 is used to gather the word co-occurrence information through the entire dataset. the point-wise mutual information (pmi) is employed to measure the relationship between the words i and j as follows: the adjacency matrix a of the graph is then computed elementwise, as follows: since the graph is undirected, the adjacency matrix is symmetric. finally, the adjacency matrix is normalized by ## system overview in this section, we specify our uni-and bi-modal models. figure 1 depicts the three uni-modal models bertc (1a), gcan (1b), and vit (1c), which form the basis of our further experiments. the bi-modal models are constructed based on trained uni-modal models and our proposed model fusion network, which is further detailed in section 3.2. finally, we apply soft and hard voting ensembles on the trained candidate models. ## experimental setup in the following, we describe our data processing and training pipeline in more detail. ## results in summary, we investigated two configurations, displayed in ## conclusion this paper presents our ensemble-based approach to address two sub-tasks of the semeval-2022 mami competition. the challenge aims to identify misogynous memes and classify them intopotentially overlapping-categories. we train dif-ferent text models, an image model, and via our proposed fusion network, we combine these in a number of different bi-modal models. among the uni-modal systems, all text models show a far better performance than the image model. as expected, our proposed graph convolutional attention network (gcan), which also considers the graph structure of the input data while using pre-trained roberta word embeddings as node features, consistently outperforms the pretrained roberta model. the proposed fusion network further improves the performance by combining the ideas of streamweighting and representation fusion. we additionally adopt 10-fold cross-validation and use a dataset-level soft voting ensemble to obtain better and more robust results. finally, our model-level hard voting ensemble integrates the soft voting ensemble predictions of our best uni-and bi-modal models. our experiments indicate that this layered ensemble approach can significantly improve the model accuracy. ultimately, our submitted system results in an f1-score of 0.755 for sub-task a and 0.709 for sub-task b. overall, we believe that the identification of misogyny in memes is best addressed through bimodal recognition, considering both textual and image information. concerning the text-based clas-sification, we found a graph convolutional attention neural network to be beneficial as an integrative model for transformer embeddings. this helps in the text classification, when the documents are short, as for the given meme classification task. to cope with the bi-modality of the task at hand, we have implemented a range of systems for integrating the information from both streams. an idea that proved to be effective here was that of bringing together the strengths of early fusion and decision fusion in a joint framework. this allowed us to dynamically adjust the contributions of the two modalities through dynamic stream weighting, while still being able to combine information at the feature level across the streams, thanks to the representation fusion branch of our bi-modal systems.