PAIC at S em E val-2022 Task 5: Multi-Modal Misogynous Detection in MEMES with Multi-Task Learning And Multi-model Fusion (2022)

Abstract: This paper describes our system used in the SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification (MAMI). Multimedia automatic misogyny recognition consists of the identification of misogynous memes, taking advantage of both text and images as sources of information. The task will be organized around two main subtasks: Task A is a binary classification task, which should be identified either as misogynous or not misogynous. Task B is a multi-label classification task, in which the types of misogyny should be identified in potential overlapping categories, such as stereotype, shaming, objectification, and violence. In this paper, we proposed a system based on multi-task learning for multi-modal misogynous detection in memes. Our system combined image features with text features to train a multi-label classification. The prediction results were obtained by the simple weighted average method of the results with different fusion models, and the results of Task A were corrected by Task B. Our system achieves a test accuracy of 0.755 on Task A (ranking 3rd on the final leaderboard) and the accuracy of 0.731 on Task B (ranking 1st on the final leaderboard).

## introduction in the era of mass online communication, more and more people like to share their thoughts on social media platforms. social media platforms provide users with the ability to express themselves freely.however, it has also led to a rise in cyber hate, such as bullying, sarcasm, and misogyny, on the internet. a study has shown that women have a strong presence online, particularly in image-based social media such as twitter and instagram: 78% of women use social media multiple times per day compared to 65% of men. the web has opened up a whole new world of opportunities for women, but systematic inequality and discrimination in the real world are also found in online spaces in the form of offensive content against them. now that we live in a world where everything is connected through social media, hate speech against women used to be limited to a specific place or time. there's no need to wait for a specific time or place. all you have to do is type a few keys on the keyboard. because people on big social media sites make millions of posts every day, it is not possible to manually check all misogynistic posts. to help human curators, it is important to make algorithms that can tell when users post inappropriate content. memes are one of the most popular ways to communicate on social media platforms. a meme is essentially an image characterized by a pictorial content with an overlaying text a posteriori introduced by human, with the main goal of being funny and/or ironic. most of them are made to be funny, but in a short time, people started to use them as a form of hate against women, which led to sexist and aggressive messages in online environments that made the sexual stereotypes and gender inequality in the real world even worse. misogyny is a type of offensive speech directed at women. it is common on all social media platforms and is becoming more and more of a problem. previous work on automatic misogyny detection mostly focused on text mode, and they came up with a bunch of supervised methods, like traditional machine learning methods with lexical features and deep learning methods. it can't be certain, though, if memes are misogynistic if only text mode is used to detect them. memes are intuitively important for automatic misogyny identification tasks. in this paper, we describe a system that we submitted for detecting misogynous memes. more specifically, we introduce an ensemble model that uses a multi-task learning mechanism based on multi-modal inputs (image information and text information). in our system, image feature representation was extracted by the pre-trained model of convnext, and text feature representation was ex-tracted by different pre-trained models (deberta-v3large and roberta-large model). the two features then undergo representation fusion, where they are transformed into reconstructed representation vectors and pumped into a classification layer to yield the final result. finally, a modality fusion layer performs a weighted average on the fusion results, which results in different text and image features, and the threshold of the final results is adjusted. ## background many researches worked on identification misogynous in texts. earlier methods extract carefully engineered discrete features from texts, including n-grams, keyword's sentiment, punctuations, emoticons, etc @xcite , @xcite . recently, researchers have used the powerful technology of deep learning to obtain a more accurate semantic representation of twitter text, including cnn (convolutional neural network), lstm (long short term memory) and pre-trained model, etc. @xcite proposed a technique based on combining several simpler classifiers into a more complex blended model. the model considers the probability of classes calculated by simpler models (logistic regression, naive bayes, and support vector machines -svm) . another method from @xcite was proposed, which three classifiers trained by using svm with a linear kernel, random forests (rf) and gradient boosted trees (gbt). in the testing stage, the same way of text pre-processing and feature extraction is applied to test instances separately, and each pair of two out of the three trained classifiers (svm+rf, svm+gbt and rf+gbt) are fused by combining the probabilities for each class by averaging. the primary driver was the renaissance of neural networks, particularly convolutional neural networks (convnets). the introduction of alexnet @xcite precipitated the "ima-genet moment", ushering in a new era of computer vision. since then, this field has developed rapidly. representative convnets like vggnet @xcite , resne(x)t @xcite , densenet @xcite , mo-bilenet @xcite and regnet focused on different aspects of accuracy, efficiency and scalability, and popularized many useful design principles. however, the introduction of visual transformer (vit) completely changed the land-scape of network architecture design, which soon replaced convnets and became state-of-the-art image classification model. although the effect of swin transformer is much better than resnet on the premise of the same size. however, due to the shift operation, it is difficult to design different networks for different sizes of inputs and restart training. and like detection transformer (detr), the convergence is too slow when training. this year, researchers from facebook ai research (fair) and uc berkeley reexamined the design space and tested the limits that pure convnet can reach, which shows that the performance of convolutional neural network is no less than that of visual transformer. this series of pure convnet models is named con-vnext. it is constructed entirely from standard convnet modules and competes with transformers in terms of accuracy and scalability. however, little has been revealed by far on how to effectively combine textual and image information to automatic misogyny identification of memes. @xcite simply concatenate manually designed features or deep learning based features of texts and images to make prediction with two modalities. @xcite proposed a hierarchical fusion model to deeply fuse three modalities. different from there works, we proposed a multitask learning and multi-model. ## system overview figure 1 shows the architecture of our proposed. in this work, we treat text and image classification tasks as two models. image features are mined using a fine-tuned image classification model based on convnext. text features are extracted using a fine-tuned sentence embedding based on the pretrained roberta model. image features and text features are fused by direct splicing. a classification layer is connected to the fused model to obtain the category probability of memes. additionally, we create another fused model that adopts the pretrained model deberta instead of roberta for text feature exaction progress. our final submitted labels are voted on by the results inferred by these two proposed models (convnext + roberta and convnext + deberta). the threshold value selected for the weighted voting method is 0.405. the classification label returns 1 when the classification probability is greater than 0.405, otherwise it returns 0. ## results in this section, the results of the experiments for our runs will be discussed and compared to the results published in ## conclusion in this paper, we proposed a system to make full use of two modes (image and text) for solving the challenging multi-mode misogynous meme detection task. we extracted the image features and text features using different pre-trained models and weighted averaged the results after fusing the features of image and text. the system performs well in identifying misogynistic memes, achieving 77.85% percent accuracy on taska and 73.1% percent accuracy on taskb.