ASR trans at S em E val-2022 Task 5: Transformer-based Models for Meme Classification (2022)

Abstract: Women are frequently targeted online with hate speech and misogyny using tweets, memes, and other forms of communication. This paper describes our system for Task 5 of SemEval-2022: Multimedia Automatic Misogyny Identification (MAMI). We participated in both the sub-tasks, where we used transformer-based architecture to combine features of images and text. We explore models with multi-modal pre-training (VisualBERT) and text-based pre-training (MMBT) while drawing comparative results. We also show how additional training with task-related external data can improve the model performance. We achieved sizable improvements over baseline models and the official evaluation ranked our system 3^{rd} out of 83 teams on the binary classification task (Sub-task A) with an F1 score of 0.761, and 7^{th} out of 48 teams on the multi-label classification task (Sub-task B) with an F1 score of 0.705.

## introduction despite its unique advantages, social media is considered to be one of the harmful elements of society, if not monitored properly. it has become a medium to express hatred towards particular groups, especially women. women have a strong presence online and have become victims of systematic inequality and discrimination which is reflected from the behavior offline. violence has increased to the point where for many girls, abuse is a day-to-day reality. a landmark survey conducted by plan international in more than 20 countries has revealed shocking accounts of escalating online violence against girls and women, with respondents exposed to explicit messages, pornographic photos, cyberstalking, and other forms of internet abuse. the most common type of online harm includes using abusive and insulting language, followed by deliberate embarrassment, body shaming, and threats of sexual violence. one of the most popular communication tools in social media is a meme. it is a combination of image and text, created typically for humor. semeval 2022 task 5 is the first misogynistic meme detection challenge that incorporates several categories such as stereotyping, shaming, objectification and violence. pre-trained language models such as bert @xcite , roberta @xcite , etc., have emerged as the state-of-the-art models for many nlp tasks such as text classification, machine translation, sequence tagging, etc., mainly due to their rich contextual embeddings. hence, we chose a transformer-based architecture to fuse both visual and textual features. two types of pretraining techniques are explored in this paper: textbased and multi-modal-based. instead of directly using a pre-trained text transformer, we further train roberta on task-related data and fine-tune the model with extended visual features extracted from a image classification network (e.g. resnet). we also use ensemble learning to combine the results of the two pre-training techniques. we achieved significant improvement over baselines in both the sub-tasks. we were ranked (1) 3 rd with an f1-macro score of 0.761 in sub-task a and (2) 7 th with a weighted f1 score of 0.705 in sub-task b among 83 teams. we release the code for models and experiments via github foot_0 the rest of the paper is organized as follows: section 2 describes the challenge, followed by a brief literature survey. section 3 explains the proposed approach in detail while section 4 presents the experimental details required to reproduce the results. results and analysis are shown in section 5. finally, conclusions are drawn in section 7. ## system overview we explored models using both single-task and multi-task learning approaches. since multi-task learning did not provide any significant improvement, our final model was trained using single-task learning. consequently, we trained models using one modality (either text or image) and both modalities for the sake of comparison. we also compared the performance of models with and without multimodal pre-training. ## experimental setup we used pytorch @xcite and hugging-face library @xcite for training and inference. all the models are trained on google colab. adamw @xcite optimizers with learning rates of 2e-5 and 5e-5 are used for training visualbert and mmbt models respectively. other text models such as roberta and ernie-2.0 also use adamw optimizer with a learning rate of 2e-5. the maximum length of the text is limited to 50. we chose a batch size of 32 for training all the models. all the hyperparameters are tuned on the validation set which is 20% of the training data. ## results and analysis we have tested transformer and non-transformerbased models with features extracted from text, images, and a combination of text and images as input. we experimented with models that employ text and multimodal pre-training like mmbt and visualbert respectively. the results of these experiments are summarized in our model (mmbt with tuned roberta) performs slightly better than visualbert without any multimodal pre-training. this is because visual-bert is pre-trained on microsoft's image annotation dataset coco and most real-world multimodal inputs are not as straightforward as a caption describing an image. for the model to adapt to the target domain, we need additional pre-training with the data of interest. in our case, instead of extra multimodal pre-training, we just trained the text model (roberta) for a few epochs using external data. this is computationally more efficient and flexible. furthermore, this approach can be easily applied to inputs with missing modalities. the performance of our model on the development set with a different number of image embeddings is shown in ## conclusion and future work we conducted experiments with models trained on features extracted from text, images, and both modalities combined for meme classification. to evaluate the text-based approach, we trained language models such as roberta, ernie-2.0, etc., and rnn-based models. subsequently, we con-cluded that the transformer-based models produced best results. we showed that further training the roberta model (tuned roberta) with taskrelated data improves the performance. this tuned roberta model combined with the features from resnet-152 without multimodal pretraining performs slightly better than visualbert. furthermore, an ensemble of these two models gave an additional performance boost. in the future, we plan to test other problem transformation approaches such as classifier chains and their variants for multi-label classification in order to accurately model label dependence and hierarchy. additionally, we plan to train visualbert from scratch on meme datasets to see if there is an improvement in its performance.