IITR C ode B usters at S em E val-2022 Task 5: Misogyny Identification using Transformers (2022)

Abstract: This paper presents our submission to task 5 ( Multimedia Automatic Misogyny Identification) of the SemEval 2022 competition. The purpose of the task is to identify given memes as misogynistic or not and further label the type of misogyny involved. In this paper, we present our approach based on language processing tools. We embed meme texts using GloVe embedding and classify misogyny using BERT model. Our model obtains an F1-score of 66.24% and 63.5% in misogyny classification and misogyny labels, respectively.

## background the dataset used for this task consists of a set of 10000 memes images whose text has already been extracted. each meme has already been labeled as misogynous for subtask a and as shaming, stereotype, objectification ,and violence for subtask b. the dataset is perfectly balanced in terms of classification of misogyny, i.e., precisely 50% of the memes are misogynous and the rest 50% are not. of these 5000 misogynous memes -• 1274 or 25.48% are labelled as shaming category. the data is provided as zip file of memes in image format and a .csv file where the memes are labelled according to the following structure @xcite . ## related work memes are essentially language of the internet but such widespread use of memes had also made them a target for spreading hateful and offensive messages by fringe web communities @xcite . many online communities accept sexism and harassment in the name of humour in the form of memes and has resulted in increased attraction of attention from academics @xcite . thus, memes had emerged as a multi-modal expression of online hate. research in the field of multi-modal sentiment analysis has been mostly focused on video and text or speech and text @xcite . sentiment analysis of memes was conducted by @xcite but it was based on correlation of the meme and online discussion in the comments. recent study for meme classification was done by @xcite where hateful memes were classified based on the protected category they attacked which were race, sex, religion, nationality, disability. the study included usage of state-of-the-art visual and textual representations to produce respective embedding of the memes which were then concatenated to train a logistic regression classifier model. facebook recently launched the hateful memes challenge to accelerate development in this field @xcite . we follow a more humble approach of using glove to embed text in the memes but tried models more sophisticated than logistic regression classifier for the classification. the study conducted by @xcite reflects that better results emerge when both text and image are considered. but we do not consider the images for the challenge and consider this challenge an opportunity to learn about nlp first-hand. ## system overview bert foot_0 model is a state-of-the-art model developed by the ai team at google @xcite . traditional nlp models are unidirectional, i.e., they read the text from left-to-right or rightto-left. on the other hand, bert is bidirectional, understanding the correlation of a word with words on both sides by reading the entire sequence of the words at once. google showed that this scheme helps better understand the statement's sentiment, making this model the best fit for the task. roberta 2 model is built on top of the bert model. it has the same architecture but differs in terms of tokenizer and pre-training scheme, i.e., much larger mini-batches and learning rates are used. the architecture similarity with the bert model makes this model suitable for this task. baseline scores are made available by the organizers and are mentioned in ## experimental setup the scoring parameters, pre-processing, language, libraries and hyper-parameters used are mentioned in this section for ease in reproduction. ## results organizers are listed in from the results we can see that random forest model has high level of over-fitting as the score difference of training and validation is highest in this model. the least difference in training and validation score appears in logistic regression model but this is mostly due to high level of under-fitting as the least training score is obtained in this model. bert model provides the best results. the model obtains a high training score and score difference in training and validation is the least for this model. the trend maintains in subtask b as well with the model obtaining highest score. ranking phase -we submit our scores obtained from bert model for the leader board. the scores obtained are 66.24% and 63.5% 5 for subtask a and b respectively. the ranks obtained for subtask a and b are 47 and 34 respectively. ## conclusion the results show that bert model is the best among those we tried for the given problem statement. however, these results are obtained using purely nlp based tools and techniques, the image component of the memes is not considered. as discussed in the background section, there is a scope of improvement if we consider the images of the 5 the score of 64.76% mentioned in the table is obtained using the same model used in making the submission for evaluation phase. this score is obtained post-evaluation with test-labels being made public by the organizers. memes as well for the classification.