Poirot at S em E val-2022 Task 5: Leveraging Graph Network for Misogynistic Meme Detection (2022)

Abstract: In recent years, there has been an upsurge in a new form of entertainment medium called memes. These memes although seemingly innocuous have transcended the boundary of online harassment against women and created an unwanted bias against them. To help alleviate this problem, we propose an early fusion model for the prediction and identification of misogynistic memes and their type in this paper for which we participated in SemEval-2022 Task 5. The model receives as input meme image with its text transcription with a target vector. Given that a key challenge with this task is the combination of different modalities to predict misogyny, our model relies on pre-trained contextual representations from different state-of-the-art transformer-based language models and pre-trained image models to get an effective image representation. Our model achieved competitive results on both SubTask-A and SubTask-B with the other competingteams and significantly outperforms the baselines.

## introduction meme culture in today's virtual climate gives us a variety of insight into the pop culture, general ideology and linguistic conversational manner of the generation. to understand the internet culture, it becomes essential to study memes @xcite and the impact it has on people on the internet. some of the most popular communication tools on social media platforms are memes. memes are essentially images characterized by the content of a picture overlaid with text that was introduced by people with the main purpose of being interesting and ironic. women have a strong presence online, especially on image-based social media like twitter, snapchat, and instagram. 78% of women use social media several times a day, compared to 65% of men @xcite . while new opportunities are being opened up for women on-line, systematic inequality and discrimination are being replicated offline from these online spaces in the form of offensive content for women. most of them were created to make funny jokes, but soon people began to use them as a form of hatred for women, leading to sexist and offensive messages in the online environment, and as a consequence, the sexual stereotyping and gender inequality of the offline world where sexuality stereotypes and gender inequality have been strengthened. this insensitive and obscene type of meme has a profound effect on a person's mental health and can exhibit harmful effects on cognitive and emotional processes leading to mental illnesses as shown in @xcite . in this work, we present team poirot's solution to semeval -2022 task 5 competition as described in detail in @xcite . we focused our efforts on our primary approach of building a multi-modal module that uses features from both images and text. furthermore, in this paper, we provide ablation studies on different modalities, relative importance of the different modalities and some training parameters, and show how by changing the module parameters, the predictions on the misogynistic identification of memes aggravates or allays. ## system overview the solution system comprises of 2 separate systems for both the sub-tasks. the approach can be broadly divided into binary approach and multilabel approach. ## conclusion we have described the systems developed by as to solve the multimedia automatic misogyny identification challenge at semeval 2022. in our best performing submission for subtask-a, we framed the problem as a binary classification task and used two separate streams of information simultaneously to identify misogyny, while for our model for subtask-b, we tried to find the semantic relation between the type of misogyny and their relative importance to solve the problem for multi-label classification. by making use of powerful, state-ofthe-art, pre-trained models for text and images, our models were able to achieve a high f1 score for both the tasks. our best performing model ranked 3rd out of the 10 teams submissions on subtask-a and 22nd out of 30 team submissions on subtask-b. as part of future work, we aim to explore alternate approaches to model the multi-label dependencies using knowledge-graph and gat networks. also, there seems to be a problem of oversmoothing when increasing the depth of the graph classification module, which we aim to resolve using effective normalization layers between the graph layers.