R 2 D 2 at S em E val-2022 Task 5: Attention is only as good as its Values! A multimodal system for identifying misogynist memes (2022)

Abstract: This paper describes the multimodal deep learning system proposed for SemEval 2022 Task 5: MAMI - Multimedia Automatic Misogyny Identification. We participated in both Subtasks, i.e. Subtask A: Misogynous meme identification, and Subtask B: Identifying type of misogyny among potential overlapping categories (stereotype, shaming, objectification, violence). The proposed architecture uses pre-trained models as feature extractors for text and images. We use these features to learn multimodal representation using methods like concatenation and scaled dot product attention. Classification layers are used on fused features as per the subtask definition. We also performed experiments using unimodal models for setting up comparative baselines. Our best performing system achieved an F1 score of 0.757 and was ranked 3^{rd} in Subtask A. On Subtask B, our system performed well with an F1 score of 0.690 and was ranked 10^{th} on the leaderboard. We further show extensive experiments using combinations of different pre-trained models which will be helpful as baselines for future work.

## introduction internet and social media sites have played an integral role in bringing people together by providing a simple yet effective way of communication. over recent times, internet memes have become a popular choice for sharing sentiment on the internet. a meme is an approach, concept, idea, or style that spreads through social media within a society, often to express a trend, topic, or significance represented by it @xcite . memes shared on the internet are often harmless and used to express humour; however, recent trends have increased their usage to spread hate or cause social unrest @xcite . hate speech and, in particular, hate against women has seen an exponential rise in social media platforms @xcite . misogyny, a subset of hate-speech @xcite , is defined as hate or prejudice against women, which can be manifested in numerous ways, including social exclusion, sex discrimination, hostility, patriarchy, male privilege, belittling of women, disenfranchisement of women, violence against women, and sexual objectification @xcite . women have a strong presence online, particularly on instagram and twitter. women use social media multiple times a day compared to men @xcite . this makes it extremely important to identify and remove such content to make the internet safer for women. efforts have been made to identify misogynous textual content on social media @xcite @xcite , however, no efforts have been made to identify the misogynous content spanning multiple modalities like memes. memes are uniquely multimodal and convey information using images and text. the multimodal nature of memes allows them to combine harmless texts/images into misogynist memes when used together. this poses an exciting challenge as memes require joint language and visual understanding to infer their true meaning. semeval 2022 task 5: mami @xcite draws attention to the problem of identifying misogynous memes and further identifying the type of misogyny. the task provides a dataset of misogynist memes and its type. both images and the corresponding text was available as a source of information; the text content of the provided dataset was in english. our proposed system for both subtasks uses the late fusion of visual and textual features obtained from pre-trained models. we use separate pre-trained models for each modality, i.e. text and image, and fuse the features to learn multimodal representation for memes. we experimented with simple concatenation of image and text features; and scaled dot product attention @xcite , followed by a convolution layer to learn multimodal features. once multimodal features are learnt, we stack classification layers on them as per the subtask requirements. we also performed extensive experiments using a single modality (text/image) to set up comparative baselines. we used vit (vision transformer) for image feature extraction. we experimented with various plms for textual feature extraction like bidirectional encoder representations from transformers(bert), robustly optimized bert pretraining approach(roberta), mpnet, and decoding-enhanced bert with disentangled attention(deberta). the results of unimodal experiments showed that text-only models had a superior performance than image-only models. we experimented with feature concatenation and scaler dot product attention for multimodal models. feature concatenation led to only minor performance gains. in case of scaler dot product attention, choice of query(q) @xcite turned out to be an essential factor. our experiments showed that image features as query performed significantly well for all models and outperformed all unimodal and concatenation baselines. our best performing model was a voting ensemble of attention based multimodal models and achieved an f1 score of 0.757 with a 3 rd rank on the official leaderboard for subtask a. for subtask b we used bert and vit based attention model, which performed well, attaining an f1 score of 0.690 and 10 th rank on the leaderboard. our code available at github foot_0 for method replicability. ## background identifying misogynous content is critical to making the internet accessible and safe for women. in recent times there has been an exponential rise in hateful content and, in particular, the phenomenon of hate against women on social media @xcite . there have been previous attempts to identify hate/toxic content on social media platforms ( @xcite @xcite ) @xcite but none deal specifically with identifying the hate against women. the first benchmark dataset to identify misogynous content was proposed in @xcite . the task and the papers of ami@evalita 2018 @xcite and ami@ibereval2018 @xcite highlight the difficulties and barriers involved in automatically identifying misogynist content on social media. workshop on trolling, aggression and cyberbullying (trac) shared task @xcite contained a subtask to identify gender-based identification of hateful content. there has been a rise in multimodal content over the internet in the form of memes. most efforts to identify toxic and misogynous content consider only the textual content. due to the rapid increase of multimodal content, efforts have been made to analyse it. memotion analysis @xcite aimed to perform sentiment analysis on internet memes. it involved identifying offensive sentiment as part of its subtasks. @xcite used a feature fusion model using lstms, grus with attention and attained the best results in identifying offensive memes. hateful memes challenge @xcite aimed to study and identify the hateful nature of internet memes. analysing memes is an intrinsically difficult task as it requires multimodal reasoning capable of understanding textual and visual features. the common strategy used in analysing memes involved learning features for each modality and then fusing the features to represent joint features. work done in (pranesh and shekhar, 2020) uses the simple concatenation of text and visual features for meme sentiment analysis. @xcite used early fusion models like lxmert @xcite , uniter @xcite uses transformer @xcite architecture and neural attention to learn joint representation. social media provides a platform for numerous people to express and share their thoughts. misogyny which can be simplified as hate or prejudice against women, is on the rise in social media platforms. hence, it is crucial to identify and remove such content from the internet. attempts have been made to identify hateful/toxic and misogynist content on the internet, but none focuses on multimodal content like memes. semeval 2022 task 5: mami -multimedia automatic misogyny identification @xcite aims to study the misogynist nature of memes and is divided into two subtasks which we define as: subtask a: given a labelled dataset d of internet memes and their text, the objective of the task is to learn a classification function that can predict if a meme is misogynous or not. subtask b: given a labelled dataset d of internet memes and their text, the objective of the task is to learn a multilabel classification function that can predict the type of misogyny m for a given misogynous meme, where @xmath0, shaming, objectification and violence}. ## conclusion this paper describes our approach for semeval 2022 task 5: mami -multimedia automatic misogyny identification. we propose a dot product attention-based mechanism for learning multimodal representation from independent text/image features. our work also describes a comprehensive set of experiments using unimodal/multimodal models using different pre-trained models. our system performed well, attaining 3 rd rank on subtask a and 10 th in subtask b. our experiments highlight the non-commutative nature of dot product attention, with the choice of the query being a critical design decision. our results showed that textual features dominate over image features in multimodal understanding. in the future, we would like to explore more on how attentions learn multimodal features and further compare the role of individual modalities in multimodal tasks.