T eam O tter at S em E val-2022 Task 5: Detecting Misogynistic Content in Multimodal Memes (2022)

Abstract: We describe our system for the SemEval 2022 task on detecting misogynous content in memes. This is a pressing problem and we explore various methods ranging from traditional machine learning to deep learning models such as multimodal transformers. We propose a multimodal BERT architecture that uses information from both image and text. We further incorporate common world knowledge from pretrained CLIP and Urban dictionary. We also provide qualitative analysis to support out model. Our best performing model achieves an F1 score of 0.679 on Task A (Rank 5) and 0.680 on Task B (Rank 13) of the hidden test set. Our code is available at https://github.com/paridhimaheshwari2708/MAMI .

## introduction in this era of the internet, memes have become a new form of communication, which predominantly contain an image and a small caption. while their general purpose is to invoke humour or irony, they are also increasingly being used as a source of harmful, offensive and misogynistic content. detecting such content in an automated manner is an important problem to avoid the spread of hate. memes pose a unique multimodal challenge as their underlying implication is not just a simple combination of the image and text, but a subtle inference that comes naturally to humans. another complexity is that memes are highly contextual and the component image and text pieces might be completely uncorrelated. understanding this fusion of modalities is a challenging task for machines. our aim is to automatically identify misogynistic multimodal memes using machine learning. ## related work the task of identifying misogyny in memes is a relatively new area and is closely related to hate de- * these authors contributed equally tection. while there has been a lot of work on identifying hateful content in unimodal data @xcite , there is little work on multimodal hate detection. recently, facebook hateful memes challenge @xcite explored fusion of text and vision models along with advanced architectures like cross-modal bert @xcite . a major problem with these large pretrained models is the domain gap between memes and training data. some works try to solve this with better pretraining @xcite and disentangling hate from meme representations @xcite . in this work, we build on these technologies for our specific use-case of misogyny detection and incorporate common world knowledge from urban dictionary @xcite and clip @xcite to address the domain gap. ## conclusion and future work our work focused on the task of misogyny detection in multimodal memes. we demonstrated that using a combination of visual and textual, i.e, multimodal features outperforms the unimodal counterparts. in addition to simple baselines, we have also experimented with advanced dl architectures inspired from vqa and multimodal transformers. further, we have shown how incorporating common world knowledge from urban dictionary and pretrained clip can significantly help in identifying misogynistic content, along with qualitative evidence. finally, the proposed joint learning paradigm can exploit the synergy between the two tasks, instead of training models independently.