JRLV at S em E val-2022 Task 5: The Importance of Visual Elements for Misogyny Identification in Memes (2022)

Abstract: Gender discrimination is a serious and widespread problem on social media and online in general. Besides offensive messages, memes are one of the main means of dissemination for such content. With these premises, the MAMI task was proposed at the SemEval-2022, which consists of identifying memes with misogynous characteristics. In this work, we propose a solution to this problem based on Mask R-CNN and VisualBERT that leverages the multimodal nature of the task. Our study focuses on observing how the two sources of data in memes (text and image) and their possible combinations impact performances. Our best result slightly exceeds the higher baseline, but the experiments allowed us to draw important considerations regarding the importance of correctly exploiting the visual information and the relevance of the elements present in the memes images.

## introduction even though many advances and initiatives have been carried on in the last decades, misogyny and gender discrimination still represent a serious social problem. comments and posts with hate messages or bad jokes having the female gender as the target are published every day on social networks. memes are the most popular means of communication on the internet. a meme basically is an image with overlayed text. the combination of image and text should transmit in a clear and direct way a message, which is often funny or ironic. however, many users use them to spread hate or discriminatory messages against certain categories of people. considering the number of contents that are published online every moment, social network administrators require automatic tools to identify and remove this type of memes. such tools can be an effective way to fight and stop sexist and misogynous manifestations online. for the semeval-2022 it has been proposed the multimedia automatic misogyny identification (mami) task @xcite , which consists of identifying misogynous memes. in this work, we present our proposed solution for subtask a of mami, which is a binary classification problem where a meme must be categorized as misogynous or non-misogynous. our solution exploits the multimodal nature of the available data (a meme is represented by a pair image -overlayed text) by using a visualbert model together with a mask r-cnn. our solution classified 35 th out of 69 participants, getting an f1 score 2% higher than the best baseline. beyond metric values, our work highlights the different importance of various visual elements and how their usage can affect the performances. ## task overview the purpose of mami is identifying misogyny contents and messages within memes through the use of visual and textual information. organizers set up this task as a competition, providing an annotated dataset and proposing two different sub tasks. ## related works hate speech detection in text data has been deeply studied in the last few years. state-of-the-art approaches usually apply transformers-based methods achieving impressive results. @xcite for example proposed different hybrid architectures to fine-tune a bert model @xcite for detecting offensive tweets. however, identifying hate content in multimodal data requires correlating visual and textual information and introduces an additional challenge. multimodal transformers, such as visualbert @xcite and vilbert @xcite , are currently the state-of-the-art models for these types of problems. in 2020 facebook ai organized the hateful memes challenge @xcite . the competition was similar to mami and consisted of categorizing a meme as hateful or non-hateful. the dataset contained memes with various types of hate messages (e.g. targeting an ethnicity, a religion, or the sexual orientation of people). winning solutions all used ensembles of multimodal transformers-based networks. @xcite ) won the competition using a complex and task-specific pipeline to extract additional information from the data with which to fine-tune multimodal transformers networks. @xcite and @xcite , who ranked respectively second and third, proposed simpler ensemble methods based on visualbert and vilbert-derived architectures. many other works regarding meme analysis for hate speech detection can be found in the literature. each has its own peculiarities, often related to information extraction techniques, especially from the visual content. among them, the most used are text removal, object detection, image captioning, and entity recognition, such as in @xcite , @xcite and @xcite . first of all, we exploit single modalities separately, through state-of-the-art deep models, to investigate their relevance and the quantity of information they carry on. subsequently, we combine image and text embeddings to estimate the performance of a multimodal approach. the strategies mentioned above can be considered as three baseline models in which the information from each modality is retrieved without taking into account the other one and then used both alone or together. afterward, we adopt early fusion methods to create a more informative embedding of a whole meme, extracting information jointly from both modalities with the same model. this allows to directly obtain a final representation depending on both text and image, thus removing the need for a fusion point between unimodal embeddings. ## conclusions and discussion in this work we demonstrate how leveraging the multimodal nature of data allows to achieve a significant boost in performance when facing tasks involving memes. moreover, we confirm that an early fusion point between modalities implemented using visualbert, the current state-of-the-art architecture for vision-and-language tasks, can lead to satisfactory results to the mami task. the reliability of this model allows us to focus on other crucial aspects of the problem, such as the data to use as input to the network. while unimodal transformers successfully identify offensive content in textual data, state-of-the-art computer vision models obtain lower results on this task when analyzing visual information only. catching the message of a meme requires understanding the context created by the text and the various elements depicted in the image. a deep learning model will struggle to form this context if the visual data are analyzed as a whole. therefore, we have to first extract the relevant parts from the image and use their features separately as input to our multimodal network. by doing this, visualbert is capable of correlating text and visual scene to create the aforementioned context. our experiments showed that the most relevant elements in the images are people, likely the target or the offender mentioned in the text. despite these discoveries, the extraction of information from the pictures remains the main obstacle of this task. as a future work, we plan to thoroughly investigate the content analysis of images. in particular we would like to enrich the preprocessing stage, performing captioning and entity detection of the image, in order to transpose some visual information in textual format. (a) misogynous meme. (b) not misogynous meme.