S em E val-2022 Task 5: Multimedia Automatic Misogyny Identification (2022)

Abstract: The paper describes the SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification (MAMI),which explores the detection of misogynous memes on the web by taking advantage of available texts and images. The task has been organised in two related sub-tasks: the first one is focused on recognising whether a meme is misogynous or not (Sub-task A), while the second one is devoted to recognising types of misogyny (Sub-task B). MAMI has been one of the most popular tasks at SemEval-2022 with more than 400 participants, 65 teams involved in Sub-task A and 41 in Sub-task B from 13 countries. The MAMI challenge received 4214 submitted runs (of which 166 uploaded on the leader-board), denoting an enthusiastic participation for the proposed problem. The collection and annotation is described for the task dataset. The paper provides an overview of the systems proposed for the challenge, reports the results achieved in both sub-tasks and outlines a description of the main errors for a comprehension of the systems capabilities and for detailing future research perspectives.

## introduction women have a strong presence online, particularly in image-based social media such as twitter and instagram: 78% of women use social media multiple times per day compared to 65% of men @xcite . however, while new opportunities for women have been opened on the web, systematic inequality and discrimination offline is replicated in online spaces in the form of offensive contents against them @xcite @xcite @xcite . a popular commu-nication tool in social media platforms are image macros popularly connoted as "memes" @xcite . an internet meme is usually an image communicating pictorial content with an overlaid text that is added a posteriori by the meme author, with the main goal of being funny and/or ironic @xcite . although many memes are created with humorous intent, others have political or activist ambitions. few familiar with the format would be surprised to learn that memes can be used to express hate against women, via sexist and aggressive messages in online environments @xcite that subsequently amplify the sexual stereotyping and gender inequality of the offline world @xcite . in order to counter this phenomenon, the multimedia automatic misogyny identification (mami) shared task has been organised at semeval-2022 @xcite . the proposed challenge consists of the identification of misogynous memes, taking advantage of both text and images available as sources of information. the task is organised around two main sub-tasks: -sub-task a: a basic task of misogynous meme identification, where a meme should be categorised either as misogynous or not misogynous; -sub-task b: an advanced task, where the type of misogyny should be recognised among potential overlapping categories such as stereotype, shaming, objectification and violence. some other tasks related to this topic, but that did not consider the same data and a multimodal approach have been previously organised in the same area of interest, i.e. ami@evalita @xcite , ami@ibereval @xcite , hateval @xcite and offenseeval (zampieri et al., 2020). however, the proposed mami challenge is a step forward the previous ones for two main reasons: (1) it is focused on multi-modality and (2) the type of misogynous contents are expressed in a completely different form, i.e. in the former challenge the presence of hateful contents was explicit within the text, while here it is often implicit. ## dataset and annotation process candidate memes have been collected by focusing on the following main types of misogyny: examples of the above mentioned types of misogynous memes are presented in figure 1 . the procedure for collecting relevant memes for this shared task consisted of: (1) searching the most popular social media platforms, such as twitter and reddit; and (2) downloading samples from websites dedicated to meme creation and sharing, such as 9gag, knowyourmeme and imgur, by site scraping and manual download. in both cases, in order to collect a proper number of misogynous memes, 4 main activities have been performed: (1) searching for threads dedicated to memes with women as the subject; (2) searching for threads or conversations dedicated to or written by persons who identify as anti-women or antifeminist (such as the mgtow website and the related threads on reddit); (3) exploring discussions in recent events involving famous women (such as michelle obama); (4) searching by keywords and/or hashtags such as #girl, #girlfriend, #women, #feminist. the final collection is composed of 15k memes that have been labelled by human annotators (duplicates have been previously removed). among the labelled memes we obtained an adequate number of misogynous and non misogynous memes. the final benchmark dataset released for the mami challenge is composed of 10k memes for training and 1k for testing (balanced between classes). the dataset has been labelled using crowd-sourcing platforms according to the following primary ques stereotype, objectification, violence)? in the last case, i.e. related to the misogyny category, multiple overlapping labels have been considered. the memes were shown one at a time to avoid bias introduced by the annotators seeing multiple memes simultaneously. memes were annotated by 3 observers and the final label was given according to the majority of the labels (2/3). the text of the memes have been transcribed using google cloud vision foot_1 . we report an example of a meme that has been provided to the participants as training example, which is composed of raw image (figure 2 ) and the corresponding labels available through a csv file ( we estimated the inter-annotator agreement using the fleiss-κ coefficient @xcite . in particular, we used the traditional fleiss-κ measure for estimating the agreement related to the misogynous vs not misogynous annotation necessary for sub-task a, while we adopted the fleiss-κ with the masi (jaccard) index @xcite to calculate the agreement between annotators on multiple (overlapping) annotations necessary for sub-task b. regarding the agreement on the misogynous vs not misogynous annotations, we estimated a coefficient equal to 0.5767, while for the type of misogyny labelling we derived a coefficient equal to 0.3373. we report in we also used two multi-label models introduced for sub-task b and detailed in the following paragraph. sub-task b. systems have been evaluated using weighted-average f1-measure. in particular, the f1-measure has been computed for each label and then the average has been weighted by the number of true instances for each label. for sub-task b, the baselines are grounded on: -baseline flat multi-label: a multi-label model, based on the concatenation of deep image and text representations for predicting simultaneously if a meme is misogynous and the corresponding type; -baseline hierarchical multi-label: a hierarchical multi-label model, based on text representations for predicting whether a meme is misogynous or not and, if misogynous, the corresponding type. ## participant systems and results mami has been one of the most popular tasks in semeval-2022, with 65 teams that joined sub-task a and 41 teams that participated in sub-task b. we received a total of 4,214 submissions, of which 166 submitted to the leader-board. among the teams joining the mami challenge, 41 groups have provided the details about their participation (team name, number of team members, country, and description of their system). in appendix b ( as a general overview of the results, we report in when considering the max values, it emerges that sub-task b seems to be more difficult than sub-task a, while the median values indicates that for the 50% of the systems both tasks are equally challenging. in regards to the models adopted by the participants, it has been observed that the majority of the teams exploited pre-trained models, distinguished in text-based, where the most used ones are based on bert @xcite such as roberta @xcite , and image-based models, where the most adopted ones are based on visualbert @xcite . among these systems, considered by 90% of the teams, half of them adopted an ensemble strategy to make the final prediction. the remaining ones adopted either traditional neural networks (30%) or multi-task (20%) approaches to classify the memes. few teams exploited models, such as clip @xcite and vilbert @xcite , to jointly learn the characteristics of misogynous and not misogynous memes, and the related misogyny categories. ## error analysis in order to gain deeper insight into the prediction capabilities of the systems and delineate the open issues about the recognition and classification of misogynous memes, we conducted a detailed error analysis on both sub-tasks, considering all participating teams. the error distributions and the types of the most common errors in regards to the labels to be predicted are detailed in the following subsections. we considered memes misclassified by at least 25%, 50% and 75% of the teams, distinguishing false positive (fp) and false negative (fn), according to the labels available in each sub-task. for the memes misclassified by at least 75% of the teams, we reported the most frequent types of errors by analysing the visual and textual content of the memes. ## conclusions the high number of participating teams at the mami challenge at semeval-2022 confirms the growing interest of the research community not only in detecting abusive language but also pictorial content as sources of information. overall, results and error analysis confirm that the detection of misogynous memes is challenging, with many open issues that need to be addressed. first of all, the fact that the most predominant error in misogyny recognition relates to the misclassification of not misogynous memes as misogynous ones suggests that some potential issues could be related to biased models. the research community is therefore encouraged to pay attention not only to accuracy metrics, but also to ensure models are unbiased before applying them in a real context. another open issue relates to the capability of the systems to model the dynamics of the memes. every day different memes, with different images and different text are generated on the web and shared online. ziming zhou, han zhao, jingjing dong, ning ding, xiaolong liu, and kangli zhang. 2022. dd-tig at semeval-2022 task 5: investigating the relationships between multimodal and unimodal information in misogynous memes detection and classification. in the 16th international workshop on semantic evaluation.