Findings of the CONSTRAINT 2022 Shared Task on Detecting the Hero, the Villain, and the Victim in Memes (2022)

Abstract: We present the findings of the shared task at the CONSTRAINT 2022 Workshop: Hero, Villain, and Victim: Dissecting harmful memes for Semantic role labeling of entities. The task aims to delve deeper into the domain of meme comprehension by deciphering the connotations behind the entities present in a meme. In more nuanced terms, the shared task focuses on determining the victimizing, glorifying, and vilifying intentions embedded in meme entities to explicate their connotations. To this end, we curate HVVMemes, a novel meme dataset of about 7000 memes spanning the domains of COVID-19 and US Politics, each containing entities and their associated roles: hero, villain, victim, or none. The shared task attracted 105 participants, but eventually only 6 submissions were made. Most of the successful submissions relied on fine-tuning pre-trained language and multimodal models along with ensembles. The best submission achieved an F1-score of 58.67.

## introduction the unwarranted spread of misinformation @xcite , propaganda @xcite , fake news @xcite , covid-19 infodemic @xcite , hate speech @xcite , and other harmful content @xcite has plagued social media. lately, memes have emerged as a powerful multimodal means to disseminate malicious content due to their ability to circumvent censorship norms @xcite and to their fast-spreading nature. with an aptly crafted combination of images and text, a seemingly naïve meme can easily become a source of harmful information diffusion. as a result, exploring the noxious side of memes has become a pressing research topic; see also recent surveys on harmful memes @xcite and on multimodal disinformation detection @xcite . while meme analysis has been studied in a variety of contexts, such as hate speech @xcite harmfulness @xcite , emotions @xcite , misinformation @xcite , sarcasm @xcite , offensiveness @xcite , and propaganda @xcite , limited forays have been made on comprehending the role of the entities that make up a meme. this is our main focus here: on identifying the hero, the villain, and the victim entities present in a meme. given a meme and a list of the entities it involves, the task is to identify which entity plays what role. such categorization of the entities in the meme can help understand the entity-specific connotation and their nature, attitudes, decisions, and demeanour. for instance, when the meme creators intend to spread misinformation and hatred towards minority communities or to defame certain individuals, politicians, or organizations, they would depict the target entities as villains. similarly, when the intent is to shed light on the deplorable state of certain entities or to glorify them, these entities would be portrayed as victims or as heroes, respectively. fig. 1 depicts apt examples for hero, villain, and victim categorization of the entities in a meme. the meme in fig. 1a draws a comparison between abraham lincoln, john f. kennedy, barack obama, and donald trump, where the former three are portrayed as heroes, while donald trump is shown in negative light, as a villain. similarly, fig. 1b mocks jill stein and the green party as villains for allegedly getting bribed by the rich. fig. 1c on the other hand, frames the republican party as a villain, for their inconsiderate views on the poor, the minorities, and women, thus making them the victims. in conclusion, through depictions of heroism, villainy, and victimization, memes act as an appealing means to propagate certain views about the targeted entities. (a) [green party, jill stein] while some previous meme studies have sought to identify harmfulness and the entities @xcite or the categories that are being targeted, e.g., a person, a group, an organization, or society @xcite , none of them has scrutinized the entity's connotation. our shared task aims to bridge this gap. we release hvvmemes, a meme dataset with about 7,000 memes on covid-19 and us politics, where each meme is annotated with a list of entities, each labeled with its role: hero, villain, victim, or other. the shared task attracted 105 teams, and nine of them made official submissions. most teams finetuned pre-trained language and multimodal models or used ensembles, with the best system achieving an f1-score of 58.67. we discuss the submissions and their approaches in more detail in section 5. despite the growing body of research on meme analysis, understanding the connotation underlying the individual entities in the meme remains a challenging endeavour. their camouflaged semantics, satirical outlook, and cryptic nature make their analysis a daunting task @xcite . moreover, categorizing the entities as heroes, villains, or victims requires real-world and commonsense knowledge, which often are not present in popular pre-trained language models. thus, it should not be surprising that, as the shared task's results show, off-the-shelf multimodal models, as well as various ensembles thereof, struggle with this task @xcite . this highlights that the current stateof-the-art visual-linguistic models are unable to grasp the veiled information present in the memes. thus, we hope that the dataset and task will foster further research in this interesting direction. more details about the shared task is available at http://constraint-lcs2.github.io/ 2 related work studies on online targeting. previous work studied affective content in the context of harmful discourse in social media @xcite @xcite . sarcastic content was detected by leveraging data sparseness @xcite towards studying aspect-based sentiment analysis. @xcite established enhancements in target detection by examining generic concept extraction for hate speech detection. targeted protected categories were characterised by harmful online engagements whilst addressing societal bias along with explainability @xcite . for affective target characterisation, sequence modeling was explored in a hierarchical formulation of stacked bigrus @xcite as well as in lowresource scenarios @xcite . most approaches did not consider the variability in target referencing and the associated affective spectrum @xcite . finally, @xcite discussed hero/villain/victim analysis of news text; unlike their work, here we focus on multi-modality and memes. studies on detecting harmful memes. the constant transitioning of harmful memes from unfiltered and largely anonymous communities and platforms such as 4chan, reddit, and gab to more mainstream social media has made the entire social media ecosystem both sensitive and vulnerable to extremism @xcite . research on offense @xcite , hate speech @xcite , and online harm detection @xcite has found the availability of large datasets and the use of multi-modal frameworks crucial for these tasks. additional contextual cues involving commonsense knowledge @xcite , semantic entities, cues about the protected categories @xcite , along with other meta information, have also been explored for characterising various aspects of the online harm conveyed by memes. most such tasks address affect detection at various levels of granularity, sometimes organised in a taxonomy. still, none of these tasks has focused on explicitly modeling the complex narrative framework of the memetic discourse surrounding the specific entities referred to in the meme. with this in mind, here we attempt to alleviate a few associated challenges by exploring the feasibility of entity-specific visual-semantic role labelling for memes. other related shared tasks. several shared tasks have targeted the broad field of harmful social media content. some tasks investigated the characterisation of offensive language, hate speech, profanity, and associated fine-grained attributes such as implicit and explicit implications in binary, multiclass, multi-label, and hierarchical settings @xcite @xcite . their coverage has been fairly comprehensive in terms of the languages covered including arabic, danish, greek, english, turkish, and dravidian languages like tamil, malayalam, kannada as well as german and english/indo-aryan code-mixing @xcite @xcite @xcite . they also address harmful content dissemination, targeting various protected categories such as religious affiliation, national origin, sex, etc. @xcite . other efforts have targeted misinformation, propaganda, and persuasiveness detection @xcite @xcite , where the goal is to detect verifiable claims, their veracity, span, and check-worthiness. persuasive technique detection has also been explored for images besides text-based content, e.g., @xcite introduced the task of propaganda in memes. some tasks have attempted to address affect concerning various targets. @xcite focused on stance prediction for given targets, i.e., whether the comment is in favour or against the target, both in supervised and in unsupervised scenarios. @xcite modeled sarcastic targeting of specific entities. @xcite focused on sentiment analysis in twitter. in contrast, here we focus not only on the polarity of the target entity, but also on understanding complex connotations such as glorification, vilification, and victimisation in memes. this is both challenging and important, as memetic discourse has taken over a sizable portion of online engagement and as it requires specialised moderation given its multimodal nature. ## dataset curation towards curating a dataset that would enable the identification of hero, villain, and victim as roles in memes, we leveraged and reannotated the harmeme dataset released in @xcite , and we call this new dataset hvvmemes. harmeme includes 3,544 memes about covid-19 and 3,552 memes about us politics, which are annotated for harmfulness as well as for target type, in case the meme is harmful, with four categories for the latter: individual, organisation, community, and society. ## participation and results the total of 105 teams registered for the competition, and nine of them made submissions to the leaderboard, making a total of 71 attempts to improve their scores. the teams tried a variety of approaches, and below we discuss the approaches by the six teams who also submitted a system description paper with information about their runs. it is remarkable that they achieved this result using just the text input. they formulated the problem as a multiple choice question answering task (mcqa), and they used an ensemble of three modules: twitter-xlmroberta + covid-bert @xcite ) + bert-tweet (nguyen et al., 2020). they further added a sentiment module trained using roberta, with the final classification layer comprising support vector machine (svm). a major drawback of this approach is that they ignored the image as an input altogether. they built an ensemble of visualbert (li et al., 2019) + uniter (chen et al., 2020) + oscar (li et al., 2020) + ernie-vil (yu et al., 2021), combined using an svm. to handle the disproportionately large number of other examples, they introduced lossreweighting. the lack of sufficient external knowledge and position information about the ocr text with the image restricted their system performance. their source code is available at https://github.com/ zjl123001/dd-tig-constraint system bert r-bert d-bert clip eb7 ofa vilt vit vb u o e-v svm xgb bf vader w-p shiroe c1pher zhouziming smontariol iiitdwd rabindra.nath the evaluation results for the above systems are shown in ## conclusion understanding and interpreting the connotations behind the entities in a meme is a difficult problem, which we pioneered in this shared task. given a meme and a list of entities, the task asks to detect the role of each entity as a hero, a villain, a victim, or other. we curated hvvmemes, a largescale meme dataset of 7,000 memes spanning the domains of covid-19 and us politics, annotated with the entities they refer to as well as with their role. the shared task attracted 105 registered participants, out of which nine made official submissions, and six submitted papers describing their systems. we hope that our dataset and task setup will enable further research towards understanding how entities are portrayed in memes.