Characterizing the Entities in Harmful Memes: Who is the Hero, the Villain, the Victim? (2023)

Abstract: Memes can sway people’s opinions over social media as they combine visual and textual information in an easy-to-consume manner. Since memes instantly turn viral, it becomes crucial to infer their intent and potentially associated harmfulness to take timely measures as needed. A common problem associated with meme comprehension lies in detecting the entities referenced and characterizing the role of each of these entities. Here, we aim to understand whether the meme glorifies, vilifies, or victimizes each entity it refers to. To this end, we address the task of role identification of entities in harmful memes, i.e., detecting who is the ‘hero’, the ‘villain’, and the ‘victim’ in the meme, if any. We utilize HVVMemes – a memes dataset on US Politics and Covid-19 memes, released recently as part of the CONSTRAINT@ACL-2022 shared-task. It contains memes, entities referenced, and their associated roles: hero, villain, victim, and other. We further design VECTOR (Visual-semantic role dEteCToR), a robust multi-modal framework for the task, which integrates entity-based contextual information in the multi-modal representation and compare it to several standard unimodal (text-only or image-only) or multi-modal (image+text) models. Our experimental results show that our proposed model achieves an improvement of 4% over the best baseline and 1% over the best competing stand-alone submission from the shared-task. Besides divulging an extensive experimental setup with comparative analyses, we finally highlight the challenges encountered in addressing the complex task of semantic role labeling within memes.

## introduction due to their pervasive nature, online social media platforms have emerged as a conducive medium for information exchange. unfortunately, their democratic nature has fostered unabated dissemination of hate speech @xcite , misinformation @xcite , fake news @xcite , propaganda @xcite , and other harmful content. such information manifests itself in various ways, and more recently, in the form of memes. though typically intended to burlesque and lampoon world events, political outlook, or daily routine, an ostensibly innocuous meme can readily become a multi-modal cause of distress with a dexterous blend of images and texts. due to their viral nature and ability to circumvent censorship @xcite , social media instigators and hatemongers are increasingly using memes as a powerful medium for disseminating spiteful content. therefore, investigating the dark side of memes has risen as a pertinent research problem both in industry and academia @xcite . motivation. while there have been many studies that have analyzed memes through the lens of emotions @xcite , sarcasm @xcite , hate speech @xcite , misinformation @xcite , offensiveness @xcite , and harmfulness @xcite , there has been much less focus on analyzing semantic roles present within memes. understanding these roles is critical for comprehending memes and narrative framing of various social entities. caricaturing these entities with nefarious motives can lead to misinformation propagation, social calumny, and hatred towards minority communities. in addition to the dark portrayals, memes sometimes depict the sorrowful state of certain entities, illustrate their heroism, etc. consider the memes in fig. 1 . the meme in fig. 1 (a) portrays jill stein and the green party as heroes for their feminist views. fig. 1 (b ) draws a comparison between adolf hitler and donald trump, thus portraying the latter as a villain for his anti-immigrant views. fig. 1 (c ), on the other hand, depicts the plight of ola and uber drivers, who are out of work and victims of the lock-downs due to the covid-19 pandemic. thus, through depictions of heroism, villainy, and victimization, memes act as an alluring means to spread entityrelevant information and opinions. challenges. despite growing interest in analyzing memes, identifying the underlying connotations for the entities framed therein remains a challenging task @xcite . memes are obscure due to their highly cryptic semantics, and satirical content @xcite . moreover, categorizing the entities as a hero, a villain, or a victim requires real-world, contextual, temporal, spatial, and commonsense knowledge, which makes the task highly complex and subjective even for humans. therefore, off-the-shelf multi-modal models that stand out well on conventional visuallinguistic tasks often flounder for memes as they are presumably inept to comprehend and capture the veiled information and multi-modal nuances present in a meme @xcite . our contributions. in this work, we propose a powerful approach to tackle the novel task of identifying the roles (the hero, the villain, and the victim) of the entities present in memes. we model the problem as a role identification task and report the results for several unimodal and multi-modal baselines to benchmark the task (and the dataset) and assess its feasibility. we then proffer vector a vision and commonsense enriched version of deberta @xcite for the task at hand. as meme text often contains satirical content, which tends to contradict the meme image, it is necessary to consider mutual information from both modalities. also, meme content is often stated in a non-obvious way, necessitating commonsense and world knowledge. thus, our vector attempts to infuse the relevant visual and commonsense knowledge with linguistic representations. utilizing con-ceptnet @xcite , we generate an entityrelevant knowledge graph to represent commonsense knowledge relevant to the meme. moreover, we employ a distinct multi-modal information fusion strategy based on optimal transport. we appropriate optimal transport-based kernel embedding (otke) @xcite for cross-modal correspondence. this technique by @xcite marries the concepts of optimal transport theory with kernel techniques to provide robust and adaptable cross-modal adaptation. our qualitative analysis underscores the importance of vision and commonsense knowledge integration, as vector outperforms several competitive baselines. our contributions are summarized as follows: ## dataset we employ hvvmemes, a dataset released as part of constraint@acl-2022. it contains english memes on two topics: 3, 552 memes about covid-19 (c) and 3, 381 memes related to us politics (p). the dataset primarily captures connotative roles: hero, villain, and victim, for different entities referenced within memes. ## proposed approach this section outlines our proposed model vector (visual-semantic role detector) and its varied components. as previously noted, role detection for memetic entities is challenging and requires real-world, contextual, and commonsense knowledge. thus, we propose a neuro-symbolic approach that integrates commonsense-enriched modelling via graph (kg) structure into the language modelling-based architecture @xcite ). kg's can be considered as discrete symbolic knowledge, which we leverage along with multimodal neural modelling. as shown in fig. 2 , vector houses two primary sub-modules. the vision module leverages cross-modal interaction between the visual-linguistic signals to grasp optimal contextual information. the commonsense module integrates commonsense cues through an entity-based knowledge graph. lastly, the information fusion module coalesces the information obtained via attentionbased fusion. in the following subsections, we go over the specifics of each module. text module: we use deberta @xcite as our backbone model as it gives the best results amongst the text-only baselines (see vision module: meme contents often contain contradicting text and image pairs. therefore, it is required to incorporate information from these modalities to understand memes. instead of using the traditional cross-modal attention to facilitate interaction between the two modalities, we utilize an optimal transport-based kernel interaction @xcite . to begin with, we use a vision transformer @xcite for generating the image representations e @xmath1. the text and the image representations undergo dimensionality reduction by non-linear transformation, followed by a self-attention layer @xcite as given by equations 1. this spawns vectors z ′ m , e ′ @xmath2. we concatenate these two vectors and pass them to the optimal transportbased kernel embedding (otke) layer to bring about cross-modal interaction. it transforms the feature vectors to a reproducing kernel hilbert space (rkhs) @xcite followed by a weighted pooling scheme using weights determined by the transport plan between the set and a trainable reference. such a fusion technique provides a theoretically grounded adaptive vector for the task. this yields vector z @xmath3, given by equation 2 below: commonsense module: due to their cryptic nature, identifying the intent of a meme is challenging. their satirical and non-obvious way of conveying a message often requires commonsense comprehension. thus, in our commonsense module, we generate a commonsense knowledge graph based on the entities in the meme. similarly to @xcite , we extract all the nouns and noun phrases in the meme ocr and the meme caption. we extract commonsense relation pairs having @xmath4. noun chunks, commonsense entities, and the meme entity in question form the nodes of the commonsense graph. each noun chunk from the ocr text is connected. the same applies to the noun chunks in the meme caption. a special aggregator token connects ocr and caption-based nodes. thus, for each entity e @xmath5, e i2 , . . . , e in }, we have a commonsense graph g @xmath6, e ij ), where v ij and e ij are the nodes and the edges of the graph. the graph having edges between various entities, noun chunks, and nouns, being undirected, represents a generic "connectivity". therefore, an edge between nodes a and b indicates that they have some association. by doing this, we wanted to capture the commonsense-based 'proximity' of different entities/concepts within the vectorized space for common sense concepts. v ij represents a set node (or vertices) constituting a commonsense graph corresponding to each entity. for each node in the commonsense graph, we generate an embedding of size d g using the last layer representations from deberta. in order to facilitate the interaction between the nodes, the commonsense graph goes through two rounds of graph convolutions @xcite followed by a max-pooling operation to spawn an aggregated graph embedding e @xmath7. similarly to the vision module, the textual representations z and the commonsense graph representation e g undergo non-linear transformation and dimensionality reduction followed by self-attention @xcite . this generates contextual vectors z ′ ## baselines unimodal systems: we use a variety of textbased and image-based models as our baseline systems. starting with the text baselines, we use bert @xcite , and variants thereof such as @xcite , roberta @xcite , xlnet @xcite , and deberta @xcite . for the image-based baselines, we start with a representation based on resnet-50 @xcite , followed by vision transformer (vit) @xcite and swin @xcite , which is hierarchical version of vit that uses shifted windows. we also include the recently proposed convnext @xcite and beit @xcite , which uses selfsupervised pre-training of vision transformers. multi-modal systems: we use various variants of multi-modal pre-trained systems from the mmf framework. mmf transformer is a library transformer model that uses visual and language tokens with self-attention. mmbt: multi-modal bitransformer @xcite captures the two modalities' intra-modal and inter-modal dynamics. vilbert is a vision and language bert @xcite , a strong model with the task-agnostic joint representation of images and text. vilbert cc is pre-trained on conceptual captions @xcite based pretext task. visual bert @xcite , also pre-trained using ms coco @xcite , implicitly aligns the input text and regions in the input image using self-attention. clip @xcite leverages image-text contrastive pretraining. ## experiments experimental details we present benchmarking results in for the villain category, bert can be observed (c.f. interestingly, none of the text-only models, except for deberta large, could beat the f1 score of 0.395 for deberta-base. the lower performance is primarily due to inadequate categorical representation within the dataset. moreover, the inherent complexity in distinguishing villains from victims confuses the model. the modelling efficacy solicited for such a scenario is suggested by over 0.10 absolute point enhancement by debertalarge, which has 4x more backbone parameters 2154 other category, having the majority representation with over 6k unique references within memes, projects the highest f1 scores with an average of 0.879. finally, the overall macro-f1 scores reflect the category-wise trend observed, with deberta large leading with an f1 score of 0.540. debertabase follows it with 0.509 and the rest with an average f1 score of 0.444. for intuitive reasons, the visual modality, not indicative of complex role semantics, yields poor performance compared to text-only models. as a result, none of the image-only models (convnext, vitbeit, and swin) are observed to make any headway in correctly detecting a hero reference in memes (c.f. ## interpretability the predictive capacity of vector can be assessed by comparing its interpretability with that of visu-albert, the best-performing baseline. we investigate a reasonably complex example depicted in fig 4 , for which vector accurately predicts all the entity roles, but visualbert wrongly predicts all of them, as villains. attention map for vector reveals that it primarily attends to the face of 'barack obama', while leveraging other contextual cues, likely via the common-sense and visual description-based signals, leading to a correct assignment of roles. on the other hand, visualbert, having a faster r-cnn-based image encoder, primarily pretrained for the common object-detection task, fails to uniquely attend to different semantically referred entities within the meme image and additionally adds noisy features related to objects like a tie, without any additional contextual knowledge. this validates vector's superior discriminatory capacity and interpretability over visualbert. ## generalizability since, the test set consists of all unique memes, there is a higher chance that there are many entities as part of it that are not seen during the training stage. ## conclusion this paper addresses a recently proposed task of identifying the roles of entities in harmful memes and discusses its challenges. we further presented numerous unimodal and multi-modal baselines to benchmark hvvmemes. moreover, we proffer vector, a contextual knowledge-enriched multimodal framework that bolsters the multi-modal representations with entity-based external knowledge using a cross-modal attention scheme. vector shows noteworthy improvements over the baselines, thus justifying contextual knowledge inclusion. as for future investigations, we plan to conceive a more symbolic system with graph-based entity linking, commonsense knowledge, and visual concepts.