DISARM : Detecting the Victims Targeted by Harmful Memes (2022)

Abstract: Internet memes have emerged as an increasingly popular means of communication on the web. Although memes are typically intended to elicit humour, they have been increasingly used to spread hatred, trolling, and cyberbullying, as well as to target specific individuals, communities, or society on political, socio-cultural, and psychological grounds. While previous work has focused on detecting harmful, hateful, and offensive memes in general, identifying whom these memes attack (i.e., the ‘victims’) remains a challenging and underexplored area. We attempt to address this problem in this paper. To this end, we create a dataset in which we annotate each meme with its victim(s) such as the name of the targeted person(s), organization(s), and community(ies). We then propose DISARM (Detecting vIctimS targeted by hARmful Memes), a framework that uses named-entity recognition and person identification to detect all entities a meme is referring to, and then, incorporates a novel contextualized multimodal deep neural network to classify whether the meme intends to harm these entities. We perform several systematic experiments on three different test sets, corresponding to entities that are (i) all seen while training, (ii) not seen as a harmful target while training, and (iii) not seen at all while training. The evaluation shows that DISARM significantly outperforms 10 unimodal and multimodal systems. Finally, we demonstrate that DISARM is interpretable and comparatively more generalizable and that it can reduce the relative error rate of harmful target identification by up to 9 % absolute over multimodal baseline systems.

## introduction social media offer the freedom and the means to express deeply ingrained sentiments, which can be done using diverse and multimodal content such as memes. besides being popularly used to express benign humour, internet memes have also been misused to incite extreme reactions, hatred, and to spread disinformation on a massive scale. numerous recent efforts have attempted to characterize harmfulness @xcite , hate speech @xcite , and offensiveness @xcite within memes. most of these efforts have been directed towards detecting malicious influence within memes, but there has been little work on identifying whom the memes target. besides detecting whether a meme is harmful, it is often important to know whether the meme contains an entity that is particularly targeted in a harmful way. this is the task we are addressing here: detecting the entities that a meme targets in a harmful way. harmful targeting in memes is often done using satire, sarcasm, or humour in an explicit or an implicit way, aiming at attacking an individual, an organization, a community, or society in general. for example, fig. 1a depicts justin trudeau, the prime minister of canada, as communally biased against canadians, while favoring alleged killings by muslims, whereas fig. 1b shows an arguably benign meme of the same person expressing subtle humour. essentially, the meme in fig. 1a harmfully targets justin trudeau directly, while causing indirect harm to canadians and to muslims as well. note that in many cases interpreting memes and their harmful intent requires some additional background knowledge for the meme to be understood properly. hence, an automated system for detecting the entities targeted by harmful memes faces two major challenges: (i) insufficient background context, (ii) complexity posed by the implicit harm, and (iii) keyword bias in a supervised setting. to address these challenges, here we aim to address the task of harmful target detection in memes by formulating it as an open-ended task, where a meme can target an entity not seen on training. an end-to-end solution requires (i) identifying the entities referred to in the meme, and (ii) deciding whether each of these entities is being targeted in a harmful way. to address these two tasks, we perform systematic contextualization of the multimodal information presented within the meme by first performing intra-modal fusion between an external knowledge-based contextualized-entity and the textually-embedded harmfulness in the meme, which is followed by cross-modal fusion of the contextualized textual and visual modalities using low-rank bi-linear pooling, resulting in an enriched multimodal representation. we evaluate our model using three-level stress-testing to better assess its generalizability to unseen targets. we create a dataset, and we propose an experimental setup and a model to address the aforementioned requirements, making the following contributions: foot_0 : 1. we introduce the novel task of detecting the entities targeted by harmful memes. ## related work misconduct on social media. the rise in misconduct on social media is a prominent research topic. some forms of online misconduct include rumours @xcite , fake news @xcite @xcite , misinformation @xcite , disinformation @xcite , hate speech @xcite @xcite , trolling @xcite , and cyber-bullying @xcite . web-entity detection along with fair face classification @xcite and semisupervised learning-based classification @xcite were also used for the hateful meme classification task. other noteworthy research includes using implicit models, e.g., topic modelling and multimodal cues, for detecting offensive analogy @xcite and hateful discrimination @xcite in memes. @xcite argued that online attention can be garnered immensely via fauxtography, which could eventually evolve towards turning into memes that potentially go viral. to support research on these topics, several datasets for offensiveness, hate speech, and harmfulness detection have been created @xcite @xcite @xcite . most of the above studies attempted to address classification tasks in a constrained setting. however, to the best of our knowledge, none of them targeted the task of detecting the specific entities that are being targeted. here, we aim to bridge this gap with focus on detecting the specific entities targeted by a given harmful meme. ## dataset the harm-p dataset @xcite consists of 3,552 memes about us politics. each meme is annotated with its harmful label and the social entity that it targets. the targeted entities are coarsely classified into four social groups: individual, organization, community, and the general public. while these coarse classes provide an abstract view of the targets, identifying the specific targeted person, organization, or community in a fine-grained fashion is also crucial, and this is our focus here. all the memes in this dataset broadly pertain to us politics domain, and they target well-known personalities or organizations. to this end, we manually re-annotated the memes in this dataset with the specific people, organizations, and communities that they target. extending harm-p (ext-harm-p). towards generalizability, we extend harm-p by redesigning the existing data splits as shown in ## proposed approach our proposed model disarm, as depicted in fig. 4 , is based on a fusion of the textual and the visual modalities, explicitly enriched via contextualised representations by leveraging clip @xcite . we chose clip as a preferred encoder module for contextualization, due to its impressive zero-shot multimodal embedding capabilities. at first, valid entities are extracted automatically, as part of the process of creating training/validation sets. then, for each meme, we first obtain the contextualized-entity (ce) representation by fusing the clip-encoded context and the entity representation. ce is then fused with bert-based @xcite embeddedharmfulness (eh) encoding fine-tuned on the ocr-extracted text and entities as inputs. we call the resulting fusion output a contextualizedtext (ct) representation. ct is then fused with the contextualized-image (ci) representation, obtained using the clip encoder for the image. we, henceforth, refer to the resulting enriched representation as the contextualized multimodal (cmm) representation. we modify the multimodal lowrank bi-linear pooling @xcite to fuse the input representation into a joint space. this approach, as can be seen in the subsequent sections below, not only can capture complex cross-modal interactions, but it also provides an efficient fusion mechanism towards obtaining a context-enriched representation. finally, we use this representation to train a classifier for our task. we describe each module in detail below. ## experiments we experiment with various unimodal (image/text-only) and multimodal models, including such pre-trained on multimodal datasets such as ms coco @xcite and cc @xcite . we train disarm and all unimodal baselines using pytorch, while for the multimodal baselines, we use the mmf framework. foot_3 foot_4 ## conclusion and future work we introduced the novel task of detecting the targeted entities within harmful memes and we highlighted the inherent challenges involved. towards addressing this open-ended task, we extended harm-p with target entities for each harmful meme. we then proposed a novel multimodal deep neural framework, called disarm, which uses an adaptation of multimodal low-rank bi-linear pooling-based fusion strategy at different levels of representation abstraction. we showed that disarm outperforms various uni/multi-modal baselines in three different scenarios by 4%, 7%, and 13% increments in terms of macro-f1 score, respectively. moreover, disarm achieved a relative error rate reduction of 9% over the best baseline. we further emphasized the utility of different components of disarm through ablation studies. we also elaborated on the generalizability of disarm, thus confirming its modelling superiority over vilbert via interpretability analysis. we finally analysed the shortcomings in disarm that lead to incorrect harmful target predictions. in the present work, we made an attempt to elicit some inherent challenges pertaining to the task at hand: augmenting the relevant context, effectively fusing multiple modalities, and pretraining. yet, we also leave a lot of space for future research for this novel task formulation.