DISARM : Detecting the Victims Targeted by Harmful Memes
Abstract
Internet memes have emerged as an increasingly popular means of communication on the web.
Although memes are typically intended to elicit humour, they have been increasingly used to spread hatred, trolling, and cyberbullying, as well as to target speciﬁc individuals, communities, or society on political, sociocultural, and psychological grounds.
While previous work has focused on detecting harmful, hateful, and offensive memes in general, identifying whom these memes attack (i.e., the ‘victims’) remains a challenging and underexplored area.
We attempt to address this problem in this paper.
To this end, we create a dataset in which we annotate each meme with its victim(s) such as the name of the targeted person(s), organization(s), and community(ies).
We then propose DISARM (Detecting vIctimS targeted by hARmful Memes), a framework that uses named-entity recognition and person identiﬁcation to detect all entities a meme is referring to, and then, incorporates a novel contextualized multimodal deep neural network to classify whether the meme intends to harm these entities.
We perform several systematic experiments on three different test sets, corresponding to entities that are (i) all seen while training, (ii) not seen as a harmful target while training, and (iii) not seen at all while training.
The evaluation shows that DISARM signiﬁcantly outperforms 10 unimodal and multimodal systems.
Finally, we demonstrate that DISARM is interpretable and comparatively more generalizable and that it can reduce the relative error rate of harmful target identiﬁcation by up to 9% absolute over multimodal baseline systems.
Introduction
Social media platforms offer the freedom and the means to express deeply ingrained sentiments, which can be done using diverse and multimodal content such as memes.
Besides being popularly used to express benign humour, Internet
[Figure]
memes are also misused to incite extreme reactions, hatred, and to spread disinformation on a massive scale.
Numerous recent efforts have attempted to characterize harmfulness (Pramanick et al., 2021b), hate speech (Kiela et al., 2020), offensiveness (Suryawanshi et al., 2020), etc.
within memes.
Most of these efforts have been directed towards detecting such malicious inﬂuence within memes, but there has been little work on identifying whom the memes target .
Besides detecting whether a meme is harmful, it is often important to know whether the meme contains an entity that is particularly targeted in a harmful way.
This motivates us to address the problem of detecting the entities that meme targets in a harmful way.
The harmful targeting in memes is often done using satirical, sarcastic, or humorous elements.
This involves either explicit or implicit ways to imply harm.
Such stealth techniques are often used to implicate an individual, an organization, a community, or society, in general.
For example, Fig.
1a depicts Justin Trudeau as communally biased – against Canadians – while favoring alleged killings by Muslims, whereas Fig.
1b shows a benign meme expressing subtle humour.
Essentially, the meme in Fig.
1a harmfully targets Justin Trudeau directly, while causing indirect harm to Canadians and to Muslims as well.
Also, a large number of memes require some addi- tional background context for holistic comprehension.
Hence, some challenges that indicate how intricate it is for an automated system to accurately detect harmful targeting in memes are the following: (i) insufﬁcient background context , (ii) complexity posed by the implicit harm, and (iii) keyword bias in a supervised setting.
We aim to address the task of harmful target detection from memes by posing it as an open-ended task.
The end-to-end solution primarily requires (i) identiﬁcation of the entities mentioned within a meme, and (ii) a multimodal framework that helps in detecting whether the referenced entity is being harmfully targeted in a given meme.
Essentially, we perform systematic contextualization of the multimodal information presented within memes, by ﬁrst performing intra-modal fusion between external knowledge-based contextualized-entity and textually-embedded harmfulness in memes.
This is followed by cross-modal fusion of contextualized textual and visual modalities using low-rank bi-linear pooling, resulting in an enriched multimodal feature.
We evaluate using three-level stress testing towards assessing their generalizability.
We aim to address the aforementioned requirements, and we make the following contributions 1 :
We introduce a novel task of detecting harmful targets within a meme.
We create a new dataset Ext-Harm-P , by extending Harm-P (Pramanick et al., 2021b) via re-annotating the memes for the ﬁne-grained entities they target.
We propose DISARM , a novel multimodal neural architecture that models contextualized multimodal features, towards detecting the harmful targeting in memes.
We empirically showcase that DISARM outperforms 10 unimodal and multimodal baselines by 4% , 7% , and 13% increment in the macroF1 scores in three different evaluation setups.
We ﬁnally discuss DISARM ’s generalizability and interpretability.
Related Work
Misconduct on Social Media.
The rise in misconduct on social media has brought a range of related studies under active investigation.
Some forms of online misconduct include rumors (Zhou et al., 2019), fake news (Aldwairi and Alwahedi, 2018; Shu et al., 2017), misinformation (Ribeiro et al., 2021), disinformation (Alam et al., 2021), hate speech (MacAvaney et al., 2019a; Zhang and Luo, 2018), trolling (Cook et al., 2018), and cyber-bullying (Kowalski et al., 2014; Kim et al., 2021).
Some notable work in this direction includes stance (Graells-Garrido et al., 2020) and rumour veracity prediction, explored in a multi-task learning framework (Kumar and Carley, 2019), wherein the authors proposed a Tree LSTM for characterizing online conversations.
Wu and Liu (2018) explored user and social network feature embeddings towards classifying a message trajectory as genuine vs. fake.
User’s mood along with the online contextual discourse was studied by Cheng et al.
(2017) to demonstrate better modelling for trolling behaviour prediction in contrast with using just the user’s behavioural history.
Relia et al.
(2019) studied the synergy between discrimination based on race, ethnicity and national origin in the physical and in the virtual space.
Studies Focusing on Memes.
Recent efforts have shown interest in incorporating extra contextual information for meme analysis.
Shang et al.
(2021a) proposed knowledge-enriched graph neural networks that use common-sense knowledge for offensive memes detection.
Pramanick et al.
(2021a) focused on detecting COVID-19 related harmful memes and highlighted the challenge of inherent biases within existing multimodal systems.
Pramanick et al.
(2021b) further released another dataset for US Politics and proposed a multimodal framework for harmful meme detection.
The Hateful Memes detection challenge by
Facebook (Kiela et al., 2020) introduced the task of classifying a meme as either hateful or nonhateful.
Different approaches such as feature augmentation, attention mechanism, and multimodal loss re-weighting were attempted (Das et al., 2020;
Sandulescu, 2020; Zhou and Chen, 2020; Lippe et al., 2020) as part of this task.
Sabat et al.
(2019) studied hateful memes by highlighting the importance of visual cues such as structural template, graphic modality, causal depiction, etc.
Interesting approaches such as web-entity detection along with fair face classiﬁcation (Karkkainen and Joo, 2021) and semi-supervised learning-based classiﬁcation (Zhong, 2020) were also used for the hateful meme classiﬁcation task.
Other notewor-
[Table]
thy work includes implicit models and topic modelling of multimodal cues for detecting offencive analogy (Shang et al., 2021b) and hateful discrimination (Mittos et al., 2020) in memes.
Wang et al.
(2020) argued that online attention can be garnered immensely via fauxtography content, which could eventually evolve towards becoming memes that go viral.
Several datasets including the ones about offence, hate speech, harmfulness, etc. have been proposed (Suryawanshi et al., 2020; Kiela et al., 2020; Pramanick et al., 2021a,b; Gomez et al., 2019).
Most of these studies attempt to address classiﬁcation tasks in a constrained setting.
However, to the best of our knowledge, none of them addressed the task of detecting the speciﬁc targets of hate, offence, harm, etc.
We intend to explore precisely this task in this work for harmful memes.
Dataset
The Harm-P dataset (Pramanick et al., 2021b) consists of 3 , 552 US politics memes.
Each meme is annotated with its harmful label and the social entity that it targets.
The target entities are coarsely classiﬁed into four social groups – individual, organization, community, and the general public.
While these coarse classes provide an abstract view of targets, identifying the targeted person, organization, or community in a ﬁne-grained fashion is also crucial.
All the memes in this dataset broadly pertain to US Politics domain , and they target well-known personalities or organizations.
To this end, we manually re-annotated this dataset with speciﬁc entities of people, organizations, and communities that harmful memes target.
Extending Harm-P ( Ext-Harm-P ).
Towards generalizability, we extend Harm-P by reformulating existing splits as shown in Table 1.
We call the resulting dataset Ext-Harm-P .
For training, we use the harmful memes provided as part of the original dataset (Pramanick et al., 2021b) and re-annotate them for the ﬁne-grained entities being targeted harmfully as positive sam- ples ( harmful targets).
This is matched with twice as many negative samples ( not-harmful targets).
For negative targets, top-2 entities from the original entity lexicon, which are not labeled for harmfulness and have the highest lexical similarity with the meme text are selected (Ferreira et al., 2016).
This atleast ensures lexical similarity with entities referenced within a meme, thereby facilitating a confounding effect (Kiela et al., 2020) as well.
For the test set, all the entities are ﬁrst extracted automatically using named entity recognition (NER) and person identiﬁcation (PID) 2 .
This is followed by manual annotation of the test set towards ensuring a robust evaluation.
Dataset Annotation process
Since, assessing harmfulness in memes is a highly subjective task, annotators were requested to follow 4 key steps towards annotating each meme as enlisted below, to ensure labeling consistency.
We consider an example, depicted in Fig.
2 to demonstrate the steps taken while annotating 3 .
Annotators were requested to:
Understand a meme and its background context clearly.
The argument being made in the example meme, depicted in Fig.
2 is reasonably self-explanatory, due to its descriptiveness. 2.
Enlist all the valid candidate entities that are referenced within a given meme.
For the sample meme (Fig. 2), valid entities are bill clinton, hillary clinton, white house, donald trump and democrat .
Assign suitable entities as harmful , as positive cases for harmful targeting. Entities bill clin- [Figure]
donald trump barack obama hillary clinton mike [Figure] [Figure] [Figure]
[Figure]
muslim islam russian
[Figure]
ton, hillary clinton and democrat are insinuated in the meme, for depicted bias over the appointment of their kin for government roles.
Finally, assign harmless references under notharmful category.
In this case (Fig. 2), donald trump and white house would be annotated as not-harmful references, for not being the subject of implied insinuation.
We had 3 annotators, and a consolidator.
The inter-annotator agreement before consolidation was Fleiss Kappa=0.48 (moderate), and after consolidation was 0.64 (substantial).
Analyzing Harmful Targeting in Memes.
Memes in Ext-Harm-P are about US Politics , hence they prominently feature entities like Joe
Biden and Donald Trump , both harmfully and harmlessly.
Although the ratio between either type of referencing vary across individuals , organizations , and communities categories (Fig.
The top-5 harmfully referenced individuals and organizations are observed to be subjected to a higher amount of harm, as against the support they garner.
In contrast, the stacked plots for the top-5 harmfully targeted communities: Mexicans, Black, Muslim, Islam , and Russian depict relatively higher support in the form of harmless referencing on social media (Fig. 3c).
Proposed Approach
DISARM , as depicted in Fig. 4, models the fusion of textual and visual modalities, explicitly enriched via contextualised representations by leveraging CLIP Radford et al.
CLIP was empirically chosen as a preferred encoder module for contextualization, due to its impressive zero-shot multimodal embedding capabilities.
At ﬁrst, valid entities are extracted automatically, as part of the train/val set creation.
Then for each meme, we ﬁrst obtain the contextualized-entity ( CE ) representa- [Figure] tion by fusing the CLIP (Radford et al., 2021) encoded context and the entity representation.
CE is then fused with BERT-based (Devlin et al., 2019) embedded-harmfulness ( EH ) encoding ﬁne-tuned over OCR-extracted text and entities as inputs.
We call the resulting fusion output as contextualizedtext ( CT ) representation.
CT is then fused with the contextualized-image ( CI ) representation, obtained using the CLIP encoder for image.
We, henceforth, refer to the resulting enriched representation as the contextualized multimodal ( CMM ) representation.
We modify multimodal low-rank bi-linear pooling (Kim et al., 2017), to fuse input features represented in joint embedding space.
This approach, as can be noted from the subsequent sections, is empirically observed to not only capture complex cross-modal features, but also provide an efﬁcient fusion mechanism towards obtaining context-enriched features.
Finally, CMM is used to train a classiﬁcation head for our task.
We describe each module in detail below.
Low-rank Bi-linear Pooling (LRBP).
We begin by revisiting low-rank bi-linear pooling to set the necessary background.
Due to many parameters in bi-linear models, Pirsiavash et al.
(2009) suggested a low-rank bi-linear (LRB) approach to reduce the rank of the weight matrix W i .
Con- sequently parameters, and hence the complexity is reduced.
The weight matrix W i is re-written as W i =
U
i V Ti , where U i ∈ R N × d and
V i ∈ R M × d , effectively putting an upper bound of min( N, M ) on the value of d .
Therefore, the low-rank bi-linear models can be expressed as follows:
[Equation]
where 1 ∈ R d : column vector of ones, and ◦ :
Hadamard product.
f i in Equation 1 can be further re-written to obtain f as follows:
[Equation]
where f ∈ { f i } , P ∈ R d × c , b ∈ R c .
d and c : output and LRB hyper-parameters, respectively.
Following (Kim et al., 2017), we introduce a non-linear activation based formulation for the
LRBP.
Kim et al.
(2017) argued that non-linearity both before and after the Hadamard product complicates the gradient computation.
This addition to
Equation 2, can be represented as follows:
[Equation]
We slightly modify multimodal low-rank bilinear pooling ( MMLRBP ).
Instead of directly projecting the input x ∈ R N and y ∈ R M to a lower dimension d , we ﬁrst project the input modalities in a joint space ( N ).
We then perform LRBP as expressed in Equation 3, by using jointly embedded representations x mm ∈ R N × d and y
mm ∈ R N × d to obtain a multimodal fused feature f mm , as expressed below:
[Equation]
Structured Context.
Towards modelling auxiliary knowledge, we curate contexts for the memes in Ext-Harm-P .
First, we use meme text as the search query 4 to retrieve relevant contexts.
We treat the title and the ﬁrst paragraph from the resulting top document, towards modelling the required context and represent it as con .
Contextualized-entity
Representation (CE).
Towards modelling the context-enriched entity, we ﬁrst obtain the embedding of a given entity ent .
Since we have a ﬁnite set of entities referenced in the memes in our dataset, we perform a lookup in the embedding matrix ( ∈ R V × H ) to obtain the corresponding entity embedding ent ∈ R H , with H =
300 being the embedding dimension and V is the vocabulary size.
The embedding matrix is jointly trained from scratch , during training.
We project the obtained entity representation ent into 512 dimensional space, and we call it e .
To augment a given entity with relevant contextual information, we fuse it with contextual representation c ∈ R 512 , obtained by encoding the associated context ( con ) using CLIP text-encoder (Radford et al., 2021).
We perform this fusion using our adaptation of multimodal low-rank bi-linear pooling (Equation 4).
This gives contextualized-entity ( CE ) representation c ent as shown below:
[Equation]
where c ent ∈ R 512 , P 1 ∈ R 256 × 512 , b ∈ R 512 ,
U 1 ∈ R 512 × 256 and V 1 ∈ R 512 × 256 .
Contextualized-Text (CT)
Representation.
Once we obtain the contextualized-entity embedding c ent , we concatenate it with the BERT encoding for the combined representation of the
OCR-extracted text and the entity ( o ent ∈ R 768 ).
We call this encoding embedded-harmfulness ( EH ) representation.
The concatenated feature ∈
R 1280 is then projected non-linearly into a lower dimension using a dense layer of size 512.
We term the resultant vector c txt as contextualized-text (CT) representation.
[Equation]
where W ∈ R 1280 × 512 .
Contextualized Multimodal (CMM) Representation.
Once we obtain the contextualized-text representation c txt ∈
R 512 , we again perform multimodal low-rank bi-linear pooling using Equation 4 to fuse it with the contextualizedimage representation c img ∈ R 512 , obtained using
CLIP image-encoder (Radford et al., 2021).
The operation is expressed as
[Equation]
where c mm ∈ R 512 , P 2 ∈ R 256 × 512 , U 2 ∈
R 512 × 256 and V 2 ∈ R 512 × 256 .
Notably, we learn two different projection matrices P 1 and P 2 , for the two fusion operations performed as part of
Equations 5 and 7, respectively since the fused representations at the respective steps are obtained using different modality-speciﬁc interactions.
Classiﬁcation Head.
Towards modelling the binary classiﬁcation for a given meme and a corresponding entity as either harmful or non-harmful, we use a shallow multi-layer perceptron with a single dense layer of size 256, which represents a condensed representation for classiﬁcation.
We ﬁnally map this layer to a single dimension output via a sigmoid activation.
We use binary crossentropy for the back-propagated loss.
Experiments
We experiment with various state-of-the-art unimodal (image/text-only) and multimodal baseline systems, including the ones that are pre-trained using multimodal datasets such as MS COCO (Lin et al., 2014) and CC (Sharma et al., 2018).
We train DISARM and all unimodal baselines using PyTorch and multimodal baselines using the
MMF framework 5 6 .
For evaluation, we use commonly used macro-average versions of metrics such as accuracy, precision, recall and F1 score.
The harmful class recall is relevant for our study as it characterizes the model’s performance, towards detecting harmfully targeting memes correctly.
The results reported are averaged across ﬁve independent runs.
Evaluation Strategy.
Towards examining a realistic setting, we pose our evaluation strategy as an open-class one.
We train all the systems with the set having positive (harmful) samples and twice as many negative (not-harmful) samples.
We then evaluate using open-class testing, for all referenced entities (some possibly unseen during training) per meme, effectively making the evaluation more realistic.
To this end, we formulate three testing scenarios as follows, with their
Harmful (H) and Not-harmful (N) sample counts:
(a) Test set A (316H, 296N) – Includes examples with entities seen during training.
(b) Test set B (27H, 94N) – The examples in this set correspond to the entities that are unseen as harmful , during training.
(c) Test set C (16H, 76N) – Only entities that are completely unseen during the training are considered.
Baseline Models.
Our baselines include both unimodal and multimodal models as follows:
– Unimodal Systems : (cid:73) VGG16, VIT: For the unimodal (image-only) systems, we use two well-known models: VGG16 (Simonyan and Zisserman, 2015) and VIT (Vision Transformers) that emulate a Transformer based application jointly over textual tokens and image patches (Dosovitskiy et al., 2021).
(cid:73) GRU, XLNet: For the unimodal (text-only) systems, we use GRU (Cho et al., 2014), which adaptively captures temporal dependencies, and XLNet (Yang et al., 2020), which implements a generalized auto-regressive pre-training strategy.
– Multimodal Systems : (cid:73) MMF Transformer:
This is a multimodal Transformer model that utilizes visual and language tokens with selfattention 7 .
(cid:73) MMBT: Multimodal Bitransformer (Kiela et al., 2019) captures the intramodal and the inter-modal dynamics.
(cid:73) ViLBERT CC: Vision and Language BERT (Lu et al., 2019), pre-trained on CC (Sharma et al., 2018), is a strong model with task-agnostic joint representation.
(cid:73) Visual BERT COCO: Visual BERT (Li et al., 2019), pre-trained on the MS COCO dataset (Lin et al., 2014).
Experimental Results.
We compare the performance of several unimodal and multimodal systems (pre-trained and otherwise) and DISARM along-with its variants.
All systems are evaluated using the 3-way testing strategy described above.
We then perform ablation studies over contextualized-entity , its fusion with embeddedharmfulness resulting into contextualized-text and the ﬁnal fusion with contextualized-image , yielding the contextualized-multimodal modules of
DISARM 8,9 .
This is followed by interpretability analysis.
Finally, we discuss the limitations of
DISARM by performing error analysis 10 .
All Entities Seen During Training : Towards unimodal text-only baseline evaluation, the GRUbased system yields a relatively lower harmful recall 0 .
74 along-with an overall better F1 0 .
[Table] m & [Table]
a r i [Table] [Table] in comparison to XLNet’s 0 . 82 and a lower F of 0 .
67 , as shown in Table 2.
The lower harmful precision 0 .
65 and not-harmful recall of 0 .contribute to the lower F1 score for XLNet.
Amongst image-only unimodal systems, VGGbased (image-only) system performs better with not-harmful recall 0 .
81 , but is poor for detecting the harmful memes correctly with a lower harmful recall value of 0 . 68 .
Whereas, VIT has a relatively better harmful class recall 0 .
Overall, the unimodal results (Table 2) indicate the efﬁcacy of self-attention over convolution for images and RNN
Multimodally pre-trained models such as VisualBERT (MS COCO (Lin et al., 2014)) and
ViLBERT (Conceptual Captions (Sharma et al., 2018)), yield moderate F1 scores of 0 .
70 and 0 .
68 , and harmful recall values of 0 .
78 and 0 .
77 , respectively (Table 2).
Fresh training facilitates more meaningful results in favour of not-harmful precision ( 0 . 78 and 0 .
78 respectively) and harmful recall ( 0 . 84 and 0 .
82 respectively).
Overall, ViLBERT yields the most balanced performance with 0 .
75 F1 score.
It can be inferred from these results (Table 2) that multimodal pre-training could leverage domain relevance.
Multimodal low-rank bi-linear pooling is observed to distinctly enhance the performance by 4% and 6% F1 scores.
The improvements can be attributed to the fusion of the CE and EH representations, respectively with CI , instead of a simple concatenation (Table 2).
This is more prominent for CE with 0 .
78 F1, effectively implying the importance of the background context.
Finally,
DISARM is observed to yield a balanced F1 score of 0 .
78 , with a reasonable precision of 0 .
74 for non-harmful and the best recall of 0 .
86 for the harmful categories, respectively.
All Entities Unseen as Harmful Targets
During
Training : With Test Set B, the evaluation is made slightly more challenging (Table 2) in terms of the entities to be assessed, as these were never seen as part of the training process as harmful .
Unimodal systems mostly perform poorly in terms of both precision and recall for harmful class, with the exception of XLNet (Table 2) with harmful class recall as 0 . 56 .
For the multimodal baselines, the performance of the systems that are pre-trained using
COCO (VisualBERT) and CC (ViLBERT) yields moderate recall of 0 .
64 and 0 .
71 for the harmful class in contrast to what we see for Test Set A in
Table 2.
This could be due to additional commonsense reasoning facilitated by such systems, on a test set that is more open-ended compared to Test
Set A. Their non-pre-trained versions along with
MM Transformer and MMBT achieve better F scores, but with low harmful class recall.
Multimodal fusion using MMLRBP is observed (Table 2) to obtain an improved harmful class recall for CE ( 0 . 52 ) and lower values for EH ( 0 . 37 ) based fusion with CI , respectively.
This reconﬁrms the utility of context.
In comparison,
DISARM yields a balanced F1 score of
.with the best precision values 0 .
83 and 0 .
38 , along with decent recall values of 0 .
79 and 0 .
69 for not-
[Figure]
Target Candidate → democratic party
Context → Politics tears families apart during bruising political season, when many Americans drop friends and family members who have different political views.
harmful and harmful memes, respectively.
All Entities Unseen During Training : The results decline in this scenario (similarly to Test Set B), except for the harmful class recall score for XLNet ( 0 . 62 ), as shown in Table 3.
In the current scenario (Test Set C), none of the entities being assessed during testing is seen during the training phase.
For multimodal baselines, we see a similar trend for VisualBERT (COCO) and ViLBERT (CC), with the harmful class recall of 0 .
72 for ViLBERT (CC) being signiﬁcantly better than 0 .for VisualBERT (COCO).
This again emphasizes the need for the afﬁnity between the pre-training dataset and the downstream task at hand.
In general, the precision for the harmful class is very low.
We observe (Table 3) signiﬁcant increase in the harmful class recall for MMLRBP -based multimodal fusion of CI with CE ( 0 .
69% ), as against a decrease in the same with EH ( 0 .
In comparison, DISARM yields a low, yet the best harmful precision of 0 .
36 and a moderate recall value 0 .
70 (See Table 3).
Also, besides yielding reasonable precision and recall values of 0 . 86 and 0 . 76 , respectively, for the not-harmful class, DISARM exhibits better average precision, recall, and F scores of 0 .
73 and 0 .
64 , respectively.
Generalizability of DISARM.
The generalizability of DISARM follows from the characteristic modelling and context-based fusion.
DISARM demonstrates the potential for detecting harmful targeting for a diverse set of entities.
Speciﬁcally, the three-way testing setup inherently captures the efﬁcacy with which DISARM can detect unseen harmful targets.
Prediction for entities completely unseen during training yields better results (Table 2 and 3), and suggests possibly induced bias in the former scenario.
Furthermore, it is a direct consequence of the fact that we were able to incorporate only a limited set of 246 potential targets.
Overall, we argue that DISARM generalizes well for unseen entities with 0 .
6498 and 0 .
6412 macro-F1 scores, as compared to ViLBERT’s 0 .
5782 and MMBT’s 0 .
5146 , for Test Sets B and C, respectively.
Comparative
Diagnosis.
Despite marginally better harmful recall of ViLBERT (CC) for Test
Sets B (Table 2) and C (Table 3), the overall balanced performance of DISARM appears to be reasonably justiﬁed based on the comparative interpretability analysis between the attention maps for both the systems.
Fig.
5 shows attention maps for an example meme.
It depicts a meme that is correctly predicted for harmfully targeting democratic party by DISARM and incorrectly by ViLBERT.
As visualised in Fig.
5a, harmfully-inclined word killing effectively attends not only to baby , but also to democrats and racist .
The relevance is depicted via different color schemes and intensities, respectively.
Interestingly, killing also attends to democratic party , both as part of OCR-extracted text and the target-candidate, jointly encoded by
BERT.
Multimodal attention being leveraged by
DISARM depicted (via CLIP encoder) in Fig.
5b, demonstrates the utility of contextualised attention over male ﬁgure depicted, who represents insinuation of democratic party .
Also, DISARM has a relatively focused ﬁeld of vision, shown in Fig.
5c as compared to a relatively scattered one for ViLBERT (Fig. 5d).
This demonstrates a better multimodal modelling capacity of DISARM as compared to that of ViLBERT.
Conclusion and Future Work
In this work, we introduced a novel task of detecting targeted entities within harmful memes and highlighted the inherent challenges involved.
Towards addressing this open-ended task, we extended Harm-P with target entities for each harmful meme.
We then proposed a novel multimodal deep neural framework, called DISARM that employs an adaptation of multimodal low-rank bilinear pooling-based fusion strategy at different levels of feature abstraction.
We showed that
DISARM outperforms various uni/multi-modal baselines in three different scenarios by 4% , 7% , and 13% increments in the macro-F1 score, respectively.
Also, DISARM achieved a relative error rate reduction of 9% over the best baseline.
We further emphasized the utility of different components of DISARM through ablation studies.
We also elaborated on the generalizability of DISARM and established its modelling efﬁcacy over that of
ViLBERT via.
interpretability analysis.
We ﬁnally analysed the shortcomings in DISARM that lead to incorrect harmful target predictions.
Through this work, we made an attempt towards eliciting a few inherent challenges pertaining to the task at hand – augmenting relevant context, effectively fusing multiple modalities, and pre-training.
This reinstates the required motivation and leaves scope for future investigations in this direction.
Acknowledgments
The work was partially supported by a Wipro research grant, Ramanujan Fellowship, the Infosys
Centre for AI, IIIT Delhi, and ihub-Anubhuti-iiitd
Foundation, set up under the NM-ICPS scheme of the Department of Science and Technology, India.
It is also part of the Tanbih mega-project, developed at the Qatar Computing Research Institute, HBKU, which aims to limit the impact of “fake news,” propaganda, and media bias by making users aware of what they are reading, thus promoting media literacy and critical thinking.
Ethics and Broader Impact
Reproducibility.
We present detailed hyperparameter conﬁgurations in Appendix A and Table 4.
The source code, and the dataset ExtHarm-P are available at https://github.
com/LCS2-IIITD/DISARM
User Privacy.
The information depicted/used doesn’t include any personal information.
Copyright aspects are attributed to the dataset source.
Annotation.
The annotation was conducted by
NLP experts or linguists in India.
The annotators were fairly treated and were duly compensated.
We conducted several discussion sessions to make sure all annotators could understand the distinction between harmful vs. not-harmful referencing.
Biases.
Any biases found in the dataset are unintentional, and we do not intend to cause harm to any group or individual.
We acknowledge that detecting harmfulness can be subjective, and thus it is inevitable that there would be biases in our goldlabelled data or the label distribution.
This is addressed by working on a dataset that is created using general keywords about US Politics, and also by following a well-deﬁned schema, which sets explicit deﬁnitions during annotation.
Misuse Potential.
We state that this dataset can be potentially used for ill-intended purposes, like biased targeting of individuals/communities/organizations, etc. that may or may not be related to demographics and other information within the text.
Intervention with human moderation would be required to ensure that this does not occur.
Intended Use.
We make use of the existing dataset in our work in line with the intended usage prescribed by its creators and solely for research purposes.
This applies in its entirety to its further usage as well.
We commit to releasing our dataset aiming to encourage research in studying harmful targeting in memes on the web.
We distribute the dataset for research purposes only, without a license for commercial use.
We believe that it represents a useful resource when used appropriately.
Environmental Impact.
Finally, due to the requirement of GPUs/TPUs large-scale Transformers require a lot of computations, contributing to global warming (Strubell et al., 2019).
However, in our case, we do not train such models from scratch; rather, we ﬁne-tune them on relatively small datasets.
Moreover, running on a CPU for inference, once the model has been ﬁne-tuned, is perfectly feasible, and CPUs contribute much less to global warming.
A Implementation Details and Hyperparameter Values
We train all the models using PyTorch on NVIDIA Tesla V100 GPU, with 32 GB dedicated memory, CUDA-11.2 and cuDNN-8.1.1 installed.
For the unimodal models, we import all the pre-trained weights from the TORCHVISION.MODELS 11 , a sub-package of the PyTorch framework.
We initialize the remaining weights randomly using a zero-mean Gaussian distribution with a standard deviation of 0.02.
We train DISARM in a setup considering only harmful class data from Harm-P (Pramanick et al., 2021b).
We extend it by manually annotating for harmful targets, followed by including not-harmful samples using automated entity extraction (textual and visual) strategies for train/val splits and manual annotation (for both harmful and not-harmful) for the test split.
We train all models we experiment with, using the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 1 e − 4 , with a weight decay of 1 e − 5 and a Binary Cross-Entropy (BCE) loss as the objective function.
We extensively ﬁnetuned our experimental setups, based upon different architectural requirements to ﬁnalise on aforementioned hyper-parameters.
We also use earlystopping for saving the best intermediate checkpoints as well.
Table 4 gives more detail about the hyper-parameters we used for training.
On average, it took approx.
2:30 hours to train a multimodal neural model.
B Ablation Study
In this section, we present some ablation studies for CE , EH , CT and CI based sub-modules of
DISARM , examined in isolation and combinations, and ﬁnally for DISARM using CMM representation.
Test Set A : As observed in the comparisons made with the other baseline systems for the Test Set A in Table 2, the overall range of the F1 scores is relatively higher with the least value observed to be 0 .
66 for XLNet (text-only) model. The results for unimodal systems, as cab be observed from Table 5, is satisfactory with values of 0 .
73 , and 0 .
77 for CE EH , and CI based unimodal systems, respectively.
For multimodal systems, we can observe distinct lead for the MMLRBP -based fusion strategy, for both CE and EH based systems over the concatenation-based approach, except for EH ’s recall drop by 7% .
Finally DISARM yields the best overall F1 score of 0 .
Test Set B : With context not having any harmfulness cues for a given meme, when considered in isolation, the unimodal CE based module performs the worst with 0 .
48 F1 and 0 .
07 harmful recall, in the open-ended setting of Test Set
B.
In contrast, EH yields an impressive F1 score of 0 .and
a harmful recall of 0 .
This relative gain of 7% in the F1 score could be due to the presence of explicit harmfulness cues.
The complementary effect of considering contextual information can be inferred from the joint modeling of CE and EH , to obtained CT , that enhances the F1 and harmful recall by 2% and 3% , respectively (Table 5).
Unimodal assessment of CI performs moderately with 0 .
51 F1 score, but with a poor harmful recall of 0 . 15 .
MMLRBP , towards jointmodeling of CE and CI yields a signiﬁcant boost in the harmful recall with value 0 .
52 (Table 5).
On the other hand, MMLRBP -based fusion of EH and CI yields 0 .
54 F1 score, which is 1% below that for the unimodal EH system.
This emphasizes the importance of accurately modeling the embedded harmfulness, besides augmenting with additional context.
The complementary effects of CE , EH , and CI are observed for DISARM with a balanced F1 score of 0 .
65 and a competitive harmful recall value of 0 . 69 .
Test Set C :
As observed in the previous scenario (Test Set B), the unimodal models for CE yield a low F1 score of 0 .
48 and the worst harmful recall value of 0 .
Much better performance is observed for unimodal setups including EH , and its joint modelling with CE with improved F scores of 0 . 56 and 0 .
58 , respectively, along with [Table] the harmful recall score of 0 . 56 and 0 . 57 , respectively.
CI based unimodal evaluation again yields a moderate F1 score of 0 .
53 (Table 5), along with a poor harmful recall of 0 . 19 , which shows its inadequacy for modelling harmful targeting on its own.
For multimodal setups, the joint modelling of CE and CI beneﬁts from MMLRBP based fusion, yielding a gain of 7% and 13% in F1 and harmful recall, respectively.
This conﬁrms the importance of contextual multimodal semantic alignment.
Correspondingly, joint multimodal modelling of EH and CI regresses the unimodal afﬁnity within the EH .
Finally, DISARM outperforms all other systems in this category with the best F score of 0 .
64 , with a decent harmful recall score of 0 . 7 .
The results reported in this work are for the comparison and analysis of the most optimal set of design and baseline choices.
We have performed extensive experiments as part of preliminary investigations, with different contextual modelling strategies, attention mechanisms, modelling choices, etc., to reach a conclusive architectural conﬁguration, that indicates promise towards addressing the task of target detection from harmful memes.
C Error Analysis
It is evident from the results shown in Table 2 and 3, that DISARM still has short-comings.
Examples like the one shown in Fig.
6 are seemingly harmless , both textually and visually, but imply serious harm to a person of color in an implicit way.
Such complexity can be challenging to model, without providing additional context like people of colour face racial discrimination all over the world .
This is also analogous to a fundamental challenge associated with detecting implicit hate (MacAvaney et al., 2019b).
Despite modelling contextual in-
[Figure]
Target Candidate → person of color
Context
→
During the evening of the
VP debates, Joe Biden settled down on his soft couch with a glass of warm milk to watch this.
formation explicitly in DISARM , it misclassiﬁes this meme.
Although the context obtained for this meme pertains to its content (Fig. 6), it does not relate to global racial prejudice , which is key to ascertaining it as a harmfully targeting meme.
Moreover, besides context, visuals and the message embedded within the meme do not convey deﬁnite harm when considered in isolation.
This error can be inferred clearly from the embedded- [Figure] harmfulness, contextualised-visuals, and the visuals being attended by DISARM as depicted in Fig.
6a, 6b and 6c respectively.
On the other hand, as shown in the visual attention plot for ViLBERT in Fig.
6d, the ﬁeld of view being attended to encompasses the visuals of Kamala Harris , who is the person of colour being primarily targeted through the meme.
Besides the distinct attention on the primary target-candidate within the meme,
ViLBERT could have leveraged the pre-training it received from Conceptual Captions (CC) (Sharma et al., 2018), a dataset known for its diverse coverage of complex textual descriptions.
This essentially highlights the importance of multimodal pretraining using the dataset that is not as generic as
MS COCO (Lin et al., 2015), but facilitate modelling of the complex real-world multimodal information, especially for tasks related to memes.
D Annotation Guidelines
Before discussing details about the annotation process, revisiting the deﬁnition of harmful memes would set the pretext towards consideration of harmful targeting and not-harmful referencing.
According to Pramanick et al.
(2021b), abuse, offence, disrespect, insult or insinuation of a targeted entity or any socio-cultural or political ideology, belief, principle, or doctrine associated with that entity amounts to the expression of harm.
Another common understanding 12,13,14 about the harmful content is that it could be anything online which causes a person distress.
It is an extremely subjective phenomenon, wherein what maybe be harmful to some, might not be considered an issue by others.
This makes it signiﬁcantly challenging to characterize and hence study it via the computational lens.
Based on a survey of 52 participants, Scheuerman et al.
(2021) deﬁnes online harm to be any violating content that results in any (or a combination) of four categories: (i) physical harm, (ii) emotional harm, (iii) relational harm and (iv) ﬁnancial harm.
With this pretext, we deﬁne below 2 types of referencing that we have investigated in our work, within the context of internet memes: (i) harmful and (ii) not-harmful .
D. Reference types Harmful.
The understanding about harmful referencing ( targeting ) in memes, can be sourced back to the deﬁnition of harmful memes by Pramanick et al.
(2021b), wherein a social entity is subjected to some form of ill-treatment like mental abuse, psycho-physiological injury, proprietary damage, emotional disturbance, or public image damage, based on their background (bias, social background, educational background, etc.) by a meme author.
Not-harmful.
Not-harmful referencing in memes is any benign mention (or depiction) of a social entity via humour, limerick, harmless pun or any content that does not cause distress.
Any reference that is not harmful, comes under this category.
D. Characteristics of harmful targeting
There are several factors that collectively facilitate characterisation of harmful targeting in memes.
Few are enlisted below:
A prominent way of harmfully targeting an entity in a meme is by leveraging sarcastically harmful analogies, framed via either textual or visual instruments (Fig. 7a).
[Table]
There could be multiple entities being harmfully targeted within a meme as depicted in Fig.
Hence, annotators were asked to provide all targets as harmful, without exception.
Harmful targeting within a meme could have visual depictions, that are either gory, violent, graphically sensitive or pornographic (Fig. 7b).
Any meme that insinuates an entity on either social, political, professional, religious grounds, can cause harm (Fig.
7c and 7d).
Any meme that implies an explicit/implicit threat to an individual, community, national or international entity is harmful (Fig. 7d and 7e).
Whenever there is any ambiguity regarding the harmfulness of any reference being made, we requested authors to proceed with the best of their understanding.
E Ext-Harm-P Characteristics E. Lexical Analysis
Interestingly, a signiﬁcant number of memes are disseminated making references to popular individuals like Joe Biden, Donald Trump, etc. , as can be observed for individual sub-category (for both harmful and not-harmful categories), in Table 6.
It can be noticed for harmful – organization in Table 6, top-5 harmfully targeted organizations include top-2 leading political organizations ( democratic and republican party ), which are of signiﬁcant political relevance, followed by libertarian party , a media house ( CNN ) and ﬁnally government .
Whereas, non-harmfully referenced organizations includes biden camp and trump administration , that are mostly leveraged for harmfully targeting (or otherwise) the associated public ﬁgure.
Finally, communities like mexicans, black, muslim, islam and russian are often immensely prejudiced online.
Whereas, non-harmfully targeted communities like trump supporters and african american are not targeted as often as the aforementioned ones Table 6.
This largely emphasizes the inherent bias that multimodal content like memes exhibit, which has a direct inﬂuence on the efﬁcacy of machine/deep learning-based systems.
The reasons for this bias are mostly linked to societal behaviour at the organic level, and the limitations posed by current techniques to process such data.
Distinct mutual exclusion for harmful vs. not-harmful categories for community shows the inherent bias that could pose a challenge, even for the best multimodal deep neural systems.
The high pervasiveness of a few prominent keywords could effectively lead to increasing bias towards them for speciﬁc eventualities.
Whereas, the signiﬁcant overlap observed in
Table 6 for the enlisted entities, between harmful and not-harmful individuals, highlight the need for sophisticated multimodal systems that can effectively reason towards making a complex decision like detecting harmful targeting within memes.
E. Meme-message Length Analysis
Most of the harmful memes are observed to be created using texts of length 16 − 18 (Fig. 8).
Whereas, not-harmful meme-text lengths are have a relatively higher std.-dev., possibly due to diversity of not-harmful messages.
Trump and Republic party have meme-text length distributions similar for not-harmful category; skewing left, but gradually decreasing towards the right.
This suggests varying content generation pattern amongst meme creators (Fig.
Meme-text length distribution for Biden closely approximates a normal distribution with the low std.-dev.
Both the categories would pre-dominantly entail creating memes with shorter text lengths, due to the popularity of Biden amongst humorous content creators.
A similar trend could be seen for the democratic party as well, where most of the samples are observed to be falling within 50 − 75 memetext length range.
The overall harmful and notharmful meme-text length distribution is observed to be fairly distributed across different meme-text lengths for mexican .
Whereas, the amount of harm intended towards black community is observed [Figure] to be signiﬁcantly more, as compared to moderately distributed not-harmful memes depicted by the corresponding meme-text length distribution in Fig.