Meme-ingful Analysis: Enhanced Understanding of Cyberbullying in Memes Through Multimodal Explanations (2024)

Abstract: Internet memes have gained significant influence in communicating political, psychological, and sociocultural ideas. While meme are often humorous, there has been a rise in the use of memes for trolling and cyberbullying. Although a wide variety of effective deep learning-based models have been developed for detecting offensive multimodal memes, only a few works have been done on explainability aspect. Recent laws like “right to explanations” of General Data Protection Regulation, have spurred research in developing interpretable models rather than only focusing on performance. Motivated by this, we introduce MultiBully-Ex, the first benchmark dataset for multimodal explanation from code-mixed cyberbullying memes. Here, both visual and textual modalities are highlighted to explain why a given meme is cyberbullying. A Contrastive Language-Image Pretraining (CLIP) projection based multimodal shared-private multitask approach has been proposed for visual and textual explanation of a meme. Experimental results demonstrate that training with multimodal explanations improves performance in generating textual justifications and more accurately identifying the visual evidence supporting a decision with reliable performance improvements.

## introduction the tremendous increase in multimodal content due to the widespread use of social media platforms renders human moderation of such information untenable @xcite . memes, which are images with tiny text descriptions embedded in them, have become a popular kind of multimodal content on ## related works cyberbullying is very reliant on linguistic subtlety. researchers have recently provided a lot of attention to automatically identifying cyberbullying in social media. in this section, we will review recent works on the detection and explainability aspects of cyberbullying. detection: researchers have made significant strides in detecting meme-based cyberbullying and offensive content. maity et al. (2022a) created multibully, a twitter and reddit dataset in codemixed language, proposing two multitask platforms for detecting bullying memes, sentiment, and emotion. pramanick et al. (2021) extended the harmeme dataset and developed a deep multimodal network to detect harmful memes, focusing on covid-19 and us politics. other notable works include kiela et al. (2020)'s hate speech detection with 69.47% accuracy using visual-bert, gomez et al. (2020)'s mmhs150k dataset of 150k tweets, and suryawanshi et al. (2020)'s multioff dataset for identifying offensive meme content, which showcased a fusion method combining text and image modalities. explainability: lime (ribeiro et al., 2016) and shap (lundberg and lee, 2017) have been used to advance both textual and visual explainability in machine learning models. zaidan et al. (2007) improved sentiment classification by employing human-annotated "rationales." mathew et al. (2020) introduced hatexplain, finding that training with human rationales reduced bias. karim et al. (2021) developed an explainable hate speech detection in bengali, highlighting crucial words. hee et al. (2022) visualized how vilbert and visualbert models captured slurs in hateful memes, discovering the image modality's significant contribution. while most studies used explainability to justify model outputs, our task uniquely focuses on explainability as the output itself, specifically designed to offer textual and visual explanations for cyberbullying memes. this represents the first effort to generate mexccm. ## multimodal bully explanations dataset (multibully-ex) to create multibully-ex, we utilize multibully dataset foot_1 @xcite , which includes 3222 bully and 2632 nonbully memes. we selected this dataset because it is the only openly available meme dataset on cyberbullying in a code-mixed setting. our work focuses on jointly extracting textual rationales (words or phrases) and visual masks (image segmentation) to localize salient regions to explain cyberbullying detection tasks. hence we only considered the bully memes for further annotation. ## methodology formulation of mexcm: formally, given a meme (m ) with textual modality @xmath0, t 2 , ..., t n } and visual modality @xmath1, where w is the width and h is the height of an image, we intend ocr clip text encoder clip image encoder image text joint features visual projection textual projection cross modal projection neck positional encoding multi-head self attention add & norm fnn add & norm text-vision fusion add & norm lx visual gate autoregressive @xmath2. > textual explanation bidirectional encoder positional encoding lx multi-head self attention add & norm to learn textual justification along with visual evidence which is defined as follow: (1) textual explanation: textual explanation is the process of extracting pertinent rationales @xmath3, r 2 , .., r k } from the textual modality t of a meme m , which contributes to its classification as a cyberbullying instance. ## results and discussion for a fair comparison with proposed models, we have set up standard baselines such as bart @xcite , t5 @xcite , vg-bart, vg-t5 @xcite , and deeplabv3 @xcite , mobilenetv3 @xcite , fully convolutional networks (fcn) @xcite , unet @xcite for textual and visual explainability, respectively. detailed explanations on baselines, evaluation metrics and training details are given in appendix b). our proposed model can be utilized in a single task (keeping one task-specific private layers) or multitask (keeping both visual and textual private layers) settings. in single task setting, there is no gating mechanism. ## conclusion and future work to encourage more research on explainable meme cyberbullying detection in code-mixed language, we introduced multibully-ex, manually annotated with textual and visual explanations. this work introduces a clip projection-based multimodal