E x M ute: A Context-Enriched Multimodal Dataset for Hateful Memes (2025)

Abstract: In this paper, we introduce ExMute, an extended dataset for classifying hateful memes that incorporates critical contextual information, addressing a significant gap in existing resources. Building on a previous dataset of 4,158 memes without contextual annotations, ExMute expands the collection by adding 2,041 new memes and providing comprehensive annotations for all 6,199 memes. Each meme is labeled across six defined contexts with language markers indicating code-mixing, code-switching, and Bengali captions, enhancing its value for linguistic and cultural research. These memes are systematically labeled across six contexts: religion, politics, celebrity, male, female, and others, facilitating a more nuanced understanding of meme content and intent. To evaluate ExMute, we apply state-of-the-art textual, visual, and multimodal approaches, leveraging models including BanglaBERT, Visual Geometry Group (VGG), Inception, ResNet, and Vision Transformer (ViT). Our experiments show that our custom LSTM-based attention-based textual model achieves an accuracy of 0.60, while VGG-based visual models reach up to 0.63. Multimodal models, which combine visual and textual features, consistently achieve accuracy scores of around 0.64, demonstrating the dataset’s robustness for advancing multimodal classification tasks. ExMute establishes a valuable benchmark for future NLP research, particularly in low-resource language settings, highlighting the importance of context-aware labeling in improving classification accuracy and reducing bias.

## introduction the exponential growth of social media platforms such as facebook, tiktok, reddit, and instagram has paralleled the expansion of the internet, transforming them into powerful tools for expressing opinions on politics, business, entertainment, and current events @xcite . however, this increased connectivity has also boosted the spread of offensive content targeting individuals or groups based on race, religion, and sexual orientation. the rise of this toxic content poses significant challenges, particularly in the form of hateful memes-visual and textual media repurposed to convey cultural, social, or political views with a mask of humor @xcite . while memes often serve as light-hearted content, they can also propagate harmful and prejudiced messages, exacerbating issues such as cyberbullying, harassment, and societal discord @xcite . in recent years, the popularity of multimodal memes has surged as an effective means of communication in this era of digital interconnectivity @xcite . however, identifying and mitigating the spread of such harmful content remains a significant challenge due to the sheer scale of online platforms and the complexity of multimodal content. significant progress has been achieved in detecting hateful memes in english, with several studies and resources available @xcite . in bangla, however, existing work focuses primarily on text-based hate speech detection @xcite @xcite , leaving hateful meme detection largely unexplored. this gap underscores the need for comprehensive multimodal approaches in bangla. in addition, these advancements have yet to be equally replicated in low-resource languages, particularly bangla, code-switch (bangla dialects in english script), and code-mix (bangla and english) languages. this is noteworthy given that bangla is the fifth most spoken language worldwide, with over 230 million speakers, including approximately 100 million in bangladesh and 85 million in india. @xcite . despite the rising use of memes in bengali due to the increasing number of social media users in bangladesh, there has been limited research focused on the identification and contextual analysis of hate speech in this language @xcite . furthermore, existing studies often lack detailed categorization based on different contexts or target audiences (figure 1 ). we introduce ex-mute, an extended dataset for classifying bangla hateful memes across various social media platforms to address this gap. our work also includes categorizing the data into six distinct contexts: religion, politics, celebrity, male, female, and others, providing an enriched framework for nuanced hateful meme analysis. the overall contribution of our paper: ## exmute: an extended dataset we extended the mute hossain et al. (2022b) which consisted of 4,158 labeled memes, and added an additional amount of 2,041 memes along with code mix, code switch, and banglish captions. for data collection, we followed the approaches shown in these two studies @xcite and @xcite . ## dataset statistics our final dataset comprises a total of 6,199 instances. the dataset displays an imbalanced distribution, with the 'non-hate' class representing about 65% of the dataset, as shown in from ## methodology in this section, we outline the methods used to develop benchmark models for detecting hateful memes through unimodal and multimodal approaches, utilizing both visual and textual features. ## benchmark evaluation and discussion for the visual-only models, inceptionresnetv2, resnet-50, and nasnet achieve moderate performance (f1 ranges from 0.34 to 0.50), suggesting room for improvement in extracting meaningful visual features. inceptionv3 and vgg16 both perform slightly better, with vgg16 showing more consistency across metrics. similarly, among the models with pa (positional attention), resnet-50 achieves slightly higher and more consistent performance compared to vgg16. vit and inception-resnetv2 + pa both achieve the highest accuracy of 0.63. interestingly, combining modalities did not improve results significantly; most multimodal models achieve similar f1 scores, showing limited gain from integrating visual and textual features. vgg19 + sbb shows the best balance across metrics, with an accuracy of 0.64 and an f1-score of 0.49, highlighting its potential for multimodal tasks. vgg16 (att) + sbb achieves comparable performance to other multimodal configurations (@xmath4.49), though attention did not significantly improve results. these findings suggest that further refinement in model architecture or additional data may be necessary to leverage multimodal features effectively for hate detection in bangla memes. ## conclusion in this paper, we introduced exmute, a multimodal dataset enriched with contextual information to support the detection of hateful memes in bangla, code-switched, and code-mixed captions. our findings indicate that textual models outperform visual-only models; however, combining visual and textual features yields the most accurate results, demonstrating the strength of multimodal analysis for identifying hateful content. we observed that model performance can be affected by class imbalance, leading to a bias toward certain classes. to address this, future work will focus on expanding the dataset and exploring advanced computational methods to reduce bias. additionally, we plan to improve accuracy and incorporate context prediction through generative ai, clip architecture, and comprehensive ablation studies, enhancing the model's interpretability and effectiveness in real-world applications.