M eme D etox N et: Balancing Toxicity Reduction and Context Preservation (2025)

Abstract: Toxic memes often spread harmful and offensive content and pose a significant challenge in online environments. In this paper, we present MemeDetoxNet, a robust framework designed to mitigate toxicity in memes by leveraging fine-tuned pre-trained models. Our approach utilizes the interpretability of CLIP (Contrastive Language-Image Pre-Training) to identify toxic elements within the visual and textual components of memes. Our objective is to automatically assess the immorality of toxic memes and transform them into morally acceptable alternatives by employing large language models (LLMs) to replace offensive text and blurring toxic regions in the image. As a result, we proposed MemeDetoxNet that has three main primitives: (1) detection of toxic memes, (2) localizing and highlighting toxic visual and textual attributes, and (3) manipulating the toxic content to create a morally acceptable alternative. Empirical evaluation on several publicly available meme datasets shows a reduction in toxicity by approximately 10-20%. Both qualitative and quantitative analyses further demonstrate MemeDetoxNet’s superior performance in detoxifying memes compared to the other methods. These results underscore MemeDetoxNet’s potential as an effective tool for content moderation on online platforms.

## introduction in the context of the proliferation of social media, the right to freedom of expression has increasingly been prominent in spreading toxic content through various posts @xcite . among these posts, memes have become 1 codes are available at this link: https://github.com/ gitanjali1801/memedetoxnet.git the most popular among social media users, deliberately combining visual and textual elements to convey toxic messages. due to the high volume of such content on social media platforms, manual moderation of toxic memes is becoming increasingly challenging. automated techniques are, therefore, necessary to detect and address such harmful content efficiently. previous research has predominantly focused on developing robust deep-learning models that learn cross-modal interactions from scratch to identify these memes @xcite @xcite @xcite @xcite , often overlooking the critical necessity of applying safety filters that detoxify such memes by moderating both the textual and visual modalities. a straightforward approach to detecting toxic memes involves training a lightweight safety classifier on a specialized dataset @xcite . however, such methods' lack of interpretability and limited generalizability restrict their effectiveness in diverse, open-world scenarios @xcite . to address these challenges, there is a growing need for artificial intelligence (ai) models that can operate effectively in open-world environments by grounding their decisions in ethical principles @xcite . such models must learn how funda-mental facts about the world relate to human values and morality, enabling them to make nuanced judgments beyond simple classification. in this paper, we adopt a proactive approach to ethical and moral considerations by not only detecting toxic memes but also actively detoxifying them. we aim to transform harmful memes on social media into non-offensive alternatives, preventing the spread of such toxic material. to bridge the gap in existing content moderation approaches, we propose memedetoxnet, a novel framework designed to not only detect toxic memes but also convert them into morally acceptable alternatives. memedetoxnet operates through three main primitives: (1) identifying immoral memes using a meme immorality recognizer, (2) localizing and highlighting toxic visual and textual attributes that make them immoral, and (3) manipulating the toxic content to create a morally acceptable alternative (refer figure 1 ). using the fine-tuning of the clip model, memedetoxnet evaluates each meme's visual and textual components. it segments and identifies the harmful attributes in both modalities through interpretability. once identified, the model applies a detoxification process by replacing toxic text with a more neutral alternative using a large language model (llm) and by blurring the image regions responsible for toxicity based on the relevance scores. importantly, this process preserves the original context and intent of the meme while removing its harmful aspects. the primary motivation of our work is the recognition that simply detecting toxic memes is insufficient; a more proactive approach is needed to combat the spread of harmful content online. by significantly reducing toxicity in memes with the help of a modular framework, our work seeks to build a foundation for a more comprehensive and generalized approach to content moderation. ## related work previous research in toxic meme identification primarily relied on fusing unimodal approaches, where either the text or image was analyzed separately. pre-trained models, such as bert, resnet, and vision transformers, were used to extract textual and visual features independently @xcite @xcite , which were then combined for classification @xcite . however, these methods struggled to capture the nuanced inter-play between image and text-an essential aspect for understanding the full meaning conveyed by memes. as research advanced, pre-trained multimodal models, such as clip, blip, visualbert etc., were introduced to address this limitation, integrating both textual and visual information @xcite @xcite @xcite , demonstrating that multimodal approaches are significantly more effective than unimodal methods. despite advancements in multimodal approaches for toxic meme identification, the area of meme detoxification remains largely underexplored. while researchers have made significant progress in detoxifying textual data through content moderation techniques @xcite @xcite , these efforts have predominantly focused on textbased generation models, such as gpt-3 @xcite @xcite and controlled generation systems like ctrl @xcite , which have successfully generated non-toxic alternatives for offensive text. emerging techniques like safe latent diffusion @xcite and concept erasure @xcite , though primarily applied to creative tasks, show promise for content moderation by bridging the gap between textimage relationships in multimodal content @xcite @xcite @xcite . however, these methods have not been adequately extended to memes' unique challenges. detoxifying memes requires a more sophisticated approach that can effectively address the interplay of both modalities, an area that current research has yet to explore fully. while meme categorization has advanced with the development of image-text retrieval algorithms, llm-based methods, contrastive-learning approaches, and scene-graph-based techniques @xcite @xcite @xcite @xcite , the area of meme detoxification remains relatively unexplored. despite progress in toxic meme detection, understanding the underlying causes of toxicity through interpretability techniques remains a challenge. additionally, current toxic detection models often lack transparency, making it difficult to alter specific harmful attributes without compromising the overall context. to address these gaps, we propose memedetoxnet, a multimodal framework to detoxify memes. by leveraging the interpretability of the clip model, we employ gradcam and attention scores to identify specific image regions and text attributes that contribute to toxic interpretations. this level of interpretability, often lacking in current approaches, allows us to implement targeted interventions that replace harmful content while preserving the meme's overall context. ## dataset for our experiments, we utilized a diverse set of publicly available meme datasets, including english-language datasets (mami @xcite , hateful memes @xcite , and memotion2 @xcite and the hindi-english code-mixed mimic dataset @xcite . ## proposed methodology this section illustrates our proposed memedetoxnet framework to detoxify memes. the overall workflow of our proposed memedetoxnet model is shown in figure 2 , and its components are discussed below. ## results analysis this section presents the performance of memedetoxnet compared to baseline and stateof-the-art models. we also analyze its impact on the meme immorality recognizer (mir), trained on multiple morality datasets, to assess how detoxification influences toxicity reduction and immoral content recognition. ## error analysis despite memedetoxnet's strong performance, it encounters challenges in specific scenarios, categorized into the following error types (refer appendix section e for details): ## conclusion in this work, we introduced memedetoxnet, a novel approach for detoxifying offensive memes while preserving their meaning and context. by leveraging a clip-based model fine-tuned on multimodal datasets and employing interpretability methods like gradcam, we demonstrated the model's ability to identify and mitigate toxic elements in both text and images. a balanced focus on knowledge relevance and context preservation achieves effective detoxification across various datasets, ensuring that harmful content is removed without significantly altering the original intent of the meme. our results indicate that memedetoxnet outperforms several sota models in toxicity reduction, making it a promising model for addressing offensive meme content in the real world. in the future, we aim to extend our approach to address other forms of toxic online content and explore additional modalities, such as audio and video, to enhance further the model's ability to moderate diverse types of harmful content.