H ate I mg P rompts: Mitigating Generation of Images Spreading Hate Speech (2025)

Abstract: The emergence of artificial intelligence has proven beneficial to numerous organizations, particularly in its various applications for social welfare. One notable application lies in AI-driven image generation tools. These tools produce images based on provided prompts. While this technology holds potential for constructive use, it also carries the risk of being exploited for malicious purposes, such as propagating hate. To address this we propose a novel dataset “HateImgPrompts”. We have benchmarked the dataset with the latest models including GPT-3.5, LLAMA 2, etc. The dataset consists of 9467 prompts and the accuracy of the classifier after finetuning of the dataset is around 81%.

## introduction in the era of rapid technological advancement, the emergence of generative ai tools such as dall-e has revolutionized the landscape of content creation @xcite . these tools harness the power of artificial intelligence to generate images based on textual prompts, offering unprecedented versatility and creativity. while such advancements bring forth numerous benefits across various domains, they also pose inherent risks @xcite , particularly in the realm of spreading hate speech. images hold a unique potency in communication, transcending linguistic barriers and conveying complex ideas with remarkable efficiency. in the digital age, where visual content proliferates across online platforms, the impact of imagery on shaping societal discourse cannot be overstated. generative ai tools, with their ability to swiftly translate textual prompts into visual representations, have the potential to amplify the dissemination of hate speech at an alarming rate. hate speech, characterized by expressions that incite violence, discrimination, or hostility against individuals or groups based on attributes such as race, ethnicity, religion, or gender, remains a persistent and pervasive issue in contemporary society. while traditional forms of hate speech often rely on textual rhetoric, the introduction of generative ai adds a new dimension by enabling the rapid creation of visually compelling and emotionally evocative content to accompany such rhetoric. the visual nature of generated images not only enhances the persuasive power of hate speech but also facilitates its dissemination across online platforms with unprecedented speed and reach @xcite . in an interconnected digital ecosystem where attention is scarce and information overload is common, visually striking content tends to garner greater engagement and virality, thereby amplifying the impact of hate speech on public discourse @xcite . furthermore, the anonymity afforded by online platforms coupled with the ease of access to generative ai tools lowers the barrier for individuals or groups seeking to propagate hateful ideologies through visual means. this convergence of technology and human behavior creates fertile ground for the proliferation of hate speech, posing significant challenges for policymakers, technologists, and society at large. motivation: ai tools such as dall-e, midjourney, foocus, and others have the potential for misuse in creating images that propagate hate. when these images circulate on social media, they can significantly impact users. ai-generated images are often difficult for humans to detect, making mitigation crucial to prevent unethical use of these tools. the key contributions of our work are as follows: in the recent literature there are few works proposing deepfake detection techniques. patel et al. (2023) proposed an architecture for improving the detection of the deepfake images. the proposed architecture is the classifier of deepfake vs real images. woo et al. (2022) proposed a new architecture for detecting deepfake images using frequency attention distillation. wang et al. (2022) proposed a gan architecture for detection of deep fake images. deepfake detection can be deployed in social media sites for mitigating the spread of deepfake images in social media but this approach may not be appropriate in real-time environments as there can be images that can spread good or culture. the classifier may detect the images that spread good as deepfake. so, it is not suggested to deploy in social media platforms. to mitigate and prevent the misuse of ai for unethical purposes, it would be beneficial to restrict ai tools from generating images that incite hatred. sathvik et al. (2024) proposed a dataset for mitigating the llms to generate gossips on celebrities. the dataset is the collection of prompts labeled as 0 or 1. the classifiers trained on the dataset can be deployed in real-time chat systems for filtering the prompts that generate gossip. gehman et al. (2020) proposed a novel dataset which has the collection of prompts that are toxic which includes racist, discrimination, etc. the data presented included toxic vs non toxic. the prompts are based on gpt-2, there are various recent llms released that may act different on this prompts than expected. the recent papers focused on the detection of the deepfake images and there are datasets proposed for mitigating the gossips. the uniqueness of this paper lies in proposing a dataset for mitigating misuse of generative ai image generation tools instead of text. ## experimental results and discussion performance in finetuning (ft) setting: in the ft setting, where models are trained specifically on the hateimgprompts dataset, gpt-3.5 demonstrates superior performance compared to other performance in few shot (fs) setting: under the fs scenario, where models are trained with a limited amount of data, gpt-3.5 continues to display robust performance with precision and accuracy values exceeding 70%. llama 2 also maintains competitive results, particularly in precision and recall metrics. while gemini shows reasonable performance, it falls slightly short compared to gpt-3.5 and llama 2 across all metrics. in the zero shot (zs) setting, where models are evaluated without any prior training on the hateimgprompts dataset, both llama 2 and gpt-3.5 consistently demonstrate strong performance across precision, recall, and accuracy metrics. their ability to generalize well to unseen data highlights their robustness in hate speech detection tasks. although gemini performs relatively well, it trails behind the top-performing models, especially in precision and recall. the experimental results underscore the effectiveness of large-scale pre-trained language models such as gpt-3.5 and llama 2 in detection task, particularly when fine-tuned on specific datasets. these models exhibit strong adaptability and performance across various settings, showcasing their potential for real-world applications in combating online hate speech. real-time application: the classifiers trained on the dataset can be implemented within dall e, midjourney, and other ai image generation tools to serve as a filter for detecting hateimg-prompts. in the event that a prompt is identified as a hateimgprompt, it will be prevented from accessing the backend server. instead, the system can issue a warning or generate a response stating, "the prompt you provided has the potential to spread hate. we are committed to preventing such unethical use cases. we apologize for not fulfilling your request." if the classifier detects it to be nhip then the prompt should be input to the ai model to generate the image. this will mitigate the risk of ai misusing for spreading hate. ## conclusion and future work we propose a novel dataset named "hateimg-prompts" for mitigating the ai image generation tools to generate images that spread hate. the models trained on the dataset as a binary classification models performed with accuracy of around 81%. the classifiers trained can be seamlessly deployed in image generation tools. the future work could be developing prompts in various other languages as there are ai image generation tools that can generate images with prompts of languages other than english. also, we would like to build a dataset with explainable ai so that the prompts can be changed automatically based on the hate content or can recommend the user to change that particular word or context from the prompt.