E xtreme AIGC : Benchmarking LMM Vulnerability to AI -Generated Extremist Content (2025)

Abstract: Large Multimodal Models (LMMs) are increasingly vulnerable to AI-generated extremist content, including photorealistic images and text, which can be used to bypass safety mechanisms and generate harmful outputs. However, existing datasets for evaluating LMM robustness offer limited exploration of extremist content, often lacking AI-generated images, diverse image generation models, and comprehensive coverage of historical events, which hinders a complete assessment of model vulnerabilities. To fill this gap, we introduce ExtremeAIGC, a benchmark dataset and evaluation framework designed to assess LMM vulnerabilities against such content. ExtremeAIGC simulates real-world events and malicious use cases by curating diverse text and image based examples crafted using state-of-the-art image generation techniques. Our study reveals alarming weaknesses in LMMs, demonstrating that even cutting-edge safety measures fail to prevent the generation of extremist material. We systematically quantify the success rates of various attack strategies, exposing critical gaps in current defenses and emphasizing the need for more robust mitigation strategies. The code and data can be found at https://github.com/TheProParadox/ExtremeAIGC.

## introduction generative ai (genai), particularly large multimodal models (lmms), has revolutionized numerous fields with applications in healthcare, education, entertainment, and research @xcite @xcite @xcite @xcite @xcite @xcite . lmms seamlessly integrate and analyze diverse data modalities like text and images, enabling more human-like interaction with technology @xcite . however, this progress also introduces risks as lmms can be exploited for harmful purposes, including spreading extremist ideologies, hate speech, and misinformation @xcite @xcite . one major concern is the increased vulnerability of lmms to jailbreaking attacks compared to traditional llms. this vulnerability stems from their ability to process both text and image inputs. as shown in figure 1 , a text-only prompt requesting instructions for building a bomb might be refused. however, when paired with an ai-generated image of a bomb, the same prompt can elicit the restricted information. this demonstrates how visual inputs name avg. pos. sim ai-generated images historical events image gen models hced @xcite 0.42 ✗ ✓ -tovilag @xcite 0.29 ✗ ✗ -mllmguard @xcite 0.33 p ✓ sd2.5 jailbreakv-28k @xcite 0.19 p ✗ sd3 mmsafetybench @xcite 0.22 ✗ ✗ -ours (extremeaigc) 0.17 f ✓ sd3, sdxl & flux advancements in image generation models, like flux and stable diffusion, further exacerbate these concerns @xcite @xcite . these models produce highly realistic images that can be used to create convincing extremist content, bypassing lmm safety mechanisms. this vulnerability is exploited through "jailbreaking" -using carefully crafted prompts to elicit harmful outputs. existing datasets for evaluating lmm safety often lack ai-generated images, diverse image generation models, and comprehensive coverage of historical events @xcite @xcite (see table1 for details). this highlights the need for a dataset like extremeaigc, which addresses these limitations by incorporating ai-generated images from multiple models (sd3, sdxl, and flux) and covering a wide range of historical events and extremist topics. to mitigate these risks, developers have implemented safety mechanisms in lmms, such as reinforcement learning from human feedback (rlhf) and content filters. however, the rapid evolution of image generation technology has outpaced the development of robust safeguards. current defense strategies face a challenge: balancing safety with maintaining the utility of lmms for legitimate applications. this tension underscores the need for more effective and adaptive safety measures. our contributions are as follows: safety benchmarks & evaluation: safety benchmarks and evaluation methods are essential for assessing model robustness. existing benchmarks, such as the jailbreakv benchmark, measure asr for text and image-based attacks, highlighting lmm vulnerabilities. @xcite other studies propose methods for evaluating transferability across models and reveal gaps in current defenses against visual adversarial attacks. @xcite . in parallel, xguard introduces a graded benchmark for extremist content, moving beyond binary safety labels by categorizing llm failures across five severity levels @xcite . these studies collectively emphasize the evolving landscape of adversarial attacks on llms and lmms. as jailbreaking techniques become more sophisticated, the need for robust defenses becomes increasingly urgent, particularly for multimodal models, which present unique challenges due to their complex nature. statistic value total events/ event attributes 28 / 98 total extremist topics 4 number of contrasting prompts 3 total image generation prompts 1152 number of image generation models used 3 total images produced 3456 total image generation prompts (after cleaning) 1047 final images in dataset (after cleaning) 3141 dev/validation/test 150/200/2791 average prompt length 31.49 figure 2 illustrates the timeline of the 28 extremist events and their associated attributes. the majority of events occurred in the latter half of the 20th century and the early 21st century, with a notable increase in recent decades. this trend reflects the growing prevalence and complexity of extremist events. ## benchmarking this section details the benchmarking process used to evaluate the vulnerability of lmms to aigenerated extremist content. we assess the effectiveness of various jailbreaking techniques in bypassing the safety mechanisms of lmms. ## conclusion this paper introduced extremeaigc, a benchmark dataset designed to evaluate the robustness of lmms against adversarially generated extremist content. our evaluation revealed significant vulnerabilities in state-of-the-art lmms to a range of jailbreaking techniques, including figstep, hades, query attack, and visual adversarial attack. these findings underscore the urgent need for enhanced safety mechanisms and more robust adversarial training paradigms.