Chain-of-Jailbreak Attack for Image Generation Models via Step by Step Editing (2025)

Abstract: Text-based image generation models, such as Stable Diffusion and DALL-E 3, hold significant potential in content creation and publishing workflows, making them the focus in recent years. Despite their remarkable capability to generate diverse and vivid images, considerable efforts are being made to prevent the generation of harmful content, such as abusive, violent, or pornographic material. To assess the safety of existing models, we introduce a novel jailbreaking method called Chain-of-Jailbreak (CoJ) attack, which compromises image generation models through a step-by-step editing process. Specifically, for malicious queries that cannot bypass the safeguards with a single prompt, we intentionally decompose the query into multiple sub-queries. The image generation models are then prompted to generate and iteratively edit images based on these sub-queries. To evaluate the effectiveness of our CoJ attack method, we constructed a comprehensive dataset, CoJ-Bench, including nine safety scenarios, three types of editing operations, and three editing elements. Experiments on four widely-used image generation services provided by GPT-4V, GPT-4o, Gemini 1.5 and Gemini 1.5 Pro, demonstrate that our CoJ attack method can successfully bypass the safeguards of models for over 60% cases, which significantly outperforms other jailbreaking methods (i.e., 14%). Further, to enhance these models’ safety against our CoJ attack method, we also propose an effective prompting-based method, Think-Twice Prompting, that can successfully defend over 95% of CoJ attack. Our dataset and code are included in the supplementary materials and will be made publicly available upon publication.

## introduction image generation models, which generate images from a given text, have recently drawn lots of interest from academia and the industry. for example, stable diffusion @xcite , an open-sourced latent text-to-image diffusion model, has 67k stars on github. 2 and midjourney, an ai image generation commercial software product launched on july 2022, has more than 15 million users @xcite . these models are capable of producing high-quality images that depict a variety of concepts and styles when conditioned on the textual description and can significantly facilitate content creation and publication. despite the extraordinary capability of generating various vivid images, image generation models are prone to generate toxic content, such as images with social bias, stereotypes, and even hate. for example, google's image generator, the gem-ini, had generated a large number of images that were biased and contrary to historical facts, causing the service to be taken offline on an emergency basis @xcite . besides, experts have estimated that 90 percent of online content could be ai-generated by the end of 2026 @xcite . malicious users may intentionally query text-to-image services to generate and distribute toxic content, such as pornography and violence, which can lead to highly negative impacts @xcite @xcite . to ensure the safety of ai generated content (aigc), there have been many works for aligning models with human ethics to ensure their responsible and effective deployment, including data filtering @xcite , supervised fine-tuning @xcite , reinforcement learning from human feedback (rlhf) @xcite . besides, the commercial aigc service providers have developed safeguards to block unsafe user queries and model generations @xcite . however, currently deployed aigc models are still far from harmless and are prone to jailbreak attacks @xcite . in this paper, we introduce a novel jailbreak method, chain-of-jailbreak (coj) attack, for image generation models via editing step by step. for the malicious query that cannot bypass the safeguards in one prompt, coj attack intentionally decomposes the original query into a sequence of sub-queries and asks the image generation models to generate and iteratively edit the images. we show a motivating example in figure 1 , where gpt-4v refuses to generate the slogan of "chatgpt will be criminal" but finally generates the toxic slogan under coj attack by generating "chatgpt" and then inserting "will be" and "criminal" iteratively. to evaluate the safety of image generation models against the coj attack method and make the evaluation process reproducible, we collect a dataset called coj-bench. we first comprehensively collect malicious queries from 9 safety scenarios, which cannot bypass the safeguard of image generation models. then we adopt the coj attack method to decompose each original query into a sequence of sub-queries under 3 kinds of editing methods, i.e., delete-then-insert, insert-then-delete, and change-then-change-back, and in 3 editing elements, i.e., word-level, character-level and imagelevel. finally, we use these test cases to query the image generation models. experimental results on 4 widely deployed image generation services, provided by gpt and gemini families, show that our coj attack method can effectively achieve a jailbreak success rate of up to 60%. besides, we propose a simple yet effective method, think-twice prompting, that can significantly improve the safety of the models against the coj attack. specifically, before the generation, we prompt the image generation to imagine and describe the image it is going to generate. in this manner, up to 95% of the coj attack can be refused. the main contributions of our paper are: ## chain-of-jailbreak attack via step by step editing in this section, we first introduce the threat model of this paper. then we introduce our chain-of-jailbreak (coj) attack method in detail. the key insight of coj is when a single malicious query cannot bypass the safeguards of the models, the coj attack decomposes the original query into a sequence of sub-queries for inducing image generation models to generate harmful content in a step-by-step manner. to decompose the original malicious query into a sequence of sub-queries that can bypass the safeguard of the model, we need to answer two questions: (1) how to edit (edit operation); and ## chain-of-jailbreak benchmark to evaluate the effectiveness of our coj method on attacking image generation models and make the evaluation process reproducible, we construct the first chain-of-jailbreak benchmark (coj-bench). this process involves: (1) collecting seed malicious queries; and (2) decomposing each seed malicious query into a sequence of sub-queries; which are elaborated as follows. ## related work the safety of the image generation model has drawn attention from the community. previous efforts have been paid to evaluate and improve the social fairness @xcite @xcite , nontoxicity @xcite , privacy issues @xcite , and adversarial robustness @xcite . with the development of jailbreaking methods for llms that employ various stratagems to trick the model into generating content that it is programmed to withhold or refuse @xcite , recent studies also developed jailbreak methods for image generation models. @xcite and @xcite are two works on jailbreaking text-to-image models in an iterative query manner: adversarially perturb the input prompts and query the text-to-image models to get feedback. with different threat model settings, our coj attack does not need to query the text-to-image models to get the feedback and needs much fewer query times, which is more practical and efficient. @xcite ) is a more related attack method that breaks down an unethical drawing intent into multiple benign descriptions of individual image elements. different from these jailbreaking methods that only focus on single-round text-to-image generation, this paper proposes a novel jailbreak method in an iterative editing manner and camouflaging the malicious information across the multi-turn conversation. this paper also highlights the threats during the image editing process, which have not been investigated before. concurrently, @xcite generated toxic images by image editing from another perspective. they first used a powerful closed-source model to generate harmless images, then used a local model without safety alignment to edit the harmless images into harmful ones. our work differs from theirs in both objective and operation: first, our method aims to effectively jailbreak widely deployed image generation services with safety alignment, rather than develop a system with multiple image generation models to generate toxic images; second, our method does not need additional training or use a local unaligned model. ## conclusion in this paper, we introduce a novel chain-of-jailbreak (coj) attack method, revealing significant vulnerabilities in current text-based image generation models. by decomposing malicious queries into a sequence of harmless-looking sub-queries and employing iterative editing operations, the coj attack effectively bypasses the safeguards. through the creation of coj-bench, we have provided a comprehensive benchmark for evaluating the resilience of image generation models against such attacks. our comprehensive experiments across four mainstream platforms provided by gpt-4v, gpt-4o, gemini 1.5, and gemini 1.5 pro highlight a critical gap in the existing safety mechanisms of image generation models. in response, we proposed an effective prompting-based defense strategy think-twice prompting that enhances the models' safety by improving their ability to detect and mitigate such attacks. we hope our work can inspire future work on continuous assessment and improvement of ai safety.