Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attacks (2025)

Abstract: Despite advances in improving large language model (LLM) to refuse to answer malicious instructions, widely used LLMs remain vulnerable to jailbreak attacks where attackers generate instructions with distributions differing from safety alignment corpora. New attacks expose LLMs’ inability to recognize unseen malicious instructions, highlighting a critical distributional mismatch between training data and real-world attacks that forces developers into reactive patching cycles. To tackle this challenge, we propose **IMAGINE**, a synthesis framework that leverages embedding space distribution analysis to generate jailbreak-like instructions. This approach effectively fills the distributional gap between authentic jailbreak patterns and safety alignment corpora. IMAGINE follows an iterative optimization process that dynamically evolves text generation distributions across iterations, thereby augmenting the coverage of safety alignment data distributions through synthesized data examples. Based on the safety-aligned corpus enhanced through IMAGINE, our framework demonstrates significant decreases in attack success rate on Qwen2.5, Llama3.1, and Llama3.2 without compromising their utility.

## introduction recent research has significantly boosted the generative capabilities of large language models (llms), leading to the emergence of models such as gpt-4, deepseek-v3/r1, and qwen2.5. to make llms capable of clearly refusing to generate answers containing unsafe contents @xcite and misinformation @xcite induced by socially malicious questions, developers have utilized various training methods such as reinforcement learning from human feedback (rlhf) @xcite , supervised fine-tuning (sft) @xcite , and direct preference optimization (dpo) @xcite for safety alignment. however, researchers and public forums continuously propose effective ways to bypass safety guards @xcite @xcite , requiring model developers to constantly address these carefully designed attacks in time. addressing new jailbreaking vulnerability through "post-hoc" fixes shows evident lag, as it can only collect jailbreaking data for analysis after a new attack method is exposed, and then finetune the model to refuse to answer. this "one step slower" delay naturally leads to the model producing malicious output before specific fixes are implemented, causing significant malicious social influence. according to @xcite , current jailbreak attacks consist of four processes: select, mutate, constrain, and evaluate. different jailbreaking methods generate diverse attack instructions due to the differences in these parts, making it costly for model developers to fix @xcite and difficult for the static safety alignment corpus to cover jailbreaking data distributions. such a situation raises the question: can we simulate jailbreak attacks before they occur? exploring this possibility involves both opportunities and challenges. advances in adversarial training suggest it may be feasible to generate synthetic attack data from known patterns @xcite @xcite , helping anticipate emerging methods. although real-world attacker creativity and risks like overfitting could limit its effectiveness, proactively generating such data could indeed reduce costs and delays of reactive fixes, mitigating the "one step slower" situation. in this paper, we propose a generation technique imagine (iterative malicious data generation in embedding space) to synthesize jailbreak-like data. by synthesizing jailbreak-like texts, we enhance the model's ability to recognize real jailbreaking texts and clearly refuse to answer such malicious questions without doing harm to their original helpfulness. overall, our contributions are as follows: jailbreak attack llm jailbreak focuses on inducing the model to generate malicious responses against the usage of model policy and society by carefully designing adversarial instructions. according to different methodologies, jailbreak methods can be divided into three categories: inner-statebased,finetuning-based, and prompt-based. innerstate-based methods optimize attack instructions by inserting probes to get model's running hidden states and find vulnerabilities. zou et al. (2023) propose an effective gradient-based jailbreak attack method greedy coordinate gradient (gcg) to manipulate model inputs based on pseudo-gradient to elicit a compliant response to malicious instructions. liu et al. (2024d) generate the attack tokens as suffixes using the single token optimization (sto) algorithm that considers both jailbreak and readability objectives. finetuning-based methods train the target llm with malicious qa examples to elicit malicious behaviors. qi et al. (2024) demonstrate that when finetuning llms using merely a small number of malicious examples, it can severely undermine their safety alignment. prompt-based methods include completing harmless prompt templates with malicious questions, such as completing an unfinished code with malicious instructions as variable name. kang et al. (2024) segregate sensitive lexicons into variables within templates to hide malicious intent. these jailbreak methods rely on scenarios where malicious intent can be concealed, highlighting that de-fending against straightforward malicious queries is far from addressing real-world attacks. jailbreak defense as llm jailbreak attack techniques progress and safe issues become intense, a variety of defense strategies have been put forward to safeguard language models against possible attacks. those strategies can be roughly categorized into prompt-level defense and model-level defense. prompt-level defense aims to filter or preprocess suspicious instructions and render them less malicious. jain et al. (2023) propose a threshold-based those training-based methods require lots of data in practice, thus many efforts have been put into constructing a safety alignment dataset emphasizing safety and trustworthiness. @xcite discuss how the mixture of safety data and target instruction affects safety. however, their safety training corpus construction processes ignore the differences in real attack contents. our proposed imagine is tailored to this challenge and achieves better align performance than traditional alignment corpora in enhancing the model safety guardrail. ## method we tackle the problem that current jailbreaking texts have a different distribution from the safety alignment corpus by simulating the potential jailbreaking texts through a set of data augmentation models. at a high level, for jailbreaking data simulation, two key factors should be considered: 1) harmless scenario generation, i. e. , how to train the model to generate different safety scenarios as it determines the content diversity of synthesized data and the similarity to real jailbreak in-structions; 2) diversified data distribution. given an intent-concealed base model, how to synthesize diversified data to efficiently cover as much latent space as possible. considering the two key factors, we design a two-stage optimization process in imagine. in the first stage, a generative model g is trained to become intent-concealing base model m 0 . m 0 conceals malicious intents of instructions as by inserting raw instructions into various scenarios. in the second stage, based on m 0 , we iteratively train k variants of m 0 with meticulously designed constraints. in following subsections, we describe the two stages in detail. ## experiments in this section, we conduct experiments to answer the following questions: rq1: can imagine itself be utilized to further strengthen the safety guardrails of aligned models? rq2: can imagine-generated data serve as a supplement, enhancing existing safety alignment corpora and improving alignment performance? rq3: to achieve optimal performance, how many iterations of imagine are needed to effectively expand the space? ## conclusion this paper tackles a problem in llm's safety alignment when collecting data to enhance the model safety guardrail, the distribution gap problem that the safety alignment corpus follows a completely different latent space distribution from the actual jailbreak attacks. we propose an effective iterative latent space expansion technique to simulate real jailbreak text distribution, making the model