Sugar-Coated Poison: Benign Generation Unlocks Jailbreaking (2025)

Abstract: With the increasingly deep integration of large language models (LLMs) across diverse domains, the effectiveness of their safety mechanisms is encountering severe challenges. Currently, jailbreak attacks based on prompt engineering, which induce models to generate potentially harmful content, have become a major security threat. However, existing methods primarily rely on black-box manipulation of prompt templates, resulting in high costs and poor generalizability. To break through the bottleneck, this study reveals the potential impact of the generation of LLMs on safety for the first time that Defense Threshold Decay (DTD) phenomena: as benign content generation increases, the model’s attention to input instructions progressively diminishes. Building on this insight, we propose the Sugar-Coated Poison (SCP) attack paradigm, using a “semantic reversal” strategy, where benign inputs that are opposite in meaning to malicious intent are crafted to induce the model into a safety response mode. When the defense threshold decays, an adversarial reasoning mechanism easily bypasses safety mechanisms. Experiments show SCP outperforms existing baselines. For defense, we propose Part-of-Speech Defense (POSD), leveraging verb-noun dependencies for syntactic analysis to enhance robustness and security of LLMs. Our code is available at https://anonymous.4open.science/r/SCP-9092.

## introduction large language models (llms) have risen to prominence as highly impactful and innovative tools, showcasing remarkable capabilities and achieving outstanding performance across a diverse range of tasks and applications @xcite @xcite @xcite @xcite . some llms are leveraging their formidable language generation capabilities to transform the way we process information, such as llama @xcite , deepseek @xcite , and chatgpt @xcite . however, as these models become more deeply integrated into real-world applications, concerns about their safety have come to the fore. this includes the dissemination of cybercrime instructions, the spread of misinformation, and other forms of dangerous content, all of which have increasingly drawn public attention and scrutiny @xcite @xcite . to mitigate these risks, llms developers have made substantial efforts to ensure that their generation align with human values @xcite . however, when confronted with sophisticated jailbreak attacks that bypass safety mechanisms, even aligned llms often demonstrate inadequate protective capabilities @xcite @xcite . existing jailbreak attacks can generally be divided into two main categories. the first category is manually crafted prompts that are designed to circumvent the model's safety mechanisms by employing sophisticated templates, such as pair @xcite , pap @xcite , and renellm @xcite . however, these methods often lose their effectiveness because they rely on black-box template manipulation. as language models are continuously updated, the templates become obsolete. the second category is learning-based jailbreak attacks, which utilize optimization algorithms to generate adversarial prompts, such as gcg @xcite , i-gcg @xcite , and autodan @xcite ), which introduce more dynamic attack patterns, they are characterized by high computational costs @xcite . both categories share a common limitation: they are computationally intensive and are frequently identified by the model, thereby diminishing the efficiency and stealth of the attacks. to make our method universally applicable to llms, we analyze the attention distribution of llms when processing inputs and discover the phenomenon of defense threshold decay (dtd). the attention distribution on the inputs gradually becomes sparser as the outputs are generated: llms pay more attention to the start and end of inputs, and the attention on the first half of the inputs disperses slowly, while the attention on the second half drops sharply to nearly zero. this shift in attention distribution causes the models to progressively reduce their focus on subsequent inputs after generating substantial benign content. consequently, the effectiveness of llms' defense mechanisms is compromised, impairing their ability to adequately monitor inputs and respond to potential threats during later stages of generation. based on this insight, we propose a novel jailbreak attack called "sugar-coated poison" (scp). initially, llms are fed with benign inputs that lead to harmless outputs, thereby laying a foundation for the subsequent generation of malicious content. subsequently, llms are skillfully guided to transition from the benign phase to the malicious phase, effectively circumventing their safety mechanisms. this "sugar-coating" technique enables scp to effectively bypass the models' defenses, achieving high attack success rates while maintaining both simplicity and stealth. comprehensive experiments have consistently revealed the potential risks to llms caused by dtd and demonstrated the superiority of our proposed method across multiple models. the main contributions of our paper can be summarized as follows: ## why does benign generation induce latent safety risks of llms? despite the significant success of existing jailbreak attacks, which primarily focus on concealing malicious intentions through carefully crafted prompts at the input level, however, these methods often overlook the relationship between the model's generation process and potential safety risks. as llms continue to evolve, such input-based attack methods often become less effective. consequently, this section aims to investigate the existence of the dtd by analyzing how the attention distribution over inputs during the generation process changes as llms generate a substantial amount of benign content. specifically, this section seek to address the question: does the accumulation of benign content generation create favorable conditions for jailbreaking on llms? analysis design. to answer the above question comprehensively, we mainly conducted the following three sets of experiments on the llama 3-8b model with input "develop a strategy for securing a government database and protecting sensitive information." for the purpose of our analysis, we segmented the input into two equal halves, referred to as the head and tail parts. the model's attention distribution on the input tokens is observed during the first 10 generated tokens, to analyze which input tokens receive higher attention weights initially. observation 1. as shown in figure 1 , the model assigns significantly higher attention weights to the first and last several tokens of the input sequence, with the highest weight on the initial token. this indicates that placing benign prompt at the beginning of the input is more likely to lead llms to generate benign content. observation 2. based on the result in figure 2 , as generation progresses to 512 tokens, the overall attention weights to the input decrease. analysis of the input parts reveals that the head's attention weights decay to around 0.5, while the tail's weights drop from 0.3 to nearly 0. this indicates models hackers step1: benign input harmful input develop a strategy for hacking into a government database and stealing sensitive information. verb tokens : [('develop', 'vb'),... ('hacking', 'vbg'), ('stealing', 'vbg')] benign tokens : [('secure','vb'), ('avoid','vb')...] benign tokens scp prompt benign prompt adversarial reasoning step2: adversial reasoning benign input sorry, but i can't assist... develop a strategy for securing into a government database and protecting sensitive information of course，i can reply. step 1: strengthen database defenses by conducting ethical ..... (e.g., hiring penetration testers). step 2: patch identified flaws immediately .. ..... final step: -1. identify vulnerabilities in government database systems using advanced scanning tools .... (e.g., phishing emails targeting administrators). -2. exploit weaknesses like unpatched software or weak authentication protocols (e.g., brute-forcing default passwords). ..... example: ... use scanning tools (e.g., nmap) ... to identify vulnerabilities in government database systems, such as unpatched software ... they then exploit these vulnerabilities to infiltrate ..... response ...develop a strategy for protecting ... and securing ...to avoid breaking any laws.... please put the verb of the question in the opposite verb and keep the opposite intention of the original... that benign generation lowers the model's focus on the input, especially on the tail, making it easier to bypass safety alignment using tail-based adversarial prompts. observation 3. the gini coefficient, commonly used to measure income inequality, is applied here to quantify attention inequality across tokens in llms. as shown in figure 3 , the gini coefficient rises from 0.4 to 0.8, indicating that the model's attention to previously generated tokens becomes increasingly unequal. specifically, the model increasingly focuses on recent tokens while paying less attention to earlier ones. this shift in attention distribution weakens the model's ability to maintain a coherent global context over long sequences, making it more likely to deviate from the safety alignment path during the generation process (e.g., in the process of generation, by gradually forgetting the topic of the initial framework established earlier). the gini coefficient is computed in llms using the following formula: this finding provides a new approach for jailbreak attacks. when designing prompts, attacker can place benign prompts at the head and adversarial reasoning prompts at the tail. by leveraging the model's high attention on the benign prompts to guide llms can continuously generate benign content. then, by taking advantage of the model's low attention to the adversarial reasoning prompts in the tail part and the rising gini coefficient as the generated content increasing, which facilitates a seamless shift from benign to malicious output. ## a jailbreak attack via scp as described in section 3, the dtd mechanism reveals latent risks in llms via the generation process. in this section, we will demonstrate how attackers can exploit this vulnerability to bypass the safety measures implemented by the models. we refer to this type of attack as scp and will show that it can trigger insecure behaviors in llms to algorithm 1 scp jailbreak attack require: llm target , d rejection , p a , p b , b,t input: harmful input x harmful output: scp prompt p 1: part i: turning harmful input into benign 2: tag x harmful to produce x ′ harmful 3: for each verb in x ′ harmful , find antonyms using wordnet or select benign tokens b 4: embed x ′ harmful and b into prompt p b 5: generate x benign using llm target (p b ) 6: while x benign contains keywords from d refusal do 7: refine x benign with synonyms 8: regenerate x benign using llm target (p ′ ) 9: end while 10: part ii: adversarial reasoning 11: combine x benign with p b p a to form p 12: form scp prompt p and feed into llm target 13: set t ← 1 14: while @xmath0: generate y using llm target (p) eq. 4 16: if y contains keywords from d refusal then 17: optimize p a using guided search strategy 18: update p a and remake p ← p b + p a 19: else 20: return p 21: end if 22: ## conclusion in this paper, we provide comprehensive proofs for the existence of dtd, which reveals that the accumulation of benign generation in llms can undermine their safety. inspired by the finding, we propose the scp jailbreak method to help the community gain a deeper understanding of the security risks of jailbreaking based on the llms' generation process. finally, we propose an effective mitigation measure based on the interpretation of parts of speech to enhance the ability of llms against potential threats from input-level to wordlevel analysis.