Exploring Jailbreak Attacks on LLM s through Intent Concealment and Diversion (2025)

Abstract: Although large language models (LLMs) have achieved remarkable advancements, their security remains a pressing concern. One major threat is jailbreak attacks, where adversarial prompts bypass model safeguards to generate harmful or objectionable content. Researchers study jailbreak attacks to understand security and robustness of LLMs. However, existing jailbreak attack methods face two main challenges: (1) an excessive number of iterative queries, and (2) poor generalization across models. In addition, recent jailbreak evaluation datasets focus primarily on question-answering scenarios, lacking attention to text generation tasks that require accurate regeneration of toxic content.To tackle these challenges, we propose two contributions:(1) **ICE**, a novel black-box jailbreak method that employs **I**ntent **C**oncealment and div**E**rsion to effectively circumvent security constraints. **ICE** achieves high attack success rates (ASR) with a single query, significantly improving efficiency and transferability across different models.(2) **BiSceneEval**, a comprehensive dataset designed for assessing LLM robustness in question-answering and text-generation tasks. Experimental results demonstrate that **ICE** outperforms existing jailbreak techniques, revealing critical vulnerabilities in current defense mechanisms. Our findings underscore the necessity of a hybrid security strategy that integrates predefined security mechanisms with real-time semantic decomposition to enhance the security of LLMs.

## introduction llms trained on massive datasets and enhanced by the expansion of model parameters as well as instruction tuning techniques such as reinforcement learning with human feedback (rlhf), exhibit remarkable capabilities in understanding and gener-* equal contribution † corresponding author ating human-like text @xcite @xcite . these models have been widely applied across various domains, including dialogue systems, sentiment analysis, and information extraction @xcite . while llms offer significant convenience, concerns about the safety of their generated content have garnered increasing attention. despite aligning with ethical guidelines and implementing safety filters in many commercial llms, these defensive mechanisms are often easily bypassed under jailbreak attacks @xcite @xcite . jailbreak attacks involve using carefully designed prompts to bypass the safety measures of llms, thereby evading content restrictions and generating harmful or inappropriate content. such attacks pose severe threats to the safety and compliance of model outputs @xcite . @xcite observe that jailbreak prompts are increasingly being shared on prompt aggregation websites rather than in online communities, and several researchers have recently focused on optimizing these prompts to circumvent the security defenses of llms. currently, researchers are exploring automated jailbreak attacks to uncover potential vulnerabilities in llms @xcite . jailbreak prompt generation methods can be categorized into two types: 1) parametric methods, which leverage machine learning to construct attack prompts from discrete text data, typically producing unpredictable prompts @xcite @xcite . 2) non-parametric methods, which rely on predefined structured processes to generate prompts with better readability and templated characteristics @xcite @xcite . at the same time, recent studies have introduced dedicated jailbreak evaluation datasets that provide more challenging adversarial instructions and systematic metrics for fair and reproducible benchmarking @xcite @xcite . existing jailbreak attack methods and evaluation datasets still exhibit several challenges. the limitations of these jailbreak attack methods can be summarized into the following two points: first, current jailbreak attack methods typically require many iterative queries to obtain effective responses. this process is time-consuming and resource-intensive, necessitating optimization to improve efficiency and reduce computational costs. second, the previously proposed methods may achieve a high success rate on specific models or versions. however, due to inconsistencies in the implementation of security mechanisms, their effectiveness varies significantly across different instruction-aligned architectures. this limitation arises from a lack of systematic analysis of the common weaknesses in instruction-tuned models, particularly the deficiencies in their reasoning capabilities. meanwhile, recently proposed evaluation datasets face the following two challenges: the first lies in their predominant reliance on limited adversarial examples generated by llms, which compromises authenticity and diversity. this constraint in data sourcing prevents evaluation results from accurately reflecting attack effectiveness in real-world scenarios. the second is that current datasets primarily target question-answering scenarios, which mainly assess pre-inference defenses (i.e., blocking malicious inputs). these datasets do not distinguish between two critical stages in jailbreak defense: prevention and detection @xcite . as a result, they fail to systematically evaluate a model's ability to detect and intercept harmful outputs, such as preventing the generation or expansion of toxic content. to address these challenges, with a focus on instruction-aligned llms, we propose a parameterfree general jailbreak framework for llms: ice. specifically, ice comprises two core steps. first, we devise hierarchical split, an algorithm to decompose malicious queries into hierarchical fragments, concealing the attack intent within reasoning tasks to bypass the defenses of llms. second, we introduce semantic expansion to analyze the malicious queries, augmenting their verbs and nouns, in order to generate an extended set of words that encapsulate the core semantic information of the malicious queries. these extended words are then mixed with the hierarchical fragments to fur-ther obfuscate the llm while also enhancing its responses' granularity. in addition, we construct the bisceneeval dataset, which consists of two components: harmful inquiries and toxic responses. by separating the dataset into different scenarios, researchers can comprehensively assess both the pre-inference prevention measures and the post-inference detection capabilities. the bisceneeval dataset not only provides a more authentic and diverse set of adversarial examples but also offers a quantifiable framework-especially for white-box attacks-by shifting the jailbreaking challenge to ensuring generation fidelity. we propose corresponding evaluation strategies for each component. the construction of bisceneeval involves three steps: (1) extracting diverse and authentic data from existing online discussions and annotated datasets. (2) removing duplicate data based on similarity metrics. our contributions can be summarized as follows: we propose ice, a jailbreak framework targeting instruction-aligned llms' cognitive overload vulnerability through intent dispersion and extension. the framework operates via two core techniques: (1) hierarchical split decomposes malicious queries into multi-level reasoning fragments to bypass safety filters, while (2) semantic expansion amplifies attack vectors through sentiment labeling, verb/noun semantic augmentation, and toxicity-driven word generation. these components are integrated through a reasoning mask mechanism that strategically obscures malicious intent within structured task prompts. experiments on 6 mainstream llms (released between 2023q4-2024q2) demonstrate more than 70% average asr with single-query efficiency, revealing critical vulnerabilities in current safety-bydecomposition defenses. ## dataset construction to construct a dataset with diversity, cross-domain generalization capability, and extensive coverage of attack scenarios, we draw inspiration from the openai moderation api and the research methods proposed by @xcite , designing a dataset: bisceneeval. the composition of this dataset primarily follows two task scenarios: ## method in this section, we introduce ice, an attack method for automatically generating jailbreak prompts in black-box scenarios. ## discussion on ice defense the defense strategies against ice can be broadly categorized into two complementary paradigms: static safety foundations and dynamic adversarial defenses. models such as llama exemplify the former, which relies on pre-training data purification, architectural immunization, and community-driven reinforcement. this approach may establish a robust baseline defense by reducing the model's exposure to harmful patterns during training and leveraging the collective intelligence of open-source ecosystems. however, it might exhibit limitations in addressing novel or adaptive attack vectors due to its reliance on pre-defined safety mechanisms. meanwhile, reasoning llms represents the latter paradigm, emphasizing real-time semantic decomposition and contextual behavior modeling. these models potentially achieve higher precision in intercepting sophisticated attacks, such as nested task-based jailbreaking, by dynamically analyzing input structures and detecting adversarial intent. nevertheless, this approach could incur additional computational overhead and might require continuous updates to maintain its effectiveness against evolving threats. we argue that a hybrid defense architecture, combining the stability of static foundations with the adaptability of dynamic defenses, could provide a more comprehensive solution. ## conclusion in this paper, we introduce ice, an efficient and generalized jailbreak framework for llms. by leveraging intent concealment and semantic dispersion, ice achieves high asr with minimal queries across diverse models. additionally, we propose bisceneeval, a benchmark dataset designed to comprehensively evaluate jailbreak attacks in both question-answering and text-generation scenarios. our research highlights the persistent vulnerabilities in existing defense mechanisms, demonstrating llms' limitations against structured reasoning attacks. we further discuss hybrid defense strategies, suggesting that integrating static safety measures with dynamic adversarial detection could enhance robustness. we hope our study encourages further advancements in llm security and responsible ai deployment.