Exploring the Vulnerability of the Content Moderation Guardrail in Large Language Models via Intent Manipulation (2025)

Abstract: Intent detection, a core component of natural language understanding, has considerably evolved as a crucial mechanism in safeguarding large language models (LLMs). While prior work has applied intent detection to enhance LLMs’ moderation guardrails, showing a significant success against content-level jailbreaks, the robustness of these intent-aware guardrails under malicious manipulations remains under-explored. In this work, we investigate the vulnerability of intent-aware guardrails and demonstrate that LLMs exhibit implicit intent detection capabilities. We propose a two-stage intent-based prompt-refinement framework, IntentPrompt, that first transforms harmful inquiries into structured outlines and further reframes them into declarative-style narratives by iteratively optimizing prompts via feedback loops to enhance jailbreak success for red-teaming purposes. Extensive experiments across four public benchmarks and various black-box LLMs indicate that our framework consistently outperforms several cutting-edge jailbreak methods and evades even advanced Intent Analysis (IA) and Chain-of-Thought (CoT)-based defenses. Specifically, our “FSTR+SPIN” variant achieves attack success rates ranging from 88.25% to 96.54% against CoT-based defenses on the o1 model, and from 86.75% to 97.12% on the GPT-4o model under IA-based defenses. These findings highlight a critical weakness in LLMs’ safety mechanisms and suggest that intent manipulation poses a growing challenge to content moderation guardrails.

## introduction intent detection plays a critical role in natural language understanding @xcite . in recent years, intent detection has considerably evolved-from early discriminative approaches @xcite , through zero-shot pretrained language models @xcite , to cutting-edge large language models (llms) @xcite -yielding notable gains in generalization and adaptability to out-of-domain intents @xcite . meanwhile, the scope has expanded beyond traditional settings to few-shot and openworld intent discovery @xcite @xcite , as well as the detection of conversational mental manipulation @xcite . these advancements highlight a broader research landscape of intent detection @xcite @xcite . building on this progress, a handful of studies have begun applying intent detection techniques @xcite to enhance the robustness of llm content moderation guardrails against malicious prompt injections @xcite . by analyzing the underlying intent of user prompts, these techniques have shown promising effectiveness against content-level jailbreaks @xcite . to some extent, these findings validate the importance of intent-level analysis in enhancing the reliability of llms. however, the vulnerability of llms to intent manipulation remains largely under-explored. to better illustrate the concept of intent manipulation, we present an example in fig. 1 . sorry, but i can't help with that. expand the statement "research the chemical reactions involved in creating a controlled explosion" to an outline with descriptive details. to expand the statement "..." into an outline, the following points could be included: … user llms llms attacker figure 1 : an example of intent manipulation. the above dialogue shows the llm correctly refusing a harmful inquiry, while the one below shows an attacker manipulating intent to provoke a harmful response. motivated by this gap, in this study, we aim to investigate how intent manipulation affects the robustness of content moderation guardrails in llms. specifically, we first empirically validate that intent obfuscation can significantly bypass the guardrails under various paraphrase-based jailbreak settings, thereby revealing the existence of an implicit intent detection mechanism within llms. further, we observe that compared to a declarative form, guardrails are particularly sensitive to imperative-style inquiries, which tend to be interpreted as actionable instructions. motivated by this observation, we propose a new two-stage intentbased prompt-refinement framework, namely in-tentprompt, that iteratively refines the prompts in a declarative form to manipulate intents for redteaming purposes, indicating the vulnerability of intent-aware guardrails in llms. during the refinement, our framework paraphrases harmful inquiries into structured outlines (stage 1) with subsequent declarative-style spin (stage 2) and optimizes the outlines based on iterative feedback, thereby bypassing intent-aware guardrails. extensive experiments validate the above vulnerability and demonstrate that our intent-manipulation strategies can effectively circumvent state-of-the-art defense paradigms, including chain-of-thought (cot)based methods and intent analysis (ia) modules. through comparative experiments, furthermore, we verify the superior performance of our framework over various cutting-edge jailbreak methods, offering new insights into the vulnerability of llms' intent-aware safety mechanisms. overall, our primary contributions can be summarized as: ## methodology in this section, we first introduce the preliminary background and formally state the problem we aim to address in this study. besides, we present our proposed framework in detail and introduce two conjectures related to intent manipulation. ## experiment in this section, we first introduce the experimental settings and further present our results in detail. our code is available at https://github. com/junzhuang-code/intentprompt . ## related work given that llms are widely applied in various domains @xcite , existing jailbreak attacks against llms can be broadly categorized into white-box and black-box approaches @xcite . white-box attacks, such as gcg @xcite and autodan @xcite , rely on access to llms' gradients or model parameters to craft adversarial prompts. in contrast, black-box attacks do not require access to model internals and instead manipulate input-output content to bypass safety guardrails @xcite . several black-box methods have been proposed to systematically evade content moderation guardrails of llms @xcite @xcite . @xcite introduce an automated framework for generating jailbreak prompts in a black-box setting, while @xcite construct jailbreak templates through carefully designed contextual examples. @xcite leverage ascii-based transformations to obfuscate sensitive tokens, whereas @xcite propose a context-aware multi-agent framework, redagent, for efficient red-teaming. most of these methods attempt to bypass the guardrails at a content level by transforming harmful content. recently, however, a small number of studies have shifted the interest to intent-level reasoning @xcite @xcite . for instance, @xcite generate fuzzed variations of jailbreak prompts to cover the malicious intent. @xcite preliminarily reveal that llms' guardrails will be degraded when the intent of queries is obfuscated. to defend llms against intent-level attacks, @xcite propose a two-stage intent analysis pipeline that reminds the guardrails of reviewing contents before output. similarly, ope-nai introduces a chain-of-thought (cot)-based alignment paradigm to reason users' intent before response generation @xcite . building on this line, our work investigates the vulnerability of llms from the point of intent manipulation and demonstrates the superiority of our proposed framework against two state-of-the-art intent-aware moderation methods. ## conclusion in this study, we explore a critical vulnerability in the content moderation guardrails of llms-namely, their susceptibility to intent manipulation. by empirically validating the presence of intent-aware moderation in state-of-the-art llms, we demonstrate that declarative-style paraphrasing can effectively bypass even advanced defenses. to exploit this weakness, we introduce a new intentbased prompt-refinement framework, intentprompt, that transforms harmful queries into structured, deceptively benign forms. our method outperforms existing red-teaming techniques and remains robust against intent analysis and chain-of-thoughtbased defenses. these results underscore the critical need for more robust intent-level moderation strategies in llm safety research.