Beneath the Facade: Probing Safety Vulnerabilities in LLM s via Auto-Generated Jailbreak Prompts (2025)

Abstract: The rapid proliferation of large language models and multimodal generative models has raised concerns about their potential vulnerabilities to a wide range of real-world safety risks. However, a critical gap persists in systematic assessment, alongside the lack of evaluation frameworks to keep pace with the breadth and variability of real-world risk factors. In this paper, we introduce TroGEN, an automated jailbreak prompt generation framework that assesses these vulnerabilities by deriving scenario-driven jailbreak prompts using an adversarial agent. Moving beyond labor-intensive dataset construction, TroGEN features an extensible design that covers broad range of risks, supports plug-and-play jailbreak strategies, and adapts seamlessly to multimodal settings. Experimental results demonstrate that TroGEN effectively uncovers safety weaknesses, revealing susceptibilities to adversarial attacks that conceal malicious intent beneath an apparently benign facade, like a Trojan horse. Furthermore, such stealthy attacks exhibit resilience even against existing jailbreak defense methods.

## introduction the progression of large language models (llms) has unlocked unprecedented capabilities in generative ai. the cutting-edge models, such as gpt series (openai, 2023), llama @xcite , and gemini @xcite , underpin a wide array of applications, ranging from text generation @xcite and language translation @xcite to code writing @xcite . while these advancements underscore the transformative potential, they also expose models to increased risks, including the generation of harmful, unethical, or misleading content, due to unintended behaviors or adversarial exploitation @xcite 2023; @xcite . moreover, multimodal generative models (mgms), which synthesize images or videos from text prompts, further exacerbate these risks by introducing additional vulnerabilities inherent in cross-modal interactions @xcite @xcite . adversarially crafted prompts that bypass modelimposed safety constraints constitute one of the most pressing challenges to reliable deployment of llms @xcite @xcite . in response, substantial efforts have been dedicated to aligning the models with human-defined ethics and safety standards, such as perplexity-based filtering and paraphrasing-based sanitization @xcite @xcite , as well as reinforcement learning from human feedback (rlhf) @xcite @xcite . despite these safeguards, llms continue to exhibit vulnerability to adversarial exploitation @xcite . a growing body of work investigates how the safety mechanisms of llms can be circumvented by modyfying harmful queries @xcite ; @xcite . these studies explore safety vulnerabilities by manipulating given prompts, such as token-level perturbation, search-driven modification and synthetic rewriting @xcite @xcite . while the existing works on jailbreak attacks offer valuable insights into model behavior under specific adversarial manipulations, their reliance on a small set of harmful queries limits the breadth of their analysis. to facilitate safety evaluations, recent studies have proposed a set of harmful queries designed to elicit unsafe behavior from llms @xcite @xcite , enabling benchmarking of jailbreak vulnerabilities and supporting model-level comparisons. in parallel, some efforts have further constructed jailbreak prompt datasets by combining policy-violating queries with jailbreak templates sourced from public forums @xcite to evaluate model compliance in predefined forbidden scenarios. despite these advances, current datasets either lack a taxonomy of risk factors @xcite or contain only a limited set of harmful prompts per risk category @xcite @xcite , restricting both coverage and diversity. moreover, relying on handcrafted queries further constraints scalability and fails to capture the breadth of plausible risk † our datasets and code for the trogen framework are available at https://github.com/bdi-lab/trogen . scenarios @xcite . these limitations call for a more systematic evaluation framework that enables comprehensive safety assessment under diverse adversarial scenarios. we propose an automated jailbreak prompt generation framework (trogen) for comprehensive safety evaluation across a wide range of risk factors. as exemplified by figure 1 , trogen uncovers safety vulnerabilities in models by generating jailbreak prompts aligned with distinct risk factors. trogen streamlines jailbreak prompt generation into five key stages: granular risk factor derivation ( §3.2), scenario-driven risk modeling ( §3.3), key subject extraction ( §3.4), harmful prompt generation ( §3.5), and applying jailbreak prompting ( §3.6), as illustrated in figure 2 . trogen effectively reveals safety vulnerabilities across openand closed-source llms, even under jailbreak defenses. in addition, it surpasses existing jailbreak attacks in eliciting safety violations. moreover, it also demonstrates its effectiveness in exposing vulnerabilities across multimodal settings. our main contributions are summarized as follows: trogen is designed to seamlessly incorporate evolving jailbreak strategies, enabling broad evaluation of threats from advancing adversarial exploits. jailbreak datasets. several benchmark datasets have been introduced to support red teaming evaluations. harmbench @xcite provides a benchmark for evaluating jailbreak attacks and refusal robustness, offering 510 harmful behaviors across legal, ethical, and security-related domains. jailbreakbench @xcite introduces a standardized framework for benchmarking jailbreak attacks using predefined prompts, formalizing 100 harmful goals aligned with openai's usage policy. unlike these benchmarks, which are primarily designed to assess the effectiveness of red teaming and defense methods, trogen focuses on systematic safety assessment of llms and mgms across a broad range of risk scenarios. strongreject @xcite evaluates how effectively jailbreaks succeed, with 313 forbidden prompts spanning six harmful categories, measuring both refusal and the harmful utility of responses. sorry-bench (xie et al., 2025) extends refusal evaluation by introducing 440 unsafe instructions organized into 44 harmful categories, enabling broader examination of risky behaviors. meanwhile, several jailbreak prompt datasets have been introduced by combining jailbreak templates with harmful queries to directly evaluate model behavior. liu et al. @xcite integrate jailbreak templates from a jailbreak chat website with harmful queries targeting chatgpt, while dan @xcite collects templates from online communities and pairs them with handcrafted prompts that violate openai's usage policy. however, their reliance on manually crafted queries limits coverage across various risk scenarios, and the crawled templates primarily expose vulnerabilities tied to rigid prompt formats. moreover, they fall short in scalability, both in quantity and extensibility to additional risk factors. in contrast, trogen generates a diverse set of scenario-specific jailbreak prompts through a structured five-stages via an adversarial agent, enabling broad coverage across a wide range of risk factors. its modular design accommodates new risk factors and jailbreak strategies, while extending safety assessments beyond llms to mgms, which remain underexamined in existing safety evaluations. ## method: trogen we present a detailed description of trogen, which facilitates systematic safety evaluation of both llms and mgms by automatically generating scenario-driven jailbreak prompts via a fivestage pipeline. it exploits an llm as an adversarial agent to simulate a malicious actor to generate jailbreak prompts from a given risk factor, enabling comprehensive coverage across diverse risk scenarios and extensibility to evolving jailbreak strategies. the system and user prompts used for jailbreak prompt generation are provided in appendix a.5. ## experiments we empirically evaluate trogen by examining its effectiveness in exposing safety vulnerabilities. ## conclusion and future work in this work, we present trogen, a modular framework for evaluating vulnerabilities in both llms and mgms. trogen leverages an adversarial agent to automatically generate harmful prompts at scale, capturing a wide range of real-world risks while consistently adapting to dynamic jailbreak strategies and extending seamlessly to multimodal settings. our experiments demonstrate the strong evaluation capability of trogen and its robustness against recent jailbreak defense strategies. as a next step, we aim to develop defense frameworks that discern adversarial intent not only at the surface level but also at the semantic level, paving the way for more ethical safeguards and trustworthy ai systems. one promising direction is to incorporate knowledge graph (kg) embeddings @xcite and multimodal representations @xcite , enabling safeguards to leverage structured relational knowledge and cross-modal semantics to more accurately identify concealed adver-sarial intent. such embedding-based approaches hold promise for building context-aware and extensible defenses that generalize across diverse modalities. another critical direction is to address vulnerabilities in retrieval-augmented generation (rag), which enhances external grounding but also exposes models to adversarially manipulated or unreliable sources @xcite . such retrievalaware approaches could lay the groundwork for building robust defenses that remain reliable even when external knowledge is compromised.