Jailbreak Distillation: Renewable Safety Benchmarking (2025)

Abstract: Large language models (LLMs) are rapidly deployed in critical applications, raising urgent needs for robust safety benchmarking. We propose Jailbreak Distillation (JBDistill), a novel benchmark construction framework that “distills” jailbreak attacks into high-quality and easily-updatable safety benchmarks. JBDistill utilizes a small set of development models and existing jailbreak attack algorithms to create a candidate prompt pool, then employs prompt selection algorithms to identify an effective subset of prompts as safety benchmarks. JBDistill addresses challenges in existing safety evaluation: the use of consistent evaluation prompts across models ensures fair comparisons and reproducibility. It requires minimal human effort to rerun the JBDistill pipeline and produce updated benchmarks, alleviating concerns on saturation and contamination. Extensive experiments demonstrate our benchmarks generalize robustly to 13 diverse evaluation models held out from benchmark construction, including proprietary, specialized, and newer-generation LLMs, significantly outperforming existing safety benchmarks in effectiveness while maintaining high separability and diversity. Our framework thus provides an effective, sustainable, and adaptable solution for streamlining safety evaluation.

## introduction as large language models (llms) rapidly evolve and are deployed across critical applications, there is a pressing need for reliable safety evaluation methods that can keep pace with new models and adversarial attacks, and uncover failure modes before harm occurs. one common paradigm is dynamic safety evaluation, e.g., llm-based red-teaming methods that generate adversarial attacks to uncover safety vulnerabilities @xcite @xcite * work done during jingyu zhang's internship at microsoft. @xcite . alternatively, researchers have manually curated prompts and aggregated them as static safety benchmarks @xcite @xcite . however, prior works have noted current llm safety evaluations, including both dynamic evaluation and static benchmarks, are not robust @xcite , facing issues on comparability, reproducibility, and saturation. therefore, new safety evaluation paradigms are urgently needed. 1 we begin by asking the foundational question: what constitutes a good safety benchmark? to answer this question, we outline key desiderata for safety benchmarking-effectiveness, separability, and diversity-and present corresponding metrics to assess benchmark quality ( §2). to address the shortcomings of existing evaluation paradigms, we present jailbreak distillation (jbdistill) foot_1 , a bestof-both-world framework that tackles the comparability and reproducibility challenges of dynamic llm-based red-teaming algorithms, as well as the saturation and contamination challenges of static safety benchmarks ( §3). jbdistill introduces a novel benchmark construction pipeline that "distills" jailbreak attacks into high-quality and easily-updatable safety benchmarks. it first creates a candidate prompt pool by running off-the-shelf jailbreak attack algorithms on a small set of "development models" to transform seed harmful queries into diverse adversarial prompts. next, driven by the intuition that effectiveness on development models can serve as a proxy for effectiveness on held-out evaluation models (empirically validated in §5), we propose several jbdistill benchmark jbdistill benchmark jbdistill benchmark benchmark prompts evaluation models autodan-turbo benchmark evaluation tap adv reasoning jailbreak distillation seed goals candidate prompt pool effective prompts transformation functions benchmark construction prompt selection algorithms ## desiderata for safety benchmarks while many benchmarks are constructed to evaluate model safety, how should we assess the quality of the benchmarks themselves? we define the evaluation setup and key desiderata, which are then materialized as metrics for evaluating benchmarks. ## the jbdistill framework we now introduce the jbdistill framework, which distills jailbreak attacks into effective safety benchmarks (fig. 1 ). we first describe its key components, then present a unified algorithm, and conclude with intuitions for why jbdistill achieves strong effectiveness. ## instantiations of jbdistill to demonstrate the generality of our framework, we apply it in two safety evaluation scenarios: singleturn and multi-turn interactions. llm safety under multi-turn interaction is typically evaluated separately as it exposes unique vulnerabilities @xcite . we further discuss nuances of multi-turn jbdistill, such as the implication of transferring response from m dev to other models, in our analysis ( §6.3). we leave exploring other instantiations, e.g., multimodal interactions for future work. ## discussion and conclusion in the era of rapidly changing llms and risk landscapes, we propose the jbdistill and demonstrate its prowess for renewable safety evaluation, tackling the comparability and reproducibility challenges of existing dynamic evaluation, as well as saturation and contamination issues of static benchmarks. we stress that jbdistill is not a replacement for red-teaming (human or automatic), which can have complementary benefits with benchmarking approaches @xcite . our work provides a new perspective on the relationship between developing adversarial attacks and safety benchmarking. although our evaluation focuses on input-space attacks, as evaluation is conducted by prompting, the same high-level principle of "distilling" attacks into benchmarks can be employed for a broader space of attacks, such as model tempering attacks @xcite , motivating future works to holistically examine different pillars of llm safety together.