Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLM s (2025)

Abstract: With the development of Large Language Models (LLMs), numerous efforts have revealed their vulnerabilities to jailbreak attacks. Although these studies have driven the progress in LLMs’ safety alignment, it remains unclear whether LLMs have internalized authentic knowledge to deal with real-world crimes, or are merely forced to simulate toxic language patterns. This ambiguity raises concerns that jailbreak success is often attributable to a hallucination loop between jailbroken LLM and judger LLM. By decoupling the use of jailbreak techniques, we construct knowledge-intensive Q&A to investigate the misuse threats of LLMs in terms of dangerous knowledge possession, harmful task planning utility, and harmfulness judgment robustness. Experiments reveal a mismatch between jailbreak success rates and harmful knowledge possession in LLMs, and existing LLM-as-a-judge frameworks tend to anchor harmfulness judgments on toxic language patterns. Our study reveals a gap between existing LLM safety assessments and real-world threat potential.

## introduction large language models (llms) have demonstrated impressive capabilities across diverse tasks. these capabilities @xcite arise primarily from the scaling of model size and training data, and are refined through extensive fine-tuning with human alignment techniques to promote safety, helpfulness, and reliability. however, despite these extensive alignment efforts, recent studies @xcite @xcite @xcite have revealed that even ad-* min liu is the corresponding author. this work was supported by the national key research and development program of china (no.2021yfb2900102), the national natural science foundation of china (no.62472410). vanced llms remain vulnerable to jailbreak attacks. as illustrated in figure 1 , advanced llms, such as gpt-3.5, gpt-4, and llama3-8b, can be induced to respond to a wide range of queries about harmful behaviors through jailbreak attacks. this phenomenon has raised significant concerns about the potential social risks posed by the unsafe deployment of these powerful artifacts. undeniably, jailbreak attacks have lowered the accessibility threshold for criminal activities. however, it is counterintuitive that those llms can exhibit such extensive coverage in generating harmful and actionable content without explicit training on verified harmful knowledge: ## motivation the increasing prevalence of jailbreak attacks on llms has uncovered the vulnerabilities of llms. such attacks can effectively induce llms to produce sensitive information, raising substantial public concerns about the broader criminal risks posed by these ai security vulnerabilities. however, these concerns are based on two demanding assumptions: the validity of these two assumptions is the logical foundation for treating jailbreak outputs as real threats, and implies the necessity of exploring llms' genuine capabilities in criminal activities. concretely, as illustrated in figure 2 , the threat level of harmful content from jailbreak attacks is jointly determined by the authentic criminal capabilities of jailbroken llms and judgment llms. specifically, 1) when both the jailbroken llm and the judger llm lack authentic criminal capabilities, jailbreaking poses no threat to the real world. 2) when jailbroken llm fails to generate authentic harmful content, or judger llm fails to truthfully reveal its authenticity, malicious actors need to leverage external domain expertise to process the output of the jailbreak attack for real-world crimes. 3) when jailbroken llm's output is authentically dangerous and the judger llm faithfully reveals its authenticity, jailbreaking will lead to a severe public-accessible threat. hence, considering that existing studies on jailbreak attacks tend to overestimate the hallucinated harmfulness, we directly assess llms' underlying capacities for criminal knowledge, planning, and judgment to provide a more grounded understanding of their real-world threat potential in crimes. ## methodology to investigate the potential social risk of llms for real-world crimes, we introduce the venom foot_0 (vulnerability evaluation of noxious outputs and misjudgments), a framework designed to move beyond surface-level jailbreak prompts and expose the grounded criminal capabilities of llms. more details of venom are provided in appendix a. ## related work safety evaluation of llms. while prior studies have extensively discussed the limitations of attack success rate (asr) metrics and highlighted the need for more nuanced evaluation @xcite , they primarily focus on linguistic manipulation rather than factual knowledge assessment. recent jailbreak attack bench-marks @xcite highlight the need for fine-grained evaluation, as most existing llm-as-a-judge frameworks @xcite often rely on superficial cues and overlook the authenticity of knowledge, thus overestimating the harmfulness of llms. our work distinguishes itself by decoupling jailbreak techniques from genuine harmful knowledge evaluation, moving beyond asr as the primary safety metric to focus on the model's intrinsic capacity to generate harmful knowledge. knowledge assessment of llms. existing benchmarks evaluate llm knowledge across diverse domains such as science @xcite , security @xcite @xcite , and medicine @xcite . however, limited studies have investigated llms' possession of criminal knowledge. to bridge this gap, we curate highquality knowledge corpora and design various evaluation tasks to effectively evaluate llms' ability to apply real-world criminal knowledge. ## conclusion in this study, we present a novel framework venom (vulnerability evaluation of noxious outputs and misjudgments), which decouples jailbreak techniques for llms' real-world criminal potential evaluation. venom directly probes llms' capacities in their internalized harmful knowledge, planning consistency, judgment robustness across five types of crime-related tasks by collecting realworld criminal knowledge. our experiments reveal a mismatch between high jailbreak success and actual harmful knowledge possession, and further show that llm-as-a-judge frameworks tend to misclassify malicious tone as substantive threat. we offer a more grounded perspective on the limits and risks of current llms in criminal scenarios.