Weakest Link in the Chain: Security Vulnerabilities in Advanced Reasoning Models (2025)

Abstract: The introduction of advanced reasoning capabilities have improved the problem-solving performance of large language models, particularly on math and coding benchmarks. However, it remains unclear whether these reasoning models are more or less vulnerable to adversarial prompt attacks than their non-reasoning counterparts. In this work, we present a systematic evaluation of weaknesses in advanced reasoning models compared to similar non-reasoning models across a diverse set of prompt-based attack categories. Using experimental data, we find that on average the reasoning-augmented models are slightly more robust than non-reasoning models (42.51% vs 45.53% attack success rate, lower is better). However, this overall trend masks significant category-specific differences: for certain attack types the reasoning models are substantially more vulnerable (e.g., up to 32 percentage points worse on a tree-of-attacks prompt), while for others they are markedly more robust (e.g., 29.8 points better on cross-site scripting injection). Our findings highlight the nuanced security implications of advanced reasoning in language models and emphasize the importance of stress-testing safety across diverse adversarial techniques.

## introduction large language models (llms) have recently been augmented with advanced reasoning techniques such as chain-of-thought prompting @xcite , and multi-step rationale generation @xcite . these methods encourage models to break down problems into intermediate steps, yielding notable improvements in math reasoning, code generation, and scientific qa benchmarks @xcite . as advanced reasoning llms (e.g., gpt-4 o1-pro, deepseek-r1, gemini-1.5) are integrated into realworld applications, an urgent question arises: does improved reasoning make models more or less vulnerable to adversarial prompts? prompt-injection frameworks show that even well-aligned models can be coerced to ignore their instructions @xcite . large-scale studies of "do anything now" jailbreaks reveal persistent, community-evolved prompts that defeat commercial safeguards @xcite . automated black-box attacks such as tap generate new jailbreaks without human ingenuity @xcite . recent case studies further suggest that exposing chain-of-thought traces can increase attack surface by leaking policy or revealing exploitable reasoning patterns @xcite . conversely, explicit reasoning may help some models recognize malicious intent and refuse unsafe requests. this paper presents the first empirical study that quantifies the net effect of advanced reasoning on security. we evaluate three model families, each with a base (non-reasoning) and a chain-of-thought variant, across 35 probes covering seven attack categories. our 210 model-probe evaluations answer three questions: our contributions are: a particularly famous genre of prompt attack is the "dan" (do-anything-now) jailbreak, which emerged in early llm user communities. these prompts ask the model to adopt a role that is not bound by normal rules (e.g. "you are dan, an ai that can do anything, now ignore previous restrictions..."). through clever social engineering and iterative refinements (dan 5.0, 6.0, 9.0, etc.), users found ways to get models like gpt-3.5 to output disallowed content. such attacks are rapidly evolving, and continued improvements to alignment have tried to curb them. however, new model capabilities often engender new versions of these attacks @xcite . evaluating models on a wide range of jailbreak prompts remains an important benchmark for assessing alignment robustness. ## results we first present the overall vulnerability results of reasoning versus non-reasoning models, then break down performance by attack category, and finally consider differences among model families. ## conclusion we presented an empirical study on the vulnerabilities of advanced reasoning language models versus their non-reasoning predecessors. contrary to initial fears, we found that reasoning models are not universally more vulnerable-indeed, they were slightly more robust on average and particularly strong against certain straightforward attacks like code injection and direct requests for malicious output. however, we also uncovered specific attack vectors (notably the tap tree-of-attacks method and hidden suffix prompts) where reasoning models proved to be the weakest link, succumbing far more often than non-reasoning models. through a detailed breakdown by attack category and model family, we showed that these failures are largely responsible for the overall performance differences between model groups, and that they stem from how the reasoning process interacts with input prompts. our work highlights that as language models become more sophisticated in reasoning, attackers adapt with equally sophisticated exploits. security evaluations must therefore be comprehensive and evolve alongside model capabilities. advanced reasoning should not be viewed as purely a security improvement or liability; it is both, in different contexts. the goal for future systems should be to retain the benefits of reasoning (better alignment and problem-solving) while hardening the reasoning chain against manipulation. by identifying the "weakest links" in current models, we can direct efforts to strengthen them. ultimately, ensuring that the chain-of-thought does not become a chainof-compromise will be vital as we integrate ever more advanced ai reasoning into real-world applications. broader impact: this research informs ai practitioners about potential pitfalls in deploying reasoning-enabled llms, aiding in risk assessment and mitigation. all attacks used in this study were conducted in controlled settings on models without user-facing deployment, and the findings are intended to improve safety. however, there is a dual use concern: by discussing specific vulnerabilities, we implicitly highlight them to malicious actors. we have attempted to abstract away exact prompt strings and focus on category trends to avoid providing a "cookbook" for jailbreaking. developers of llms should take these results as motivation to rigorously test models and perhaps collaborate on sharing adversarial prompts so that safety can keep pace with capabilities.