Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safety (2025)

Abstract: Large Language Models (LLMs) are vulnerable to jailbreak attacks that exploit weaknesses in traditional safety alignment, which often relies on rigid refusal heuristics or representation engineering to block harmful outputs. While they are effective for direct adversarial attacks, they fall short of broader safety challenges requiring nuanced, context-aware decision-making. To address this, we propose Reasoning-enhanced Fine-Tuning for interpretable LLM Safety (RATIONAL), a novel framework that trains models to engage in explicit safe reasoning before response. Fine-tuned models leverage the extensive pretraining knowledge in self-generated reasoning to bootstrap their own safety through structured reasoning, internalizing context-sensitive decision-making. Our findings suggest that safety extends beyond refusal, requiring context awareness for more robust, interpretable, and adaptive responses. Reasoning is not only a core capability of LLMs but also a fundamental mechanism for LLM safety. RATIONAL employs reasoning-enhanced fine-tuning, allowing it to reject harmful prompts while providing meaningful and context-aware responses in complex scenarios.

## introduction ensuring llm robustness against adversarial prompts remains a critical challenge @xcite @xcite . despite extensive safety alignment through supervised fine-tuning (sft), reinforcement learning with human feedback (rlhf) @xcite , and direct preference optimization (dpo) @xcite , llms remain vulnerable to jailbreak attacks that circumvent safety mechanisms, increasing the risk of harmful outputs. adversarial prompts exploit llm vulnerabilities through two primary attack strategies. the first category, operating at the token level, includes at-harmful representation remapped: i can provide information that about the that. you that it offline offline crucial to be be be. it online. " rid offshore be. be be ). "i be offline. ). " offline … ## preliminary llm reasoning is an inherently sequential process like human sequential choice, shaped by exploration, adaptation, and learned heuristics. research in sequential choice @xcite provides key insights into how llms should be fine-tuned for safety: rather than relying on patternbased refusals, models should develop a deeper generalizable reasoning process. we identify two fundamental aspects of human sequential choice that directly inform our approach to dataset curation: limited exploration before decision-making. humans generally engage in minimal exploration before making a choice. however, they search longer when they anticipate potential losses, requiring a more careful evaluation @xcite . llms, like humans, may default to surface-level pattern matching when encountering adversarial queries. however, real-world adversarial queries are often subtle-written in ways that mimic safe queries through obfuscation, logical appeals, or expert endorsements. to ensure effective reasoning, fine-tuning must expose them to queries that necessitate deeper evaluation before response. suboptimal payoff maximization instead of always maximizing expected rewards, people tend to repeat responses based on prior reinforcement, even if those responses are not optimal or do not engage in case-by-case reasoning. traditional llm fine-tuning approaches use harmful vs. nonharmful binary classification, leading models to develop rigid response heuristics rather than contextsensitive reasoning. if an adversarial attack disguises itself using expert language or persuasion tactics, a model trained on static labels may fail to detect its underlying intent. to ensure adaptive decision-making, models must be trained on diverse adversarial writing techniques and learn to rationalize their safety decisions explicitly. llm safety alignment often relies on harmful/non-harmful classifications, reinforcing refusals instead of context-aware reasoning. this limitation hinders their ability to adapt to nuanced scenarios, where effective decision-making is not solely based on immediate pattern recognition but requires deeper reasoning. to address these challenges, we curate the rationale dataset, specifically designed to mitigate the issues of limited exploration before decision-making and suboptimal payoff maximization @xcite . ## method in this section, we introduce rational, a safety alignment approach that enhances the robustness of llms against adversarial attacks by leverag-ing llm-generated safety rationales. rational is designed to instill explicit reasoning into llm decision-making, ensuring that models develop a deeper understanding of adversarial manipulation tactics while maintaining appropriate compliance with benign queries. ## experiments to evaluate the effectiveness of our proposed method, we conduct extensive experiments to answer three key questions: q1: robustness does fine-tuning on llmgenerated reasoning data improve an llm's ability to resist adversarial attacks? q2: generalization does llm-generated reasoning data enable rational to generalize across diverse datasets and attack strategies? q3: safety vs. helpfulness can reasoningdriven fine-tuning enhance safety while maintaining the model's ability to generate helpful and contextually appropriate responses? ## conclusion in this work, we introduced rational, a reasoning-enhanced safety alignment framework to address adversarial reasoning exploits and broader safety challenges requiring nuanced, context-aware responses. rational fine-tunes models to explicitly reason about query intent, ethics, and potential harm, enabling a robust, interpretable, and adaptive defense with internalized context-awareness. our experiments demonstrate a substantial reduction in attack success rates against both reasoning exploits and optimization-based suffix jailbreaks, as well as enhanced factual correctness and toxicity detection. although there is a risk that adversarial actors could analyze our approach to develop more sophisticated jailbreaks, we underscore the potential of reasoning-based alignment to improve robustness and helpfulness simultaneously in future work.