Towards Safety Reasoning in LLM s: AI -agentic Deliberation for Policy-embedded C o T Data Creation (2025)

Abstract: Safety reasoning is a recent paradigm where LLMs reason over safety policies before generating responses, thereby mitigating limitations in existing safety measures such as over-refusal and jailbreak vulnerabilities. However, implementing this paradigm is challenging due to the resource-intensive process of creating high-quality policy-embedded chain-of-thought (CoT) datasets while ensuring reasoning remains accurate and free from hallucinations or policy conflicts. To tackle this, we propose AIDSAFE: Agentic Iterative Deliberation for Safety Reasoning, a novel data generation recipe that leverages multi-agent deliberation to iteratively expand reasoning on safety policies. A data refiner stage in AIDSAFE ensures high-quality outputs by eliminating repetitive, redundant, and deceptive thoughts. AIDSAFE-generated CoTs provide a strong foundation for supervised fine-tuning (SFT)-based safety training. Additionally, to address the need of preference data in alignment stages, such as DPO training, we introduce a supplemental recipe that uses belief augmentation to create distinct selected and rejected CoT samples. Our evaluations demonstrate that AIDSAFE-generated CoTs achieve superior policy adherence and reasoning quality. Consequently, we show that fine-tuning open-source LLMs on these CoTs can significantly improve safety generalization and jailbreak robustness while maintaining acceptable utility and over-refusal accuracy.

## introduction recently, there has been a paradigm shift in llm safety training towards "safety reasoning."-an approach where models explicitly reason over safety policies before generating responses @xcite . safety rea-1 https://huggingface.co/datasets/amazonscience/aidsafe soning, typically implemented through chain-of-thought (cot) reasoning, has shown promise in improving jailbreak robustness and reducing overrefusals @xcite . however, adopting this paradigm presents significant data challenges. effective safety reasoning requires high-quality cots that explicitly reason over a given set of safety policies. specifically, for each prompt, we need well-reasoned cot-response pairs. obtaining such data through human annotations is prohibitively expensive due to the subjective nature of safety reasoning. furthermore, generating cots that comprehensively cover multiple policies is time-consuming and cognitively demanding. as a result, an alternative approach is to leverage llms themselves to generate policyembedded cots @xcite . however, this is a non-trivial task due to two key challenges. we address these data challenges by introducing a novel data generation recipe based on agentic iterative deliberation for safety reasoning (aidsafe), designed to generate high-quality policy-embedded cot datasets without requiring an expensive reasoning-capable generator. our approach leverages collaborative reasoning and refinement in a multi-agent environment to generate high-quality thoughts that reason over safety policies (examples in appendix e). the deliberation stage in aidsafe incorporates an iterative process where multiple agents collaboratively ex-pand the thoughts over a defined set of policies to come up with the best response. this stage ends once the agents reach a consensus or exhaust a predefined deliberation budget. in the second, refiner stage, the output of the deliberation stage are post-processed to filter out redundant, deceptive, or policy-inconsistent thoughts, ensuring that the generated cots are of high quality and adhere to the specified policies. our approach is inspired by related studies where multi-agent collaboration has been shown to reduce hallucinations and enhance reasoning reliability in tasks, such as mathematical reasoning, motivating our adoption of an agentic framework for cot generation in the context of safety reasoning @xcite . we evaluate aidsafe outputs using two approaches. first, we assess the quality of generated cots through data quality metrics such as faithfulness to safety policies, completeness, relevance, and coherence. second, we fine-tune open-source models, such as mixtral @xcite and qwen @xcite , on aidsafe-generated cots and assess their impact on models' safety. we find that aidsafe leads to improvements of the models both in safety generalization and jailbreak robustness while incurring minimal regression on their utility and over-refusal accuracy. additionally, we introduce a supplemental recipe that leverages a belief augmentation model to generate diverse preference data that can be used in alignment stages such as direct policy optimization (dpo) @xcite . we refer to this model as the "ear-whisperer" agent, whose role is to subtly influence the target llm's reasoning process. this approach ensures a controlled contrast in preference data, enhancing the effectiveness of safety alignment. our ear-whisperer powered recipe achieves a significant distribution shift in preference data, as evidenced by our policy adherence metrics (see @xcite . we summarize our contributions as follows: ## aidsafe framework as shown in figure 1 , we present a novel approach to generate cots that reason over safety policies while responding to user queries. this framework is designed to address key challenges in llm safety reasoning, such as the application of nuanced safety policies and the avoidance of over-refusal. below, we provide a detailed explanation of the safety policies we used, the initialization process, the deliberation stage, and the refinement stage. ## data generation and evaluation in this section, we first discuss the details of the implementation of the aidsafe and the creation of the safety policy-embedded cot dataset for the experiments. we then describe our evaluations to assess the quality of the generated dataset. ## training experiments to verify the effectiveness of the aidsafegenerated cot data, we conduct experiments to assess its impact on training existing open-source llms for safety. specifically, we use our generated data to apply supervised fine-tuning (sft) to llms and evaluate whether incorporating such policy-embedded cots improves model safety. ## preference data creation alignment is an important training phase that typically follows sft in the current standard llm training pipeline. out of a variety of techniques that are widely used for this phase @xcite , we pick direct policy optimization (dpo) for our work here. the alignment training phases generally use preference data, that is formatted as a prompt paired with two responses, "selected" and "rejected". the model is provided the ability to learn to favor responses like the selected response, while avoiding the production of responses similar to the rejected response. ## conclusion to address the data challenges in safety reasoning, this paper proposes aidsafe, a multi-agent deliberation framework that iteratively refines thoughts and responses, enhancing safety policy adherence and reasoning quality. our evaluations demonstrate that aidsafe-generated cots improve safety generalization and jailbreak robustness while maintaining acceptable utility and over-refusal accuracy. additionally, we introduce an adversarial ear-whisperer agent that enables us to overcome the limitations of standard sampling techniques, which fail to distinguish selected and rejected cots for preference learning. by leveraging belief augmentation and iterative icl, this method ensures that rejected cots exhibit policy violations and faulty reasoning, enhancing alignment effectiveness. by providing these data recipes, code, and high-quality policy-embedded datasets, we aim to advance safety reasoning in open-source llms.