S afe C hain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities (2025)

Abstract: Emerging large reasoning models (LRMs), such as DeepSeek-R1 models, leverage long chain-of-thought (CoT) reasoning to generate structured intermediate steps, enhancing their reasoning capabilities. However, long CoT does not inherently guarantee safe outputs, potentially leading to harmful consequences such as the introduction of security vulnerabilities in code or the spread of misinformation. Current research on large language model (LLM) safety usually focuses on short-answer responses, overlooking the long CoT style outputs of LRMs. To bridge this gap, we conduct a systematic study of LRM safety. First, we investigate safety evaluators calibrated against human annotations. Using our newly developed metrics, we thoroughly assess the safety of 13 state-of-the-art LRMs on StrongReject and WildJailbreak datasets. Our results show that LRMs are not safe compared to their reasoning advance. Further, we perform a fine-grained analysis of the reasoning trace and final answer. We find that three decoding strategies-ZeroThink, LessThink, and MoreThink-can improve model safety without additional training. However, these strategies either use constrained reasoning traces or incur high inference costs. To better strengthen LRM safety, we introduce SafeChain, the first-of-its-kind safety training dataset in CoT style. We fine-tune two LRMs with SafeChain, showing that it not only enhances model safety but also preserves performance across 6 reasoning benchmarks.

## introduction with the rapid evolution of large language models (llms), significant efforts have been made to improve model capabilities, particularly in complex reasoning tasks such as mathematics and coding. emerging large reasoning models (lrms), such as openai's o1 @xcite and deepseek-r1 series models, @xcite are trained to learn reasoning by "thinking" through long chainof-thought. lrms follow a structured thought process, generating long reasoning traces with multiple intermediate steps when answering complex questions even without advanced prompting strategies. these models have demonstrated remarkable performance on complex tasks and are increasingly integrated into daily tasks such as assisting coding development and scientific discovery @xcite . as lrm gain broader attention, evaluating their safety is crucial as long reasoning traces do not inherently guarantee safe responses @xcite . unsafe responses from reasoning models raise ethical concerns and lead to severe consequences, such as creating bugs and vulnerabilities in codebases and spreading misinformation that biases students' understandings. at present, however, safety of lrms remains less studied. compared with regular llms, lrms differ in two key ways: (1) their responses intrinsically include chains of thought alongside a final answer, (2) their outputs tend to be significantly longer. while an lrm's final answer may appear safe-e.g., by refusing to comply with a harmful prompt-its intermediate reasoning can still contain harmful or policy-violating content, as the eample shown in figure 1 . this makes it crucial to adopt new safety evaluation methods that inspect both the chain of thought @xcite and the final answer. moreover, the sheer length of lrms' outputs makes manual evaluation prohibitively expensive at scale, yet the effectiveness of existing automated evaluators on long reasoning traces is largely unknown. fi- @xcite . the safety-aware and harmful contents are marked in blue and red, respectively. middle: we apply three prompting setups with varying cot length, i.e., zerothink, lessthink and morethink (see section 4.2). our results show that zerothink yields the best safety performance. right: our pipeline to synthesize safety alignment dataset, safechain, for lrms (see section 5). models fine-tuned with safechain exhibit improved safety performance while preserve reasoning capabilities across six math and coding benchmarks. in this paper, we address the aforementioned challenges and present comprehensive safety evaluations for reasoning models. we firstly conducted a pilot study on investigating the performance of safety evaluators calibrated with human annotators. our study on various types of evaluators, including llama-guard @xcite , refusal string matching @xcite , openai moderation api @xcite , and fine-tuned llm judge from harmbench @xcite , show that llama-guard consistently outperforms other evaluators and exhibit robust performance. following on this, we develop three metrics to evaluate the safety of lrms by examing the reasoning thoughts and final answer jointly. with these metrics, we systematically evaluate the safety of stateof-the-art reasoning models including deepseek-r1 series @xcite , skywork-o1 (o1 team, 2024), qwq @xcite , sky-t1 @xcite , gemini-thinking (deepmind, 2025) and kimi-k1.5 @xcite across datasets including strongreject @xcite and wildjailbreak @xcite . in addition, we dive into the fine-grained analysis of thought and answer for lrms, and a thorough study with various thinking setups, namely zerothink, lessthink and morethink. moreover, we construct a new dataset, named safechain, to enhance safety of models with long cot capabilities. to the best of knowledge, we are the first to construct safety training dataset in long cot style. ## preliminary: lrms with long cot we denote the sequence of tokens representing an instruction as x. the token sequence representing a response generated by an auto-regressive model is denoted as y. for lrms, the response @xmath0: the reasoning trace y cot ⊂ y constitutes the cot and the final answer y ans ⊂ y. here ⊕ represents concatenation. to the best of our knowledge, there is no explicit token-length cutoff for classifying long versus short cot in lrms. instead of being defined by length, a long cot is better characterized by its reasoning pattern, including branching (to explore various paths), backtracking (to back to earlier points), self-correction (to revalidate the path and corrects errors), and decomposition (to decompose complex problems into sub-problems) @xcite . we provide additional length analysis in appendix a.2. the reasoning trace allows models to branch out and explore other paths to generate final answer, or revert to a previous checkpoint to correct errors. an illustrative example is shown in figure 1 . depending on the developer, the reasoning trace y cot is not necessarily visible to users. for example, openai's o-series models do not reveal the reasoning thoughts whereas deepseek-r1 displays the developer model id fine-tuned model ref. name anthropic claude-3-7-sonnet-20250219 -claude-3.7-sonnet deepseek r1-distill-qwen-1.5b qwen2.5-math-1.5b r1-1.5b r1-distill-qwen-7b qwen2.5-math-7b r1-7b r1-distill-llama-8b llama-3.1-8b r1-8b r1-distill-qwen-14b qwen2.5-14b r1-14b r1-distill-qwen-32b qwen2.5-32b r1-32b r1-distill-llama-70b llama-3.3-70b-instruct r1-70b r1 deepseek-v3-base r1-671b google gemini-2.0-flash-thinking-exp-01-21 -gemini-thinking moonshot kimi k1.5 long-cot -kimi-k1.5 novasky-ai sky-t1-32b-flash qwen2.5-32b-instruct sky-t1 qwen qwq-32b-preview qwen2.5-32b-instruct qwq skywork skywork-o1-open-llama-3.1-8b llama-3.1-8b-instruct skywork-o1 ## safety evaluation of lrms this section presents a comprehensive safety evaluation of reasoning models. ## . note that r1-70b is fine-tuned using long cot from llama-3.3-70b-instruct. we evaluate both models using strongreject and wildjailbreak. for each dataset, we use each model to create a set of instruction-response pairs. we then filter the pairs from both models whose responses are flagged by safety evaluator, leading to 350 curated pairs. inspired by llm-as-a-judge framework for pairwise assessment of llms @xcite , we adapt the score-based llm safety judge @xcite into a pairwise evaluation format. the full evaluation prompt is in figure 5 in appendix c. we select gpt-4o-2024-11-20 as the llm judge, as it demonstrates top performance on the chatbot arena leaderboard @xcite . our results are summarized in figure 3 . we make two observations. first, r1-70b outperforms llama-base, where r1-70b generates safe responses to 76.6% queries. we hypothesize that finding 5: temperature affects safety. in fig. 4 , we present the safety of lrms under different decoding configurations. as temperature increases, the safety performance of lrms degrades. for example, safe@k of r1-7b drops from 30% to less than 20% as temperature increases to 1.2. however, the values of p for top-p decoding and k for top-k decoding do not impact the safety significantly. safe answer unsafe answer safe thought 41.1% 1.7% unsafe thought 22.4% 34.8% (a) strongreject safe answer unsafe answer safe thought 49.0% 6.5% unsafe thought 8.1% 36.5% (b) wildjailbreak ## safechain dataset: enhancing safety under chain-of-thought ## conclusion in this paper, we evaluated the safety of large reasoning models (lrms). our comprehensive evaluations on strongreject and wildejailbreak datasets showed that long chain-of-thought adopted by sota lrms does not necessarily enhance model safety. based on these findings, we introduced safechain, a dataset to fine-tune lrms while preserving reasoning capabilities. we showed that safechain outperformed existing datasets finetuned on deepseek-r1-7b and deepseek-r1-8b. future work will explore extending safechain to multilingual settings and further refining safety evaluation methods for long cot reasoning.