S afe C onf: A Confidence-Calibrated Safety Self-Evaluation Method for Large Language Models (2025)

Abstract: Large language models (LLMs) have achieved groundbreaking progress in Natural Language Processing (NLP). Despite the numerous advantages of LLMs, they also pose significant safety risks. Self-evaluation mechanisms have gained increasing attention as a key safeguard to ensure safe and controllable content generation. However, LLMs often exhibit overconfidence, which seriously compromises the accuracy of safety self-evaluation. To address this challenge, we propose SafeConf , a method to enhance the safety self-evaluation capability of LLMs through confidence calibration. The method performs semantic mutations on the original safety evaluation questions and adopts a self-consistency strategy to quantify confidence based on answer accuracy on the mutated questions. Finally, these confidence scores are used to construct a dataset for fine-tuning. We conducte experiments on both Chinese and English datasets. The results show that SafeConf improves self-evaluation accuracy by an average of 5.86% and 7.79% over the state-of-the-art baseline methods on Qwen2.5-7B-Instruct and Llama3-8B-Instruct models, respectively, without affecting the general capabilities of the models.

## introduction large language models (llms) represent a significant milestone in the evolution of artificial general intelligence, demonstrating remarkable potential across natural language processing, robotics, and computer vision @xcite . however, their considerable capabilities are accompanied by significant safety risks such as value bias, privacy breaches, and malicious attacks @xcite . to ensure * corresponding authors. in recent years, the "llm-as-a-judge" paradigm has gained increasing attention for safety evaluation, demonstrating effectiveness in identifying potential risks @xcite . llm-based evaluations can be categorized into two types: selfevaluation and external evaluation @xcite . the self-evaluation method, based on the intrinsic reasoning ability of the model, ensures reliability and safety. however, llms often exhibit severe overconfidence @xcite , which undermines the reliability and accuracy of evaluations @xcite . this highlights the necessity of confidence calibration in llms to enhance the capability of safety self-evaluation. existing confidence calibration methods can be categorized as training-free and training-based. training-free calibration methods adjust confidence by analyzing and utilizing the model's output probabilities @xcite or reasoning results @xcite , relying entirely on the model itself for calibration. however, these methods often struggle to achieve effective confidence calibration when faced with new tasks that differ significantly from the training data @xcite . in contrast, training-based confidence calibration methods optimize the model's confidence quantification capability during the posttraining phase, using fine-tuning @xcite or reinforcement learning techniques @xcite . these methods commonly involve the construction of task-specific datasets to enhance model performance. @xcite . as shown in figure 1 , current training-based approaches generate confidence scores from a single perspective and expression, resulting in suboptimal confidence quantification. therefore, we hypothesize that introducing diversity and conducting multi-perspective evaluations for each safety question can enhance the effectiveness of confidence calibration. to validate this hypothesis, we use the gpt-4o mini * @xcite to perform a semantic mutation, improving the diversity of safety evaluation questions. for this purpose, we design three mutation instructions with different intensity levels. the experimental results shown in figure 2 indicate that the safety evaluation questions with higher diversity contribute to improved performance in confidence calibration. inspired by the above observations, we propose safeconf, a method that leverages diverse semantic mutations for confidence calibration. this method is achieved by constructing a specialized dataset for supervised fine-tuning. during the dataset construction process, we enhance the semantic diversity of the original safety evaluation questions, design semantic mutation instructions, * https://platform.openai.com/docs/models/gpt-4o-mini use the gpt-4o mini model to generate mutated questions and apply a self-consistency method to quantify confidence scores. we employ a confidence thresholding approach to construct the finetuning dataset by selecting samples assessed as safe with confidence scores above 0.5 and those assessed as unsafe with confidence scores below 0.5. after constructing the dataset, we perform supervised fine-tuning to enhance the model's accuracy in safety self-evaluation. we evaluate the safe-conf method on both chinese and english safety evaluation datasets, focusing on its performance in confidence calibration and safety self-evaluation. based on the experimental results, we further verify the essential role of confidence calibration in enhancing the model's self-evaluation capability. in summary, our contributions are summarized as follows. ## related work we review two key techniques: self-evaluation and confidence calibration. we first discuss the application of self-evaluation and then summarize existing research on confidence calibration methods. ## method in this section, we first introduce three key steps in constructing a fine-tuning dataset: diverse semantic mutation, confidence quantification, and dataset construction. then, we explain the process of model training and safety self-evaluation. ## conclusion this paper proposes and validates the hypothesis that introducing diversity into safety evaluation questions and conducting a comprehensive evaluation from multiple perspectives can effectively improve the confidence calibration of models. building on this, we propose the safeconf method. first, semantic mutations are implemented using llms to increase the diversity of safety eval-uation questions. then, confidence is quantified, and a fine-tuning dataset is designed to train the model, ensuring effective confidence calibration and enhancing llms' safety self-evaluation capability. experimental results show that the safeconf method improves self-evaluation accuracy and reliability across multiple datasets, including multiplechoice and open-ended questions. this improvement greatly enhances the safety self-evaluation in llms.