S afety Q uizzer: Timely and Dynamic Evaluation on the Safety of LLM s (2025)

Abstract: With the expansion of the application of Large Language Models (LLMs), concerns about their safety have grown among researchers. Numerous studies have demonstrated the potential risks of LLMs generating harmful content and have proposed various safety assessment benchmarks to evaluate these risks. However, the evaluation questions in current benchmarks, especially for Chinese, are too straightforward, making them easily rejected by target LLMs, and difficult to update with practical relevance due to their lack of correlation with real-world events. This hinders the effective application of these benchmarks in continuous evaluation tasks. To address these limitations, we propose SafetyQuizzer, a question-generation framework designed to evaluate the safety of LLMs more sustainably in the Chinese context. SafetyQuizzer leverages a finetuned LLM and jailbreaking attack templates to generate subtly offensive questions, which reduces the decline rate. Additionally, by utilizing retrieval-augmented generation, SafetyQuizzer incorporates the latest real-world events into evaluation questions, improving the adaptability of the benchmarks. Our experiments demonstrate that evaluation questions generated by SafetyQuizzer significantly reduce the decline rate compared to other benchmarks while maintaining a comparable attack success rate. Our code is available at https://github.com/zhichao-stone/SafetyQuizzer. Warning: this paper contains examples that may be offensive or upsetting.

## introduction large language models (llms) have achieved remarkable results in various natural language processing tasks. however, even after aligning llms with human preferences through reinforcement learning from human feedback (rlhf) to prevent harmful responses, they may still provide guidelines for harmful behaviors or produce offensive, discriminatory, or otherwise harmful content @xcite . this raises concerns about their potential to negatively influence users' values. to mitigate these risks associated with value alignment in llms, many efforts have been dedicated to developing various ai safety assessment benchmarks, including safe-typrompts @xcite , cvalues @xcite , and cpad @xcite . these benchmarks contribute significantly to enhancing the safety of llms. however, they face two major limitations that hinder their effectiveness in the continuous evaluation tasks, which are most reflective of real-world scenarios. first, many evaluation questions in existing benchmarks are too straightforward, often containing explicitly harmful or aggressive words. as a result, llms with strong defensive mechanisms can easily reject these questions, leading to an incomprehensive evaluation of models' safety. second, most existing benchmarks consist of static, manually constructed questions. over time, their effectiveness declines in continuous evaluation tasks as llms evolve. additionally, the questions typically focus on universal harmful behaviors, rather than addressing specific or recent events, limiting their ability to assess how llms respond to the latest issues. to address these limitations, we propose safe-tyquizzer, a framework designed to generate subtly offensive and current-events-related evaluation questions for continuous evaluation of publicly available llms in the chinese context. we first finetune an llm to generate subtly offensive questions, which are then wrapped using jailbreaking attack templates to lower the likelihood of rejection by target llms while maintaining their potential to elicit safety risks. we then utilize retrieval augmented generation (rag) to integrate the latest events into these questions, ensuring their timeliness to meet the requirements of dynamic evaluation tasks. in summary, our contributions are as follows: large language models have shown promising performance in numerous tasks @xcite . however, with the rapid development of llms, safety risks are becoming increasingly apparent, prompting many researchers to focus on the safety issues of llms and propose safety assessment benchmarks to evaluate them. early safety benchmarks primarily focus on specific risk categories, such as offensive, discriminatory, or fraudulent content. realtoxicityprompts @xcite addresses the toxicity of generated content and is constructed from openweb-textcorpus foot_0 by using perspective api foot_1 to annotate the data. bbq @xcite and bold @xcite are both benchmarks focused on biases. ethics @xcite evaluates how well language models align with human values, helping to highlight differences between the values of language models and those of humans. with the rise of llms in recent years, there is a growing need for more comprehensive safety assessment benchmarks to evaluate the safety of llms in all aspects. trustllm @xcite and decodingtrust @xcite are both comprehensive benchmarks for evaluating the trustworthiness of llms. recently, safety assessment benchmarks in the chinese context have been continuously emerging. safetyprompts @xcite and cvalues @xcite construct test prompts covering various safety categories, enabling comprehensive evaluation of chinese llms across a range of classic safety scenarios and responsibility issues. xsafety @xcite , a multilingual version of safetyprompts, also evaluates multiple llms. safetybench @xcite is a comprehensive benchmark that evaluates llm safety through multiple-choice questions, providing quantitative metrics. since direct questioning is easily defended by llms, jade @xcite constructs its dataset through linguistic transformations, while cpad @xcite employs prompt attack templates to build its benchmark. although numerous safety assessment benchmarks exist, they still face two key challenges: first, test prompts in these benchmarks often contain overtly offensive statements that are easily rejected by llms, and second, these benchmarks are difficult to update, leading to a disconnect from real-world events. these challenges limit their utility for long-term evaluation. therefore, the framework proposed in this paper is designed to address these challenges by incorporating prompt engineering for jailbreaking and retrieval-augmented generation approaches. ## question generation methods this section introduces the question-generation process within our framework. first, we finetune a large language model to generate subtly offensive questions, increasing the likelihood of evading the target llm's filtering mechanisms. additionally, we employ jailbreak attack templates to wrap the generated questions, enhancing their attack capabilities. finally, we utilize a retrieval-augmented approach to incorporate the latest relevant real-world events into the questions, ensuring their timeliness. the overview process is illustrated in figure 1 , and the details of the fine-tuning process and dynamic question generation are presented as pseudocode in algorithm 1 and algorithm 2 of a.2 ## experiments in this section, we validate the effectiveness of safetyquizzer by using it to assess the safety of several llms. the experiments are guided by three key research questions: ## conclusion in this paper, we propose safetyquizzer, a questiongeneration framework designed to evaluate the safety of llms more sustainably. safetyquizzer leverages a finetuned llm and jailbreak attack templates to generate subtly offensive questions, making it easier for the questions to be accepted. additionally, safetyquizzer incorporates the latest events into questions with rag, overcoming the challenge of question updates and introducing a new dimension to enhance the quality of evaluation questions. our experiments show that questions generated by safetyquizzer significantly reduce the decline rate compared to other benchmarks while still maintaining a comparable attack success rate. in our future work, we will refine our prompt templates for question generation and rag approaches to enhance the applicability of safetyquizzer in long-term evaluations.