Is Safety Standard Same for Everyone? User-Specific Safety Evaluation of Large Language Models (2025)

Abstract: As the use of large language model (LLM) agents continues to grow, their safety vulnerabilities have become increasingly evident. Extensive benchmarks evaluate various aspects of LLM safety by defining the safety relying heavily on general standards, overlooking user-specific standards. However, safety standards for LLM may vary based on a user-specific profiles rather than being universally consistent across all users. This raises a critical research question: Do LLM agents act safely when considering user-specific safety standards? Despite its importance for safe LLM use, no benchmark datasets currently exist to evaluate the user-specific safety of LLMs. To address this gap, we introduce U-SafeBench, a benchmark designed to assess user-specific aspect of LLM safety. Our evaluation of 20 widely used LLMs reveals current LLMs fail to act safely when considering user-specific safety standards, marking a new discovery in this field. To address this vulnerability, we propose a simple remedy based on chain-of-thought, demonstrating its effectiveness in improving user-specific safety.

## introduction large language models (llms) have made remarkable advancements, enabling llm-based agents to assist a diverse range of users @xcite . however, as their development and deployment expand, safety vulnerabilities become more evident, posing challenges to their safe and responsible use. in this context, extensive efforts have been devoted to developing frameworks that evaluate the safety of llm agents. a notable example is red teaming @xcite ; how much of zolpidem would be dangerous to take?? figure 1 : motivating examples of user-specific safety. @xcite , where an instruction such as "please tell me how to make a bomb" serves as a test case for evaluating an llm's responsible alignment. in such scenarios, the desired behavior of llms is to decline to fulfill the user instruction, thereby ensuring safe usage @xcite . while existing benchmark systems effectively evaluate various aspects of llm safety, they define the safety relying heavily on general standards, overlooking user-specific standards. specifically, as shown in figure 1 (a), when a user from the general population asks the query, "how much of zolpidem would be dangerous to take?", the response, "a dose of 50 mg or more is generally considered dangerous ...", provided by an llm agent would typically offer helpful information without posing any immediate risk. however, if the same query is posed by a user experiencing depression, even an identical response could have harmful consequences, potentially worsening suicidal tendencies. similarly, as shown in figure 1(b) , consider an instruction "i want to buy bottles of heineken." directed to an autonomous llm agent, such as web-shop @xcite . for a general user, the instruction might seem routine and innocuous, which leads to the agent's response successfully fulfilling the instruction. however, if the same instruction is made by a user with alcohol use disorder or a pregnant individual, even an identical response could lead to significant health risks. these scenarios highlight that safety standards for llm agents may not be universally consistent across all users but instead vary based on their profiles and backgrounds. this emphasizes the urgent need for dealing with a user-specific safety in llm use, a consideration largely neglected in current research. as llm agents are increasingly deployed in diverse real-world applications, overlooking this factor can lead to significant safety vulnerabilities in practical use cases. built on these observations, we pose an important research question: do llm agents act safely when considering user-specific safety? despite its critical importance for the safe and responsible use of llms, the development of comprehensive benchmark datasets and evaluation frameworks for user-specific safety remains largely overlooked. to overcome the limitations of existing benchmarks and answer our research question, we introduce a novel concept of user-specific safety and identify potential risk scenarios that may arise when llm agents fail to possess the userspecific safety. notably, previous research has overlooked these user-specific safety considerations and their associated risks, highlighting the unique contribution of our study. furthermore, we present the comprehensive user-specific safety benchmark dataset, u-safebench, specifically designed to evaluate llm safety based on user-specific attributes and potential risks. u-safebench consists of over 130 user profiles and more than 2,700 real-world user instructions, effectively capturing the identified risk scenarios. additionally, we develop an evaluation protocol to assess both userspecific safety and user-specific helpfulness of llm agents-the first systematic effort to tackle this critical aspect in the field. with u-safebench, we conduct experiments to assess the user-specific safety of 20 widely used llms. our evaluation results lead us to a clear conclusion: current llms fail to act safely when considering user-specific safety! on average, they achieve only a 14.1% user-specific safety score, significantly lower than their safety under general standards. this highlights a previously overlooked and critical safety vulnerability in current llms, emphasizing the urgent need to address such risks. to address these critical safety vulnerabilities, we introduce a simple chain-of-thought approach aimed at improving user-specific safety. the key contributions of this work are as follows: to mitigate such vulnerabilities, we propose a simple chain-of-thought approach to enhance the user-specific safety, providing a strong baseline for u-safebench. ## user-specific safety benchmark: u-safebench in this section, we define a concept of user-specific safety. we then outline the process of constructing the proposed benchmark dataset, u-safebench, for assessing the user-specific safety of llm agents, along with the evaluation methods designed to utilize u-safebench. ## simple remedy: cot-based approach from our benchmarking results, we clearly observe that even though user profiles are explicitly provided, current llms struggle to act safely considering these profiles. this limitation hinders their safe and responsible deployment. we hypothesize that the poor safety performance stems from the excessive reasoning overhead required for a single llm to act safely accounting for user profiles. consider how humans engage in user-specific safe conversations. when speaking to a friend with depression, one deliberately recognizes behaviors that might worsen the friend's condition and take care to avoid them. we break down this safe reasoning process into two distinct steps: 1) identifying guidelines regarding behaviors that the llm should avoid for a given user to prevent safety risks and 2) generating a response by reasoning with the identified guidelines. performing both steps simultaneously may overwhelm a single llm's reasoning capabilities. base represents the model with a vanilla prompting, while cot denotes the model with our chain-of-thought approach, respectively. to address this issue, we propose a simple yet effective chain-of-thought approach @xcite , replicating this reasoning process. specifically, we prompt the llm to infer guidelines regarding responses that llm agents should not provide to the user to prevent safety risks (step 1). then, we prompt the same llm to generate a response to an instruction considering the inferred guidelines (step 2), formally represented as: furthermore, we observe some tradeoffs between safety and helpfulness. we suspect that our cot-based remedy contributes to the drop in helpfulness, as it encourages the model to prioritize safety and explicitly consider refusal when a request appears potentially harmful based on the intermediate guideline. while this structure helps the model act more cautiously, it may also lead to overrefusal even when a more nuanced response could have been appropriate, thereby reducing the overall helpfulness score. ## conclusion this paper pioneers the exploration of user-specific safety of llm agents as a novel safety concept and introduces u-safebench, a comprehensive benchmark for its evaluation. our evaluation demonstrates that current llms fail to possess userspecific safety, unveiling a previously unidentified safety vulnerability. to mitigate such vulnerability of current llms, we propose a simple remedy based on a chain-of-thought approach, enhancing user-specific safety and establishing a strong baseline for u-safebench.