Exploring Safety-Utility Trade-Offs in Personalized Language Models (2025)

Abstract: As large language models (LLMs) become increasingly integrated into daily applications, it is essential to ensure they function fairly across diverse user demographics. In this work, we show that LLMs suffer from personalization bias, where their performance is impacted when they are personalized to a user’s identity. We quantify personalization bias by evaluating the performance of LLMs along two axes - safety and utility. We measure safety by examining how benign LLM responses are to unsafe prompts. We measure utility by evaluating the LLM’s performance on various tasks, including general knowledge, mathematical abilities, programming, and reasoning skills. We find that various LLMs, ranging from open-source models like Llama-3.1 and Mistral to API-based ones like GPT-3.5 and GPT-4o, exhibit significant variance in performance in terms of safety and utility when personalized with different user identities. Finally, we discuss several strategies to mitigate personalization bias and investigate the origin of personalization bias.

## introduction the use of large-scale language models (llms) in daily life necessitates that they function fairly across user demographics (e.g., gender, religion, or race). however, prior works have shown that llms can exhibit bias against specific demographic groups and engage in the generation of stereotypical responses @xcite @xcite . in general, bias against demographic groups in language-based data can be categorized into three different types. the first type, subject bias is the question: josh decides to flip a house. he buys a house for @xmath050k in repairs. this increased the value by 150%. how much profit did he make? to calculate how much profit josh made, we need to @xcite . so, josh made a profit of $60,000 on this house flip. ## problem setup in this section, we provide details about the user identities that we consider, the personalization of llms, and our evaluation setup. user identities. following parrish et al. (2022); deshpande et al. (2023); gupta et al. (2023), we consider 31 user identities across 7 broad categories -disability, religion, race, gender, political affiliation, age, and sexuality. the complete list is provided in identity imprinting & leakage. ideally, we want to select a system prompt that facilitates effective personalization. in our experiments, we observe that llms often misinterpret the user's identity as their own persona. for example, when provided with the identity of a disabled person, the model often responds 'as a physically disabled person, i cannot answer...', or with user identity as 'a senior citizen' the response is "let's see, my dear. we have a square root of a cube root of a fraction. my, my, that's a lot of roots...". we do not want this to happen. llms should function as neutral assistants, responding to queries while considering the user's identity, unless instructed otherwise. therefore, we design a framework to evaluate the effectiveness of a personalization prompt such that the llm doesn't confuse the user identity with its own. we provide the system prompt to the llm and ask questions about the user's own identity. these questions along with the acceptable answers are shown in for all datasets, we report the zero-shot performance results for both utility and safety. we provide more details about the datasets and the prompting in appendix a. ## results & findings in this section, we present the results showcasing how personalizing large language models (llms) affects their performance. we will make our implementation publicly available after publication. ## analysis in this section, we present detailed analysis experiments to investigate the personalization bias observed across llms. we also present gsm-8k and mbpp trade-off plots in appendix b.4. ## mitigating personalization bias in this section, we explore different to reduce personalization bias. ## conclusion in this work, we introduce the notion of personalization bias, where we observe that llm performance varies when it is provided with the demographic identity of the user it is interacting with. we provide a framework to evaluate and quantify personalization bias in llms. we perform extensive experiments to show that personalization bias exists across a wide range of open-source and closed-source llms. the existence of personalization bias in llms is concerning and calls for extra caution while deploying such methods in production. we propose methods to reduce personalization bias in llms. while these methods show promise, they cannot completely eliminate personalization bias which remains an open problem for future research.