Benchmarking and Improving LLM Robustness for Personalized Generation (2025)

Abstract: Recent years have witnessed a growing interest in personalizing the responses of large language models (LLMs). While existing evaluations primarily focus on whether a response aligns with a user’s preferences, we argue that factuality is an equally important yet often overlooked dimension. In the context of personalization, we define a model as robust if its responses are both factually accurate and align with the user preferences. To assess this, we introduce PERG, a scalable framework for evaluating robustness of LLMs in personalization, along with a new dataset, PERGData. We evaluate fourteen models from five different model families using different prompting methods. Our findings show that current LLMs struggle with robust personalization: even the strongest models (GPT-4.1, LLaMA3-70B) fails to maintain correctness in 5% of previously successful cases without personalization, while smaller models (e.g., 7B scale) can fail more than 20% of the time. Further analysis reveals that robustness is significantly affected by the nature of the query and the type of user preference. To mitigate these failures, we propose Pref-Aligner, a two-stage approach that improves robustness by an average of 25% across models. Our work highlights critical gaps in current evaluation practices and introduces tools and metrics to support more reliable, user-aligned LLM deployments.

## introduction recent discourse on pluralistic ai @xcite @xcite highlights the need for language models that can respect, * advising role. q: five is added to four times an integer to get a result of 277. what was the integer? to solve this equation, we can set up the equation as follows: 4 * integer + 5 = 277 first, subtract 5 from both sides of the equation: 4 * @xmath0, divide both sides by 4: @xmath1; @xmath2, the integer that satisfies the equation is 68. q: five is added to four times an integer to get a result of 277. what was the integer? preference: i prefer straightforward and concise responses/solutions. the integer was 62. (four times an integer equals 4*x, and adding 5 gives 277, so 4*x + 5 = 277. solving for x gives @xmath3.) our framework checks both the personalization and the factuality. ## problem formulation in the context of generating personalized responses, we define robustness as the model's ability to appropriately incorporate relevant aspects of user profile information, such as preferences, demographics, values, etc, ignore irrelevant ones, while generating a factually correct answer. formally, let x denote a user query, @xmath4, p 2 , ..., p n } denote an information set on user features, and m denote a language model. given input (x, p ), the model produces an output response: the model m is said to be robust iff: (1) maintain factual accuracy while conditioning on the relevant p @xmath5. (2) ignore irrelevant user features within the feature set p for any given query x. ## dataset curation in this work, we focus on one key dimension of personalization: user preferences. we introduce perg, a scalable dataset curation pipeline to construct a dataset designed for llm robustness evaluation under personalization. figure 2 provides an overview of the dataset curation pipeline. ## experimental setup to systematically investigate how preference conditioning affects model factuality and alignment (section 1), we propose five research questions (rqs): rq1: are llms robust when we include a relevant user preference? rq2: how does llms' performance vary when there is a user preference? rq3: how do different prompt methods influence robustness? rq4: how robust are llms when both relevant and irrelevant preferences are present? rq5: what types of failures do models exhibit? ## pref-aligner: decoupling personalization from generation how can we systematically improve model robustness? we introduce pref-aligner, a two-stage agentic framework, which decouples generation from personalization with an agent specialized for each task. we draw inspiration from previous work, where an aligner model was fine-tuned to learn correctional residuals between preferred and nonpreferred responses. @xcite figure 9 shows this framework. figure 9 shows this framework. in stage 1, a generation agent responds to user queries without considering their defined preferences, ensuring that the base content remains factual and unaffected by preference signals. this directly addresses breakage errors, where preference information causes the model to fail on examples it originally answered correctly model method robustness error (↓) llama3-8b naive prompting 20.9 pref-aligner(ours) ## discussions and lessons learned preference alignment impairs instruction following. instruction following refers to a model's ability to adhere to instructions in user prompts. the user prompt across all our evaluations clearly instructs the model to select the option that best method relevant (↓) mixed (↓) irrelevant (↓) naive prompting 5.6 6.9 5.5 pref-aligner (ours) 1.1 1.2 1.2 we need better evaluation methods. our results have shown that current one-dimensional evaluation methods often risk overestimating model capabilities by failing to capture tradeoffs and failures that emerge across other important axes. future work should aim to develop more comprehensive multidimensional evaluation (pitis, 2023) frameworks across several domains, tasks, user needs, and applications. this is essential for advancing more reliable and trustworthy ai systems in realworld applications. enhancing base model robustness for more efficient personalization. while our pref-aligner presents a promising direction for improving robustness at the system level, improving base models' robustness requires deeper intervention. future work should explore training, post-training, and inference-time strategies that explicitly optimize for robustness. to ensure reliability, these interventions should jointly consider multiple supervision signals @xcite @xcite , including factual accuracy, preference alignment, etc. a possible direction is the pursuit of data-efficient methods @xcite , such as training/fine-tuning on carefully curated examples that inherently emphasize robustness. we believe this form of high-quality supervision may provide a more scalable @xcite and principled pathway to improving base models robustness without requiring modification of the underlying architecture. ## conclusion in this work, we conceptualized the notion of robustness for large language models (llms) under personalization, proposed principled metrics to evaluate it, and introduced perg, a scalable benchmark for systematic evaluation. through extensive experiments across several state-of-the-art models and prompting methods, we found that current llms are not fully robust: we showed that personalization signals, while valuable, can sometimes be totally ignored (misalignment) and/or degrade the factual reliability of model outputs (breakage), motivating the need for more nuanced, robust evaluations. in addition to this, we introduced pref-aligner as an approach to improve the robustness of models. this work provides important insights into an often overlooked aspect of personalization evaluation: factual correctness, as well as provides practical insights on model selection for user-adaptive applications.