P ersona L ens: A Benchmark for Personalization Evaluation in Conversational AI Assistants (2025)

Abstract: Large language models (LLMs) have advanced conversational AI assistants. However, systematically evaluating how well these assistants apply personalization—adapting to individual user preferences while completing tasks—remains challenging. Existing personalization benchmarks focus on chit-chat, non-conversational tasks, or narrow domains, failing to capture the complexities of personalized task-oriented assistance. To address this, we introduce PersonaLens, a comprehensive benchmark for evaluating personalization in task-oriented AI assistants. Our benchmark features diverse user profiles equipped with rich preferences and interaction histories, along with two specialized LLM-based agents: a user agent that engages in realistic task-oriented dialogues with AI assistants, and a judge agent that employs the LLM-as-a-Judge paradigm to assess personalization, response quality, and task success. Through extensive experiments with current LLM assistants across diverse tasks, we reveal significant variability in their personalization capabilities, providing crucial insights for advancing conversational AI systems.

## introduction the emergence of large language models (llms) has significantly advanced conversational ai assistants, enabling them to engage in sophisticated, multi-turn dialogues and handle complex, task-oriented interactions across diverse domains @xcite @xcite . unlike traditional task-oriented dialogue (tod) systems, which relied on rigid, domain-specific pipelines for slot-filling and intent recognition, llm-based assistants offer greater flexibility and generalization across tasks. this advancement has broadened their applicability, from customer support @xcite and virtual personal assistants @xcite to educational tools * work done during an internship at amazon. here, n is the number of user profiles, m is the number of domains, and t is the total tasks. a binary mask µ is generated to filter out domains which are not of interest of the user, excluding related preferences and past interactions. to use the benchmark, a user profile is selected along with a task and its situational context, ensuring that the task is not from a filtered domain. thus, the total data points are slightly less than n ⇥ t . personalization benchmarks exist, but they have limitations when applied to task-oriented ai assistants. personachat @xcite focuses on chit-chat interactions, lacking the task-oriented structure necessary for assistants where personalization and goal completion are deeply intertwined. lamp @xcite targets personalized language tasks but is not designed for conversational contexts. other datasets such as pens @xcite and cornell-rich @xcite suffer from narrow domain coverage, limiting their applicability to broader assistant scenarios. moreover, they often rely heavily on human-in-the-loop methods @xcite @xcite , which are costly and difficult to scale. to address these challenges, we propose per-sonalens, a benchmark specifically designed to assess personalization in task-oriented conversational ai assistants. unlike existing benchmarks, it incorporates rich contextual information, such as user preferences, past interactions, and situational factors, allowing for a fine-grained assessment of personalization across over 100 tasks spanning 20 domains. our benchmark employs two agents: a user agent (u ) that simulates real users with diverse demographic profiles and rich preferences; and a judge agent (j ) that assesses the personalization capability of ai assistants based on user preferences, historical user-assistant interactions, and current situational context of the user. u interacts with the ai assistant under evaluation, with a particular task and goal, generating a dialogue that is subsequently evaluated by j . personalens enables scalable and automated evaluation of any ai assistant while preserving the complexity and dynamism of real-world assistant-user interactions. through empirical validation, we confirm its reliability and use it to evaluate multiple llm assistants, uncovering key insights into their personalization capabilities. our key contributions are as follows: ## the personalens benchmark personalens is designed to evaluate the personalization capabilities of ai assistants in multi-turn, task-oriented dialogues. unlike existing benchmarks, which often lack depth in contextual and demographic information, our benchmark captures rich user profiles and realistic interaction scenarios across multiple domains. the benchmark comprises three main components: (1) a diverse set of 1,500 user profiles containing demographic information, preferences, and interaction histories, (2) a collection of 111 tasks across 20 domains with associated situational contexts, and (3) two llmpowered agents for simulating users and evaluating personalization quality, respectively. this section details the creation, design, and evaluation of these components. an illustration of our benchmark is provided in figure 1 . ## experimental setup our experiments evaluate the personalization capabilities of various llm assistants, including both open-source and proprietary models, using our proposed benchmark. we assess their ability to provide personalized responses tailored to user preferences, while completing the goal of the user's task. we evaluate 4 model families: claude (claude 3 sonnet, claude 3 haiku, claude 3.5 haiku; anthropic, 2024), llama 3.1 instruct (8b, 70b; @xcite , as well as mistral 7b @xcite and mixtral 8x7b @xcite . for a consistent evaluation setup across all as, we implement u using the claude 3 sonnet and j using the claude 3.5 sonnet foot_3 . the benchmark consists of 1,500 user profiles and 111 tasks across 20 domains, resulting in 122,133 unique user-task scenarios (98,115 for t sd and 24,018 for t md ). for computational feasibility, all experiments reported in this paper are conducted on a randomly sampled subset of 50 user profiles, comprising 3,283 single-domain dialogues and 813 multi-domain dialogues. we use a set of evaluation metrics to assess model performance. task completion (tc) is a binary metric indicating whether a model successfully completes a given task. the task completion rate (tcr) measures the percentage of successfully completed tasks across the benchmark. personalization (p) is a 1-4 scale metric, measuring the extent to which assistant responses in a dialogue are tailored to the user, with 4 being the perfect score of personalization. in addition, we also measure dialogue quality generated by u and a. we measure naturalness, which rates human-likeness on a 1-5 scale, and coherence, which scores response consistency on a 1-5 scale. further details on llm configurations, evaluation prompts, and annotation guidelines are provided in appendix b. ## related work personalization in conversational ai early approaches to personalization in dialogue systems relied on leveraging user personas to generate responses aligned with predefined attributes @xcite . with the advent of llms, dynamic personalization strategies have emerged, including prompt engineering with ## conclusion we introduce a benchmark for evaluating personalization in conversational assistants across diverse domains and user preferences. our benchmark assesses personalization through user-assistant simulation, systematically measuring task completion and personalization quality across diverse task settings. through extensive experiments, we analyze the impact of different prompting strategies, the role of contextual information, and cross-domain personalization dynamics. our findings highlight key challenges in multi-domain personalization, showing that larger models exhibit better adaptability but still struggle with cross-domain consistency. we also demonstrate that interaction history is the most valuable contextual factor for improving personalization, reinforcing the need for dynamic user modeling. future work can explore more advanced user simulation techniques, better retrieval mechanisms for historical interactions, and fine-tuning strategies to enhance personalization.