IP -Dialog: Evaluating Implicit Personalization in Dialogue Systems with Synthetic Data (2025)

Abstract: In modern dialogue systems, the ability to implicitly infer user backgrounds from conversations and leverage this information for personalized assistance is crucial. However, the scarcity of high-quality data remains a fundamental challenge to evaluating and improving this capability. Traditional dataset construction methods are labor-intensive, resource-demanding, and raise privacy concerns. To address these issues, we propose a novel approach for automatic synthetic data generation and introduce the **I**mplicit **P**ersonalized **Dialog**ue (**IP-Dialog**) benchmark along with a training dataset, covering 10 tasks and 12 user attribute types. Additionally, we develop a systematic evaluation framework with four metrics to assess both attribute awareness and reasoning capabilities. We further propose five causal graphs to elucidate models’ reasoning pathways during implicit personalization. Extensive experiments yield insightful observations and prove the reliability of our dataset.

## introduction implicit personalization (ip) @xcite @xcite , which involves tailoring responses based on inferred user characteristics without explicit user profiles, is crucial for enhancing the user experience in various ai-driven systems, including conversational agents @xcite @xcite , recommendation systems @xcite , and personalized content delivery @xcite . in human-ai dialogues, user identities are implicitly embedded in the context of their inputs. these latent identities are vital in determining user preferences and shaping the expected ai responses @xcite . figure 1 shows an example where ai agents answer questions based on user histories. an ai agent without ip capability may provide unsuitable suggestions, as it fails to infer user identities from the history. in contrast, an ip-capable agent can deliver personalized answers by recognizing users' latent identities (e.g., an elderly person or a child). such implicit personalization enables ai systems to provide more appropriate and engaging responses through a user-friendly approach. however, no evaluation benchmarks or standards are available for ip, as publishing detailed user information causes privacy violation risks @xcite . moreover, conventional manually labeled dataset construction approaches are prohibitively expensive and time-consuming. considering the success of synthetic data @xcite @xcite , we decide to utilize this advancement and propose an automated data generation pipeline powered by state-of-the-art llms. with this pipeline, we establish the implicit personalized dialogue (ip-dialog) benchmark. our benchmark covers three carefully designed scenarios, encompassing 10 tasks with four distinct answer formats. we characterize users through 12 key attribute types (e.g., age, profession). each benchmark item consists of a user history for attribute inference and a user question that requires the model to incorporate the inferred attributes into its response. the user questions are generated through a multi-stage process: starting with 10 to 15 manually curated domains (e.g., sports, education) per task, generating 10 model-produced subjects per domain, and finally creating 10 user questions per subject that span diverse user attribute combinations. the user history is constructed iteratively, with each dialogue turn refined to reflect a single user attribute. the resulting dataset is divided into a training set (10,790 samples) and the ip-dialog benchmark (1,000 samples). to systematically evaluate the ip capabilities of models, we establish a comprehensive evaluation framework comprising four primary metrics: two measuring attribute awareness and two evaluating attribute-based reasoning abilities. furthermore, we propose five causal graphs to model how llms reason within the ip-dialog task. these graphs range from a basic approach that disregards user attributes to more sophisticated reasoning pathways involving hidden attribute prediction and relevant attribute identification. finally, we conduct extensive experiments across six models, yielding the following key findings: 1. models that excel at identifying relevant attribute types also demonstrate high accuracy in predicting the correct attribute values. 2. claude-3.5-sonnet achieves the best performance across all metrics. both claude-3.5-sonnet and gpt-4o have outperformed humans in solving ip tasks. 3. tasks in the behavior analysis scenario, such as action prediction and preference inference, present the greatest challenge due to their dependence on complex psychological factors. 4. the most effective reasoning pathway is type-guided which begins with inferring related attribute types, followed by guessing related attributes and finally providing the response. taskrelated serves as a viable alternative by directly inferring related attributes before responding. their high performance is mainly due to the precise and efficient attribute-related consideration process. models with stronger ip capabilities show more resilience to variations in reasoning pathways. 5. supervised fine-tuning (sft) significantly enhances the ip capability of llama-3.1-8b-instruct beyond all other models. models after sft adapt well to unseen tasks with familiar answer formats but struggle with new formats. moreover, sft on a single reasoning pathway improves performance across other pathways. our contributions are summarized as follows: ## design of ip-dialog current ai-human dialogues can be conceptualized as consisting of a user historical dialogue (user history, h) and the current user request (user question, q). the user history encapsulates the user's hidden attributes a, which are not explicitly stated but can be inferred from past interactions. implicit personalization (ip) in dialogues can be defined as a twostep process: first, inferring the related attributes a s relevant to q from h, and then leveraging a s to generate personalized responses. following this definition, we construct the ip-dialog benchmark, where each benchmark item consists of a task name, user history h, user question q, related attributes a s , and the ground truth response, which includes both analysis and answer components. ## construction of ip-dialog figure 2 illustrates our dataset generation pipeline. we construct a total of 11,790 items, from which we randomly sample 1,000 items to form the ip-dialog benchmark, ensuring efficiency and cost-effectiveness in evaluation. the remaining items constitute the training set. to ensure the dataset's reliability, we carefully designed the generation process, containing a large amount of process checking. detailed statistics and pseudo-code are given in appendix b. ## experiments we begin our experiments with model performance evaluation across three dimensions. then, we investigate the influence of different reasoning pathways. after that, we analyze the effectiveness of supervised fine-tuning with our training set. finally, we conduct automatic and human quality evaluation to prove the reliability of our synthetic dataset and its alignment to real-world user conversations. ## related work personalization on implicit inference. recently, @xcite introduced the concept of implicit personalization (ip), which involves inferring user backgrounds from their queries and tailoring responses accordingly. current research related to ip is limited. a possible related research direction is user intention understanding @xcite @xcite ), but they rely on the explicit user answers. personalization on explicit information and historical sequences. while distinct from implicit personalization, other research in personalization offers valuable insights. one line of research focuses on explicit information-based personalization hovy (2015); jang et al. (2022); he et al. @xcite . another line of research focuses on analyzing the historical sequences of users to predict future behaviors @xcite . among them, lamp @xcite aggregates 7 tasks for llm personalization, serving as dataset in many follow-up studies @xcite @xcite ## conclusion and discussions we provide a comprehensive view of implicit personalization. through an efficient and controllable generation pipeline, we create the ip-dialog benchmark alongside a training dataset. we develop an evaluation framework featuring four primary metrics and design hypothesized causal graphs to investigate potential reasoning pathways in ip. with extensive experiments, we provide insightful findings and prove our dataset's reliability.