P ic P ersona- TOD : A Dataset for Personalizing Utterance Style in Task-Oriented Dialogue with Image Persona (2025)

Abstract: Task-Oriented Dialogue (TOD) systems are designed to fulfill user requests through natural language interactions, yet existing systems often produce generic, monotonic responses that lack individuality and fail to adapt to users’ personal attributes. To address this, we introduce PicPersona-TOD, a novel dataset that incorporates user images as part of the persona, enabling personalized responses tailored to user-specific factors such as age or emotional context. This is facilitated by first impressions, dialogue policy-guided prompting, and the use of external knowledge to reduce hallucinations. Human evaluations confirm that our dataset enhances user experience, with personalized responses contributing to a more engaging interaction. Additionally, we introduce a new NLG model, Pictor, which not only personalizes responses, but also demonstrates robust performance across unseen domains.

## introduction task-oriented dialogue (tod) is one of the core tasks of dialogue systems, which is designed to fulfill user requests, such as assisting users at customer service desks @xcite and tourist centers @xcite . a tod system is typically divided into the following sub-modules: (1) dialogue state tracking (dst) for tracking the user's requests, (2) policy module for determining system actions such as database (db) searches or dialogue terminations, and (3) natural language generation module (nlg) for converting dialogue policies and db results into natural language responses @xcite . among these components, the responses generated by the nlg module are used to directly interact with the user; therefore, nlg responses significantly influence the overall user experience. although extensive research has improved system responses @xcite ; hosseini-asl et al., 2020; @xcite @xcite , the primary focus has remained on enhancing the accuracy of information conveyance. as a result, the style of the generated response is monotonic and lacks individuality, which hinders the system from forming age-appropriate and emotionally resonant connections with users @xcite . in an effort to improve the naturalness and engagement of system responses, recent approaches have curated new tod datasets that support personalized response styles by incorporating user personas into the dialogue. for instance, @xcite included age and gender information in the dialogues, @xcite integrated emotion, and @xcite personalized system responses by mirroring users' noun and verb phrases. although these approaches provide personalized responses to some degree, the persona modality has been limited to textual information, which lacks details and concurrency about the users they interact with. meanwhile, in the field of open dialogue systems, the integration of user personas has been a long-standing point of interest @xcite @xcite , 2017; @xcite , with recent advancements highlighting the potential of the visionary persona approach @xcite . a visionary persona can capture subtle facial expressions and provide a rich understanding of the user's context, which is similar to how people interpret visual and non-verbal cues in real-life interactions. this approach is particularly valuable in situations where prior textual user profiles are unavailable, such as first-time interactions where no historical information is present. despite these advantages, visionary personas have primarily been utilized in chit-chat @xcite or counseling scenarios @xcite and have not yet been explored in current tod systems. taking these considerations into account, we introduce a new tod dataset that incorporates realistic user images as part of the user persona, enabling personalized system responses in terms of greetings, formality, age sensitivity, and emotional awareness (figure 1 ). in constructing picpersona-tod, we use the user's first impression and dialogue policy-guided prompts, which effectively distill the personalization capabilities of large language model (llm). additionally, we incorporate external knowledge from google maps and wikipedia to reduce hallucinations in personalized responses. furthermore, we implement a meticulous filtering process to ensure stylistic appropriateness, semantic accuracy, and overall naturalness, resulting in a well-refined personalized tod dataset. despite the highly automated process, our dataset demonstrates higher human preference in both user experience ( § 4.1) and personalization ( § 4.2) compared to other datasets and methods. from a label alignment perspective, analysis with dst and policy modules show that picpersona-tod maintains information accuracy. specifically, we present an nlg model called pictor, which demonstrates the ability to generate robustness in personalization, even in unseen domains. in summary, the contributions of this work are threefold: first, we introduce picpersona-tod, a novel tod dataset that integrates user images into personas and provides personalized system responses. second, we present a highly automated dataset generation framework that efficiently creates realistic and personalized datasets. third, through human evaluation, we demonstrate that our dataset enhances user experience through personalization, with benchmark results confirming that personalization does not compromise performance in other critical tasks. ## picpersona-tod dataset in this section, we introduce picpersona-tod, the first personalized tod dataset based on user image persona. to construct a high-quality personalized tod dataset, we hypothesize that it should meet three criteria: (1) the user's utterances should be consistent with their image, (2) the system's responses should be appropriately personalized to the user image, and (3) the synthesized dataset should align with the sub-task labels of tod tasks, such as dst and dialogue policy prediction, while maintaining the information. to address these criteria, our dataset construction pipeline comprises five key stages: (1) user image collection and dialogue dataset extension, (2) user image and utterance alignment, (3) user utterance style transfer, (4) system response personalization, and (5) data filtering. for data construction, we mainly employed gpt-4o @xcite as the primary language model. the overall process is illustrated in figure 2 and the used prompts are in appendix i. ## picpersona-tod analysis in this section, we analyze our dataset in comparison to other datasets( § 3.1), followed by an analysis across key dimensions such as word difficulty, politeness ( § 3.2), and emotions ( § 3.3). ## baselines in this section, we introduce pictor, an nlg baseline trained on the picpersona-tod dataset to generate personalized responses. additionally, we provide models for dst and policy prediction, allowing for comparisons with other datasets. ## conclusion in this paper, we have introduced picpersona-tod, a novel dataset that personalizes system responses based on a user's visual persona in the tod domain. specifically, picpersona-tod incorporates personalized responses in terms of greetings, age, politeness, and emotions. through user satisfaction experiments, we have demonstrated that picpersona-tod enhances personalization while retaining the original information. additionally, we have proposed and analyzed a baseline model, which includes nlg (pictor), dst, and policy prediction. our results show that this method improves personalization without compromising performance in other critical tasks. we believe this work advances research on personalized tod with multimodal user personas, enabling more natural and humanlike interactions.