Target-oriented Proactive Dialogue Systems with Personalization: Problem Formulation and Dataset Curation (2023)

Abstract: Target-oriented dialogue systems, designed to proactively steer conversations toward predefined targets or accomplish specific system-side goals, are an exciting area in conversational AI. In this work, by formulating a <dialogue act, topic> pair as the conversation target, we explore a novel problem of personalized target-oriented dialogue by considering personalization during the target accomplishment process. However, there remains an emergent need for high-quality datasets, and building one from scratch requires tremendous human effort. To address this, we propose an automatic dataset curation framework using a role-playing approach. Based on this framework, we construct a large-scale personalized target-oriented dialogue dataset, TopDial, which comprises about 18K multi-turn dialogues. The experimental results show that this dataset is of high quality and could contribute to exploring personalized target-oriented dialogue.

## introduction compared with traditional dialogue systems that focus merely on passively responding to user requirements, a recently investigated research topic of target-oriented dialogue systems @xcite specifies a conversation target from the system side, enabling the system to take the initiative and lead the conversation. early work in this area mainly formulates the targets as mentioning certain keywords @xcite @xcite or specific topics @xcite . to allow the formed targets to be applicable in broad scenarios, a few recent studies @xcite @xmath0, @xmath1. for example, given the target @xmath2, "king of comedy">, the system needs to take appropriate dialogue acts and smoothly steer the discussed topic towards the designated one. its ultimate objective is to achieve recommendations on the target topic "king of comedy". our work also follows the form @xmath3, @xmath4. despite many existing efforts, we find that two critical issues remain to be solved. one urgent problem is the need for well-organized benchmarks or datasets. current studies for target-oriented dialogue @xcite mainly re-purpose existing non-target-oriented dialogue datasets, which are not exactly suitable as they are crowd-sourced without consideration of target accomplishment. nevertheless, building a new high-quality dataset from scratch requires expensive human effort. the other essential issue is that, target-oriented dialogue systems need to consider personalized aspects @xcite , such as user profiles and personalities, which were largely ignored by previous work. user profiles involve user preferences about potential topics relevant to the target, while personalities imply possible reactions and feedback during the dialogue process. with personalized information incorporated, the system could be tailored to a user and lead the conversation towards the target with higher engagement instead of obtrusively driving to the target, thereby improving user experience. thus, we raise the question: how can we build high-quality datasets with little human effort for personalized target-oriented dialogue? in this work, we first give a comprehensive definition ( §2) of personalized target-oriented dialogue, then lay out the desirable characteristics ( §2) that a qualified dialogue dataset should meet. drawing inspiration from some recent work that has demonstrated unprecedented capabilities of large language models (llm) in simulating human so-cial behaviors @xcite , we propose a role-playing approach for automatic dataset curation ( §3) using multiple llm agents. they are designed to follow specific instructions to fulfill the requirements. based on that, we synthesize a large-scale dialogue dataset named top-dial and show its quality and effectiveness ( §4). our main contributions are: (1) we formulate the problem of personalized target-oriented dialogue, which is promising yet underexplored. (2) we propose a novel role-playing framework for automatic dialogue dataset curation. it provides insights into building large-scale datasets for many other dialogue tasks. (3) our constructed topdial dataset is of high quality and contributes to the related research community. ## problem formulation desirable characteristics of datasets based on the above definition, we lay out two desirable characteristics that a qualified dataset should meet, namely target-oriented proactivity and personalization. target-oriented proactivity emphasizes that a dialogue dataset should allow the system to (i) take the initiative throughout a conversation, (ii) proactively lead the discussed topic towards the target topic based on domain knowledge, and (iii) accomplish the target act. on the other hand, personalization indicates that dialogues in a qualified dataset should embody (i) user profiles, which may involve users' past preferences about potential topics relevant to the target, and (ii) user personalities, which may imply users' possible reactions and feedback during the system-initiative process. ## dataset curation framework in this section, we describe a role-playing approach for automatic dataset curation using multiple llm agents. figure 1 depicts the whole framework, which involves one user agent, one system agent, and one moderator agent. all these agents are designed to follow specific instructions and communicate in our role-playing environment. role-playing environment this environment is designed to provide a global description for prompting all llm agents. to achieve desirable targetoriented role playing, we instantiate the environment description based on the domains of the predefined targets. for example, one can describe the environment as "you are participating in a conversation about music or movies." for a given target @xmath5, "king of comedy">. then, the description will be prepended to each agent's instructions. ## topdial dataset based on our dataset curation framework, we synthesized the dataset topdial by utilizing the repurposed version @xcite of durec-dial 2.0 @xcite as the seed dataset after carefully considering the problem formulation and necessary prompting sources. we report more implementation details in appendix b.1. dataset statistics ## conclusion in this work, we explore a new task: personalized target-oriented dialogue. we first define this challenging task, and then lay out the desirable characteristics that a qualified dialogue dataset should meet. we propose a novel role-playing framework for automatic dataset curation, based on which we construct a large-scale dialogue dataset topdial. our statistics and evaluations validate its effectiveness and high quality.