Faithful Persona-based Conversational Dataset Generation with Large Language Models (2024)

Abstract: High-quality conversational datasets are essential for developing AI models that can communicate with users. One way to foster deeper interactions between a chatbot and its user is through personas, aspects of the user’s character that provide insights into their personality, motivations, and behaviors. Training Natural Language Processing (NLP) models on a diverse and comprehensive persona-based dataset can lead to conversational models that create a deeper connection with the user, and maintain their engagement. In this paper, we leverage the power of Large Language Models (LLMs) to create a large, high-quality conversational dataset from a seed dataset. We propose a Generator-Critic architecture framework to expand the initial dataset, while improving the quality of its conversations. The Generator is an LLM prompted to output conversations. The Critic consists of a mixture of expert LLMs that control the quality of the generated conversations. These experts select the best generated conversations, which we then use to improve the Generator. We release Synthetic-Persona-Chat, consisting of 20k conversations seeded from Persona-Chat. We evaluate the quality of Synthetic-Persona-Chat and our generation framework on different dimensions through extensive experiments, and observe that the losing rate of Synthetic-Persona-Chat against Persona-Chat during an AI detection test decreases from 17.2% to 8.8% over three iterations.

## introduction every person is a story. systems that interact with people must understand their underlying stories to effectively engage with them. unfortunately, many existing datasets used for training conversational ˚work done during an internship at google inc., mountain view, usa 1 dataset will be publicly available on github agents do not sufficiently model their users. personas -abstract user representations that express the "story" of a person based on their background and preferences -have been widely used for humancentered design in a variety of domains, including marketing, system design, and healthcare @xcite . prior persona-based conversational datasets, like persona-chat (pc) @xcite , suffer from several limitations, such as small size, static dialogues that cannot easily be updated with new topics, irrelevant utterances, and contradictory persona attributes @xcite . in this paper, we propose a novel framework for generating large, dynamic, persona-based conversational datasets that capture the breadth and depth of human experience. personas @xcite have been widely used in a variety of domains and applications, including creating narratives for patients and sharing educational messages in healthcare @xcite , targeting users in marketing @xcite , and communicating with workers in management @xcite . conversational agents use personas to generate more interesting and engaging conversations with their users @xcite . creating persona-based datasets is difficult: the process is labor-intensive, the outputs must be updated to reflect current events and new concepts, and there are often quality concerns. existing persona-based datasets have resulted from laborintensive data collection processes @xcite involving humans to create or validate personas, create fictional personabased conversations, and ensure the conversations are coherent. moreover, even after these datasets are created, it is difficult to update them with the latest topics @xcite , such as current events, new concepts, products, or social trends @xcite . finally, existing persona-based datasets do not guarantee faithfulness, a criterion we introduce to describe the alignment between participants' utterances and their personas. in this paper, we introduce a new framework for generating large, customized persona-based conversational datasets that uses unsupervised llms to reduce human labor, introduces methods to generate, expand, and update personas automatically, and enforces a set of quality criteria including faithfulness to ensure dialogues are human-like. our persona-based conversational dataset generation framework consists of a three-level pipeline: ## definitions we define the faithful persona-based dialogue generation task. we begin by defining the personabased dialogue generation task. we then formally define the faithfulness criteria as a desired quality for the generated dialogues. throughout this section, we use π to refer to persona attributes (individual sentences which, together, form the user persona), u to refer to user profiles, and d to refer to conversations (dialogues). ## method in this section, we introduce our method to generate persona-based conversations. we create such conversations with minimum human input, starting from an initial dataset. our process consists of three steps, as shown in figure 2 : user generation, user pairing, and conversation generation. the first component augments a set of seed persona attributes π 0 into an expanded set of persona attributes π e , from which it creates user profiles. the second component pairs user profiles as interlocutors of a conversation. the third and final component uses an iterative process to generate high-quality conversations among user profile pairs. we detail each of these components below. ## evaluation we evaluate different aspects of our dataset generation framework, and the resulting dataset -referred to as synthetic-persona-chat -which is created using an instruction fine-tuned llm with 24 billion parameters @xcite . we compare synthetic-persona-chat (spc) against the widely used persona-chat (pc) dataset across different dimensions. we begin by evaluating the quality of the personas we generate. we then evaluate spc using both automatic metrics, and human assessment. we analyze other aspects of spc, such as toxicity and diversity in appendices b.1 and b.1. ## related work large language models (llms) have been used for data augmentation @xcite , generation @xcite @xcite , and evaluation @xcite . one of the earliest works in this area @xcite used llms to create a large text dataset from a small, labeled one. this idea was followed by @xcite which leveraged llms to create datasets without any human data. @xcite evaluated the performance of different llms on the data augmentation task. several conversational dataset generation methods focused on the structure of the conversational data @xcite @xcite . @xcite illustrated how llms can effectively generate synthetic training data for task-oriented dialogue models. persona-based conversations have been a popular research topic in nlp @xcite . one of the earliest works in this area is persona-chat, by @xcite , which proposed the persona-chat dataset and evaluation metrics that have become a benchmark for persona-based conversation generation @xcite . many subsequent works have used this dataset to train and evaluate @xcite their models, including dialogpt @xcite , blenderbot @xcite , and personachatgen @xcite . personachatgen automated the process of creating persona based conversations of persona-chat using llms. a challenge in generating synthetic datasets is to ensure the quality of the conversation including data faithfulness, fidelity, diversity, and consistency @xcite @xcite @xcite . several works have focused on creating and using high quality training datasets @xcite , and creating quality filtering components to their conversation dataset generation @xcite . evaluation of the resulting conversational datasets is also challenging @xcite . @xcite recently introduced the paradigm of interactive evaluation of conversations with llms. ## conclusion and future work we developed a novel framework for generating high-quality persona-based conversations using llms, resulting in the creation of synthetic-persona-chat, comprising 20k conversations. we hope this dataset will support future endeavors in developing persona-aware conversational agents, including the generation of domain-specific multisession conversations for specialized, task-oriented interactions. while we focused on a persona-based dataset generation task, our generator-critic approach can be generalized to other use cases, such as generating other specialized datasets, etc.