PLACES : Prompting Language Models for Social Conversation Synthesis (2023)

Abstract: Collecting high quality conversational data can be very expensive for most applications and infeasible for others due to privacy, ethical, or similar concerns. A promising direction to tackle this problem is to generate synthetic dialogues by prompting large language models. In this work, we use a small set of expert-written conversations as in-context examples to synthesize a social conversation dataset using prompting. We perform several thorough evaluations of our synthetic conversations compared to human-collected conversations. This includes various dimensions of conversation quality with human evaluation directly on the synthesized conversations, and interactive human evaluation of chatbots fine-tuned on the synthetically generated dataset. We additionally demonstrate that this prompting approach is generalizable to multi-party conversations, providing potential to create new synthetic data for multi-party tasks. Our synthetic multi-party conversations were rated more favorably across all measured dimensions compared to conversation excerpts sampled from a human-collected multi-party dataset.

## introduction training dialogue models typically requires an abundance of data, as with any machine learning task. however, collecting high quality data is difficult and expensive, especially for dialogue tasks where there often is no "right answer" when developing the trajectory of a conversation. typically dialogue data are sourced from crowdworkers and the quality of annotations, evaluations, and conversations can vary considerably @xcite , often necessitating guardrails such as credentialbased worker selection or defensive task design for quality control @xcite . ## related work recently, the zero-and few-shot learning capabilities of large pre-trained language models have overtaken state-of-the-art performance on many classical natural language processing tasks, including dialogue @xcite . many plms such as t5 @xcite , gpt-j (wang and komatsuzaki, 2021), gpt-3 @xcite , and opt @xcite have become the backbone of several dialogue-specific models (e.g., peng et al. (2022 in particular, in-context learning, where few-shot examples are provided in the input prompt of a plm, has been found to provide valuable information in guiding generation output @xcite @xcite . as a result, many recent efforts in prompting plms have sought to augment various natural language processing datasets @xcite @xcite @xcite . prompting has become a viable "solution" for augmentation in dialogue tasks, which have traditionally been considered challenging due to the difficulty of augmenting dialogue context @xcite . however, prompt-based augmentation strategies are uncontrolled forms of generation, which may result in generation mistakes for labeled datasets @xcite @xcite . in contrast, other recent studies have instead proposed language augmentation strategies that use complex, highly-controlled frameworks that often involve fine-tuning generators @xcite @xcite . such complex augmentation frameworks require larger amounts of seed data to maintain a ground-truth language distribution @xcite , and are more costly than prompting plms @xcite . however, in the context of dataset synthesis, seed data and label correctness are less random in-context examples target @xmath0. bob recently got engaged. alice: congrats on your engagement! when do you think you will have your wedding? bob: thank you!! we're thinking of having it in november. alice: that's amazing! will you pick a fancy destination? ... <conversation 2> <conversation 3> the following is a conversation between alice and bob about pets. alice likes cats but bob likes dogs. result: alice: what kind of pet do you have? bob: i have a dog. her name is hope. alice: are you an animal person? bob: i do love animals. i think they're so special. alice: i just adopted a kitten from a shelter. her name is mimi. bob: aww, i hope she's doing well. you must be a really good pet parent. alice: thanks! i just wanted to do what i could to make sure she was safe. bob: that's the greatest thing anyone can do for a pet. well, i should probably go now. alice: ok! have a good night. topic: relationships background info: bob got engaged topic: pets background info: alice likes cats. bob like dogs. recipe figure 2: example of the components of a prompt (left) used by opt 30b to generate a synthetic conversation about pets (right). conversations in the prompt are prefixed by recipes. blue text: topic labels. red text: seed background information metadata. ## conversation generation in this section, we discuss our methods for conversation generation. we first detail the construction of our example conversations, then describe their application to prompting plms. ## synthetic conversation evaluation in figure 1 , the top-left is taken from dailydialog, whereas the top-right is generated synthetically. the bottom-left is generated synthetically and the bottom-right is taken from mpc. ## triadic and multi-party conversations the vast majority of dialogue tasks and conversational datasets focus on dyadic conversations (e.g. li et al. (2017); gopalakrishnan et al. (2019); smith et al. (2020); rashkin et al. (2019)), following the traditional speaker-listener paradigm (engelhardt et al., 2006). in contrast, the literature on multi-party social conversation is rather scarce, not only in terms of conversation generation but as a task altogether. however, while it is an understudied research area, it is incredibly important, because dyadic conversations do not capture the full reality of in-person, human-human social conversations, nor the full potential of dialogue agents. to name a few applications, dialogue agents have the potential to supplement classroom learning with multiple parties, serving as a third mediating party in a debate or discussion between two people, or to provide companionship and support in virtual group settings. a major reason why these lines of work remain unsolved is that there are few largescale multi-party dialogue datasets. many existing multi-party datasets are scripted corpora such as meld @xcite or mpdd @xcite or hla-chat @xcite . other multi-party corpora are collected for highly domain-specific purposes, such as multi-party empathetic dialogue @xcite . such corpora are also typically collected through asynchronous online platforms, rather than natural conversation. these platforms exist in the form of forums and online chat platforms such as ubuntu irc @xcite or reddit @xcite . other more natural multiparty conversational datasets are license-protected speech datasets (e.g. chime @xcite ) which have been constructed for tasks such as speaker attribution. we find that we can apply our prompting approach to generate synthetic, open-domain, multiparty social conversations following the same structure as our synthetic dyadic conversations foot_5 . as in the dyadic case, we generate triadic conversations using optional background information for each speaker. we consider the "multi-party chat" corpus (mpc) @xcite , a text-based, open-domain conversation dataset collected in realtime online sessions at the university of albany, and meld, which contains scripted multi-party dialogues from the popular sitcom "friends." we directly compare our synthetically generated conversations against mpc and meld. we inform the amt workers that they will read conversation excerpts. in addition to the questions in in additionally, we checked the linguistic diversity for each speaker. in terms of distinct-n, each speaker's lexical diversity is comparable (figure 4 ) as well as the number of words per turn (12.2, 12.2, and 13.5 for speakers 1, 2, and 3 respectively). the triadic conversations tended to be slightly longer than the average dyadic conversation (11.5 turns/conversation versus 9.29 turns/conversation). ## discussion overall, we find that prompting plms to generate synthetic conversations is promising. ## error analysis we examine the dyadic and triadic conversations which received low scores (1/5) across multiple dimensions. ## conclusion in this work, we presented an application of prompting plms to create synthetic conversations. these synthetic conversations are comparable in terms of quality and lexical diversity to actual humanhuman datasets, and can be used as training data for dialogue models. this opens avenues in generative language work such as collaborative and creative writing, story generation, as well as synthesis of new conversational tasks. here, we presented one example -synthesizing a multi-party conversational dataset. this presents a unique opportunity to further study multi-party dialogue modeling.