Controllable Dialogue Simulation with In-context Learning (2022)

Abstract: Building dialogue systems requires a large corpus of annotated dialogues. Such datasets are usually created via crowdsourcing, which is expensive and time-consuming. In this paper, we propose Dialogic, a novel dialogue simulation method based on large language model in-context learning to automate dataset creation. Seeded with a few annotated dialogues, Dialogic automatically selects in-context examples for demonstration and prompts GPT-3 to generate new dialogues and annotations in a controllable way. Our method can rapidly expand a small set of dialogue data with minimum or zero human involvement and parameter update and is thus much more cost-efficient and time-saving than crowdsourcing. Experimental results on the MultiWOZ dataset demonstrate that training a model on the simulated dialogues leads to even better performance than using the same amount of human-generated dialogues under the challenging low-resource settings, with as few as 85 dialogues as a seed. When the full training set is given, our method can still serve as an effective data augmentation method to further improve performance. Human evaluation results also show that our simulated dialogues have near-human fluency and annotation accuracy. The code and data are available at .

## introduction task-oriented dialogue (tod) systems can assist users in completing tasks such as booking a restaurant or making an appointment. building such a dialogue system requires a large corpus of annotated dialogues @xcite , which is costly to obtain in terms of money and time. one popular approach to collecting and annotating task-oriented dialogues is crowdsourcing via a wizard-of-oz setup @xcite @xcite , where crowdworkers produce conversations. significant annotation efforts are further needed to label intent, entities, etc. prior work has been proposed to minimize the cost and effort in data collection by hiring crowdworkers or leveraging user simulators to interact with existing dialogue systems @xcite @xcite @xcite . however, the dependency on existing dialogue systems leave the developers with a classic chicken-and-egg problem. in addition, developing such user simulators typically requires considerable handcrafting and human involvement. in recent years, large language models (llms) @xcite @xcite @xcite demonstrate strong in-context learning capability. provided with a few in-context examples, the llms, such as @xcite , can generate text with similar patterns without finetuning. this capability has been leveraged to synthesize training data in a few nlp tasks @xcite . although there have been methods that generate training data for a single component in the tod systems @xcite , there hasn't been a plausible solution to generate whole dialogues with annotations for endto-end training due to its complex nature of involving multi-turn interactions, multiple possible logic flows, and multiple types of annotations. to address the challenge, we introduce a controllable dialogue simulation method dialogic for dialogue dataset creation. seeded with a few seed dialogues, dialogic automatically selects incontext examples for demonstration and prompts llms such as gpt-3 to generate annotated dialogues in a controllable way. dialogic can play the roles of both user and system simulator. figure 1 illustrates a partial example. for the user side, gpt-3 is prompted first to generate the turn9 . this paper has two key insights. first, leveraging the in-context learning ability of llms, our method can simulate both the user and system side to generate annotated dialogues by learning from a few examples. except for the minimal efforts in collecting the small seed dataset and training an auxiliary model on that, the simulation process is free of human involvement and parameter update, making our method much cheaper and faster than crowdsourcing in dataset creation. specifically, a large-scale and high-quality dataset such as multi-woz @xcite can be created using our method within only several hours. second, we design controllable dialogue generation strategies to overcome the deficiency of gpt-3 in lack of reliability and interpretability. we also investigate effective representations and selection strategies of in-context dialogue examples for llms to better leverage their in-context learning capabilities. we conduct experiments on multiwoz2.3 @xcite dataset. remarkably, in the challenging low resource settings where as low as only 85 seed dialogues (1% of the whole training dataset) are given, the dialogues simulated by our method lead to even better model performance than the same amount of human-generated dialogues. dia-logic can also serve as an effective data augmentation method when the full training set is provided. human evaluations indicate that our simulated dialogues have comparable fluency, annotation accu-racy, and more diverse dialogue flows than humangenerated dialogues. our results demonstrate the promise of leveraging large language model to automate the complex dialogue dataset creation. we have released the code and simulated data to facilitate future studies. foot_1 2 related work ## method in this paper, we introduce a novel method dia-logic to simulate annotated dialogues for building task-oriented dialogue systems based on language model in-context learning. the only requirements are a small seed dataset d s consisting of a few annotated dialogues and an ontology o that includes all slots and possible slot values for each domain. an auxiliary tod model m such as simple-tod (hosseini-asl et al., 2020) and pptod @xcite ) trained on d s will be used to verify and revise generated annotations. our goal is to expand d s by generating new dialogues. for each turn of the dialogues, we need to generate the user utterance u , belief state b, database (db) query result q, dialog act a, and system responses s (we omit the turn index for brevity). we will elaborate the design of our method using a well-studied task-oriented dialogue dataset multiwoz @xcite @xcite , which covers 7 domains such as hotel and restaurant, and 24 slots such as hotel-area and restaurant-food (see appendix a for more details). to simulate the low-resource environment, we use 1%, 5%, 10% of the training dataset as the seed dataset d s . ## conclusion in this paper, we propose a dialogue simulation method based on large language model in-context learning to automate dataset creation. our proposed method can generate dialogues and annotations given only a few seed dialogues. the simulation process requires zero or minimum human involvement and model training, making our method much more cost-efficient and time-saving than crowdsourcing. human and automatic evaluations demonstrate that the simulated dialogues have comparable quality to human-generated ones, which shows the potential of our method as an alternative to crowdsourcing in dialogue dataset creation.