Diverse and Effective Synthetic Data Generation for Adaptable Zero-Shot Dialogue State Tracking (2024)

Abstract: We demonstrate substantial performance gains in zero-shot dialogue state tracking (DST) by enhancing training data diversity through synthetic data generation.Existing DST datasets are severely limited in the number of application domains and slot types they cover due to the high costs of data collection, restricting their adaptability to new domains.This work addresses this challenge with a novel, fully automatic data generation approach that creates synthetic zero-shot DST datasets.Distinguished from previous methods, our approach can generate dialogues across a massive range of application domains, complete with silver-standard dialogue state annotations and slot descriptions.This technique is used to create the D0T dataset for training zero-shot DST models, encompassing an unprecedented 1,000+ domains. Experiments on the MultiWOZ benchmark show that training models on diverse synthetic data improves Joint Goal Accuracy by 6.7%, achieving results competitive with models 13.5 times larger than ours.

## introduction a critical task for building task-oriented dialogue (tod) systems is dialogue state tracking (dst), which aims to maintain a structured representation of the key task-related information provided throughout a dialogue. conventionally, the state representation is composed of a set of task-specific slot-value pairs, where slots are information types provided by a predefined slot schema. while dst has been studied in fully supervised @xcite @xcite and fewshot settings @xcite @xcite , these settings rely on a substantial amount of labeled training examples within the targeted task domain. to this end, zero-shot dst has recently gained attention, as it requires the dst model to adapt to an unseen target domain for which no training examples are available @xcite @xcite . leveraging slot descriptions to perform crosstask transfer is shown to be effective for zero-shot dst @xcite @xcite . in this approach, a model is trained to interpret the slot descriptions to perform dst using gold supervision in several data-rich domains. during inference, the model interprets new slot descriptions to perform dst in unseen target domains without any training data. however, for this approach to succeed, sufficiently diverse training data must be available to enable the model to generalize and handle new slot types. we hypothesize that existing training data for dst is a bottleneck, as the two most popular datasets for dst training, @xcite and sgd @xcite , only cover 7 and 16 domains, respectively. this work aims to explore the impact of increasing training data diversity on zero-shot dst performance. since traditional methods of creating diverse dst training data are costly and difficult to scale, we develop a novel, fully automatic data generation approach for zero-shot dst. this approach leverages the capabilities of instruction-tuned large language models (llms) to create new task domains from scratch. synthetic dialogues are generated for each domain, and are automatically annotated for dialogue state, complete with descriptions of labeled slots. this approach is leveraged to generate a synthetic dst dataset of unprecedented diversity, including over 1, 000 task domains. experiment results demonstrate a substantial performance boost provided by this synthetic data on standard benchmarks. in summary, our contributions are: we make all models, code, and data publicly available to support future work. 1 ## dst data generation this section presents our fully automatic data generation approach to support training dst models capable of zero-shot domain transfer. our goal is to create a set of dialogue data covering many diverse task domains, with silver dialogue state labels and natural language slot descriptions. given the exceptional zero-shot performance of instructiontuned large language models (llms) @xcite @xcite , our approach explores using instruction-tuned llms for data generation. we use gpt 2 in all of our presented experiments, although any llm can be used for our approach in principle. musician talks to sound technician to adjust levels. information types 1. specific frequency ranges that need to be changed 2. desired tone or feel of the overall mix 3. desired volume levels for individual instruments and vocals dialogue artist: right now, the guitar is a bit too loud compared to the vocals. can we bring the guitar down and bring the vocals up? tech: got it. what levels are we at right now? artist: the guitar is at about 80, vocals at 60. tech: okay, i'm lowering the guitar to 60 and bringing the vocals up to 70. how does that sound? question-answer pair generation what level is the guitar at now? the guitar is now at 60. what level are the vocals at now? the vocals are now at 70. how does the new sound balance sound? [unknown] slot-value translation guitar @xmath0: current guitar volume [60, 5, etc.] vocal level: current vocal volume [70, 23, etc.] sound balance: quality of the sound mix [balanced, vocal-forward, etc.] scenario derivation dialogue generation slot description generation state annotation the approach consists of four stages, which are summarized in figure 1 . first, domains are derived through an iterative process of generating and refining dialogue scenario descriptions ( §3.1). next, a dialogue is crafted based on the scenario description and a generated unstructured information list corresponding to the scenario ( §3.2). third, each turn in each dialogue is automatically annotated with silver dialogue state labels ( §3.3). finally, a slot description is composed for each silver slotvalue pair annotation ( §3.4). all prompts included in the approach are provided in appendix b. ## new dataset for zero-shot tracking using our dst data generation approach ( §3), we create a diverse 0-shot tracking dataset: d0t. since we aim to measure the impact of increasing the diversity of dst training resources, we generate d0t to include unprecedented 1, 000+ domains and 5 dialogues per domain. applying automatic state annotation ( §3.3) to the generated dialogues yields 324, 973 slot-value pairs in state updates. since compiling each dialogue state s @xmath1, u t ) produces an excessive ≈ 6.5 million total slot-value pairs for dst training, slotvalue pairs are downsampled using a method that maintains slot type diversity. we randomly sample exactly 1 example for each of the original 324, 973 slot-value updates from the set of final slot-values where that slot is filled (non-empty), resulting in @xmath2, 973 filled slot-value examples. to include examples of empty slots, we randomly sample m empty slot-value pairs from the final compiled states, where @xmath3.5 * @xmath4, 487. ## conclusion the costly nature of dst data collection has been a limiting factor for the domain diversity of existing datasets for years. by introducing the first automatic data generation method capable of creating new domains and slot definitions for dst, this work both reveals and alleviates a performance bottleneck caused by the limited domain coverage of existing dst data. training on the synthetic, domain-diverse d0t dataset produces substantial performance gains (e.g. +6.7% average jga) for zero-shot dst, and this performance gain is stable even when testing on domains with no similar analog in synthetic data. these results show the power of domain diversity for training zero-shot dst models, as it allows our models to achieve competitive or better performance to llm-based dst approaches with over 13.5× the parameters. the success of our data generation approach also demonstrates the potential of llm-based data generation to alleviate the high costs of traditional data collection. our work advances data generation methods for dst as the first that is fully automatic, including creating new task domains without human guidance. by continuing to improve the diversity and correctness of synthetic datasets, we anticipate even greater advancements in zeroshot dst performance, driving the development of more robust and adaptable dialogue systems. we look forward to future research and application development in task-oriented dialogue that builds upon our experimental insights and released models and data.