S ynth DST : Synthetic Data is All You Need for Few-Shot Dialog State Tracking (2024)

Abstract: In-context learning with Large Language Models (LLMs) has emerged as a promising avenue of research in Dialog State Tracking (DST). However, the best-performing in-context learning methods involve retrieving and adding similar examples to the prompt, requiring access to labeled training data. Procuring such training data for a wide range of domains and applications is time-consuming, expensive, and, at times, infeasible. While zero-shot learning requires no training data, it significantly lags behind the few-shot setup. Thus, ‘ Can we efficiently generate synthetic data for any dialogue schema to enable few-shot prompting? ' Addressing this question, we propose , a data generation framework tailored for DST, utilizing LLMs. Our approach only requires the dialogue schema and a few hand-crafted dialogue templates to synthesize natural, coherent, and free-flowing dialogues with DST annotations. Few-shot learning using data from results in 4-5\% improvement in Joint Goal Accuracy over the zero-shot baseline on MultiWOZ 2.1 and 2.4. Remarkably, our few-shot learning approach recovers nearly 98% of the performance compared to the few-shot setup using human-annotated training data.

## introduction dialogue state tracking (dst) is an integral task in task-oriented dialogue systems that predicts the user intentions for each turn by mapping them to predefined slot-value pairs @xcite . dst systems capture important information essential to model the downstream dialogue policy and help generate actionable responses @xcite . prior literature has typically framed can we schedule the taxi to depart at 03:15? when do you want to book a taxi, and is there anything else i can help you with? ## methodology figure 2 outlines the synthdst's data generation pipeline. it utilizes just the dialogue schema and a set of handcrafted templates to generate fluent dialogues with dialog state annotations. specifically, each dialog generated using synthdst comprises a quartet of dialogue history, system turn, user turn, and the current turn's dialog states. synthdst employs a three-step approach for generating dialogue data. we explain each step below. ## dataset quality analysis is the data generated by synthdst of good quality? as synthdst is a humaninvolvement-free synthetic data generation approach (except for template definition), assessing its quality is crucial. consequently, we conducted a human evaluation on 200 dialogues from our 1% dataset split. four evaluators, experienced in dialogue systems research, evaluate the data. given the generated samples containing the dialogue history, average system utterance, average user utterance, and new dialogue state, the evaluators assessed the dialogues on four dimensions, namely, grammar, coherence, naturalness, and annotations. the annotations are rated from 1-3, whereas the others are graded on a 1-5 scale. the detailed scales are given in appendix a.3. in figure 3 , we present the results of our human evaluation. the majority of the dataset demonstrates high scores for grammar, indicating grammatical correctness and minimal mistakes. for coherency, both the mean and median scores exceed 4, signifying that the dialogues are mostly coherent and logically structured. while naturalness exhibits slightly more variability, the mean, and median still surpass 4, indicating that most dialogues maintain a natural conversational flow resembling real-world conversations. lastly, the annotations scale attains a median of 3 and a @xmath1.5, suggesting that most of the annotations are correct. is synthdst more cost-effective than human annotation? creating the multiwoz dataset involved 1,249 workers and incurred a cost of approximately @xmath040. moreover, generating 1% equivalent data requires less than $1 while maintaining the dst performance. thus, synthdst presents a cost-effective method to collect dst data. ## conclusion and future work in this work, we present synthdst, a synthetic data generation framework that leverages the dialogue schema to create coherent dialogues with dst annotations using a template-guided llm-based approach. this framework enables the use of in-context learning for dst without human-annotated training data. performance with synthdst reaches close to the performance with training data on dialogue state tracking. this opens the possibility of supporting new domains without needing cumbersome and expensive training data collection. moreover, it also reduces while we experiment only with the multiwoz datasets, synthdst can readily be extended to other corpora. while synthdst predominantly relies on the close-sourced openai gpt-3 model, it would be interesting to see how it performs with open-sourced llms. we encourage further research that validates its performance across diverse domains and models. moreover, our approach does not incorporate safeguards to detect hallucinations in llm-generated data, which is a direction for future investigations.