Show, Don’t Tell: Demonstrations Outperform Descriptions for Schema-Guided Task-Oriented Dialogue (2022)

Abstract: Building universal dialogue systems that operate across multiple domains/APIs and generalize to new ones with minimal overhead is a critical challenge. Recent works have leveraged natural language descriptions of schema elements to enable such systems; however, descriptions only indirectly convey schema semantics. In this work, we propose Show, Don’t Tell, which prompts seq2seq models with a labeled example dialogue to show the semantics of schema elements rather than tell the model through descriptions. While requiring similar effort from service developers as generating descriptions, we show that using short examples as schema representations with large language models results in state-of-the-art performance on two popular dialogue state tracking benchmarks designed to measure zero-shot generalization - the Schema-Guided Dialogue dataset and the MultiWOZ leave-one-out benchmark.

## introduction task-oriented dialogue (tod) systems need to support an ever-increasing variety of services. since many service developers lack the resources to collect data and train models, zero and few-shot transfer to unseen services is critical to the democratization of dialogue agents. recent approaches to generalizable tod systems primarily rely on combining two techniques: large language models like bert @xcite and t5 @xcite , and schemaguided modeling -i.e. using natural language descriptions of schema elements (intents and slots) as model inputs to enable transfer to unseen services @xcite . models combining the two currently hold state-of-the-art (sota) results on dialogue state tracking (dst) @xcite @xcite . ## show, don't tell following sota models, we pose dst as a seq2seq task @xcite and finetune t5 on dst datasets. the model input consists of a prompt to convey api semantics and context to represent the current dialogue instance. the target contains ground truth belief states corresponding to the context. we compare against two baselines: we modify the prompt formats above to utilize demonstrations instead of descriptions as described below and illustrated in figure 1 . in both the t5-* and sdt-* approaches, the context is the serialized dialogue history for the current dialogue instance. the final model input is formed by concatenating the prompt and the context strings, and the target string is the same as t5-*, containing only a single slot value for *-ind models and the entire turn's belief state for *-seq models. for both t5-* and sdt-*, we enumerate the categorical slot values in multiple-choice format in the prompt and task models with decoding the multiple choice letter corresponding to the correct categorical value. more details on prompt design and its impact on performance are provided in appendix a. creating prompt examples: it is imperative that sdt prompts contain enough information to infer the semantics for all slots in a schema. for sdt-ind, we create individual utterances that showcase a single slot. for sdt-seq, we create example dialogues where all slots in the schema are used. ## experimental setup datasets: we conduct experiments on two dst benchmarks: schema-guided dialogue (sgd) @xcite and multiwoz 2.1 @xcite . for multiwoz, we evaluate on the leave-one-out setup @xcite , where models are trained on all domains but one and evaluated on the holdout domain. additionally, we apply the rec-ommended trade pre-processing script foot_0 for fair comparison with other work. for both datasets, we created concise example dialogues modeled after dialogues observed in the datasets. implementation: we train sdt models by finetuning pretrained t5 1.1 checkpoints. for sdt-seq, we select one example dialogue for each service to create a prompt and use that prompt across all dialogue instances of that service, across training and evaluation. we do the same for sdt-ind but create one prompt per slot instead of per service. unless otherwise noted, all t5-based models are based on t5-xxl (11b parameters). appendices b and c contain more details on training and baselines respectively. ## related work prior approaches focused on framing dst as question answering @xcite @xcite . many multiwoz crossdomain models leverage slot names/descriptions @xcite @xcite . pretrained generative llms @xcite have enabled framing nlp tasks as seq2seq problems. some dst papers @xcite look at settings with no train-test discrepancy. many studies explore the efficacy of task-specific prompts @xcite . @xcite and prime lms with examples for dialogue tasks, but without finetuning. @xcite finetunes language models to teach them to use prompts to generalize across nlp tasks. ## conclusion we study the use of demonstrations as lm prompts to convey the semantics of apis in lieu of natural language descriptions for tod. while taking similar effort to construct, demonstrations outperform description-based prompts in our experiments across dst datasets (sgd and multiwoz), model sizes, and training data sizes, while being more robust to changes in schemata. this work provides developers of tod systems with more options for api representations to enable transfer to unseen services. in future work, we would like to explore this representation for other tod tasks (e.g. dialogue management and response generation).