Show, Don’t Tell: Demonstrations Outperform Descriptions for Schema-Guided Task-Oriented Dialogue
Abstract
Building universal dialogue systems that can seamlessly operate across multiple domains/APIs and generalize to new ones with minimal supervision and maintenance is a critical challenge.
Recent works have leveraged natural language descriptions for schema elements to enable such systems; however, descriptions can only indirectly convey schema semantics.
In this work, we propose Show, Don’t Tell , a prompt format for seq2seq modeling which uses a short labeled example dialogue to show the semantics of schema elements rather than tell the model via descriptions.
While requiring similar effort from service developers, we show that using short examples as schema representations with large language models results in stronger performance and better generalization on two popular dialogue state tracking benchmarks: the Schema-Guided Dialogue dataset and the MultiWoZ leave-one-out benchmark.
Introduction
Task-oriented dialogue (TOD) systems need to support an ever-increasing variety of services.
Since many service developers lack the resources to collect labeled data and/or the requisite ML expertise, zero and few-shot transfer to unseen services is critical to the democratization of dialogue agents.
New approaches to TOD that can generalize to new services primarily rely on combining two techniques: large language models like BERT (Devlin et al., 2019) and T5 (Raffel et al., 2020) and schema-guided modeling i e. using natural language descriptions of schema elements (intents and slots) as model inputs to enable inference on unseen services (Rastogi et al., 2020a,b).
Models combining the two currently show state-of-the-art results on dialogue state tracking (DST) (Heck et al., 2020;
Lee et al., 2021a; Zhao et al., 2022).
However, description-based schema representations have drawbacks. Writing precise natural language descriptions requires manual effort and can be tricky, and descriptions only provide indirect supervision about how to interact with a service compared to an example.
Furthermore, Lee et al.
(2021b) showed that state-of-the-art schemaguided DST models may not be robust to variation in schema descriptions, causing signiﬁcant accuracy drops.
We propose using a single dialogue example with state annotations as an alternative to the description-based schema representation, similar to one-shot priming (Brown et al., 2020).
Rather than tell the model about schema element semantics in natural language, we show the schema through a demonstration, as in Figure 1.
Applying our approach, Show, Don’t Tell (SDT) , to two SotA DST models consistently results in superior accuracy and generalization to new APIs across both the Schema-Guided Dataset (SGD) (Rastogi et al., 2020b) and MultiWoZ Leave-OneOut
(Budzianowski et al., 2018; Lin et al., 2021b) benchmarks, while being more data-efﬁcient and robust to schema variations.
Show, Don’t Tell
Following SoTA models, we pose DST as a seq2seq task (Wu et al., 2019;
Zhao et al., 2021a), where the seq2seq language model (in our case, T5) is ﬁnetuned on a DST dataset.
During ﬁnetuning and evaluation, the model input consists of a prompt and context , and the target contains ground truth belief states.
We compare against two baselines:
• T5-ind
(Lee et al., 2021a): Model input comprises the dialogue history as context concatenated with one slot description as the prompt.
The target is the value of that slot in the dialogue
state.
Inference is done per slot - i.e. values for different slots are independently decoded.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
• T5-seq
(Zhao et al., 2022): Model input comprises the descriptions of all slots as the prompt, followed by the dialogue history as the context.
The target is the sequence of slot-value pairs in the dialogue state - i.e. the dialogue state is decoded sequentially in a single pass.
We modify the prompt formats above to utilize demonstrations instead of descriptions as described below and illustrated in Figure 1.
• SDT-ind : A prompt P indi comprises a single utterance labeled with a slot value pair formatted as
[Equation]
where u indi is a user utterance where slot i is active and sv i is the slot value pair.
[ ex ] , [ slot ] are special delimiter tokens, and ; denotes concatenation.
• SDT-seq : A prompt P seq comprises a single labeled dialogue formatted as: P seq =
[ ex ]; u 1 ; ... ; u n ; [ slot ]; sv 1 ; ... ; sv m where u j is an utterance, and other symbols are shared with SDT-ind.
In simple terms, the prompt is constructed by concatenating all utterances in the example dialogue followed by all slot-value pairs in the ﬁnal dialogue state.
The
context in both formats is a concatenation
of the dialogue history for the current training example
The ﬁnal model input is formed by con-
catenating
the prompt and the context strings.
The target string is the same as
T5-*
, containing only a single slot value for *-ind models and the entire turn’s belief state for *-seq models. For both T5-* (baseline) and SDT-*, we enumerate the categorical slot values in multiple-choice format in the prompt and task models with decoding the correct multiple choice letter.
More details on the prompt design and its impact on performance are provided in Appendix C.
Formulating prompt examples: It is imperative that SDT prompts contain sufﬁcient information to infer the semantics for all slots in the schema.
This is easy for SDT-ind, which uses a separate prompt for each slot.
However, for SDT-seq, we only choose example dialogues where all slots in the schema are used.
Experimental Setup
Datasets: We conduct experiments on two DST benchmarks:
Schema-guided Dialogue (SGD) (Rastogi et al., 2020b) and MultiWOZ 2.
(Budzianowski et al., 2018;
Eric et al., 2020).
For
MultiWOZ, we evaluate on the leave-one-out setup (Wu et al., 2019; Lin et al., 2021a), where models are trained on all domains but one and evaluated on the holdout domain.
Additionally, we apply the recommended TRADE pre-processing script 1 for fair comparison with other work.
For both datasets, we created concise prompt dialogues modeled after dialogues observed in the datasets.
Implementation: We train SDT models by ﬁnetuning pretrained T5 1.1 checkpoints.
For both datasets, we select one example prompt per service schema (for SDT-seq) or slot (for SDT-ind), [Table] and use the same prompt for all examples for that service/slot across training and evaluation.
Unless otherwise noted, all T5-based models (T5/SDTseq/ind) are ﬁnetuned on T5-XXL (11B parameters).
See appendices A and B for more details on training and baselines respectively.
Results 4. Results on SGD
Table 1 contains results on the SGD test set.
Since
SDT results may depend on the choice of example turn/dialogue provided in the prompt, 5 different versions of prompts are created for each service using different examples.
We report the average JGA across these versions and the 95% conﬁdence intervals.
SDT-seq achieves the highest JGA, showing major gains, particularly on unseen services, over its description-based counterpart T5-seq and the next-best model T5-ind.
SDT-ind is comparable to its counterpart T5-ind, and better than T5-seq.
Based on these results, conveying service semantics via a single dialogue example appears more effective than using natural language descriptions.
We hypothesize that SDT-seq outperforms SDTind because the full dialogue prompts used in SDTseq demonstrate more complex linguistic patterns (e.g. coreference resolution, long term dependencies) than the single utterance prompts of SDT-ind. On the other hand, T5-seq does not outperform T5ind because no additional information is conveyed to the model through stacking descriptions.
Also all-else-equal, decoding all slots in one pass is more challenging than decoding each slot independently. We also experimented with using more than one dialogue to prompt SDT-seq, but we did not see an increase in performance.
4. MultiWOZ Results
Table 2 summarizes results for the MultiWOZ 2.
leave-one-out setup.
Comparing T5-seq and SDTseq, both ﬁnetuned on T5-XXL, SDT achieves [Table] state-of-the-art results on the overall task by +2% and in 3 of the 5 domains.
4. Impact of Model Size
T5’s XXL size may be unsuitable in a number of settings; consequently, we measure SDT’s performance on SGD across other model sizes in Table 3.
For the base and large model sizes, both SDT variations offer higher JGA than their description-based counterparts, possibly due to smaller T5 models being less capable of inferring unseen slots with just a description. Additionally, SDT-ind outperforms SDT-seq for these sizes.
Model
[Table]
4. Data Efﬁciency
To examine the data efﬁciency of SDT models
[Table]
we train SDT-seq in a low-resource setting with 0.16% (10-shot), 1%
[Table]
and 10% of the SGD training data and evaluate on the entire test set. For 10-shot, we randomly sample 10 training dialogues from every service; for 1% and 10%, we sample uniformly across the entire dataset.
SDT-seq demonstrates far higher data efﬁciency than T5-seq (Table 4)
[Table]
4. Robustness
Large LMs are often sensitive to the choice of prompt (Zhao et al., 2021b; Reynolds and Mc- Example Dialogue Error
[Figure] Donell, 2021). To this end, we evaluate SDT-seq on the SGD-X (Lee et al., 2021b) benchmark, which comprises 5 variants with paraphrased slot names and descriptions per schema (see Appendix Figure 4).
Table 5 shows SDT-seq achieves the highest average JGA ( JGA v 1 − 5 ) and lowest schema sensitivity ( SS JGA ), indicating it is the most robust of the compared models.
Even so, the JGA decline indicates SDT-seq is sensitive to how slot names are written.
[Table]
*Results from Lee et al. (2021b). (cid:72) Result of using T5-seq with only slot names and not descriptions, from
Zhao et al.
Discussion 5. Writing descriptions vs. demonstrations
We note that the information provided to SDT is not identical to what is provided to typical schemaguided models, as SDT exchanges natural language descriptions for a demonstration of identifying slots in a dialogue.
However, we argue that from the developer standpoint, creating a single example is similar in effort to writing descriptions, so we consider the methods comparable.
Creating the SDT-seq prompts for all 45 services in SGD took an experienced annotator ∼ 2 hours, compared to ∼ 1.5 hours for generating slot descriptions.
SDTind prompts are even simpler to write because they relax the requirement for creating a coherent dialogue where all slots are used. One advantage of descriptions is that they can be easier to generate than a succinct dialogue that covers all slots.
However, given the performance gain, example-based prompts may be a better choice for many settings, especially for smaller model sizes where the gain is more pronounced.
5. Descriptions plus demonstrations
Training with descriptions has proven effective for improving DST performance (Zhao et al., 2022;
Lee et al., 2021a), and our experiments show that demonstrations are even more effective. We combined both together to see if the two are complementary or overlapping, and we ﬁnd that performance does not improve above using demonstrations alone (Appendix Table A1. We hypothesize that demonstrations already convey slot semantics sufﬁciently and descriptions become extraneous. 5. Prompting vs. traditional ﬁnetuning
To understand the impact of using a single demonstration as a prompt vs. traditional ﬁnetuning, we ﬁnetune T5-seq a second time on the same set of dialogues used in SDT-seq prompts; it therefore has access to both slot descriptions as well as a single demonstration for each service.
In this case, T5-seq is provided strictly more information than SDT-seq.
T5-seq with ﬁnetuning obtains a JGA of 87.7% on SGD, on par with T5-ind but still lower than
SDT-seq, suggesting that dialogue examples are better used as prompts (Le Scao and Rush, 2021).
Interestingly, ﬁnetuning on more than one dialogue example per service did not improve performance (Appendix Figure 3).
Figure 2 compares some common errors made by T5-seq and SDT-seq.
The patterns suggest that SDT’s demonstrations are helpful when multiple slots are similar to each other (#1) and when prompt dialogues closely match target dialogues
However, SDT can be limited by its prompt. For instance, in #3 it has only seen the "music" value for the event_type slot in the prompt, potentially resulting in under-predicting the other categorical value ("theater").
Related Work
Prior approaches focused on framing DST as question answering (Ruan et al., 2020; Ma et al., 2019;
Zhang et al., 2021).
Many MultiWoZ cross-domain models leverage slot names/descriptions (Wu et al., 2019; Lee et al., 2019; Lin et al., 2021a).
Pretrained generative LLMs (Raffel et al., 2020;
Brown et al., 2020) have enabled framing NLP tasks as seq2seq problems.
Some DST papers (Zhao et al., 2021a; Feng et al., 2021) look at settings with no train-test discrepancy.
Many studies explore the efﬁcacy of task-speciﬁc prompts (Jiang et al., 2020; Liu et al., 2021).
Madotto et al.
(2020) and prime LMs with examples for dialogue tasks, but without ﬁnetuning.
Wei et al.
(2021) ﬁnetunes language models to understand prompts for a different task.
Conclusion
We study the use of demonstrations as LM prompts to convey the semantics of APIs in lieu of natural language descriptions for TOD.
While taking similar effort to construct, demonstrations outperform description-based prompts in our experiments across DST datasets (SGD and MultiWOZ), model sizes, and training data sizes, while being more robust to changes in schemata.
This work provides developers of TOD systems with more options for
API representations to enable transfer to unseen services.
In future work, we would like to explore this representation for other TOD tasks (e.g. dialogue management and response generation).
Ethical Considerations
We proposed a more efﬁcient way of building TOD systems by leveraging demonstrations in place of descriptions, leading to increased accuracy with minimal/no data preparation overhead.
We conduct our experiments on publicly-available TOD datasets in English, covering domains which are popular for building conversational agents.
We hope our work leads to building more accurate TOD systems with similar or less overhead, and encourages further research in the area
Jeffrey Zhao, Mahdis Mahdieh, Ye Zhang, Yuan Cao, and Yonghui Wu. 2021a
Effective sequence-tosequence dialogue state tracking.
In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 7486–7493, Online and Punta Cana, Dominican Republic.
Association for Computational Linguistics.
Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021b. Calibrate before use: Improving few-shot performance of language models. A SDT Model Details
All T5 checkpoints used are available publicly 2 .
For all experiments, we use a sequence length of 2048, dropout of 10% and a batch size of 16.
We used a constant learning rate of 1 e − 3 or 1 e − 4 .
All models were trained for 50k steps or until convergence, and each experiment was conducted on either 64 or 128 TPU v3 chips (Jouppi et al., 2017).
B Baseline Models
For SGD, we compare against SGP-DST
(Ruan et al., 2020), MRC+WD-DST (Ma et al., 2019), T5-seq (Zhao et al., 2022) and T5-ind (Lee et al., 2021a).
For MultiWOZ, we compare against TRADE (Wu et al., 2019), SUMBT (Lee et al., 2019), TransferQA
(Lin et al., 2021a), and T5-seq.
Transfer QA is based on T5-large, and T5-ind and T5-seq are based on T5-XXL in this paper unless otherwise noted.
C Prompt Design
We experimented with various formats for the SDT prompt before arriving at the ﬁnal format.
Below, we list alternative designs that we tried and their impact on JGA, as evaluated on the SGD test set.
C. Categorical value strings vs. multiple choice answers
We found that JGA dropped -2% when we tasked the model with decoding categorical values instead of multiple choice answers - e.g. payment_method=debit card instead of payment_method=
b (where b is linked to the value debit card in the prompt as described in
Section 2).
We found that when tasking the model to decode categorical values, it would often decode related yet invalid values, which we counted as false in our evaluation.
For example, instead of debit card , the model might decode bank balance .
C. Slot IDs vs. slot names
When we delexicalized slot names with slot IDs,
JGA dropped -5%.
One downside of this approach is that the model lost access to valuable semantic information conveyed by the slot name.
Another downside is that the model could not distinguish two slots that had the same value in the prompt.
For example, if the prompt was "I would like a petfriendly hotel room with wiﬁ" and the corresponding slots were 1=True (has_wiﬁ) and 2=True (pets_allowed), it is ambiguous which ID refers to which slot.
The potential upside of using slot IDs was to remove dependence on the choice of slot name, but this did not succeed for the reasons above.
C. Decoding active slots vs. all slots
We experimented with training the model to only decode active slots rather than all slots with none values when they were inactive.
JGA dropped0.4%, which we hypothesized could be a result of greater dissimilarity between the slot-value string in the prompt (which contained all slots by construction) and the target, which only contained a subset of slots
. C. In-line annotations vs. dialogue+slots concatenated
We hypothesized that bringing the slot annotation in the prompt closer to where it was mentioned in the dialogue might help the model better understand the slot’s semantic meaning.
We changed the format as follows:
• Original: [example] [user]
I
would like a pet-friendly hotel room with wifi [system] I found ...
[slot]
has_wifi=
True • In-line: [example] [user] I would like a pet-friendly hotel room with
wifi [has_wifi=
True] [system] I found ...
However, this decreased JGA by more than20%.
We hypothesized that this was likely due to a mismatch between the prompt’s annotations and the target string format, which remained the same. [Table]