Grounding Description-Driven Dialogue State Trackers with Knowledge-Seeking Turns (2023)

Abstract: Schema-guided dialogue state trackers can generalise to new domains without further training, yet they are sensitive to the writing style of the schemata. Augmenting the training set with human or synthetic schema paraphrases improves the model robustness to these variations but can be either costly or difficult to control. We propose to circumvent these issues by grounding the state tracking model in knowledge-seeking turns collected from the dialogue corpus as well as the schema. Including these turns in prompts during finetuning and inference leads to marked improvements in model robustness, as demonstrated by large average joint goal accuracy and schema sensitivity improvements on SGD and SGD-X.

## introduction task-oriented dialogue (tod) agents provide natural language interfaces that users can interact with to access a wide variety of services, from airline search @xcite to complex customer service applications @xcite . to enable this, agents track key information communicated by the user as the conversation progresses. this is known as dialogue state tracking (dst). commonly, the dialogue state is represented as a sequence of task-specific slot-value pairs foot_1 . a common dst assumption is that the set of slots and values a user may communicate, the domain ontology, is known at design time. hence, extensive data collection and annotation is needed to support new domains, which hinders the scalability of this approach. @xcite address this issue by creating the schema-guided dialogue dataset (sgd). in sgd, the information available to a tod agent is organised as schemas foot_2 describing services with which users can interact. each service has user intents representing the tasks users can complete by interacting with the agent (e.g. find restaurants). several slots are associated with each intent and the schema provides a natural language description for each intent and slot. the insight motivating description-driven dst is that these descriptions alone can be used in tracking the dialogue state in a form close to natural language. this has emerged as a powerful approach for few-shot and zero-shot dst @xcite and benefits from recent advances in language modelling. for example, @xcite finetune t5 @xcite to generate the dialogue state conditioned on the dialogue history and a descriptive prompt containing all intent and slot descriptions in a service schema. the use of natural language in the descriptive prompt enables the underlying language model to generalise to new services, whose schemas are not seen in training. while the reliance on natural language is a strength, @xcite show that state-of-theart (sota) schema-guided dst models are not robust to the style of descriptive prompts: in sgd, the training schema contains a single description per slot or intent, and models trained with prompts composed from this schema alone are prone to overfitting. @xcite show this limitation can be mitigated by increasing prompt diversity. they use a large number of human annotators alongside expert curation to create diverse schema paraphrases that are used for model robustness improvement. this is a costly process that is not easily scalable. as an alternative to additional human annotation of the schema, we show that the sgd training dialogues themselves exhibit sufficient diversity of expression such that they can be used to overcome the lack of diversity in the sgd schema descriptions. we ground dst prompts in the dialogue context by concatenating the schema descriptions with dialogue turns extracted from the sgd cor- seq2seq (t5) [state] 1=tomorrow [intent] i1) d3st target dialogue states d3st encoder input 0=name of the city 1=date for the weather i1) get the weather of a certain location on a date slots: city: name of the city date: date for the weather intents: getweather: get the weather of a certain location on a date weather_1 schema [usr] what's the weather like tomorrow? ## related work neural classification is effective for dst when the domain ontology is fixed and known at design time @xcite . adapting such models to track new slots and domains requires annotated conversational data and thus data scarcity is a long-standing issue in dst research @xcite . scarcity has been addressed by copyenhanced generation @xcite , reading comprehension @xcite and adapting pretrained language models to generate the state given the dialogue context alone @xcite . these were improved upon by transfer learning from question-answering tasks @xcite , which in turn was outperformed by schema-guided models @xcite @xcite . recently, @xcite apply in-context tuning @xcite to dst, creating training prompts which contain a dialogue and its target state sequence. their model thus learns from dst task demonstrations. @xcite investigate the robustness of sota schema-guided dialogue state trackers to schema changes. this is a new line of research, as previous work concerns other robustness issues that generally affect dst, such as variations in the conversational data distribution, noise, and adversarial perturbations @xcite . through extensive, crowdsourced foot_3 , schema paraphrase collection, @xcite report that dst performance degrades substantially when models trained on one set of prompts are evaluated on manually paraphrased prompts. by contrast, @xcite report little degradation in dst with backtranslated prompts, suggesting that backtranslation is a weak proxy for actual human variability. @xcite perform data augmentation (da) for robust dst, finding that prompts obtained via automatic paraphrasing lag in quality relative to manual paraphrases. ours is the first work to address the gap between synthetic methods, such as backtranslation, and manual paraphrasing. we show that the gains from manual paraphrasing can be achieved by mining the existing annotated dialogues used for training the dst model in the first place. ## robust dst with grounded prompts we review d3st, a sota description-driven dst model (section 3.1). we then describe our grounding method that extracts turns from the corpus (section 3.2) and uses them to design prompts for robust dst with d3st (sections 3.3 & 3.4). ## conclusion grounding d3st and data augmentation with knowledge-seeking turns are effective for robust schema-guided dst. both improve d3st robustness by a large margin compared to strong baselines and yield similar benefits as training on large, diverse collection of human-written prompts. our proposed approach is competitive with or outperforms other sota dst models on sgd and sgd-x. we have also showed how prompt engineering can be applied to boost model robustness through grounded prompt ensembling, a novel technique that uses a single model for ensembling.