More Robust Schema-Guided Dialogue State Tracking via Tree-Based Paraphrase Ranking (2023)

Abstract: The schema-guided paradigm overcomes scalability issues inherent in building task-oriented dialogue (TOD) agents with static ontologies. Rather than operating on dialogue context alone, agents have access to hierarchical schemas containing task-relevant natural language descriptions. Fine-tuned language models excel at schema-guided dialogue state tracking (DST) but are sensitive to the writing style of the schemas. We explore methods for improving the robustness of DST models. We propose a framework for generating synthetic schemas which uses tree-based ranking to jointly optimise lexical diversity and semantic faithfulness. The robust generalisation of strong baselines is improved when augmenting their training data with prompts generated by our framework, as demonstrated by marked improvements in average Joint Goal Accuracy (JGA) and schema sensitivity (SS) on the SGD-X benchmark.

## introduction dst is concerned with tracking user goals in taskoriented conversations. the goals are represented as key-value pair sequences, with the keys known as slots (e.g. hotel name). pre-trained language models (plms) @xcite have helped shift focus from systems that can only track slots drawn from a database or domain ontology @xcite to models that do not require re-training to parse goals in new domains. the schema-guided dialogue (sgd) dataset @xcite facilitates this shift with a large-scale set of conversations grounded in 45 service apis or schemas that describe the domains, slots and user intents that annotate the conversations (appendix a). test set dialogues are grounded in 6 schemas seen during training and 15 unseen ones. neural models perform impressively on the difficult schema-guided dst task @xcite , 1 code will be released here: https://bit.ly/3wyb7fl but @xcite show that the uniformity of the descriptive language of the schemas facilitates this. they create the sgd-x benchmark to evaluate robust zero-shot generalisation of dst models. this is achieved by grounding the sgd test set conversations in five schema variants increasingly dissimilar to the sgd schemata foot_0 . to perform well, a dst model should correctly track the state of a dialogue when conditioned, in turn, on prompts constructed from the five variants. we show how to improve dst robustness by introducing controlled variability in the data. we contribute to robust dst research by (1) a flexible framework for generating and ranking diverse outputs of a paraphrase model based on a treeclustering algorithm designed to control lexical diversity and semantic similarity; (2) combine stateof-the-art paraphrase models and language generation metrics to generate increasingly diverse schemata paraphrases; (3) show that augmenting the training dataset with these schemata improves the robustness and generalisation performance of strong dst baselines. ## related work input variety, data scarcity and domain shifts affect the robustness of dst models. @xcite investigate the former. they employ word-level data augmentation (da) @xcite , turn paraphrasing and speech disfluency modeling to approximate their field performance. turn and dialogue generation are effective in low-resource settings @xcite but are very difficult to scale to new domains and are not effective in the high-resource setting we consider @xcite . this also applies to word-and sentence-level meth-ods @xcite . @xcite find word-order changes and deletions to be ineffective in the high-resource, schema-guided setting we consider. schema-guided dst tackles both data scarcity and novel domains by using api definitions to prompt plms @xcite ). yet @xcite demonstrate the lack of robustness of schema-guided dst models to prompt styles and vocabulary, creating a new research direction. they show that augmenting the training data with synthetic prompts obtained via backtranslation significantly improves models' ability to track states under meaning-preserving prompt transformations. backtranslation is also applied to improve dst robustness to linguistic variation inherent in user communication @xcite , which is orthogonal to the prompt style and vocabulary robustness setting we consider. reinforcement learning has also been applied @xcite , but works only in the very constrained single-domain, ontology-driven setting. other tod-relevant da approaches apply to policy learning @xcite and responsegeneration @xcite . addressing the dearth of augmentation methods designed to ensure prompt robustness of schemaguided dst models, we propose to generate schemas by ranking large paraphrase candidate lists with learned metrics in a tree ranking scheme. ## tree-based paraphrase ranking tree construction a large pool of schema candidates is created by generating paraphrases given grids of generation parameters (eg temperature, number of beams). the set is filtered to address generation failures (eg toxic and hallucinated words). we optionally filter candidates with an entailment model to increase semantic faithfulness @xcite (see appendix b.1). the tree constructor (algorithm 1) takes as input an object (node) that stores a metric value, val, and the candidate paraphrases which are split at that node, sents. a list of metrics to be computed between each candidate and the input is provided by the user. this enables our framework to build arbitrary-depth trees with custom user metrics. each unique list of metric values describing the distance between the input and a candidate generates a path in the tree (lines 5-13). the n-ary tree constructed in this way has the property that level-order traversal of the first level can yield diverse candidates with respect to the metric it encodes. in practice, the metrics measure lexical and semantic distances between their inputs. algorithm 1: tree building 1: def build_tree(root: node, inp: str, cands: list[str], metrics: list[callable]): data: root, inp input , cands input paraphrases, metrics objects to eval. dist. between input & cand. result: tree splitting cands according to metrics 2: curr ← root ; 3: for c in cands: 4: curr ← root ; 5: for m in metrics: 6: @xmath0, c) ; 7: next ← get_child (curr.children, m_val) ; 8: if next is null: 9: next ← node (@xmath1, @xmath2; 10: curr.children.add(next) 11: else: 12: next.sents.add(c) ; 13: curr ← next 14: ranking our ranker input is the tree and a list of decision functions, with elements corresponding to each level in the tree, f_dec. without loss of generality, we assume that the first level encodes a metric with respect to which the user wishes to maximise diversity (eg lexical distance). as shown in figure 1 , our algorithm traverses breadth-first the level for which diversity is to be maximised. each subtree returned in the traversal is traversed depthfirst, guided by the decision functions. for example, in figure 1 we show that the node @xmath3.77 is selected by applying the max decision function to the children of @xmath4, and that applying min to the children of @xmath5.77 selects the leaf @xmath6. see algorithm 3 (appendix b) for details. ## conclusion and future work we presented a simple tree-based ranking algorithm for optimising lexical diversity and semantic faithfulness during schema generation. the synthetic schemas improve both the dst models' robustness to schemata writing style and their generalisation. our framework will allow researchers working on paraphrase generation and semantic faithfulness to measure the generalisation of their models in a way that may be difficult to capture by existing benchmarks: it can generate schemata paraphrases and train sota dialogue state trackers which were shown to benefit from augmentation with high quality, crowdsourced paraphrases.