Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics (2022)

Abstract: Recent works that revealed the vulnerability of dialogue state tracking (DST) models to distributional shifts have made holistic comparisons on robustness and qualitative analyses increasingly important for understanding their relative performance. We present our findings from standardized and comprehensive DST diagnoses, which have previously been sparse and uncoordinated, using our toolkit, CheckDST, a collection of robustness tests and failure mode analytics. We discover that different classes of DST models have clear strengths and weaknesses, where generation models are more promising for handling language variety while span-based classification models are more robust to unseen entities. Prompted by this discovery, we also compare checkpoints from the same model and find that the standard practice of selecting checkpoints using validation loss/accuracy is prone to overfitting and each model class has distinct patterns of failure. Lastly, we demonstrate how our diagnoses motivate a pre-finetuning procedure with non-dialogue data that offers comprehensive improvements to generation models by alleviating the impact of distributional shifts through transfer learning.

## introduction a crucial skill for task-oriented dialogue models, which serve as backbones to modern digital assistants, is slot filling, formally known as dialogue state tracking (dst). it requires understanding the users' intents to populate slots in api queries that request information needed to fulfill their goals. to encourage the development of dst models, various large-scale datasets that double as benchmarks have been developed, such as multiwoz @xcite , taskmaster @xcite , and figure 1 : top: performance differences on current benchmarks provide limited information on how dst models compare in robustness and various modes of failure. bottom: checkdst diagnoses paints a richer summary that highlights strengths and weaknesses through standardized and comprehensive comparisons of metrics that isolate various perturbations and failure modes. higher is better for all metrics. as shown in the bar chart on the top side of figure 1 , recent state-of-the-art approaches in dst leaderboards such as multiwoz @xcite ) are compared to one another on a single metric that provides limited information and their modes of failure. moreover, since these models demonstrate a sharp drop in joint goal accuracy (jga), the average accuracy of correctly predicting all slots for a dialogue turn, when exposed with realistic distributional shifts @xcite @xcite , it is increasingly important to shift the focus from answering the question "is one dst model better than another?" to "how does one dst model compare to another?" through robustness tests and qualitative analyses to understand relative performance. yet these efforts have mostly been sparse and uncoordinated, precluding in-depth comparisons and opportunities to discover improvements that combine strengths from different models. to address this gap, we first consolidate previous efforts in robustness testing and qualitative analyses to design our toolkit, checkdst foot_0 . checkdst facilitates comprehensive diagnosis by quantifying robustness with augmented test sets that represent various perturbations in isolation and measuring the frequency of common failure modes. for measuring robustness, we introduce consistency jga (cjga) to place more emphasis on prediction consistency and obviate the assumption that the perturbations should be more difficult than their corresponding original samples. with a checkdst diagnosis, we can quickly examine a rich comparison summary that highlights relative performance in multiple dimensions, as illustrated in the bottom of figure 1 . evaluating a subset of models from two major classes of state-of-the-art models, span-based classification models and simpletod-style generation models (henceforth denoted as classification models and generation models, respectively), on the multi-woz @xcite ) leaderboard, we demonstrate the value of our diagnosis through the revelation that robustness is not proportionate to higher jga and that the model classes have clearly different strengths and weaknesses. in particular, while classification models attain higher jga, generation models are significantly more robust to various perturbations except to swapped named entities. this disconnect between jga and robustness found in inter-model comparisons naturally prompts comparisons between different checkpoints from the same model. by diagnosing multiple intermediate checkpoints with checkdst, we show that the current standard practice of selecting a checkpoint according to performance on the validation set is prone to overfitting, regardless of model class. finally, we demonstrate how the findings from our diagnoses with checkdst can help develop more robust dst models with strategies that alleviate the trade-off between jga and robustness and mitigate the failure modes. we share our experiments with prefinedst, building on the robustness of generation models via a pre-finetuning procedure with instruction prompts, similarly to the t0 model @xcite , with non-dialogue tasks to acquire skills that should intuitively boost robustness. we show that comprehensive improvements that reduce the trade-off between jga and robustness can be achieved without directly exposing the model to similar perturbations used in the test sets during training. in summary, our contributions include: checkdst diagnosis that achieves a comprehensive improvement in robustness and jga through a multitasking pre-finetuning step with instruction-prompts. ## dst diagnostics with checkdst to conduct standardized robustness tests and qualitative analyses, we first consolidate previous efforts to design checklist for dialogue state tracking (checkdst), a toolkit for quantifying robustness with a collection of augmented test sets and diagnostics that help identify commonly known failure modes. in this section, we introduce consistent jga (cjga) and summarize the perturbations and failure modes of interest that are shown in ## prefinedst the weaknesses exposed by checkdst guide us toward approaches that can boost robustness without compromising jga. inspired by the strong results of massive multi-task learning on many nlp tasks in recent work, such as muppet @xcite , t0 @xcite and flan @xcite , and the ease of expanding a generation model to end-to-end task-oriented dialogue @xcite , we explore a simple multi-tasking approach for a generation model that we name prefinedst, short for pre-finetuned dst, to train a more well-rounded dst model. as shown in figure 3 , prefinedst first prefinetunes a pre-trained bart-large model with the same method as t0, but with a narrower set of tasks that can address the robustness issues captured by checkdst. we select tasks that require understanding paraphrases, generating exact spans of text from the context, and resolving coreferences, with the expectation that skills required to perform well on them will be transferred to the fine-tuning step on a downstream dst task and eventually be reflected in better scores on checkdst. details on the chosen tasks and implementation details can be found in appendix c. pre-finetuning with non-target data is a promising avenue for a robust dst model. overall, results in prefinedst is most directly comparable to soloist and muppet in that they all incorporate a pre-finetuning step. thus, it is notable that prefinedst achieves a higher jga than both while simultaneously achieving comparable or better results in all dimensions measured by checkdst, except for f swap . this indicates some degree of successful knowledge transfer from the pre-finetuning tasks, but the relatively lower f swap signals that generalizing to correctly copying unseen entities through pre-finetuning is ineffective. in conclusion, using a simple and intuitive approach, prefinedst shows that pre-finetuning with non-target datasets is a promising direction for boosting robustness. we leave it to future work to leverage checkdst as a guide to explore more sophisticated pre-finetuning strategies and non-target tasks to improve on prefinedst, especially for handling unseen entities. ## related work pretrained language models continue to make impressive strides on nlp benchmarks, surpassing human baseline scores on many of them @xcite @xcite @xcite . these results lead to questions of whether these models are acquiring the intelligence required for their performance to be robust or instead are taking advantage of spurious correlations @xcite . many works show that the latter is the case and seek adversarial techniques to test these models to new limits @xcite @xcite and train them to be more robust @xcite @xcite . robustness in dialogue models has also been similarly questioned. perturbations to the dialogue history have exposed that dialogue models do not effectively use dialogue structure information @xcite and commonsense probes show that they struggle with commonsense reasoning @xcite . specifically for the dialogue state tracking task, several works report drops in performance for conversations with entities unseen during training @xcite @xcite or with adversarially created dialogue flows @xcite . @xcite and @xcite recently initiated a rigorous study into the robustness of tod models to realistic natural language perturbations. we extend their work to establish a framework that further facilitates robustness analysis with additional metrics that capture coreference resolution performance and frequency of well-known problems to generation models. moreover, we propose cjga, a simple yet rigorous metric that enables measuring robustness in dst without making assumptions about the difficulty of perturbations. prefinedst is motivated by the recent line of work that uses generation models for dst. sim-pletod @xcite first reported viability of formulating tod tasks in a completely end-to-end manner with a generation model and soloist @xcite added a pretraining step to improve on data efficiency. prefinedst, inspired by recent work on impressive results from massive multi-tasking prefinetuning @xcite @xcite , extends simpletod and soloist by adding more prefinetuning tasks. concurrent work from @xcite explore a similar procedure for taskoriented dialogue, but their analysis is centered around zero-and few-shot generalization. ## conclusion we shared our findings enabled by checkdst, a toolkit that standardizes a comprehensive diagnosis of dst models. we confirm that benchmark results are insufficient for understanding relative performance as relative robustness is not correlated to benchmark performance. we find generation models to be a more promising direction as they are significantly more robust to language variety without making any task-specific design choices. we also observed that the standard practice of choosing a checkpoint with validation loss or accuracy is prone to overfitting, exacerbating the trade-off between jga and robustness for both classification and generation models. finally, we use the robustness issues exposed by checkdst to guide the development of prefinedst, a model that attains well-rounded improvements through a prefinetuning step that multi-tasks on reasoning skills. moving forward, we encourage future work on task-oriented dialogue to conduct checkdst diagnosis for a comprehensive comparison and analysis of dst performance, which we believe will make the search for ideas to build robust task-oriented dialogue models significantly more efficient.