A Comparative Study on Schema-Guided Dialogue State Tracking (2021)

Abstract: Frame-based state representation is widely used in modern task-oriented dialog systems to model user intentions and slot values. However, a fixed design of domain ontology makes it difficult to extend to new services and APIs. Recent work proposed to use natural language descriptions to define the domain ontology instead of tag names for each intent or slot, thus offering a dynamic set of schema. In this paper, we conduct in-depth comparative studies to understand the use of natural language description for schema in dialog state tracking. Our discussion mainly covers three aspects: encoder architectures, impact of supplementary training, and effective schema description styles. We introduce a set of newly designed bench-marking descriptions and reveal the model robustness on both homogeneous and heterogeneous description styles in training and evaluation.

## introduction from early frame-driven dialog system gus @xcite to virtual assistants (alexa, siri, and google assistant et al.), frame-based dialog state tracking has long been studied to meet various challenges. in particular, how to support an everincreasing number of services and apis spanning multiple domains has been a focal point in recent years, evidenced by multi-domain dialog modeling @xcite @xcite and transferable dialog state tracking to unseen intent/slots @xcite @xcite . recently, @xcite proposed a new paradigm called schema-guided dialog for transferable dialog state tracking by using natural language description to define a dynamic set of service schemata. as shown in figure 1 , the primary motivation is that these descriptions can offer effective knowledge sharing across different services, e.g., connecting semantically similar concepts across heterogeneous apis, thus allowing a unified model to handle unseen services and apis. with the publicly available schema-guided dialog dataset (sg-dst henceforward) as a testbed, they organized a state tracking shared task composed of four subtasks: intent classfication (intent), requested slot identification (req), categorical slot labeling (cat), and noncategorical slot labeling (noncat) @xcite . many participants achieved promising performance by exploiting the schema description for dialog modeling, especially on unseen services. despite the novel approach and promising results, current schema-guided dialog state tracking task only evaluates on a single dataset with limited variation in schema definition. it is unknown how this paradigm generalizes to other datasets and other different styles of descriptions. in this paper, we focus our investigation on the study of three aspects in schema-guided dialog state tracking: (1) schema encoding model architectures (2) supplementary training on intermediate tasks (3) various styles for schema description. to make a more general discussion on the schema-guided dialog state tracking, we perform extensive empirical studies on both sg-dst and multiwoz 2.2 datasets. in summary, our contributions include: ## schema-guided dialog state tracking a classic dialog state tracker predicts a dialog state frame at each user turn given the dialog history and predefined domain ontology. as shown in figure 1 , the key difference between schema-guided dialog state tracking and the classic paradigm is the newly added natural language descriptions. in this section, we first introduce the four subtasks and schema components in schema-guided dialog state tracking, then we outline the research questions in our paper. subtasks. as shown in figure 1 , the dialog state for each service consists of 3 parts: active intent, requested slots, user goals (slot values). without loss of generality, for both sg-dst and multiwoz 2.2 datasets, we divide their slots into categorical and non-categorical slots by following previous study on dual-strategies @xcite . thus to fill the dialog state frame for each user turn, we solve four subtasks: intent classification (intent), requested slot identification (req), categorical slot labeling (cat), and non-categorical slot labeling (noncat). all subtasks require matching the current dialog history with candidate schema descriptions for multiple times. schema components. figure 1 shows three main schema components: service, intent, slot. for each intent, the schema also describes optional or required slots for it. for each slot, there are flags indicating whether it is categorical or not. categorical means there is a set of predefined candidate values (boolean, numeric or text). for instance, has_live_music in figure 1 is a categorical slot with boolean values. non-categorical, on the other hand, means the slot values are filled from the string spans in the dialog history. new questions. these added schema descriptions pose the following three new questions. we discuss each of them in the following sections. q1. how should dialogue and schema be encoded? §5 q2. how do different supplementary trainings impact each subtask? §6 q3. how do different description styles impact the state tracking performance? §7 our work is related to three lines of research: multisentence encoding, multi-domain and transferable dialog state tracking. however, our focus is on the comparative study of different encoder architectures, supplementary training, and schema description style variation. thus we adopt existing strategies from multi-domain dialog state tracking. ## datasets to the best of our knowledge, at the time of our study, sg-dst and multiwoz 2.2 are the only two publicly available corpus for schema-guided dialog study. we choose both of them for our study. in this section, we first introduce these two representative datasets, then we discuss the generalizibility in domain diversity, function overlapping, data collecting methods. ## dialog & schema representation and inference (q1) in this section, we focus on the model architecture for matching dialog history with schema descriptions using pretrained bert @xcite 3 . to support four subtasks, we first extend dual-encoder and cross-encoder to support both sentence-level matching and token-level prediction. then we propose an additional fusion-encoder strategy to get faster inference without sacrificing much accuracy. we summarize different architectures in figure 2 . then we show the classification head and results for each subtask. 3 we use bert-base-cased for all main experiments. other pretrained language models can be easily adapted to our study figure 2: dual-encoder, cross-encoder and fusion encoder, shaded block will be cached during training ## supplementary training (q2) besides the pretrain-fintune framework used in §5, @xcite propose to add a supplementary training phase on an intermediate task after the pretraining, but before finetuning on target task. it shows significant improvement on the target tasks. moreover, large amount pretrained and finetuned transformer-based models are publicly accessible, and well-organized in model hubs for sharing, training and testing foot_3 . given the new task of schemaguided dialog state tracking, in this section, we study our four subtasks with different intermediate tasks for supplementary training. ## impact of description styles (q3) previous work on schema-guided dialog @xcite are only based on the provided descriptions in sg-dst dataset. recent work on modeling dialog state tracking as reading comprehension @xcite only formulate the descriptions as simple question format with existing intent/slot names, it is unknown how it performs when compared to other description styles. moreover, they only conduct homogeneous evaluation where training and test data share the same descrip-tion style. in this section, we also investigate how a model trained on one description style will perform on other different styles, especially in a scenario where chat-bot developers may design their own descriptions. we first introduce different styles of descriptions in our study, and then we train models on each description style and evaluate on tests with corresponding homogeneous and heterogeneous styles of descriptions. given the best performance of cross-encoder shown in the previous section and its popularity in dstc8 challenges, we adopt it as our model architecture in this section. ## conclusion in this paper, we studied three questions on schemaguided dialog state tracking: encoder architectures, impact of supplementary training, and effective schema description styles. the main findings are as follows: by caching the token embedding instead of the single cls embedding, a simple partial-attention fusion-encoder can achieve much better performance than dual-encoder, while still infers two times faster than cross-encoder. we quantified the gain via supplementary training on two intermediate tasks. by carefully choosing representative description styles according to recent works, we are the first of doing both homogeneous/heterogeneous evaluations for different description style in schema-guided dialog. the results show that simple name-based description performs well on intent and req tasks, while noncat tasks benefits from richer styles of descriptions. all tasks suffer from inconsistencies in description style between training and test, though to varying degrees. our study are mainly conducted on two datasets: sg-dst and multiwoz 2.2, while the speedaccuracy balance of encoder architectures and the findings in supplementary training are expected to be dataset-agnostic, because they depend more on the nature of the subtasks than the datasets. based on our proposed benchmarking descriptions suite, the homogeneous and heterogeneous evaluation has shed the light on the robustness of cross-style schema-guided dialog modeling, we believe our study will provide useful insights for future research.