Preview, Attend and Review: Schema-Aware Curriculum Learning for Multi-Domain Dialogue State Tracking (2021)

Abstract: Existing dialog state tracking (DST) models are trained with dialog data in a random order, neglecting rich structural information in a dataset. In this paper, we propose to use curriculum learning (CL) to better leverage both the curriculum structure and schema structure for task-oriented dialogs. Specifically, we propose a model-agnostic framework called Schema-aware Curriculum Learning for Dialog State Tracking (SaCLog), which consists of a preview module that pre-trains a DST model with schema information, a curriculum module that optimizes the model with CL, and a review module that augments mispredicted data to reinforce the CL training. We show that our proposed approach improves DST performance over both a transformer-based and RNN-based DST model (TripPy and TRADE) and achieves new state-of-the-art results on WOZ2.0 and MultiWOZ2.1.

## introduction dialog state tracking (dst) extracts users' goals in task-oriented dialog systems, where dialog states are often represented in terms of a set of slot-value pairs @xcite . due to the language variety of multi-turn dialogs, the concepts of slots and values are often indirectly expressed in the conversation (such as co-references, ellipsis, and diverse appearances), which are a major bottleneck for improving dst performance @xcite . many existing dst methods have focused on designing better model architectures to tackle the problems @xcite @xcite , but still neglect the full exploitation of two important aspects of structural information. the first is curriculum structure in a dataset. such a structure relies on a measure of the difficulty of examples, which can be used to guide the model training in an easy-to-hard manner, imitating the meaningful learning order in human curricula. this paradigm is called curriculum learning (cl) @xcite and has been shown useful in various other problems @xcite . dst training examples also vary greatly in their difficulty levels. as shown in figure 1 , for the same slot 'taxi-departure', a user can either inform its value 'nandos' explicitly in a simple utterance or convey her intention implicitly via multi-round interactions, requiring a complex inference process to find the value 'golden house' referred by the slot 'restaurant-name'. however, cl has been rarely studied in dst, and models are often trained with dialog data in a random order. in addition, schema structure is prominent in multi-domain task-oriented dialogs. a schema is specified by a collection of all possible slots and their values, which describes semantic relations among them. some previous work utilized the structure via an extra schema graph in a regular training process @xcite @xcite . we propose to incorporate schema information into cl through a pre-curriculum process, in which a dst model can be pre-trained with schema-related objectives to prepare for upcom-ing dst examples. to reinforce the cl training, we can also expand those examples with frequent mispredictions during cl based upon the schema, enabling the model to accumulate more experience and perform better on similar cases. built on these motivations, we propose a novel framework named as schema-aware curriculum learning for dialog state tracking (saclog), which consists of three components: 1) a preview module that pre-trains the base part of a dst model (e.g., bert and rnn) with objectives capturing the connections between the schema and dialog contexts, 2) a curriculum module that organizes training data from easy to hard and optimizes the model with cl, and 3) a review module which leverages schema-based data augmentation to extend mispredicted data to boost the cl training process further. the proposed approach is model-agnostic, in the sense that it can be incorporated into different dst models. to the best of our knowledge, this is the first attempt to apply cl to the dst task. we show that our proposed approach improves dst performance over both a transformer-based and rnn-based dst model (trippy and trade) and achieves new state-ofthe-art results on woz2.0 and multiwoz2.1. ## problem formulation we denote a dialog context containing t turns as x @xmath0, u 1 ), ..., (r t , u t )}, where r i and u i represent system and user utterance at the i-th turn respectively. dst is tasked to extract turn-level or discourse-level dialog states in the form of a set of slot-value pairs given x t . a turn-level dialog state y @xmath1, v t ), @xmath2, u t ) at current turn t, where s is a predefined set of slot s in the schema and v t is the corresponding value foot_0 of the slot s. a discourse-level dialog state z t is the accumulation of l t , representing all slot-value pairs that have been expressed over the course of the dialog until the t-th turn. we denote a dialog data for dst as d @xmath3, y t , z t } and the training dataset as d. ## schema-aware curriculum learning in this section, we first introduce the core curriculum module about how to apply the basic cl to the dst task; we then describe the preview and review module, which exploit the schema structure to facilitate the cl training process. the overall framework of saclog is shown in figure 2 . ## experiments two popular datasets, woz2.0 @xcite and multiwoz2.1 @xcite , are used to verify our approach. woz2.0 is a single-domain dataset with 1,200 dialogs and 3 slots. multi-woz2.1 is a multi-domain dialog dataset with 10,438 dialogs, where there are 30 slots spanning 7 domains. the data splits (train/valid/test) of woz2.0 and multiwoz2.1 are 600/200/400 and 8438/1000/1000, respectively. we use the joint goal accuracy (jga), the ratio of dialog data whose z t is correct, as the evaluation metric. we apply saclog onto trippy @xcite , a transformer-based dst model, and trade @xcite , an rnn-based dst model, to show its effect. the slot encoder and the dialog context encoder are weight-shared. we use a bert base as the encoder and the [cls] embedding as the slot embedding in trippy, and use a bi-gru as the encoder and the concatenation of the first and last hidden state as the slot embedding for trade. we also follow trippy to add 2 new slot operations (i.e. refer/dontcare) into the classification types of l cls . implementation details. for the preview module, we use adam (kingma and ba, 2015) with a fixed learning rate 3e-5 for 3 epochs in the pretraining. the batch size for l aux is 14 and the batch size for l seq and l cls is 64. for the curriculum module, we perform a warm-up strategy for adam optimizer with a maximum learning rate 1e-4. before cl, we train models on full dataset for 2 epochs. after all subsets are accumulated, we then train for 10 extra epochs with a minimum learning rate 1e-6. we set the bucket number @xmath4. the batch size is 36 and models multiwoz2.1 woz2.0 glad @xcite 35.57% * * 88.1±0.4% sumbt @xcite 46.65% * * 91.0±1.0% dst-picklist @xcite 53.30% -trippy @xcite 55.29±0.28% 92.7±0.2% simpletod @xcite 55.72% -chan @xcite 58.55% -trippy + convbert 58.70% 93.1±0.3% the maximum length is 256. to simplify the review process, we conduct data augmentation after the cl training is finished. ## related work curriculum learning (cl) has attracted increasing research interests in various nlp tasks, such as machine translation @xcite , general language understanding @xcite , reading comprehension @xcite and open-domain chatbots @xcite @xcite . yet, the research on using cl in task-oriented dialog systems is limited. there has been some work @xcite on using cl in dialog policy learning, but applying cl to dst has not been investigated. learning a structural inductive bias during pretraining has been shown beneficial in downstream tasks that require parsing semantics, such as textto-sql @xcite and table cell recognition @xcite . there are also many works @xcite @xcite on dialog augmentation. we aim to integrate these methods to build a general cl framework for dst. ## conclusion in this paper, we propose a model-agnostic framework named as schema-aware curriculum learning for dst, which exploits both the curriculum structure and the schema structure in task-oriented dialogs and shows to substantially improve dst performances. in the future, we plan to investigate cl approaches on other dialog modeling tasks.