From Machine Reading Comprehension to Dialogue State Tracking: Bridging the Gap (2020)

Abstract: Dialogue state tracking (DST) is at the heart of task-oriented dialogue systems. However, the scarcity of labeled data is an obstacle to building accurate and robust state tracking systems that work across a variety of domains. Existing approaches generally require some dialogue data with state information and their ability to generalize to unknown domains is limited. In this paper, we propose using machine reading comprehension (RC) in state tracking from two perspectives: model architectures and datasets. We divide the slot types in dialogue state into categorical or extractive to borrow the advantages from both multiple-choice and span-based reading comprehension models. Our method achieves near the current state-of-the-art in joint goal accuracy on MultiWOZ 2.1 given full training data. More importantly, by leveraging machine reading comprehension datasets, our method outperforms the existing approaches by many a large margin in few-shot scenarios when the availability of in-domain data is limited. Lastly, even without any state tracking data, i.e., zero-shot scenario, our proposed approach achieves greater than 90% average slot accuracy in 12 out of 30 slots in MultiWOZ 2.1.

## introduction building a task-oriented dialogue system that can comprehend users' requests and complete tasks on their behalf is a challenging but fascinating problem. dialogue state tracking (dst) is at the heart of task-oriented dialogue systems. it tracks the state of a dialogue during the conversation between a user and a system. the state is typically defined as the (slot name, slot value) pair that represents, given a slot, the value that the user provides or system-provided value that the user accepts. despite the importance of dst in task-oriented dialogues systems, few large datasets are available. to address this issue, several methods have been proposed for data collection and bootstrapping the dst system. these approaches either utilize wizard-of-oz setup via crowd sourcing @xcite or machines talking to machines (m2m) framework @xcite . currently the most comprehensive dataset with state annotation is multiwoz @xcite , which contains seven domains with around 10, 000 dialogues. however, compared to other nlp datasets, multiwoz is still relatively small, especially for training data-intensive neural models. in addition, it is also a non-trivial to get a large amount of clean labeled data given the nature of task-oriented dialogues @xcite . another thread of approaches have tried to utilize data in a more efficient manner. these approaches @xcite usually train the models on several domains and perform zero-shot or few-shot learning on unseen domains. however, these methods require slot definitions to be similar between the training data and the unseen test data. if such systems are given a completely new slot type, the performance would degrade significantly. therefore, these approaches still rely on considerable amount of dst data to cover a broad range of slot categories. we find machine reading comprehension task (rc) @xcite as a source of inspiration to tackle these challenges. the rc task aims to evaluate how well machine models can understand human language, whose goals are actually similar to dst. ultimately, dst focuses on the contextual understanding of users' request and inferring the state from the conversation, whereas rc focuses on the general understanding of the text regardless of its format, which can be either passages or conversations. in addition, recent advances have shown tremendous success in rc tasks. thus, if we could formulate the dst task as a rc task, it could benefit dst in two aspects: first, we could take advantage of the fastgrowing rc research advances; second, we could make use of the abundant rc data to overcome the data scarcity issue in dst task. building upon this motivation, we formulate the dst task into an rc task by specially designing a question for each slot in the dialogue state, similar to @xcite . then, we divide the slots into two types: categorical and extractive, based on the number of slot values in the ontology. for instance, in multiwoz, slots such as parking take values of {yes, no, don't care} and can thus be treated as categorical. in contrast, slots such as hotel-name may accept an unlimited number of possible values and these are treated as extractive. accordingly, we propose two machine reading comprehension models for dialogue state tracking. for categorical slots, we use multiple-choice reading comprehension models where an answer has to be chosen from a limited number of options. and for the extractive dialogue state tracking, span-based reading comprehension are applied where the answer can be found in the form of a span in the conversation. to summarize our approach and contributions: thirdly, in zero-shot setting where no state tracking data is used for training, our models still achieve considerable average slot accuracy. more concretely, we show that 13 out of 30 slots in multiwoz 2.1 can achieve an average slot accuracy of greater than 90% without any training. ## related works traditionally, dialogue state tracking methods @xcite @xcite @xcite assume a fully-known fixed ontology for all slots where the output space of a slot is constrained by the values in the ontology. however, such approaches cannot handle previously unseen values and do not scale well for slots such as restaurantname that can take potentially unbounded set of values. to alleviate these issues, rastogi et al. (2017); goel et al. (2018) generate and score slot-value candidates from the ontology, dialogue context ngrams, slot tagger outputs, or a combination of them. however, these approaches suffer if a reliable slot tagger is not available or if the slot value is longer than the candidate n-grams. xu and hu (2018) proposed attention-based pointing mechanism to find the start and end of the slot value to better tackle the issue of unseen slot values. gao et al. (2019) proposed using a rc framework for state tracking. they track slot values by answering the question "what is the value of the slot?" through attention-based pointing to the dialogue context. chao and lane (2019); rastogi et al. (2019) utilize bert to encode the dialogue context and then point to slot-value span in the encoded context. although these approaches are more practical and scalable, they suffer when the exact slot value does not appear in the context as expected by the backend database or if the value is not pointable. more recently, hybrid approaches have attempted to combine the benefits of both using predefined ontology (closed vocabulary) and dynamically generating candidate set or pointing (open vocabulary) approaches. goel et al. (2019) select between the two approaches per slot based on dev set. wu et al. there are recent efforts into building or adapting dialog state tracking systems in low source data scenarios @xcite ; @xcite . the general idea in these approaches is to treat all but one domain as in-domain data and test on the remaining unseen domain either directly (zero shot) or after fine-tuning on small percentage (1%-10%) of the unseen domain data (few shot). a major drawback of these approaches is that they require several labeled in-domain examples in order perform well on the unseen domain. this limits these approaches to in-domain slots and slot definitions and they do not generalize very well to new slots or completely unseen target domain. this also requires large amount of labeled data in the source domain, which may not be available in realworld scenario. our proposed approach, on the other hand, utilizes domain-agnostic qa datasets with zero or a small percentage of dst data and significantly outperforms these approaches in lowresource settings. ## conclusion task-oriented dialogue systems aim to help users to achieve a variety of tasks. it is not unusual to have hundreds of different domains in modern taskoriented virtual assistants. how can we ensure the dialogue system is robust enough to scale to different tasks given limited amount of data? some approaches focus on domain expansion by training on several source domains and then adapting to the target domain. while such methods can be successful in certain cases, it is hard for them to generalize to other completely different out-of-domain tasks. machine reading comprehension provides us a clear and general basis for understanding the context given a wide variety of questions. by formulating the dialogue state tracking as reading comprehension, we can utilize the recent advances in reading comprehension models. more importantly, we can utilize reading comprehension datasets to mitigate some of the resource issues in task-oriented dialogue systems. as a result, we achieve much higher accuracy in dialogue state tracking across different domains given limited amount of data compared to the existing methods. as the variety of tasks and functionalities in a dialogue system continues to grow, general methods for tracking dialogue state across all tasks will become increasingly necessary. we hope that the developments suggested here will help to address this need.