A Probabilistic End-To-End Task-Oriented Dialog Model with Latent Belief States towards Semi-Supervised Learning (2020)

Abstract: Structured belief states are crucial for user goal tracking and database query in task-oriented dialog systems. However, training belief trackers often requires expensive turn-level annotations of every user utterance. In this paper we aim at alleviating the reliance on belief state labels in building end-to-end dialog systems, by leveraging unlabeled dialog data towards semi-supervised learning. We propose a probabilistic dialog model, called the LAtent BElief State (LABES) model, where belief states are represented as discrete latent variables and jointly modeled with system responses given user inputs. Such latent variable modeling enables us to develop semi-supervised learning under the principled variational learning framework. Furthermore, we introduce LABES-S2S, which is a copy-augmented Seq2Seq model instantiation of LABES. In supervised experiments, LABES-S2S obtains strong results on three benchmark datasets of different scales. In utilizing unlabeled dialog data, semi-supervised LABES-S2S significantly outperforms both supervised-only and semi-supervised baselines. Remarkably, we can reduce the annotation demands to 50% without performance loss on MultiWOZ.

## introduction belief tracking (also known as dialog state tracking) is an important component in task-oriented dialog systems. the system tracks user goals through multiple dialog turns, i.e. infers structured belief states expressed in terms of slots and values (e.g. in figure 1 ), to query an external database @xcite . different belief tracking models have been proposed in recent years, either trained independently @xcite @xcite i need to find a thai restaurant that's in the south section of the city. there are three restaurants in the south part of town that serve thai food. do you have a cuisine preference? belief state db # match: 3 restaurant-food: thai ; restaurant-area: south figure 1 : the cues for inferring belief states from user inputs and system responses. the system response reveals the belief state either directly in the form of word repetition (red), or indirectly in the form of the database query result (green) determined by the belief state. @xcite or within end-to-end (e2e) trainable dialog systems @xcite @xcite @xcite . existing belief trackers mainly depend on supervised learning with human annotations of belief states for every user utterance. however, collecting these turn-level annotations is labor-intensive and time-consuming, and often requires domain knowledge to identify slots correctly. building e2e trainable dialog systems, called e2e dialog systems for short, even further magnifies the demand for increased amounts of labeled data @xcite . notably, there are often easily-available unlabeled dialog data such as between customers and trained human agents accumulated in real-world customer services. in this paper, we are interested in reducing the reliance on belief state annotations in building e2e task-oriented dialog systems, by leveraging unlabeled dialog data towards semi-supervised learning. intuitively, the dialog data, even unlabeled, can be used to enhance the performance of belief tracking and thus benefit the whole dialog system, because there are cues from user inputs and system responses which reveal the belief states, as shown in figure 1 . technically, we propose a latent variable model for task-oriented dialogs, called the latent belief state (labes) dialog model. the model generally consists of multiple (e.g. t ) turns of user inputs u 1:t and system responses r 1:t which are observations, and belief states b 1:t which are latent variables. basically, labes is a conditional generative model of belief states and system responses given user inputs, i.e. p θ (b 1:t , r 1:t |u 1:t ). once built, the model can be used to infer belief states and generate responses. more importantly, such latent variable modeling enables us to develop semi-supervised learning on a mix of labeled and unlabeled data under the principled variational learning framework @xcite . in this manner, we hope that the labes model can exploit the cues for belief tracking from user inputs and system responses. furthermore, we develop labes-s2s, which is a specific model instantiation of labes, employing copy-augmented seq2seq @xcite based conditional distributions in implementing p θ (b 1:t , r 1:t |u 1:t ). we show the advantage of our model compared to other e2e task-oriented dialog models, and demonstrate the effectiveness of our semisupervised learning scheme on three benchmark task-oriented datasets: camrest676 @xcite , in-car @xcite and multiwoz @xcite across various scales and domains. in supervised experiments, labes-s2s obtains state-of-the-art results on camrest676 and in-car, and outperforms all the existing models which do not leverage large pretrained language models on multiwoz. in utilizing unlabeled dialog data, semi-supervised labes-s2s significantly outperforms both supervised-only and prior semisupervised baselines. remarkably, we can reduce the annotation requirements to 50% without performance loss on multiwoz, which is equivalent to saving around 30,000 annotations. ## related work on use of unlabeled data for belief tracking. classic methods such as self-training @xcite , also known as pseudo-labeling @xcite , has been applied to belief tracking @xcite . recently, the pretraining-and-finetuning approach has received increasing interests @xcite @xcite . the generative model based semisupervised learning approach, which blends unsupervised and supervised learning, has also been studied @xcite . notably, the two approaches are orthogonal and could be jointly used. our work belongs to the second approach, aiming to leverage unlabeled dialog data beyond of using general text corpus. a related work close to ours is sedst @xcite , which also perform semi-supervised learning for belief tracking. remarkably, our model is optimized under the principled variational learning framework, while sedst is trained with an ad-hoc combination of posterior regularization and auto-encoding. experimental in §6.2 show the superiority of our model over sedst. see appendix a for differences in model structures between sedst and labes-s2s. end-to-end task-oriented dialog systems. our model belongs to the family of e2e task-oriented dialog models @xcite @xcite @xcite @xcite . we borrow some elements from the sequicity @xcite ) model, such as representing the belief state as a natural language sequence (a text span), and using copy-augmented seq2seq learning @xcite . but compared to sequicity and all its follow-up works @xcite @xcite , a feature in our labes-s2s model is that the transition between belief states across turns and the dependency between system responses and belief states are well statistically modeled. this new design results in a completely different graphical model structure, which enables rigorous probabilistic variational learning. see appendix a for details. latent variable models for dialog. latent variables have been used in dialog models. for non task-oriented dialogs, latent variables are introduced to improve diversity @xcite @xcite , control language styles @xcite or incorporate knowledge @xcite in dialog generation. for task-oriented dialogs, there are prior studies which use latent internal states via hidden markov models @xcite or variational autoencoders @xcite to discover the underlying dialog structures. in @xcite and @xcite , dialog acts are treated as latent variables, together with variational learning and reinforcement learning, aiming to improve response generation. to the best of our knowledge, we are the first to model belief state as discrete latent variables, and propose to learn these structured representations via the variational principle. ## latent belief state dialog models we first introduce labes as a general dialog modeling framework in this section. for dialog turn t, let u t be the user utterance, b t be the current belief state after observed u t and r t be the corresponding system response. in addition, denote c t as the dialog context or model input at turn t, such as c t {r t-1 , u t } as in this work. note that c t can include longer dialog history depending on specific implementations. let d t be the database query result which can be obtained through a databaselookup operation given the belief state b t . our goal is to model the joint distribution of belief states and system responses given the user inputs, p θ (b 1:t , r 1:t |u 1:t ), where t is the total number of turns and θ denotes the model parameters. in labes, we assume the joint distribution follows the directed probabilistic graphical model illustrated in figure 2 , which can be formulated as: ## labes-s2s: a copy-augmented seq2seq instantiation in the above probabilistic dialog model labes, the belief state decoder p θ (b t |b t-1 ,c t ) and the response decoder p θ (r t |c t ,b t ,d t ) can be flexibly im-plemented. in this section we introduce labes-s2s as an instantiation of the general labes model based on copy-augmented seq2seq conditional distributions @xcite , which is shown in figure 3 (a) and described in the following. the responses are generated through two seq2seq processes: 1) decode the belief state given dialog context and last turn's belief state and 2) decode the system response given dialog context, the decoded belief state and database query result. ## results and analysis in our experiments, we report both the best result and the statistical result obtained from multiple independent runs with different random seeds. details are described in the caption of each table. the implementation details of our model is available in appendix c. results are organized to show the advantage of our proposed labes-s2s model over existing models ( §6.1) and the effectiveness of our semi-supervised learning method ( §6.2). ## conclusion and future work in this paper we are interested in reducing belief state annotation cost for building e2e task-oriented dialog systems. we propose a conditional generative model of dialogs -labes, where belief states are modeled as latent variables, and unlabeled dialog data can be effectively leveraged to improve belief tracking through semi-supervised variational learning. furthermore, we develop labes-s2s, which is a copy-augmented seq2seq model instantiation of labes. we show the strong benchmark performance of labes-s2s and the effectiveness of our semi-supervised learning method on three benchmark datasets. in our experiments on multi-woz, we can save around 50%, i.e. around 30,000 belief state annotations without performance loss. there are some interesting directions for future work. first, the labes model is general and can be enhanced by, e.g. incorporating largescale pre-trained language models, allowing other options for the belief state decoder and the response decoder such as transformer based. second, we can analogously introduce dialog acts a 1:t as latent variables to define the joint distribution p θ (b 1:t , a 1:t , r 1:t |u 1:t ), which can be trained with semi-supervised learning and reinforcement learning as well. calendar scheduling, weather information retrieval and point-of-interest navigation. the multiwoz @xcite dataset is a large-scale human-human multi-domain dataset containing dialogs in seven domains including attraction, hotel, hospital, police, restaurant, train, and taxi. it is more challenging due to its multi-domain setting, complex ontology and diverse language styles. as there are some belief state annotation errors in mul-tiwoz, we use the corrected version multiwoz 2.1 @xcite in our experiments. we follow the data preprocessing setting in @xcite , whose data cleaning is developed based on @xcite .