Advancing Semi-Supervised Task Oriented Dialog Systems by JSA Learning of Discrete Latent Variable Models (2022)

Abstract: Developing semi-supervised task-oriented dialog (TOD) systems by leveraging unlabeled dialog data has attracted increasing interests. For semi-supervised learning of latent state TOD models, variational learning is often used, but suffers from the annoying high-variance of the gradients propagated through discrete latent variables and the drawback of indirectly optimizing the target log-likelihood. Recently, an alternative algorithm, called joint stochastic approximation (JSA), has emerged for learning discrete latent variable models with impressive performances. In this paper, we propose to apply JSA to semi-supervised learning of the latent state TOD models, which is referred to as JSA-TOD. To our knowledge, JSA-TOD represents the first work in developing JSA based semi-supervised learning of discrete latent variable conditional models for such long sequential generation problems like in TOD systems. Extensive experiments show that JSA-TOD significantly outperforms its variational learning counterpart. Remarkably, semi-supervised JSA-TOD using 20% labels performs close to the full-supervised baseline on MultiWOZ2.1.

## introduction task-oriented dialog (tod) systems are designed to help users to achieve their goals through multiple turns of natural language interaction. the system needs to parse user utterances, track dialog states, query a task-related database (db), decide actions and generate responses, and to do these iteratively across turns. the information flow in a task-oriented dialog is illustrated in figure 1 . recent studies recast such information flow in a tod system as conditional generation of tokens and base on pretrained language models (plms) such as gpt2 @xcite and t5 @xcite as the model backbone. fine-tuning a plm over annotated dialog datasets such as multi-woz @xcite via supervised learning has shown promising results @xcite @xcite , but requires manually labeled dialog states and system acts (if used). notably, there are often easily-available unlabeled dialog data such as in customer-service logs and online forums. this has motivated the development of semi-supervised leaning (ssl) for tod systems, which aims to leverage both labeled and unlabeled dialog data. a broad class of ssl methods builds a latent variable model (lvm) of observations and labels and blends unsupervised and supervised learning. unsupervised learning with a lvm usually maximizes the marginal log-likelihood, which is often intractable to compute. variational learning @xcite introduces an auxiliary inference model and, instead, maximizes the evidence lower bound (elbo) of the marginal log-likelihood. this approach of variational learning of lvms has been studied for semi-supervised tod systems such as in @xcite ; @xcite ; @xcite ; @xcite . particularly, discrete latent variables are mostly used, since dialog states and system acts are often modeled as taking discrete values. however, for variational learning of discrete latent variable models, the monte-carlo gradient estimator for the inference model parameter is known to have high-variance. most previous studies use the gumbel-softmax trick @xcite or the straight-through trick @xcite empirically, which in fact are biased estimators. another drawback of variational learning is that it indirectly optimizes the lower bound of the target marginal log-likelihood, which leaves an uncontrolled gap between the target and the bound, depending on the expressiveness of the inference model. recently, an alternative algorithm, called joint stochastic approximation (jsa) @xcite , has emerged for learning discrete latent variable models with impressive performances. jsa directly optimizes the marginal likelihood and completely avoids gradient propagation through discrete latent variables. in this paper, we propose to apply jsa to semi-supervised learning of the latent state tod models, which is referred to as jsa-tod. we develop recursive turnlevel metropolis independence sampling (mis) to enable the successful application of jsa, which needs posterior sampling of the latent states from the whole dialog session. to our knowledge, jsa-tod represents the first work in developing jsa based semi-supervised learning of discrete latent variable conditional models for such long sequential generation problems like in tod systems. extensive experiments show that jsa-tod significantly outperforms its variational learning counterpart in semi-supervised learning. remarkably, semi-supervised jsa-tod using 20% labels performs close to the supervised-only baseline using 100% labels on multiwoz2.1. the code and data are released at https://github.com/cycrab/jsatod. ## preliminary: joint stochastic approximation (jsa) stochastic approximation (sa) refers to an important family of iterative stochastic optimization algorithms for stochastically solving a root finding problem, which has the form of expectations being equal to zeros @xcite . within the sa framework, the joint stochastic approximation (jsa) algorithm is recently developed @xcite for learning a broad class of latent variable models, particularly for learning models with discrete latent variables. interestingly, jsa amounts to coupling an sa version of expectation-maximization (saem) @xcite with an adaptive markov chain monte carlo (mcmc) procedure. based on jsa, the annoying difficulty of propagating gradients through discrete latent variables and the drawback of indirectly optimizing the target log-likelihood can be gracefully addressed. consider a latent variable generative model p θ (z, x) for observation x and latent variable z, with parameter θ. like in variational methods, jsa also jointly trains the target model p θ (z, x) together with an auxiliary amortized inference model q ϕ (z|x). the difference is that jsa directly maximizes w.r.t. θ the marginal log-likelihood and simultaneously minimizes w.r.t. ϕ the inclusive kl divergence kl(p θ (z|x)||q ϕ (z|x)) between the posterior and the inference model, pooled over the training dataset: the optimization problem eq. ( 1 ) can be solved by setting the gradients to zeros and applying the sa algorithm to find the root for the resulting simultaneous equations, which has the exact form of expectations equal to zeros: algorithm 1 the jsa algorithm repeat monte carlo sampling: draw κ over 1, • • • , n, pick the data-point x (κ) the jsa algorithm can be intuitively understood as a stochastic extension of the well-known em algorithm @xcite . since the latent variable z (κ) is unknown for data-point x (κ) , the monte carlo sampling step in jsa fills the missing value for z (κ) through sampling p θ (z (κ) |x (κ) ), which is analogous to the e-step in em. then in the parameter updating step, z (κ) is treated as if being known, and used to optimize over θ and ϕ by performing gradient ascent using ∇ θ log p θ (z (κ) , x (κ) ) and ∇ ϕ log q ϕ (z (κ) |x (κ) ) respectively. this is analogous to the m-step in em, but with the proposal q ϕ being adapted as well. in summary, we could refer to the underlying mechanism of jsa as propose, accept/reject, and optimize (or, for short, the paro mechanism), which establishes jsa as a simple, solid and effective approach to learning discrete latent variable models. ## conclusion and future work this paper represents a progress towards building semi-supervised tod systems by learning latent state tod models. traditionally, variational learning is often used; notably, the recently emerged jsa method has been shown to surpass variational learning, particularly in learning of discrete latent variable models. this paper represents the first application of jsa in its conditional sequential version, particularly for such long sequential generational problems like in tod systems. extensive experiments clearly show the superiority of jsa-tod over its variational learning counterpart, not only in benchmark metrics for semi-supervised tod systems but also from the latent state prediction performances and the variances of the gradients of the inference model. since discrete latent variable models are widely used in many natural language procession tasks, we hope the results presented in this paper will encourage the community to further explore the applications of jsa and improve upon current approaches.