V ARIATIONAL M ESSAGE P ASSING WITH S TRUCTURED I NFERENCE N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a variational message-passing algorithm for a class of models that combine deep models with probabilistic graphical models.
Our algorithm is a natural-gradient algorithm whose messages automatically reduce to stochasticgradients for the deep components of the model.
Using a special-structure inference network, our algorithm exploits the structural properties of the model to gain computational efﬁciency while retaining the simplicity and generality of deeplearning algorithms.
By combining the strength of two different types of inference procedures, our approach offers a framework that simultaneously enables structured, amortized, and natural-gradient inference for complex models.
I NTRODUCTION
To analyze real-world data, machine learning relies on models that can extract useful patterns.
Deep
Neural Networks (DNNs) are a popular choice for this purpose because they can learn ﬂexible representations.
Another popular choice are probabilistic graphical models (PGMs) which can ﬁnd interpretable structures in the data.
Recent work on combining these two types of models hopes to exploit their complimentary strengths and provide powerful models that are also easy to interpret (Johnson et al., 2016; Krishnan et al., 2015;
Archer et al., 2015; Fraccaro et al., 2016).
To apply such hybrid models to real-world problems, we need efﬁcient algorithms that can extract useful structure using the model.
However, the two ﬁelds of deep learning and PGMs traditionally use different types of algorithms.
For deep learning, stochastic-gradient (SG) methods are the most popular choice, e.g., those based on back-propagation.
These algorithms are not only widely applicable, but can also employ amortized inference to enable fast inference at test time (Rezende et al., 2014; Kingma & Welling, 2013).
On the other hand, most popular algorithms for PGMs exploit the model’s graphical conjugacy structure to gain computational efﬁciency, e.g., variational message passing (VMP) (Winn & Bishop, 2005), expectation propagation (Minka, 2001), Kalman ﬁltering, and more recently stochastic variational inference (SVI) (Hoffman et al., 2013).
In short, the two ﬁelds of deep learning and probabilistic modelling employ fundamentally different inferential strategies and a natural question is, whether we can design algorithms that combine their respective strengths.
There have been several attempts to design such methods in the recent years, e.g., Krishnan et al.
; Fraccaro et al.
; Archer et al.
; Johnson et al.
Our work in this paper is inspired by the previous work of Johnson et al.
(2016) that aims to combine message-passing, natural-gradient, and amortized inference.
Our proposed algorithm in this paper simpliﬁes and generalizes the algorithm of Johnson et al.
To do so, we propose Structured Inference Networks (SIN) that combine the strengths of standard inference networks used in deep learning with structured PGM models.
We derive conditions under which such inference networks can enable efﬁcient inference procedure.
By using a recent VMP method of Khan & Lin (2017), we derive a variational message-passing algorithm whose messages automatically reduce to stochastic-gradients for the deep components of the model, while perform natural-gradient updates for PGM part.
Overall, our algorithm enables Structured, Amortized, and
Natural-gradient (SAN) updates and therefore we call our algorithm the SAN algorithm.
We show that our algorithm give comparable performance to the method of Johnson et al.
(2016) while simplifying and generalizing it.
[Figure]
Latent state-space model y n x n z n θ NN θ PGM
N (a) Model y n x n z n θ PGM φ NN φ PGM
N (b) Inference Network yyyyxxxxθ PGM θ NN (c) Model yyyyxxxxθ PGM φ PGM φ NN (d) Inference Network
T HE M ODEL AND C HALLENGES WITH I TS I NFERENCE
We consider the modelling of data vectors y n by using local latent vectors x n .
Following previous works (Johnson et al., 2016; Archer et al., 2015; Krishnan et al., 2015), we model the output y n given x n using a neural network with parameters θ NN , and capture the correlations among data vectors y :=
{ y 1 , y 2 , . . .
, y N } using a probabilistic graphical model (PGM) over the latent vectors x := { x 1 , x 2 , . . .
, x N } .
Speciﬁcally, we use the following joint distribution:
[Equation]
| x n , θ NN ) (cid:35) (cid:124) (cid:123)(cid:122) (cid:125) DNN (cid:34) p (
[Equation]
θ PGM ) (cid:35) (cid:124) (cid:123)(cid:122) (cid:125)
PGM (cid:34) p (
[Equation]
PGM ) (cid:35) (cid:124) (cid:123)(cid:122) (cid:125) Hyperprior , (1) where θ NN and θ PGM are parameters of a DNN and PGM respectively, and θ := { θ NN , θ PGM } .
This combination of probabilistic graphical model and neural network is referred to as structured variational auto-encoder (SVAE) by Johnson et al.
SVAE employs a structured prior p ( x | θ PGM ) to extract useful structure from the data.
SVAE therefore differs from VAE (Kingma & Welling, 2013) where the prior distribution over x is simply a multivariate Gaussian distribution p ( x ) =
N ( x | 0 , I ) with no special structure.
To illustrate this difference, we now give an example of a structured prior which is useful to obtain clustering of the outputs y n .
Example (Mixture-Model Prior) : Suppose we wish to group the output y n into K distinct clusters, in this case the standard Gaussian prior, which is used in VAE, is not a useful prior.
In
SVAE, we could instead use a mixture-model prior over x n (Johnson et al., 2016),
[Equation]
where z n ∈ { 1 , 2 , . .
, K } is the mixture indicator for n ’th data example, and π k are mixing proportions that sum to 1.
Each cluster can be modelled, for example, by using a Gaussian mixture model (GMM) where each mixture component p ( x n | z n = k ) :
= N ( x n | µ k , Σ k ) is a
Gaussian with mean µ k and covariance Σ k , in which case θ PGM := { µ k , Σ k , π k } Kk =1 .
The graphical model of an SVAE with a GMM prior is shown in Figure 1a.
Such structured-prior is useful to discover cluster of outputs y n by clustering the latent variables x n .
Given such models, our goal is to approximate the posterior distribution p ( x , θ | y ) .
In particular, in a similar spirit to VAE, we would like to approximate the posterior of x by using an inference network.
In VAE, such inference networks are obtained by using functions parameterized using a
DNN, as shown below where the generative model for VAE is shown in the left hand side while the
[Equation]
The distribution q is typically an exponential-family distribution whose natural-parameters are modelled by using a DNN f φ with parameters φ .
The same function f φ ( · ) is used for all n which reduces the number of variational parameters and enables sharing of statistical strengths across n .
This leads to both faster training and testing (Rezende et al., 2014).
Unfortunately, for SVAE, such inference networks may give inaccurate predictions since they ignore the prior structure of the PGM prior p ( x | θ PGM ) .
For example, suppose y n is a time-series and we model x n using a dynamical system as depicted in Fig. 1c.
In this case, the inference network of (3) is not an accurate approximation since it ignores the time-series structure in x .
This might result in inaccurate predictions of distant future observations, e.g., prediction for an observation ygiven the past data { y 1 , y 2 , y 3 } would be inaccurate because the inference network has no path connecting x 10 to x 1 , x 2 , or x 3 .
In cases like these where the prior structure plays an important role in prediction, it is important to incorporate this structure in the inference network.
A solution to this problem is to use an inference network with the same structure as the model but to replace all its edges by neural networks (Krishnan et al., 2015; Fraccaro et al., 2016).
This solution is reasonable when the PGM itself is complex, but this might be too aggressive when the PGM is a simple model, e.g., for the linear dynamical system in Fig. 1c.
Using DNNs in such cases would dramatically increase the number of parameters which will lead to a possible deterioration in both speed and performance.
Johnson et al.
(2016) propose an alternative algorithm to incorporate the structure of the PGM part.
They consider optimizing the standard mean-ﬁeld lower bound: L ( λ x , λ θ ) :
=
E q [log p ( y , x , θ ) − log q ( x | λ x ) q ( θ | λ θ )] when p ( x | θ PGM ) is a conditionally-conjugate distribution and q are minimal exponential-family distributions with natural parameters
λ x and λ θ respectively.
To incorporate an inference network, they deﬁne the following surrogate lower-bound where the DNN likelihood in L is replaced by the inference network of (3),
[Equation]
and constrain λ x to be its local optimizer, i.e., λ ∗ x ( λ θ , φ )
= arg max λ x ˆ L ( λ x , λ θ , φ ) .
When the factors q ( x n | f φ ( y n )) are chosen to be conjugate to p ( x | θ PGM ) , this optimization can be performed efﬁciently using VMP.
To exploit this efﬁciency, Johnson et al.
(2016) propose a two-stage procedure where, in every iteration, they compute λ ∗ x ( λ θ , φ ) for a given λ θ and φ , and then optimize the mean-ﬁeld objective L ( λ ∗ x ( λ θ , φ ) , λ θ ) .
The hope is that λ ∗ x can be computed efﬁciently using
VMP, while the update of λ θ can be obtained using a natural-gradient method similar to SVI, and φ can be optimized using SGD as in VAE.
This two-stage optimization is equivalent to an implicitly-constrained optimization problem.
Such problems are typically more difﬁcult to solve than their unconstrained counterparts, especially when the constraints are nonconvex (Heinkenschloss, 2008).
Theoretically, the convergence of such methods is difﬁcult to guarantee when the constraints are violated.
In practice, this makes the implementation difﬁcult because in every iteration the VMP updates need to run long enough to reach close to a local optimum of the surrogate lower bound.
Another disadvantage of the method of Johnson et al.
(2016) is that it applies only under restrictive assumptions on the PGM prior.
For example, it does not work for PGMs that contain non-conjugate factors because in that case VMP cannot be used to optimize the surrogate lower bound.
In addition, it does not apply directly when λ x is constrained, and in case q ( x | λ x ) has additional latent variables (e.g., indicator variables z n in GMM), the VMP iterations may need to run quite long to reach a local optimum.
Due to these issues, the method of Johnson et al.
(2016) presents serious implementation challenge for generic PGMs.
S TRUCTURED I NFERENCE N ETWORKS
Our goal is to design an inference network that preserves the PGM’s probabilistic structure while enabling inference with a computationally-efﬁcient message-passing algorithm.
We propose the following structured inference network (SIN) which consists of two types of factors,
[Equation]
The DNN factor here is similar to (3) while the PGM factor is an exponential-family distribution which has a similar graph structure as the PGM prior p ( x | θ PGM ) .
The role of the DNN term is to enable ﬂexibility while the role of the PGM term is to incorporate the model’s PGM structure into the inference network.
Both factors have their own parameters, where φ NN is the parameter of the DNN and φ PGM is the natural parameters of the PGM factor.
The set of parameters is φ = { φ NN , φ PGM } .
How should we choose the two factors?
As we will show soon that, for tractable inference, these factors need to satisfy the following two conditions.
The ﬁrst condition is that the normalizing constant Z ( φ ) is easy to evaluate and differentiate with respect to φ .
Note that both the factors are normalized distribution, but their product may not be.
The second condition is that it is easy to draw samples x ∗ ( φ ) ∼ q ( x | y , φ ) , where we denote a sample by x ∗ ( φ ) to show the direct dependence on φ .
An additional desirable feature is to be able to compute the gradient of x ∗ ( φ ) by using the reparameterization trick, although this is not necessary.
Now, we will show that given these two conditions we can easily compute the necessary gradients to derive the message passing algorithm.
We show that when the above two conditions are met, the gradient of the lower bound can be computed easily, and then give details of the message passing algorithm in the next section.
For now, we assume that θ is a deterministic variable for now (we will relax this in the next section).
The variational lower bound in this case can be written as follows:
[Equation]
The ﬁrst three terms are almost identical to the lower bound for the standard VAE, and the last two terms are a result of adding the PGM structure to the inference network.
As a result, if we can generate samples x ∗ ( φ ) from SIN, the gradient can be straightforwardly computed using the reparameterization trick similar to VAE.
This is shown below.
[Equation]
[Table] SIN
[Equation]
where we drop the explicit dependence of x ∗ ( φ ) over φ .
In the ﬁrst line, the gradient with respect to θ NN is identical to the back-propagation-step through the decoder in VAE, while the gradient with respect to θ PGM is the gradient of log-likelihood of PGM.
Both quantities can be easily computed.
In the second line, the gradient with respect to φ NN is identical to the gradient of the encoder in VAE, plus the gradient of Z ( φ ) .
The third line shows the gradient with respect to φ PGM , which is equal to the gradient through the log-likelihoods of the PGMs, plus the gradient of Z ( φ ) .
Therefore, given that x ∗ can be drawn easily and gradient of Z ( φ ) can be computed, we can compute all gradients easily.
Not only that, we can also reuse the implementation for inference in VAE and PGMs.
When p ( x | θ PGM ) and q ( x | φ PGM ) factorize into simple factors, which is expected since both are structured graphical models, we can further reduce computation by computing stochastic approximations of their gradients.
The only tricky part is the computation of Z ( φ ) which might require a full inference through the graphical model.
If the graphical model is too complex (e.g., Gaussian process prior), then this might be difﬁcult since an exact Bayesian inference in such models is not computationally efﬁcient.
In this paper, we therefore only focus on PGMs where Z ( φ ) is tractable.
We now give many examples and illustrate the choices of the two factors for a variety of PGM priors.
When p ( x | θ PGM ) is a conjugate exponential-family distribution, choosing the two factors is a very easy task.
In this case, we can let q ( x | φ PGM )
= p ( x | φ PGM ) , i.e., the second factor is the same distribution as the PGM prior but with different set of parameters φ PGM .
To illustrate this, we give an example below when the PGM prior is a linear dynamical system.
Example (SIN for Linear Dynamical System (LDS)) :
When y n is a time series, we can model the latent x n using an LDS.
p ( x | θ ) :
= N ( x 0 | µ 0 , Σ 0 ) (cid:81)
Nn =1 N ( x n |
Ax n − 1 , Q ) , where A is the transition matrix, Q is the covariance for the process noise, and µ 0 and Σ 0 are the mean and covariance of the initial distribution.
Therefore, θ PGM := { A , Q , µ 0 , Σ 0 } .
In our inference network, we choose q ( x | φ ) =
p ( x | φ ) as show below, where φ PGM := { ¯ A , ¯ Q , ¯ µ 0 , ¯ Σ 0 } and, since the PGM is a Gaussian, we choose the DNN factor to be a Gaussian as well:
[Equation]
y , φ ) :=
[Equation]
where m n := m φ NN ( y n ) and V n :=
V φ NN ( y n ) are mean and covariance parameterized by a
DNN with parameter φ NN .
The graphical model and SIN are shown in Figure 1c.
The above SIN is a conjugate model where the marginal likelihood and distributions can be computed in O ( N ) using the forward-backward algorithm, a.k.a.
Kalman smoother
(Bishop, 2006).
We can also compute the gradient of Z ( φ ) as shown in (Kokkala et al., 2015).
When the PGM prior have additional structure and latent variables, e.g., the GMM prior has cluster indicators z n , we might want to incorporate this into the network, as shown in the example below.
Example (SIN for GMM prior): The prior shown in (2) has additional structure where we marginalize over the indicator variable z n .
To mimic this structure in SIN, we choose the PGM factor as shown below with parameters φ PGM := { ¯ µ k , ¯ Σ k , ¯ π k } Kk =1 , while keeping the DNN part to be a Gaussian distribution similar to the LDS case.
[Equation]
The model and SIN are shown in Figure 1a.
Fortunately, due to conjugacy of the Gaussian and
Multinomial distribution, we can marginalize x n to get a closed-form expression for log Z ( φ ) :
= (cid:80) n log (cid:80) k
N ( m n | ¯ µ k , V n + ¯ Σ k ) ¯ π k .
We an also easily sample by ﬁrst sampling from the marginal distribution of z n given by q ( z n = k | y , φ ) ∝ N (cid:0) m n | ¯ µ k , V n + ¯ Σ k (cid:1)
¯ π k .
Given z n , we can easily sample x n from the following marginal: q ( x n | z n = k, y , φ ) =
N ( x n | (cid:101) µ n , (cid:101) Σ n ) , where (cid:101) Σ − 1 n = V − 1 n + ¯ Σ − 1 k
and (cid:101) µ n = (cid:101) Σ n ( V − 1 n m n + ¯ Σ − 1 k µ k ) .
In all of the above examples, we are able to satisfy the two conditions even when using the same structure as the model.
However, this may not be possible for general conditionally-conjugate exponential family distribution.
In such cases, we can still obtain samples from a tractable structured mean-ﬁeld approximation using VMP.
We illustrate this on the switching state-space model in Appendix B. A drawback of our method in this case is that we need to run VMP long enough to get a sample, very similar to the method of Johnson et al.
The advantage, however, is that this process is completely independent of θ , therefore we do not have to compute gradient of Z ( φ ) with respect to θ , which makes the implementation easier.
We also expect the convergence to not be a big problem because, even if we stop early, we are simply generating samples from an approximate posterior.
In addition, we can handle non-conjugate factors as we show in our next example.
When the PGM prior contains some non-conjugate factors, we might replace them by their closest conjugate approximations while making sure that the inference network captures the useful structure present in the posterior distribution.
We illustrate this point for a Student’s t mixture model.
Example (SIN for Student’s t Mixture Model) :
To handle outliers in the data, we might want to use the Student-t mixture component in the mixture model shown in (2), i.e., we set p ( x n | z n = k )
=
T ( x n | µ k , Σ k , γ k ) with mean µ k , scale matrix Σ k and degree of freedom γ k .
The Student’s t-distribution is not conjugate to the multinomial distribution, therefore, if we use it as the PGM factor in SIN, we will not be able to satisfy both conditions easily.
Even though our model contains a t-distribution components, we can still use the SIN shown in (12) that uses a
GMM factor.
Our method therefore simpliﬁes inference by choosing an inference network which has a simpler form than the original model.
In theory, one can do this even when all factors are non-conjugate, however, the approximation error might be quite large in some cases for this approximation to be useful.
We experimented this for non-linear dynamical system and found that capturing non-linearity was essential for extremely non-linear dynamics.
V ARIATIONAL M ESSAGE P ASSING WITH I NFERENCE N ETWORKS
In the previous section, we introduced the structured inference network and described how to compute the gradient of the lower bound.
We will now show that when θ PGM is a random variable with a prior distribution, we can use VMP udpates to estimate its variational approximation, while continuing to use stochastic-gradient updates for the rest of the parameters.
In this section, we assume an exponential-family prior p ( θ PGM | η PGM ) with natural parameter η PGM .
Our goal is to approximate the posterior distribution over x and θ PGM , and compute point estimates of θ NN , φ NN , and φ PGM .
Similar to Johnson et al.
(2016), we want to perform natural-gradient updates for θ PGM .
For this reason, we will use the method of Khan & Lin (2017).
This method is a generalization of VMP and
SVI, and enables us to derive natural-gradient updates for θ PGM .
The method uses a mirror-descent algorithm in the mean-parameter space of the exponential family.
When the Kullback-Leibler (KL) divergence is used for the mirror-descent step, the step is equivalent to the natural-gradient descent in the natural-parameter space.
Khan & Lin (2017) show that the mirror-descent update is equivalent to the SVI update when all the neighbours of a node are conjugate.
In our setup, their method is useful since it enables natural-gradient updates even when the PGM part is not conjugate.
We start by deriving the variational lower bound.
We use the following mean-ﬁeld variational approximation: p ( x , θ | y )
≈ q ( x | y , φ ) q
( θ PGM | λ PGM ) where the ﬁrst term is SIN introduced in the previous section, and the second term is an exponential-family distribution with natural parameter λ PGM .
The variational lower bound corresponding to the mean-ﬁeld approximation can be expressed in terms of L SIN derived in the previous section.
[Equation]
We will use a mirror-descent update with the KL divergence for q ( θ PGM | λ PGM ) because we want natural-gradient updates for it.
We denote the mean parameter of this distribution by µ PGM .
For the rest of the parameters, we will use the usual Euclidean distance.
Denoting the values at iteration t with a superscipt t and using Eq.
19 in (Khan & Lin, 2017) with these divergences, we get:
[Equation]
where β 1 to β 4 are scalars, (cid:104) , (cid:105) is an inner product, and ∇L t is the gradient at the value in iteration t .
As shown in Khan & Lin (2017), the maximization for λ PGM can be obtained in closed-form:
[Equation]
This update is a natural-gradient update since the gradient with respect to µ PGM is the natural gradient.
When the prior on θ PGM is conjugate to p ( x | θ PGM ) , this gradient can be computed using the
VMP or SVI implementation.
Otherwise, it can be approximated either using stochastic gradients or using the reparameterization trick, as discussed in (Khan & Lin, 2017).
It is easy to see that the update of the rest of the parameters is equal to a stochastic-gradient descent update.
We can verify this by simply taking the derivative of (15) or (16), and setting it to zero.
The gradient ∇L t can be obtained by using a stochastic approximation by ﬁrst sampling θ PGM ∼ q ( θ PGM | λ PGM ) , and then approximating the gradient as follows:
∇L ( λ PGM , φ PGM , θ NN , φ NN ) ≈ ∇L SIN ( θ , φ ) .
Therefore, the update can be obtained using the gradients given in (8)-(10).
In practice, we use adaptive-gradient methods to update (instead of using SGD).
The ﬁnal algorithm is outlined in Algorithm 1.
Since our algorithm enables Structured, Amortized, and Natural-gradient (SAN) updates, we call it the SAN algorithm.
Our updates conveniently separate the PGM and DNN computations.
Step 3-6 operate on the PGM part, for which we can use existing implementation for the PGM.
Step 7 operates on the DNN part, for which we can reuse
VAE implementation.
Our algorithm not only generalizes previous works, but also simpliﬁes the implementation by enabling reusing existing software.
E XPERIMENTS AND R ESULTS
The main goal of our experiment is to show that our SAN algorithm gives similar results to the method of Johnson et al.
The advantage of our algorithm over the method of Johnson et al.
(2016) is that our algorithm is simpler and more general than their method, and does not involve a double-loop procedure.
For this reason, we apply our algorithm to the two examples considered in Johnson et al.
(2016), namely the latent GMM and latent LDS.
In this section, we show results on GMM.
An additional result for LDS is included in
Appendix A.
Our results show that the SAN algorithm can learn complex representations with interpretable structure, similar to the method of
Johnson et al.
For latent GMM, we compare SAN to three algorithms: ﬁrst, the standard variational EM algorithm applied to GMM (referred to as ‘GMM’), second, the VAE approach of Kingma & Welling (2013), and third, the SVAE approach of Johnson et al.
We use two datasets.
The ﬁrst dataset is the synthetic two-dimensional Pinwheel dataset ( N = 5000 and D = 2 ), and the second dataset is the
Auto dataset ( N = 392 and D = 6 , available in the UCI repository).
The latter contains information about cars.
The dataset also contains a ﬁve-class label which indicates the number of cylinders in a car.
We use these labels to validate our results.
For both datasets we use 70 % data for training.
For all methods, we tune their stepsizes, the number of mixture components, and the latent dimensionality on a validation set.
We train the GMM baseline on the full dataset, and, for VAE and SVAE, we use minibatches of size 64 .
Our neural-network likelihood consists of two layers with 50 hidden units and an output layer of dimensionality 6 for the Auto dataset and 2 for the Pinwheel dataset.
In Figure 2, we compare the performance during training.
In the left ﬁgure, we compare to SVAE and GMM, where we see that SAN converges faster than SVAE.
Both methods achieve similar performance upon convergence and perform better than GMM.
In the right ﬁgure, we compare to VAE and observe similar trends.
In this ﬁgure, the performance of GMM is represented as a constant because training converges after a few iterations already.
We found that the implementation
[Figure]
GMMVAE
SVAE (Johnson et.
al.)
[Figure] [Figure]
provided by
(2016) does not perform well included
[Figure]
In Figure 3, we show samples generated from the learned distribution p ( y ) for all methods.
On the top of this plot, we show the data with true labels.
These labels were not used during training.
The plots (a)-(c) show results on Pinwheel dataset, while plots (d)-(e) shows results on the Auto dataset.
For the Auto dataset, each label corresponds to the number of cylinders present in a car.
We observe that SAN can learn meaningful partition of the data space, while VAE does not posses this capability.
We also observe that SAN and VAE can learn ﬂexible patterns while GMM does not do so.
These results show that with SAN we are able to combine complementary characteristics of VAE and
GMM.
For both datasets, our method learns ﬂexible representations containing meaningful structural information.
We also consider a non-conjugate model, the latent Student’s t mixture model (latent TMM) discussed in Section 3.
The goal of this experiment is to show that, unlike the method of Johnson et al.
, our method works even when the model contains non-conjugate factors.
The Student’s t-distribution is very similar to the Gaussian distribution but is more robust to outliers thanks to its heavy tails.
To show the latent TMM’s advantage over the latent GMM, we add artiﬁcial outliers to the Pinwheel dataset by introducing random Gaussian noise.
We ﬁx the degree of freedom for the
Student’s t-distribution to 5 and use the structured inference network as explained in Section 3.
We test our model on four noisy data sets with different levels of noise and report the test MSE averaged over three runs in Figure 4.
We compare to the GMM baseline, as well as to SAN algorithm applied to latent GMM (instead of latent TMM).
In Figure 4, we can see that latent TMM is more robust than latent GMM and GMM.
The performance of each of the three tested models degrades with an increasing number of outliers, but the drop of the latent TMM is far less dramatic (mind the log-scale of the y-axis).
In fact, with 70% of outliers, the latent TMM still performs better than the latent GMM with only 10% of outliers.
This experiment illustrates that a conjugate SIN can be used for inference on a model with a nonconjugate PGM.
D ISCUSSION AND C ONCLUSION
We propose a message-passing algorithm for the models that use both deep networks and probabilistic graphical models.
We discussed the challenges involved when designing an algorithm to exploit the structural properties.
Our algorithm can automatically adjust its messages and adapt it to the type of the node.
For the DNN part, the messages reduce to computation of stochastic gradients, while on the PGM part, the messages are equal to natural gradients.
Experimental comparisons conﬁrm that our method gives similar results to existing methods, while using simpler updates.
Our experimental results in this paper are limited to small scale data.
We found that it is nontrivial to implement a message-passing framework that goes well with the deep learning framework.
Our current implementation is based on AutoGrad and TensorFlow, both of which are slow due to additional communication overhead due to messages passing.
We are going to pursue this direction in the future and investigate good platforms to integrate the capabilities of these two different ﬂavors of algorithms.
A R ESULTS FOR L ATENT L INEAR D YNAMICAL S YSTEM
In this experiment, we apply our SAN algorithm to the latent LDS discussed in Section 3.
For comparison, we compare our method, Structured Variational Auto-Encoder (SVAE)
(Johnson et al., 2016), and LDS on the Dot dataset used in Johnson et al.
Our results show that our method achieves comparable performance to SVAE.
For LDS, we perform batch learning for all model parameters using the EM algorithm.
For SVAE and SAN, we perform mini-batch updates for all model parameters.
We use the same neutral network architecture as in Johnson et al.
(2016), which contains two hidden layers with tanh activation function.
We repeat our experiments 10 times and measure model performance in terms of the following mean absolute error for τ -steps ahead prediction.
The error measures the absolute difference between the ground truth and the generative outputs by averaging across generated results.
[Equation]
(18) where N is the number of testing time series with T time steps, d is the dimensionality of observation y , and observation y ∗ t + τ,n denotes the ground-truth at time step t + τ .
[Figure]
From Figure 5, we can observe that our method performs as good as SVAE and outperforms LDS.
Our method is slightly robust than SVAE.
In Figure 6, there are generated images obtained from all methods.
From Figure 6, we also see that our method performs as good as SAVE and is able to recover the ground-truth observation.
B SIN FOR S WITCHING LDS
In SLDS, we introduce discrete variable z n ∈ { 1 , 2 , . .
, K } that are samples using a Markov model: p ( z n = i |
z n − 1 = j ) =
π ij such that π ij sum to 1 over all i given j .
The transition for LDS is deﬁned conditional on z n : p ( x n | x n − 1 , z n =
i, θ PGM ) :
= N ( x n | A i x n − 1 , Q i ) where A i and
Q i are parameters for the i ’th indicator.
These two dynamics put together deﬁne the SLDS prior p ( x , z | θ PGM ) .
We can use the following SIN which uses the SLDS prior as the PGM factor but with parameters φ PGM instead of θ PGM .
The expression for q ( x , z | y , φ ) is shown below:
[Equation]
Even though the above model is a conditionally-conjugate model, the partition function is not tractable and sampling is also not possible.
However, we can use a structured mean-ﬁeld approximation.
First, we can combine the DNN factor with the Gaussian observation of SLDS factor and then use a mean-ﬁeld approximation q ( x , z | y , φ )
≈ q ( x | λ x ) q
( z | λ z ) , e.g., using the method of Ghahramani & Hinton (2000).
This will give us a structured approximation where the edges between y n and x n and z n and z n − 1 are maintained but x n and z n independent of each other.
[Figure]