Paper:
Composing graphical models with neural networks for structured representations and fast inference
We propose a general modeling and inference framework that combines the complementary strengths of probabilistic graphical models and deep learning methods. Our model family composes latent graphical models with neural network observation likelihoods. For inference, we use recognition networks to produce local evidence potentials, then combine them with the model distribution using efficient message-passing algorithms. All components are trained simultaneously with a single stochastic variational inference objective. We illustrate this framework by automatically segmenting and categorizing mouse behavior from raw depth video, and demonstrate several other example models.

Review:
This paper presents a general modeling and inference framework that combines the strengths of probabilistic graphical models and the flexibility of neural network models. Graphical models are used to represent structured probability distributions and variational autoencoder ideas are used to learn the nonlinear data manifold in an approach that is referred to as the structured variational autoencoder (SVAE).  A major challenge of hybrid models is that inference is often difficult. The main idea of this paper is to learn recognition networks that output conjugate graphical model potentials, allowing for the use of tractable graphical model inference algorithms. They outline an algorithm for doing inference in the SVAE model.  The SVAE model is then applied to both a synthetic dataset and a real dataset. In the synthetic data experiment, a latent linear dynamical system (LDS) SVAE is trained to predict the trajectory of a dot that is bouncing back and forth in a sequence of images. The model seems to do a very good job on this relatively simple task.  Next, they apply a LDS SVAE to model depth video recordings of mouse behavior, where the model seems to predict future frames relatively well. Finally, they apply a latent switching linear dynamical system (SLDS) SVAE model to try to learn discrete states corresponding to natural behavioral units.
Combining PGMs and deep learning/neural networks is a very active and promising area of research. This is an interesting paper, with it’s main contribution to the existing literature being that it presents a model that can account for discrete latent variables. This new capability suggests that it could be used in a variety of interesting applications, including the type of behavior representation modeling shown in one of the experiments.  Overall, the organization of the paper is excellent, and the writing is clear. The technical part is however quite dense. Algorithm 1 in Section 4 is hard to follow. Some of the symbols in the algorithm don’t seem to be defined anywhere (and if they are, they’re too hard to find).   One of the strengths of the paper is that the approach is quite general. The very general description of the method might make it hard to figure out how to apply the algorithm to a specific new model. It might be helpful to include in the appendix a detailed description of the algorithm applied to the examples in Figure 3.  My main criticism is that there is no comparison to existing baselines, except for the "toy" example in Figure 1. I do believe that there are advantages to "structured" representations (at the very least, in terms of interpretability), but it would have been nice to see the results achieved by some baseline (existing) method. E.g., in Figure 5, it’s hard for someone who is unfamiliar with the data to determine if the model is making good predictions without having something to compare to.   Section 6.3 and Figure 6 should be explained in more detail. It seems that one of the more interesting contributions of this approach is that the latent switching linear dynamical system (SLDS) model can identify discrete latent behavioral states that influence the observed dynamics. Without more detail, it’s very hard to tell what is going on. For example, Section 6.3 states that there are 30 discrete states, but Figure 6 only shows two of these - are the other states semantically meaningful as well, or were these two picked because they were the only useful ones?