Paper:
DEEP VARIATIONAL BAYES FILTERS: UNSUPERVISED LEARNING OF STATE SPACE MODELS FROM RAW DATA
We introduce Deep Variational Bayes Filters (DVBF), a new method for unsupervised learning and identification of latent Markovian state space models. Leveraging recent advances in Stochastic Gradient Variational Bayes, DVBF can overcome intractable inference distributions via variational inference. Thus, it can handle highly nonlinear input data with temporal and spatial dependencies such as image sequences without domain knowledge. Our experiments show that enabling backpropagation through transitions enforces state space assumptions and significantly improves information content of the latent embedding. This also enables realistic long-term prediction.
We introduce Deep Variational Bayes Filters (DVBF), a new method for unsupervised learning and identification of latent Markovian state space models. Leveraging recent advances in Stochastic Gradient Variational Bayes, DVBF can overcome intractable inference distributions via variational inference. Thus, it can handle highly nonlinear input data with temporal and spatial dependencies such as image sequences without domain knowledge. Our experiments show that enabling backpropagation through transitions enforces state space assumptions and significantly improves information content of the latent embedding. This also enables realistic long-term prediction.
1 INTRODUCTION
Estimating probabilistic models for sequential data is central to many domains, such as audio, natural language or physical plants, Graves (2013); Watter et al. (2015); Chung et al. (2015); Deisenroth & Rasmussen (2011); Ko & Fox (2011). The goal is to obtain a model p(x1:T ) that best reflects a data set of observed sequences x1:T . Recent advances in deep learning have paved the way to powerful models capable of representing high-dimensional sequences with temporal dependencies, e.g., Graves (2013); Watter et al. (2015); Chung et al. (2015); Bayer & Osendorfer (2014).
Time series for dynamic systems have been studied extensively in systems theory, cf. McGoff et al. (2015) and sources therein. In particular, state space models have shown to be a powerful tool to analyze and control the dynamics. Two tasks remain a significant challenge to this day: Can we identify the governing system from data only? And can we perform inference from observables to the latent system variables? These two tasks are competing: A more powerful representation of system requires more computationally demanding inference, and efficient inference, such as the well-known Kalman filters, Kalman & Bucy (1961), can prohibit sufficiently complex system classes.
Leveraging a recently proposed estimator based on variational inference, stochastic gradient variational Bayes (SGVB, Kingma & Welling (2013); Rezende et al. (2014)), approximate inference of latent variables becomes tractable. Extensions to time series have been shown in Bayer & Osendorfer (2014); Chung et al. (2015). Empirically, they showed considerable improvements in marginal data likelihood, i.e., compression, but lack full-information latent states, which prohibits, e.g., long-term sampling. Yet, in a wide range of applications, full-information latent states should be valued over compression. This is crucial if the latent spaces are used in downstream applications.
Our contribution is, to our knowledge, the first model that (i) enforces the latent state-space model assumptions, allowing for reliable system identification, and plausible long-term prediction of the observable system, (ii) provides the corresponding inference mechanism with rich dependencies, (iii) inherits the merit of neural architectures to be trainable on raw data such as images or other sensory inputs, and (iv) scales to large data due to optimization of parameters based on stochastic gradient descent, Bottou (2010). Hence, our model has the potential to exploit systems theory methodology for downstream tasks, e.g., control or model-based reinforcement learning, Sutton (1996).
2 BACKGROUND AND RELATED WORK
2.1 PROBABILISTIC MODELING AND FILTERING OF DYNAMICAL SYSTEMS
We consider non-linear dynamical systems with observations xt ∈ X ⊂ Rnx , depending on control inputs (or actions) ut ∈ U ⊂ Rnu . Elements of X can be high-dimensional sensory data, e.g., raw images. In particular they may exhibit complex non-Markovian transitions. Corresponding timediscrete sequences of length T are denoted as x1:T = (x1,x2, . . . ,xT ) and u1:T = (u1,u2, . . . ,uT ).
We are interested in a probabilistic model1 p(x1:T | u1:T ). Formally, we assume the graphical model p(x1:T | u1:T ) = ∫ p(x1:T | z1:T ,u1:T ) p(z1:T | u1:T ) dz1:T , (1)
where z1:T , zt ∈ Z ⊂ Rnz , denotes the corresponding latent sequence. That is, we assume a generative model with an underlying latent dynamical system with emission model p(x1:T | z1:T ,u1:T ) and transition model p(z1:T | u1:T ). We want to learn both components, i.e., we want to perform latent system identification. In order to be able to apply the identified system in downstream tasks, we need to find efficient posterior inference distributions p(z1:T | x1:T ). Three common examples are prediction, filtering, and smoothing: inference of zt from x1:t−1, x1:t, or x1:T , respectively. Accurate identification and efficient inference are generally competing tasks, as a wider generative model class typically leads to more difficult or even intractable inference.
The transition model is imperative for achieving good long-term results: a bad transition model can lead to divergence of the latent state. Accordingly, we put special emphasis on it through a Bayesian treatment. Assuming that the transitions may differ for each time step, we impose a regularizing prior distribution on a set of transition parameters β1:T :
(1) = ∫∫
p(x1:T | z1:T ,u1:T )p(z1:T | β1:T ,u1:T ) p(β1:T ) dβ1:T dz1:T (2)
To obtain state-space models, we impose assumptions on emission and state transition model,
p(x1:T | z1:T ,u1:T ) = T∏ t=1 p(xt | zt), (3)
p(z1:T | β1:T ,u1:T ) = T−1∏ t=0 p(zt+1 | zt,ut,βt). (4)
Equations (3) and (4) assume that the current state zt contains all necessary information about the current observation xt, as well as the next state zt+1 (given the current control input ut and transition parameters βt). That is, in contrast to observations, zt exhibits Markovian behavior.
A typical example of these assumptions are Linear Gaussian Models (LGMs), i.e., both state transition and emission model are affine transformations with Gaussian offset noise,
zt+1 = Ftzt +Btut +wt wt ∼ N (0,Qt), (5) xt = Htzt + yt yt ∼ N (0,Rt). (6)
Typically, state transition matrix Ft and control-input matrix Bt are assumed to be given, so that βt = wt. Section 3.3 will show that our approach allows other variants such as βt = (Ft,Bt,wt). Under the strong assumptions (5) and (6) of LGMs, inference is provably solved optimally by the well-known Kalman filters. While extensions of Kalman filters to nonlinear dynamical systems exist, Julier & Uhlmann (1997), and are successfully applied in many areas, they suffer from two major drawbacks: firstly, its assumptions are restrictive and are violated in practical applications, leading to suboptimal results. Secondly, parameters such as Ft and Bt have to be known in order to perform posterior inference. There have been efforts to learn such system dynamics, cf. Ghahramani & Hinton (1996); Honkela et al. (2010) based on the expectation maximization (EM) algorithm or Valpola & Karhunen (2002), which uses neural networks. However, these algorithms are not applicable in cases
1Throughout this paper, we consider u1:T as given. The case without any control inputs can be recovered by setting U = ∅, i.e., not conditioning on control inputs.
where the true posterior distribution is intractable. This is the case if, e.g., image sequences are used, since the posterior is then highly nonlinear—typical mean-field assumptions on the approximate posterior are too simplified. Our new approach will tackle both issues, and moreover learn both identification and inference jointly by exploiting Stochastic Gradient Variational Bayes.
2.2 STOCHASTIC GRADIENT VARIATIONAL BAYES (SGVB) FOR TIME SERIES DISTRIBUTIONS
Replacing the bottleneck layer of a deterministic auto-encoder with stochastic units z, the variational auto-encoder (VAE, Kingma & Welling (2013); Rezende et al. (2014)) learns complex marginal data distributions on x in an unsupervised fashion from simpler distributions via the graphical model
p(x) = ∫ p(x, z) dz = ∫ p(x | z)p(z) dz.
In VAEs, p(x | z) ≡ pθ(x | z) is typically parametrized by a neural network with parameters θ. Within this framework, models are trained by maximizing a lower bound to the marginal data log-likelihood via stochastic gradients:
ln p(x) ≥ Eqφ(z|x)[ln pθ(x | z)]−KL(qφ(z | x) || p(z)) =: LSGVB(x, φ, θ) (7) This is provably equivalent to minimizing the KL-divergence between the approximate posterior or recognition model qφ(z | x) and the true, but usually intractable posterior distribution p(z | x). qφ is parametrized by a neural network with parameters φ.
The principle of VAEs has been transferred to time series, Bayer & Osendorfer (2014); Chung et al. (2015). Both employ nonlinear state transitions in latent space, but violate eq. (4): Observations are directly included in the transition process. Empirically, reconstruction and compression work well. The state space Z , however, does not reflect all information available, which prohibits plausible generative long-term prediction. Such phenomena with generative models have been explained in Theis et al. (2015).
In Krishnan et al. (2015), the state-space assumptions (3) and (4) are softly encoded in the Deep Kalman Filter (DKF) model. Despite that, experiments, cf. section 4, show that their model fails to extract information such as velocity (and in general time derivatives), which leads to similar problems with prediction.
Johnson et al. (2016) give an algorithm for general graphical model variational inference, not tailored to dynamical systems. In contrast to previously discussed methods, it does not violate eq. (4). The approaches differ in that the recognition model outputs node potentials in combination with message passing to infer the latent state. Our approach focuses on learning dynamical systems for controlrelated tasks and therefore uses a neural network for inferring the latent state directly instead of an inference subroutine.
Others have been specifically interested in applying variational inference for controlled dynamical systems. In Watter et al. (2015) (Embed to Control—E2C), a VAE is used to learn the mappings to and from latent space. The regularization is clearly motivated by eq. (7). Still, it fails to be a mathematically correct lower bound to the marginal data likelihood. More significantly, their recognition model requires all observations that contain information w.r.t. the current state. This is nothing short of an additional temporal i.i.d. assumption on data: Multiple raw samples need to be stacked into one training sample such that all latent factors (in particular all time derivatives) are present within one sample. The task is thus greatly simplified, because instead of time-series, we learn a static auto-encoder on the processed data.
A pattern emerges: good prediction should boost compression. Still, previous methods empirically excel at compression, while prediction will not work. We conjecture that this is caused by previous methods trying to fit the latent dynamics to a latent state that is beneficial for reconstruction. This encourages learning of a stationary auto-encoder with focus of extracting as much from a single observation as possible. Importantly, it is not necessary to know the entire sequence for excellent reconstruction of single time steps. Once the latent states are set, it is hard to adjust the transition to them. This would require changing the latent states slightly, and that comes at a cost of decreasing the reconstruction (temporarily). The learning algorithm is stuck in a local optimum with good reconstruction and hence good compression only. Intriguingly, E2C bypasses this problem with its data augmentation.
This leads to a key contribution of this paper: We force the latent space to fit the transition—reversing the direction, and thus achieving the state-space model assumptions and full information in the latent states.
3 DEEP VARIATIONAL BAYES FILTERS
3.1 REPARAMETRIZING THE TRANSITION
The central problem for learning latent states system dynamics is efficient inference of a latent space that obeys state-space model assumptions. If the latter are fulfilled, the latent space must contain all information. Previous approaches emphasized good reconstruction, so that the space only contains information necessary for reconstruction of one time step. To overcome this, we establish gradient paths through transitions over time so that the transition becomes the driving factor for shaping the latent space, rather than adjusting the transition to the recognition model’s latent space. The key is to prevent the recognition model qφ(z1:T | x1:T ) from directly drawing the latent state zt. Similar to the reparametrization trick from Kingma & Welling (2013); Rezende et al. (2014) for making the Monte Carlo estimate differentiable w.r.t. the parameters, we make the transition differentiable w.r.t. the last state and its parameters:
zt+1 = f(zt,ut,βt) (8) Given the stochastic parameters βt, the state transition is deterministic (which in turn means that by marginalizing βt, we still have a stochastic transition). The immediate and crucial consequence is that errors in reconstruction of xt from zt are backpropagated directly through time.
This reparametrization has a couple of other important implications: the recognition model no longer infers latent states zt, but transition parameters βt. In particular, the gradient ∂zt+1/∂zt is well-defined from (8)—gradient information can be backpropagated through the transition.
This is different from the method used in Krishnan et al. (2015), where the transition only occurs in the KL-divergence term of their loss function (a variant of eq. (7)). No gradient from the generative model is backpropagated through the transitions.
Much like in eq. (5), the stochastic parameters includes a corrective offset term wt, which emphasizes the notion of the recognition model as a filter. In theory, the learning algorithm could still learn the transition as zt+1 = wt. However, the introduction of βt also enables us to regularize the transition with meaningful priors, which not only prevents overfitting the recognition model, but also enforces meaningful manifolds in the latent space via transition priors. Ignoring the potential of the transition over time yields large penalties from these priors. Thus, the problems outlined in Section 2 are overcome by construction.
To install such transition priors, we split βt = (wt,vt). The interpretation of wt is a sample-specific process noise which can be inferred from incoming data, like in eq. (5). On the other hand, vt
are universal transition parameters, which are sample-independent (and are only inferred from data during training). This corresponds to the idea of weight uncertainty in Hinton & Van Camp (1993). This interpretation leads to a natural factorization assumption on the recognition model:
qφ(β1:T | x1:T ) = qφ(w1:T | x1:T ) qφ(v1:T ) (9)
When using the fully trained model for generative sampling, i.e., sampling without input, the universal state transition parameters can still be drawn from qφ(v1:T ), whereas w1:T is drawn from the prior in the absence of input data.
Figure 1 shows the underlying graphical model and the inference procedure. Figure 2a shows a generic view on our new computational architecture. An example of a locally linear transition parametrization will be given in section 3.3.
3.2 THE LOWER BOUND OBJECTIVE FUNCTION
In analogy to eq. (7), we now derive a lower bound to the marginal likelihood p(x1:T | u1:T ). After reflecting the Markov assumptions (3) and (4) in the factorized likelihood (2), we have:
p(x1:T | u1:T ) = ∫∫ p(β1:T ) T∏ t=1 pθ(xt | zt) T−1∏ t=0 p(zt+1 | zt,ut,βt) dβ1:T dz1:T
Due to the deterministic transition given βt+1, the last term is a product of Dirac distributions and the overall distribution simplifies greatly:
p(x1:T | u1:T ) = ∫ p(β1:T ) T∏ t=1 pθ(xt | zt) ∣∣∣ zt=f(zt−1,ut−1,βt−1)
dβ1:T( = ∫ p(β1:T )pθ(x1:T | z1:T ) dβ1:T )
The last formulation is for notational brevity: the term pθ(x1:T | z1:T ) is not independent of β1:T and u1:T . We now derive the objective function, a lower bound to the data likelihood:
ln p(x1:T | u1:T ) = ln ∫ p(β1:T )pθ(x1:T | z1:T )
qφ(β1:T | x1:T ,u1:T ) qφ(β1:T | x1:T ,u1:T ) dβ1:T
≥ ∫ qφ(β1:T | x1:T ,u1:T ) ln ( pθ(x1:T | z1:T )
p(β1:T )
qφ(β1:T | x1:T ,u1:T )
) dβ1:T
= Eqφ [ln pθ(x1:T | z1:T )− ln qφ(β1:T | x1:T ,u1:T ) + ln p(β1:T )] (10) = Eqφ [ln pθ(x1:T | z1:T )]−KL(qφ(β1:T | x1:T ,u1:T ) || p(β1:T )) (11) =: LDVBF(x1:T , θ, φ | u1:T )
Our experiments show that an annealed version of (10) is beneficial to the overall performance:
(10′) = Eqφ [ci ln pθ(x1:T | z1:T )− ln qφ(β1:T | x1:T ,u1:T ) + ci ln p(w1:T ) + ln p(v1:T )] Here, ci = max(1, 0.01 + i/TA) is an inverse temperature that increases linearly in the number of gradient updates i until reaching 1 after TA annealing iterations. Similar annealing schedules have been applied in, e.g., Ghahramani & Hinton (2000); Mandt et al. (2016); Rezende & Mohamed (2015), where it is shown that they smooth the typically highly non-convex error landscape. Additionally, the transition prior p(v1:T ) was estimated during optimization, i.e., through an empirical Bayes approach. In all experiments, we used isotropic Gaussian priors.
3.3 EXAMPLE: LOCALLY LINEAR TRANSITIONS
We have derived a learning algorithm for time series with particular focus on general transitions in latent space. Inspired by Watter et al. (2015), this section will show how to learn a particular instance: locally linear state transitions. That is, we set eq. (8) to
zt+1 = Atzt +Btut +Ctwt, t = 1, . . . , T, (12)
where wt is a stochastic sample from the recognition model and At,Bt, and Ct are matrices of matching dimensions. They are stochastic functions of zt and ut (thus local linearity). We draw
vt = { A (i) t ,B (i) t ,C (i) t | i = 1, . . . ,M } ,
from qφ(vt), i.e.,M triplets of matrices, each corresponding to data-independent, but learned globally linear system. These can be learned as point estimates. We employed a Bayesian treatment as in Blundell et al. (2015). We yield At,Bt, and Ct as state- and control-dependent linear combinations:
At = M∑ i=1 α (i) t A (i) t
αt = fψ(zt,ut) ∈ RM
Bt = M∑ i=1 α (i) t B (i) t
Ct = M∑ i=1 α (i) t C (i) t
The computation is depicted in fig. 2b. The function fψ can be, e.g., a (deterministic) neural network with weights ψ. As a subset of the generative parameters θ, ψ is part of the trainable parameters of our model. The weight vector αt is shared between the three matrices. There is a correspondence to eq. (5): At and Ft, Bt and Bt, as well as CtC>t and Qt are related.
We used this parametrization of the state transition model for our experiments. It is important that the parametrization is up to the user and the respective application.
4 EXPERIMENTS AND RESULTS
In this section we validate that DVBF with locally linear transitions (DVBF-LL) (section 3.3) outperforms Deep Kalman Filters (DKF, Krishnan et al. (2015)) in recovering latent spaces with full information. 2 We focus on environments that can be simulated with full knowledge of the
2We do not include E2C, Watter et al. (2015), due to the need for data modification and its inability to provide a correct lower bound as mentioned in section 2.2.
ground truth latent dynamical system. The experimental setup is described in the Supplementary Material. We published the code for DVBF and a link will be made available at https://brml. org/projects/dvbf.
4.1 DYNAMIC PENDULUM
In order to test our algorithm on truly non-Markovian observations of a dynamical system, we simulated a dynamic torque-controlled pendulum governed by the differential equation
ml2ϕ̈(t) = −µϕ̇(t) +mgl sinϕ(t) + u(t),
m = l = 1, µ = 0.5, g = 9.81, via numerical integration, and then converted the ground-truth angle ϕ into an image observation in X . The one-dimensional control corresponds to angle acceleration (which is proportional to joint torque). Angle and angular velocity fully describe the system.
Figure 3 shows the latent spaces for identical input data learned by DVBF-LL and DKF, respectively, colored with the ground truth in the top row. It should be noted that latent samples are shown, not means of posterior distributions. The state-space model was allowed to use three latent dimensions. As we can see in fig. 3a, DVBF-LL learned a two-dimensional manifold embedding, i.e., it encoded the angle in polar coordinates (thus circumventing the discontinuity of angles modulo 2π). The bottom row shows ordinary least-squares regressions (OLS) underlining the performance: there exists a high correlation between latent states and ground-truth angle and angular velocity for DVBF-LL. On the contrary, fig. 3b verifies our prediction that DKF is equally capable of learning the angle, but extracts little to no information on angular velocity.
The OLS regression results shown in table 1 validate this observation.3 Predicting sin(ϕ) and cos(ϕ), i.e., polar coordinates of the ground-truth angle ϕ, works almost equally well for DVBF-LL and DKF, with DVBF-LL slightly outperforming DKF. For predicting the ground truth velocity ϕ̇, DVBF-LL
3Linear regression is a natural choice: after transforming the ground truth to polar coordinates, an affine transformation should be a good fit for predicting ground truth from latent states. We also tried nonlinear regression with vanilla neural networks. While not being shown here, the results underlined the same conclusion.
Table 1: Results for pendulum OLS regressions of all latent states on respective dependent variable.
Dependent ground truth variable
DVBF-LL DKF Log-Likelihood R2 Log-Likelihood R2
sin(ϕ) 3990.8 0.961 1737.6 0.929 cos(ϕ) 7231.1 0.982 6614.2 0.979 ϕ̇ −11139 0.916 −20289 0.035
(a) Generative latent walk. (b) Reconstructive latent walk.
...
...
... 51 10 15 20 40 45
(c) Ground truth (top), reconstructions (middle), generative samples (bottom) from identical initial latent state.
Figure 4: (a) Latent space walk in generative mode. (b) Latent space walk in filtering mode. (c) Ground truth and samples from recognition and generative model. The reconstruction sampling has access to observation sequence and performs filtering. The generative samples only get access to the observations once for creating the initial state while all subsequent samples are predicted from this single initial state. The red bar indicates the length of training sequences. Samples beyond show the generalization capabilities for sequences longer than during training. The complete sequence can be found in the Appendix in fig. 7.
shows remarkable performance. DKF, instead, contains hardly any information, resulting in a very low goodness-of-fit score of R2 = 0.035.
Figure 4 shows that the strong relation between ground truth and latent state is beneficial for generative sampling. All plots show 100 time steps of a pendulum starting from the exact same latent state and not being actuated. The top row plots show a purely generative walk in the latent space on the left, and a walk in latent space that is corrected by filtering observations on the right. We can see that both follow a similar trajectory to an attractor. The generative model is more prone to noise when approaching the attractor.
The bottom plot shows the first 45 steps of the corresponding observations (top row), reconstructions (middle row), and generative samples (without correcting from observations). Interestingly, DVBF works very well even though the sequence is much longer than all training sequences (indicated by the red line).
Table (2) shows values of the lower bound to the marginal data likelihood (for DVBF-LL, this corresponds to eq. (11)). We see that DVBF-LL outperforms DKF in terms of compression, but only
Table 2: Average test set objective function values for pendulum experiment.
Lower Bound = Reconstruction Error − KL divergence DVBF-LL 798.56 802.06 3.50
DKF 784.70 788.58 3.88
(a) Latent walk of bouncing ball. (b) Latent space velocities.
Figure 5: (a) Two dimensions of 4D bouncing ball latent space. Ground truth x and y coordinates are combined into a regular 3×3 checkerboard coloring. This checkerboard is correctly extracted by the embedding. (b) Remaining two latent dimensions. Same latent samples, colored with ball velocities in x and y direction (left and right image, respectively). The smooth, perpendicular coloring indicates that the ground truth value is stored in the latent dimension.
with a slight margin, which does not reflect the better generative sampling as Theis et al. (2015) argue.
4.2 BOUNCING BALL
The bouncing ball experiment features a ball rolling within a bounding box in a plane. The system has a two-dimensional control input, added to the directed velocity of the ball. If the ball hits the wall, it bounces off, so that the true dynamics are highly dependent on the current position and velocity of the ball. The system’s state is four-dimensional, two dimensions each for position and velocity.
Consequently, we use a DVBF-LL with four latent dimensions. Figure 5 shows that DVBF again captures the entire system dynamics in the latent space. The checkerboard is quite a remarkable result: the ground truth position of the ball lies within the 2D unit square, the bounding box. In order to visualize how ground truth reappears in the learned latent states, we show the warping of the ground truth bounding box into the latent space. To this end, we partitioned (discretized) the ground truth unit square into a regular 3x3 checkerboard with respective coloring. We observed that DVBF learned to extract the 2D position from the 256 pixels, and aligned them in two dimensions of the latent space in strong correspondence to the physical system. The algorithm does the exact same pixel-to-2D inference that a human observer automatically does when looking at the image.
4.3 TWO BOUNCING BALLS
Another more complex environment4 features two balls in a bounding box. We used a 10-dimensional latent space to fully capture the position and velocity information of the balls. Reconstruction and generative samples are shown in fig. 6. Same as in the pendulum example we get a generative model with stable predictions beyond training data sequence length.
5 CONCLUSION
We have proposed Deep Variational Bayes Filters (DVBF), a new method to learn state space models from raw non-Markovian sequence data. DVBFs perform latent dynamic system identification, and subsequently overcome intractable inference. As DVBFs make use of stochastic gradient variational Bayes they naturally scale to large data sets. In a series of vision-based experiments we demonstrated that latent states can be recovered which identify the underlying physical quantities. The generative model showed stable long-term predictions far beyond the sequence length used during training.
ACKNOWLEDGEMENTS
Part of this work was conducted at Chair of Robotics and Embedded Systems, Department of Informatics, Technische Universität München, Germany, and supported by the TACMAN project, EC Grant agreement no. 610967, within the FP7 framework programme.
We would like to thank Jost Tobias Springenberg, Adam Kosiorek, Moritz Münst, and anonymous reviewers for valuable input.
A SUPPLEMENTARY TO LOWER BOUND
A.1 ANNEALED KL-DIVERGENCE
We used the analytical solution of the annealed KL-divergence in eq. (10) for optimization:
Eqφ [− ln qφ(w1:T | x1:T ,u1:T ) + ci ln p(w1:T )] =
ci 1
2 ln(2πσ2p)−
1 2 ln(2πσ2q ) + ci
σ2q + (µq − µp)2
2σ2p − 1 2
B SUPPLEMENTARY TO IMPLEMENTATION
B.1 EXPERIMENTAL SETUP
In all our experiments, we use sequences of 15 raw images of the respective system with 16×16 pixels each, i.e., observation space X ⊂ R256, as well as control inputs of varying dimension and interpretation depending on the experiment. We used training, validation and test sets with 500 sequences each. Control input sequences were drawn randomly (“motor babbling”). Additional details about the implementation can be found in the published code at https://brml.org/ projects/dvbf.
B.2 ADDITIONAL EXPERIMENT PLOTS
B.3 IMPLEMENTATION DETAILS FOR DVBF IN PENDULUM EXPERIMENT
• Input: 15 timesteps of 162 observation dimensions and 1 action dimension
• Latent Space: 3 dimensions
• Observation Network p(xt|zt) = N (xt;µ(zt), σ): 128 ReLU + 162 identity output
• Recognition Model: 128 ReLU + 6 identity output
q(wt|zt,xt+1,ut) = N (wt;µ, σ), (µ, σ) = f(zt,xt+1,ut)
• Transition Network αt(zt): 16 softmax output
• Initial Network w1 ∼ p(x1:T ): Fast Dropout BiRNN with: 128 ReLU + 3 identity output
• Initial Transition z1(w1): 128 ReLU + 3 identity output
• Optimizer: adadelta, 0.1 step rate
• Inverse temperature: c0 = 0.01, updated every 250th gradient update, TA = 105 iterations
• Batch-size: 500
B.4 IMPLEMENTATION DETAILS FOR DVBF IN BOUNCING BALL EXPERIMENT
• Input: 15 timesteps of 162 observation dimensions and 2 action dimension • Latent Space: 4 dimensions • Observation Network p(xt|zt) = N (xt;µ(zt), σ): 128 ReLU + 162 identity output • Recognition Model: 128 ReLU + 8 identity output
q(wt|zt,xt+1,ut) = N (wt;µ, σ), (µ, σ) = f(zt,xt+1,ut)
• Transition Network αt(zt): 16 softmax output • Initial Network w1 ∼ p(x1:T ): Fast Dropout BiRNN with: 128 ReLU + 4 identity output • Initial Transition z1(w1): 128 ReLU + 4 identity output • Optimizer: adadelta, 0.1 step rate • Inverse temperature: c0 = 0.01, updated every 250th gradient update, TA = 105 iterations • Batch-size: 500
B.5 IMPLEMENTATION DETAILS FOR DVBF IN TWO BOUNCING BALLS EXPERIMENT
• Input: 15 timesteps of 202 observation dimensions and 2000 samples • Latent Space: 10 dimensions • Observation Network p(xt|zt) = N (xt;µ(zt), σ): 128 ReLU + 202 sigmoid output • Recognition Model: 128 ReLU + 20 identity output
q(wt|zt,xt+1,ut) = N (wt;µ, σ), (µ, σ) = f(zt,xt+1,ut)
• Transition Network αt(zt): 64 softmax output • Initial Network w1 ∼ p(x1:T ): MLP with: 128 ReLU + 10 identity output • Initial Transition z1(w1): 128 ReLU + 10 identity output • Optimizer: adam, 0.001 step rate • Inverse temperature: c0 = 0.01, updated every gradient update, TA = 2 105 iterations • Batch-size: 80
B.6 IMPLEMENTATION DETAILS FOR DKF IN PENDULUM EXPERIMENT
• Input: 15 timesteps of 162 observation dimensions and 1 action dimension • Latent Space: 3 dimensions • Observation Network p(xt|zt) = N (xt;µ(zt), σ(zt)): 128 Sigmoid + 128 Sigmoid + 2 162
identity output • Recognition Model: Fast Dropout BiRNN 128 Sigmoid + 128 Sigmoid + 3 identity output • Transition Network p(zt|zt−1,ut−1): 128 Sigmoid + 128 Sigmoid + 6 output • Optimizer: adam, 0.001 step rate • Inverse temperature: c0 = 0.01, updated every 25th gradient update, TA = 2000 iterations • Batch-size: 500

Review:
This paper presents a variational inference based method for learning nonlinear dynamical systems. Unlike the deep Kalman filter, the proposed method learns a state space model, which forces the latent state to maintain all of the information relevant to predictions, rather than leaving it implicit in the observations. Experiments show the proposed method is better able to learn meaningful representations of sequence data.
The proposed DVBF is well motivated, and for the most part the presentation is clear. The experiments show interesting results on illustrative toy examples. I think the contribution is interesting and potentially useful, so I’d recommend acceptance.
The SVAE method of Johnson et al. (2016) deserves more discussion than the two sentences devoted to it, since the method seems pretty closely related. Like the DVBF, the SVAE imposes a Markovianity assumption, and it is able to handle similar kinds of problems. From what I understand, the most important algorithmic difference is that the SVAE q network predicts potentials, whereas the DVBF q network predicts innovations. What are the tradeoffs between the two?  Section 2.2 says they do the latter in the interest of solving control-related tasks, but I’m not clear why this follows. 
Is there a reason SVAEs don’t meet all the desiderata mentioned at the end of the Introduction?
Since the SVAE code is publicly available, one could probably compare against it in the experiments. 
I’m a bit confused about the role of uncertainty about v. In principle, one could estimate the transition parameters by maximum likelihood (i.e. fitting a point estimate of v), but this isn’t what’s done. Instead, v is integrated out as part of the marginal likelihood, which I interpret as giving the flexibility to model different dynamics for different sequences. But if this is the case, then shouldn’t the q distribution for v depend on the data, rather than being data-independent as in Eqn. (9)?