Paper:
Sequential Neural Models with Stochastic Layers
How can we efficiently propagate uncertainty in a latent state representation with recurrent neural networks? This paper introduces stochastic recurrent neural networks which glue a deterministic recurrent neural network and a state space model together to form a stochastic and sequential neural generative model. The clear separation of deterministic and stochastic layers allows a structured variational inference network to track the factorization of the model's posterior distribution. By retaining both the nonlinear recursive structure of a recurrent neural network and averaging over the uncertainty in a latent path, like a state space model, we improve the state of the art results on the Blizzard and TIMIT speech modeling data sets by a large margin, while achieving comparable performances to competing methods on polyphonic music modeling.

Review:
The authors combine state space models with RNNs. The RNNs form a stochastic layer while the state space model creates a stochastic layer. Their model is trained with the tractable variational evidence lower bound (ELBO). The model is evaluated on several speech and music datasets and also compare it to similar models like STORN, VRNN and Deep Kalman Filters.
Updates after Rebuttal:  In the rebuttal the authors said (to clarify a concern of mine): "In the caption to Figure 1, we mentioned setting u_t=x_{t-1}, e.g. for language modelling. We used u_t=x_{t-1} for STORN and VRNN so that they could be compared more easily with the SRNN architecture in Figure 2. We will reiterate this in Figure’s 4 caption and elsewhere to keep it clear." I overlooked this information and it should indeed be mentioned in other captions and in the text. It is correct that when setting the action equal to the time-shifted observations all four models will have a more similar structure. However similar structure does not mean that the models are comparable (e.g. DKF still uses actions instead of observation as u_{1:T}). But this also means that there now is almost no difference between VRNN and the proposed method. The only difference is the missing connection from z to the deterministic hidden state. This does indeed "separate the noisy stochastic units from the deterministic ones", as the authors said, but this is not favorable when someone wants good generalization capabilities. After this clarification from the authors and the resulting changes in the model structure, I have to reduce the rating in the technical quality section and the novelty section.     Initial review: The generative model for STORN in Figure 4(a) is wrong. STORN has no input from any action u_t and the deterministic hidden state d_t needs input from the last generated observation x_{t-1}. VRNN in Figure 4(b) is also missing a connection from the last generative sample x_{t-1} to the hidden state d_t.  Technical quality: Experiments only evaluate negative log likelihoods and reconstruction quality. The generative model is never evaluated in itself, e.g., visualisations that show that the state transition is creating valid sequences even for multiple time-steps.  Novelty/originality: The algorithm is different from STORN and VRNN but does not provide any novel improvements upon them. The experiments show that other algorithms perform even better on most datasets.  Potential impact or usefulness: Impact is similar to STORN or VRNN since no clear improvements.  Clarity and presentation: Quite clear.