Paper:
We propose an efficient inference method for switching nonlinear dynamical systems. The key idea is to learn an inference network which can be used as a proposal distribution for the continuous latent variables, while performing exact marginalization of the discrete latent variables. This allows us to use the reparameterization trick, and apply end-to-end training with stochastic gradient descent. We show that the proposed method can successfully segment time series data (including videos) into meaningful "regimes", by using the piece-wise nonlinear dynamics.

Review:
Thank you for an interesting read.
As far as I understand, the paper claims two contributions:
1. A combination of collapsed variational inference and amortised inference for SNLDS, to make the training pipeline fully differentiable;
2. An improved loss function upon the variational lowerbound (ELBO) to force the model to use the discrete states.
======= novelty =======
The 1st idea is combinatorial: 
1. The forward-backward algorithm is a standard inference method for HMM-like sequence models; collapsed variational inference has been investigated extensively in 2000s when hierarchical Bayes models were actively developed; amortised inference is widely used in variational auto-encoders. 
2. The combination of the above two inference methods on S(N)LDS is new to the best of my knowledge. However, this combination has been proposed on a similar model called Kalman VAE ( Fraccaro et al. 2017), where the sequence model can be viewed as a "soft" version of SLDS. 
The 2nd idea is interesting but not very well explained to some extent:
1. The goal of the modified objective function is to encourage the model to use the discrete states (instead of pushing all useful information to the continuous states). It is interesting as it regularises the *exact posterior* of the discrete states conditioned on the *approximately inferred* continuous states. 
2. I believe the entropy regulariser is non-differentiable as it is based on a *histogram* estimate of the temporally averaged discrete state distribution. How exactly is this regulariser implemented? 
3. I agree adding the KL regulariser can avoid the iterating assignment pathology, however, is random assignment of the regime preferred in any case? From the introductory example, I think contiguous segments are preferred.
======= significance =======
Experiments consider 3 synthetic examples for sequence segmentation (so that ground truth is available). The proposed approach performs significantly better which is a good sign. The paper also provides useful analysis on the effects of balancing parameter tempering which is always welcome.
However, two baselines are missing:
1. To claim the significance of the collapsed variational inference approach, a non-collapsed inference version of the proposed SNLDS model needs to be compared. The authors did discuss this and mentioned possible workarounds (e.g. using the Gumbel-softmax trick for discrete state inference), but the comparison is not reported. If compared, this will serve well as an ablation study for the inference method.
2. The paper also provides comparisons across models, but I do think the Kalman VAE model needs to be compared to SNLDS. Both models are more flexible than the original SLDS, but the complexity is added in different ways. Since I think the inference mechanisms are similar (both using forward-backward inference for top-level latents and amortised inference for bottom-level latents), this comparison would provide a better ablation study on the modelling side.
======= clarity =======
1. The paper presentation is overall clear to me, although I found the many sentences in parenthesis a bit distracted, so I would suggest maybe using footnotes for them instead. 
2. For readers who are less familiar with HMMs/forward-backward algorithms, the papers can be difficult to understand, as it skips all the detailed computation of the gamma terms. I would suggest adding the details in the appendix, and/or visualise the intuition using e.g. message passing on factor graphs.
3. I found the related work well presented with most relevant papers, although I do think the Kalman VAE approach is highly relevant which needs to be cited and discussed.
======== references ========
Fraccaro et al. (2017). A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning. NeurIPS 2017