Paper:
Video prediction is a challenging task: models have to account for the inherent uncertainty of the future. Most works in the literature are based on stochastic imageautoregressive recurrent networks, raising several performance and applicability issues. An alternative is to use fully latent temporal models which untie frame synthesis and dynamics. However, no such model for video prediction has been proposed in the literature yet, due to design and training difficulties. In this paper, we overcome these difficulties by introducing a novel stochastic temporal model. It is based on residual updates of a latent state, motivated by discretization schemes of differential equations. This first-order principle naturally models video dynamics as it allows our simpler, lightweight, interpretable, latent model to outperform prior state-of-the-art methods on challenging datasets.

Review:
Summary:
The paper proposes a video prediction method based on State-Space Models. The paper describes two main contributions:
1. By learning dynamics in the latent state space, the method avoids the high computational cost and accumulating image reconstruction errors of autoregressive models that condition on generated frames.
2. To model dynamics, the paper proposes a residual update rule inspired by Euler’s method to solve ODEs. According to this rule, the update to the latent state y_t is modeled as an additive residual f(y_t, z_{t+1}). This has the advantage that the step size of the discretization can be adjusted freely, e.g. between training and inference. 
The paper provides extensive experimental comparison of their model to the SVG and SAVP models on several standard datasets. The paper further contains experiments illustrating features of the model such as disentangling dynamics and content, and interpolation of dynamics in the latent space.
Decision:
The paper is written clearly and the mathematical treatment and experiments appear rigorous. The idea of predicting video using state-space models is interesting and promising. However, as described below, the paper overstates its novelty and falls short of showing the advantages of the method beyond incremental improvements on frame-wise image quality metrics. I therefore suggest rejection in its current version.
Supporting arguments and suggestions:
1. The idea to use fully latent models for video prediction, to untie frame synthesis and dynamics, is not new and the paper does not fully cite this literature. For example, [1] and [2] perform unsupervised, non-autoregressive video prediction. The differences to these models should be discussed.
2. The advantages of the residual update rule are not made clear enough. The parallels to the ODE literature seem tenuous. The main advantage described in the paper is the ability to synthesize videos at different frame rates, but interpolation over such short time horizons is not a hard problem. At least, the paper should compare to existing methods for frame interpolation. Apart from interpolation (variable step size), it appears that the update rule could be changed from y_{t+1} = y_t + f(y_t, z_{t+1}) to y_{t+1} = f(y_t, z_{t+1}) without impact to the model. How is it different from the standard VRNN formulation [3]? More experiments to show the advantage of the proposed update rule would be helpful.
3. Some of the experiments seem like interesting starting points but do not support general claims. For example, Fig 2 (b) shows that the proposed dynamics model is better than an MLP or GRU on deterministic Moving MNIST, but is this also true on real datasets, which have much more complex dynamics? Similarly, the interpolation in Figure 9 is intriguing, but it would be helpful to describe and test how this ability is useful for applications of the predictive model.
4. The comparisons use frame-wise metrics of image quality (PSNR, SSIM, LPIPS). Even though they are common in the literature, these metrics are unsuitable for comparing long video sequences due to their stochasticity. The metrics are probably dominated by relatively uninteresting features such as the quality of the static background. Metrics for comparing entire videos exist (e.g. FVD [4]) and should be used. Even better, the paper should demonstrate the usefulness of the model for downstream tasks such as reinforcement learning, although I understand that this may be out of scope.
Minor comments:
- As far as I know, the correct term for error terms is residual, not residue.
- What do the error bars in the figures show? Please add this information to the figure legends.
[1] Wichers et al., 2018, https://arxiv.org/pdf/1806.04768.pdf
[2] Minderer et al., 2019, https://arxiv.org/abs/1906.07889
[3] Chung et al, 2015, https://arxiv.org/abs/1506.02216
[4] Unterthiner et al., 2018, https://arxiv.org/abs/1812.01717