Paper:
In this work we introduce a new framework for performing temporal predictions in the presence of uncertainty. It is based on a simple idea of disentangling components of the future state which are predictable from those which are inherently unpredictable, and encoding the unpredictable components into a low-dimensional latent variable which is fed into a forward model. Our method uses a supervised training objective which is fast and easy to train. We evaluate it in the context of video prediction on multiple datasets and show that it is able to consistently generate diverse predictions without the need for alternating minimization over a latent space or adversarial training.

Review:
Summary: 
I like the general idea of learning "output stochastic" noise models in the paper, but the idea is not fully explored (in terms of reasonable variations and their comparative performance).  I don't fully understand the rationale for the experiments: I cannot speak to the reasons for the GAN's failure (GANs are not easy to train and this seems to be reflected in the results); the newly proposed model seems to improve with samples simply because the evaluation seems to reward the best sample.  I.e., with enough throws, I can always hit the bullseye with a dart even when blindfolded.
Comments:
The model proposes to learn a conditional stochastic deep model by training an output noise model on the input x_i and the residual y_i - g(x_i).  The trained residual function can be used to predict a residual z_i for x_i.  Then for out-of-sample prediction for x*, the paper appears to propose sampling a z uniformly from the training data {z_i}_i (it is not clear from the description on page 3 that this uniformly sampled z* = z_i depends on the actual x* -- as far as I can tell it does not).  The paper does suggest learning a p(z|x) but does not provide implementation details nor experiment with this approach.
I like the idea of learning an "output stochastic" model -- it is much simpler to train than an "input stochastic" model that is more standard in the literature (VAE, GAN) and there are many cases where I think it could be quite reasonable.  However, I don't think the authors explore the idea well enough -- they simply appear to propose a non-parametric way of learning the stochastic model (sampling from the training data z_i's) and do not compare to reasonable alternative approaches.  To start, why not plot the empirical histogram of p(z|x) (for some fixed x's) to get a sense of how well-behaved it is as a distribution.  Second, why not simply propose learning exponential family models where the parameters of these models are (deep nets) conditioned on the input?  One could even start with a simple Gaussian and linear parameterization of the mean and variance in terms of x.  If the contribution of the paper is the "output stochastic" noise model, I think it is worth experimenting with the design options one has with such a model.
The experiments range over 4 video datasets.  PSNR is evaluated on predicted frames -- PSNR does not appear to be explicitly defined but I am taking it to be the metric defined in the 2nd paragraph from the bottom on page 7.  The new model "EEN" is compared to a deterministic model and conditional GAN.  The GAN never seems to perform well -- the authors claim mode collapse, but I wonder if the GAN was simply hard to train in the first place and this is the key reason?  Unsurprisingly (since the EEN noise does not seem to be conditioned on the input), the baseline deterministic model performs quite well.  If I understand what is being evaluated correctly (i.e., best random guess) then I am not surprised the EEN can perform better with enough random samples.  Have we learned anything?