Paper:
PredRNN: Recurrent Neural Networks for Predictive Learning using Spatiotemporal LSTMs
The predictive learning of spatiotemporal sequences aims to generate future images by learning from the historical frames, where spatial appearances and temporal variations are two crucial structures. This paper models these structures by presenting a predictive recurrent neural network (PredRNN). This architecture is enlightened by the idea that spatiotemporal predictive learning should memorize both spatial appearances and temporal variations in a unified memory pool. Concretely, memory states are no longer constrained inside each LSTM unit. Instead, they are allowed to zigzag in two directions: across stacked RNN layers vertically and through all RNN states horizontally. The core of this network is a new Spatiotemporal LSTM (ST-LSTM) unit that extracts and memorizes spatial and temporal representations simultaneously. PredRNN achieves the state-of-the-art prediction performance on three video prediction datasets and is a more general framework, that can be easily extended to other predictive learning tasks by integrating with other architectures.

Review:
This paper deals with predictive learning (mainly video prediction), using a RNN type of structure. A new (to my knowledge) variation of LSTM is introduced, called ST-LSTM, with recurrent connections not only in the forward time direction.
The predictive network is composed of ST-LSTM blocks. Each of these block resemble a convolutional LSTM unit, with some differences to include an additional input. This extra input comes from the last layer of the previous time step, and enters at the first layer of the current time step, it is then propagated through the layers at the current time step.
The resulting ST-LSTM is similar to the combination of two independent LSTM units, interacting through concatenation of the memories (figure 2(left)).
I couldn't find an explicit formulation of the loss (the closest I could find is equation 1). I am assuming that it is a MSE loss, but this should be confirmed, since other forms are possible.
Results are presented on the Moving MNIST (synthetic) and KTH (natural) datasets, showing both PSNR/SSIM and generations. The authors compare their PredRNN with other baselines, and show the best results on these datasets. On Moving MNIST, there is a comparison of different LSTM schemes, and ST-LSTM show the best results, which is a good result.
However, the experimental section could be stronger by having more datasets (the two datasets presented have little ambiguity in their future, it could be interesting to see how the model preforms in less contained dataset, such as Sports1m, UCF101 or the Google "Push" dataset, for instance). Although this paper present a complex (and, as it seems, good) structure for the predictor, the loss function is almost never mentioned. As (among others) [17] and [19] mention, a simple loss such as the MSE cannot work well when the future distribution is multimodal. The paper would also be stronger if compared with other methods (non LSTM-based), such as the ones presented in section 1.2. In particular, VPNs and GANs seem like strong competitors.
Notes:
- Figure 1: Although the figure is clear, I do not quite understand what the orange arrows mean (compared to the black arrows).
- Figure 2(right): As I understand it, each W box should also have an Xt input (not just W1)