Paper:
In this paper we developed a hierarchical network model, called Hierarchical Prediction Network (HPNet) to understand how spatiotemporal memories might be learned and encoded in a representational hierarchy for predicting future video frames. The model is inspired by the feedforward, feedback and lateral recurrent circuits in the mammalian hierarchical visual system. It assumes that spatiotemporal memories are encoded in the recurrent connections within each level and between different levels of the hierarchy. The model contains a feed-forward path that computes and encodes spatiotemporal features of successive complexity and a feedback path that projects interpretation from a higher level to the level below. Within each level, the feed-forward path and the feedback path intersect in a recurrent gated circuit that integrates their signals as well as the circuit's internal memory states to generate a prediction of the incoming signals. The network learns by comparing the incoming signals with its prediction, updating its internal model of the world by minimizing the prediction errors at each level of the hierarchy in the style of predictive self-supervised learning. The network processes data in blocks of video frames rather than a frame-to-frame basis. This allows it to learn relationships among movement patterns, yielding state-of-the-art performance in long range video sequence predictions in benchmark datasets. We observed that hierarchical interaction in the network introduces sensitivity to memories of global movement patterns even in the population representation of the units in the earliest level. Finally, we provided neurophysiological evidence, showing that neurons in the early visual cortex of awake monkeys exhibit very similar sensitivity and behaviors. These findings suggest that predictive self-supervised learning might be an important principle for representational learning in the visual cortex.

Review:
Summary:
The paper presents a novel architecture for video prediction consisting of a feed-forward path with sparse convolutions and an LSTM generating predictions of chunks of video based on the sequence of input chunks. A feedback path links the LSTMs of the different sparse prediction modules. Experiments in video prediction are performed on moving-MNIST and the KTH action recognition dataset and the model achieves state-of-the-art performance on both. Interestingly, the model is exhibits prediction suppressions effects as have been observed during neurophysiological experiments in the inferotemporal cortex of macaque monkeys. The proposed method exhibits prediction suppression effects also in the lower layers, motivating a neurophysiological experiment in the earlier V1/V2 regions, which yielded an observation similar to the model’s prediction.
Strengths:
The performance improvements over competing methods on Moving-MNIST and KTH presented in the experimental section are significant. The analysis seems fairly thorough.
Weaknesses and requests for clarification:
- The description of the sparse predictive module is difficult to follow, and I am not sure I understood it completely. I find it a bit unintuitive to start the description with the errors, instead of explaining what is computed from beginning to end. The section reads more like a loose description of isolated parts instead of an integrated whole. Maybe walking the reader step-by-step through one complete iteration of the computation helps to clarify this. Also, not every character in equations 1-5 and the algorithm has been defined. For example, what is L? 
- The text makes it sound like the idea of using 3d convolutions in a convLSTM is novel. 3D convLSTMs have been previously used in 3d vision, see 
Choy, C. B., Xu, D., Gwak, J., Chen, K., & Savarese, S. (2016, October). 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European conference on computer vision (pp. 628-644). Springer, Cham.
The application of 3d convLSTMs to video might be new, but the mentioned paper by Choy et al. (2016) should be cited.
- You mention that padding is used for rows and columns. Are you using padding on the temporal axis as well?
- The paper seems to be written in a rush, as it contains way too many typos and grammar mistakes, e.g. “a hierarchical of” (should be “a hierarchy of“ or just “hierarchical”), “feedforwad”, “Expriment” (section 4 heading), “achievedbetter”, “trained monkeys to image pairs”, “pervious”, “perserves”, “processure”, “sequnence” “viusal”. Many typos could have been caught by a spellcheck! This would improve readability a lot!
- The citations are not properly formatted: (1) If the author names are used as part of the sentence, use e.g. Lotter et al. (2016), else (2) If the author names are not part of the sentence, use (Lotter et al., 2016). These two styles are mixed randomly in the current draft. This makes the manuscript, which already contains a lot of language mistakes, difficult to read.
- Abbreviations that are used but not introduced: CNN, IT, PSTH, DCNN, LSTM.
- The related work section could benefit from referring to some of the related work in neuroscience.
- Adding a sentence explaining the intuition behind using SatLU in equation (1) might be helpful
To summarize my feedback: I think experimental results and analysis are strong, but the presentation is strongly lacking! The description of the approach definitely needs to be improved to make replication of the results easier. It might help to have someone who doesn’t know the model already read the description and explain it back to you while revising the draft. I hope I could provide some helpful suggestions. I would recommend the manuscript for acceptance, if the presentation is significantly improved!