Paper:
EIDETIC 3D LSTM: A MODEL FOR VIDEO PREDICTION AND BEYOND
Spatiotemporal predictive learning, though long considered to be a promising selfsupervised feature learning method, seldom shows its effectiveness beyond future video prediction. The reason is that it is difficult to learn good representations for both short-term frame dependency and long-term high-level relations. We present a new model, Eidetic 3D LSTM (E3D-LSTM), that integrates 3D convolutions into RNNs. The encapsulated 3D-Conv makes local perceptrons of RNNs motion-aware and enables the memory cell to store better short-term features. For long-term relations, we make the present memory state interact with its historical records via a gate-controlled self-attention module. We describe this memory transition mechanism eidetic as it is able to effectively recall the stored memories across multiple time stamps even after long periods of disturbance. We first evaluate the E3D-LSTM network on widely-used future video prediction datasets and achieve the state-of-the-art performance. Then we show that the E3D-LSTM network also performs well on the early activity recognition to infer what is happening or what will happen after observing only limited frames of video. This task aligns well with video prediction in modeling action intentions and tendency.

Review:
# 1. Summary
This paper presents a model for future video prediction, which integrates 3D convolutions into RNNs. The internal operations of the RNN are modified by adding historical records controlled via a gate-controlled self-attention module. The authors show that the model is effective also for other tasks such as early activity recognition.
Strengths:
* Nice extensive experimentation on video prediction and early activity recognition tasks and comparison with recent papers
* Each choice in the model definition are motivated, although some clarity is still missing (see below)
Weaknesses:
* Novelty: the proposed model is a small extension of a previous work (Wang et al., 2017) 
# 2. Clarity and Motivation
In general, the paper is clear and general motivation makes sense, however some points need to be improved with further discussion and motivation:
A) Page 2 “Unlike the conventional memory transition function, it learns the size of temporal interactions. For longer sequences, this allows attending to distant states containing salient information”: This is not obvious. Can the authors add more details and motivate these two sentences? How is long-term relations are learned given Eq. 1? 
B) Page 5 “These two terms are respectively designed for short-term and long-term video modeling”: How do you make sure that Recall(.) does not focus on the short-term modeling instead? Not clear why this should model long-term relations.
C) Page 5 and Eq 1: motivation why layer norm is required when defining C_t^k is not clear
D) What if the Recall is instead modeled as attention? The idea is to consider only C_{1:t-1}^k (not consider R_t) and have an attentional model that learn what to recall based only on C. Also, why does Recall need to depend on R_t?
E) Page 5 “to minimize the l1 + l2 loss over every pixel in the frame”: this sentence is not clear. How does it relate to Eq. 2?
# 3. Novelty
Novelty is the major concern of this paper. Although the introduced new concepts and ideas are interesting, the work seems to be an extension of ST-LSTM and PredRNN where Eq 1 is slightly modified by introducing Recall. 
In addition the existing relation between the proposed model and ST-LSTM is not clearly state. Page 2, first paragraph: here the authors should state that model is and extension of ST-LSTM and highlight what are the difference and advantage of the new model.
# 4. Significance of the work
This paper deals with an interesting and challenging topic (video prediction) as well as it shows some results on the early activity recognition task. These are definitively nice problem which are far to be solved. From the application perspective this work is significant, however from the methodological perspective it lacks a bit of significance because of the novelty issues highlighted above.
# 5. Experimentation
The experiments are robust with nice comparisons with recent methods and ablation study motivating the different components of the model (Table 1 and 2). Some suggested improvements:
A) Page 7 “Seq 1 and Seq 2 are completely irrelevant, and ahead of them, another sub-sequence called prior context is given as the input, which is exactly the same as Seq 2”: The COPY task is a bit unclear and need to be better explained. Why are Seq. 1 and 2 irrelevant? I would suggest to rephrase this part.
B) Sec. 4.2, “Dataset and setup”: which architecture has been used here?
C) Sec. 4.3, “Hyper-parameters and Baselines“: the something-something dataset is more realising that the other two “toy” dataset. Why did the authors choose to train a 2 layers 3D-CNN encoders, instead of using existing pretrained 3D CNNs? I would suspect that the results can improve quite a bit.
# 6. Others
* The term “self-supervised auxiliary learning” is introduced in the abstract, but at this point it’s meaning is not clear. I’d suggest to either remove it or explain its meaning.
* Figure 1(a): inconsistent notation with 2b. Also add citation (Wang et al., 2017) since it ie the same model of that paper
-------
# Post-discussion
I increased my rating: even if novelty is not high, the results support the incremental ideas proposed by the authors.