Original: E IDETIC 3D LSTM: A M ODEL FOR V IDEO P REDICTION AND B EYOND
Revised: E IDETIC 3D LSTM: A M ODEL FOR V IDEO P REDICTION AND B EYOND

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Spatiotemporal predictive learning, though long considered to be a promising selfsupervised feature learning method, seldom shows its effectiveness beyond future video prediction.
Revised: Spatiotemporal predictive learning, though long considered to be a promising selfsupervised feature learning method, seldom shows its effectiveness beyond future video prediction.

Original: The reason is that it is difﬁcult to learn good representations for both short-term frame dependency and long-term high-level relations.
Revised: The reason is that it is difﬁcult to learn good representations for both short-term frame dependency and long-term high-level relations.

Original: We present a new model, Eidetic 3D LSTM (E3D-LSTM), that integrates 3D convolutions into RNNs.
Revised: We present a new model, Eidetic 3D LSTM (E3D-LSTM), that integrates 3D convolutions into RNNs.

Original: The encapsulated 3D-Conv makes local perceptrons of RNNs motion aware and enables the memory cell to store better short-term features.
Revised: The encapsulated 3D-Conv makes local perceptrons of RNNs motion-aware and enables the memory cell to store better short-term features.

Original: For long-term relations, we make the present memory state interact with its historical records via a gate-controlled self-attention module.
Revised: For long-term relations, we make the present memory state interact with its historical records via a gate-controlled self-attention module.

Original: We describe this memory transition mechanism eidetic as it is able to effectively recall the stored memories across multiple timestamps even after long periods of disturbance.
Revised: We describe this memory transition mechanism eidetic as it is able to effectively recall the stored memories across multiple time stamps even after long periods of disturbance.

Original: We ﬁrst evaluate thespatiotemporal modeling capability of the E3D-LSTM model ontwo widely-used future video prediction datasets and achieve the state of the art performance.
Revised: We ﬁrst evaluate the E3D-LSTM network on widely-used future video prediction datasets and achieve the state-of-the-art performance.

Original: Then we demonstrate thatwith a self-supervised auxiliary learning strategy, the E3D-LSTM network  performs well on  early activity recognition to infer what is happening  after observing only limited frames of  video 
Revised: Then we show that the E3D-LSTM network also performs well on the early activity recognition to infer what is happening or what will happen after observing only limited frames of video. This task aligns well with video prediction, as action intentions and tendency are important to achieve the state-of-the-art performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A fundamental problem in spatiotemporal predictive learning is how to effectively learn good representations for video inference or reasoning.
Revised: A fundamental problem in spatiotemporal predictive learning is how to effectively learn good representations for video inference or reasoning.

Original: Currently, recurrent neural networks (RNNs) remain to be the most promising models in this ﬁeld, and have achieved state-of-the-art results on a number of future video prediction benchmarks (Wang et al., 2018b; Oliu et al., 2018).
Revised: Currently, recurrent neural networks (RNNs) remain to be the most promising models in this ﬁeld, and have achieved state-of-the-art results on a number of future video prediction benchmarks (Wang et al., 2018b; Oliu et al., 2018).

Original: However, beyond frames prediction, RNN based models are less effective in learning high-level video representations or capturing long-term relations.
Revised: However, beyond frames prediction, RNN based models are less effective in learning high-level video representations or capturing long-term relations.

Original: On the other hand, recent studies demonstrate that 3D Convolutional
Revised: On the other hand, recent studies demonstrate that 3D Convolutional

Original: Neural networks (3D-CNNs) surpass RNNs in learning better representations for action classiﬁcation (Carreira & Zisserman, 2017; Tran et al., 2015).
Revised: Neural networks (3D-CNNs) surpass RNNs in learning better representations for action classiﬁcation (Carreira & Zisserman, 2017; Tran et al., 2015).

Original: For instance, variants of 3D-CNNs, such as
Revised: For instance, variants of 3D-CNNs, such as

Original: Inﬂated 3D-CNNs, have signiﬁcantly increased action classiﬁcation accuracy over the UCF 101 and
Revised: Inﬂated 3D-CNNs, have signiﬁcantly increased action classiﬁcation accuracy over the UCF 101 and

Original: Kinetics datasets.
Revised: Kinetics datasets.

Original: These 3D-CNN architectures have no recurrent structures but instead employ 3D convolution (3D-Conv) and 3D pooling operations to preserve temporal information of the input sequences which would be otherwise discarded in classical 2D convolution operations.
Revised: These 3D-CNN architectures have no recurrent structures but instead employ 3D convolution (3D-Conv) and 3D pooling operations to preserve temporal information of the input sequences which would be otherwise discarded in classical 2D convolution operations.

Original: Motivated by the recent success of 3D-CNNs, in this paper we propose a new model for spatiotemporal predictive learning based on both recurrent modeling (for temporal dependency) and feedforward 3D-Conv modeling (for local dynamics).
Revised: Motivated by the recent success of 3D-CNNs, in this paper we propose a new model for spatiotemporal predictive learning based on both recurrent modeling (for temporal dependency) and feedforward 3D-Conv modeling (for local dynamics).

Original: A plausible approach, of course, is to simply stack 3D-Convs and each RNN unit in a feed-forward way using 3D-Convs for either perceiving ﬁne-grained features from raw videos or combining high-level representations.
Revised: A plausible approach, of course, is to simply stack 3D-Convs and each RNN unit in a feed-forward way using 3D-Convs for either perceiving ﬁne-grained features from raw videos or combining high-level representations.

Original: However, as shown in our experiments, these straightforward extensions may not outperform the baseline RNN model.
Revised: However, as shown in our experiments, these straightforward extensions may not outperform the baseline RNN model.

Original: We attribute these ﬁndings to that RNNs and 3D-CNNs represent two very different mechanisms for the same purpose of spatiotemporal modeling, and connecting them directly fails to exploit their complementary advantages.
Revised: We attribute these ﬁndings to that RNNs and 3D-CNNs represent two very different mechanisms for the same purpose of spatiotemporal modeling, and connecting them directly fails to exploit their complementary advantages.

Original: Therefore, it remains challenging and requires principled approaches to design an effective spatiotemporal network.
Revised: Therefore, it remains challenging and requires principled approaches to design an effective spatiotemporal network.

Original: To this end, we propose a new model called Eidetic 3D LSTM (E3D-LSTM) for spatiotemporal predictive learning.
Revised: To this end, we propose a new model called Eidetic 3D LSTM (E3D-LSTM) for spatiotemporal predictive learning.

Original: We introduce an eidetic 3D memory to: a) memorize local appearance and motion in a short spatiotemporal volume, and b) recall the long-range historical context by learning to attend to previous memory states.
Revised: We introduce an eidetic 3D memory to: a) memorize local appearance and motion in a short spatiotemporal volume, and b) recall the long-range historical context by learning to attend to previous memory states.

Original: Regarding the short-term dependency , in many cases, spatiotemporal modeling mainly depends on temporally nearby appearances and on-going short-term motions.
Revised: Regarding the short-term dependency , in many cases, spa- tiotemporal predictive modeling mainly depends on temporally nearby appearances and on-going short-term motions.

Original: All the information is encapsulated into the eidetic 3D memory cell with a short time convolution window, and used in recurrent transitions.
Revised: All the information is encapsulated into the eidetic 3D memory cell with a short time convolution window, and used in recurrent transitions.

Original: Our experimental results show that integrating 3D-Conv deep into RNNs is effective for modeling local representations in a consecutive manner.
Revised: Our experimental results show that integrating 3D-Conv deep into RNNs is effective for modeling local representations in a consecutive manner.

Original: On the other hand, for long-term interactions , which is important for predicting non-stationary or periodical videos as well as learning high-level video representations, we exploit a self-attention mechanism controlled by revised recurrent gates to recall temporally distant memory.
Revised: On the other hand, for long-term interactions , which is important for predicting non-stationary or periodical videos as well as learning high-level video representations, we exploit a self-attention mechanism controlled by revised recurrent gates to recall temporally distant memory.

Original: The current memory state of E3DLSTM is learned to attend to all previous relevant moments.
Revised: The current memory state of E3D-LSTM is learned to attend to all previous relevant moments.

Original: Our experimental results verify that this attention mechanism is beneﬁcial for long-term memorization.
Revised: Our experimental results verify that this attention mechanism is beneﬁcial for long-term memorization.

Original: We describe this memory transition mechanism eidetic as it is able to effectively recall the stored memories across multiple timestamps even after long periods of disturbance.
Revised: We describe this memory transition mechanism eidetic as it is able to effectively recall the stored memories across multiple time stamps even after long periods of disturbance.

Original: To the best of our knowledge, the proposed E3D-LSTM model is among the ﬁrst approaches that leverage 3D-Conv in RNNs.
Revised: To the best of our knowledge, the proposed E3D-LSTM model is among the ﬁrst approaches that leverage 3D-Conv in RNNs.

Original: We empirically validate it on two spatiotemporal predictive tasks and an early activity recognition task over three benchmarks: a) on future video prediction, it achieves the best-published accuracy on two classical benchmarks; b) on early activity recognition, it outperforms the state-of-the-art action recognition methods.
Revised: We empirically validate it on standard spatiotemporal predictive tasks and an early activity recognition task over three benchmarks: a) on future video prediction, it achieves the best-published accuracy on two classical benchmarks; b) on early activity recognition, it outperforms the state-of-the-art action recognition methods.

Original: In addition, we show that self-supervised learning can further improve the performance of early activity recognition.
Revised: In addition, we show that selfsupervised learning can further improve the performance of early activity recognition.

Original: We present ablation studies to verify the effectiveness of all modules in the proposed E3D-LSTM model.
Revised: We present ablation studies to verify the effectiveness of all modules in the proposed E3D-LSTM model.

Original:  R ELATED W ORK AND P ROBLEM C ONTEXT
Revised:  R ELATED W ORK AND P ROBLEM C ONTEXT

Original: Spatiotemporal Predictive Learning Models.
Revised: Spatiotemporal Predictive Learning Models.

Original: In recent years, RNNs have been extensively used in sequence prediction and future frame prediction.
Revised: In recent years, RNNs have been extensively used in sequence prediction and future frame prediction.

Original: Srivastava et al.
Revised: Srivastava et al.

Original: (2015) adopted the LSTM-based sequence to sequence model (Sutskever et al., 2014) from language modeling for learning video representations.
Revised: (2015) adopted the LSTM-based sequence to sequence model (Sutskever et al., 2014) from language modeling for learning video representations.

Original: Shi et al.
Revised: Shi et al.

Original: (2015) proposed the convolutional LSTM by integrating convolutions into recurrent state transitions for sequence prediction.
Revised: (2015) proposed the convolutional LSTM by integrating convolutions into recurrent state transitions for sequence prediction.

Original: The convolutional LSTM model is extended by
Revised: The convolutional LSTM model is extended by

Original: Finn et al.
Revised: Finn et al.

Original: (2016) to predict future states of robotic environments.
Revised: (2016) to predict future states of robotic environments.

Original: Villegas et al.
Revised: Villegas et al.

Original: (2017) leveraged optical ﬂow to help capture short-term video dynamics for video sequence prediction.
Revised: (2017) leveraged optical ﬂow to help capture short-term video dynamics for video sequence prediction.

Original: Xu et al.
Revised: Xu et al.

Original: (2018) proposed a two-stream RNN that deals with structural video content in separate streams.
Revised: (2018) proposed a two-stream RNN that deals with structural video content in separate streams.

Original: In (Kalchbrenner et al., 2017), a sophisticated model that extends recurrent structures to estimate local dependencies between adjacent pixels.
Revised: In (Kalchbrenner et al., 2017), a sophisticated model that extends recurrent structures to estimate local dependencies between adjacent pixels.

Original: While this video pixel network model is able to describe image sequences, the computational load is prohibitively high.
Revised: While this video pixel network model is able to describe image sequences, the computational load is prohibitively high.

Original: The above-mentioned recurrent models predict future frame mainly based on sequentially updated memory states.
Revised: The above-mentioned recurrent models predict future frame mainly based on sequentially updated memory states.

Original: When the memory cell is refreshed, older memories will be discarded immediately.
Revised: When the memory cell is refreshed, older memories will be discarded immediately.

Original: In contrast, the proposed E3D-LSTM model maintains a list of historical memory records and revokes them when necessary, thereby facilitating long-range video reasoning.
Revised: In contrast, the proposed E3D-LSTM model maintains a list of historical memory records and revokes them when necessary, thereby facilitating long-range video reasoning.

Original: While in spirit this idea is similar to the self-attention module in feed-forward networks (Vaswani et al., 2017; Wang et al., 2018a), we exploit it to correlate long-term and short-term video representations in this work.
Revised: While in spirit this idea is similar to the self-attention module in feed-forward networks (Vaswani et al., 2017; Wang et al., 2018a), we exploit it to correlate long-term and short-term video representations in this work.

Original: Another signiﬁcant difference between the above-mentioned prior work and the proposed model is that we use 3D-Convs as basic operations inside the E3D-LSTM instead of fully-connected or 2D convolution operations.
Revised: Another signiﬁcant difference between the above-mentioned prior work and the proposed model is that we use 3D-Convs as basic operations inside the E3D-LSTM instead of fully-connected or 2D convolution operations.

Original: We show using 3D-Convs to model recurrent state-to-state transitions can signiﬁcantly improve prediction performance.
Revised: We show using 3D-Convs to model recurrent state-to-state transitions can signiﬁcantly improve prediction performance.

Original: This idea is motivated by recent advances in video classiﬁcation (a high-level representation learning task)
Revised: This idea is motivated by recent advances in video classiﬁcation (a high-level representation learning task)

Original: (Ji et al., 2013; Tran et al., 2015; Carreira & Zisserman, 2017).
Revised: (Ji et al., 2013; Tran et al., 2015; Carreira & Zisserman, 2017).

Original: We note that Vondrick et al.
Revised: We note that Vondrick et al.

Original: (2016) and Tulyakov et al.
Revised: (2016) and Tulyakov et al.

Original: (2018) also introduced 3D-CNNs for spatiotemporal predictive learning.
Revised: (2018) also introduced 3D-CNNs for spatiotemporal predictive learning.

Original: However, these networks are feed-forward and do not capture temporal consistency effectively.
Revised: However, these networks are feed-forward and do not capture temporal consistency effectively.

Original: To minimize uncertainties for video prediction, numerous methods have applied adversarial training or variational auto-encoder for spatiotemporal predictive models (Mathieu et al., 2016; Vondrick et al., 2016; Denton & Fergus, 2018; Bhattacharjee &Das, 2017; Tulyakov et al., 2018;
Revised: Future prediction errors of an imperfect model can be categorized by two factors: a) the “systematic errors” caused by a lack of modeling ability to the deterministic variations; b) the stochastic, inherent uncertainty of the future. We aim to minimize the ﬁrst factor in this work. For the second factor, numerous methods have applied adversarial training or variational auto-encoders to video prediction, for example (Mathieu et al., 2016; Vondrick et al., 2016; Denton & Fergus, 2018; Bhattacharjee &

Original:  Lu et al., 2017; Wichers et al., 2018).
Revised: Das, 2017; Tulyakov et al., 2018; Lu et al., 2017; Wichers et al., 2018).

Original: In this work, we propose a new network to account for spatiotemporal sequences in which prediction uncertainty is minimized by maximizing the capacity of learning the relationship between past and future data.

Original: Convolutional Recurrent Networks.
Revised: Convolutional Recurrent Networks.

Original: Our model is closely related to convolutional recurrent networks.
Revised: Our model is closely related to convolutional recurrent networks.

Original: In the ConvLSTM network (Shi et al., 2015), all state transitions are implemented with 2D convolutions.
Revised: In the ConvLSTM network (Shi et al., 2015), all state transitions are implemented with 2D convolutions.

Original: As such, the transition function is no longer permutation invariant and able to better perceive relations in a spatiotemporal neighborhood.
Revised: As such, the transition function is no longer permutation invariant and able to better perceive relations in a spatiotemporal neighborhood.

Original: The spatiotemporal LSTM (ST-LSTM) 
Revised: The spatiotemporal LSTM (ST-LSTM) is char-

Original: (Wang et al., 2017), is characterized by delivering two memory states separately: memory M in a zigzag

Original: RNN Unit RNN Unit
Revised: RNN Unit RNN Unit

Original: RNN Unit
Revised: RNN Unit

Original: Frame 3:T+2 Frame 1:T Frame 2:T+
Revised: Frame 3:T+2 Frame 1:T Frame 2:T+

Original: Frame T+
Revised: Frame T+

Original: Frame T+
Revised: Frame T+

Original: Frame T+3 2D CNN
Revised: Frame T+3 2D CNN

Original: Decoders 2D CNN Decoders 2D CNN Decoders
Revised: Decoders 2D CNN Decoders 2D CNN Decoders

Original: Classifier 3D CNN Encoders 3D CNN
Revised: Classifier 3D CNN Encoders 3D CNN

Original: Encoders 3D CNN Encoders (a) 3D-CNN at Bottom
Revised: Encoders 3D CNN Encoders (a) 3D-CNN at Bottom

Original: RNN Unit 2D CNN Encoders
Revised: RNN Unit 2D CNN Encoders

Original: RNN Unit
Revised: RNN Unit

Original: RNN Unit 2D CNN Encoders 2D CNN Encoders
Revised: RNN Unit 2D CNN Encoders 2D CNN Encoders

Original: Frame T+
Revised: Frame T+

Original: Frame T+2 Frame T+
Revised: Frame T+2 Frame T+

Original: Frame T+
Revised: Frame T+

Original: Frame T+2 Frame T
Revised: Frame T+2 Frame T

Original: Classifier 3D CNN Decoders 3D CNN
Revised: Classifier 3D CNN Decoders 3D CNN

Original: Decoders 3D CNN
Revised: Decoders 3D CNN

Original: Decoders (b) 3D-CNN on Top
Revised: Decoders (b) 3D-CNN on Top

Original: Frame 1:T
Revised: Frame 1:T

Original: Frame T+ 3D CNN Decoders 3D CNN
Revised: Frame T+ 3D CNN Decoders 3D CNN

Original: Decoders 3D CNN Decoders
Revised: Decoders 3D CNN Decoders

Original: E3D-LSTM
Revised: E3D-LSTM

Original: E3D-LSTM
Revised: E3D-LSTM

Original: E3D-LSTM 3D CNN Encoders 3D CNN
Revised: E3D-LSTM 3D CNN Encoders 3D CNN

Original: Encoders 3D CNN Encoders
Revised: Encoders 3D CNN Encoders

Original: Classifier
Revised: Classifier

Original: Frame τ :T+ τ
Revised: Frame τ :T+ τ

Original: Frame 2 τ :T+

Revised: Frame 2 τ :T+

Original: τ
Revised: τ

Original: Frame T+2 τ + 1 Frame T+ τ +(c) E3D-LSTM Network  direction and memory C being passed horizontally (see Appendix A for details).
Revised: Frame T+2 τ + 1 Frame T+ τ +(c) E3D-LSTM Network acterized by delivering two memory states separately (Wang et al., 2017): memory M in a zigzag direction and memory C being passed horizontally (see Appendix A for details).

Original: In this model,
Revised: In this model,

Original: M provides greater capability to model short-term motions, and C is adopted from fully-connected
Revised: M provides greater capability to model short-term motions, and C is adopted from fully-connected

Original: LSTMs (Hochreiter & Schmidhuber, 1997) to ease the vanishing gradient problem.
Revised: LSTMs (Hochreiter & Schmidhuber, 1997) to ease the vanishing gradient problem.

Original: Although the
Revised: Although the

Original: ST-LSTM performs well on video prediction benchmarks, it does not capture long-term video relations effectively.
Revised: ST-LSTM performs well on video prediction benchmarks, it does not capture long-term video relations effectively.

Original: The forget gates of memory C tend to respond strongly to short-term features, thereby easily falling into a saturated zone (with values between 0 and 0 . 1 ) and interrupting longrange information ﬂows.
Revised: The forget gates of memory C tend to respond strongly to short-term features, thereby easily falling into a saturated zone (with values between 0 and 0 . 1 ) and interrupting longrange information ﬂows.

Original: We adopt the zigzag updating route of memory M from the ST-LSTM, while improving the forgetting mechanism in updating the temporal memory C .
Revised: We adopt the zigzag updating route of memory M from the ST-LSTM, while improving the forgetting mechanism in updating the temporal memory C .

Original: We also increase the dimensions of memory states and take 3D-Convs as the basic operators for state transitions.
Revised: We also increase the dimensions of memory states and take 3D-Convs as the basic operators for state transitions.

Original:  E IDETIC 3D LSTM
Revised:  E IDETIC 3D LSTM

Original: This section ﬁrst presents the Eidetic 3D LSTM for perceiving and memorizing both short-term and long-term representations in videos.
Revised: This section ﬁrst presents the Eidetic 3D LSTM for perceiving and memorizing both short-term and long-term representations in videos.

Original: We then discuss a scheduled multi-task learning strategy that uses predictive learning as an auxiliary self-supervised task for activity recognition.
Revised: We then discuss a scheduled multi-task learning strategy that uses predictive learning as an auxiliary self-supervised task for activity recognition.

Original: 3. 3D C ONVOLUTIONS IN R ECURRENT N ETWORK
Revised: 3. 3D C ONVOLUTIONS IN R ECURRENT N ETWORK

Original: An ideal predictive model relies on effective learning of video representations.
Revised: An ideal predictive model relies on effective learning of video representations.

Original: RNNs and 3D-CNNs are network architectures of different mechanisms for modeling spatiotemporal data.
Revised: RNNs and 3D-CNNs are network architectures of different mechanisms for modeling spatiotemporal data.

Original: In this work, we aim to leverage the strength of each one in a uniﬁed architecture and start the discussion with two plausible extensions of stacking 3D-Convs and RNN units.
Revised: In this work, we aim to leverage the strength of each one in a uniﬁed architecture and start the discussion with two plausible extensions of stacking 3D-Convs and RNN units.

Original: Figure 1(a) and 1(b) illustrate two hybrid baseline networks which add 3D-CNNs before or after stacked spatiotemporal LSTMs.
Revised: Figure 1(a) and 1(b) illustrate two hybrid baseline networks which add 3D-CNNs before or after stacked spatiotemporal LSTMs.

Original: However, we ﬁnd that integrating the 3D-Convs outside the LSTM unit performs noticeably worse than the baseline RNN model.
Revised: However, we ﬁnd that integrating the 3D-Convs outside the LSTM unit performs noticeably worse than the baseline RNN model.

Original: To this end, we propose a “deeper” integration of 3D-Convs inside the LSTM unit in order to incorporate the convolutional features into the recurrent state transition over time.
Revised: To this end, we propose a “deeper” integration of 3D-Convs inside the LSTM unit in order to incorporate the convolutional features into the recurrent state transition over time.

Original: Figure 1(c) shows the overall encoder-decoder architecture.
Revised: Figure 1(c) shows the overall encoder-decoder architecture.

Original: In this model, a consecutive of T input frames are ﬁrst encoded by a few layers of 3D-Convs to obtain high-dimension feature maps.
Revised: In this model, a consecutive of T input frames are ﬁrst encoded by a few layers of 3D-Convs to obtain high-dimension feature maps.

Original: The 3D-Conv feature maps are directly fed into a novel E3D-LSTM to model the longterm spatiotemporal interaction.
Revised: The 3D-Conv feature maps are directly fed into a novel E3D-LSTM to model the longterm spatiotemporal interaction.

Original: Finally, the E3D-LSTM hidden states are decoded by a number of stacked 3D-Conv layers to get the predicted video frames.
Revised: Finally, the E3D-LSTM hidden states are decoded by a number of stacked 3D-Conv layers to get the predicted video frames.

Original: For classiﬁcation tasks, the hidden states can be directly used as the learned video representation.
Revised: For classiﬁcation tasks, the hidden states can be directly used as the learned video representation.

Original: 3. E IDETIC M EMORY T RANSITION
Revised: 3. E IDETIC M EMORY T RANSITION

Original: The architecture of the proposed Eidetic 3D LSTM is illustrated in Figure 2, where 
Revised: The architecture of the proposed Eidetic 3D LSTM is illustrated in Figure 2, where the red arrow indicates short-term information ﬂow and the blue arrow denotes long-term information ﬂow.

Revised: There are 4 inputs: X t , the 3D-Conv feature maps from encoders or hidden states from the previous E3DLSTM layer; H kt − 1 , the hidden states from previous time stamp; C kt − 1 , the memory states from previous time stamp; and M k − 1 t , the previous spatiotemporal memory states described earlier.

Original: We use recurrent 3D-Convs as motion-aware perceptrons to extract short-term appearance and local motions in continuous space-time ﬁelds and store them in a small spatiotemporal volume.
Revised: We use recurrent 3D-Convs as motion-aware perceptrons to extract short-term appearance and local motions in continuous space-time ﬁelds and store them in a small spatiotemporal volume.

Original: As such, video appearance and short-term motions can be encoded in R T × H × W × C tensors, in whicheach dimension indicates temporal depth, spatial size, and the number of feature map channels, re-
Revised: As such, video appearance and short-term motions can be encoded in R T × H × W × C tensors, in which

Original: Forget Gate
Revised: Forget Gate

Original: h t-1k c t-1k x t m tkm tk c tk h tk g t i t
Revised: h t-1k c t-1k x t m tkm tk c tk h tk g t i t

Original: f t o t ′ i t ′ g t ′ f t (a) Spatiotemporal LSTM
Revised: f t o t ′ i t ′ g t ′ f t (a) Spatiotemporal LSTM

Original: Recall Gate
Revised: Recall Gate

Original: Layer Norm

Original: Softmax
Revised: Softmax

Original: C t-1k

Original: C 1:t-1k

Original: C tk
Revised: C tk

Original: H t-1k
Revised: H t-1k

Original: H tk
Revised: H tk

Original: X t
Revised: X t

Original: M tkM tk- ′ F t′ G t
Revised: M tk ′ F t

Original: G t
Revised: G t

Original: O t ′ I t
Revised: O t ′ I t

Original: I t
Revised: I t

Original: R t F t ( [Figure]
Revised: R t C t-1k M tk- {

Original: spectively. By inﬂating the memory state along the time dimension, we found that the proposed
Revised: LayerNorm

Original: E3D-LSTM becomes more capable of characterizing and memorizing local or short-term motions.
Revised: C t-2k C t − τ k C t − τ + 1 k C t − τ :t-1 k ′

Original: For long-term frame interaction, we improve the recurrent transition functions for updating the memory state over time. The key equations for layer k at timestamp t are presented as follows. There are four inputs: X t , the 3D-Conv feature maps from encoders or hidden states from the previous E3D-LSTM layer; M k − 1 t, the previous spatiotemporal memory; H kt − 1 and C kt − 1 , the hidden states and memory states from previous timestamp, described by
Revised: G t F t

Original: [Equation]
Revised: [Figure]

Original: R t = σ ( W xr ∗ X t + W hr ∗ H kt − 1 + b r ) I (cid:48) t = σ ( W (cid:48)
Revised: each dimension indicates temporal depth, spatial size, and the number of feature map channels, respectively.

Original: xi ∗ X t + W mi ∗ M k − 1 t + b (cid:48) i ) G (cid:48) t = tanh( W (cid:48) xg ∗ X t + W mg ∗ M k − 1 t + b (cid:48) g )
Revised: By inﬂating the memory state along the time dimension, we found that the proposed

Original: F (cid:48) t = σ ( W (cid:48) xf ∗ X t + W mf ∗ M k − 1 t + b (cid:48) f )
Revised: E3D-LSTM becomes more capable of characterizing and memorizing local or short-term motions.

Original: Recall ( R t , C k 1: t − 1 ) = softmax( R t · C k 1: t − 1 T ) · C k 1: t − C kt = I t (cid:12) G t + LayerNorm ( F t (cid:12) C kt − 1 + Recall ( R t , C k 1: t − 1 ))
Revised: To capture the long-term frame interactions, we improve the recurrent transition function of the memory states by proposing a new memory RECALL mechanism:

Original: M kt = I (cid:48) t (cid:12) G (cid:48) t + F (cid:48) t (cid:12) M k − 1 t O t = σ ( W xo ∗ X t + W ho ∗ H kt − 1 + W co ∗ C
Revised: [Equation]

Original: kt + W mo ∗ M kt + b o ) H kt = O t (cid:12) tanh( W 1 × 1 ∗ [ C kt , M kt ]) , (1) where σ is the sigmoid function, ∗ is the 3D-Conv operation, (cid:12) is the Hadamard product, and · is the matrix product after reshaping R t and C k 1: t − 1 from R T × H × W × C to R THW × C matrices.
Revised: where σ is the sigmoid function, ∗ is the 3D-Conv operation, (cid:12) is the Hadamard product, and · is the matrix product after reshaping R t / C kt − τ : t − 1 into R THW × C / R τTHW × C matrices.

Original: The blue part of Equation 1 highlights the memory scheduling and updating mechanism.

Original: Unlike the conventional memory transition function, it learns the size of temporal interactions. For longer sequences, this allows attending to distant states containing salient information.
Revised: T is the temporal length of a gate tensor or a single memory state, and τ is the number of memory states that are concatenated along the temporal dimension.

Original: Three terms are involved in computing C kt .
Revised: Three terms are involved in computing C kt .

Original: The ﬁrst one I t (cid:12) G t encodes local video appearance and motions.
Revised: The ﬁrst one I t (cid:12)G t encodes local video appearance and motions.

Original: The second one  captures short-term changes between adjacent timestamps, where the forget gate decides which part of the previous memory can be used and to what extent.
Revised: The second one C kt − 1 can be viewed as a residual connection from the previous memory state controlled by the forget gate F t , which captures short-term changes between adjacent time stamps.

Original: In this process, the accessible memory ﬁeld is limited and ﬁxed.
Revised: In this process, the accessible memory ﬁeld is ﬁxed and limited.

Original: Assuming all recurrent memory states C k 1: t − 1 as a gradually expanded storage with a continuous address space, the forget gate only seeks the nearest partition with an extreme ﬁrst-in-ﬁrst-out method.

Original: Therefore, we introduce the third term of memory transition function, modeling long-term video relations according to local motion and appearance (as encoded in X t and H kt − 1 ).
Revised: Therefore, we introduce the third term of memory transition function, modeling long-term video relations according to local motion and appearance (as encoded in X t and H kt − 1 ).

Original: The Recall ( R t , C k 1: t − 1 ) function is implemented as an attentive module with a normalized Gaussian function to compute the relationship between the encoded local patterns and the whole memory space.
Revised: The RECALL function is implemented as an attentive module with a normalized Gaussian function to compute the relationship between the encoded local patterns and the whole memory space.

Original: A set of parameterized gates R t , acting as memory access instructions, control where and what to attend in historical memory records.
Revised: A set of parameterized gates R t , acting as memory access instructions, control where and what to attend in historical memory records.

Original: These two terms are respectively designed for short-term and long-term video modeling.
Revised: These two terms are respectively designed for short-term and long-term video modeling.

Original: We integrate them in a uniﬁed network by applying layer normalization (Ba et al., 2016) to their element-wise sum.
Revised: We integrate them in a uniﬁed network by applying layer normalization (Ba et al., 2016) to their element-wise sum, in order to mitigate the covariant shift and stabilize the training process, as it has been commonly used in RNNs.

Original: The memory tensor C kt is named eidetic 3D memory and the entire unit is called E3D-LSTM.
Revised: The hyper-parameter τ in C kt − τ : t − 1 decides how many historical memory states are attended by the recall gate R t . To involve more long-term relations, in most experiments, we take C k 1: t − 1 as the inputs of the RECALL function and do not ﬁx τ . Whereas in particular, we enable online recognition by setting τ to 5 .

Revised: Also, we found that removing the forget gate F t reduce the number of network parameters and make our model ﬁt better with the datasets.

Revised: Unlike the conventional memory transition function, the RECALL function learns the size of temporal interactions.

Revised: For longer sequences, this allows attending to distant states containing salient information.

Original: Our work is partially motivated by self-attention mechanisms (Lin et al., 2017; Vaswani et al., 2017).
Revised: Our work is partially motivated by self-attention mechanisms (Lin et al., 2017; Vaswani et al., 2017).

Original: However,  the attention  is not applied over the output states but during the memory state transition.
Revised: However, in our model, the attention mechanism is not applied over the output states but during the memory transitions.

Original: In our model, the attention mechanism is used to evoke past memories from distant timestamps for memorizing and distilling useful information from what has been perceived.
Revised: It is used to evoke past memories from distant time stamps for memorizing and distilling useful information from what has been perceived.

Original: We show that learning attention over previous memory states is beneﬁcial in recalling the long-range historical context.
Revised: We show that learning attention over previous memory states is beneﬁcial in recalling the long-range historical context.

Revised: The memory tensor C kt is named eidetic 3D memory and the entire unit is called E3D-LSTM.

Revised: We also exploit the same RECALL method to correlate M 1: k t along the vertical memory transition ﬂow, but it turns out to be less helpful.

Revised: With the updated memory state C kt , the ﬁnal output hidden states are calculated as follows:

Revised: [Equation]

Revised: where W 1 × 1 × 1 is the 1 × 1 × 1 convolutions for the transformation of the channel number.

Original: 3. S ELF - SUPERVISED A UXILIARY L EARNING
Revised: 3. S ELF - SUPERVISED A UXILIARY L EARNING

Original: In many  video datasets, there are often not enough supervisions or annotations over time for training a satisfactory RNN.
Revised: For many supervised tasks such as video action recognition, there are often not enough supervisions or annotations over time for training a satisfactory RNN.

Original: As an auxiliary measure to this problem, future video prediction is considered as a promising representation learning approach that is more densely supervised over time and might extract useful featuresin the latent space to assist video-level understanding.
Revised: As an auxiliary measure to this problem, future video prediction is considered as a promising representation learning approach that is more densely supervised over time and might extract useful features to assist video understanding.

Original: We consider two tasks:  pixel-level future frames prediction and a high-level classiﬁcation task (early activity recognition in our case).
Revised: We consider two tasks: the pixel-level future frames prediction and another video-level classiﬁcation task (early activity recognition in our case).

Original: As usual, we make  these two models share the same network backbone in the end-to-end training using a multi-task learning objective calculated from:
Revised: As usual, we make the models for these two tasks share the same network backbone in the end-to-end training using a multi-task learning objective:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where (cid:98) X and X are respectively predicted and ground truth future frames.
Revised: where (cid:98) X and X are respectively predicted and ground truth future frames.

Revised: (cid:107) · (cid:107) F is the Frobenius norm.

Original: (cid:98) Y and Y are high-level predictions and corresponding ground truth classes.
Revised: (cid:98) Y and Y are high-level predictions and corresponding ground truth classes.

Original: L ce is the cross-entropy loss for classiﬁcation, and λ is the weight factor.
Revised: L ce is the crossentropy loss for classiﬁcation, and λ is the weight factor.

Original: Although improving both tasks requires proper long short-term contextual representations, there is no guarantee that features learned with pixel-level supervisions will fully align with any highlevel objectives.
Revised: Although improving both tasks requires proper long short-term contextual representations, there is no guarantee that features learned with pixel-level supervisions will fully align with any highlevel objectives.

Original: We thus introduce a scheduled learning strategy where the objective function is gradually inclined from one task to the other in a curriculum learning manner (Bengio et al., 2009).
Revised: We thus introduce a scheduled learning strategy where the objective function is gradually inclined from one task to the other in a curriculum learning manner (Bengio et al., 2009).

Original: Speciﬁcally, we apply a linear decay to λ over the number of iterations i :
Revised: Speciﬁcally, we apply a linear decay to λ over the number of iterations i :

Original: [Equation]
Revised: [Equation]

Original: where λ (0) and η are respectively maximum and minimum values of λ ( i ) , (cid:15) controls the decreasing speed of the role of the auxiliary task.
Revised: where λ (0) and η are respectively maximum and minimum values of λ ( i ) , (cid:15) controls the decreasing speed of the role of the auxiliary task.

Original: We call this approach the Self-supervised Auxiliary Learning .
Revised: We call this approach the Self-supervised Auxiliary Learning .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the proposed E3D-LSTM model on two tasks: future video prediction and early activity recognition.
Revised: We evaluate the proposed E3D-LSTM model on two tasks: future video prediction and early activity recognition.

Original: These two tasks are of great importance with numerous applications that require effective spatiotemporal  models.
Revised: These two tasks are of great importance with numerous applications that require effective spatiotemporal predictive models.

Original: We demonstrate that the E3D-LSTM model performs favorably against the state-of-the-art models on three challenging datasets.
Revised: We demonstrate that the E3D-LSTM model performs favorably against the state-of-the-art models on three challenging datasets.

Original: The source code and trained models will be made available to the public.
Revised: The source code and trained models will be made available to the public.

Original: 4. F UTURE V IDEO P REDICTION : M OVING MNIST
Revised: 4. F UTURE V IDEO P REDICTION : M OVING MNIST

Original: We ﬁrst evaluate the E3D-LSTM model against the state-of-the-art video prediction models on a commonly used synthetic benchmark dataset with moving digits.
Revised: We ﬁrst evaluate the E3D-LSTM model against the state-of-the-art video prediction models on a commonly used synthetic benchmark dataset with moving digits.

Original: All experiments are conducted using TensorFlow (Abadi et al., 2016) and trained with the ADAM optimizer (Kingma & Ba, 2015) to minimize the l 1 + l 2 loss over every pixel in the frame.
Revised: All experiments are conducted using TensorFlow (Abadi et al., 2016) and trained with the ADAM optimizer (Kingma & Ba, 2015) to minimize the l 1 + l 2 loss over every pixel in the frame.

Original: For fair comparisons, we ensure all models to have comparable numbers of parameters, and apply the same scheduled sampling strategy (Bengio et al., 2015) in order to reduce the difﬁculty of training recurrent models.
Revised: For fair comparisons, we ensure all models to have comparable numbers of parameters, and apply the same scheduled sampling strategy (Bengio et al., 2015) in order to reduce the difﬁculty of training recurrent models.

Original: Dataset and Setup.
Revised: Dataset and Setup.

Original: The moving MNIST dataset is constructed by randomly sampling two digits from the original MNIST dataset and making them ﬂoat and bounce at boundaries with a constant velocity and angle inside a black canvas of 64 × 64 pixels.
Revised: The moving MNIST dataset is constructed by randomly sampling two digits from the original MNIST dataset and making them ﬂoat and bounce at boundaries with a constant Inputs Ground Truth Ours

Revised: PredRNN++

Original: The whole dataset has a ﬁxed number of entries, 10 , 000 sequences for training, 3 , 000 for validation and 5 , 000 for test.
Revised: ConvLSTM VPN Baseline PredRNN (a) 10 → 10 Prediction PredRNN++

Revised: Ours

Revised: Prior Context (same as Seq. 2)

Revised: Seq.

Revised: 2 Inputs

Revised: Seq.

Revised: 2 Predictions (the first line is the expected ground truth)

Revised: ConvLSTM

Original: We stack 4 E3D-LSTMs in the architecture illustrated in Figure 1(c), leaving out 3D-CNN encoders for this task.
Revised: Seq. 1 Inputs Seq. 1 Predictions (

Original: To retain the shape of hidden states over time, the integrated 3D-Conv operators are
Revised: [Figure] Test

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: velocity and angle inside a black canvas of 64 × 64 pixels. The whole dataset has a ﬁxed number of entries, 10 , 000 sequences for training, 3 , 000 for validation and 5 , 000 for test. We stack 4 E3D-LSTMs in the architecture illustrated in Figure 1(c), leaving out 3D-CNN encoders for this task.

Original:  composed of a 2 × 5 × 5 (time × height × width) convolutions and a corresponding transposed convolution with the same ﬁlter size.
Revised: To retain the shape of hidden states over time, the integrated 3D-Conv operators are composed of a 2 × 5 × 5 (time × height × width) convolutions and a corresponding transposed convolution with the same ﬁlter size.

Original: The number of hidden state channels of each E3D-LSTM is 64 .
Revised: The number of hidden state channels of each E3D-LSTM is 64 .

Original: The temporal stride is set to 1 and there is one overlapping frame over consecutive timestamps.
Revised: The temporal stride is set to 1 and there is one overlapping frame over consecutive time stamps.

Original: A single 3D-Conv layer is used as the decoder to map motion-aware hidden states to output frames.
Revised: A single 3D-Conv layer is used as the decoder to map motion-aware hidden states to output frames.

Revised: We also ﬁnd that removing the forget gate F t could make our model ﬁt better with this dataset.

Original: The E3D-LSTM model is evaluated against the state-of-the-art methods including the ConvLSTM network (Shi et al., 2015), DFN (De Brabandere et al., 2016), CDNA (Finn et al., 2016), VPN baseline model with CNN decoders (Kalchbrenner et al., 2017), PredRNN (Wang et al., 2017),
Revised: The E3D-LSTM model is evaluated against the state-of-the-art methods including the ConvLSTM network (Shi et al., 2015), DFN (De Brabandere et al., 2016), CDNA (Finn et al., 2016), VPN baseline model with CNN decoders (Kalchbrenner et al., 2017), PredRNN (Wang et al., 2017),

Original: PredRNN++ (Wang et al., 2018b) and FRNN (Oliu et al., 2018).[Figure]
Revised: PredRNN++ (Wang et al., 2018b) and FRNN (Oliu et al., 2018).

Original: Main Results.
Revised: Main Results.

Original: Table 1 shows the performance of the evaluated models using a common setting in the literature: generating 10 future frames given the previous 10 observations (denoted as 10 → 10 ).
Revised: Table 1 shows the performance of the evaluated models using a common setting in the literature: generating 10 future frames given the previous 10 observations (denoted as 10 → 10 ).

Original: We use the per-frame structural similarity index measure (SSIM)
Revised: We use the per-frame structural similarity index measure (SSIM)

Original: (Wang et al., 2004) and per-frame mean square error (MSE) for evaluation.
Revised: (Wang et al., 2004) and per-frame mean square error (MSE) for evaluation.

Original: The SSIM ranges between − 1 and 1 , representing the similarity between the generated image and the ground truth.
Revised: The SSIM ranges between − 1 and 1 , representing the similarity between the generated image and the ground truth.

Original: As shown in the second column ( 10 → 10 ) of Table 1, our model performs well against the stateof-the-art methods in both metrics.
Revised: As shown in the second column ( 10 → 10 ) of Table 1, our model performs well against the state-of-the-art methods in both metrics.

Original: The results show that the E3D-LSTM network is effective in modeling spatiotemporal data for video prediction.
Revised: The results show that the E3D-LSTM network is effective in modeling spatiotemporal data for video prediction.

Original: Figure 3(a) shows the qualitative comparisons in which our model predicts future frames from entangled digits better than other methods.
Revised: Figure 3(a) shows the qualitative comparisons in which our model predicts future frames from entangled digits better than other methods.

Original: Copy Test.
Revised: Copy Test.

Original: We evaluate the proposed model using the Copy Test setting where the task is to memorize useful information in a longer input sequence when the recurrent disturbance is present.The
Revised: We evaluate the proposed model using the Copy Test setting where the task is to memorize useful information in a longer input sequence when the recurrent disturbance is present.

Original: [Table] input clip consists of three sub-sequences, as illustrated in Figure 3(b).
Revised: The input clip consists of three sub-sequences, as illustrated in Figure 3(b).

Original: Seq 1 and Seq 2 are completely irrelevant, and ahead of them, another sub-sequence called prior context is given as the input, which is exactly the same as Seq 2.
Revised: Seq 1 and Seq 2 are completely irrelevant, and ahead of them, another sub-sequence called prior context is given as the input, which is exactly the same as Seq 2.

Original: Frames indicated by black arrows are inputs and those indicated by red arrows are expected outputs.
Revised: Frames marked by black arrows are inputs and those marked by red arrows are expected outputs.

Original: There are two goals to achieve: a) to predict 10 future framesgiven 2 past inputs of Seq 1; and b) to perform the same task with Seq 2.
Revised: There are two training objective: a) to predict 10 future frames of Seq 1; and b) to predict 10 future frames of Seq 2.

Original: At the test time, we only evaluate the prediction result of Seq 2, since an ideal spatiotemporal network is expected to discover valuable cues from prior context, despite the inﬂuence of some irrelevant recurrent states (brought by Seq 1 in this case).
Revised: At the test time, we only evaluate the prediction result of Seq 2. The copy test evaluates the modeling capability of long-range video frame relations. A well-designed model should make precise predictions regarding Seq 2, as it has seen all frames of [Table]

Original: Copy test is an interesting and important task as it evaluates the long-range spatiotemporal modeling capability to memorize relevant information in the input sequence.
Revised: this sequence before. However, this task is difﬁcult for previous LSTM networks. Because Seq 1 is completely irrelevant, the attempt of making predictions of Seq 1 can erase its memory of Seq 2.

Original: The results are presented in the third column (Copy) of Table 1.
Revised: The results are presented in the third column (Copy) of Table 1.

Original: All baseline models suffer from the inﬂuence brought by irrelevant frames in Seq 2 and tend to gradually forget the salient information in the prior context.
Revised: All baseline models suffer from the inﬂuence brought by irrelevant frames in Seq 2 and tend to gradually forget the salient information in the prior context.

Original: However, thanks to the eidetic 3D memory, our E3D-LSTM model captures the long-term video frame interactions and performs well in both metrics.
Revised: However, thanks to the eidetic 3D memory, our E3D-LSTM model captures the long-term video frame interactions and performs well in both metrics.

Original: A careful inspection of the attention weight shows that the E3D-LSTM model can better attend to useful historical representations across multiple timestamps.
Revised: A careful inspection of the attention weight shows that the E3D-LSTM model can better attend to useful historical representations across multiple time stamps.

Original: The copy test suggests that the E3D-LSTM network is capable of modeling long-range periodical motions effectively.
Revised: The copy test suggests that the E3D-LSTM network is capable of modeling long-range periodical motions effectively.

Original: Ablation Study.
Revised: Ablation Study.

Original: We conduct a series of ablation studies and summarize the results in Table 2.
Revised: We conduct a series of ablation studies and summarize the results in Table 2.

Original: First, on the ﬁrst two rows, we show two alternative 3D-LSTM models with 3D-Convs outside the recurrent unit, including 3D-CNN at Bottom (Figure 1(a)) and 3D-CNN on Top (Figure 1(b)).
Revised: First, on the ﬁrst two rows, we show two alternative 3D-LSTM models with 3D-Convs outside the recurrent unit, including 3D-CNN at Bottom (Figure 1(a)) and 3D-CNN on Top (Figure 1(b)).

Original: The performance drop validates the integration of 3D-Convs and RNN units via the eidetic 3D memory.
Revised: The performance drop validates the integration of 3D-Convs and RNN units via the eidetic 3D memory.

Original: Second, the third baseline method is a special case where all 3D convolutional ﬁlters in our model are reduced to 2D.
Revised: Second, the third baseline method is a special case where all 3D convolutional ﬁlters in our model are reduced to 2D.

Original: The results demonstrate the effect of capturing local spatiotemporal patterns by the 3D memory within an individual recurrent state.
Revised: The results demonstrate the effect of capturing local spatiotemporal patterns by the 3D memory within an individual recurrent state.

Original: Finally, the contribution of the memory attention mechanism can be isolated in the fourth baseline method.
Revised: Furthermore, the contribution of the memory attention mechanism can be isolated in the fourth baseline method.

Original: Note that all evaluated models are trained with a similar number of parameters for fair comparisons, and the performance gain comes from design options rather than increased model parameters.
Revised: Note that all evaluated models are trained with a similar number of parameters for fair comparisons, and the performance gain comes from design options rather than increased model parameters.

Original: 4. F UTURE V IDEO P REDICTION : KTH A CTION
Revised: 4. F UTURE V IDEO P REDICTION : KTH A CTION

Original: We evaluate the proposed E3D-LSTM model on video prediction of real-world  human actions.
Revised: We evaluate the proposed E3D-LSTM model on video prediction of real-world datasets, including a human action dataset as follows and a trafﬁc ﬂow dataset as described in Appendix B.

Original: Dataset and Setup.
Revised: Dataset and Setup.

Original: The KTH action dataset (Schuldt et al., 2004) contains 25 individuals performing 6 types of actions, including walking, jogging, running, boxing, hand waving and hand clapping.
Revised: The KTH action dataset (Schuldt et al., 2004) contains 25 individuals performing 6 types of actions, including walking, jogging, running, boxing, hand waving and hand clapping.

Original: On average, each video clip lasts 4 seconds.
Revised: On average, each video clip lasts 4 seconds.

Revised: We follow the experimental setup in (Villegas et al., 2017) by using person 1 - 16 for training and 17 - 25 for testing.

Original: Each frame is resized to 128 × 128 pixels.
Revised: Each frame is resized to 128 × 128 pixels.

Original: We follow the experimental setup in (Villegas et al., 2017) by using person 1 - 16 for training and 17 - 25 for testing, and then dividing these videos into a training set of 108 , 717 sequences and a test set of 4 , 086 sequences.
Revised: We employ the same E3D-LSTM network architecture detailed in Section 4.1.

Original: Models are trained to predict next 10 frames from the previous 10 observations.
Revised: Models are trained to predict next 10 frames from the previous 10 observations.

Original: At the test time, the prediction horizon  is extended to 20 or 40 timestamps into the future.
Revised: The prediction horizon at the test time is extended to 20 or 40 time stamps.

Original: We employ a similar setting detailed in Section 4.1.

Original: Same as prior work, we use SSIM and Peak Signal to Noise Ratio (PSNR) as evaluation metrics.

Original: The training process is stopped after 20 epochs.

Original: Results.
Revised: Results.

Original: Table 3 shows quantitative results of the proposed model and state-of-the-art methods.
Revised: Table 3 shows quantitative results of the proposed model and state-of-the-art methods.

Original:  Consistent with the observations on the moving MNIST dataset, the E3D-LSTM model performs favorably against the state-of-the-art methods across three settings of predicting future 10 frames, 20 frames, and copy test.
Revised: Same as prior work, we use SSIM and PSNR as metrics. Consistent with the observations on the moving MNIST dataset, the E3D-LSTM model performs favorably against the state-of-the-art methods across three settings of predicting future 10 frames, 20 frames, and copy test.

Original: These empirical results demonstrate the effectiveness of the E3D-LSTM model for modeling spatiotemporal data.[Figure]
Revised: These empirical results demonstrate the effectiveness of the E3D-LSTM model for modeling spatiotemporal data.

Original: Figure 4 shows representative predicted frames.
Revised: Figure 4 shows representative predicted frames.

Original: In the top half (predicting the next 40 frames based on the 10 previous frames), our model predicts more accurate motion trajectories into the future, whereas the state-of-the-art methods (PredRNN++ and ConvLSTM) incorrectly predict the person moving out of the scenes.
Revised: In the top half (predicting the next 40 frames based on the 10 previous frames), our model predicts more accurate motion trajectories into the future, whereas other methods (PredRNN++ and ConvLSTM) incorrectly predict the person moving out of the scenes.

Original: The lower half shows the copy test providing the expected outputs as prior inputs.
Revised: The lower half shows the copy test providing the expected outputs as prior inputs.

Original: We directly apply models, which are trained under the ﬁrst setting, to this test.
Revised: We directly apply models, which are trained under the ﬁrst setting, to this test.

Original: Without the prior context, it would be difﬁcult to predict human motions for some cases.
Revised: Without the prior context, it would be difﬁcult to predict human motions for some cases.

Original: With prior inputs, our model beneﬁts from the most from its memories and responds well to rapid appearance change.
Revised: With prior inputs, our model beneﬁts from the most from its memories and responds well to rapid appearance change.

Original: In contrast, the PredRNN++ and ConvLSTM methods are not able to useful spatiotemporal patterns from distant observations due to the lack of modeling long-term data relations.
Revised: In contrast, the PredRNN++ and ConvLSTM methods are not able to useful spatiotemporal patterns from distant observations due to the lack of modeling long-term data relations.

Original: 4. E ARLY A CTIVITY R ECOGNITION : S OMETHING - SOMETHING
Revised: 4. E ARLY A CTIVITY R ECOGNITION : S OMETHING -S OMETHING

Original: To validate that the E3D-LSTM model can learn high-level video representations effectively, we carry out experiments on early activity recognition.
Revised: To validate that the E3D-LSTM model can learn high-level video representations effectively, we carry out experiments on early activity recognition.

Original: The task is to predict an activity category in a video after only observing a fraction of frames.
Revised: The task is to predict an activity category in t=11 t= PredRNN++ Ours ConvLSTM

Revised: Inputs

Revised: Prediction Ground Truth

Revised: PredRNN++

Revised: Ours

Revised: ConvLSTM

Revised: Inputs

Revised: Prediction Ground Truth

Revised: Prior Inputs t=3 t=5 t=7 t= t=13 t=15 t=17 t=19 t=21 t=23 t=25 t=27 t=29 t=31 t=33 t=35 t=37 t=39 t=41 t=43 t=45 t=47 t=

Revised: [Table]

Revised: a video after only observing a fraction of frames.

Revised: We choose not to evaluate on the full-length video for the activity recognition task, because when a model sees the full-length video, it may make decisions solely based on the scene information, e.g. seeing only the last frame is enough to recognize many actions.

Revised: As a result, the full-length video task may not align well with our previous video prediction tasks, in which the sequential tendency and causality are important.

Original: Dataset and Setup.
Revised: Dataset and Setup.

Original: The something-something dataset (Goyal et al., 2017) is a recent benchmark for activity/action recognition.
Revised: The something-something dataset (Goyal et al., 2017) is a recent benchmark for activity/action recognition ( https://20bn.com/datasets/something-something ).

Original: We use the standard  subset( https://20bn.com/datasets/ something-something ) which contains 56 , 769 short video clips for the training set and 7 ,clips for the validation set on 41 action categories.
Revised: We use the standard and ofﬁcial subset which contains 56 , 769 short videos for the training set and 7 , 503 videos for the validation set on 41 action categories.

Original: The video length ranges between 2 and 6 seconds with 24 fps.
Revised: The video length ranges between 2 and 6 seconds with 24 fps.

Original: We adopt the early activity recognition setting (Ma et al., 2016; Zeng et al., 2017; Zhou et al., 2018), where a model predicts an action type after observing the ﬁrst 25% or 50% frames of each video.
Revised: We adopt the early activity recognition setting (Ma et al., 2016; Zeng et al., 2017; Zhou et al., 2018), where a model predicts an action type after observing the ﬁrst 25% or 50% frames of each video.

Original: As these actions appear in diverse scenes and involve interaction with different objects, it is challenging to predict actions even for humans (See Figure 5).
Revised: As these actions appear in diverse scenes and involve interaction with different objects, it is challenging to predict actions even for humans (See Figure 5).

Original: There are only subtle differences between some actions in this dataset, such as “Poking a stack of [Something] so that the stack collapses” versus “Poking a stack of [Something] without the stack collapsing”, or “Pouring [Something] into [Something]” versus “Trying to pour [Something] into [Something], but missing so it spills next to it”.
Revised: There are only subtle differences between some actions in this dataset, such as “Poking a stack of [Something] so that the stack collapses” versus “Poking a stack of [Something] without the stack collapsing”, or “Pouring [Something] into [Something]” versus “Trying to pour [Something] into [Something], but missing so it spills next to it”.

Original: To make a correct prediction, a model needs to exploit spatiotemporal cues to understand the subtle differences between actions.
Revised: To make a correct prediction, a model needs to exploit spatiotemporal cues to understand the subtle differences between actions.

Original: Namely, one can evaluate the model effectiveness for high-level video tasks.
Revised: Namely, one can evaluate the model effectiveness for high-level video tasks.

Original: Recognizing early action accurately requires predictions into future frames, which can only be achieved using an effective model based on historical observations.
Revised: Recognizing early action accurately requires predictions into future frames, which can only be achieved using an effective model based on historical observations.

Original: Hyper-parameters and Baselines.
Revised: Hyper-parameters and Baselines.

Original: We use the architecture illustrated in Figure 1(c) as our model, which consists of 2 layers of 3D-CNN encoders, 4 layers of E3D-LSTMs, and 2 layers of 3D-CNN decoders.
Revised: We use the architecture illustrated in Figure 1(c) as our model, which consists of 2 layers of 3D-CNN encoders, 4 layers of E3D-LSTMs, and 2 layers of 3D-CNN decoders.

Original: The 3D-CNN encoders take 4 consecutive 224 × 224 raw frames, encode them into 2 × 56 × 56 × 64 feature maps at each timestamp, and then feed them into E3D-LSTM.
Revised: The 3D-CNN encoders take 4 consecutive 224 × 224 raw frames, encode them into 2 × 56 × 56 × 64 feature maps at each time stamp, and then feed them into E3D-LSTM.

Original: Each encoder layer has 64 ﬁlters (the ﬁlter dimensions are 2 × 5 × 5 ).
Revised: Each encoder layer has 64 ﬁlters (the ﬁlter dimensions are 2 × 5 × 5 ).

Original: We use the same hyper-parameters for E3D-LSTMs as for video prediction.
Revised: We use the same hyper-parameters for E3D-LSTMs as for video prediction.

Original: The decoder layers map the output of E3D-LSTMs back to RGB space, which is an 1 × 3 matrix, predicting the next frame following the inputs.
Revised: The decoder layers map the output of E3D-LSTMs back to RGB space, which is an 1 × 3 matrix, predicting the next frame following the inputs.

Revised: We train the network to predict the next 10 frames using the front 25% or 50% frames of the video.

Original: We train
Revised: Note that we

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: the network to predict the next 10 frames using the front 25% or 50% frames of the video.

Original: Note that we do not extend any predictive states into the future at the test time.
Revised: do not extend any predictive states into the future at the test time.

Original: For both training and testing, we concatenate hidden representations of the top recurrent units with respect to the last 16 input timestamps (considering the ﬁrst 25% video snippets usually have about 20 to 30 frames), and feed them into the classiﬁer for activity recognition.
Revised: For both training and testing, we concatenate hidden representations of the top recurrent units with respect to the last 16 input time stamps (considering the ﬁrst 25% video snippets usually have about 20 to 30 frames), and feed them into the classiﬁer for activity recognition.

Original: The classiﬁer contains 2 layers of 3D-Convs with 128 ﬁlters (ﬁlter dimensions: 2 × 3 × 3 , ﬁlter strides: 2 × 2 × 2 ) followed by a 2 × 2 × 2 pooling layer.
Revised: The classiﬁer contains 2 layers of 3D-Convs withﬁlters (ﬁlter dimensions: 2 × 3 × 3 , ﬁlter strides: 2 × 2 × 2 ) followed by a 2 × 2 × 2 pooling layer.

Original: They transform the concatenated recurrent features from 16 × 56 × 56 × 64 to 1 × 7 × 7 × 128 , then pass them to a 512-channel fully-connected layer followed by a 41 -way classiﬁcation.
Revised: They transform the concatenated recurrent features from 16 × 56 × 56 × 64 to 1 × 7 × 7 × 128 , then pass them to a 512-channel fully-connected layer followed by a 41 -way classiﬁcation.

Original: We also exploit the self-supervised auxiliary learning approach and train the model with an objective function in Equation 2.
Revised: We also exploit the self-supervised auxiliary learning approach and train the model with an objective function in equation 3.

Original: We set λ ( i ) in Equation 3 to 10 in the beginning ( i = 0 ), and decrease it with a speed of 2 × 10 − 5 per iteration, lower bounded by η = 0 .
Revised: We set λ ( i ) in equation 4 to 10 in the beginning ( i = 0 ), and decrease it with a speed of 2 × 10 − 5 per iteration, lower bounded by η = 0 .

Original: We evaluate the E3D-LSTM model against the state-of-the-art feed-forward 3D-Conv architectures including C3D/I3D (Diba et al., 2016; Carreira & Zisserman, 2017), Separable 3D-CNN (Xie et al., 2018; Qiu et al., 2017) and (2+1)D-CNN (Tran et al., 2018).
Revised: We evaluate the E3D-LSTM model against the state-of-the-art feed-forward 3D-Conv architectures including C3D/I3D (Diba et al., 2016; Carreira & Zisserman, 2017), Separable 3D-CNN (Xie et al., 2018; Qiu et al., 2017) and (2+1)D-CNN (Tran et al., 2018).

Original: Thesespatiotemporal networks achieve the state-of-the-art results on the UCF-101 and Kinetics benchmark datasets for action recognition.
Revised: These networks achieve the stateof-the-art results on the UCF-101 and Kinetics benchmark datasets for action recognition.

Original: For fair comparisons, we train these baseline models using similar backbones to the E3D-LSTM network (i.e., 2 layers of 3D-Convs as encoders, 4 layers of 3D-Convs for spatiotemporal modeling, and 2 FC layers for classiﬁcation).
Revised: For fair comparisons, we train these baseline models using similar backbones to the E3D-LSTM network.

Original: Results.
Revised: Results.

Original: Table 4 shows the classiﬁcation accuracy of the E3D-LSTM network against the stateof-the-art feed-forward 3D-CNNs.
Revised: Table 4 shows the classiﬁcation accuracy of the E3D-LSTM network against the stateof-the-art feed-forward 3D-CNNs.

Original: The E3D-LSTM model performs favorably against the other methods in two settings of using the ﬁrst 25% and 50% frames, showing its effectiveness in learning high-level spatiotemporal representations.
Revised: The E3D-LSTM model performs favorably against the other methods in two settings of using the ﬁrst 25% and 50% frames, showing its effectiveness in learning high-level spatiotemporal representations.

Original: Figure 5 shows two pairs of video activities that are easy to confuse, especially with such limited observations.
Revised: Figure 5 shows two pairs of video activities that are easy to confuse, especially with such limited observations.

Original: For instance, our model correctly forecasts[Table] [Table] the collapse of books, while only a tendency of it has been shown explicitly within the ﬁrst 25% frames.
Revised: For instance, our model correctly forecasts the collapse of books, while only a tendency of it has been shown explicitly within the ﬁrst 25% frames.

Original: This reasoning ability comes from the integrated design of our model to capture both shortterm motions and long-term dependencies.
Revised: This reasoning ability comes from the integrated design of our model to capture both shortterm motions and long-term dependencies.

Original: On the other hand, as the feed-forward 3D-CNN models long-term relations by sampling and assembling, it does not perform well in ﬁnding the temporal dependencies between cause and effect.
Revised: On the other hand, as the feed-forward 3D-CNN models long-term relations by sampling and assembling, it does not perform well in ﬁnding the temporal dependencies between cause and effect.

Original: We note that Zhou et al.
Revised: We note that Zhou et al.

Original: (2018) introduced a feed-forward
Revised: (2018) introduced a feed-forward

Revised: [Table]

Revised: [Table]

Original: CNN model and also reported early recognition results on the same dataset.
Revised: CNN model and also reported early recognition results on the same dataset.

Original: It is not meaningful to compare these two methods in terms of accuracy as our model is trained only using 25% - 50% frames of a video instead of the entire video in (Zhou et al., 2018).
Revised: It is not meaningful to compare these two methods in terms of accuracy as our model is trained only using 25% - 50% frames of a video instead of the entire video in (Zhou et al., 2018).

Original: Moreover, the two methods are trained using different backbone networks and different splits of datasets
Revised: Moreover, the two methods are trained using different backbone networks and different splits of datasets

Original: [Table]
Revised: [Table]

Original: A number of recent studies show that separating temporal and spatial convolution operations in a 3D-CNN model leads to better results (Xie et al., 2018; Qiu et al., 2017; Tran et al., 2018).
Revised: A number of recent studies show that separating temporal and spatial convolution operations in a 3D-CNN model leads to better results (Xie et al., 2018; Qiu et al., 2017; Tran et al., 2018).

Original: This observation is validated by our results shown in Table 4.
Revised: This observation is validated by our results shown in Table 4.

Original: However, it seems counter-intuitive since such separation leads to a pseudo-3D convolution, in which spatial and temporal ﬁlters are independent.
Revised: However, it seems counter-intuitive since such separation leads to a pseudo-3D convolution [Table] in which spatial and temporal ﬁlters are independent.

Original: Interestingly, such separation in our model leads to performance loss, suggesting the 3D convolution in the E3D-LSTM jointly captures the temporal and spatial information.
Revised: Interestingly, such separation in our model leads to performance loss, suggesting the 3D convolution in the E3D-LSTM jointly captures the temporal and spatial information.

Original: Ablation Study.
Revised: Ablation Study.

Original: We conduct similar ablation studies as in Section 4.1 and summarize the results in Table 5.
Revised: We conduct similar ablation studies as in Section 4.1 and summarize the results in Table 5.

Original: The results from the ﬁrst two rows show that our deeper integration of 3D-Convs inside
Revised: The results from the ﬁrst two rows show that our deeper integration of 3D-Convs inside

Original: RNNs is helpful not only for pixel-level video prediction, but also for high-level activity recognition.
Revised: RNNs is helpful not only for pixel-level video prediction, but also for high-level activity recognition.

Original: The results on row 3 and 4 show the contribution of the two important components in the proposed Eidetic 3D LSTM: a) 3D convolution features, and b) memory attention mechanism.
Revised: The results on row 3 and 4 show the contribution of the two important components in the proposed Eidetic 3D LSTM: a) 3D convolution features, and b) memory attention mechanism.

Original: Both components are useful and important for modeling spatiotemporal data effectively.
Revised: Both components are useful and important for modeling spatiotemporal data effectively.

Original: Finally, Table 6 shows applying self-supervisedpredictive training in different settings.
Revised: Table 6 shows applying self-supervised training in different settings.

Original: The proposed self-supervised auxiliary learning approach performs better than other alternatives, including using video prediction models as network initialization, or training the model under these two tasks with a ﬁxed objective function ratio.
Revised: The proposed self-supervised auxiliary learning approach performs better than other alternatives, including using video prediction models as network initialization, or training the model under these two tasks with a ﬁxed objective function ratio.

Revised: We enable online early activity recognition by making the classiﬁer only depend on a concatenation of the last 5 recurrent output states.

Revised: Using equation 1, we ﬁx the length of the attended memory states by setting τ to 5 .

Revised: Such settings are applied to both the training and test processes.

Revised: Table 7 shows the experimental results.

Revised: Despite the slight decrease of accuracy, it enables an online prediction.

Original:   C ONCLUSION
Revised: [Table]  C ONCLUSION

Original: Spatiotemporal predictive learning has shown signiﬁcant improvements in a variety of applications, such as weather forecasting, trafﬁc ﬂow prediction, and physical interaction simulation.
Revised: Spatiotemporal predictive learning has shown signiﬁcant improvements in a variety of applications, such as weather forecasting, trafﬁc ﬂow prediction, and physical interaction simulation.

Original: Although considered to be a promising self-supervised feature learning paradigm, it seldom shows its effectiveness beyond video prediction.
Revised: Although considered to be a promising self-supervised feature learning paradigm, it seldom shows its effectiveness beyond video prediction.

Original: In this paper, we presented the E3D-LSTM model based on 3D convolutional recurrent units for this task.
Revised: In this paper, we presented the E3D-LSTM model based on 3D convolutional recurrent units for this task.

Original: In this model, we integrated 3D-Convs into state transitions to perceive short-term motions and designed a memory attentive module controlled by recurrent gates to capture the long-term video frame interaction.
Revised: In this model, we integrated 3D-Convs into state transitions to perceive short-term motions and designed a memory attentive module controlled by recurrent gates to capture the long-term video frame interaction.

Original: We evaluated the E3D-LSTM model on the Moving MNIST and KTH action datasets for video prediction, and on the Something-Something dataset for early activity recognition.
Revised: We evaluated the E3D -LSTM model on the Moving MNIST and KTH action datasets for video prediction, and on the Something-Something dataset for early activity recognition.

Original: Experimental results demonstrate that the E3D-LSTM model performs favorably against the state-of-the-art methods on video prediction tasks.
Revised: Experimental results demonstrate that the E3D-LSTM model performs favorably against the state-of-the-art methods on video prediction tasks.

Original: A K EY E QUATIONS OF S PATIOTEMPORAL LSTM
Revised: A K EY E QUATIONS OF S PATIOTEMPORAL LSTM

Original: Operations inside a ST-LSTM unit at timestamp t and layer l are shown as follows.
Revised: Operations inside a Spatiotemporal LSTM unit at time stamp t and layer k are shown as follows:

Original: There are four inputs: X t , the raw frame or hidden states from the previous layer; M l − 1 t , the previous spatiotemporal memory; H lt − 1 and C lt − 1 , the previous hidden states and memory states.

Original: [Equation]
Revised: [Equation]

Original: where σ is the sigmoid function, ∗ is the convolution operator, and (cid:12) denotes the Hadamard product.
Revised: where σ is the sigmoid function, ∗ is the convolution operator, and (cid:12) denotes the Hadamard product.

Revised: There are four inputs: X t , the raw frame or hidden states from the previous layer; M k − 1 t , the previous spatiotemporal memory; H kt − 1 and C kt − 1 , the previous hidden states and memory states.

Original: Two sets of gate structures, including input gate i t and i (cid:48) t , forget gate f t and f (cid:48) t , as well as the output gate o t , control the information ﬂow in space-time domain.
Revised: Two sets of gate structures, including input gate i t and i (cid:48) t , forget gate f t and f (cid:48) t , as well as the output gate o t , control the information ﬂow in space-time domain.

Original: All of them can be presented by
Revised: All of them can be presented by

Original: R H
Revised: R H

Original: × W × C dimensional tensors, where the ﬁrst two dimensions are the width and height of feature maps, and the last one is the number of feature map channels.
Revised: × W × C dimensional tensors, where the ﬁrst two dimensions are the width and height of feature maps, and the last one is the number of feature map channels.

Revised: B T RAFFIC F LOW P REDICTION : A R EAL V IDEO P REDICTION A PPLICATION

Revised: We further evaluate our method on the TaxtBJ dataset, which contains real-world trafﬁc ﬂow data for analysis and prediction (in consecutive heat maps).

Revised: Predicting urban trafﬁc conditions is a complex setting, as the heat maps are very noisy and we do not have any underlying or additional information that can facilitate this task.

Revised: [Figure]

Revised: Dataset and Setup.

Revised: The TaxiBJ dataset is collected from the chaotic real-world environment using

Revised: GPS monitors of taxicabs Beijing.

Revised: Each frame is a 32 × 32 × 2 heat map.

Revised: The last dimension denotes the entering and leaving trafﬁc ﬂow intensities at the same area.

Revised: We split the whole dataset into a training set and a test set as described in (Zhang et al., 2017).

Revised: We train the networks to predictframes (the next 2 hours) from 4 observations.

Revised: We use the same network architecture and training setups as the one on Moving MNIST and KTH datasets.

Revised: Results.

Revised: We report MSE at every time stamp in Table 8 where lower scores indicate better prediction results.

Revised: We also show a prediction example in Figure 6.

Revised: Furthermore, we visualize the differences between the generated heat maps and the ground truth heat maps.

Revised: Overall, the E3D-LSTM model outperforms the other methods, with the lowest differences intensities in most areas.

Revised: [Table]