L ONG H ISTORY S HORT -T ERM M EMORY FOR L ONG -T ERM V IDEO P REDICTION
Anonymous authors
Paper under double-blind review
A BSTRACT
While video prediction approaches have advanced considerably in recent years, learning to predict long-term future is challenging — ambiguous future or error propagation over time yield blurry predictions.
To address this challenge, existing algorithms rely on extra supervision (e.g., action or object pose), motion ﬂow learning, or adversarial training.
In this paper, we propose a new recurrent unit,
Long History Short-Term Memory (LH-STM).
LH-STM incorporates long history states into a recurrent unit to learn longer range dependencies.
To capture spatiotemporal dynamics in videos, we combined LH-STM with the Context-aware Video
Prediction model (ContextVP).
Our experiments on the KTH human actions and
BAIR robot pushing datasets demonstrate that our approach produces not only sharper near-future predictions, but also farther into the future compared to the state-of-the-art methods.
I NTRODUCTION
Learning the dynamics of an environment and predicting consequences in the future has become an important research problem.
A common task is to train a model that accurately predicts pixel-level future frames conditioned on past frames.
It can be utilized for intelligent agents to guide them to interact with the world, or for other video analysis tasks such as activity recognition.
An important component of designing such models is how to effectively learn good spatio-temporal representations from video frames.
The Convolutional Long Short-Term Memory (ConvLSTM) network (Xingjian et al., 2015) has been a popular model architecture choice for video prediction.
However, recent stateof-the-art approaches produce high-quality predictions only for one or less then ten frames (Lotter et al., 2016;
Villegas et al., 2017a; Byeon et al., 2018).
Learning to predict long-term future video frames remains challenging due to 1) the presence of complex dynamics in high-dimensional video data, 2) prediction error propagation over time, and 3) inherent uncertainty of the future.
In order to predict reasonably long-term video frames, some approaches represent video dynamics by hierarchical methods (Wichers et al., 2018; Villegas et al., 2017b) or action-conditioned models (Finn et al., 2016;
Oh et al., 2015).
To efﬁciently model video dynamics, spatio-temporal ConvLSTM models have also been studied (Wang et al., 2017; 2018a;b).
Another popular direction is to learn multiple plausible futures via an adversarial loss (Mathieu et al., 2015) or a Variational Autoencoder (VAE) (Denton & Fergus, 2018; Babaeizadeh et al., 2017; Lee et al., 2018).
In this paper, we attempt to solve two of the issues mentioned above: learning complex dynamics of videos and minimizing long-term prediction error by fully observing the history.
We propose a novel modiﬁcation of the ConvLSTM structure, Long History Short-Term Memory (LH-STM).
LH-STM learns to interconnect history states and the current input by a History SoftSelection Unit (HistSSel) and double memory modules.
The weighted history states computed by our HistSSel units are combined with the history memory and then used to update the current states by the update memory.
The proposed method brings the power of higher-order RNNs (Soltani &
Jiang, 2016) to ConvLSTMs, which have been limited to simple recurrent mechanisms so far.
The
HistSSel unit acts as a short-cut to the history, so the gradient ﬂow in the LSTM is improved.
More powerful RNNs are likely to be necessary to solve the hard and unsolved problem of long-term video prediction, which is extremely challenging for current architectures.
Moreover, by disentangling the history and update memories, our model can fully utilize the long history states.
This structure can better model long-term dependencies in sequential data.
In this paper, the proposed modiﬁcation is integrated into the ConvLSTM-based architectures to solve the long-term video prediction problem: Context-aware Video Prediction (ContextVP) model.
The proposed models can fully leverage long-range spatio-temporal contexts in real-world videos.
Our experiments on the
KTH human actions and the BAIR robot pushing datasets show that our model produces sharp and realistic predictions for more frames into the future compared to recent state-of-the-art long-term video prediction methods.
R ELATED W ORK
Learning Long-Term Dependencies with Recurrent Neural Networks (RNN): While Long ShortTerm Memory (LSTM) has been successful for sequence prediction, many recent approaches aim to capture longer-term dependencies in sequential data.
Several works have proposed to allow dynamic recurrent state updates or to learn more complex transition functions.
Chung et al.
(2016) introduced the hierarchical multiscale RNN that captures a hierarchical representation of a sequence by encoding multiple time scales of temporal dependencies.
Koutnik et al.
modiﬁed the standard RNN to a
Clockwork RNN that partitions hidden units and processes them at different clock speeds.
Neil et al.
(2016) introduced a new time gate that controls update intervals based on periodic patterns.
Campos et al.
(2017) proposed an explicit skipping module for state updates.
Zilly et al.
(2017) increased the recurrent transition depth with highway layers.
Fast-Slow Recurrent Neural Networks (Mujika et al., 2017) incorporate ideas from both multiscale (Schmidhuber, 1992; El Hihi & Bengio, 1996; Chung et al., 2016) and deep transition (Pascanu et al., 2013;
Zilly et al., 2017) RNNs.
The advantages of the above approaches are efﬁcient information propagation through time, better long memory traces, and generalization to unseen data.
Alternative solutions include the use of history states, an attention model or skip connections.
Soltani & Jiang (2016) investigated a higher order RNN to aggregate more history information and showed that it is beneﬁcial for long range sequence modeling.
Cheng et al.
(2016) deployed an attention mechanism in LSTM to induce relations between input and history states.
Gui et al.
(2018) incorporated into an LSTM, dynamic skip connections and reinforcement learning to model long-term dependencies.
These approaches use the history states in a single LSTM by directly adding more recurrent connections or adding an attention module in the memory cell.
These models are used for one dimensional sequence modeling, whereas our proposed approach separates the history and update memories that learn to encode the long-range relevant history states.
Furthermore, our approach is more suitable for high-dimensional ( e.g. , video) prediction tasks.
Video Prediction: The main issue in long-term pixel-level video prediction is how to capture longterm dynamics and handle uncertainty of the future while maintaining sharpness and realism.
Oh et al.
(2015) introduced action-conditioned video prediction using a Convolutional Neural Network (CNN) architecture.
Villegas et al.
(2017b)
and Wichers et al.
(2018) focused on a hierarchical model to predict long-term videos.
Their model estimates high-level structure before generating pixel-level predictions.
However, the approach by Villegas et al.
(2017b) requires object pose information as ground truth during training.
Finn et al.
(2016) used ConvLSTM to explicitly model pixel motions.
To generate high-quality predictions, many approaches train with an adversarial loss (Mathieu et al., 2015; Wichers et al., 2018; Vondrick et al., 2016; Vondrick & Torralba, 2017; Denton et al., 2017;
Lee et al., 2018).
Another active line of investigation is to train stochastic prediction models using
VAEs (Denton & Fergus, 2018; Babaeizadeh et al., 2017; Lee et al., 2018).
These models predict plausible futures by sampling latent variables and produce long-range future predictions.
The Spatio-temporal LSTM (Wang et al., 2017; 2018a) was introduced to better represent the dynamics of videos.
This model is able to learn spatial and temporal representations simultaneously.
Byeon et al.
(2018) introduced a Multi-Dimensional LSTM-based approach (Stollenga et al., 2015) for video prediction.
It contains directional ConvLSTM-like units that efﬁciently aggregate the entire spatio-temporal contextual information.
Wang et al.
(2018b) recently proposed a memory recall function with 3D ConvLSTM.
This work is the most related to our approach.
It uses a set of cell states with an attention mechanism to capture the long-term frame interaction similar to the work of Cheng et al.
With 3D-convolutional operations, the model is able to capture short-term and long-term information ﬂow.
In contrast to this work, the attention mechanism in our model is used for a set of hidden state.
We also disentangle the history and update memory cells to better memorize and restore the relevant information.
By integrating a double memory LH-STM into the context-aware video prediction (ContextVP) model (Byeon et al., 2018), our networks can capture the entire spatio-temporal context for a long-range video sequence.
M ETHOD
In this section, we ﬁrst describe the standard ConvLSTM architecture and then introduce the LH-STM.
Finally, we explain the ConvLSTM-based network architectures for multi-frame video prediction using the Context-aware Video Prediction model (ContextVP)
(Byeon et al., 2018).
3. C ONVOLUTIONAL LSTM
Let X n 1 = { X 1 , ..., X n } be an input sequence of length n .
X k ∈ R h × w × c is the k -th frame, where k ∈ { 1 , ..., n } , h is the height, w the width, and c the number of channels.
For the input frame X k , a ConvLSTM unit computes the current cell and hidden states ( C k , H k ) given the cell and hidden states from the previous frame, ( C k − 1 , H k − 1 ):
[Equation]
by computing the input, forget, output gates i k , f k , o k , and the transformed cell state ˆ C k :
[Equation]
where σ is the sigmoid function, W and M are 2D convolutional kernels for input-to-state and state-to-state transitions, ( ∗ ) is the convolution operation, and ( (cid:12) ) is element-wise multiplication.
The size of the weight matrices depends on the size of convolutional kernel and the number of hidden units.
3. L ONG H ISTORY S HORT -T ERM M EMORY (LH-STM)
LH-STM is an extension of the standard ConvLSTM model by integrating a set of history states into the LSTM unit.
The history states include the spatio-temporal context of each frame in addition to the pixel level information in the frame itself.
Figure 1 illustrates the differences between a standard
RNN, Higher-Order RNN (Soltani & Jiang, 2016), and our proposed model (Double and Single
LH-STM).
[Figure]
History Soft-Selection Unit (HistSSel): The HistSSel unit computes the relationship between the recent history and the earlier ones using dot-product similar to (Vaswani et al., 2017) 1 .
This
H k − m
H k −~ H k − m V
H k −~ H k − 2 V ~ H k − m K ~ H k − 2 K ….
~ H k − 1 Q ~ H k − 1 Q ….
Softmax
H k − m : k −
W k − m V
W k − m K
W k − 2 V
W k − 2 K
W k − 1 Q
Queries (
[Figure] [Figure]
(K) Values (V) (
[Figure]
X k −^ X k −
LH-STM
LH-STM
LH-STM LH-STM
LH-STM
LH-STM LH-STM
LH-STM LH-STM
LH-STM LH-STM
LH-STM ^
X k
^
X k
+
X k −
X k
LH-STM mechanism can be formulated as SoftSel ( Q, K, V ) =
softmax ( W Q Q · W Q K ) · W Q V .
It consists of queries (Q), keys (K) and values (V).
It computes the dot products of the queries and the keys; and then applies the a softmax function.
Finally, the values (V) are weighted by the outputs of softmax function.
The queries, keys, and values can be optionally transformed by the W Q , W K , and W V matrices.
Using this mechanism, HistSSel computes the relationship between the last hidden state H k - 1 and the earlier hidden states H k - m : k - 2 at time step k
(See Figure 2).
H k - m : k - 2 is the set of previous hidden states, ( H k - m , H k - m - 1 , · · · H k - 3 , H k - 2 ) .
We will show in Section 4.2 the beneﬁt of using history states versus using the past input frames directly.
The history soft-selection mechanism can be formulated as follows:
[Equation]
Single LH-STM:
A simple way to employ HistSSel in ConvLSTM (Equation 1) is to add
HistSSel ( H k - 1 , H k - m : k - 2 ) in addition to the input and the previous states ( X k , H k - 1 , C k - 1 ) in the
Equation 6 Figure 1c shows a diagram of the computation.
This direct extension is named Single
LH-STM : H k = SingleConvLSTM
( X k , H k - 1 , C k - 1 , HistSSel ( H k - 1 , H k - m : k - 2 ) ).
[Equation]
Double LH-STM: To effectively learn dynamics from the history states, we propose Double LH-STM .
It contains two ConvLSTM blocks, History LSTM (H-LSTM) and Update LSTM (U-LSTM).
The goal of the Double LH-STM is to explicitly separate the long-term history memory and the update memory.
By disentangling these, the model can better encode complex long-range history and keep track of the their dependencies.
Figure 1d illustrates a diagram of Double LH-STM.
The H-LSTM block explicitly learns the complex transition function from the (possibly entire) set of past hidden states, H k - m : k - 1 , 1 < m < k .
If m = k , H-LSTM incorporates the entire history up to the time step k − 1 .
The U-LSTM block updates the states H k and C k for the time step k , given the input X k , previous cell state C k − 1 , and the output of the H-LSTM, H (cid:48)
k − 1 . The History-LSTM (H-LSTM) and Update-LSTM (U-LSTM) can be formulated as:
[Equation]
k + M i ∗ H (cid:48) k - 1 + b i ) , f k = σ ( W f ∗ X k + M f ∗ H (cid:48)(cid:48) k - 1 + b f ) , o k = σ ( W o ∗ X k + M o ∗ H (cid:48)
k {+- 
 + b o )+} , ˆ C k = tanh( W ˆ c ∗ X k + M ˆ c ∗ H (cid:48)(cid:48) k - 1 + b ˆ c ) ,
C k = f k (cid:12) C k - 1 + i k (cid:12) ˆ C k , H k = o k (cid:12) tanh( C k ) .
3. I MPLEMENTATION D ETAILS
Model Architectures: ConvLSTM (Xingjian et al., 2015; Stollenga et al., 2015) is a popular building block for spatio-temporal sequence forecasting problems.
In this paper, we focused on the recent state-of-the-art ConvLSTM-based architectures proposed by Byeon et al.
(2018): the context-aware video prediction model (ContextVP-4).
It can capture sufﬁcient spatio-temporal context for video prediction and achieves state-of-the-art results on KTH human actions and BAIR robot pushing dataset.
In this paper, the ConvLSTM block is replaced with LH-STM.
Additionally, we exploit two types of skip connections to avoid the problem of vanishing gradients (Hochreiter et al., 2001) and allow long-term prediction: (a) between previous and the current recurrent layers (blue dotted line in
Figure 3) and (b) across layers (green solid line in Figure 3).
See Appendix B for the effectiveness of both skip connections.
ContextVP-4 (Byeon et al., 2018) consists of 4 layers of ﬁve directional Parallel Multi-Dimensional
LSTM (PMD) units (Stollenga et al., 2015) along the h
+ , h − , w + , w − , and t − recurrence directions.
A PMD unit along the time direction ( t − ) is mathematically equivalent to a standard ConvLSTM (Equation 6).
By using LSTM connectivities across the temporal and spatial dimensions, each processing layer covers the entire context in a video.
Following Byeon et al.
(2018), we also included weighted blending, directional weight sharing (DWS), and two skip connections between the layers- 3 and 2 - 4 (see Figure 3).
See Appendix B for details of the network architectures.
Skip connection in a recurrent block (R-Skip): A skip connection is commonly used in LSTM layers (Melis et al., 2017; Merity et al., 2017; Wang et al., 2018a) for long-term gradient propagation.
Our proposed model concatenates outputs of the previous and current LSTM layers (see Figure 3).
Gated skip connection across layers (L-Skip): Unlike in (Byeon et al., 2018)
, We added the gated skip connection across layers.
In our model, the multiplicative gate is added to control the ﬂow of information across layers.
It stabilizes learning as depth increases.
A layer is formulated as:
[Equation]
where H l = X l · W Hl + b H and T l = σ ( X l · W Tl + b T ) .
X l is the input at the l th layer, T is a gate, and H l is the transformation of the input at the layer l .
Loss: The loss function to train the networks is the combination of L 1 -norm, L 2 -norm, and perceptual loss (Johnson et al., 2016; Zhang et al., 2018):
[Equation]
where Y and ˆ X are the target and the predicted frames, respectively.
λ is a weight for each function.
Lp -norm is deﬁned as L p ( Y, ˆ X ) =
||
Y − ˆ
X
|| p .
p = 1 for L 1 -norm, and p = 2 for
L 2 -norm.
The perceptual loss computes the cosine distance between the feature maps extracted from a VGG-16 network pre-trained on the ImageNet dataset (Simonyan & Zisserman, 2014) as
L pl ( y, ˆ x ) = 1 − 1 l (cid:80) l 1 h l × w l (cid:80) h l ,w l ( φ ( y ) l · φ (ˆ x ) l ) , where φ ( y ) l and φ (ˆ x ) l are the feature maps of the target and the predicted frames Y and ˆ X , respectively at the layer l .
The size of the feature map at layer l is h l × w l .
E XPERIMENTS
We evaluated our LH-STM model on the KTH human actions (Laptev et al., 2004) and BAIR actionfree robot pushing (Ebert et al., 2017) datasets.
Videos in the KTH human actions dataset consist of six human actions including walking, jogging, running, boxing, hand waving, and hand clapping.
The BAIR action-free robot pushing dataset includes randomly moving robotic arms, which push objects on a table.
The videos have a resolution of 64 × 64 pixels.
We show quantitative comparisons by computing frame-wise Structural Similarity (SSIM)
(Wang et al., 2004), Peak Signal-to-Noise
Ratio (PSNR), and Visual Information Fidelity (VIF) (Sheikh & Bovik, 2005) between the ground truth and generated video sequences.
4. KTH H UMAN A CTIONS D ATASET
We follow the experimental setup of Lee et al.
We evaluate our methods on 128 × 128 pixels (Table 2).
We train all models to predict the next 10 frames, given 10 prior ones.
For testing, all models recursively predict 40 frames.
Our Double LH-STM outperforms all competing methods on all metrics including the base model (ContextVP).
Our Double LH-STM model consistently outperforms other methods for the 40 predicted frames.
The improvement is larger for longer-term frame prediction into the future (40 frames).
Qualitative results can be found in Figure 8 and Appendix C (Figure 14 and Figure 15).
Our models are compared with FRNN (Oliu et al., 2018), PredRNN (Wang et al., 2017),
PredRNN++ (Wang et al., 2018a), E3D-LSTM
Wang et al., 2018b), VAE and deterministic models (Lee et al., 2018) 3 .
Our model outperforms all other methods.
4. S TATE H ISTORY VS I NPUT H ISTORY
The proposed LH-STM models use past hidden states.
Alternatively, input frames can be directly applied to LH-STM as illustrated in Figure 4.
We compared the performance of using input and state history with the same LH-STM models and the same model size in Table 1 and ??
(Double LH-STM w/ input history) on the KTH human actions dataset.
Figure 8 shows a qualitative comparison of using state history (Double LH-STM) and input history (Double, input history).
The results show the importance of using history states with LH-STM and that the state history is more suitable for videos.
[Figure]
History
[Table]
VIF
Input
Single 25.
[Table]
Double 26.
State
Single
[Table]
Double 28.
[Table]
[Figure] [Figure]
[Figure] [Figure]
4. L ONGER P REDICTION
To test the performance of longer-term prediction (more than 40 frames)
[Figure]
we additionally trained the
KTH human actions dataset with a resolution of 64 × 64 pixels.
Figure 5 and Figure 6 compares the performance of 80 frame prediction with Double LH-STM and two state-of-the-art models (SAVPVAE and SAVP-Deterministic (Lee et al., 2018)) on all actions and three actions only (hand waving, hand clapping, and boxing), respectively.
We separate these actions to show that the performance at longer frames are not due to the background copy as a human in the videos may disappear with other actions (running, jogging, and walking).
The experimental setup is the same as the experiments of a resolution 128 × 128 (Section 4.1).
From this comparison, we can observe that the PSNR and SSIM scores of Double LH-STM does not degrade over a long period of time.
Notably, the performance of our model at 80 frames is similar to that of the baseline at ≈ 30 frames.
Input
Predicted Frames (top: Ground truth) t =
Double LH-STM (ours)
Single LH-STM (ours)
Double, input history
ContextVP (Byeonetal.,2018)
SAVP-Det (Leeetal.,2018) SAVP-VAE (Leeetal.,2018)
[Table]
4. BAIR R OBOT P USHING D ATASET
Our experimental setup follows that of Lee et al.
All models are conditioned on ﬁve frames to predict the next ten frames 4 .
The models are tested to predict 25 frames.
We evaluated our
LH-STM models (Single and Double) and compared them with the base model (ContextVP), the
VAE [Table] and deterministic models (Lee et al., 2018) 3 in Table 3.
Our Double LH-STM achieved better
PSNR and SSIM scores than the deterministic and VAE models.
It can be seen from the per-frame [Table] prediction results (Figure 7), that the performance LH-STM is higher over time.
We found that this dataset contains diverse random motion types which makes the long-term prediction task challenging.
Qualitative results can be found in Appendix C (Figure 17 and Figure 18).
C ONCLUSION
In this paper [Table] we have introduced a new RNN structure, LH-STM, which incorporates long history states into the recurrent unit to learn longer range dependencies.
This module is combined with the
ContextVP architecture allowing context aware long-term video prediction.
The resulting model produces more realistic predictions for a longer period of time compared to the recent state-of-the-art methods.
The proposed approach can easily be extended to standard LSTM or Gated Recurrent Units (GRU) and applied to tasks that require a long-term memory trace, e [Table] g., language modeling or speech recognition.
A N ETWORK D ETAILS
Each layer of ContextVP-4 contains ﬁve Parallel Multi-Dimensional LSTM (PMD) units (see Figure 9).
The parameters of opposite directions are shared.
We followed the same network architecture as Byeon et al.
(2018) except for the number of neurons.
LH-STM (h+)
LH-STM (h-)
LH-STM (w+)
LH-STM (w-) LH-STM (t-)
LH-STM (h+)
LH-STM (h-)
LH-STM (w+)
LH-STM (w-) LH-STM (t-)
LH-STM (h+)
LH-STM (h-)
LH-STM (w+)
LH-STM (w-) LH-STM (t-)
LH-STM (h+)
LH-STM (h-)
LH-STM (w+)
LH-STM (w-)
LH-STM (t-)
W - B l e
[Figure]
O u t p u t
L a y e r prediction
W - B l e n d
i n g
Blending:
[Equation]
(9) where H is the output of the PMD units from h + , h − , w + , w − , and t − directions, and W is the weight matrix.
Directional Weight-Sharing: Weights and biases of the PMD units in opposite directions ( i.e. , the h + and h − directions, w + and w − directions) are shared.
We adjusted the number of neurons, so all models have similar model size ( ≈ 17M).
LH-STM
Double contains 64-64-128-128 neurons for the input size of 64 × 64 pixels and 64-128-128-128 for 128 × 128 .
LH-STM Single contains 128-128-128-128 neurons for the input size of 64 × 64 pixels and 64-128-128-256 for 128 × 128 .
The initial learning rate is 1 e − 4 found by grid search between 1 e − 3 and 1 e − 5 .
The learning rate is decayed every 10 epochs with the decay rate 0 .
Weights are initialized using the Xavier’s normalized initializer (Glorot & Bengio, 2010) and the recurrent states are all initialized to zero.
B A BLATION S TUDY
We performed a series of ablation studies to evaluate the importance of each component of our proposed model 1) RNN type (standard ConvLSTM, Single LH-STM, or Double LH-STM); 2) different losses ( L 1 , L 2 , and the perceptual loss); adding skip connections 3) in recurrent blocks; and 4) across layers (see Section 3.3 for details).
We evaluated performance on the KTH human actions dataset with a resolution of 64 × 64 pixels.
The weights for the loss, λ 1 , λ 2 , and λ pl , are tested between zero to one.
The results of this experiment are summarized in Table 4.
We found that performance improves by adding skip connections in recurrent blocks and across layers.
Double
LH-STM outperforms the standard ConvLSTM and Single LH-STM.
Combination of L 1 , L 2 , and
L pl also improves the performance (with λ 1 = 1 , λ 2 = 1 , λ pl = 1 ).
C Q UALITATIVE R ESULTS
shows prediction results on the KTH human actions dataset, and ?? presents the results on the BAIR action free robot pushing dataset.
[Table]
D R ETRAINING P RED RNN++
As mentioned in the paper
[Table]
we re-trained PredRNN++ on the KTH human actions dataset using the provided code by Wang et al.
(2018a)
( https://github.com/Yunbo426/predrnn-pp ).
We conducted grid hyper-parameter search for the learning rate and the patch size.
We found in the code that there is an option to divide the input frames into smaller patches.
The learning rate search was performed in the range between 1 e − 3 and 1 e − 4 .
The range of patch size was 1, 2, 4 for both frame resolutions ( 64 × 64 and 128 × 128 pixels).
The reported results in the paper are for a learning rate of 1 e − 4 and the patch size 1 (full resolution).
[Figure]
[Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure] [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]