LMVP: Video Predictor with Leaked Motion Information
Abstract
We propose a Leaked Motion Video Predictor (LMVP) to predict future frames by capturing the spatial and temporal dependencies from given inputs.
The motion is modeled by a newly proposed component, motion guider, which plays the role of both learner and teacher.
Speciﬁcally, it learns the temporal features from real data and guides the generator to predict future frames.
The spatial consistency in video is modeled by an adaptive ﬁltering network.
To further ensure the spatio-temporal consistency of the prediction, a discriminator is also adopted to distinguish the real and generated frames.
Further, the discriminator leaks information to the motion guider and the generator to help the learning of motion.
The proposed LMVP can effectively learn the static and temporal features in videos without the need for human labeling.
Experiments on synthetic and real data demonstrate that LMVP can yield state-of-the-art results.
Introduction
Video combines structured spatial and temporal information in high dimensions.
The strong spatiotemporal dependencies among consecutive frames in video greatly increases the difﬁculty of modeling.
For instance, it is challenging to effectively separate moving objects from background and predict a plausible future movement of the former [2, 4, 11, 14, 15, 21].
Though video is large in size and complex to model, video prediction is a task that can leverage the extensive online video data without the need of human labeling.
Learning a good video predictor is an essential step toward understanding spatio-temporal modeling.
These concepts can also be applied to various tasks, like weather forecasting, trafﬁc-ﬂow prediction, and disease control [16, 18, 17].
The recurrent neural network (RNN) is a widely used framework for spatio-temporal modeling.
In most existing works, motion is estimated by the subtraction of two consecutive frames and the background is encoded by a convolutional neural network (CNN) [1, 9, 11, 12].
The CNN ensures spatial consistency, while temporal consistency is considered by the recurrent units, encouraging motion to smoothly progress through time.
However, information in two consecutive frames is usually insufﬁcient to learn the dynamics.
Using 3D convolution to generate future frames can avoid these problems [13, 14], although generating videos by 3D-convolution usually lacks sharpness.
We propose a Leaked Motion Video Predictor (LMVP) for robust future-frame prediction.
LMVP generates the prediction in an adversarial framework: we use a generative network to predict next video frames, and a discriminative network to judge the generated video clips.
For the motion part, we propose to learn the dynamics by introducing a motion guider, connecting the generator and the discriminator.
The motion guider learns the motion feature through training on real video clips, and guides the prediction process by providing possible motion features.
At the same time, in contrast with estimating motions by subtracting two consecutive frames, we allow the discriminator to leak high-level extracted dynamic features to the motion guider to further help the prediction.
Such dynamic features provide more informative guidance about dynamics to the generator.
The spatial dependencies of video are imposed by a convolutional ﬁlter network conditioned on the current frame.
This idea is inspired by a conventional signal processing technique named adaptive ﬁlter, which can increase the ﬂexibility of the neural network [5].
It is assumed that, each pixel of the predicted frame is a nonlinear function of the neighborhood pixels of the current one, where the nonlinear function is implemented via LMVP as a deep neural network.
Models
The video frames are represented as x ∈ R T × H × W × C , where T is the total number of frames,
H is the frame height, W is the width and C is the channel number.
Given the ﬁrst T 0 ( T 0 < T ) frames, the task is to predict the following T − T 0 frames.
x t and ˆ x t represent for real and predicted video frame at time t , respectively.
The model framework is given in Figure 1.
It mainly contains a generator G , a motion guider M , and a discriminator D .
D distinguishes between the real and predicted video clips.
M learns the temporal dependencies among the video through the features leaked from D , and generator G uses the output of motion guider M to predict the next frame based on the current.
2.1 Leaked Features from D as Motion Signals
The discriminator D (shown in top of Figure 1) is designed as both a discriminator and a motion feature extractor.
The bottom layers of D is a feature extractor F ( · ; θ F ) , followed by a classiﬁer between real/fake samples, parameterized by θ C .
Mathematically, given input video clips x t − c : t , we have D ( x t − c : t ; θ D ) =
CNN ( F ( x t − c : t ; θ F ); θ C ) , where θ D = ( θ F , θ C ) .
The extracted motion feature from x t − c : t is denoted f dt = F ( x t − c : t ; θ F ) , which is the input of motion guider M .
The feature extractor F is implemented as a convolutional network.
The output f dt is expected to capture motion from x t − c : t .
The difference between f d t +1 and f d t is treated as the dynamic motion feature between two consecutive frames, which is denoted as m t .
In contrast of the direct subtraction of two consecutive frames [11], our dynamic motion feature is extracted from two consecutive video clips of length c .
Since previous video frames are also included, it can still give reasonable output even if the model fails at previous time step.
The discriminator loss can be written as
[Equation]
2.2 Learning and teaching game of M
To utilize the leaked motion information f dt from D , we introduce a motion guider module M , which is inspired by the leaky GAN model [3] for text generation task.
The structure of M is displayed in the green dotted box in Figure 1.
M has a recurrent structure that takes the extracted motion feature f dt as input at each time step t , and outputs a predicted motion feature ˆ m t . Speciﬁcally, the motion guider plays two roles in the model: learner and teacher.
As a learner, M learns the motion in video via leaked feature from D from real video.
At time step t , M receives the leaked information f
dt exacted by D , and predicts the dynamic motion feature between the video clip of time t and t + 1 , by forcing ˆ m t =
M ( f dt ; θ M ) close to m t = f dt +1 − f dt .
Denoting the parameters of the motion guider as θ M , the learner loss function can be written as
[Equation]
Note that only real video samples are used to update θ M .
The superscript L means “Leaner”.
As a teacher, M serves as a guider by providing predicted dynamic motion features to G .
During this step, θ M is ﬁxed while the generator is updated under the guidance of M .
Given the leaked features ˆ f dt = F (ˆ x t − c : t ; θ F ) of predicted data ˆ x t − c : t at time t , the output ˆ m t =
M ( ˆ f dt ; θ M ) serves as an input to the generator to predict the next frame ˆ x t +1 .
This is detailed in the next section.
Since the dynamic motion feature is extracted from a real video clip instead of a single frame, M is robust against fail predictions at previous time step, i.e., even if the previous predicted frame diverges from the ground truth.
2.3 Generating the next frame under guide from M
The structure of the generator is shown at the bottom of Figure 1.
It contains a spatial feature encoder, a temporal feature encoder, and a ﬁlter network.
The spatial feature encoder is designed to learn the static background structure f st , while temporal feature f mt are computed from dynamic motion feature ˆ m t .
Note that only predicted samples have their motion guider output
ˆ m t ﬂow back to the generator.
The spatio-temporal features f st and f mt are concatenated and further fed into the ﬁlter network.
The next frame is predicted by applying the generated adaptive ﬁlter from [ f st , f mt ] on the current frame.
This technique is also known as visual transformation in [14].
As mentioned in Section 2.2, M is updated during the learning step while ﬁxed during teaching step.
When generating the next frame ˆ x t +1 in the teaching step, M is ﬁxed and outputs the motion guide ˆ m t .
Speciﬁcally, to ensure G generates the next frames following the guidance of M , the dynamic motion feature between the generated video clips at time t + 1 and t , denoted as ˆ f dt +1 − ˆ f dt should be close to the predicted dynamic motion feature ˆ m t taught by the guider M .
Then, the generator is updated by minimizing the following loss function:
[Equation]
where θ G includes all parameters in the generator G .
Note that ˆ x t and ˆ f dt are the predicted output and leaked motion feature from time step t − 1 , respectively.
The superscript T in L TM indicates M as a “Teacher”.
The total loss function for the generator is
[Equation]
L recons is the reconstruction loss function of x t +1 , ˆ x t +1 , including a pixel-wise cross-entropy/MSE loss and the gradient difference loss (GDL) [8].
The whole model is updated iteratively for each component.
A pseudo-algorithm is given in Alg.
1 in the Appendix.
The discriminator loss (1) is ﬁrst evaluated and θ D is updated.
Then the motion guider parameters θ M are updated using only real samples of x .
The generated parameters θ G is updated using loss function (4).
In practice, Adam [6] is used to perform the gradient descent optimization.
Experiments
BCE SSIM PSNR ConvLSTM [20] 8 . 96 × 10 −0 .10 .
FPM [10] 8 .
{+33 × 10 −- - DFN [5] 6 . 
 × 10 −
 LMVP 6 .+}
13 × 10 −0 .19 .
Moving MNIST :
Each video in the Moving
MNIST dataset [10] has 20 frames in total, with two handwritten digits bouncing inside a 64 ×patch.
Given 10 frames, the task is to predict the motion of the digits of the following 10 frames.
We follow the same training and testing procedure as [10].
Evaluation metrics include Binary Cross Entropy (BCE), Peak Signal to Noise
Ratio (PSNR), and Structural Similarity Index
Measure (SSIM) [19] between the ground truth x and the prediction ˆ x .
Small values of BCE or large values of SSIM and PSNR indicate good prediction results.
In this task, we need to keep the digit shape the same across time (spatial consistency) while giving them reasonable movements (temporal consistency).
Table 1 gives the comparison of LMVP and baseline models.
The BCE of LMVP achieves 0 .
061 per pixel over 10 frames, which is better than state-of-the-art models [20, 10, 5, 11].
The predictions from LMVP and DFN [5] are shown in Figure 4.
Input is given in the ﬁrst row, followed by ground truth of the output, and results from DFN model and our LMVP model.
To prove that our model has consistently good result in to the future, Figure 5 in the Appendix gives the SSIM and PSNR comparison over ( t = T 0 + 1 , · · · , T ) .
LMVP achieves higher SSIM and PSNR scores than other baseline models through all time steps.
[Table]
Highway Drive : The dataset contains videos were collected from a car-mounted camera during car driving on a highway.
The videos contain rich temporal dynamics, including both selfmotion of the car and the motion of other objects in the scene [7].
Following the setting used in [5], we split the approximately 20 , 000 frames of the 30 -minutes video into a training set of 16 , 000 frames and a test set of 4 , 000 frames.
Each frame is of size 64 × 64 .
The task is to predict three frames in the future given the past three.
The prediction results are compared in Table 2 and two samples from the test set are selected in
Figure 3.
In the prediction results of DFN, the rail of the guidepost becomes curving.
However, in the prediction results of LMVP, the rail keeps straight in the ﬁrst and second predicted frames.
To help the visual comparison, this part has been highlighted by a red circle.
Conclusion
We have proposed the Leaked Motion Video Predictor (LMVP) to handle the spatio-temporal consistency in video prediction.
For the dynamics in video, the motion guider learns motion features from real data and guides the prediction.
Since the motion guider learns features from video sequences, it is more robust compared to using only single frames as input.
For structures of the background, the adaptive ﬁlter generates input-aware ﬁlters when predicting the next frame, ensuring spatial consistency.
Further, A discriminator is adopted to further improve the prediction result.
On both synthetic and real datasets, LMVP shows superior results over the state-of-the-art approaches.
A Model training
The model is ﬁrst pre-trained by iteratively updating the parameters of D and G .
In each iteration, we ﬁrst update θ D by minimizing the loss L dis in Equation (1); then, θ F , θ M , and θ G are jointly updated by minimizing the loss L gen in Equation (4) with γ = 0 .
We found that the above pre-training technique can empirically stabilize the generation process and learn useful leaked information from discriminator.
In the main algorithm loop, D , M , and G are trained iteratively.
The algorithm outline is given in Algorithm 1.
Firstly, θ D is updated according to discriminator loss L dis while θ M and θ G are kept ﬁxed.
Secondly, L M ( θ M ) deﬁned in Equation (2) is evaluated to update θ M while θ D and θ G remain unchanged.
The third step is to update θ G by minimizing loss L gen in Equation (4) with γ > 0 .
Note that, in both pre-train and main algorithm loop, all the initial hidden states in recurrent architecture are set to zero.
The gradient is updated by Adam [6].
1: Input : Training videos V = { x 1: T } .
2: Output : Parameters θ G , θ D , θ F and θ M . 3:
Initialize the discriminator, generator, and motion guider with random weights.
Initial hidden states in the model are set to zeros.
Pre-train θ D using videos in training dataset as positive samples and output from generator as negative samples.
: Pre-train θ G , θ F and θ M
leaked features
θ F according to
L recons .
: Repeat the two pre-train steps iteratively
convergence.
do
Sample a mini-batch of real video clips { x
and generate fake video clips
x
according
using real data 13:
Compute motion guide (learner) loss L LM in Equation (2) using m t and f dt computed above.
Fix θ G and θ D , update θ M by ∂ L LM ∂ θ M .
15: // Train generator 16:
Compute the prediction ˆ x t +1 =
G ( x t , m t ; θ G ) and loss L TM in (3) using ˆ m t and ˆ f dt from generated samples in Equation (4).
Fix θ M and θ D , update θ G by ∂ L gen ∂ θ G .
18: end for
B Experiment Result on Moving MNIST Dataset
The predictions generated by a LMVP model and a DFN model [5] are displayed in Figure 4.
Frames in the ﬁrst line are input sequences and ground truth sequences.
Frames generated by a DFN model and our LMVP model are shown in the second and the third line, respectively.
Visually, the prediction of LMVP is better than DFN.
This is further conﬁrmed quantitatively in Table 3.
[Table]
To demonstrate it more clearly, Figure 5 displays the prediction evaluation of DFN and our model over different time step ( t = T 0 + 1 , · · · , T ) .
LMVP achieves higher SSIM and PSNR scores than other baseline models through all time steps.
      7LPH6WHSV  
 
 
 66 ,
[Figure]
')1/09     
[Figure]

[Figure]
 
[Figure]
')1/09
The proposed model gets higher SSIM and PSNR scores than baselines through all time steps.