U NSUPERVISED H IERARCHICAL V IDEO P REDICTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Much recent research has been devoted to video prediction and generation, but mostly for short-scale time horizons.
The hierarchical video prediction method by Villegas et al.
(2017) is an example of a state of the art method for long term video prediction.
However, their method has limited applicability in practical settings as it requires a ground truth pose (e.g., poses of joints of a human) at training time.
This paper presents a long term hierarchical video prediction model that does not have such a restriction.
We show that the network learns its own higherlevel structure (e.g., pose equivalent hidden variables) that works better in cases where the ground truth pose does not fully capture all of the information needed to predict the next frame.
This method gives sharper results than other video prediction methods which do not require a ground truth pose, and its efﬁciency is shown on the Humans 3.6M and Robot Pushing datasets.
I NTRODUCTION
It is hypothesized that learning to predict the future and the effect of their actions is an important quality for intelligent agents that interact with their environment.
This is a complicated task, as typical use-cases require predicting the outcome of interactions between the agent and objects, over multiple time-steps.
Concretely in this work we are looking at the task of predicting the pixels of future video frames given the ﬁrst few observed frames.
We also consider the action-conditional setting, in which we are given the action that the agent is taking and are tasked to predict the pixel-level outcome of that action in the future.
The method from Villegas et al.
(2017) can generate long term video predictions, but requires ground truth human pose annotations.
In this work we explore ways to generate videos using a hierarchical model without requiring a ground truth pose or other high-level structure annotations for each frame.
The method is hierarchical in the sense that it learns to generate a high level structure, then makes next frame predictions based on that structure.
R ELATED WORK
Patch-level prediction The video prediction problem has been initially studied at the patch level (Sutskever et al., 2009;
Michalski et al., 2014; Mittelman et al., 2014; Srivastava et al., 2015).
This work showed promising results on synthetic data (e.g., bounding balls), but did not scale to predicting higher-resolution videos.
Frame level prediction on realistic videos.
More recently, the video prediction problem has been formulated at the entire frame level.
Most of the recent work is based on the convolutional encoder/decoder framework.
Finn et al.
(2016) proposed a network that can perform next-level video frame prediction by explicitly predicting movement.
For each pixel in the previous frame, the network outputs a distribution over locations that pixel is predicted to move. The movements are averaged to get the ﬁnal prediction. The network is trained end to end to minimize L2 loss.
Mathieu et al.
(2016) proposed adversarial training with multiscale convolutional networks to generate sharper pixel-level predictions in comparison to conventional L2 loss.
Villegas et al.
(2017) proposed a network that decomposes motion and content in video prediction and showed improved performance over Mathieu et al.
Lotter et al.
(2017) proposed a deep predictive coding network where each layer learns to predict the lower-level difference between the future frame and current frame.
As an alternative approach to convolutional encoder/decoder networks, Kalchbrenner et al.
(2016) proposed an auto-regressive generation scheme for improved prediction performance.
Despite promise, all these work have not been demonstrated for long-term prediction on high-resolution natural videos beyond (cid:25) 20 frames.
Long-term prediction.
Oh et al.
(2015) proposed action-conditional convolutional encoderdecoder architecture that has demonstrated impressive long-term prediction performance on video games (e.g., Atari games),
but it has not been applied for predicting challenging real videos.
2. H IERARCHICAL VIDEO PREDICTION (V ILLEGAS ET AL ., 2017)
Villegas et al.
(2017) demonstrated a long-term prediction method using hierarchical prediction where the groundtruth human pose is assumed to be given as supervision.
Our method is based off of this one, so we describe it in detail here.
2.1. I NFERENCE AND ARCHITECTURE
To generate the image at timestep t , the following procedure is used.
First, a convolutional neural network encoder generates an embedding vector from the previous groundtruth image: e t (cid:0) 1 =
CNN ( img t (cid:0) 1 ) .
This encoding represents the pose of a person.
Next, a multilayer LSTM predictor network predicts what that encoding will be in a future time step.
For some number of context frames, the predictor makes its prediction based off of the encoding from the ground truth image.
After the predictor network has enough context, it makes its predictions based off of its previous predictions (Fig. 1 provides a helpful visual).
For example if there are C context frames, the following is used to generate the encoding at step t .
[Equation]
H t is the hidden state of the LSTM at timestep t .
Note that only the encoding of the context frames are used, not the subsequent frames. Similar to e t in the above, p t reprsents the predicted pose.
Once p t is obtained, a visual analogy network (VAN) (Reed et al., 2015) is used to generate the corresponding image at time t .
The VAN applies the transformation that occurred between two images to a given query image.
In this case, we want the ﬁrst frame of the video to be transformed in the same way as the encoding was transformed from the ﬁrst to t -th time step.
The VAN does this by mapping images to a space where analogies can be represented by additions and subtractions, and then mapping the result back to image space.
The equation used to obtain the predicted image at timestep t using the VAN is d img t =
V AN ( e 1 ; p t ; img 1 ) , where the VAN is deﬁned as
[Equation]
Where g is a hardcoded function to transform the pose into a heatmap image.
2.1. T RAINING
The disadvantage of this method is that the training relies on ground truth pose annotations.
The encoder is trained to produce the pose given the image.
The predictor is trained to predict that pose into the future.
The VAN is trained to generate the image given the pose.
P ROPOSED METHOD
Our method uses a similar network architecture to Villegas et al.
but we present ways of training the network that do not require a ground truth pose.
In our method, e t and p t represent the network’s own higher-level structure (e.g., pose equivalent hidden variables) which the network learns as it is trained.
e t and p t have the same dimensionality, which is much lower than the image, usually less than 256 dimensions.
In our case, there isn’t a straightforward way to transform the encoding into a feature map so the part of the VAN that maps the encoding is a fully connected network instead of a convolutional neural network.
As a result, the weights are not shared between the fully connected network which processes the encoding, and the conv-net which processes the image.
The equation for the VAN becomes
[Equation]
Note that f enc is a fully connected network, and f img is a conv net.
f dec is a deconv network.
3. T RAINING
There are several ways these networks can be trained.
In Villegas et al.
(2017), they are each trained separately with the ground truth human pose.
In this work, we explore alternative ways of training these networks in the absence of any ground truth pose or other high-level structure annotations. We use the same procedure as Villegas et al.
(2017) at inference time.
3.1. E ND TO E ND
One option is to connect the networks the same way as in inference time and train them end to end (E2E).
In this method, the L2 loss of of the predicted image is optimized:
[Equation]
There are no constraints on what kind of encoding the encoder produces, or what kind of predictions the predictor makes.
Because of how the networks are connected, the encoder will produce an encoding whose future state is easily predicted by the predictor.
Likewise, the predictor will make predictions which the VAN can use to produce images which are similar to the ground truth.
The encoder and predictor will not have to represent information that is present in the ﬁrst ground truth frame, since the VAN will have access to the ﬁrst frame.
The size of e t and p t is a hyper parameter of this approach.
Figure 1 represents a diagram of this method.
The predictor predicts the subsequent encodings, which the VAN uses to produce the level predictions.
The average of the losses is minimized.
This is also the conﬁguration of every at inference time, even if the predictor and VAN are trained separately.
3.1. E NCODER P REDICTOR WITH E NCODER VAN
An alternative way to train the combined network is to explicitly train the encoder so that e t is easy to predict into the future, and so that the VAN can use e t to produce the next frame.
We call this method Encoder Predictor with Encoder VAN, or EPEV.
The encoder and predictor are trained together so the e t is easy to predict and the predictor predicts that encoding into the future.
To accomplish this, the difference between e t and p t , L 2 ( e t ; p t ) is minimized.
The encoder is also trained with the VAN so the VAN can use e t to produce the image and so that the encoder generates an informative encoding.
This is done by minimizing the loss of the VAN given the encoder output:
L 2 ( d img e t ; img t ) where d img e t = V AN ( e 1 ; e t ; img 1 ) .
The network is trained to minimize the sum of these two losses: min( P Tt =1 L 2 ( d img e t ; img t )
+ (cid:11)L 2 ( e t ; p t )) , where (cid:11) is a hyper-parameter that controls the degree to which the e t will be easy to predict vs. informative enough so the VAN can produce a good image.
See ﬁgure 2 for a diagram of the encoder and predictor trained together and ﬁgure 3 for the encoder and VAN trained together.
where the encoder and predictor are trained together.
The encoder is trained to produce an encoding that is easy to predict, and the predictor is trained to predict that encoding into the future.
The average of the losses is minimized.
where the encoder and VAN are trained together.
The encoder is trained to produce an encoding that is informative to the VAN, while the VAN is trained to output the image given the encoding.
The average of the losses is minimized.
This is similar to an autoencoder.
Separate gradient descent procedures (or optimizers, in TensorFlow parlance) could be used to minimize L 2 ( d img e t ; img t ) and L 2 ( e t ; p t ) , but we ﬁnd that minimizing the sum works better experimentally.
With this method, the predictor will predict the encoder outputs in future time steps, and the VAN will use the encoder output to produce the frame.
3.1. E2E WITH POSE
The end to end approach can also be augmented if the dataset has information about the ground truth pose or any other high-level frame annotations.
In this method, the ﬁrst part of the e t and p t is optimized to represent the pose, and the rest of e t and p t is trained the same way as in the
E2E approach.
At each training step a separate optimizer minimizes each loss.
In this method, we can think of e t and p t as the concatenation of two vectors, one representing the pose, and another representing other information the network can represent.
If e t =
[ e pose t ; e remaining t ] and p t =
[ p pose t ; p remaining t ] , the following losses are minimized:
[Table]
The loss representing how well the encoder infers the pose: min( P Tt =1 L 2 ( e pose t ; pose t )) .
The loss representing how well the predictor predicts the pose: min( P Tt =1 L 2 ( p pose t ; pose t )) .
The end to end loss: min( P Tt =1 L 2 ( d img t ; img t )) .
These losses are minimized with separate optimizers in this method.
Minimizing the end to end loss ensures that the VAN will learn to use the pose provided by the predictor network, and that the encoder and predictor will learn to produce additional information besides the pose that is useful to the VAN.
3.1. I NDIVIDUAL
In order to compare to a baseline we also implemented the method where each of the networks are trained individually as in Villegas et al.
The main difference between this method and Villegas et al.
(2017) is we do not use a adversarial loss (Goodfellow et al., 2014).
See section 5 for a discussion of how an adversarial loss could be added to our method.
E XPERIMENTS
These methods were tested on two different datasets, the Robot Push dataset (Finn et al., 2016) and the Humans 3.6M dataset (Ionescu et al., 2014; Catalin Ionescu, 2011).
Videos of the results of our method are available by visiting the following URL: https://goo.gl/WA8uxc .
The EPEV method works best experimentally if (cid:11) starts small, around 1e-7, and is gradually increased to around .1 during training.
Then the encoder will ﬁrst be optimized to produce an informative encoding, then gradually optimized to also make that encoding easy to predict.
R OBOT PUSH DATASET
This dataset contains videos of a robot arm pushing objects on a table.
The current joint angles and the location of the end effector are given and we use these as the pose, for the methods which require it.
The action the robot arm is taking is fed into the predictor.
Each of the methods considered was given two frames of context, and then trained to predictsubsequent frames.
An encoding size of 16 was used for the E2E method.
The size of the pose is 12 so the encoding size of the INDIVIDUAL method is 12.
The other methods used an encoding size of 32.
We split the dataset into training, validation and test randomly.
We used 64x64 images, and the same frame rate as the original dataset.
Results from our test set are shown here.
We do not test on novel objects as in Finn et al.
We hypothesized that the methods where the network could learn its own pose equivalent would predict objects better than the INDIVIDUAL method.
To test this we manually compared The E2E and EPEV methods to the INDIVIDUAL method and evaluated which object prediction most closely matched the ground truth.
We evaluated 40 videos where objects move, the results are in Table 1.
In the INDIVIDUAL method, the predictor network can only produce the pose, so the VAN has to infer how the objects will move based on the start and end state of the arm.
We were surprised by how well the VAN could infer this.
However, from examining the videos, the EPEV method had better object predictions than the INDIVIDUAL which supports our hypothesis.
The magniﬁed part of ground truth frame 19 in ﬁgure 4 shows that the robot arm pushed the yellow object.
The EPEV and E2E methods correctly predict this, but in the INDIVIDUAL method, the robot arm covers up the yellow object instead of moving it.
More such analysis is in the appendix.
The average Peak Signal to Noise Ratio (PSNR) of the different methods we introduce are similar on this dataset.
The model from Finn et al.
(2016) gets a better PSNR than ours on this dataset.
The movement in this dataset can easily be represented by the movement of pixels, and it is relatively deterministic.
So the model from Finn et al.
(2016) which explicitly predicts the movement of objects and minimizes the L2 loss directly works well here.
4. L ONG TERM PREDICTION ON A TOY DATASET
To conﬁrm our claim that our method works well for long term predictions, we trained our method on a toy task with known factors of variation. We used a dataset with a generated shape that bounces around the image and changes size deterministically. We trained the EPEV method and the CDNA method in Finn et al.
(2016) to predict 16 frames given the ﬁrst 3 frames as context.
We don’t show the E2E method since it usually predicts blurrier images than the EPEV method.
Both methods are evaluated on predicting about 1k frames.
We added noise to the LSTM states of the predictor network during training to help predict further.
Results form a held out test set are shown here.
From visually inspecting the results of both methods, we found that when the CDNA fails, the shape disappears entirely, and when the EPEV method fails, the shape changes color.
To quantitativly evaluate both methods, we used a script to measure if a shape was present frames 1012 toand if that shape has the appropriate color.
See table 2 for the results averaged over 1k runs.
The
CDNA method predicts a shape with the correct color about 25% of the time, and the EPEV method predicts a shape with the correct color about 97% of the time.
The EPEV method sometimes fails by predicting the shape in the same location from frame to frame.
This doesn’t happen very often, as the reader can conﬁrm by examining the randomly sampled predictions in the appendix.
[Table]
Our method was also tested on the Humans 3.6M Dataset.
Only the E2E and EPEV methods were tested here, since Villegas et al.
(2017) has already shown the results using the ground truth pose.
We used subjects 1,5,6,7 and 8 for training, subject 9 for validation.
Subject 11 results are reported in this paper for testing.
We used 64 by 64 images.
We subsampled the dataset to 6.25 frames per second.
We trained the methods to predict 32 frames and the results in this paper show predicting 64 frames.
Each method is given the ﬁrst 5 frames as context frames.
So the in these images, the model predicts about 10 seconds into the future from .8 seconds of context.
We used an encoding size of 32 for the E2E method and a encoding size of 64 for the EPEV method on this dataset.
We compare our method to the CDNA method in Finn et al.
(2016) in ﬁgure 6.
We trained this method on the same data as ours and found the best learning rate for this method. Finn et al.
(2016) as the baseline.
This example is cherry picked to show results when there is signiﬁcant movement in the ground truth. See the appendix for non cherry picked results. The contrast of these images was increased to make the humans easier to see.
In CDNA from Finn et al. , the person disappeared part way through the prediction. The EPEV method, produced relatively sharp predictions up until frame 42, and a blurry human prediction at frame 63. From visually inspecting the images we found that in images where there isn’t signiﬁcant movement in the ﬁrst 5 frames of the ground truth it’s hard to tell the difference between our method and CDNA since both methods predict an image similar to the early ground truth frames.
However, when there is signiﬁcant movement in the ﬁrst 5 ground truth frames, the predictions from EPEV are sharper further into the future than CDNA. See the appendix for images where there is signiﬁcant movement in the ﬁrst 5 ground truth frames so the methods can be compared. We also collected results from the E2E method, but those blur out very quickly and are shown in the appendix. The CDNA method from Finn et al.
(2016) produces blurry images since it’s trained to minimize L loss directly (Finn et al., 2016). In the EPEV method, the predictor and VAN are trained separately. This prevents the VAN from learning to produce blurry images when the predictor isn’t conﬁdent.
The predictions will be sharp as long as the predictor network predicts a valid encoding.
We also compare our method to Villegas et al.
This method gives sharper results than ours.
We think that this is becuase Villegas et al.
(2017) uses a adversarial loss (Goodfellow et al., 2014) and since nothing besides the human is moving in this dataset, the pose works well as a high level structure. 4.3. P ERSON D ETECTOR E VALUATION
We propose to compare the methods quantitatively by considering whether the generated videos contain a recognizable person.
To do this in an automated fashion, for each of the generated frames, we ran a MobileNet (Howard et al., 2017) object detection model pretrained on the MS-COCO (Lin et al., 2014) dataset. We recorded how conﬁdent the detector was that a person (one of the MSCOCO labels) is in the image. We call this the “person score” (its value ranges from 0 to 1, with a higher score corresponding to a higher conﬁdence level). The results on each frame averaged over 1k runs are shown in Figure 7.
The person score on the ground truth frames is about 0.4.
This is likely due to the mismatch between the training set images of the model (the MS-COCO dataset images are very different in terms of image statistics compared to the Humans 3.6M data).
The person score is 0.26 on average for the images generated by the EPEV method, and 0.18 for CDNA from Finn et al.
The person score degrades very rapidly in the ﬁrst 8 frames of CDNA, but degrades more slowly in the EPEV method.
The person score of the EPEV method on frame 63 is about the same as on frameof CDNA. This conﬁrms our visual analysis that the EPEV method produces clearer predictions further into the future. The EPEV method was only trained to predict 32 frames into the future but there is no signiﬁcant drop in the person score at frame 32, showing that the EPEV method generalizes well to predicting longer sequences. 4.3. H UMAN E VALUATION
We also used a service similar to Mechanical Turk to collect comparisons of 1,000 generated videos from our EPEV method and the CDNA baseline.
The task showed videos generated by the two methods side by side and asked raters to conﬁrm whether one of the videos is more realistic.
The workers rated the EPEV method as more realistic 53.6% of the time, the CDNA method as more realistic 11.1% of the time and the videos as being about the same 35.3% of the time.
The high number of “same” responses could be because of it being difﬁcult to tell the difference between the methods when there is little movement.
C ONCLUSION AND F UTURE W ORK
On datasets where the pose does not capture all of the information needed to predict future frames, letting the network deﬁne its own high level structure in addition to the pose is an improvement upon a Villegas et al. The EPEV method generates sharper images than Finn et al.
(2016) on non deterministic datasets, and can generate further into the future on our toy dataset.
To ﬁx the problem of the EPEV method sometimes generating blurry images, an adversarial loss between the predictor and encoder would likely help with potentially uncertain scenarios
Appendices A I MPLEMENTATION DETAILS
The convolutional neural net encoder is a VGG-16 network (Simonyan & Zisserman, 2014).
In the
EPEV method, the encoder needs to be pre-trained on Imagenet (Deng et al., 2009).
If
The predictor network is a 20 layer LSTM, while the analogy network uses the deep analogy transformation described in Reed et al.
The VAN outputs a mask which controls whether to use its own output, or the pixels of the ﬁrst frame.
This allows the network to focus on learning the changing parts of the image instead of the background.
The end effector orientation is converted to quaternions (Bousmalis et al., 2016) in order to calculate the loss if the pose is used.
Layer normalization (Lei Ba et al., 2016) is used between every other layer in the VAN and predictor networks.
We used dropout (Srivastava et al., 2014) on the encoder and VAN to prevent overﬁtting.
The network overﬁts less in the I NDIVIDUAL method, so we used less dropout.
B H UMAN EVALUATION DETAILS
Each video in the comparison is generated from the same starting sequence.
The side that the EPEV method and the CDNA method are displayed on is changed randomly.
C EPEV METHOD WITH ENCODER OUTPUT FED TO VAN
To see what the encoder has learned in the EPEV method, we can obtain results from the visual analogy network given the input from the encoder.
The encoder is given the ground truth image.
The results are shown in ﬁgure 9.
The results show that the encoder encodes where the person is in the image, as well as the orientation of the arms, legs and head to some extent.
The results are not as good as one would expect from an autoencoder, since the encoder has the constraint that the encoding also has to be easy to predict.
We trained all of the methods including Finn et al. for 3 million steps using async SGD, across 32 worker machines. We used a minibatch size of 8 sequences in each step.
The minibatch size could be so small because there were multiple frames per sequence. In methods with multiple optimizers, a step is deﬁned as running each optimizer once. The hyperparameters are optimized separately for both datasets on a validation set. We used the best learning rates for each method.
E M ORE RESULTS ON SHAPES DATAEST .
[Table]
[Table] [Figure] [Table]
[Figure]