Stochastic Video Prediction with Perceptual Loss
Abstract
Predicting future states is a challenging process in the decision-making system because of its inherently uncertain nature.
Most works in this literature are based on deep generative networks such as variational autoencoder which uses pixelwise reconstruction in their loss functions.
Predicting the future with pixel-wise reconstruction could fail to capture the full distribution of high-level representations and result in inaccurate and blurred predictions.
In this paper, we propose stochastic video generation with perceptual loss (SVG-PL) to improve uncertainty and blurred area in future prediction.
The proposed model combines perceptual loss function and pixel-wise loss function for image reconstruction and future state predictions.
The model is built on a variational autoencoder to reduce high dimensionality to latent variable to capture both spatial information and temporal dynamics of future prediction.
We show that utilization of perceptual loss on video prediction improves reconstruction ability and result in clear predictions.
Improvements in video prediction could further help the decision-making process in multiple downstream applications.
Introduction
Future prediction is an important and challenging process in a decision-making process.
Learning full distribution of spatial dynamics and temporal dynamics could result accurate prediction of future states which could be utilized in multiple downstream applications such as pedestrian prediction [1], autonomous driving [2], sport games [3], networks [4], anomaly detection [5] and model-based reinforcement learning [6].
One of the challenging issues in video prediction research ﬁeld is inherently uncertain model dynamics.
For example, when a bouncing random ball hits a wall, it is uncertain that which direction and velocity the ball would bounce in future states.
Video prediction in this scenario is to predict features of the target object in future states such as direction, position, and velocity of the bouncing ball in continuous images where past states are given as input.
Recent researches in pixel-based video prediction shows noticeable advances in stochastic state prediction [6].
The application of deep generative model outperformed research approaches with traditional methods because of their improved ability to infer temporal dynamics of past and future and learn spatial dynamics of high dimensional state environments [7].
Video is a collection of consecutive static images which contains temporal information and motion patterns.
Prediction on video consists with two components which are reconstruction of spatial information and inference of temporal dynamics.
Disentangle these two and ﬁnd spatio-temporal correlation is an important research area [8].
Prediction in future state could be affected by external interventions such as occlusion, change in camera positions, direction of lights, background objects, and many other factors.
The future state is multimodal and contains uncertainty by nature.
In deterministic approach, each possible future state is learned to have same averaged probability and visually shown as a blurred area especially if time range is in long horizon.
Their deterministic model has limited application in real-world downstream applications which contains inherent uncertainty.
In stochastic approach, recent studies applied variational autoencoder [9] to reduce high dimensionality to latent variables and to better incorporate uncertainty.
Variational autoencoder (VAE) [9] has been employed for stochastic prediction problems.
Walker et al.
[10] proposed VAE conditioned on images to predict dense trajectories from pixels.
Xue et al.
[11] proposed cross convolution network with VAE to learn future frames with single input image.
Babaeizadeh et al.
[12] proposed stochastic variational video prediction based on variational inference with real-world videos.
Denton and Fergus [6] proposed stochastic video generation with learned prior.
Villegas et al.
[13] extended SVG while minimizing inductive bias and maximizing network capacities.
Previous approaches highlighted pixelwise reconstruction of future states in stochastic model.
Our approach differs from these in that per-pixel reconstruction is extended with perceptual loss function to extract high-level features.
Perceptual loss [14] has been introduced to extract and differentiate high-level feature representations from pretrained perceptual networks.
Ledig et al.
[15] and Sajjadi et al.
[16] utilized perceptual loss and adversarial loss to generate super-resolution single image.
Zhu et al.
[17] learns natural image manifold directly from data.
Reda et al.
[18] proposed spatially-displaced convolution to learn a motion vector and a kernel to synthesize images.
Hou et al.
[19] proposed perceptual loss for VAE training to extract natural visual appearance and perceptual quality of a image.
Pihlgren et al.
[20] applied perceptual loss to improve image embeddings in static settings.
These approaches improved reconstruction quality of images by utilizing high-level features from perceptual loss, however, limited research is done on video prediction with uncertainty.
Our approach utilizes both pixel-wise loss function and perceptual loss function to predict future states in continuous time frame.
In this paper, stochastic video generation with perceptual loss (SVG-PL) is proposed.
A pixelwise loss and a perceptual loss are both used to predict future states in stochastic approach to improve blurriness and uncertainty of the video prediction.
Especially high-level representations are extracted from pretrained perceptual network from natural image dataset.
This approach could improve reconstruction ability and result in clear predictions.
Also state prediction with high-level representations could help learning control policy in model-based reinforcement learning because high-level representations such as positions of a target and an acting agent are more important than pixel-wise reconstruction.
Method
The formal description of a pixel-based future state prediction is addressed.
Let x t ∈ R c ∗ w ∗ h ) is a state for given timestep t ∈ T where T, c, w, h , stands for total timesteps, channel, weight, and height.
Let X =
x ( t − n ) , x ( t − n +1) , ..., x ( t − 2) , x ( t − 1) } is a set of states from n past states to current state t − 1 .
Main goal is to predict the next frame x t based on previous sequence set X and latent variable z t .
Proposed approach consists of three module which are the Inference Module, the Prior Learning
Module and the Prediction Module.
Combination of MSE, Perceptual loss and KL loss [21] is applied to form Perceptual Reconstruction Loss function.
The Inference Module learns posterior q φ ( z t | x (1: t ) ) which is a distribution of a latent variable z where states from timestep 1 to t are given.
The latent variable z contains temporal information of states starting from timestep 1 to t .
The Prior Learning Module learns the prior p ψ ( z t | x (1: t − 1) ) at each timestep which is a distribution of a latent variable z where state from timestep 1 to t − 1 are given.
KL loss is calculated with the posterior q φ ( z t | x (1: t ) ) to have close distribution to the prior p ψ ( z t | x (1: t − 1) ) .
The Prediction
Module infers a future state image (cid:98) x t from a past state x t − 1 and a latent variable z t learned from the Inference Module and the Prior Learning Module.
Perceptual loss and MSE are both used as perceptual reconstruction loss to evaluate predicted image (cid:98) x t with original image x t .
The KL loss and the perceptual reconstruction loss are combined for learning processes.
2.1 Inference Module
The inference module is adapted with formalism of beta variational auto-encoder [22].
It learns posterior q φ ( z t | x (1: t ) ) which is a distribution of a latent variable z where states from timestep 1 to t are given.
The latent variable z has reduced dimensionality and contains temporal information about past t states.
Reduced dimensionality makes learning processes fast and easy to learn the distribution of representation.
The module is learned with maximization of the variational lower bound:
[Equation]
2.2 Prior Learning Module
The Prior Learning module is built on Stochastic Video Generation algorithm with learned prior (SVG-LP)[6].
Gaussian distribution with ﬁxed variance N (0,1) is not used here.
Instead this approach learns a prior which varies across time.
Prior p ψ ( z t | x (1: t − 1) ) is calculated at every timestep with all past frames up to timestep t − 1 and conditional gaussian distribution N ( µ ( x (1: t ) ) , φ ( x (1: t ) )) is used.
The module is learned with maximization of the variational lower bound:
[Equation]
2.3 Prediction Module
The Prediction Module infers a future state image (cid:98) x t from a past state x t − 1 and a latent variable z t learned from the Inference Module and the Prior Learning Module.
Temporal information is conditioned and high dimensionality is reduced to the latent variable z .
The inference process could infer temporal dynamics and how the future state would change based on the past states.
The prediction module predicts the future states for b period of timestep where a period of timestep of past states are given.
The prior is calculated in the prior learning module at every timestep.
2.4 Perceptual Reconstruction Loss
Loss function is built on beta variational autoencoder.
A perceptual loss function is applied to improve limited ability of pixel-wise loss function and extract high level representations.
A per-pixel loss function such as mean squared error (MSE) is vulnerable to slight deformation on predicted images which could damage the learning process.
Using perceptual loss could improve this limitations.
In perceptual reconstruction loss, pixel-wise reconstruction loss and perceptual reconstruction loss are combined.
Pre-trained VGG16 network [23] with ImageNet dataset [24] is used as perceptual network to extract high level features and MSE is used as pixel-wise loss.
The loss function of our model is described as:
[Equation]
Equation 1 shows complete loss function.
KL loss is calculated with the posterior q φ ( z t | x (1: t ) ) to have close distribution to the prior p ψ ( z t | x (1: t − 1) ) .
Equation 2 shows perceptual reconstruction loss that perceptual loss of each layer i as L ip in pretrained perceptual network are aggregated for a given timestep t .
Equation 3 shows detail of perceptual reconstruction loss.
The difference between original image x and reconstructed image (cid:98) x is calculated at each layer.
We evaluated our SVG-PL and baseline SVG-LP model on natural dataset BAIR Push [25].
Structural similarity index measure (SSIM) [26] is used for quantitative comparisons.
Past 5 frames are used to train the model to predict 10 future frames.
For each reconstructed 10 future frames, SSIM is measured compared to original future frames.
For each prediction sequences, 100 sample sequences are generated and one with best SSIM score is selected.
Since this metric provides limited comparison about perceptual qualities, qualitative comparison is also conducted.
Figure 1 shows average of SSIM results on stochastic moving BAIR Push data with SVG-PL and
SVG-LP models.
Results shows our model shows better SSIM results on predicting 10 future frames.
This shows perceptual reconstruction loss could improve accuracy of future frame prediction by extracting high-level representations than pixel-wise loss function for reconstruction.
Figure 2 shows qualitative results of future generations of BAIR Push data for seven future frames where ﬁve past frames are given.
The ﬁgure shows results of ground truth, proposed method SVGPL, and baseline method SVG respectively.
Result shows SVG-PL captures temporal dynamics of moving robot arm accurately compare to SVG-LP which shows limited representations.
As we predict continuous future states, the result shows compounding error that prediction error gets accumulated.
Conclusion and Discussion
Our model proposes perceptual reconstruction loss to predict future states in stochastic approach to improve blurriness and uncertainty of the video prediction.
Our model shows better results on future state prediction in both qualitative and quantitative metrics.
The perceptual reconstruction loss uses both pixel-wise loss and perceptual loss from pretrained perceptual network using VGG16 with
ImageNet dataset.
Utilization of perceptual loss on video prediction improves reconstruction ability and result in clear predictions.
Improvements in video prediction can help learning control policy in model-based reinforcement learning.
There are few limitations in this work and needs future works.
For the quantitative comparison only
Structural similarity index measure (SSIM) is used here.
There exists metrics which are used to compare perceptual similarities such as DeepSim [27], Learned Perceptual Image Patch Similarity (LPIPS) [28] and Berkeley Adobe Perceptual Patch Similarity (BAPPS) [28].
Using these metrics could show clear improvements in perceptual feature extraction in video prediction.
The perceptual network is pretrained with ImageNet dataset.
If the perceptual network is pretrained on the target dataset such as BAIR Push data, the model could extract better high-level representations on the target dataset.
Other algorithms than VGG16 could be used to extract perceptual representations.
Pixel-wise loss and perceptual loss are both used in this model.
Instead of giving equal weight on both losses, different weight could improve prediction results.
Acknowledgments and Disclosure of Funding
This work was supported by Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government.
[21ZR1100, A Study of Hyper-Connected Thinking Internet
Technology by autonomous connecting, controlling and evolving ways].