Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation
Anonymous Author(s)
AffiliationAddress email
Abstract
Video prediction is a challenging task.
The quality of video frames from currentstate-of-the-art (SOTA) generative models tends to be poor and generalizationbeyond the training data is difficult.
Furthermore, existing prediction frameworksare typically not capable of simultaneously handling other video-related tasks suchas unconditional generation or interpolation.
In this work, we devise a generalpurpose framework called Masked Conditional Video Diffusion (MCVD) for all ofthese video synthesis tasks using a probabilistic conditional score-based denoisingdiffusion model, conditioned on past and/or future frames.
We train the model in amanner where we randomly and independently mask all the past frames or all thefuture frames.
This novel but straightforward setup allows us to train a single modelthat is capable of executing a broad range of video tasks, specifically: future/pastprediction – when only future/past frames are masked; unconditional generation– when both past and future frames are masked; and interpolation – when neitherpast nor future frames are masked.
Our experiments show that this approach cangenerate high-quality frames for diverse types of videos.
Our MCVD models arebuilt from simple non-recurrent 2D-convolutional architectures, conditioning onblocks of frames and generating blocks of frames.
We generate videos of arbitrarylengths autoregressively in a block-wise manner.
Our approach yields SOTA resultsacross standard video prediction and interpolation benchmarks, with computationtimes for training models measured in 1-12 days using ≤ 4 GPUs.
Introduction
Predicting what one may visually perceive in the future is closely linked to the dynamics of objectsand people.
As such, this kind of prediction relates to many crucial human decision-making tasksranging from making dinner to driving a car.
If video models could generate full-fledged videos inpixel-level detail with plausible futures, agents could use them to make better decisions, especiallysafety-critical ones.
Consider, for example, the task of driving a car in a tight situation at highspeed.
Having an accurate model of the future could mean the difference between damaging a car orsomething worse.
We can obtain some intuitions about this scenario by examining the predictions ofour model in Figure 1, where we condition on two frames and predict 28 frames into the future fora car driving around a corner.
We can see that this is enough time for two different painted arrowsto pass under the car.
If one zooms in, one can inspect the relative positions of the arrow and the
Mercedes hood ornament in the real versus predicted frames.
Pixel-level models of trajectories,pedestrians, potholes, and debris on the road could one day improve the safety of vehicles.
Although beneficial to decision making, video generation is an incredibly challenging problem; notonly must high-quality frames be generated, but the changes over time must be plausible and ideallydrawn from an accurate and potentially complex distribution over probable futures.
Looking farin time is exceptionally hard given the exponential increase in possible futures.
Generating videofrom scratch or unconditionally further compounds the problem because even the structure of thefirst frame must be synthesized.
Also related to video generation are the simpler tasks of a) videoprediction, predicting the future given the past, and b) interpolation, predicting the in-between givenpast and future.
Yet, both problems remain challenging.
Specialized tools exist to solve the variousvideo tasks, but they rarely solve more than one task at a time.
Given the monumental task of general video generation, current approaches are still very limiteddespite the fact that many state of the art methods have hundreds of millions of parameters [Wu et al.
,2021, Weissenborn et al., 2019, Villegas et al., 2019, Babaeizadeh et al., 2021].
While industrialresearch is capable of looking at even larger models, current methods frequently underfit the data,leading to blurry videos, especially in the longer-term future and recent work has examined waysin improve parameter efficiency [Babaeizadeh et al., 2021].
Our objective here is to devise a videogeneration approach that generates high-quality, time-consistent videos within our computationbudget of ≤ 4 GPU) and computation times for training models ≤ two weeks.
Fortunately, diffusionmodels for image synthesis have demonstrated wide success, which strongly motivated our use ofthis approach.
Our qualitative results in Figure 1 also indicate that our particular approach does quitewell at synthesizing frames in the longer-term future (i.e., frame 29 in the bottom right corner).
One family of diffusion models might be characterized as Denoising Diffusion Probabilistic Models(DDPMs)
[Sohl-Dickstein et al., 2015, Ho et al., 2020, Dhariwal and Nichol, 2021], while anotherfamily might be characterized as Score-based Generative Models (SGMs)
[Song and Ermon, 2019, Liet al., 2019, Song and Ermon, 2020, Jolicoeur-Martineau et al., 2021a].
However, these approacheshave effectively merged into a field we shall refer to as score-based diffusion models.
Score-baseddiffusion models work by defining a stochastic process from data to noise and then reversing thatprocess to go from noise to data.
Their main benefits are that they generate very 1) high-quality and 2)diverse data samples.
One of their drawbacks is that solving the reverse process is relatively slow, butthere are ways to improve speed
[Song et al., 2020, Jolicoeur-Martineau et al., 2021b, Salimans and
Ho, 2022, Liu et al., 2022, Xiao et al., 2022].
Given their massive success and attractive properties,we focus here on developing our framework using score-based diffusion models for video predictionand generation.
Our work makes the following contributions:
A conditional video diffusion approach for video prediction and interpolation that yields
SOTA results.
A conditioning procedure based on masking past and/or future frames in a blockwise mannergiving a single model the ability to solve multiple video tasks: unconditional generation,future/past prediction, and interpolation.3.
A sliding window blockwise autoregressive conditioning procedure to allow fast and coherent
long-term generation.
A convolutional U-net neural architecture integrating recent developments with a conditionalnormalization technique we call SPAce-TIme-Adaptive Normalization (SPATIN) (Figure 3).
By conditioning on blocks of frames in the past and optionally blocks of frames even further in thefuture, we are able to better ensure that temporal dynamics are transferred across blocks of samples,i.e. our networks can learn implicit models of spatio-temporal dynamics to inform frame generation.
Unlike many other approaches, we do not have explicit model components for spatio-temporalderivatives or optical flow or recurrent blocks.
Conditional Diffusion for Video
Let x 0 ∈ R
d be a sample from the data distribution p data .
A sample x 0 can corrupted from t = 0 tot
= T through the Forward Diffusion Process (FDP) with the following transition kernel:
[Equation]
Furthermore, x t can be sampled directly from x 0 using the following accumulated kernel:
[Equation]
(2) where ¯ α t = (cid:81) ts =1 (1 − β s ) , and ϵ ∼ N ( 0 , I ) .
Generating new samples can be done by reversing the FDP and solving the Reverse Diffusion Process(RDP) starting from Gaussian noise x T .
It can be shown (Song et al.
[2021], Ho et al.
[2020]) thatthe RDP can be computed using the following transition kernel:
[Equation]
where
[Equation]
and
[Equation]
Since x 0 given x t is unknown, it can be estimated using eq.
(2): ˆ x 0 = (cid:0) x t −√ 1 − ¯ α t ϵ (cid:1) / √ ¯ α t , whereϵ θ ( x t | t ) estimates ϵ using a time-conditional neural network parameterized by θ .
This allows us toreverse the process from noise to data.
The loss function of the neural network is:
[Equation]
Note that estimating ϵ is equivalent to estimating a scaled version of the score function (i.e., thegradient of the log density) of the noisy data:
[Equation]
Thus, data generation through denoising depends on the score-function, and can be seen as noiseconditional score-based generation.
Score-based diffusion models can be straightforwardly adapted to video by considering the jointdistribution of multiple continuous frames.
While this is sufficient for unconditional video generation,other tasks such as video interpolation and prediction remain unsolved.
A conditional video predictionmodel can be approximately derived from the unconditional model using imputation [Song et al.
,2021]; indeed, the contemporary work of Ho et al.
[2022] attempts to use this technique; however,their approach is based on an approximate conditional model.
2.1 Video Prediction via Conditional Diffusion
[Figure]
We first propose to directly model the conditional distribution of video frames in the immediatefuture given past frames.
Assume we have p past frames p = (cid:8) p i (cid:9) p i =1 and k current frames in theimmediate future x
0 = (cid:8) x i 0 (cid:9) k
i =1 .
We condition the above diffusion models on the past frames topredict the current frames:
[Figure]
[Equation]
Given a model trained as above, video prediction for subsequent time steps can be achieved byblockwise autoregressively predicting current video frames conditioned on previously predictedframes (see Figure 2).
We use variants of the network shown in Figure 3 to model ϵ θ in equation (6)here and for equations (7) and (8) below.
[Figure]
Real Past t =t =
t =t = t =t =
t =t =
t =t = 10 t = 11 t =
Prediction →
2.2 Video Prediction + Generation via Masked Conditional Diffusion
Our approach above allows video prediction, but not unconditional video generation.
As a secondapproach, we extend the same framework to video generation by masking (zeroing-out) the pastframes with probability p mask
= 1 / 2 using binary mask m p .
The network thus learns to predict thenoise added without any past frames for context.
Doing so means that we can perform conditional aswell as unconditional frame generation, i.e., video prediction and generation with the same network.
This leads to the following loss ( B is the Bernouilli distribution):
[Equation]
We hypothesize that this dropout-like [Srivastava et al., 2014] approach will also serve as a form ofregularization, improving the model’s ability to perform predictions conditioned on the past.
We seepositive evidence of this effect in our experiments – see the MCVD past-mask model variants in Table3 versus comparable models without past-masking.
Note that masking is used only during training.
2.3 Video Prediction + Generation + Interpolation via Masked Conditional Diffusion
We now have a design for video prediction and generation, but it still cannot perform video interpolation nor past prediction from the future.
As a third and final approach, we show how to build ageneral model for solving all four video tasks.
Assume we have p past frames, k current frames, andf future frames f = (cid:8)
f i (cid:9) f i =1 .
We randomly mask the p past frames with probability p mask
= 1 / 2 ,and similarly randomly mask the f future frames with the same probability (but sampled separately).
Thus, future or past prediction is when only future or past frames are masked.
Unconditional generation is when both past and future frames are masked.
Video interpolation is when neither past norfuture frames are masked.
The loss function for this general video machinery is:
[Equation]
[Figure]
2.4 Our Network Architecture
For our denoising network we use a U-net architecture [Ronneberger et al., 2015, Honari et al., 2016,
Salimans et al., 2017] combining the improvements from Song et al.
[2021] and Dhariwal and Nichol[2021].
This architecture uses a mix of 2D convolutions [Fukushima and Miyake, 1982], multi-headself-attention
[Cheng et al., 2016], and adaptive group-norm [Wu and He, 2018].
We use positionalencodings of the noise level ( t ∈ [0 , 1] ) and process it using a transformer style positional embedding:
[Equation]
where d = 1 , . . .
, D/ 2 , D is the number of dimensions of the embedding, and c = 10000 .
Thisembedding vector is passed through a fully connected layer, followed by an activation functionand another fully connected layer.
Each residual block has an fully connected layer that adapts theembedding to the correct dimensionality.
To provide x t , p , and f to the network, we separately concatenate the past/future conditional framesand the noisy current frames in the channel dimension.
The concatenated noisy current framesare directly passed as input to the network.
Meanwhile, the concatenated conditional frames arepassed through an embedding that influences the conditional normalization akin to SPatially-Adaptive(DE)normalization (SPADE)
[Park et al., 2019]; to account for the effect of time/motion, we callthis approach SPAce-TIme-Adaptive Normalization (SPATIN).
In addition to SPATIN, we also trydirectly concatenating the conditional and noisy current frames together and passing them as theinput.
In our experiments below we show some results with SPATIN and some with concatenation(concat).
For simple video prediction with equation 6, we experimented with 3D convolutions and 3Dattention However, this requires an exorbitant amount of memory, and we found no benefit in using3D layers over 2D layers at the same memory (i.e., the biggest model that fits in 4 GPUs).
Thus, wedid not explore this idea further.
We also tried and found no benefit from gamma noise [Nachmaniet al., 2021], L1 loss, and F-PNDM sampling
[Liu et al., 2022].
Related work
Score-based diffusion models have been used for image editing [Meng et al., 2022, Saharia et al.
,2021, Nichol et al., 2021] and our approach to video generation might be viewed as an analogyto classical image inpainting, but in the temporal dimension.
The GLIDE or Guided Language
to
Image Diffusion for Generation and Editing approach of Nichol et al.
[2021] uses CLIP-guideddiffusion for image editing, while Denoising Diffusion Restoration Models (DDRM)
Kawar et al.[2022] additionally condition on a corrupted image to restore the clean image.
Adversarial variants ofscore-based diffusion models have been used to enhance quality [Jolicoeur-Martineau et al., 2021a]or speed
[Xiao et al., 2022].
Contemporary work to our own such as that of Ho et al.
[2022] and Yang et al.
[2022] alsoexamine video generation using score-based diffusion models.
However, the Video Diffusion Models(VDMs) work of Ho et al.
[2022] approximates conditional distributions using a gradient method forconditional sampling from their unconditional model formulation.
In contrast, our approach directlyworks with a conditional diffusion model, which we obtain through masked conditional training,thereby giving us the exact conditional distribution as well as the ability to generate unconditionally.
Their experiments focus on: a) unconditional video generation, and b) text-conditioned videogeneration, whereas our work focuses primarily on predicting future video frames from the past,using our masked conditional generation framework.
The Residual Video Diffusion (RVD) of Yanget al.
[2022] is only for video prediction, and it uses a residual formulation to generate framesautoregressively one at a time.
Meanwhile, ours directly models the conditional frames to generatemultiple frames in a block-wise autoregressive manner.
lowed by frames
3, 5, 10 and by bottom.
Recurrent neural network (RNN) techniqueswere early candidates for modern deep neural architectures for video prediction and generation.
Early work combined RNNs with a stochasticlatent variable (SV2P) Babaeizadeh et al.
[2018]and was optimized by variational inference.
Thestochastic video generation (SVG) approach of
Denton and Fergus [2018] learned both prior anda per time step latent variable model, which influences the dynamics of an LSTM at each step.
The model is also trained in a manner similarto a variational autoencoder, i.e., it was anotherform of variational RNN (vRNN).
To addressthe fact that vRNNs tend to lead to blurry results,
Castrejón et al.
[2019] (Hier-vRNN) increasedthe expressiveness of the latent distributions using a hierarchy of latent variables.
We compare qualitative result of SVG and Hier-vRNNwith the MCVD concat variant of our methodin Figure 4.
Other vRNN-based models include
SAVP Lee et al.
[2018], SRVP Franceschi et al.[2020], SLAMP Akan et al.
The well known Transformer paradigm [Vaswani et al., 2017] from natural language processing hasalso been explored for video.
The Video-GPT work of Yan et al.
[2021] applied an autoregressive GPTstyle [Brown et al., 2020] transformer to the codes produced from a VQ-VAE [Van Den Oord et al.
The Video Transformer work of Weissenborn et al.
[2019] models video using 3-D spatiotemporal volumes without linearizing positions in the volume.
They examine local self-attention oversmall non-overlapping sub-volumes or 3D blocks.
This is done partly to accelerate computationson TPU hardware.
Their work also observed that the peak signal-to-noise ratio (PSNR) metric andthe mean-structural similarity (SSIM) metrics
[Wang et al., 2004] were developed for images, andhave serious flaws when applied to videos.
PSNR prefers blurry videos and SSIM does not correlate well to perceptual quality.
Like them, we focus on the recently proposed Frechet Video Distance(FVD)
[Unterthiner et al., 2018], computed over entire videos and which is sensitive to visual quality,temporal coherence, and diversity of samples.
Rakhimov et al.
[2020] (LVT) used transformers topredict the dynamics of video in latent space.
Le Moing et al.
[2021] (CCVS) also predict in latentspace, that of an adversarially trained autoencoder, and also add a learnable optical flow module.
Generative Adversarial Network (GAN) based approaches to video generation have also beenstudied extensively.
Vondrick et al.
[2016] proposed an early GAN architecture for video, using aspatio-temporal CNN.
Villegas et al.
[2017] proposed a strategy for separating motion and contentinto different pathways of a convolutional LSTM based encoder-decoder RNN.
Saito et al.
[2017](TGAN) predicted a sequence of latents using a temporal generator, and then the sequence of framesfrom those latents using an image generator.
TGANv2 Saito et al.
[2020] improved its memoryefficiency.
MoCoGAN Tulyakov et al.
[2018] explored style and content separation, but within a
CNN framework.
Yushchenko et al.
[2019] used the MoCoGAN framework by re-formulating thevideo prediction problem as a Markov Decision Process (MDP).
FutureGAN Aigner and Körner[2018] used spatio-temporal 3D convolutions in an encoder decoder architecture, and elements ofthe progressive GAN Karras et al.
[2018] approach to improve image quality.
TS-GAN Munozet al.
[2021] facilitated information flow between consecutive frames.
TriVD-GAN Luc et al.
[2020]proposes a novel recurrent unit in the generator to handle more complex dynamics, while DIGAN Yuet al.
[2022] uses implicit neural representations in the generator.
Video interpolation was the subject of a flurry of interest in the deep learning community a number ofyears ago [Niklaus et al., 2017, Jiang et al., 2018, Xue et al., 2019, Bao et al., 2019].
However, thesearchitectures tend to be fairly specialized to the interpolation task, involving optical flow or motionfield modelling and computations.
Frame interpolation is useful for video compression; therefore,many other lines of work have examined interpolation from a compression perspective.
However,these architectures tend to be extremely specialized to the video compression task
[Yang et al., 2020].
The Cutout approach of DeVries and Taylor [2017] has examined the idea of cutting out smallcontinuous regions of an input image, such as small squares.
Dropout [Srivastava et al., 2014] at the
FeatureMap level was proposed and explored under the name of SpatialDropout in Tompson et al.[2015].
Input Dropout [de Blois et al., 2020] has been examined in the context of dropping differentchannels of multi-modal input imagery, such as the dropping of the RGB channels or depth mapchannels during training, then using the model without one of the modalities during testing, e.g. intheir work they drop the depth channel.
Regarding our block-autoregressive approach, previous video prediction models were typically either1) non-recurrent: predicting all n frames simultaneously with no way of adding more frames (most
GAN-based methods), or 2) recurrent in nature, predicting 1 frame at a time in an autoregressivefashion.
The benefit of the non-recurrent type is that you can generate videos faster than 1 frame at atime while allowing for generating as many frames as needed.
The disadvantage is that it is slowerthan generating all frames at once, and takes up more memory and compute at each iteration.
Ourmodel finds a sweet spot in between in that it is block-autoregressive: generating k < n frames at atime recurrently to finally obtain n frames.
Experiments
We show the results of our video prediction experiments on test data that was never seen duringtraining in Tables 1 - 4 for Stochastic Moving MNIST (SMMNIST) 1 , KTH 2 , BAIR 3 , and Cityscapes 249respectively.
We present unconditional generation results for BAIR in Table 5 and UCF-101 5 in
Table 6, and we present interpolation results for SMMNIST, KTH, and BAIR in Table 7.
Datasets : We generate 128x128 images for Cityscapes and 64x64 images for the other datasets.
See our Appendix and supplementary material for additional visual results.
Our choice of datasetsis in order of progressive difficulty: 1) SMMNIST: black-and-white digits; 2) KTH: grayscale single-humans; 3) BAIR: color, multiple objects, simple scene; 4) Cityscapes: color, natural complexnatural driving scene; 5) UCF101: color, 101 categories of natural scenes.
We process these datasetssimilarly to prior works.
For Cityscapes, each video is center-cropped, then resized to 128 × 128 .
For
UCF101, each video clip is center-cropped at 240×240 and resized to 64×64, taking care to maintainthe train-test splits.
[Table]
Unless otherwise specified, we setthe mask probability to 0.5 whenmasking was used.
For sampling,we report results using the samplingmethods DDPM
[Ho et al., 2020] or
DDIM [Song et al., 2020] with only100 sampling steps, though our models were trained with 1000, to makesampling faster.
We observe thatthe metrics are generally better using
DDPM than DDIM (except for UCF-101).
Using 1000 sampling steps could yield better results.
Note that all our models are trained to predict only 4-5 current frames at a time, unlike other modelsthat predict ≥ 10.
We use these models to then autoregressively predict longer sequences for predictionor generation.
This was done in order to fit the models in our GPU memory budget.
Despite thisdisadvantage, we find that our MCVD models perform better than many previous SOTA methods.
[Table]
Metrics : As mentioned earlier,we primarily use the FVD metric for comparison across models as FVD measures both fidelityand diversity of the generatedsamples.
Previous works compare Frechet Inception Distance(FID)
[Heusel et al., 2017] and
Inception Score (IS) [Salimanset al., 2016], adapted to videos byreplacing the Inception networkwith a 3D-convolutional network that takes video input.
FVD is computed similarly to FID, but usingan I3D network trained on the huge video dataset Kinetics-400.
We also report PSNR and SSIM.
Ablation studies : In Table 3 we compare models that use concatenated raw pixels as input to U-Netblocks (concat) to SPATIN variants.
We also compare no-masking to past-masking variants, i.e.models which are only trained predict the future vs. models which are regularized by being trained forprediction and unconditional generation.
It can be seen that our model works across different choicesof past frames and generates better quality for shorter videos.
This is expected from models of thiskind.
Moreover, it can be seen that the model trained on the two tasks of Prediction and Generation(i.e.
, the models with past-mask) performs better than the model trained only on Prediction!
In addition, the appendix contains an ablation study in Table 9 on the different design choices: concatvs concat past-future-mask vs spatin vs spatin future-mask vs spatin past-future-mask.
It can be seenthat concat is, in general, better than spatin.
It can also be seen that the past-future-mask variant,which is a general model capable of all three tasks, performs better at the individual tasks than themodels trained only on the individual task.
This was demonstrated in Table 3 as well.
This shows thatthe model gains very helpful insights while generalizing to all three tasks, which it does not whiletraining only on the individual task.
We conducted preliminary experiments with a larger number of frames.
Since the models with alarger number of frames were bigger, we could only run them for a shorter time with a smaller batchsize than the smaller models.
In general, we found that larger models did not substantially improvethe results.
We attribute this to the fact that using more frames means that the model should be given more capacity, but we could not increase it due to our computational budget constraints.
Weemphasize that our method works very well with fewer computational resources.
Examining these results we remark that we have SOTA performance for prediction on SMMNIST,
BAIR and the challenging Cityscapes evaluation.
Our Cityscapes model yields an FVD of 145.5,whereas the best previous result of which we are aware is 418.
The quality of our Cityscapes resultsare illustrated visually in Figure 1, 2 and in the additional examples provided in our Appendix.
Whileour completely unconditional generation results are strong, we note that when past masking is usedto regularize future predicting models, we see clear performance gains in Table 3.
Finally, in Table 7we see that our interpolation results are SOTA by a wide margin, across experiments on SMMNIST,
KTH and BAIR – even compared to architectures much more specialized for interpolation.
It can be seen that our proposed method generates better quality videos, even though it was trainedon a shorter number of frames than other methods.
It can also be seen that using masking improvesthe quality of generated frames than without masking.
[Table]
[Table]
Conclusion
[Table]
We have shown how to obtain SOTAvideo prediction and interpolation results with randomly masked conditional video diffusion models usinga relatively simple architecture.
Wefound that past-masking was ableto improve performance across allmodel variants and configurationstested.
We believe our approach maypave the way forward toward highquality larger-scale video generation.
[Table]
Limitations.
Videos generated bythese models are still small comparedto real movies, and they can still become blurry or inconsistent when the number of generated frames is very large.
Our unconditionalgeneration results on the highly diverse UCF-101 dataset are still far from perfect.
More work isclearly needed to scale these models to larger datasets with more diversity and with longer durationvideo.
As has been the case in many other settings, simply using larger models with many moreparameters is a strategy that is likely to improve the quality and flexibility of these models – we werelimited to 4 GPUs for our work here.
There is also a need for faster sampling methods capable ofmaintaining quality over time.
Given our strong interpolation results, conditional diffusion models for prediction which skip frameswhen generating future blocks of frames could make it possible to generate much longer, butconsistent video through a strategy of first generating sparse distant frames in a block, followed by aninterpolative diffusion step for the missing frames.
Reporting average of the best metrics out of n trajectories per test sample.
↓ ( p + f ) and ↑ k is harder.
We used MCVD spatin for SMMNIST and KTH, and MCVD concat past-future-mask for BAIR.
We also include results on SMMNIST for a "pure" model trained without any masking.
[Table]
Broader Impacts.
High-quality video generation is potentially a powerful technology that couldbe used by malicious actors for applications such as creating fake video content.
Our formulationfocuses on capturing the distributions of real video sequences.
High-quality video prediction couldone day find use in applications such as autonomous vehicles, where the cost of errors could be high.
Diffusion methods have shown great promise for covering the modes of real probability distributions.
In this context, diffusion-based techniques for generative modelling may be a promising avenue forfuture research where the ability to capture modes properly is safety critical.
Another potential pointof impact is the amount of computational resources being spent for these applications involving thehigh fidelity and voluminous modality of video data.
We emphasize the use of limited resourcesin achieving better or comparable results.
Our submission provides evidence for more efficientcomputation involving fewer GPU hours spent in training time.