Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation
Anonymous Author(s)
AffiliationAddress email
Abstract
Video prediction is a challenging task.
The quality of video frames from currentstate-of-the-art (SOTA) generative models tends to be poor and generalizationbeyond the training data is difficult.
Furthermore, existing prediction frameworksare typically not capable of simultaneously handling other video-related tasks suchas unconditional generation or interpolation.
In this work, we devise a generalpurpose framework called Masked Conditional Video Diffusion (MCVD) for all ofthese video synthesis tasks using a probabilistic conditional score-based denoisingdiffusion model, conditioned on past and/or future frames.
We train the model in amanner where we randomly and independently mask all the past frames or all thefuture frames.
This novel but straightforward setup allows us to train a single modelthat is capable of executing a broad range of video tasks, specifically: future/pastprediction – when only future/past frames are masked; unconditional generation– when both past and future frames are masked; and interpolation – when neitherpast nor future frames are masked.
Our experiments show that this approach cangenerate high-quality frames for diverse types of videos.
Our MCVD models arebuilt from simple non-recurrent 2D-convolutional architectures, conditioning onblocks of frames and generating blocks of frames.
We generate videos of arbitrarylengths autoregressively in a block-wise manner.
Our approach yields SOTA resultsacross standard video prediction and interpolation benchmarks, with computationtimes for training models measured in 1-12 days using ≤ 4 GPUs.
Introduction
Predicting what one may visually perceive in the future is closely linked to the dynamics of objectsand people.
As such, this kind of prediction relates to many crucial human decision-making tasksranging from making dinner to driving a car.
If video models could generate full-fledged videos inpixel-level detail with plausible futures, agents could use them to make better decisions, especiallysafety-critical ones.
Consider, for example, the task of driving a car in a tight situation at highspeed.
Having an accurate model of the future could mean the difference between damaging a car orsomething worse.
We can obtain some intuitions about this scenario by examining the predictions ofour model in Figure 1, where we condition on two frames and predict 28 frames into the future fora car driving around a corner.
We can see that this is enough time for two different painted arrowsto pass under the car.
If one zooms in, one can inspect the relative positions of the arrow and the
Mercedes hood ornament in the real versus predicted frames.
Pixel-level models of trajectories,pedestrians, potholes, and debris on the road could one day improve the safety of vehicles.
Although beneficial to decision making, video generation is an incredibly challenging problem; notonly must high-quality frames be generated, but the changes over time must be plausible and ideallydrawn from an accurate and potentially complex distribution over probable futures.
Looking farin time is exceptionally hard given the exponential increase in possible futures.
Generating videofrom scratch or unconditionally further compounds the problem because even the structure of thefirst frame must be synthesized.
Also related to video generation are the simpler tasks of a) videoprediction, predicting the future given the past, and b) interpolation, predicting the in-between givenpast and future.
Yet, both problems remain challenging.
Specialized tools exist to solve the variousvideo tasks, but they rarely solve more than one task at a time.
Given the monumental task of general video generation, current approaches are still very limiteddespite the fact that many state of the art methods have hundreds of millions of parameters [Wu et al.
,2021, Weissenborn et al., 2019, Villegas et al., 2019, Babaeizadeh et al., 2021].
While industrialresearch is capable of looking at even larger models, current methods frequently underfit the data,leading to blurry videos, especially in the longer-term future and recent work has examined waysin improve parameter efficiency [Babaeizadeh et al., 2021].
Our objective here is to devise a videogeneration approach that generates high-quality, time-consistent videos within our computationbudget of ≤ 4 GPU) and computation times for training models ≤ two weeks.
Fortunately, diffusionmodels for image synthesis have demonstrated wide success, which strongly motivated our use ofthis approach.
Our qualitative results in Figure 1 also indicate that our particular approach does quitewell at synthesizing frames in the longer-term future (i.e., frame 29 in the bottom right corner).
One family of diffusion models might be characterized as Denoising Diffusion Probabilistic Models(DDPMs)
[Sohl-Dickstein et al., 2015, Ho et al., 2020, Dhariwal and Nichol, 2021], while anotherfamily might be characterized as Score-based Generative Models (SGMs)
[Song and Ermon, 2019, Liet al., 2019, Song and Ermon, 2020, Jolicoeur-Martineau et al., 2021a].
However, these approacheshave effectively merged into a field we shall refer to as score-based diffusion models.
Score-baseddiffusion models work by defining a stochastic process from data to noise and then reversing thatprocess to go from noise to data.
Their main benefits are that they generate very 1) high-quality and 2)diverse data samples.
One of their drawbacks is that solving the reverse process is relatively slow, butthere are ways to improve speed
[Song et al., 2020, Jolicoeur-Martineau et al., 2021b, Salimans and
Ho, 2022, Liu et al., 2022, Xiao et al., 2021].
Given their massive success and attractive properties,we focus here on developing our framework using score-based diffusion models for video predictionand generation.
Our work makes the following contributions:
A conditional video diffusion approach for video prediction and interpolation that yields
SOTA results.
A conditioning procedure based on masking past and/or future frames in a blockwise mannergiving a single model the ability to solve multiple video tasks: unconditional generation,future/past prediction, and interpolation.3.
A sliding window blockwise autoregressive conditioning procedure to allow fast and coherent
long-term generation.
A convolutional U-net neural architecture integrating recent developments with a conditionalnormalization technique we call SPAce-TIme-Adaptive Normalization (SPATIN) (Figure 3).
By conditioning on blocks of frames in the past and optionally blocks of frames even further in thefuture, we are able to better ensure that temporal dynamics are transferred across blocks of samples,i.e. our networks can learn implicit models of spatio-temporal dynamics to inform frame generation.
Unlike many other approaches, we do not have explicit model components for spatio-temporalderivatives or optical flow.
Conditional Diffusion for Video
Let x 0 ∈ R
d be a sample from the data distribution p data .
A sample x 0 can corrupted from t = 0 tot
= T through the Forward Diffusion Process (FDP) with the following transition kernel:
[Equation]
Furthermore, x t can be sampled directly from x 0 using the following accumulated kernel:
[Equation]
(2) where ¯ α t = (cid:81) ts =1 (1 − β s ) , and z ∼ N ( 0 , I ) .
Generating new samples can be done by reversing the FDP and solving the Reverse Diffusion Process(RDP) starting from Gaussian noise x T .
It can be shown (Song et al.
[2021], Ho et al.
[2020]) thatthe RDP can be computed using the following transition kernel:
[Equation]
where
[Equation]
and ˜
[Equation]
Since x 0 given x t is unknown, it can be estimated using eq.
(2): ˆ x 0 = (cid:0) x t −√ 1 − ¯ α t ϵ (cid:1) / √ ¯ α t , whereϵ θ ( x t | t ) estimates ϵ using a time-conditional neural network parameterized by θ .
This allows us toreverse the process from noise to data.
The loss function of the neural network is:
[Equation]
[Equation]
Note that estimating ϵ is equivalent to estimating a scaled version of the score function (i.e., thegradient of the log density) of the noisy data:
[Equation]
Thus, data generation through denoising depends on the score-function, and can be seen as noiseconditional score-based generation.
Score-based diffusion models can be straightforwardly adapted to video by considering the jointdistribution of multiple continuous frames.
While this is sufficient for unconditional video generation,other tasks such as video interpolation and prediction remain unsolved.
A conditional video predictionmodel can be approximately derived from the unconditional model using imputation [Song et al.
,2021]; indeed, the contemporary work of Ho et al.
[2022] attempts to use this technique; however,their approach is based on an approximate conditional model
2.1 Video Prediction via Conditional Diffusion
[Figure]
We first propose to directly model the conditional distribution of video frames in the immediatefuture given past frames.
Assume we have p past frames p = (cid:8) p i (cid:9) p i =1 and k current frames in theimmediate future x
0 = (cid:8) x i 0 (cid:9) k
i =1 .
We condition the above diffusion models on the past frames topredict the future frames:
[Equation]
[Equation]
Given a model trained as above, video prediction for subsequent time steps can be achieved byblockwise autoregressively predicting current video frames conditioned on previously predictedframes (see Figure 2).
We use variants of the network shown in Figure 3 to model ϵ θ in equation (6)here and for equations (7) and (8) below.
[Figure]
Real Past t =t =
t =t = t =t =
t =t =
t =t = 10 t = 11 t =
Prediction →
2.2 Video Prediction + Generation via Masked Conditional Diffusion
Our approach above allows video prediction, but not unconditional video generation.
As a secondapproach, we extend the same framework to video generation by masking (zeroing-out) the pastframes with probability p mask
= 1 / 2 using binary mask m .
The network thus learns to predict thenoise added without any past frames for context.
Doing so means that we can perform conditional aswell as unconditional frame generation, i.e., video prediction and generation with the same network.
This leads to the following loss ( B is the Bernouilli distribution):
[Equation]
We hypothesize that this dropout-like [Srivastava et al., 2014] approach will also serve as a form ofregularization, improving the model’s ability to perform predictions conditioned on the past.
Wesee positive evidence of this effect in our experiments – see the MCVD past-mask model variants in Table 3 versus comparable models without past-masking.
2.3 Video Prediction + Generation + Interpolation via Masked Conditional Diffusion
We now have a design for video prediction and generation, but it still cannot perform video interpolation nor past prediction from the future.
As a third and final approach, we show how to build ageneral model for solving all four video tasks.
Assume we have p past frames, k current frames, andf future frames f = (cid:8)
f i (cid:9) f i =1 .
We randomly mask the p past frames with probability p mask
= 1 / 2 ,and similarly randomly mask the f future frames with the same probability (but sampled separately).
Thus, future or past prediction is when only future or past frames are masked.
Unconditional generation is when both past and future frames are masked.
Video interpolation is when neither past norfuture frames are masked.
The loss function for this general video machinery is:
[Equation]
[Figure]
2.4 Our Network Architecture
For our denoising network we use a U-net architecture [Ronneberger et al., 2015, Honari et al., 2016,
Salimans et al., 2017] combining the improvements from Song et al.
[2021] and Dhariwal and Nichol[2021].
This architecture uses a mix of 2D convolutions [Fukushima and Miyake, 1982], multi-headself-attention
[Cheng et al., 2016], and adaptive group-norm [Wu and He, 2018].
We use positionalencodings of the noise level ( t ∈ [0 , 1] ) and process it using a transformer style positional embedding:
[Equation]
where d = 1 , . . .
, D/ 2 , D is the number of dimensions of the embedding, and c = 10000 .
Thisembedding vector is passed through a fully connected layer, followed by an activation functionand another fully connected layer.
Each residual block has an fully connected layer that adapts theembedding to the correct dimensionality.
To provide x t , p , and f to the network, we separately concatenate the past/future conditional framesand the noisy current frames in the channel dimension.
The concatenated noisy current framesare directly passed as input to the network.
Meanwhile, the concatenated conditional frames arepassed through an embedding that influences the conditional normalization akin to SPatially-Adaptive(DE)normalization (SPADE)
[Park et al., 2019]; to account for the effect of time/motion, we callthis approach SPAce-TIme-Adaptive Normalization (SPATIN).
In addition to SPATIN, we also trydirectly concatenating the conditional and noisy current frames together and passing them as theinput.
In our experiments below we show some results with SPATIN and some with concatenation(concat).
For simple video prediction with equation 6, we experimented with 3D convolutions and 3Dattention However, this requires an exorbitant amount of memory, and we found no benefit in using3D layers over 2D layers at the same memory (i.e., the biggest model that fits in 4 GPUs).
Thus, wedid not explore this idea further.
We also tried and found no benefit from gamma noise [Nachmaniet al., 2021], L1 loss, and F-PNDM sampling
[Liu et al., 2022].
Related work
Score-based diffusion models have been used for image editing [Meng et al., 2022, Saharia et al.
,2021, Nichol et al., 2021] and our approach to video generation might be viewed as an analogyto classical image inpainting, but in the temporal dimension.
The GLIDE or Guided Language
to
Image Diffusion for Generation and Editing approach of Nichol et al.
[2021] uses CLIP-guideddiffusion for image editing, while Denoising Diffusion Restoration Models (DDRM)
Kawar et al.[2022] additionally condition on a corrupted image to restore the clean image.
Adversarial variants ofscore-based diffusion models have been used to enhance quality [Jolicoeur-Martineau et al., 2021a]or speed
[Xiao et al., 2021].
Contemporary work to our own such as that of Ho et al.
[2022] and Yang et al.
[2022] alsoexamine video generation using score-based diffusion models.
However, the Video Diffusion Models(VDMs) work of Ho et al.
[2022] approximates conditional distributions using a gradient method forconditional sampling from their unconditional model formulation.
In contrast, our approach directlyworks with a conditional diffusion model, which we obtain through masked conditional training,thereby giving us the exact conditional distribution as well as the ability to generate unconditionally.
Their experiments focus on: a) unconditional video generation, and b) text-conditioned videogeneration, whereas our work focuses primarily on predicting future video frames from the past,using our masked conditional generation framework.
The Residual Video Diffusion (RVD) of Yanget al.
[2022] is only for video prediction, and it uses a residual formulation to generate framesautoregressively one at a time.
Meanwhile, ours directly models the conditional frames to generatemultiple frames in a block-wise autoregressive manner.
lowed by frames
3, 5, 10 and by bottom.
Recurrent neural network (RNN) techniqueswere early candidates for modern deep neural architectures for video prediction and generation.
Early work combined RNNs with a stochastic latent variable (SV2P) Babaeizadeh et al.
[2018a]and was optimized by variational inference.
Thestochastic video generation (SVG) approach of
Denton and Fergus [2018] learned both prior anda per time step latent variable model, which influences the dynamics of an LSTM at each step.
The model is also trained in a manner similarto a variational autoencoder, i.e., it was anotherform of variational RNN (vRNN).
To addressthe fact that vRNNs tend to lead to blurry results,
Castrejón et al.
[2019] (Hier-vRNN) increasedthe expressiveness of the latent distributions using a hierarchy of latent variables.
We compare qualitative result of SVG and Hier-vRNNwith the MCVD concat variant of our methodin Figure 4.
Other vRNN-based models include
SAVP Lee et al.
[2018], SRVP Franceschi et al.[2020], SLAMP Akan et al.
The well known Transformer paradigm [Vaswani et al., 2017] from natural language processing hasalso been explored for video.
The Video-GPT work of Yan et al.
[2021] applied an autoregressive GPTstyle [Brown et al., 2020] transformer to the codes produced from a VQ-VAE [Van Den Oord et al.
The Video Transformer work of Weissenborn et al.
[2019] models video using 3-D spatiotemporal volumes without linearizing positions in the volume.
They examine local self-attention oversmall non-overlapping sub-volumes or 3D blocks.
This is done partly to accelerate computationson TPU hardware.
Their work also observed that the peak signal-to-noise ratio (PSNR) metric andthe mean-structural similarity (SSIM) metrics
[Wang et al., 2004] were developed for images andhave serious flaws when applied to videos.
PSNR prefers blurry videos and SSIM does not correlate well to perceptual quality.
Like them, we focus on the recently proposed Frechet Video Distance(FVD)
[Unterthiner et al., 2018], computed over entire videos and which is sensitive to visual quality,temporal coherence and the diversity of samples.
Rakhimov et al.
[2020] (LVT) explored the useof transformers in predicting the dynamics of video in latent space.
Le Moing et al.
[2021] (CCVS)also predict in latent space, that of an adversarially trained autoencoder, and also add in a learnableoptical flow module.
Generative Adversarial Network (GAN) based approaches to video generation have also beenstudied extensively.
Vondrick et al.
[2016] proposed an early GAN architecture for video, using aspatio-temporal CNN.
Villegas et al.
[2017] proposed a strategy for separating motion and contentinto different pathways of a convolutional LSTM based encoder-decoder RNN.
Saito et al.
[2017](TGAN) predicted a sequence of latents using a temporal generator, and then the sequence of framesfrom those latents using an image generator.
TGANv2 Saito et al.
[2020] improved its memoryefficiency.
MoCoGAN Tulyakov et al.
[2018] explored style and content separation, but within a
CNN framework.
Yushchenko et al.
[2019] used the MoCoGAN framework by re-formulating thevideo prediction problem as a Markov Decision Process (MDP).
FutureGAN Aigner and Körner[2018] used spatio-temporal 3D convolutions in an encoder decoder architecture, and elements ofthe progressive GAN Karras et al.
[2018] approach to improve image quality.
TS-GAN Munozet al.
[2021] facilitated information flow between consecutive frames.
TriVD-GAN Luc et al.
[2020]proposes a novel recurrent unit in the generator to handle more complex dynamics, while DIGAN Yuet al.
[2022] uses implicit neural representations in the generator.
Video interpolation was the subject of a flurry of interest in the deep learning community a number ofyears ago [Niklaus et al., 2017, Jiang et al., 2018, Xue et al., 2019, Bao et al., 2019].
However, thesearchitectures tend to be fairly specialized to the interpolation task, involving optical flow or motionfield modelling and computations.
Frame interpolation is useful for video compression; therefore,many other lines of work have examined interpolation from a compression perspective.
However,these architectures tend to be extremely specialized to the video compression task
[Yang et al., 2020].
The Cutout approach of DeVries and Taylor [2017] has examined the idea of cutting out smallcontinuous regions of an input image, such as small squares.
Dropout [Srivastava et al., 2014] at the
FeatureMap level was proposed and explored under the name of SpatialDropout in Tompson et al.[2015].
Input Dropout [de Blois et al., 2020] has been examined in the context of dropping differentchannels of multi-modal input imagery, such as the dropping of the RGB channels or depth mapchannels during training, then using the model without one of the modalities during testing, e.g. intheir work they drop the depth channel.
Experiments
[Table]
[Table]
[Table] [Table] We show the results of our videoprediction experiments in Tables 14 for Stochastic Moving MNIST(SMMNIST) 1 , KTH 2 , BAIR 3 , and Cityscapes 4 respectively.
We presentunconditional generation results for BAIR in Table 5 and UCF-101 5 in Table 6, and we present interpolation results for SMMNIST, KTH, and BAIRin Table 7.
See our Appendix and supplementary material for additional visual results.
Unless otherwise specified, when we used masks, we set the mask probability to 0.5.
For sampling,we report results using the sampling methods DDPM
[Ho et al., 2020] or DDIM [Song et al.
, 2020]with only 100 sampling steps, even though our models were trained with 1000, to make sampling faster.
We observe that the metrics are generally better using DDPM than DDIM (except with
UCF-101), and that using 1000 sampling steps could yield even better results.
Note that all our models are trained to predict only 4-5 current frames at a time, unlike other modelsthat predict ≥ 10.
We use these models to then autoregressively predict longer sequences for predictionor generation.
This was done in order to fit the models in our GPU memory budget.
Despite thisdisadvantage, we find that our MCVD models perform better than many previous SOTA methods.
[Table]
[Table]
In Table 3 we compare modelsthat use concatenated raw pixelsas input to U-Net blocks (concat)to SPATIN variants.
We also compare no-masking to past-maskingvariants, i.e. models which areonly trained predict the future vs.models which are regularized bybeing trained for prediction andunconditional generation.
Examining these results we remark that we have SOTA performance for prediction on SMMNIST, BAIR and the challenging
Cityscapes evaluation.
Our Cityscapes model yields an FVD of 145.5, whereas the best previousresult of which we are aware is 418.
The quality of our Cityscapes results are illustrated visuallyin Figure 1, 2 and in the additional examples provided in our Appendix.
While our completelyunconditional generation results are strong but not SOTA, we note that when past masking is used toregularize future predicting models, we see clear performance gains in Table 3.
Finally, in Table 7we see that our interpolation results are SOTA by a wide margin, across our suite of experiments on
SMMNIST, KTH and BAIR – even compared to architectures that are much more specialized forinterpolation.
[Table]
due issues Cityscapes [2 → 28; trained on k ]
[Table]
SVG
[Table]
LP Denton and Fergus [2018]
[Table]
MCVD concat past
[Table]
Conclusion
[Table]
We have shown how to obtain SOTAvideo prediction and interpolation results with randomly masked conditional video diffusion models usinga relatively simple architecture.
Wefound that past-masking was able toimprove performance across all model variants and configurations tested.
We believe our approachmay pave the way forward toward high quality larger-scale video generation.
[Table]
Limitations.
Videos generated bythese models are still small comparedto real movies, and they can still become blurry or inconsistent when thenumber of generated frames is verylarge.
Our unconditional generationresults on the highly diverse UCF-101dataset are still far from perfect.
Morework is clearly needed to scale these models to larger datasets with more diversity and with longerduration video.
As has been the case in many other settings, simply using larger models with manymore parameters is a strategy that is likely to improve the quality and flexibility of these models – wewere limited to 4 GPUs for our work here.
There is also a need for faster sampling methods capableof maintaining quality over time.
Given our strong interpolation results, conditional diffusion modelsfor prediction which skip frames when generating future blocks of frames could make it possible togenerate much longer, but consistent video through a strategy of first generating sparse distant framesin a block, followed by an interpolative diffusion step for the missing frames.
Broader Impacts.
Video generation could eventually help with safety critical AI applications. However actors with ill intent could generate video for socially negative purposes.
[Table] [Table] Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem? [Yes] If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We will release
the code with the supplementary
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We will provide this in the supplementary material, there was
insufficient space to provide this in the main paper.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)? [Yes] Whenever possible. (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] Readers can refer to the original
references.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
However we will link to a URL after peer review.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A] We are using publicly available data collected by others. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]