Video Diffusion Models
Abstract
Generating temporally coherent high ﬁdelity video is an important milestone in generative modeling research.
We make progress towards this milestone by proposing a diffusion model for video generation that shows very promising initial results.
Our model is a natural extension of the standard image diffusion architecture, and it enables jointly training from image and video data, which we ﬁnd to reduce the variance of minibatch gradients and speed up optimization.
To generate long and higher resolution videos we introduce a new conditional sampling technique for spatial and temporal video extension that performs better than previously proposed methods.
We present the ﬁrst results on a large text-conditioned video generation task, as well as state-of-the-art results on established benchmarks for video prediction and unconditional video generation.
Supplementary material is available at https://video-diffusion.github.io/ .
Introduction
Diffusion models have recently been producing high quality results in image generation and audio generation [e g. 28, 39, 40, 16, 23, 36, 48, 60, 42, 10, 29], and there is signiﬁcant interest in validating diffusion models in new data modalities.
In this work, we present ﬁrst results on video generation using diffusion models, for both unconditional and conditional settings.
We show that high quality videos can be generated using essentially the standard formulation of the Gaussian diffusion model [46], with little modiﬁcation other than straightforward architectural changes to accommodate video data within the memory constraints of deep learning accelerators.
We train models that generate a ﬁxed number of video frames using a 3D U-Net diffusion model architecture and we enable generating longer videos by applying this model autoregressively using a new method for conditional generation.
We additionally show the beneﬁts of joint training on video and image modeling objectives.
We test our methods on video prediction and unconditional video generation, where we achieve state-of-the-art sample quality scores, and we also show promising ﬁrst results on text-conditioned video generation.
Background
A diffusion model [46, 47, 22] speciﬁed in continuous time [53, 48, 10, 28] is a generative model with latents
z = { z t | t ∈ [0 , 1] } obeying a forward process q ( z | x ) starting at data x ∼ p ( x ) .
The forward process is a Gaussian process that satisﬁes the Markovian structure:
[Equation]
where 0 ≤ s < t ≤ 1 , σ 2 t | s = (1 − e λ t − λ s ) σ 2 t , and α t , σ t specify a differentiable noise schedule whose log signal-to-noise-ratio λ t = log
[ α 2 t /σ 2 t ] decreases with t until q ( z 1 ) ≈ N ( 0 , I ) .
Training Learning to reverse the forward process for generation can be reduced to learning to denoise z t ∼ q ( z t | x ) into an estimate ˆ x θ ( z t , λ t )
≈ x for all t (we will drop the dependence on λ t to simplify notation).
We train this denoising model ˆ x θ using a weighted mean squared error loss
[Equation]
over uniformly sampled times t ∈ [0 , 1] .
This reduction of generation to denoising can be justiﬁed as optimizing a weighted variational lower bound on the data log likelihood under the diffusion model, or as a form of denoising score matching [56, 47, 22, 28].
In practice, we use the (cid:15) -prediction parameterization, deﬁned as ˆ x θ ( z t ) =
( z t − σ t (cid:15) θ ( z t )) /α
t , and train (cid:15) θ using a mean squared error in (cid:15) space with t sampled according to a cosine schedule [37].
This corresponds to a particular weighting w ( λ t ) for learning a scaled score estimate (cid:15) θ ( z t ) ≈ − σ t ∇ z t log p ( z t ) , where p ( z t ) is the true density of z t under x ∼ p ( x ) [22, 28, 48].
We also train using the v -prediction parameterization for certain models [42].
Sampling We use a variety of diffusion model samplers in this work.
One is the discrete time ancestral sampler [22] with sampling variances derived from lower and upper bounds on reverse process entropy [46, 22, 37].
To deﬁne this sampler, ﬁrst note that the forward process can be described in reverse as q ( z s | z t , x ) =
N ( z s ; ˜ µ s | t ( z t , x ) ,
˜ σ 2 s | t I ) (noting s < t ),
where
[Equation]
[Equation]
and ˜
[Equation]
Starting at z 1 ∼ N ( 0 , I ) , the ancestral sampler follows the rule
[Equation]
where (cid:15) is standard Gaussian noise, γ is a hyperparameter that controls the stochasticity of the sampler [37], and s, t follow a uniformly spaced sequence from 1 to 0.
Another sampler, which we found especially effective with our new method for conditional generation (Section 3.1), is the predictor-corrector sampler [48].
Our version of this sampler alternates between the ancestral sampler step (4) and a Langevin correction step of the form
[Equation]
where δ is a step size which we ﬁx to 0 . 1 here, and (cid:15) (cid:48) is another independent sample of standard
Gaussian noise.
The purpose of the Langevin step is to help the marginal distribution of each z s generated by the sampler to match the true marginal under the forward process starting at x ∼ p ( x ) .
In the conditional generation setting, the data x is equipped with a conditioning signal c , which may represent a class label, text caption, or other type of conditioning.
To train a diffusion model to ﬁt p ( x | c ) , the only modiﬁcation that needs to be made is to provide c to the model as ˆ x θ ( z t , c ) .
Improvements to sample quality can be obtained in this setting by using classiﬁer-free guidance [20].
This method samples using adjusted model predictions ˜ (cid:15) θ , constructed via
[Equation]
where w is the guidance strength , (cid:15) θ
( z t , c ) = 1 σ t ( z t − ˆ x θ ( z t , c )) is the regular conditional model prediction, and (cid:15)
θ ( z t ) is a prediction from an unconditional model jointly trained with the conditional model (if c consists of embedding vectors, unconditional modeling can be represented as c = 0 ).
For w > [-
-]0 this adjustment has the effect of over-emphasizing the effect of conditioning on the signal c , which tends to produce samples of lower diversity but higher quality compared to sampling from the regular conditional model [20].
The method can be interpreted as a way to guide the samples towards areas where an implicit classiﬁer p ( c | z t ) has high likelihood, and is an adaptation of the explicit classiﬁer guidance method proposed by [16].
Video diffusion models
Our approach to video generation using diffusion models is to use the standard diffusion model formalism described in Section 2 with a neural network architecture suitable for video data.
Each of our models is trained to jointly model a ﬁxed number of frames at a ﬁxed spatial resolution.
To extend sampling to longer sequences of frames or higher spatial resolutions, we will repurpose our models with a conditioning technique described later in Section 3.1.
In prior work on image modeling, the standard architecture for ˆ x θ in an image diffusion model is a
U-Net [38, 44], which is a neural network architecture constructed as a spatial downsampling pass followed by a spatial upsampling pass with skip connections to the downsampling pass activations.
The network is built from layers of 2D convolutional residual blocks, for example in the style of the Wide ResNet [65], and each such convolutional block is followed by a spatial attention block [55, 58, 11].
Conditioning information, such as c and λ t , is provided to the network in the form of an embedding vector added into each residual block (we ﬁnd it helpful for our models to process these embedding vectors using several MLP layers before adding).
We propose to extend this image diffusion model architecture to video data, given by a block of a ﬁxed number of frames, using a particular type of 3D U-Net [13] that is factorized over space and time.
First, we modify the image model architecture by changing each 2D convolution into a space-only 3D convolution, for instance, we change each 3x3 convolution into a 1x3x3 convolution (the ﬁrst axis indexes video frames, the second and third index the spatial height and width).
The attention in each spatial attention block remains as attention over space; i.e., the ﬁrst axis is treated as a batch axis.
Second, after each spatial attention block, we insert a temporal attention block that performs attention over the ﬁrst axis and treats the spatial axes as batch axes.
We use relative position embeddings [45] in each temporal attention block so that the network can distinguish ordering of frames in a way that does not require an absolute notion of video time.
We visualize the model architecture in Fig.
( z t , c ,λ t ) N 2 , M( N 2 ) 2 , M( NK ) 2 , M K ( NK ) 2 , M K ( NK ) 2 , 2 × M K ( N 2 ) 2 , 2 × M
N 2 , 2 ×
Mˆ x
The downsampling/upsampling blocks adjust the spatial input resolution height × width by a factor of 2 through each of the K blocks.
The channel counts are speciﬁed using channel multipliers M 1 ,
M 2 , ..., M K , and the upsampling pass has concatenation skip connections to the downsampling pass.
The use of factorized space-time attention is known to be a good choice in video transformers for its computational efﬁciency [2, 5, 21].
An advantage of our factorized space-time architecture, which is unique to our video generation setting, is that it is particularly straightforward to mask the model to run on independent images rather than a video, simply by removing the attention operation inside each time attention block and ﬁxing the attention matrix to exactly match each key and query vector at each video timestep.
The utility of doing so is that it allows us to jointly train the model on both video and image generation.
We ﬁnd in our experiments that this joint training is important for sample quality (Section 4).
3.1 Reconstruction-guided sampling for improved conditional generation
The videos we consider modeling typically consist of hundreds to thousands of frames, at a frame rate of at least 24 frames per second.
To manage the computational requirements of training our models, we only train on a small subset of say 16 frames at a time.
However, at test time we can generate longer videos by extending our samples.
For example, we could ﬁrst generate a video x a ∼ p θ ( x ) consisting of 16 frames, and then extend it with a second sample x b ∼ p θ ( x b | x a ) .
If x b consists of frames following x a , this allows us to autoregressively extend our sampled videos to arbitrary lengths, which we demonstrate in Section 4.3.3.
Alternatively, we could choose x a to represent a video of lower frame rate, and then deﬁne x b to be those frames in between the frames of x a .
This allows one to then to upsample a video temporally, similar to how [34] generate high resolution images through spatial upsampling.
Both approaches require one to sample from a conditional model, p θ ( x b | x a ) .
This conditional model could be trained explicitly, but it can also be derived approximately from our unconditional model p θ ( x ) by imputation, which has the advantage of not requiring a separately trained model.
For example, [48] present a general method for conditional sampling from a jointly trained diffusion model p θ ( x = [ x a , x b ]) :
In their approach to sampling from p θ ( x b | x a ) , the sampling procedure for updating z b s is unchanged from the standard method for sampling from p
θ ( z s | z t ) , with z s =
[ z a s , z b s ] , but the samples for z a s are replaced by exact samples from the forward process, q ( z a s | x a ) , at each iteration.
The samples z a s then have the correct marginal distribution by construction, and the samples z b s will conform with z a s through their effect on the denoising model ˆ x θ ([ z a t , z b t ]) .
Similarly, we could sample z a s from q ( z a s | x a , z a t ) , which follows the correct conditional distribution in addition to the correct marginal.
We will refer to both of these approaches as the replacement method for conditional sampling from diffusion models.
When we tried the replacement method to conditional sampling, we found it to not work well for our video models:
Although samples x b looked good in isolation, they were often not coherent with x a .
This is caused by a fundamental problem with this replacement sampling method.
That is, the latents z b s are updated in the direction provided by ˆ x b θ ( z t ) ≈
E q [ x b | z t ] , while what is needed instead is E q [ x b | z t , x a ] .
Writing this in terms of the score of the data distribution, we get
E q [ x b | z t , x a ] =
E q [ x b | z t ] + ( σ 2 t /α t ) ∇ z bt log q ( x a | z t ) , where the second term is missing in the replacement method.
Assuming a perfect denoising model, plugging in this missing term would make conditional sampling exact.
Since q ( x a | z t ) is not available in closed form, however, we instead propose to approximate it using a Gaussian of the form q ( x a | z t )
≈ N
[ˆ x a θ ( z t ) , ( σ 2 t /α 2 t ) I ] , where ˆ x a θ ( z t ) is a reconstruction of the conditioning data x a provided by our denoising model.
Assuming a perfect model, this approximation becomes exact as t → 0 , and empirically we ﬁnd it to be good for larger t also.
Plugging in the approximation, and adding a weighting factor w r , our proposed method to conditional sampling is a variant of the replacement method with an adjusted denoising model, ˜ x bθ , deﬁned by
[Equation]
The additional gradient term in this expression can be interpreted as a form of guidance [16, 20] based on the model’s reconstruction of the conditioning data, and we therefore refer to this method as reconstruction-guided sampling , or simply reconstruction guidance .
Like with other forms of guidance, we ﬁnd that choosing a larger weighting factor, w r > 1 , tends to improve sample quality.
We empirically investigate reconstruction guidance in Section 4.3.3, where we ﬁnd it to work surprisingly well, especially when combined with predictor-corrector samplers using Langevin diffusion [48].
Reconstruction guidance also extends to the case of spatial interpolation (or super-resolution), in which the mean squared error loss is imposed on a downsampled version of the model prediction, and backpropagation is performed through this downsampling.
In this setting, we have low resolution ground truth videos x a (e.g. at the 64x64 spatial resolution), which may be generated from a low resolution model, and we wish to upsample them into high resolution videos (e.g. at the 128x spatial resolution) using an unconditional high resolution diffusion model ˆ x θ .
To accomplish this, we adjust the high resolution model as follows:
[Equation]
where ˆ x aθ ( z t ) is our model’s reconstruction of the low-resolution video from z t , which is obtained by downsampling the high resolution output of the model using a differentiable downsampling algorithm such as bilinear interpolation.
Note that it is also possible to simultaneously condition on low resolution videos while autoregressively extending samples at the high resolution using the same reconstruction guidance method.
In Fig.
2, we show samples of this approach for extending 16x64x low resolution samples at frameskip 4 to 64x128x128 samples at frameskip 1 using a 9x128x diffusion model.
Experiments
We report our results on video diffusion models for unconditional video generation (Section 4.1), conditional video generation (video prediction) (Section 4.2), and text-conditioned video generation (Section 4.3).
We evaluate our models using standard metrics such as FVD [54], FID [19], and IS [43]; details on evaluation are provided below alongside each benchmark.
Samples and additional results are provided at https://video-diffusion.github.io/ .
Architecture hyperparameters, training details, and compute resources are listed in Appendix A.
4.1 Unconditional video modeling
To demonstrate our approach on unconditional generation, we use a popular benchmark of Soomro et al.
[49] for unconditional modeling of video.
The benchmark consists of short clips of people performing one of 101 activities, and was originally collected for the purpose of training action recognition models.
We model short segments of 16 frames from this dataset, downsampled to a spatial resolution of 64x64.
In Table 1 we present perceptual quality scores for videos generated by our model, and we compare against methods from the literature, ﬁnding that our method strongly improves upon the previous state-of-the-art.
We use the data loader provided by TensorFlow Datasets [1] without further processing, and we train on all 13,320 videos.
Similar to previous methods, we use the C3D network [51] 2 for calculating
FID and IS, using 10,000 samples generated from our model.
C3D internally resizes input data to the 112x112 spatial resolution, so perceptual scores are approximately comparable even when the data is sampled at a different resolution originally. As discussed by [64], methods in the literature are unfortunately not always consistent in the data preprocessing that is used, which may lead to small differences in reported scores between papers.
The Inception Score we calculate for real data ( ≈ 60 ) is consistent with that reported by [26], who also report a higher real data Inception score of ≈ 90 for data sampled at the 128x128 resolution, which indicates that our 64x64 model might be at a disadvantage compared to works that generate at a higher resolution.
Nevertheless, our model obtains the best perceptual quality metrics that we could ﬁnd in the literature
[Table]
4.2 Video prediction
A common benchmark task for evaluating generative models of video is video prediction , where the model is given the ﬁrst frame(s) of a video and is asked to generate the remainder.
Models that do well on this conditional generation task are usually trained explicitly for this conditional setting, for example by being autoregressive across frames.
Although our models are instead only trained unconditionally, we can adapt them to the video prediction setting by using the guidance method proposed in section 3.1.
Here we evaluate this method on two popular video prediction benchmarks, obtaining state-of-the-art results.
BAIR Robot Pushing We evaluate video prediction performance on BAIR Robot Pushing [17], a standard benchmark in the video literature consisting of approximately 44000 videos of robot pushing motions at the 64x64 spatial resolution.
Methods for this benchmark are conditioned on 1 frame and generate the next 15.
Results are listed in Table 2.
Following the evaluation protocol of [4] and others, we calculate FVD [54] using the I3D network [8] by comparing 100 × 256 model samples against the 256 examples in the evaluation set.
Kinetics-600 We additionally evaluate video prediction performance on the Kinetics-600 benchmark [27, 9].
Kinetics-600 contains approximately 400 thousand training videos depictingdifferent activities.
We train unconditional models on this dataset at the 64 × 64 resolution and evaluate on 50 thousand randomly sampled videos from the test set, where we condition on a randomly sampled subsequence of 5 frames and generate the next 11 frames.
Like previous works, we calculate
FVD and Inception Score using the I3D network [8].
See Table 3 for results.
In our reported results we sample test videos without replacement, and we use the same randomly selected subsequences for generating model samples and for deﬁning the ground truth, since this results in the lowest bias and variance in the reported FVD metric.
However, from personal communication we learned that [33, 14] instead sampled with replacement , and used a different random seed when sampling the ground truth data.
We ﬁnd that this way of evaluating raises the FVD obtained by our model slightly, from 16 .
2 to 16 .
Inception Score is unaffected.
[Table] [Table]
4.3 Text-conditioned video generation
The remaining experiments reported are on text-conditioned video generation.
In this text-conditioned video generation setting, we employ a dataset of 10 million captioned videos
[Table]
and we condition the diffusion model on captions in the form of BERT-large embeddings [15] processed using attention pooling. We consider two model sizes: a small model for the joint training ablation [Table]
and a large model for generating the remaining results (both architectures are described in detail in Appendix A) [Table] and we explore the effects of joint video-image training, classiﬁer-free guidance [Table]
and our newly proposed reconstruction guidance method for autoregressive extension and simultaneous spatial and temporal super-resolution.
We report the following metrics in this section on 4096 samples: the video metric
FVD [Table] and the Inception-based image metrics FID and IS measured by averaging activations across frames (FID/IS-avg) and by measuring the ﬁrst frame only (FID/IS-ﬁrst).
For FID and FVD, we report two numbers which are measured against the training and validation sets, respectively.
For IS, we report two numbers which are averaged scores samples, respectively. Both models are conditioned on the text prompt. In this , the text prompt, low resolution frames, and high resolution frames are visualized in sequence.
See Fig.
5 for more samples.
4.3.1 Joint training on video and image modeling
As described in Section 3, one of the main advantages of our video architecture is that it allows us to easily train the model jointly on video and image generative modeling objectives.
To implement this joint training, we concatenate random independent image frames to the end of each video sampled from the dataset, and we mask the attention in the temporal attention blocks to prevent mixing information across video frames and each individual image frame.
We choose these random independent images from random videos within the same dataset; in future work we plan to explore the effect of choosing images from other larger image-only datasets.
Table 4 reports results for an experiment on text-conditioned 16x64x64 videos, where we consider training on an additional 0, 4, or 8 independent image frames per video.
One can see clear improvements in video and image sample quality metrics as more independent image frames are added.
Adding independent image frames has the effect of reducing variance of the gradient at the expense of some bias for the video modeling objective, and thus it can be seen as a memory optimization to ﬁt more independent examples in a batch.
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
4.3.2 Effect of classiﬁer-free guidance
Table 5 reports results that verify the effectiveness of classiﬁer-free guidance [20] on text-to-video generation.
As expected, there is clear improvement in the Inception Score-like metrics with higher guidance weight, while the FID-like metrics improve and then degrade with increasing guidance weight.
Similar ﬁndings have been reported on text-to-image generation [36].
Figure 3 shows the effect of classiﬁer-free guidance [20] on a text-conditioned video model.
Similar to what was observed in other work that used classiﬁer-free guidance on text-conditioned image generation [36] and class-conditioned image generation [20, 16], adding guidance increases the sample ﬁdelity of each individual image and emphases the effect of the conditioning signal.
[Table]
4.3.3 Autoregressive video extension for longer sequences
In Section 3.1 we proposed the reconstruction guidance method for conditional sampling from diffusion models, an improvement over the replacement method of [48].
In Table 6 we present results on generating longer videos using both techniques, and ﬁnd that our proposed method indeed improves over the replacement method in terms of perceptual quality scores.
Figure 4 shows the samples of our reconstruction guidance method for conditional sampling compared to the replacement method (Section 3.1) for the purposes of generating long samples in a blockautoregressive manner (Section 4.3.3).
The samples from the replacement method clearly show a lack of temporal coherence, since frames from different blocks throughout the generated videos appear to be uncorrelated samples (conditioned on c ).
The samples from the reconstruction guidance method, by contrast, are clearly temporally coherent over the course of the entire autoregressive generation process.
Figure 2 additionally shows samples of using the reconstruction guidance method to simultaneously condition on low frequency, low resolution videos while autoregressively extending temporally at a high resolution
[Table]
[Table]
Related work
Prior work on video generation has usually employed other types of generative models, notably, autoregressive models, VAEs, GANs, and normalizing ﬂows [e.g. 3, 4, 32, 30, 14, 59, 62, 57].
Related work on model classes similar to diffusion models includes [25, 24].
Concurrent work [63] proposes a diffusion-based approach to video generation that uses an image diffusion model to predict each individual frame within a RNN temporal autoregressive model.
Our video diffusion model, by contrast, jointly models entire videos (blocks of frames) using a 3D video architecture with interleaved spatial and temporal attention, and we extend to long sequence lengths by ﬁlling in frames or autoregressive temporal extension.
Conclusion
We have introduced diffusion models for video modeling, thus bringing recent advances in generative modeling using diffusion models to the video domain.
We have shown that with straightforward extensions of conventional U-Net architectures for 2D image modeling to 3D space-time, with factorized space-time attention blocks, one can learn effective generative models for video data using the standard formulation of the diffusion model.
This includes unconditional models, text-conditioned models, and video prediction models.
We have additionally demonstrated the beneﬁts of joint image-video training and classiﬁer-free guidance for video diffusion models on both video and image sample quality metrics, and we also introduced a new reconstruction-guided conditional sampling method that outperforms existing replacement or imputation methods for conditional sampling from unconditionally trained models.
Our reconstruction guidance method can generate long sequences using either frame interpolation (or temporal super-resolution) or extrapolation in an auto-regressive fashion, and also can perform spatial super-resolution.
We look forward to investigating this method in a wider variety of conditioning settings.
Our goal with this work is to advance research on methods in generative modeling, and our methods have the potential to positively impact creative downstream applications.
As with prior work in generative modeling, however, our methods have the potential for causing harmful impact and could enhance malicious or unethical uses of generative models, such as fake content generation, harassment, and misinformation spread, and thus we have decided not to release our models.
Like all generative models, our models reﬂect the biases of their training datasets and thus may require curation to ensure fair results from sampling.
In particular, our text-to-video models inherit the challenges faced by prior work on text-to-image models, and our future work will involve auditing for forms of social bias, similar to [6, 7, 50, 12] for image-to-text and image labeling models.
We see our work as only a starting point for further investigation on video diffusion models and investigation into their societal implications, and we will aim to explore benchmark evaluations for social and cultural bias in the video generation setting and make the necessary research advances to address them.
A.1 UCF
[Table]
A.2 BAIR Robot Pushing
[Table]
A.3 Kinetics
[Table]
A.4 Text-to-video Small
[Table]
model
[Table]
Large
[Table]