Flexible Diffusion Modeling of Long Videos
Abstract
We present a framework for video modeling based on denoising diffusion probabilistic models that produces long-duration video completions in a variety of realistic environments.
We introduce a generative model that can at test-time sample any arbitrary subset of video frames conditioned on any other subset and present an architecture adapted for this purpose.
Doing so allows us to efficiently compare and optimize a variety of schedules for the order in which frames in a long video are sampled and use selective sparse and long-range conditioning on previously sampled frames.
We demonstrate improved video modeling over prior work on a number of datasets and sample temporally coherent videos over 25 minutes in length.
We additionally release a new video modeling dataset and semantically meaningful metrics based on videos generated in the CARLA self-driving car simulator.
Introduction
Generative modeling of photo-realistic videos is at the frontier of what is possible with deep learning on currently-available hardware.
Although related work has demonstrated modeling of short photorealistic videos (e g. 30 frames [36], 48 frames [6] or 64 frames [16]), generating longer videos that are both coherent and photo-realistic remains an open challenge.
A major difficulty is scaling: photorealistic image generative models [4, 8] are already close to the memory and processing limits of modern hardware.
A long video is at very least a concatenation of many photorealistic frames, implying resource requirements, long-range coherence notwithstanding, that scale with frame count.
Attempting to model such long-range coherence makes the problem harder still, especially because in general every frame can have statistical dependencies on other frames arbitrarily far back in the video.
Unfortunately fixed-lag autoregressive models impose unrealistic conditional independence assumptions (the next frame being independent of frames further back in time than the autoregressive lag is problematic for generating videos with long-range coherence).
And while deep generative models based on recurrent neural networks (RNN) theoretically impose no such conditional independence assumptions, in practice they must be trained over short sequences [12, 26] or with truncated gradients [31].
Despite this, some RNN-based video generative models have demonstrated longer-range coherence, albeit without yet achieving convincing photorealistic video generation [26, 3, 7, 20, 2].
In this work we embrace the fact that finite architectures will always impose conditional independences.
The question we ask is: given an explicit limit K on the number of video frames we can jointly model, how can we best allocate these frames to generate a video of length N > K ?
One option is to use the previously-described autoregressive model but, if K = N/ 4 , we could instead follow Ho et al.
[16] by training two models: one which first samples every 4th frame in the video, and another which (in multiple stages) infills the remaining frames conditioned on those.
To enable efficient exploration of the space of such sampling schemes, we propose a flexible architecture based on the denoising diffusion probabilistic model (DDPM) framework.
This can sample any subset of video frames conditioned on observed values of any other subset of video frames.
It therefore lets us explore a wide variety of previously untested sampling schemes while being easily repurposed for different generation tasks such as unconditional generation, video completion, and generation of videos of different lengths.
Since our model can be flexibly applied to sample any frames given any others we call it a Flexible Diffusion Model, or FDM.
Contributions (1) At the highest level, we claim to have concurrently developed one of the first denoising diffusion probabilistic model (DDPM)-based video generative models [16, 40].
To do so we augment a previously-used DDPM image architecture [15, 22] with a temporal attention mechanism including a novel relative (frame) position encoding network.
The principal contribution of this paper, regardless, is a “meta-learning” training objective that encourages learning of a video generative model that can (a) be flexibly conditioned on any number of frames (up to computational resource constraints) at any time in the past and future and (b) be flexibly marginalized (to achieve this within computational resource constraints).
(3) We demonstrate that our model can be used to efficiently explore the space of resource constrained video generation schemes, leading to improvements over prior work on several long-range video modeling tasks.
(4) Finally, we release a new autonomous driving video dataset along with a new video generative model performance metric that captures semantics more directly than the visual quality and comparison metrics currently in widespread use.
Sampling long videos
Our goal in this paper is to sample coherent photo-realistic videos v with thousands of frames (see
Fig. 1).
To sample an arbitrarily long video with a generative model that can sample or condition on only a small number of frames at once, we must use a sequential procedure.
The simplest example of this is an autoregressive scheme, an example of which is shown in Fig.
2a for a video completion task.
In this example it takes seven stages to sample a complete video, in that we must run the generative model’s sampling procedure seven times.
At each stage three frames are sampled conditioned on the immediately preceding four frames.
This scheme is appealing for its simplicity but imposes a strong assumption that, given the set of four frames that are conditioned on at a particular stage, all frames that come afterwards are conditionally independent of all frames that came before.
This restriction can be partially ameliorated with the sampling scheme shown in Fig.
2b where, in the first three stages, every second frame is sampled and then, in the remaining four stages, the remaining frames are infilled.
One way to implement this would be to train two different models operating at the two different temporal resolutions.
In the language of Ho et al.
[16], who use a similar approach, sampling would be carried out in the first three stages by a “frameskip-2” model and, in the remaining stages, by a “frameskip-1” model.
Both this approach and the autoregressive approach are examples of what we call sampling schemes .
More generally, we characterize a sampling scheme as a sequence of tuples [( X s , Y s )]
Ss =1 , each containing a vector X s of indices of frames to sample and a vector Y s of indices of frames to condition on for stages s = 1 , . . .
, S .
Algorithm
For unconditional generation, the input v can be a tensor of zeros.
For conditional generation, the observed input frames should contain their observed values.
[Figure] [Figure]
Algorithm 1 lays out how such a sampling scheme is used to sample a video.
If the underlying generative model is trained specifically to model sequences of consecutive frames, or sequences of regularly-spaced frames, then the design space for sampling schemes compatible with these models is severely constrained.
In this paper we take a different approach.
We design and train a generative model to sample any arbitrarily-chosen subset of video frames conditioned on any other subset and train it using an entirely novel distribution of such tasks.
In short, our model is trained to generate frames for any choice of X and Y .
The only constraint we impose on our sampling schemes is therefore a computational consideration that |X s | + |Y s | ≤ K for all s but, to generate meaningful videos, any valid sampling scheme must also satisfy two more constraints: (1) all frames are sampled at at least one stage and (2) frames are never conditioned upon before they are sampled.
Such a flexible generative model allows us to explore and use sampling schemes like those in Fig.
2c and Fig.
2d.
We find in our experiments that the best video sampling scheme is dataset dependent.
Accordingly, we have developed methodology to optimize such sampling schemes in a dataset dependent way, leading to improved video quality as measured by the Fréchet Video Distance [33] among other metrics.
We now review conditional DDPMs (Section 3), before discussing the FDM’s architecture, the specific task distribution used to train it, and the choice and optimization of sampling schemes in Section 4.
A review of conditional denoising diffusion probabilistic models
Denoising diffusion probabilistic models, or DDPMs [28, 15, 22, 30], are a class of generative model for data x , which throughout this paper will take the form of a 4-dimensional tensor representing multiple video frames.
We will describe the conditional extension [32], in which the modeled x is conditioned on observations y .
DDPMs simulate a diffusion process which transforms x to noise, and generate data by learning the probabilistic inverse of the diffusion process.
The diffusion process happens over timesteps 0 , . .
, T such that x 0 = x is data without noise, x 1 has a very small amount of noise added, and so on until x T is almost independent of x 0 and approximates a random sample from a unit Gaussian.
In the diffusion process we consider, the distribution over x t depends only on x t − 1 :
[Equation]
Hyperparameters α 1 , . .
. , α T are chosen to all be close to but slightly less than 1 so that the amount of noise added at each step is small.
The combination of this diffusion process and a data distribution
[Figure]
[Figure]
q ( x 0 , y ) (recalling that x 0 = x ) defines the joint distribution
[Equation]
DDPMs work by “inverting” the diffusion process: given values of x t and y a neural network is used to parameterize p θ ( x t − 1 | x t , y ) , an approximation of q ( x t − 1 | x t , y ) .
This neural network lets us draw samples of x 0 by first sampling x T from a unit Gaussian (recall that the diffusion process was chosen so that q ( x T ) is well approximated by a unit Gaussian), and then iteratively sampling x t − 1 ∼ p θ ( ·| x t , y ) for t
=
T, T − 1 , . .
The joint distribution of sampled x 0: T given y is
[Equation]
where p ( x T ) is a unit Gaussian that does not depend on θ .
Training the conditional DDPM therefore involves fitting p θ ( x t − 1 | x t , y ) to approximate q ( x t − 1 | x t , y ) for all choices of t , x t , and y .
Several observations have been made in recent years which simplify the learning of p θ ( x t − 1 | x t , y ) .
Sohl- et al.
[28] showed that when α t is close to 1, p θ ( x t − 1 | x t ) is approximately Gaussian [28].
Furthermore, Ho et al.
[15] showed that this Gaussian’s variance can be modeled well with a non-learned function of t , and that a good estimate of the Gaussian’s mean can be obtained from a “denoising model” as follows.
Given data x 0 and unit Gaussian noise ϵ , the denoising model (in the form of a neural network) is fed “noisy” data x t := √ ˜
α t x 0 + √ 1 − ˜ α t ϵ and trained to recover ϵ via a mean squared error loss.
The parameters ˜ α t := (cid:81) ti =
1 α i are chosen to ensure that the marginal distribution of x t given x 0 is q ( x t | x 0 ) as derived from Eq.
Given a weighting function λ ( t ) , the denoising loss is
[Equation]
with x t = (cid:112) ˜
α
[Equation]
x
[Equation]
+ (cid:112)
[Equation]
˜ α t ϵ
[Equation]
The mean of p θ ( x t − 1 | x t , y ) is obtained from the denoising model
’s output ϵ θ ( x t , y , t ) asα t x t − 1 − α t √ 1 − ˜ α t ϵ θ ( x t , y , t ) .
If the weighting function λ ( t ) is chosen appropriately, optimising Eq.
(4) is equivalent to optimising a lower-bound on the data likelihood under p θ .
In practice, simply setting λ ( t ) :
= 1 for all t can produce more visually compelling results in the image domain [15].
In our proposed method, as in Tashiro et al.
[32], the shapes of x 0 and y sampled from q ( · ) vary.
This is because we want to train a model which can flexibly adapt to e.g. varying numbers of observed frames.
To map Eq.
(4) to this scenario, note that both x 0 and y implicitly contain information about which frames in the video they represent (via the index vectors X and Y introduced in the previous section).
This information is used inside the neural network ϵ θ ( x t , y , t ) so that interactions between frames can be conditioned on the distance between them (as described in the following section) and also to ensure that the sampled noise vector ϵ has the same shape as x 0 .
1: X := {} ; Y := {} 2: while True do 3: n group ∼ UniformDiscrete (1 , K ) 4: s group ∼ LogUniform (1 , ( N − 1) /n group ) 5: x group ∼ Uniform (0 , N − ( n group − 1) · s group ) 6: o group ∼ Bernoulli (0 . 5) 7:
G :=
{⌊ x group + s group · i ⌋| i ∈ { 0 , . . .
, n group − 1 }} \ X \ Y 8: if |X| + |Y| + |G| > K then 9: return set2vector ( X ) , set2vector ( Y ) 10: else if |X| = 0 or o group = 0 then 11:
X :=
X ∪ G 12: else 13:
Y :
=
Y ∪ G
The FDM’s training objective forces it to work well for any ( X , Y ) pair from this broad distribution.
Training procedure and architecture
Training task distribution Different choices of latent and observed indices X
and Y can be regarded as defining different conditional generation tasks.
In this sense, we aim to learn a model which can work well on any task (i.e. any choice of X and Y ) and so we randomly sample these vectors of indices during training.
We do so with the distribution u ( X , Y ) described in Fig. 4.
This provides a broad distribution covering many plausible test-time use cases while still providing sufficient structure to improve learning (see ablation in Section 6 and more details in Appendix C).
To cope with constrained computational resources, the distribution is designed such that |X| + |Y| is upper-bounded by some pre-specified K .
Sampling from q ( x 0 , y ) in Eq.
(4) is then accomplished by randomly selecting both a full training video v and indices X , Y ∼ u ( · , · ) .
We then extract the specified frames x = v [ X ] and y = v [ Y ] (where we use v [ X ] to denote the concatenation of all frames in v with indices in X and and v [ Y ] similarly).
Architecture DDPM image models [15, 22] typically use a U-net architecture [24].
Its distinguishing feature is a series of spatial downsampling layers followed by a series of upsampling layers, and these are interspersed with convolutional res-net blocks [14] and spatial attention layers.
Since we require an architecture which operates on 4-D video tensors rather than 3-D image tensors we add an extra frame dimension to its input, output and hidden state, resulting in the architecture shown on the right of Fig.
We create the input to this architecture as a concatenation x t ⊕ y , adding an extra input channel which is all ones for observed frames and all zeros for latent frames.
For
RGB video, the input shape is therefore ( K, image height , image width , 4) .
Since the output should have the same shape as x t we only return outputs corresponding to the latent frames, giving output shape ( |X| , image height , image width , 3) .
We run all layers from the original model (including convolution, resizing, group normalization, and spatial attention) independently for each of the K frames.
To allow communication between the frames, we add a temporal attention layer after each spatial attention layer, described in more detail in the appendix.
The spatial attention layer allows each spatial location to attend to all other spatial locations within the same frame, while the temporal attention layer allows each spatial location to attend to the same spatial location across all other frames.
This combination of a temporal attention layer with a spatial attention layer is sometimes referred to as factorized attention [32, 16].
We found that, when using this architecture in conjunction with our meta-learning approach, performance could be improved by using a novel form of relative position encoding [27, 38].
This is included in our released source code but we leave its exposition to the supplementary material.
Training batch padding Although the size |X ⊕ Y| of index vectors sampled from our training distribution is bounded above by K , it can vary.
To fit examples with various sizes of index vectors into the same batch, one option would be to pad them all to length K with zeros and use masks so that the zeros cannot affect the loss.
This, however, would waste computation on processing tensors of zeros.
We instead use this computation to obtain a lower-variance loss estimate by processing additional data with “training batch padding”.
This means that, for training examples where |X ⊕ Y| <
K , we concatenate frames uniformly sampled from a second video to increase the length along the frame-dimension to K .
Masks are applied to the temporal attention mechanisms so that frames from different videos cannot attend to eachother and the output for each is the same as that achieved by processing the videos in different batches.
Sampling schemes Before describing the sampling schemes we explore experimentally, we emphasize that the relative performance of each is dataset-dependent and there is no single best choice.
A central benefit of FDM is that it can be used at test-time with different sampling schemes without retraining.
Our simplest sampling scheme, Autoreg , samples ten consecutives frames at each stage conditioned on the previous ten frames.
Long-range is similar to Autoreg but conditions on only the five most recent frames as well as five of the original 36 observed frames.
Hierarchy-2 uses a multi-level sampling procedure.
In the first level, ten evenly spaced frames spanning the non-observed portion of the video are sampled (conditioned on ten observed frames).
In the second level, groups of consecutive frames are sampled conditioned on the closest past and future frames until all frames have been sampled.
Hierarchy-3 adds an intermediate stage where several groups of variables with an intermediate spacing between them are sampled.
We include adaptive hierarchy-2, abbreviated
Ad.
hierarchy-2 , as a demonstration of a sampling scheme only possible with a model like FDM.
It samples the same frames at each stage as Hierarchy-2 but selects which frames to condition on adaptively at test-time with a heuristic aimed at collecting the maximally diverse set of frames, as measured by the pairwise LPIPS distance [41] between them.
Optimizing sampling schemes
An appealing alternative to the heuristic sampling schemes described in the previous paragraph would be to find a sampling scheme that is, in some sense, optimal for a given model and video generation/completion task.
While it is unclear how to tractably choose which frames should be sampled at each stage, we suggest that the frames to condition on at each stage can be chosen by greedily optimizing the diffusion model loss which, as mentioned in Section 3, is closely related to the data log-likelihood.
Given a fixed sequence of frames to sample at each stage [ X s ] Ss =1 we select Y s for each s to minimize Eq.
This is estimated using a set of 100 training videos and by iterating over 10 evenly-spaced values of t (which reduced variance relative to random sampling of t ).
See the appendix for further details.
We create two optimized sampling schemes: one with the same latent indices as Autoreg, and one with the same latent indices as Hierarchy-2.
We call the corresponding optimized schemes Opt.
autoreg and Opt.
hierarchy-2 .
CARLA Town01 Dataset
In addition to our methodological contributions, we propose a new video-modeling dataset and benchmark which provides an interpretable measure of video completion quality.
The dataset consists of videos of a car driving with a first-person view, produced using the CARLA autonomous driving simulator [9].
All 408 training and 100 test videos (of length 1000 frames and resolution 128 × 128 ) are produced within a single small town, CARLA’s Town01.
As such, when a sufficiently expressive video model is trained on this dataset it memorizes the layout of the town and videos sampled from the model will be recognisable as corresponding to routes travelled within the town.
We train a regression model in the form of a neural network which maps with high accuracy from any single rendered frame to ( x, y ) coordinates representing the car’s position.
Doing so allows us to plot the routes corresponding to sampled videos (see left of Fig. 5) and compute semantically-meaningful yet quantitative measures of the validity of these routes.
Specifically, we compute histograms of speeds, where each speed is estimated by measuring the distance between the regressed locations for frames spaced ten apart (1 second at the dataset’s frame rate).
Sampled videos occasionally “jump” between disparate locations in the town, resulting in unrealistically large estimated speeds.
To measure the frequency of these events for each method, we compute the percentage of our point-speed estimates that exceed a threshold of 10 m/s (the dataset was generated with a maximum simulated speed of 3 m/s).
We report this metric as the outlier percentage (OP).
After filtering out these outliers, we compute the Wasserstein distance (WD) between the resulting empirical distribution and that of the original dataset, giving a measure of how well generated videos match the speed of videos in the dataset.
We release the CARLA Town 01 dataset along with code and our trained regression model to allow future comparisons.
Experiments
We perform our main comparisons on the video completion task.
In keeping with Saxena et al.
[26], we condition on the first 36 frames of each video and sample the remainder.
We present results on three datasets: GQN-Mazes [10], in which videos are 300 frames long; MineRL Navigate [13, 26] (which we will from now on refer to as simply MineRL), in which videos are 500 frames long; and the CARLA Town01 dataset we release, for which videos are 1000 frames long.
We train FDM in all cases with the maximum number of represented frames K = 20 .
We host non-cherry-picked video samples (both conditional and unconditional) from FDM and all baselines online 3 .
Speed (m/s) 0 .0 .1 .1 .2 .
D e n s i t y
Ad.
HierarchyGround Truth
Long-range
Comparison of sampling schemes The relative performance of different sampling schemes varies significantly between datasets as shown in Table 1.
We report Fréchet Video Distances (FVDs) [33], a measure of how similar sampled completions are to the test set, on all datasets.
In addition on GQN-Mazes we we report the accuracy metric [26], which classifies videos based on which rooms are visited and measures how often a completion is given the same class as the corresponding test video.
For CARLA Town01 we report the previously described percentage outliers (PO) and Wasserstein distance (WD) metrics.
We can broadly consider the aforementioned sampling schemes as either being in the “autoregressive” family (Autoreg and Long-range) or in the “hierarchical”
[Table]
family (the remainder).
Those in the hierarchical family achieve significantly better FVDs [33] on
GQN-Mazes.
Our samples in the appendix suggest that this is related to the autoregressive methods “forgetting” the colors of walls after looking away from them for a short time.
In contrast, for MineRL the autoregressive methods tend to achieve the best FVDs.
This may relate to the fact that trajectories in MineRL tend to travel in straight lines through procedurally-generated “worlds”[13, 26], limiting the number of long-range dependencies.
Finally on CARLA Town01 we notice qualitatively different behaviours from our autoregressive and hierarchical sampling schemes.
The hierarchical sampling schemes have a tendency to occasionally lose coherence and “jump” to different locations in the town.
This is reflected by higher outlier percentages (OP) in Table 1.
On the other hand the autoregressive schemes often stay stationary for unrealistically long times at traffic lights.
This is reflected in the histogram of speeds in Fig.
6, which has a larger peak around zero than the ground truth.
The high variance of the sampling scheme’s relative performance over different datasets points to a strength of our method, which need only be trained once and then used to explore a variety of sampling schemes.
Furthermore, we point out that the best FVDs in Table 1 on all datasets were obtained using sampling schemes that could not be implemented using models trained in prior work, or over evenly spaced frames.
Comparison with baselines The related work most relevant to ours is the concurrent work of
Ho et al.
[16], who model 64-frame videos using two trained DDPMs.
The first is a “frameskip-4” model trained to generate every fourth frame and the second is a “frameskip-1” model trained on sequences of nine consecutive frames and used to “fill in” the gaps between frames generated in the first stage.
To compare against this approach, which we denote VDM , we train both a “frameskip-4” and a “frameskip-1” model with architectures identical to our own.
4 Since VDM requires
two trained
DDPMs, we train it for more GPU-hours than FDM despite the fact that FDM is meta-learning over a far broader task distribution.
We also compare against TATS [11], which embeds videos into a discrete latent space before modelling them with a transformers, and the clockwork VAE ( CWVAE ) [26], a VAE-based model specifically designed to maintain long-range dependencies within video.
Both the diffusion-based methods, FDM and VDM, achieve significantly higher FVD scores than
TATS and CWVAE.
This may point toward the utility of diffusion models in general for modeling images and video.
Table 1 also makes clear the main benefit of FDM over VDM: although there is no sampling scheme for FDM which always outperforms VDM, there is at least one sampling scheme that outperforms it on each dataset.
This speaks to the utility of learning a flexible model like FDM that allows different sampling schemes to be experimented with after training.
Optimized sampling schemes As mentioned in Section 4, another advantage of FDM is that it makes possible a model- and dataset-specific optimization procedure to determine on which frames
[Table]
to
Table
Comparison with training on a single task Training a network with our distribution over training tasks could be expected to lead to worse performance on a single task than training specifically for that task.
To test whether this is the case, we train an ablation of FDM with training tasks exclusively of the type used in our Autoreg sampling scheme, i.e. “predict ten consecutive frames given the previous ten.”
Tested with the Autoreg sampling scheme, it obtained an FVD of 82 .
on GQN-Mazes and 234 on MineRL.
As expected given the specialization to a single task, this is better than when
FDM is run with the Autoreg sampling scheme (obtaining FVDs of 86 . 4 and 281 respectively).
Ablation on training task distribution To test how important our proposed structured training distribution is to FDM’s performance, we perform an ablation with a different task distribution that samples X and Y from uniform distributions instead of our proposed structured task distribution We provide full details in the appendix, but report here that switching away form our structured training distribution made the FVD scores worse on all five tested sampling schemes on both GQN-Mazes and MineRL.
The reduction in the average FVD was 31% on GQN-Mazes and 52% on MineRL.
This implies that our structured training distribution has a significant positive effect.
Related work
Some related work creates conditional models by adapting the sampling procedure of an unconditional
DDPM [30, 18, 21, 16].
These approaches require approximations and the more direct approach that we use (explcitly training a conditional DDPM) was shown to have benefits by Tashiro et al.
We consider further comparison of these competing approaches to be outside the focus of this work, which is on modeling a small portion of video frames at a time, essentially performing marginalization in addition to conditioning.
There are a number of approaches in the literature which use VAEs rather than DDPMs for video modelling.
Babaeizadeh et al.
[2] use a VAE model which predicts frames autoregressively conditioned on a global time-invariant latent variable.
A related approach by Denton and Fergus [7] also uses a
VAE with convolutional LSTM architectures in both the encoder and decoder.
Unlike Babaeizadeh et al.
[2] the prior is learned and a different latent variable is sampled for each frame.
Babaeizadeh et al.
[3] use a VAE with one set of latent variables per frame and inter-frame dependencies tracked by a two-layer LSTM.
Their architecture intentionally overfits to the training data, which when coupled with image augmentations techniques achieves SOTA on various video prediction tasks.
Kim et al.
[20] use a variational RNN [5] with a hierarchical latent space that includes binary indicator variables which specify how the video is divided into a series of subsequences.
Both Villegas et al.
[35] and Wichers et al.
[37] target long-term video prediction using a hierarchical variational LSTM architecture, wherein high-level features such as landmarks are predicted first, then decoded into low-level pixel space.
The two approaches differ in that Villegas et al.
[35] requires ground truth landmark labels, while [37] removes this dependence using an unsupervised adversarial approach.
Fully GAN-based video models have also been proposed [1, 6] but generally suffer from “low quality frames or low number of frames or both” [1].
Discussion
We have defined and empirically explored a new method for generating photorealistic videos with long-range coherence that respects and efficiently uses fixed, finite computational resources.
Our approach outperforms prior work on long-duration video modeling as measured by quantitative and semantically meaningful metrics and opens up several avenues for future research.
For one, similar to using DDPMs for image generation, our method is slow to sample from (it takes approximatelyminutes to generate a 300 frame video on a GPU).
Ideas for making sampling faster by decreasing the number of integration steps [25, 29, 39] could be applied to our video model.
On a different note, consider the datasets on which our artifact was trained.
In each there was a policy for generating the sequences of actions that causally led to the frame-to-frame changes in camera pose.
In MineRL the video was generated by agents that were trained to explore novel Minecraft worlds to find a goal block approximately 64 meters away [13].
The CARLA data was produced by a camera attached to an agent driven by a low level proportional–integral–derivative controller following waypoints laid down by a high level planner that was given new, random location goals to drive to intermittently.
In both cases our video model had no access to either the policy or the specific actions taken by these agents and, so, in a formal sense, our models integrate or marginalize over actions drawn from the stochastic policy used to generate the videos in the first place.
Near-term future work could involve adding other modalities (e.g. audio) to FDM as well as explicitly adding actions and rewards, transforming our video generative model into a vision-based world model in the reinforcement learning sense [17, 19].
Furthermore, we point out that FDM trained on CARLA
Town01 is in theory capable of creating 100-second videos conditioned on both the first and final frame.
Doing so can be interpreted as running a “visual” controller which proposes a path between a current state and a specified goal.
Preliminary attempts to run in FDM in this way yielded inconsistent results but we believe that this could be a fruitful direction for further investigation.
Acknowledgments
We would like to thank Inverted AI, and especially Alireza Morsali, for generating the CARLA
Town01 dataset.
We acknowledge the support of the Natural Sciences and Engineering Research
Council of Canada (NSERC), the Canada CIFAR AI Chairs Program and the Intel Parallel Computing Centers program.
Additional support was provided by UBC’s Composites Research Network (CRN) and Data Science Institute (DSI).
This research was enabled in part by technical support and computational resources provided by WestGrid (www.westgrid.ca), Compute Canada (www.computecanada.ca), and Advanced Research Computing at the University of British Columbia (arc.ubc.ca).
WH acknowledges support by the University of British Columbia’s Four Year Doctoral
Fellowship (4YF) program.
A Experimental details
[Table]
N/A
N/A CWVAE
1xA
N/A N/A
AblationsonGQN-Mazes
FDM(incl.
ablations)
1xA  40AblationsonMineRL
FDM(incl.
ablations)
1xA  40The total compute required for this project, including all training, evaluation, and preliminary runs, was roughly 3.5 GPU-years.
We used a mixture of NVIDIA RTX A5000s (on an internal cluster) and
NVIDIA A100s (from a cloud provider).
Due to the expensive nature of drawing samples from both FDM and our baselines, we compute all quantitative metrics reported over the first 100 videos of the test set for GQN-Mazes and MineRL.
For CARLA Town01, the test set length is 100.
Table 3 lists the hyperparameters for all training runs reported.
We provide additional details on the implementations of each method below.
FDM Our implementation of FDM builds on the DDPM implementation 5 of Nichol and Dhariwal [22].
For experiments at 64 × 64 resolution, the hyperparameters of our architecture are almost identical to that of their 64 × 64 image generation experiments: for example we use 128 as the base number of channels, the same channel multpliers at each resolution, and 4-headed attention.
The exception is that we decrease the number of res-net blocks from 2 to 1 at each up/down-sampling step.
As mentioned in the main text, we run all layers from the image DDPM independently and in parallel for each frame, and add a temporal attention layer after every spatial attention layer.
The temporal attention layer has the same hyperparameters as the spatial attention layer (e.g. 4 attention heads) except for the addition of relative position encodings, which we describe later.
For experiments at 128 × 128 resolution, we use almost the same architecture, but with an extra block at 128 ×resolution with channel multiplier 1.
For full transparency, we release FDM’s source code.
VDM As mentioned in the main paper, we train VDM by simply training two networks, each with architecture identical to that of FDM but different training tasks.
In each of VDM’s training tasks, we use a slice of 16 or 9 frames (with frameskip 4 or 1 respectively).
We randomly sample zero or more “groups” of regularly-spaced frames to observe (where groups of frames are sampled similarly here to in FDM’s structured mask distribution in Algorithm 2), and the rest are latent.
On all datasets, we train each of the two networks forming the VDM baseline with roughly as many GPU-hours as FDM, so that VDM receives roughly twice as much training compute in total.
TATS We train TATS using the official implementation 6 along with its suggested hyperparameters.
For GQN-Mazes and MineRL we train each stage for close to a week and, following Ge et al.
[11], train them on 8 GPUs in parallel.
For all datasets, the total training computation is multiple times that of FDM.
In the included video samples from our TATS baseline, some artifacts are clearly visible.
It may be that these could be removed with further hyperparameter tuning, but we did not pursue this.
[Table]
Notably
[Table]
the datasets which we experiment on generally have a lower frame-rate than those used by
Ge et al.
[11], meaning that neighboring frames are more different and so potentially harder to model.
CWVAE We train CWVAE using the official implementation 7 and use hyperparameters as close as possible to those used in the implementation by Saxena et al.
We use 600 epochs to train
CWVAE on MineRL, as suggested by Saxena et al.
[26], and train it for more iterations on both other datasets.
On CARLA Town01, since CWVAE is not implemented for 128 × 128 images, we downsample all train and test data to 64 × 64 .
A.1 Additional evaluation metrics
We report additional evaluation metrics in Table 4.
The “Loss” refers to the average DDPM loss (Eq.
(4)) over the test set, such that an appropriate choice of λ ( t ) would yield the ELBO of the test videos under each model and sampling scheme although, as in our training loss, we use λ ( t ) :
=to de-emphasise pixel-level detail.
The commonly-used [26, 3] LPIPS, SSIM and PSNR metrics measure frame-wise distances between each generated frame around the ground-truth.
To account for stochasticity in the task, k video completions are generated for each test video and the smallest distance to the ground-truth is reported.
We report them for completeness, but do not believe that
SSIM and PSNR correlate well with video quality due to the stochastic nature of our datasets.
For example, see the CWVAE videos on MineRL at https://www.cs.ubc.ca/~wsgh/fdm which obtain higher SSIM and PSNR than other methods despite being much blurrier.
Since SSIM and
PSNR are related to the mean-squared error in pixel space, they favor blurry samples over more realistic samples.
While increasing k should counteract this effect, the effectiveness of this scales poorly with video length and and so this made little difference in the datasets we consider.
A.2 Ablation on training task distribution
We mention in the main paper that we perform an ablation on the training task distribution.
FVD scores from this ablation are reported in Table 5.
We sample from the baseline “uniform” task distribution as follows (where Uniform(a, b) should be understood to assign probability to all integers between a and b inclusive ):
Sample n total ∼ Uniform (1 , K ) .
Assign Z to be a vector of n total integers sampled without replacement from { 1 , . .
, K } .
Sample n obs ∼ Uniform (0 , n total − 1) .
Assign the first n obs entries in Z to Y and the remainder to X .
This leads to a much less structured distribution than that described in Fig.
The network trained using our proposed task distribution obtains a better FVD than our ablation on all tested combinations of dataset and sampling scheme.
Note that all networks in this experiment use the hyperparameters reported in the bottom two rows of Table 3, explaining the disparity between FVDs here and in
Table 1.
[Table]
B Relative position encodings
Relative position encoding background and use-case
Our temporal attention layer is run independently at every spatial location, allowing each spatial location in every frame to attend to its counterparts at the same spatial location in every other frame.
That is, denoting the input to a temporal attention layer z in and the output z out , we compute the
K × C slice z out : ,h,w, : =
attn ( z in : ,h,w, : ) for every spatial position ( h, w ) .
To condition the temporal attention on the frame’s positions within the video, we use relative position encodings (RPEs) [27, 38] for each pair of frames.
Let pos ( i ) =
( X ⊕Y )
i be a function mapping the index of a frame within z to its index within the full video v .
Then the encoding of the relative position of frames i and j depends only on pos ( i ) − pos ( j ) .
We write this RPE as the set of three vectors p ij = { p Qij , p Kij , p Vij } which are used in a modified form of dot-product attention (described in the following paragraph).
Since p ij must be created for every ( i, j ) pair in a sequence, computing it adds a cost which scales as O ( K 2 ) and prior work has attempted to minimize this cost by parametrizing p ij with a simple learned look-up table (LUT) as p ij :=
LUT ( pos ( i ) − pos ( j )) .
In the next paragraph we describe our alternative to the LUT, but first we describe how the RPEs are used in either case.
We use the RPEs in the same way as Shaw et al.
As in a standard transformer [34], a sequence of input vectors z in 1 , . . .
, z in K are transformed to queries, keys, and values via the linear projections q i =
W Q z i , k i =
W K z i , and v i
=
W V z i for i = 1 , . . .
, K .
Given the RPEs for all ( i, j ) pairs, and marking the operations involving them in blue, the output of the attention block is
[Equation]
Our approach to computing RPEs We argue that the simplicity of parametrizing RPEs with a
LUT is not necessary within our framework for three reasons.
In our framework, K can be kept small, so the O ( K 2 ) scaling cost is of limited concern.
Furthermore, since the temporal attention mechanism is run for all spatial locations ( h, w ) in parallel, the cost of computing RPEs can be shared between them.
The range of values that pos ( i ) − pos ( j ) can take scales with the video length N , and the average number of times that each value is seen during training scales as K 2 /N .
For long videos and small K , a look-up table will be both parameter-intensive and receive a sparse learning signal.
We propose to parameterize p ij with a learned function as p ij := f RPE ( d ij ) where d ij := pos ( i ) − pos ( j ) .
As shown in Fig.
7, f RPE passes a 3-dimensional embedding of d ij through a neural network which outputs the vectors making up p ij .
We use a network with a single C -channel hidden layer and were not able to measure any difference in the runtime between a DDPM with this network and a DDPM with a look-up table.
Figure 8 shows the effect of the RPE network on attention weights early in training.
Architectures with an network can learn the relative importance of other frames much more quickly.
After training to convergence, there was no noticeable difference in sample quality but the architecture with an network used 9.8 million fewer parameters by avoiding storing large look-up tables.
An alternative approach to our RPE network described by Wu et al.
[38] shares the look-up table entries among “buckets” of similar pos ( i ) − pos
( j ) ,
but this imposes additional hyperparameters as well as restricting network expressivity.
C Explanation of our training task distribution
Here we provide our motivation, design choices and more explanation of our training task distribution, as visualized in Fig.
4 and implemented in Algorithm 2.
Since we train our model to work with any custom sampling scheme at test time, our training distribution should be broad enough to assign some probability to any feasible choices of frames to sample and observe.
At the same time, we want to avoid purely random sampling of frame positions (as in e.g. the ablation in Appendix A.2) as this will impair performance in realistic sampling schemes.
Taking these considerations in mind, our design considerations for Algorithm 2 are simple:
The model should sample frames at multiple timescales, so we sample the spacing between frames (as on line 4 of Algorithm 2).
A log-uniform distribution is a natural fit since events in a video sequence can happen over timescales in, e.g., seconds, minutes, or hours, and the differences between these are best captured by a log scale.
The parameters of this log-uniform distribution are chosen to be the broadest possible (given the video length and the frame rate).
The user may wish to jointly sample multiple disparate sections of a video.
We therefore make it possible to sample multiple groups of frames, potentially with different timescales (this is the purpose of the while loop in Algorithm 2).
The number of frames a user may wish to sample at a time is not fixed, so we add a broad uniform distribution over this (line 3 of the algorithm).
We train the model to perform conditional generation, so we choose groups of frames to be conditioned on (line 6 of the algorithm) using the simplest appropriate distribution, Bernoulli(0.5).
The remainder of the algorithm is boilerplate, gathering the indexed frames (line 1, 7, 9-13), randomizing the position of frames within the video (line 5) and enforcing that the number of frames does not exceed K (line 8).
Note that we do not claim that e.g. this exact mechanism for ensuring that ≤ K frames are sampled is a necessary or optimal choice for achieving FDM’s performance.
It is simply a design choice.
D Sampling schemes
Figure 9 illustrates each of the sampling schemes for which we reported results.
We show the versions adapted to completing 300-frame GQN-Mazes videos from 36 initial observations, but all can be extended to e.g. different video lengths.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
D.1 Adaptive sampling schemes
As mentioned in the main text, our Ad.
hierarchy-2 sampling scheme chooses which frames to condition on at test-time by selecting a diverse set of observed of previously generated frames.
Our procedure to generate this set is as follows.
For a given stage s we define X s to be the same as the latent frame indices at the corresponding stage of the standard Hierarchy-2 sampling scheme.
We then initialize Y s with the closest observed or previously generated frame before the first index in
X s , after the last index in X s , and any observed or previously generated frames between the first and last indices of X s .
We add more observed indices to Y s in an iterative procedure, greedily adding the observed or previously generated frame with the maximum LPIPS [41] distance to it’s nearest neighbour in Y s .
Frames are added one-at-a-time in this way until Y s is the desired length (generally
K/ 2 , or 10 in our experiments).
Despite using a convolutional neural network to compute the LPIPS distances, the computational cost of computing Y s in our experiments with Ad.
hierarchy-2 is small relative to the cost of drawing samples from the DDPM.
D.2 Optimized sampling schemes
We now describe in detail our procedure for optimizing the choice of indices to condition on at each stage in a sampling scheme.
Our procedure requires that the “latent” frames are pre-specified.
Figures 9f and 9g show examples of the indices that our optimization scheme chooses to condition on for GQN-Mazes when the latent indices are set according to either our Autoreg or Hierarchy2 sampling scheme.
We emphasize that the (relatively computationally-expensive) optimization described in this section need only be performed once and then arbitrarily many videos can be sampled.
This is in contrast to the adaptive sampling scheme described in Appendix D.1, in which the sets of indices to condition on are chosen afresh (with small computational cost) for each video as it is sampled.
For each stage of the sampling scheme, we select the set of indices to condition on with a greedy sequential procedure.
Intialization of Y
This procedure begins by initializing this set of indices, Y .
In general Y can be initialized as an empty set, but it can also be initialized with indices that the algorithm is “forced” to condition on.
We initialize it to contain the closest previously sampled/observed indices before and after each latent index.
In other words, we initialize it so that there is a blue pixel between any red and black pixel in each row of Figs. 9f and 9g.
Appending to Y
On each iteration of the procedure, we estimate the DDPM loss in Eq.
(4) (with uniform weighting) for every possible next choice of index to condition on.
That is, we compute the DDPM loss when conditioning on frames at indices Y ⊕ [ i ] for every i ∈ { 1 , . . .
, N } \ X \ Y .
We estimate the loss by iterating over timesteps t ∈ { 100 , 200 , . . .
, 1000 } and, for each timestep, estimating the expectation over x 0 with 10 different training images.
We found that the iteration over a grid of timesteps, rather than random sampling, helped to reduce the variance in our loss estimates.
We then select the index resulting in the lowest loss, append it to Y , and repeat until Y is at the desired length.
We repeat the entire procedure for every stage of the sampling scheme.
E CARLA Town
The CARLA Town01 dataset was created by recording a simulated car driving programatically around the CARLA simulator’s Town01 [9].
The car is driven so as to stay close to the speed limit of roughly 3 m/s where possible, stopping at traffic lights.
The simulations run for 10 000 frames and we split each into 10 1000-frame videos.
Within each simulation, the weather and other world state (e.g. state of the traffic lights) is sampled randomly.
The car begins each simulation in a random position, and navigates to randomly selected waypoints around the town.
As soon as it reaches one, another is randomly sampled so that it continues moving.
We use a 120 degree field of view and render frames at 128 × 128 resolution.
To perform our evaluations on this dataset, we trained a regressor to map from a frame (either from the dataset or from a video model) to the corresponding town coordinates.
This is trained with ( x, y ) coordinates extracted from the simulator corresponding to the car location at each frame.
The regressor takes the form of two separate networks: a classifier mapping each frame to a cell within a 10 × 10 grid placed over the town; and a multi-headed regressor mapping from the frame to ( x, y ) coordinates in a continuous space.
The final layer of the multi-headed regressor consists of 100 linear “heads”, and which one to use for each data point is chosen depending on which cell the coordinate lies in.
These two networks are trained separately but used jointly during evaluation, when the classifier is run first and its output determines which regressor head is used to obtain the final ( x, y ) coordinate.
We found that this approach improved the test mean-squared error considerably relative to using a single-headed regressor.
The classifier was trained with data augmentation in the form of color jitter and a Gaussian blur, but we found that the multi-headed
[Figure]
contains several days the well for 30-40 minutes so should not be expected to explore the entire map.
However
-2 in particular obtains high coverage.
The sampled with Autoreg explores less of the map due to its tendency to remain stationary for long periods.
regressor did not benefit from this data augmentation so trained it without.
Both the classifier and multi-headed regressor had the Resnet128 [14] architecture, with weights pretrained on ImageNet, available for download from the PyTorch torchvison package [23].
We will release the classifier and multi-headed regressor used to evaluate our models, enabling future comparisons.
F Sampled videos
To fully appreciate our results, we invite the reader to view a collection of FDM’s samples in mp format.
9 These include video completions, unconditionally sampled videos, and the long video samples from which some frames are shown in Fig.
To summarize the long video samples in this document, we visualize the trajectories taken throughout their (30-40 minute) course on CARLA
Town01 in Fig.
Additionally, in the following pages, we show frames from uncurated samples of both video completions and unconditional video generations on each dataset.