Flexible Diffusion Modeling of Long Videos
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We present a framework for video modeling based on denoising diffusion probabilistic models that produces long-duration video completions in a variety ofrealistic environments.
We introduce a generative model that can at test-time sample any arbitrary subset of video frames conditioned on any other subset and presentan architecture adapted for this purpose.
Doing so allows us to efﬁciently compareand optimize a variety of schedules for the order in which frames in a long videoare sampled and use selective sparse and long-range conditioning on previouslysampled frames.
We demonstrate improved video modeling over prior work ona number of datasets and sample temporally coherent videos over 25 minutes inlength.
We additionally release a new video modeling dataset and semanticallymeaningful metrics based on videos generated in the CARLA self-driving carsimulator
Introduction
Generative modeling of photo-realistic videos is at the frontier of what is possible with deep learningon currently-available hardware.
Although related work has demonstrated modeling of short photorealistic videos (e.g. 30 frames [35], 48 frames [6] or 64 frames [15]), generating longer videosthat are both coherent and photo-realistic remains an open challenge.
A major difﬁculty is scaling:photorealistic image generative models [4, 8] are already close to the memory and processing limitsof modern hardware.
A long video is at very least a concatenation of many photorealistic frames,implying resource requirements, long-range coherence notwithstanding, that scale with frame count.
Attempting to model such long-range coherence makes the problem harder still, especially because ingeneral every frame can have statistical dependencies on other frames arbitrarily far back in the video.
Unfortunately ﬁxed-lag autoregressive models impose unrealistic conditional independence assumptions (the next frame being independent of frames further back in time than the autoregressive lag isproblematic for generating videos with long-range coherence).
And while deep generative modelsbased on recurrent neural networks (RNN) theoretically impose no such conditional independenceassumptions, in practice they must be trained over short sequences [11, 25] or with truncated gradients [30].
Despite this, some RNN-based video generative models have demonstrated longer-rangecoherence, albeit without yet achieving convincing photorealistic video generation [25, 3, 7, 19, 2].
In this work we embrace the fact that ﬁnite architectures will always impose conditional independences.
The question we ask is: given an explicit limit K on the number of video frames we canjointly model, how can we best allocate these frames to generate a video of length N > K ?
Oneoption is to use the previously-described autoregressive model but, if K = N/ 4 , we could insteadfollow Ho et al.
[15] by training two models: one which ﬁrst samples every 4th frame in the video,and another which (in multiple stages) inﬁlls the remaining frames conditioned on those.
To enableefﬁcient exploration of the space of such sampling schemes, we propose a ﬂexible architecture basedon the denoising diffusion probabilistic model (DDPM) framework.
This can sample any subset ofvideo frames conditioned on observed values of any other subset of video frames.
It therefore lets us explore a wide variety of previously untested sampling schemes while being easily repurposedfor different generation tasks such as unconditional generation, video completion, and generation ofvideos of different lengths.
Since our model can be ﬂexibly applied to sample any frames given anyothers we call it a Flexible Diffusion Model, or FDM.
Contributions (1) At the highest level, we claim to have concurrently developed one of the ﬁrstdenoising diffusion probabilistic model (DDPM)-based video generative models [15, 39].
To do so weaugment a previously-used DDPM image architecture [14, 21] with a temporal attention mechanismincluding a novel relative (frame) position encoding network.
The principal contribution ofthis paper, regardless, is an objective and “meta-learning” task schedule that encourages learningof a video generative model that can a) be ﬂexibly conditioned on any number of frames (up tocomputational resource constraints) at any time in the past and future and b) be ﬂexibly marginalized(to achieve this within computational resource constraints).
(3) We demonstrate that our model canbe used to efﬁciently explore the space of resource constrained video generation schemes, leading toimprovements over prior work on several long-range video modeling tasks.
(4) Finally, we release anew autonomous driving video dataset along with a new video generative model performance metricthat captures semantics more directly than the visual quality and comparison metrics currently inwidespread use.
Sampling long videos
Our goal in this paper is to sample coherent photo-realistic videos v with thousands of frames (see
Fig. 1).
To sample an arbitrarily long video with a generative model that can sample or condition ononly a small number of frames at once, we must use a sequential procedure.
The simplest example ofthis is an autoregressive scheme, an example of which is shown in Fig.
2a for a video completiontask.
In this example it takes seven stages to sample a complete video; at each stage three framesare sampled conditioned on the immediately preceding four frames.
This scheme is appealing for itssimplicity but imposes a strong assumption that, given the set of four frames that are conditioned onat a particular stage, all frames that come afterwards are conditionally independent of all frames thatcame before.
This restriction can be partially ameliorated with the sampling scheme shown in Fig.
2bwhere, in the ﬁrst three stages, every second frame is sampled and then, in the remaining four stages,the remaining frames are inﬁlled.
One way to implement this would be to train two different modelsoperating at the two different temporal resolutions.
In the language of Ho et al.
[15], who use a similarapproach, sampling would be carried out in the ﬁrst three stages by a “frameskip-2” model and, inthe remaining stages, by a “frameskip-1” model.
Both this approach and the autoregressive approachare examples of what we call sampling schemes .
More generally, we characterize a sampling schemeas a sequence of tuples [( X s , Y s )]
Ss =1 , each containing a vector X s of indices of frames to sample 72 and a vector Y s of indices of frames to condition on for stages s = 1 , . . .
, S .
Algorithm 1 lays out how such a sampling scheme is used to sample a video.
If the underlyinggenerative model is trained speciﬁcally to model sequences of consecutive frames, or sequences ofregularly-spaced frames, then the design space for sampling schemes compatible with these modelsis severely constrained.
In this paper we take a different approach.
We design and train a generative
[Figure]
model to sample any arbitrarily-chosen subset of video frames conditioned on any other subset andtrain it using an entirely novel distribution of such tasks.
In short, our model is trained to generateframes for any choice of X and Y .
The only constraint we impose on our sampling schemes istherefore a computational consideration that |X s | + |Y s |  K for all s but, to generate meaningful 81 videos, any valid sampling scheme must also satisfy two more constraints: (1) all frames are sampledat at least one stage and (2) frames are never conditioned upon before they are sampled.
Such a ﬂexible generative model allows us to explore and use sampling schemes like those in Fig.
2cand Fig.
2d.
We ﬁnd in our experiments that the best video sampling scheme is dataset dependent.
Accordingly, we have developed methodology to optimize such sampling schemes in a datasetdependent way, leading to performance gains.
We now review conditional DDPMs (Section 3), beforedelving into the FDM’s architecture, the speciﬁc task distribution used to train it, and the choice andoptimization of sampling schemes in Section 4.
A review of conditional denoising diffusion probabilistic models
Denoising diffusion probabilistic models, or DDPMs [27, 14, 21, 29], are a class of generative modelfor data x , which throughout this paper will take the form of a 4-dimensional tensor representingmultiple video frames.
We will describe the conditional extension [31], in which the modeled x isconditioned on observations y .
DDPMs simulate a diffusion process which transforms x to noise,and learn to generate data by learning the probabilistic inverse of the diffusion process.
The diffusionprocess happens over timesteps 0 , . .
, T such that x 0 = x is data without noise, x 1 has a very smallamount of noise added, and so on until x T is almost independent of x 0 and approximates a randomsample from a unit Gaussian.
In the diffusion process we consider, the distribution over x t dependsonly on x t  1 :
[Equation]
Hyperparameters ↵ 1 , . .
. , ↵ T are chosen to be close to but slightly less than 1 so that the amount ofnoise added at each step is small.
The combination of this diffusion process and a data distributionq ( x 0 , y ) (recalling that x 0 = x ) deﬁnes the joint distribution
[Equation]
DDPMs work by “inverting” the diffusion process: given values of x t and y a neural network isused to parameterize p ✓ ( x t  1 | x t , y ) , an approximation of q ( x t  1 | x t , y ) .
This neural network lets
[Figure]
[Figure] us draw samples of x 0 by ﬁrst sampling x T from a unit Gaussian (recall that the diffusion processwas chosen so that q ( x T ) is well approximated by a unit Gaussian), and then iteratively samplingx t  1 ⇠ p ✓ ( ·| x t , y ) for t =
T, T  1 , . .
The joint distribution of sampled x 0: T given y is
[Equation]
where p ( x T ) is a unit Gaussian that does not depend on ✓ .
Training the conditional DDPM thereforeinvolves ﬁtting p ✓ ( x t  1 | x t , y ) to approximate q ( x t  1 | x t , y ) for all choices of t , x t , and y .
Several observations have been made in recent years which simplify the learning of p ✓ ( x t  1 | x t , y ) .
Sohl-Dickstein et al.
[27] showed that when ↵ t is close to 1, p ✓ ( x t  1 | x t ) is approximately Gaussian [27].
Furthermore, Ho et al.
[14] showed that this Gaussian’s variance can be modeled well witha non-learned function of t , and that a good estimate of the Gaussian’s mean can be obtained froma “denoising model” as follows.
Given data x 0 and unit Gaussian noise ✏ , the denoising model (inthe form of a neural network) is fed “noisy” data x t := p ˜
↵ t x 0 + p 1  ˜ ↵ t ✏ and trained to recover ✏via a mean squared error loss.
The parameters ˜ ↵ t :=
Q ti =1 ↵ i are chosen to ensure that the marginaldistribution of x t given x 0 is q ( x t | x 0 ) as derived from Eq.
Given a weighting function  ( t ) , thedenoising loss is
[Equation]
The mean of p ✓ ( x t  1 | x t , y ) is obtained from the denoising model’s output ✏ ✓ ( x t , y , t ) as1 ↵ t x t  1  ↵ t p 1  ˜ ↵ t ✏ ✓ ( x t , y , t ) .
If the weighting function  ( t ) is chosen appropriately, optimising Eq.
(4)is equivalent to optimising a lower-bound on the data likelihood under p ✓ .
In practice, simply setting ( t ) :
= 1 for all t can produce more visually compelling results in the image domain [14].
In our proposed method, as in Tashiro et al.
[31], the shapes of x 0 and y sampled from q ( · ) vary.
Thisis because we want to train a model which can ﬂexibly adapt to e.g. varying numbers of observedframes.
To map Eq.
(4) to this scenario, note that both x 0 and y implicitly contain information aboutwhich frames in the video they represent (via the index vectors X and Y introduced in the previoussection).
This information is used inside the neural network ✏ ✓ ( x t , y , t ) so that interactions betweenframes can be conditioned on the distance between them (as described in the following section)
andalso to ensure that the sampled noise vector ✏ has the same shape as x 0 .
Training procedure and architecture
Training task distribution Different choices of latent and observed indices X and Y can be 131 regarded as deﬁning different conditional generation tasks.
In this sense, we aim to learn a model
Algorithm X , Y ⇠ u ( · ) given N, K . 1: X := {} ; Y := {} 2: while True do 3: n group ⇠ UniformDiscrete (1 , K ) 4: s group ⇠ LogUniform (1 , ( N  1) /n group ) 5: x group ⇠ Uniform (0 , N  ( n group  1) · s group ) 6: o group ⇠ Bernoulli (0 . 5) 7:
G :=
{ b x group + s group · i c | i 2 { 0 , . . .
, n group  1 }} \ X \ Y 8: if |X| + |Y| + |G| > K then 9: return set2vector ( X ) , set2vector ( Y ) 10: else if |X| = 0 or o group = 0 then 11:
X :=
X [ G 12: else 13:
Y :
=
Y [ G
The FDM’s training objective forces it to work well for any ( X , Y ) pair from this broad distribution.
which can work well on any task (i.e. any choice of X and Y ) and so we randomly sample thesevectors of indices during training.
We do so with the distribution u ( X , Y ) described in Fig. 4.
This provides a broad distribution covering many plausible test-time use cases while still providingsufﬁcient structure to improve learning (see ablation in Section 6 and more details in Appendix C).
To cope with constrained computational resources, the distribution is designed such that |X| + |Y| is 137 upper-bounded by some pre-speciﬁed K .
Sampling from q ( x 0 , y ) in Eq.
(4) is then accomplishedby randomly selecting both a full training video v and indices X , Y ⇠ u ( · , · ) .
We then extract the 139 speciﬁed frames x = v [ X ] and y = v [ Y ] (where we use v [ X ] to denote the concatenation of allframes in v with indices in X and and v [ Y ] similarly).
Architecture DDPM image models [14, 21] typically use a U-net architecture [23].
Its distinguishing feature is a series of spatial downsampling layers followed by a series of upsampling layers, andthese are interspersed with convolutional res-net blocks [13] and spatial attention layers.
Since werequire an architecture which operates on 4-D video tensors rather than 3-D image tensors we addan extra frame dimension to its input, output and hidden state, resulting in the architecture shownon the right of Fig.
We create the input to this architecture as a concatenation x t  y , adding anextra input channel which is all ones for observed frames and all zeros for latent frames.
For RGBvideo, the input shape is therefore ( K, image height , image width , 4) .
Since the output should havethe same shape as x t we only return outputs corresponding to the latent frames, giving output shape( |X| , image height , image width , 3) .
We run all layers from the original model (including convolution,resizing, group normalization, and spatial attention) independently for each of the K frames.
To allowcommunication between the frames, we add a temporal attention layer after each spatial attentionlayer, as described below.
The spatial attention layer allows each spatial location in each frame toattend to every other spatial location in the same frame, while the temporal attention layer allows it toattend to the same spatial location in every other frame.
This combination of a temporal attentionlayer with a spatial attention layer is sometimes referred to as factorized attention [31, 15].
We foundthat, when using this architecture in conjunction with our meta-learning approach, performance couldbe improved by using a novel form of relative position encoding [26, 37].
This is included in ourreleased source code but we leave its exposition to the supplementary material.
Training batch padding Although the size |X  Y| of index vectors sampled from our training 161 distribution is bounded above by K , it can vary.
To ﬁt examples with various sizes of index vectorsinto the same batch, we therefore use what we call “training batch padding”.
This means that, fortraining examples where |X  Y| < K , we concatenate frames uniformly sampled from a second 164 video to increase the length along the frame-dimension to K .
Masks are applied to the temporalattention mechanisms so that frames from different videos cannot attend to eachother.
Sampling schemes Before describing the sampling schemes we explore experimentally, we emphasize that the relative performance of each is dataset-dependent and there is no single best choice.
Acentral beneﬁt of FDM is that it can be used at test-time with different sampling schemes withoutretraining.
Our simplest sampling scheme, Autoreg , samples ten consecutives frames at each stageconditioned on the previous ten frames.
Long-range is similar to Autoreg but conditions on onlythe ﬁve most recent frames as well as ﬁve of the original 36 observed frames.
Hierarchy-2 uses amulti-level sampling procedure.
In the ﬁrst level, ten evenly spaced frames spanning the non-observedportion of the video are sampled (conditioned on ten observed frames).
In the second level, groups ofconsecutive frames are sampled conditioned on the closest past and future frames until all frameshave been sampled.
Hierarchy-3 adds an intermediate stage where several groups of variables withan intermediate spacing between them are sampled.
We include adaptive hierarchy-2, abbreviated
Ad.
hierarchy-2 , as a demonstration of a sampling scheme only possible with a model like FDM.
It samples the same frames at each stage as Hierarchy-2 but selects which frames to condition onadaptively at test-time with a heuristic aimed at collecting the maximally diverse set of frames, asmeasured by the pairwise LPIPS distance [40] between them.
Optimizing sampling schemes
An appealing alternative to the heuristic sampling schemes described in the previous paragraph would be to ﬁnd a sampling scheme that is, in some sense, optimalfor a given model and video generation/completion task.
While it is unclear how to tractably choosewhich frames should be sampled at each stage, we suggest that the frames to condition on at eachstage can be chosen by greedily optimizing the diffusion model loss which, as mentioned in Section 3,is closely related to the data log-likelihood.
Given a ﬁxed sequence of frames to sample at each stage[ X s ] Ss =1 we select Y s for each s to minimize an objective close to Eq.
This is estimated using 100training videos and a single sample of t for each.
See the appendix for further details.
We create twooptimized sampling schemes: one with the same latent indices as Autoreg, and one with the samelatent indices as Hierarchy-2.
We call the corresponding optimized schemes Opt.
autoreg and Opt.hierarchy-2 .
[Table]
CARLA Town01 Dataset
In addition to our methodological contributions
[Table]
we propose a new video-modeling dataset andbenchmark which provides an interpretable measure of video completion quality.
The dataset consistsof videos of a car driving with a ﬁrst-person view, produced using the CARLA autonomous drivingsimulator [9].
All 408 training and 100 test videos (of length 1000 frames and resolution 128 ⇥ 128 )are produced within a single small town, CARLA’s Town01.
As such, when a sufﬁciently expressivevideo model is trained on this dataset it memorizes the layout of the town and videos sampled fromthe model will be recognisable as corresponding to routes travelled within the town.
We train aregression model in the form of a neural network which maps with high accuracy from any singlerendered frame to ( x, y ) coordinates representing the car’s position.
Doing so allows us to plot theroutes corresponding to sampled videos (see left of Fig. 5) and compute semantically-meaningful yetquantitative measures of the validity of these routes.
Speciﬁcally, we compute histograms of speeds,where each speed is estimated by measuring the distance between the regressed locations for framesspaced ten apart (1 second at the dataset’s frame rate).
Sampled videos occasionally “jump” betweendisparate locations in the town, resulting in unrealistically large estimated speeds.
To measure thefrequency of these events for each method, we compute the percentage of our point-speed estimatesthat exceed a threshold of 10 m/s (the dataset was generated with a maximum simulated speed of3 m/s).
We report this metric as the outlier percentage (OP).
After ﬁltering out these outliers, wecompute the Wasserstein distance (WD) between the resulting empirical distribution and that of theoriginal dataset, giving a measure of how well generated videos match the speed of videos in thedataset.
We will release the CARLA Town 01 dataset along with code and our trained regressionmodel to allow future comparisons.
Experiments
The related work most relevant to ours is the concurrent work of Ho et al.
[15], who model 64-framevideos using two trained DDPMs.
The ﬁrst is a “frameskip-4” model trained to generate every fourthframe and a the second is a “frameskip-1” model trained on sequences of nine consecutive framesand used to “ﬁll in” the gaps between frames generated in the ﬁrst stage.
To compare against thisapproach, which we denote VDM , we train both a “frameskip-4” and a “frameskip-1” model witharchitectures identical to our own.
1 Since VDM requires two trained DDPMs, we train it for more
GPU-hours than FDM despite the fact that FDM is meta-learning over a far broader task distribution.
We also compare against the clockwork VAE ( CW-VAE ) [25], a VAE-based model speciﬁcallydesigned to maintain long-range dependencies within video.
We perform our main comparisons on the video completion task.
In keeping with Saxena et al.[25], we condition on the ﬁrst 36 frames of each video and sample the remainder.
We presentresults on three datasets: GQN-Mazes [10], in which videos are 300 frames long; MineRL three of
[Table]
Navigate [12, 25] (which we will fromnow on refer to as simply MineRL),in which videos are
;and the CARLA Town01 dataset we release, for which videos are 1000 frameslong
FDM all cases number of represented
Comparison of sampling schemes
The relative performance of different sampling schemes varies signiﬁcantly between datasets asshown in Table 1.
We report Fréchet Video Distances (FVDs) [32], a measure of how similarsampled completions are to the test set, as well as the test DDPM losses for Mazes and MineRL.
For CARLA Town01, we report the previously described percentage outliers (PO) and Wasserstein distance (WD) metrics.
We can broadly consider the mentioned sampling schemes as either being in the “autoregressive” family (Autoreg and Long-range) or in the “hierarchical” family (the remainder).
The hierarchical schemes achieve signiﬁcantly lower FVDs [32] on GQNMazes.
Our samples in the appendix suggest that this is related to the autoregressive methods“forgetting” the colors of walls after looking away from them for a short time.
In contrast for MineRL, the autoregressive methods tend to achieve the best FVDs.
This may relate to thefact that trajectories in MineRL tend to travel in straight lines through procedurally-generated“worlds”[12, 25], limiting the number of long-range dependencies.
Finally, on CARLA Town01, wenotice qualitatively different behaviours from our autoregressive and hierarchical sampling schemes.
The hierarchical sampling schemes have a tendency to occasionally lose coherence and “jump”to different locations in the town.
This is reﬂected by higher outlier percentages (OP) in Table 1.
On the other hand the autoregressive schemes often stay stationary for unrealistically long times attrafﬁc lights.
This is reﬂected in the histogram of speeds in Fig.
6, which has a larger peak aroundzero than the ground truth.
The high variance of the sampling scheme’s relative performance overdifferent datasets points to a strength of our method, which need only be trained once and then usedto explore a variety of sampling schemes.
Furthermore, we point out that the best FVDs in Table 1on
all datasets were obtained using sampling schemes that could not be implemented using modelstrained in prior work, or over evenly spaced frames. [Figure]
Optimized sampling schemes As mentioned in Section 4, another advantage of FDM is that it makespossible a model- and dataset-speciﬁc optimizationprocedure to determine on which frames to condition.
Table 2 shows the results when this procedure is usedto create sampling schemes for GQN-Mazes and MineRL.
In the ﬁrst row we show results where the latentframes are ﬁxed to be those of the Autoreg samplingscheme, and in the second row the latent frames areﬁxed to match those of Hierarchy-2.
In both casesthe results in Table 1 are improved upon, showing theutility of this optimization procedure.
Comparison with training on a single task Training a network with our distribution over trainingtasks could be expected to lead to worse performance on a single task than training speciﬁcally forthat task.
To test whether this is the case, we train an ablation of FDM with training tasks exclusivelyof the type used in our Autoreg sampling scheme, i.e. “predict ten consecutive frames given theprevious ten.”
Tested with the Autoreg sampling scheme, it obtained an FVD of 82 .
on GQN-Mazesand 234 on MineRL.
As expected given the specialization to a single task, this is better than when
FDM is run with the Autoreg sampling scheme (obtaining FVDs of 86 . 4 and 281 respectively).
Ablation on training task distribution To test how important our proposed structured trainingdistribution is to FDM’s performance, we perform an ablation with a different task distribution thatsamples X and Y from uniform distributions instead of our proposed structured task distribution Weprovide full details in the appendix, but report here that switching away form our structured trainingdistribution made the FVD scores worse on all ﬁve tested sampling schemes on both GQN-Mazes and MineRL.
This implies that our structured training distribution has a signiﬁcant positive effect.
Finally, we note that although we have not quantitatively evaluated unconditionally sampled videos,our model can be applied in this way. We host uncurated samples (conditional and unconditional) athttps://fdmolv.github.io .
Related work
Some related work creates conditional models by adapting the sampling procedure of an unconditional
DDPM [29, 17, 20, 15].
These approaches require approximations and the more direct approachthat we use (explcitly training a conditional DDPM) was shown to have beneﬁts by Tashiro et al.[31].
We consider further comparison of these competing approaches to be outside the focus ofthis work, which is on modeling a small portion of video frames at a time, essentially performingmarginalization in addition to conditioning.
There are a number of approaches in the literature which use VAEs rather than DDPMs for video modelling.
Babaeizadeh et al.
[2] use a VAE model which predicts frames autoregressively conditionedon a global time-invariant latent variable.
A related approach by Denton and Fergus [7] also uses a
VAE with convolutional LSTM architectures in both the encoder and decoder.
Unlike Babaeizadehet al.
[2] the prior is learned and a different latent variable is sampled for each frame.
Babaeizadehet al.
[3] use a VAE with one set of latent variables per frame and inter-frame dependencies trackedby a two-layer LSTM.
Their architecture intentionally overﬁts to the training data, which whencoupled with image augmentations techniques achieves SOTA on various video prediction tasks.
Kimet al.
[19] use a variational RNN [5] with a hierarchical latent space that includes binary indicatorvariables which specify how the video is divided into a series of subsequences.
Both Villegas et al.[34] and Wichers et al.
[36] target long-term video prediction using a hierarchical variational LSTMarchitecture, wherein high-level features such as landmarks are predicted ﬁrst, then decoded intolow-level pixel space.
The two approaches differ in that Villegas et al.
[34] requires ground truthlandmark labels, while [36] removes this dependence using an unsupervised adversarial approach.
Fully GAN-based video models have also been proposed [1, 6] but generally suffer from “low qualityframes or low number of frames or both” [1].
Discussion
We have deﬁned and empirically explored a new method for generating photorealistic videos withlong-range coherence that respects and efﬁciently uses ﬁxed, ﬁnite computational resources.
Ourapproach outperforms prior work on long-duration video modeling as measured by quantitative andsemantically meaningful metrics and opens up several avenues for future research.
For one, similarto using DDPMs for image generation, our method is slow to sample from (it takes approximately 16minutes to generate a 300 frame video on a GPU).
Ideas for making sampling faster by decreasingthe number of integration steps [24, 28, 38] could be applied to our video model.
On a different note, consider the datasets on which our artifact was trained.
In each there was a policyfor generating the sequences of actions that causally led to the frame-to-frame changes in camerapose.
In MineRL the video was generated by agents that were trained to explore novel Minecraftworlds to ﬁnd a goal block approximately 64 meters away [12].
The CARLA data was producedby a camera attached to an agent driven by a low level proportional–integral–derivative controllerfollowing waypoints laid down by a high level planner that was given new, random location goalsto drive to intermittently.
In both cases our video model had no access to either the policy or thespeciﬁc actions taken by these agents and, so, in a formal sense, our models integrate or marginalizeover actions drawn from the stochastic policy used to generate the videos in the ﬁrst place.
In nearterm future work we will add action (and reward) explicitly to the architecture and training objectivetransforming our video generative model into a vision-based world model in the reinforcementlearning sense [16, 18].
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
The Carla Town01 dataset we used is too large to include as supplementary material or
host anonymously.
We will provide a link if accepted.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]