F IT V ID :
H IGH
-C
APACITY
P
IXEL
-L EVEL V IDEO P REDICTION Anonymous authors Paper under double-blind review A BSTRACT
An agent that is capable of predicting what happens next can perform a variety of tasks through planning with no additional training.
Furthermore, such an agent can internally represent the complex dynamics of the real-world and therefore can acquire a representation useful for many other visual perception tasks.
This makes predicting the future frames of a video, conditioned on the observed past and potentially future actions, an interesting task which remains exceptionally challenging despite many recent advances.
Existing video prediction models have shown promising results on simple narrow benchmarks but they generate low quality predictions on real-life datasets with more complicated dynamics or broader domain.
There is a growing body of evidence that underﬁtting on the training data is one of the primary causes for the low quality predictions.
In this paper, we argue that the inefﬁcient use of parameters in the current video models is the main reason for underﬁtting.
Therefore, we introduce a new architecture, named FitVid, which is capable of ﬁtting the common benchmarks so well that it begins to suffer from overﬁtting – while having similar parameter count as the current state-of-theart models.
We analyze the consequences of overﬁtting, illustrating how it can produce unexpected outcomes such as generating high quality output by repeating the training data, and how it can be mitigated using existing image augmentation techniques.
As a result, FitVid outperforms the current state-of-the-art models across four different video prediction benchmarks on four different metrics.
I NTRODUCTION
[Figure]
Here, the visual detail on the pushed object is preserved effectively, and even the shadow of the robot’s arm is predicted correctly.
In this example, FitVid is trained on 15M video frames of 7 diffedrent robots from
RoboNet (Dasari et al., 2019).
Predicting what happens next is a cornerstone of intelligence and one of the key capabilities of humans, which we heavily rely on to make decisions in everyday life (Bubic et al., 2010).
This capability enables us to anticipate future events and plan ahead to perform temporally extended tasks.
While the machine learning literature has studied a wide range of prediction problems, one of the most direct challenges is to predict raw sensory inputs.
In particular, prediction of future visual inputs conditioned on a context of past observations – i.e., pixel-level video prediction – encapsulates the challenges of visual perception, modeling of physical events, and reasoning about uncertain behaviors.
Video prediction can be formulated as a self-supervised problem, enabling us to use a substantial amount of unlabeled data to provide autonomous systems with powerful predictive capabilities as well as learning rich representations for downstream tasks.
Already, video models have been successfully deployed in applications such as robotics (Finn & Levine, 2017;
Zhang et al., 2019), simulation (Kim et al., 2021; 2020), compression (Duan et al., 2020; Sulun &
Tekalp, 2021) and video synthesis from a single frame (Endo et al., 2019; Nam et al., 2019).
Despite recent advances in generative models in many domains, such as images (Karras et al., 2017;
Vahdat & Kautz, 2020) and text (Devlin et al., 2018; Brown et al., 2020), video prediction is still considered to be challenging.
Current state-of-the-art methods are limited to low-resolution videos (typically 64 × 64 (Villegas et al., 2019) and a maximum of 256 × 256 (Kim et al., 2021)), usually in a narrow domain, such as a single human walking Ionescu et al.
(2014), or a robotic arm pushing objects in a stationary setting Ebert et al.
Even then the quality of predicted frames tends to drop substantially after as little as 10 seconds into the future (Saxena et al., 2021).
A growing body of evidence suggests that underﬁtting is one of the primary reasons for low quality predictions.
For example, Villegas et al.
(2019) demonstrate how scaling the model, by adding more parameters, can substantially improve the prediction quality.
Similarly, Castrejon et al.
(2019) argue that blurry video prediction of variational methods is a sign of underﬁtting, exhibiting an improved test and train performance as the network capacity increases.
Wu et al.
(2021a) also observed monotonic improvement as the number of modules in a hierarchical architecture increases.
While scaling up models is common to address underﬁtting, it comes at the cost of more computation, memory, and integration risks (Wolf et al., 2019) as well as more complicated training regimes (Wu et al., 2021a).
In this paper we take a step back and address underﬁtting by instead ﬁnding an architecture that uses its parameters more efﬁciently.
We propose FitVid, a model that – with the same parameter count as current state-of-the-art models – can ﬁt the current video prediction benchmarks successfully.
In fact, FitVid ﬁts these benchmarks so well that in many cases we ﬁnd it actually begins to suffer from overﬁtting .
To the best of our knowledge, this is the ﬁrst time a video model reports substantial overﬁtting on theses benchmarks.
Importantly, we also ﬁnd that simple existing image augmentation techniques can mitigate this overﬁtting, leading to models that can both ﬁt the training set and generalize well to held-out videos.
As a result, FitVid achieves state-of-the-art on four challenging video datasets across a wide range of metrics.
Furthermore, while each of the components of our proposed model are not novel on their own, we ﬁnd that FitVid can be trained a signiﬁcantly simpler training recipe.
Prior works on video prediction, particularly those that make use of variational methods to provide for stochasticity, typically require a number of delicate design decisions to train successfully: curriculum training (Oh et al., 2015; Lee et al., 2018; Cenzato et al., 2019), a learned prior (Denton & Fergus, 2018; Castrejon et al., 2019) and annealing of the weight on the VAE KL-divergence penalty (Babaeizadeh et al., 2018).
In contrast, we show that our method actually ﬁts the training data well without any such components, training directly via optimizing the evidence lower bound with minimal hyperparameters.
R ELATED W ORK
Video prediction (Ranzato et al., 2014; Srivastava et al., 2015) has been formulated in different ways such as generating videos from a single image (Endo et al., 2019;
Nam et al., 2019; Dorkenwald et al., 2021; Yang et al., 2021;
Hu et al., 2021) or no image (Vondrick et al., 2016; Saito & Saito, 2018;
Clark et al., 2019), text to video generation (Wu et al., 2021b), video-to-video translation (Wang et al., 2019a; 2018) and data-driven simulation (Kaiser et al., 2019; Kim et al., 2021; 2020).
In this paper, our focus is on conditional video prediction, which is to predict the future frames of a video conditioned on a few initial context frames and possibly the future actions of the agents (Finn et al., 2016a; Babaeizadeh et al., 2018).
Conditional video prediction has a number of applications, including model-based reinforcement learning from pixels (Hafner et al., 2018; 2019; Kaiser et al., 2019; Rafailov et al., 2020) and robotics (Boots et al., 2014; Kalchbrenner et al., 2017a;
Ebert et al., 2017; 2018a;b; Xie et al., 2019; Paxton et al., 2019; Nair & Finn, 2020; Nair et al., 2020).
Initially, video prediction was tackled using deterministic models (Walker et al., 2015; Jia et al., 2016;
Xue et al., 2016; Walker et al., 2016; Liang et al., 2017; Byravan & Fox, 2017; Vondrick &
Torralba, 2017; Van Amersfoort et al., 2017; Liu et al., 2017; Chen et al., 2017; Lu et al., 2017).
Later on, given the common randomness and partial observability in the real-life situations, various stochastic models were proposed to capture the stochasticity of the future.
Generative adversarial networks (GANs) (Goodfellow et al., 2014) are used for video predictions (Mathieu et al., 2015;
Lee et al., 2018; Clark et al., 2019; Luc et al., 2020;
Hong et al., 2021) as sell as autoregressive models (Kalchbrenner et al., 2017b;
Reed et al., 2017; Weissenborn et al., 2019).
Flow based generative models (Dinh et al., 2014; 2016) are also adopted for video prediction (Kumar et al., 2019).
In this paper we focus on Variational Auto-Encoders (VAEs) (Kingma & Welling, 2014) which are widely used for conditional video prediction (Shu et al., 2016; Babaeizadeh et al., 2018; Denton &
Fergus, 2018; Wichers et al., 2018; Villegas et al., 2019; Franceschi et al., 2020; Castrejon et al., 2019; Yan et al., 2021; Rakhimov et al., 2020; Lee et al., 2021; Wu et al., 2021a; Walker et al., 2021).
Recently, it has been demonstrated that underﬁtting on training data plays a major role in low quality blurry predictions in VAE based models.
Villegas et al.
; Castrejon et al.
; Wu et al.
(2021a) all reported improved prediction quality as the network capacity increases.
In this paper, we are interested in addressing underﬁtting without increasing the capacity of the current models, and instead, ﬁnd a more expressive architecture which uses its capacity more efﬁciently.
[Table]
+ 1 x 1 + bn s & e bn + 1 x  bn + s w
i s h + 5 xbn
+ s w
i s h + 1 x 1 + bn s & e bn + 1 x bn + s w
i s h + 5 xbn
+ s w
i s h + 1 x 1 + bn s & e 256 512128512 512128512
10242561024 204851220483 x
+ s i
g m o i d
L S T M L S T M +
+ + + + + + + + + + + + + + + + + + +
[Figure]
and ( N × N ) is a convolutional layer with kernel size of N × N .
The strides are always one, except when down-sampling which has a stride of two.
For up-sampling we use nearest neighbour.
The number under each shows the number of ﬁlters while the top numbers indicate the input size.
To model the dynamics, we use two layers of LSTMs (Hochreiter & Schmidhuber, 1997).
T HE F IT V ID MODEL
Striving for simplicity, we propose the FitVid model for stochastic video prediction, a convolutional non-hierarchical variational model with a ﬁxed prior.
In this section we will describe the proposed model and arcitecture which is also visualized in Fig 2.
The appendix includes more architecture details and the code is available in Supplementary Material.
3. V ARIATIONAL V IDEO P REDICTION WITH F IXED P RIOR
Following prior work (Finn et al., 2016a; Babaeizadeh et al., 2018; Denton & Fergus, 2018; Rubinstein, 1997; Oprea et al., 2020; Wu et al., 2021a), we deﬁne the problem of pixel-level video prediction as follows: given the ﬁrst c frame of a video x <c = x 0 , x 1 , . . .
, x c − 1 , our goal is to predict the future frames by sampling from p ( x c : T | x <c ) .
Optionally, the predictive model can be conditioned on additional given information a t , such as the actions that the agents in the video are planning to take.
This is typically called action-conditioned video prediction.
Variational video prediction (Babaeizadeh et al., 2018) follows the variational auto-encoder (Kingma & Welling, 2014) formalism by introducing a set of latent variables z to capture the inherent stochasticity of the problem.
We factorize the likelihood model to (cid:81)
Tt =
c p θ ( x t | x <t , z ≤ t ) which is parametrized in an autoregressive manner over time; i.e. at each timestep t the video frame x t and the latent variables z t are conditioned on the past latent samples and frames.
For inference, we need to compute a marginalized distribution over the latent variables z , which is intractable.
To overcome this problem, we use variation inference (Jordan et al., 1999) by deﬁning an amortized approximate posterior q ( z | x ) = (cid:81) t q ( z t | z <t , x ≤ t ) that approximates the posterior distribution p ( z | x ) .
The approximated posterior is commonly modeled with an inference network q φ ( z | x ) that outputs the parameters of a conditionally Gaussian distribution
N ( µ φ ( x ) , σ φ ( x )) .
This network can be trained using the reparameterization trick (Kingma &
Welling, 2014) by optimizing the variational lower bound (Kingma & Welling, 2014; Rezende et al., 2014):
L ( x ) = − E q φ ( z | x ) (cid:2) log p θ ( x t : T | x <t , z ) (cid:3) + βD KL (cid:0) q φ ( z | x ) || p ( z ) (cid:1) where D KL is the
Kullback-Leibler divergence between the approximated posterior and the ﬁxed prior p ( z ) =
N ( 0 , I ) .
The hyper-parameter β represents the trade-off between minimizing frame prediction error and ﬁtting the prior (Higgins et al., 2016b;a; Denton & Fergus, 2018).
For FitVid, we set β = 1 .
3. T HE P ROPOSED A RCHITECTURE
Encoder and decoder.
Following the recent advances in image generation, we use similar residual encoding and decoding cells as NVAE (Vahdat & Kautz, 2020).
Each cell includes convolutional layers with batch-normalization (Ioffe & Szegedy, 2015) and swish (Ramachandran et al., 2017;
Elfwing et al., 2018; Hendrycks & Gimpel, 2016) as the activation function, followed by Squeeze and
Excite (Hu et al., 2018).
The encoder is made of four encoding blocks with two cells in each block.
There is down-sampling after each encoder block using a strided convolution of size three in the spatial dimensions.
The decoder also consists of four decoding blocks with two cells in each block, and a nearest neighbour up-sampling after each block.
The number of ﬁlters in each encoding block is doubled while the number of ﬁlters in each decoding block is halved from the previous one.
There is a residual skip connection between the encoder and the decoder after each cell which are ﬁxed to the output from the last context frame.
The statistics for batch-normalization is averaged across time.
Dynamics model.
Similar to Denton & Fergus (2018), the encoded frame h t is used to predict h t + using two layers of LSTMs (Hochreiter & Schmidhuber, 1997).
Likewise, q ( z t | x <t ) is also modeled using a single layer of LSTMs with h t +1 as the input that outputs the parameters of a conditionally
Gaussian distribution N ( µ φ ( x ) , σ φ ( x )) .
During the training, z is sampled from q ( z t | x <t ) while at the inference time z is sampled from the ﬁxed prior N ( 0 , I ) .
The input to the model is always the ground-truth image (which is usually referred to as teacher-forcing (Goodfellow et al., 2016; Cenzato et al., 2019)).
At inference time, the predicted image in the previous time-step is used as input to predict the next frame.
Data augmentation.
We ﬁnd that FitVid can ﬁt even very large datasets very well.
However, in many cases, even for datasets that prior models struggle to ﬁt effectively, we ﬁnd that FitVid actually begins to suffer from overﬁtting (read Section 4 and 5).
To prevent the model from overﬁtting we use data augmentation.
To the best of our knowledge, this is the ﬁrst use of augmentation in video prediction, perhaps because prior state-of-the-art models tend to underﬁt already and therefore would not beneﬁt from it.
Given the rich literature in image augmentation, we augment the videos using RandAugment (Cubuk et al., 2020).
We randomize the augmentation per video but keep the randomization constant for frames of a single video.
RandAugment substantially improves the overﬁtting, however not entirely, as it can be seen in Figure 3.
We improve the augmentation by selecting a random crop of the video before resizing it to the desired resolution at the training time, called RandCrop.
The combination of RandCrop and RandAugment successfully prevents the overﬁtting, leading to models that both ﬁt the training set and generalize well to held-out videos.
What FitVid does not need.
Prior works on variational video prediction (Finn et al., 2016b;
Babaeizadeh et al., 2018; Lee et al., 2018; Villegas et al., 2019; Wu et al., 2021a), generally require a range of additional design decisions for effective training.
Common design parameters include using curriculum training, commonly by scheduled sampling (Bengio et al., 2015), to mitigate distributional shift between training and generation time (Oh et al., 2015; Finn et al., 2016a; Lee et al., 2018; Cenzato et al., 2019); heuristically tuning β in L ( x ) to balance the prediction vs ﬁtting the prior (Denton & Fergus, 2018) by annealing it over the course of training (Babaeizadeh et al., 2018; Lee et al., 2018) or learned priors (Denton & Fergus, 2018; Villegas et al., 2019; Castrejon et al., 2019; Wu et al., 2021a).
Each of these design choices introduces hyperparameters, tuning burden, and additional work when applying a model to a new task.
FitVid does not require any of these details: we simply train FitVid by optimizing L ( x ) using Adam (Kingma & Ba, 2014).
E XPERIMENTS
To evaluate FitVid, we test it on four different real-world datasets and compare its performance with prior state-of-the-art methods, with comparable parameter count, using four different metrics.
Our main goal is to demonstrate that FitVid can in fact ﬁt on these datasets well and illustrate how augmentation can prevent FitVid from overﬁtting, resulting in state-of-the-art prediction performance.
Please visit the anonymized website sites.google.com/view/fitvid to see samples of generated videos.
The code to reproduce these experiments is available in Supplementary Material.
4. E XPERIMENTATION S ETUP
Datasets: To test FitVid, we use four datasets that cover a variety of real-life scenarios.
We use the Human3.6M dataset (Ionescu et al., 2014), which consists of actors performing various actions in a room to study the structured motion prediction.
We also use the KITTI dataset (Geiger et al., 2013b) to evaluate FitVid’s ability to handle partial observability and dynamic backgrounds.
For
[Table]
both datasets, we followed the pre-processing and testing format proposed by Wu et al.
(2021a)
and
Villegas et al.
(2019), which predicts 25-frames conditioned the previous ﬁve at 64 × 64 resolution.
To evaluate FitVid in an action-conditioned setting, we use RoboNet dataset (Dasari et al., 2019).
This large dataset includes more than 15 million video frames from 7 different robotic arms pushing objects in different bins.
It contains a wide range of conditions, including different viewpoints, objects, tables, and lighting.
Prior video prediction methods have a tendency to badly underﬁt on this dataset (Dasari et al., 2019).
We follow the same setup as Wu et al.
(2021a) and randomly selectvideos for testing.
Similar to Wu et al.
(2021a), we train FitVid to predict next ten frames given two context frames as well as the ten future actions.
Finally, to compare FitVid to a wider range of prior work, we use the BAIR robot pushing dataset (Ebert et al., 2017), which is a widely-used benchmark in the video prediction literature.
We follow the evaluation protocol of Rakhimov et al.
(2020), which predicts the next 16 frames given only one context frame and no actions.
Given the high stochasticity of robotic arm movement in
BAIR, and the variety of unseen objects in the test set, it is a great benchmark for evaluating the model’s ability to generate diverse outputs.
Metrics: We evaluate our method and prior models across four different metrics: Structural Similarity
Index Measure (SSIM) (Wang et al., 2004), Peak Signal-to-noise Ratio (PSNR) (Huynh-Thu &
Ghanbari, 2008), Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) and
Fréchet Video Distance (FVD) (Unterthiner et al., 2018).
FVD measures the overall visual quality and temporal coherence without reference to the ground truth video.
PSNR, SSIM, and LPIPS measure pixel-wise similarity to the ground-truth with LPIPS most accurately representing human perceptual similarity.
Given the stochastic nature of video prediction benchmarks, we follow the standard stochastic video prediction evaluation protocol (Babaeizadeh et al., 2018; Villegas et al., 2019; Wu et al., 2021a): we sample 100 future trajectories per video and pick the best one as the ﬁnal score for PSNR, SSIM and LPIPS.
For FVD, we use all 100 with a batch size of 256.
4. R ESULTS
Comparisons: First, we compare FitVid to GHVAE (Wu et al., 2021a) and SVG (Villegas et al., 2019).
GHVAE is a hierarchical variational video prediction model trained in a greedy manner.
SVG is a large-scale variational video prediction model with learned prior and minimal inductive bias.
We chose these two baseline because they both investigated overﬁtting by scaling the model, and achieve state-of-the-art results.
However, SVG reported no overﬁtting even for their biggest model ( ( M = 3 , K = 5) ) with 298M parameters (Villegas et al., 2019) while GHVAE (with 599M paramteres) reported "some overﬁtting" on smaller datasets (Wu et al., 2021a).
At the same time, both of these models share a similar architecture to FitVid.
Table 1 contains the results of these experiments.
As it can be seen in this table, FitVid outperforms both SVG and GHVAE across all metrics in Robonet and Human3.6M.
In KITTI, FitVid also consistently outperforms SVG while either improves or closely matches the performance of GHVAE which has more than twice as parameters.
For qualitative results, see Figures 4, 5 and 6.
Comparison to non-variational methods: To provide a comprehensive comparison with prior methods, including non-variational models, we test FitVid on BAIR dataset (Ebert et al., 2017).
As can be seen in Table 2, FitVid outperforms most of the previous models in this setting while performing comparably to Video Transformer (Weissenborn et al., 2019) with 373M parameters.
[Figure]
Effect of Augmentation
[Figure]
[Table]
A NALYSIS
In this section, we take a closer look at the results from Section [Table] to analyse the consequences of overﬁtting and the effect of regularization on the current benchmarks.
On Human3.6M as a video prediction benchmark: Human3.6M (Ionescu et al., 2014) is a common benchmark in video prediction literature (Finn et al., 2016a;
Babaeizadeh et al., 2018; Villegas et al., 2017; Wang et al., 2019b; Villegas et al., 2019; Lin et al., 2020; Franceschi et al., 2020; Guen & Thome, 2020; Wu et al., 2021a;c) which we also use to evaluate FitVid (Figure 6).
At the ﬁrst glance, it seems that the model is generating extremely detailed and human-like motions conditioned on the given context pose.
However, on closer inspection, we observe that the human subject in the predicted video is changing.
In fact, FitVid replaces the unseen human subject into a training subject which is particularly evident from the clothing.
Actually, we can ﬁnd similar video clips from the training data for each one of the predicted videos (see Figure 6).
These frames are not exactly the same, but they look notably similar.
This observation indicates that:
The model can generalize to unseen frames, poses and subjects since the test context frames are new and unseen.
FitVid correctly distinguished the subject from the background, detects the human pose and continues the video from there.
The model memorized the motion and the appearance of the training subjects.
The model morphs the test human subject into a training one, and then plays a relevant video from the memory.
This means that FitVid fails to generalize to a new subject, while still generalizing to unseen frames.
Given that the Human3.6M has
ﬁve
training and two test subjects ( Finn et al , 2016a; Villegas et al 2017) this may not be surprising
Nevertheless, this observation shows how the current low-resolution setup for Human3.6M is not suitable for large-scale video prediction.
In fact, after this observation, we traced the same behaviour in other video prediction literature and, unfortunately, it seems this is a common and overlooked issue.
For example, the same phenomena can be seen in Figure 6 from Franceschi et al.
(2020) that shows changing the test to a training subject by Struct-VRNN (Minderer et al., 2019)
and the proposed method by Franceschi et al.
(2020) (note the changed shirt).
For others examples, see Figure 7 of
Villegas et al.
and Figure 5 of Villegas et al.
See Figure 15 for a copy of these ﬁgures.
Overﬁtting and regularization:
As mentioned in Section 1, there is considerable evidence that current video prediction models tend to underﬁt when trained on large datasets (Villegas et al., 2019;
Castrejon et al., 2019; Wu et al., 2021a).
Wu et al.
(2021a), which is the current state-of-the-art model with 599 million parameters, reported "some overﬁtting" on smaller datasets such as Human3.6M and
KITTI.
However, we observe severe and clear overﬁtting with FitVid, despite having only 302 million parameters.
Figure 7 visualizes the training and evaluation LPIPS metric while training FitVid on
Human3.6M, without augmentation .
This graph demonstrates that the training keeps getting better while the test quality starts to get worse after ∼ 15K iterations.
We also observed similar behaviour on KITTI, as can be seen in Figure 7b.
These results clearly shows that FitVid is overﬁtting on
Human3.6M and KITTI, indicating that FitVid is using its parameters more efﬁciently.
As mentioned
G r o un d
T r u t h
[Figure]
t =
P r e d i c t e d t = t = t = 1st
Context Frame t
= t = t =
t =
[Figure]
P r e d i c t
[Figure]
d t = t =
t = 1st
Context Frame t
= t = t =
t =
[Figure]
r e d i c t e d
C l o s e s t T r a i n i n g
[Figure]
t = 1st
Context Frame t
= t = t =
t =
More videos can be found at sites.google.com/view/fitvid .
[Figure]
in Section 3.2, to address overﬁtting, we use augmentation which results in state-of-the-art prediction quality, as reported in Table 1.
Overﬁtting on Robonet: We did not observe any overﬁtting on RoboNet, which is expected given the fact that RoboNet is much larger compared to the other benchmarks.
Trying to ﬁnd a model that can overﬁt on RoboNet, we test a scaled version of FitVid with 500M parameters – which is still smaller compared to GHVAE with 599M parameters and reported no overﬁtting on this dataset.
This scaled version of FitVid overﬁts on RoboNet, as demonstrated in Figure 7d.
Note that we did not use this scaled version in the reported numbers of Table 1, which is generated using the 302M version.
Our goal here was to demonstrate that a scaled version of FitVid can also use its parameters more efﬁciently, compared to prior models, leading to overﬁtting on even bigger datasets such as RoboNet.
[Table]
Effect of Augmentation on SVG : There is a discrepancy between the input data for training the models in Section 4.
FitVid is trained with augmentation while the baselines are trained without any augmentation which raises a question: can the better performance of FitVid be explained only by the augmentation?
In other words, do the previous methods beneﬁt from augmentation too?
To answer this question, we retrain SVG with and without augmentation.
As demonstrated in Table 3, SVG performs worse if trained with augmented data, supporting the claim that it is underﬁtting to the raw data.
As a result, this experiment provides more support for FitVid truly overﬁtting on these datasets and therefore beneﬁting from augmentation.
Please note that we included our SVG results without augmentation too, as we could not perfectly reproduce the numbers reported by Villegas et al.
[Table]
Ablation Study
In order to ﬁgure out how much each design element in FitVid is contributing to its performamce, we ablate it and check its performance on RoboNet.
In this study we try to cover a variety of ablations including encoder/decoder ablations, adding or removing modeling features such as learned prior as well as various skip connection implementations.
As can be seen in Table 4, skip connections are a crucial component for archiving high prediction performance.
This table also demonstrates that, while modeling sotchasticity is importnat even in action conditioned settings, FitVid does not beneﬁt from a learned prior.
For more ablation studies, please look at Section A.1.
[Table]
works and almost no domain shift from training to testing.
In this section, we are interested in investigating whether FitVid is capable of generalizing to a similar but visually different task with no training data for this new domain.
Therefore, we setup a real-robot experiment, with a Franka
Emika Panda robot arm, in which the goal is to push a speciﬁc object to a predetermined goal position.
We train FitVid on RoboNet and use cross-entropy method (CEM) (Rubinstein, 1997;
Chua et al., 2018) for planning (please see Appendix for details).
As can be seen in Table 5, this agent is unable to generalize to the new domain, achieving worse performance than a random agent.
This may not be surprising given the fact that the videos in RoboNet have entirely different robots and visuals, although the robots are performing the same task (i.e. pushing objects in a bin using a robotic arm).
We then try to bring the training and test domain closer to each other by ﬁne-tuning FitVid on the data from Wu et al.
(2021a).
This data contains 5000 autonomously collected videos of a Franka
Emika Panda robot arm pushing objects around which look more similar to our setup compared to
RoboNet, but still contain different lighting, camera angle, and target objects.
This time, we observe that FitVid is relatively successful at generalizing to the new domain, succeeding in 56% of the trials.
Finally, we ﬁnd that adding data augmentation to the ﬁne-tuning improves the generalization ability of the model, achieving 78% success rate.
These results illustrate that while large distribution shift adaptation (RoboNet) remains difﬁcult, by using data augmentation FitVid is capable of adapting to a relatively new domain (from Wu et al.
(2021a) data).
C ONCLUSION
We propose FitVid, a simple and scalable variational video prediction model that can attain a signiﬁcantly better ﬁt to current video prediction datasets even with a similar parameter count as prior models.
In fact, while prior methods generally suffer from underﬁtting on these datasets, naïvely applying FitVid actually results in overﬁtting.
We therefore propose using a set of existing image augmentation techniques that prevent overﬁtting in video prediction settings, leading to state-of-the-art results across a range of prediction benchmarks.
To the best of our knowledge, this is the ﬁrst time a model reports, not only not underﬁtting, but rather substantial overﬁtting on these benchmarks.
This is particularly important because underﬁtting is usually cited as one the main reasons for low quality predictions of the future frames.
We demonstrate that simple existing image augmentation techniques can be used to prevent the mode from overﬁtting, resulting in high quality images.
In fact, while each of the components of our proposed model are not novel on their own, FitVid outperforms the current state-of-the-art models across four different video prediction benchmarks on four different metrics.
Moreover, the ﬁt attained by FitVid is so good that a number of previously used benchmarks and metrics can be fooled, resulting in undesired outcomes, which are often overlooked in the video prediction literature.
There are many ways that FitVid can be expanded.
As mentioned in the text, one of the interesting features of our proposed method is that it is simple.
It is non-hierarchical, convolutional, with no attention mechanism, no curriculum learning, and no training scheduling.
Any of these features can potentially improve the results of FitVid in order to generate even higher quality images.
Given the simplicity of FitVid, it can be easily built upon.
Another interesting direction would be to introduce new training-aware metrics for video prediction and generation to signal when a model is generating high quality videos by repeating the training data.
E THICS S TATEMENT
Videos are an abundant source of visual information about our physical world.
They contain information about objects, humans and how they interact with each other.
The goal of video prediction is to foremost learn a representation of the world, usable for downstream tasks by an agent.
Second, is to predict what happens next, conditioned on the past and the future intents, which can be used for planning.
Despite many recent advances in this ﬁeld, the present day models are still relatively low-quality and limited to narrow domains which makes their applications limited.
However, if improved, video prediction or representations learned by video prediction, can be a major step forward toward fully autonomous self-learning agents.
This paper, we believe, takes an important step towards this goal by pushing state-of-the-art forward and simplifying it in a meaningful way.
However, our model is vulnerable to the bias in the training data and if adopted widely, this can skew research in certain directions.
For example, our results may lead to higher quality models which can be scaled to generate even higher quality results.
These models will be harder to design and train and require more computational power, and potentially can be biased.
Finally, the underlying techniques for video prediction can be misused for generating high-quality videos that are misleading, depicting deliberately false situations or persons, similar to the phenomenon of deepfakes (Korshunov & Marcel, 2018).
However, we believe that our experiments are conducted in relatively speciﬁc narrow settings and conditions which will likely not generalize broadly.
Speciﬁcally, it is especially challenging to generate high-quality realistic-looking videos even with such state of the art methods.
R EPRODUCIBILITY S TATEMENT
The anonymized implementation of the model, as well as the code to train and evaluate FitVid is attached in the Supplementary Material.
This code can be used to regenerate the results reported in this paper.
The approximated computation cost of rerunning these experiments is available in
Section A.2.1.
Appendix also includes details of implementation such as the training and evaluation details in Section A.2 as well as augmentation details in Algorithm 3.
The detailed architecture of the encoder networks can be found in Table 9, decoder in Table 11, and the dynamic model in Tableof the Appendix.
The hyper-parameters used are reported in Table 8.
Table 13 includes the license of the benchmark datasets
A A PPENDIX A. M ORE ABLATION STUDIES
In order to demonstrate how the design decisions behind FitVid’s architecture affect its performance, we conduct more ablation studies.
In the ﬁrst study, we replace FitVid’s encoder and decoder blocks with the blocks from
SVG, while keeping everything else the same, and report the prediction performance on RoboNet in Table 6.
From this table, it can be seen that the new encoder and decoder blocks are much more efﬁcient at modeling the videos, resulting in improved prediction metrics.
Please note that the numbers reported for SVG are different from Table 1, because we used FitVid ’s dynamic model in this study.
In another study, we analyze the effect of the number of encoding and decoding blocks on the performance of the model.
We very the number of blocks from one to three and measure the prediction quality on RoboNet.
As it can be seen in Table 7, it’s not just about making the architecture bigger, but that we are actually beneﬁting from the particular design of our encoder and decoder i.e. having exactly two encoding blocks and two decoding blocks is the optimal spot for achieving the best results in a reasonable number of training iterations while maintaining the number of parameters in the same ballpark as previous work.
A. I MPLEMENTATION D ETAILS
In this section we describe the details of FitVid’s architecture as well as training.
Algorithm 1 and Algorithm 2 describe the high-level training and prediction process respectively.
Look at Table 8 for the used hyper-parameters
We used the same set of hyper-parameters across all experiments.
As mentioned in Section 3.2, all of the hyper- parameters are ﬁxed during the training and there is no scheduling.
Table 9 and Table 11 include the detailed architecture of the encoder and the decoder.
Table 10 describes the structure of dynamic and posterior networks.
Finally, Table 12 describes the augmentations details.
A.2. C OMPUTATION R ESOURCES
We implement FitVid using Flax (Heek et al., 2020) library for JAX (Bradbury et al., 2018).
We train FitVid on 4 × 4 TPUs (32 co-processors).
Each training step (with global batch size of 128 or local batch size of 4) takes ∼ 1020 milliseconds (i.e. 0 .
98 step per second).
In parallel, we evaluate the model every 1000 training steps on a single V100 GPU which takes about ∼ 850 milliseconds for a batch size of 128.
The models are trained for one million training iterations which takes ∼ 12 days to complete.
Algorithm
[Table]
[Table]
[Table] [Table] [Figure]
[Table]
[Figure]
u
[Figure]
[Table]
[Table]
[Table]
Algorithm 3: Video Augmentation.
end return
Hyper-parameter
[Table]
Dataset
Reference
License
RoboNet
Dasari et al.
MIT
KITTI
Geiger et al.
(2013a)
Creative Commons
Human3.6M
Ionescu et al.
License
BAIR robot pushing
Ebert et al.
GitHub Default
A. R OBOT E XPERIMENT D ETAILS A.3. E NVIRONMENT
The robot environment consists of a Franka Emika Panda robot operating over a bin which contains various objects.
The robot’s observations are 64 × 64 × 3 RGB images, and its action space consists of 3 DOF delta position control of the end effector with action magnitudes in the range
A.3. D ATA
The data used for ﬁnetuning the model used in robot experiments was taken directly from Wu et al.
(2021a).
This data consists of different viewpoint, lighting conditions, and target objects than what is used in our evaluation.
A.3. E VALUATION D ETAILS
During evaluation, the agent is speciﬁed to complete an object pushing task by a goal image.
The agent has 50 timesteps to complete the task, and a trial is measured as successful if the majority of the object overlaps with its target position at some point in the episode.
Each method is evaluated over 18 trials, of which 6 consist of objects in an “ofﬁce” setting, 6 consist of objects in a “kitchen” setting, and 6 consist of objects in a “cleaning” setting (See Figure 9).
To execute the task, the agent performs visual model predictive control using the cross entropy method (CEM).
Speciﬁcally, the agent takes in 1 frame, and predicts trajectories of 10 time-steps for 200 different sampled action sequences.
Trajectories are ranked according to their negative mean squared error to the goal image, averaged across all 10 time-steps.
The action distribution reﬁts to the top 20 actions, and repeats for 3 iterations of CEM.
Afterwards the best sequence of 10 actions is stepped in the environment in an open loop faction.
The process repeats 5 times until the end of the episode.
Ground Truth 1 s t
C o n t e x t
F r a m e t =
Predicted t =
t = t = t =
t = t = t = t =
t =
Ground Truth
Predicted
F i gu r e 11 :
M o r e d e t a il e d v i d e o fr o m F i gu r e 4 w
h
i c h ill u s t
r a t e s
F it V i d on ac ti on - c ond iti on e d
R obo
N e t (
D a s a r i e t a l ., ) .
T h
e m od e l
i s c ond iti on e d on t
h e ﬁ r s t t w o fr a m
e s a nd
i s p
r e d i c ti ng t h e n e x t t e n fr a m
e s g
i v e n t h
e f u t u r e ac ti on s o f t
h e r obo ti c a r m .
T h
e s e ﬁ gu r
e
s d e m on s t r a t e ho w t
h e p r
e
d
i c t e d m ov e m e n t s o f t h e a r m c
l o s e l y f
o ll o w s t
h e g r ound t r u t
h
g
i v e n t h a t t
h
e f u t u r e ac ti o n s i s kno w n .
T h
e m od e l a l s o p r
e d i c t s d e t a il e d m ov e m e n t s o f t h e pu s h e d ob j ec t s ( v i s i
b l e i n t h e t op e x a m p l e ) a s w e ll a s ﬁ lli
ng i n t
h
e
p r e v i ou s l y un s ee n b ac kg r ound w it
h s o m e r a ndo m ob j ec t s ( l ook a t t h e ob j ec t t
h a t a pp ea r b e h
i nd t
h e r obo ti c a r m i n t h e bo tt o m e x a m p l e ) .
A l s o no ti ce t
h e w r ong p r e d i c ti on s o f r obo t s ﬁ ng e r s i n t h e bo t t o m e x a m p l e .
Ground Truth 1 s t
C o n t e x t F r a m e t =
Predicted t =
t =t =t = t = t =
t = t = t = t =
t = t =
Ground Truth
Predicted
F i gu r e 12 :
M o r e d e t a il e d v i d e o fr o m F i gu r e 5 w h
i c h ill u s t
r a t e s
F it V i d on
K I
TT I d a t a s e t (
G e i g e r e t a l ., 2013b ) .
A s it ca n b e s ee n i n t h
i s ﬁ gu r e , t
h e m od e l g e n e r a t e s h
i gh qu a lit y p r e d i c ti on o f t
h e f u t u r e
i n a dyn a m i
c s ce n e .
N o t e ho w i n t h e t op e x a m p l e
F it V i
d k ee p s p r e d i c ti ng t
h e m ov e m e n t o f t h e s h a do w on t h e g r ound till it m ov e s ou t o f t h e fr a m e .
A f t e r t h a t , t
h e m od e l k ee p s pu s h
i ng t h e b ac kg
r ound c
l o s e r i n eac h fr a m e , i m p l y
i ng
d r i v
i ng f o r
w a r d .
W e no ti ce d t h a t t
h e qu a lit y
o f
p r e d i c ti on s d r op s ub s t a n ti a ll y f a s t
e
r w h e n t
h e r e a r e m o r e ob j ec t s i n t
h e s ce n e e .
g .
t h e d r i
v
i ng s ce n e s i n s i
d
e a c it
y a s ca n b e s ee n i n t h e bo tt o m e x a m p l e .
T h
i s i nd
i ca t e s t
h e m od e l s till f a il s t o g e n e r a li
ze t o m o r e c o m p l e
x s ce n e s w it
h
m o r
e m ov
i ng s ub j ec t s .
Ground Truth 1 s t
C o n t e x t F r a m e t =
Predicted t =
t =t =t = t = t =
t = t = t = t =
t = t =
Ground Truth
Predicted
F i gu r e 13 :
M o r e d e t a il e d v i d e o fr o m
F i gu r
ew
h
i c h ill u s t
r a t e s
F it V i d on
H u m a n .
6 M (I on e s c u e t a l .
T h
i s ﬁ gu r e d e m on s t r a t
e s e x t r e m e l y d e t a il e d a nd hu
m a n - li k e m o ti on s p
r
e
d
i c t e d by
F it V i
d , c ond iti on e d on t
h e g
i v e n c on t e x
t
fr a m e s .
H o w e v e r , on c l o s e r i n s p ec ti on , it ca n b e s ee n t h a t t h e hu
m
a n s ub j ec t i n t h e v i d e o i s c
h a ng i ng , fr o m t h e t e s t s ub j ec t t o a t r a i n i ng s u b j ec t .
T h
i s i s p a r ti c u l a r l y e v i d e n t fr o m t
h e c l o t h s .
T h
i s ph e no m e n a i
nd
i ca t e s t h a t , a lt hough
F it V i
d i s ca p a b l e o f g e n e r a li z
i ng t o t h e fr a m e s ou t
o f t r a i n i ng d i s t r i b u ti on , ho w e v e r , it m o r ph s t h e hu
m
a n s ub j ec t i n t o a f a m ili a r on e fr o m t
h e t r a i n i ng s e t a nd t h e n p l a y s t
h e v i d e o fr o m t
h e m e m o r y .
[Table]
o n t e x t
F r a m e t =
Predicted t =
t = t = t = t = t =
t =
[Table]
Truth
Predicted
F i gu r e 14 :
M o r e d e t a il e d v i d e o fr o m F i gu r
e 8 w h
i c h ill u s t
r a t e s
F it V i d on
B A I
R r obo t pu s h
i ng d a t a s e t ( E b e r t e t a l ., ) w it h
no ac ti on s .
T h
e m od e l
i s c ond iti on e d on l y on t h e ﬁ r s t fr a m
e a nd
i s p
r e d i c ti ng t h e n e x t fr a m e s .
G i v e n t h a t t h e f u t u r e
ac
ti on s o f t
h e r obo ti c a r
m
i s unkno w n , t h e p
r e d i c ti on ca n d i v e r g e s ub s t a n ti a ll
y fr o m t h e g r ound t r u t h v i d e o .
H o w e v e r , t
h
e m od e l p r
e
d i c t s m ov e m e n t s f o r t
h e ob j ec t s w h e n e v e r t h e a r m pu s h e s t
h e ob j ec t i n a n i m a g i n a r y s ce n a r i o .
I t a l s o ﬁ ll s t h e b ac kg r ound w it h r a ndo m ob j ec t s .
of
Copy of
5 from al