G ENERATING V IDEOS WITH D YNAMICS - AWARE I MPLICIT G ENERATIVE A DVERSARIAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
In the deep learning era, long video generation of high-quality still remains challenging due to the spatio-temporal complexity and continuity of videos.
Existing prior works have attempted to model video distribution by representing videos as 3D grids of RGB values, which impedes the scale of generated videos and neglects continuous dynamics.
In this paper, we found that the recent emerging paradigm of implicit neural representations (INRs) that encodes a continuous signal into a parameterized neural network effectively mitigates the issue.
By utilizing INRs of video, we propose dynamics-aware implicit generative adversarial network (DIGAN), a novel generative adversarial network for video generation.
Speciﬁcally, we introduce (a) an INR-based video generator that improves the motion dynamics by manipulating the space and time coordinates differently and (b) a motion discriminator that efﬁciently identiﬁes the unnatural motions without observing the entire long frame sequences.
We demonstrate the superiority of DIGAN under various datasets, along with multiple intriguing properties, e.g ., long video synthesis, video extrapolation, and non-autoregressive video generation.
For example, DIGAN improves the previous state-of-the-art FVD score on UCF-101 by 30.7% and can be trained on 128 frame videos of 128 × 128 resolution, 80 frames longer than the 48 frames of the previous state-of-the-art method.
I NTRODUCTION
Deep generative models have successfully synthesized realistic samples on various domains, including image (Brock et al., 2019; Karras et al., 2020b; 2021; Dhariwal & Nichol, 2021), text (Adiwardana et al., 2020; Brown et al., 2020), and audio (Dhariwal et al., 2020; Lakhotia et al., 2021).
Recently, video generation has emerged as the next challenge of deep generative models, and thus a long line of work has been proposed to learn the video distribution (Vondrick et al., 2016; Kalchbrenner et al., 2017;
Saito et al., 2017; 2020; Tulyakov et al., 2018; Acharya et al., 2018; Clark et al., 2019;
Weissenborn et al., 2020; Rakhimov et al., 2020; Tian et al., 2021; Yan et al., 2021).
Despite their signiﬁcant efforts, a substantial gap still exists from large-scale real-world videos.
The difﬁculty of video generation mainly stems from the complexity of video signals; they are continuously correlated across spatio-temporal directions.
Speciﬁcally, most prior works interpret the video as a 3D grid of RGB values, i.e ., a sequence of 2D images, and model them with discrete decoders such as convolutional (Tian et al., 2021) or autoregressive (Yan et al., 2021) networks.
However, such discrete modeling limits the scalability of generated videos due to the cubic complexity (Saito et al., 2020) and ignores the inherent continuous temporal dynamics (Gordon & Parde, 2021).
Meanwhile, implicit neural representations (INRs; Sitzmann et al.
; Tancik et al.
(2020)) have emerged as a new paradigm for representing continuous signals.
INR encodes a signal into a neural network that maps input coordinates to corresponding signal values, e.g ., 2D coordinates of images to RGB values.
Consequently, INR amortizes the signal values of arbitrary coordinates into a compact neural representation instead of discrete grid-wise signal values, requiring a large memory proportional to coordinate dimension and resolution.
In this respect, INRs have shown to be highly effective at modeling complex signals such as 3D scenes (Mildenhall et al., 2020; Li et al., 2021a).
Furthermore, INR has intriguing properties from its compactness and continuity, e.g ., reduced data memory (Dupont et al., 2021a) and upsampling to arbitrary resolution (Chen et al., 2021).
[Figure]
DIGAN can train these videos with 4 NVIDIA V100 GPUs, while the prior state-of-the-art method,
DVD-GAN, uses more than 32 (up to 512) TPUs for training 48 frame videos of the same resolution.
Several works utilize INRs for generative modeling, i.e ., samples are generated through INRs (Chan et al., 2021;
Dupont et al., 2021b; Kosiorek et al., 2021).
In particular, Skorokhodov et al.
(2021) and Anokhin et al.
(2021) exposed that INR-based image generative adversarial networks (GANs;
Goodfellow et al.
(2014)), which generate images as INRs, show impressive generation performance.
Interestingly, they further merit various advantages of INRs, e.g ., natural inter- and extra-polation, anycost inference ( i.e ., control the trade-off of quality and cost), and parallel computation, which needs a non-trivial modiﬁcation to apply under other generative model architectures.
Inspired by this, we aim to design an INR-based (or implicit) video generation model by interpreting videos as continuous signals.
This alternative view is surprisingly effective as INRs compactly encode the videos without 3D grids and naturally model the continuous spatio-temporal dynamics.
While na ¨ ıvely applying INRs for videos is already fairly effective, we found that a careful design of separately manipulating space and time signiﬁcantly improves the video generation.
Contribution.
We introduce dynamics-aware implicit generative adversarial network (DIGAN), a novel INR-based GAN architecture for video generation.
Our idea is two-fold (see Figure 2):
• Generator: We propose an INR-based video generator that decomposes the motion and content (image) features, and incorporates the temporal dynamics into the motion features.
2 To be speciﬁc, the proposed generator encourages the temporal coherency of videos by regulating the variations of motion features and enhancing the expressive power of motions with an extra nonlinear mapping.
Furthermore, our proposed generator can create videos with diverse motions sharing the initial frame by conditioning a random motion vector to the content vector.
• Discriminator:
Given generated INRs of videos, we propose a motion discriminator that efﬁciently detects the unnatural motions from a pair of images (and their time difference) instead of a long sequence of images.
Speciﬁcally, DIGAN utilizes a 2D convolutional network for the motion discriminator, unlike prior works that employ computationally heavier 3D convolutional networks to handle the entire video at once.
Such an efﬁcient discriminating scheme is possible since the INRs of videos can easily produce highly correlated frames at arbitrary times.
We demonstrate the superiority of DIGAN on various datasets, including UCF-101 (Soomro et al., 2012), Tai-Chi-HD (Siarohin et al., 2019), Sky Time-lapse
(Xiong et al., 2018), and a food class subset of Kinetics-600 (Carreira et al., 2018) datasets, e.g ., it improves the state-of-the-art results of Fr´echet video distance (FVD; Unterthiner et al.
(2018), lower is better) on UCF-101 from 833 to 577 (+30.7%).
Furthermore, DIGAN appreciates various intriguing properties, including,
• Long video generation : synthesize long videos of high-resolution without demanding resources on training, e.g ., 128 frame video of 128 × 128 resolution (Figure 1) • Time interpolation and extrapolation : ﬁll in the interim frames to make videos transit smoother, and synthesize the out-of-frame videos (Figure 4, Table 2) • Non-autoregressive generation : generate arbitrary time frames, e.g ., previous scenes (Figure 5), and fast inference via parallel computing of multiple frames (Table 3) • Diverse motion sampling : sample diverse motions from the shared initial frames (Figure 6) • Space interpolation and extrapolation : upsample the resolution of videos (Figure 7, Table 4) and create zoomed-out videos while preserving temporal consistency (Figure 8)
To the best of our knowledge, we are the ﬁrst to leverage INRs for video generation.
We hope that our work would guide new intriguing directions for both video generation and INRs in the future.
R ELATED WORK
Image generation.
Image generation has achieved remarkable progress, with the advance of various techniques, including generative adversarial networks (GANs; Goodfellow et al.
(2014)), autoregressive models (Ramesh et al., 2021), and diffusion models (Dhariwal & Nichol, 2021).
In particular,
GANs have been considered as a de facto model for image generation, producing high-resolution images with fast inference (Brock et al., 2019; Karras et al., 2020b; 2021).
Our model is inspired by recent GAN architectures (Karras et al., 2020b; Skorokhodov et al., 2021) and training techniques (Zhao et al., 2020; Karras et al., 2020a), and extends these methods for video generation.
Video generation.
Following the success of GANs on images, most prior works on video generation considered the temporal extension of image GANs (Vondrick et al., 2016;
Saito et al., 2017; 2020; Tulyakov et al., 2018; Acharya et al., 2018; Clark et al., 2019;
Yushchenko et al., 2019; Kahembwe & Ramamoorthy, 2020; Gordon & Parde, 2021; Tian et al., 2021; Fox et al., 2021; Munoz et al., 2021).
Another line of works (Kalchbrenner et al., 2017; Weissenborn et al., 2020; Rakhimov et al., 2020; Yan et al., 2021) train autoregressive models over pixels or discretized embeddings.
Despite their signiﬁcant achievement, a large gap still exists between generated results and large-scale realworld videos.
We aim to move towards longer video generation by exploiting the power of implicit neural representations.
We provide more discussion on related ﬁelds in Appendix C.
Implicit neural representations.
Implicit neural representations (INRs) have recently gained considerable attention, observing that high-frequency sinusoidal activations signiﬁcantly improve continuous signal modeling (Sitzmann et al., 2020; Tancik et al., 2020).
In particular, INR has proven its efﬁcacy in modeling complex signals such as static (Chen et al., 2021;
Park et al., 2021; MartinBrualla et al., 2021) and dynamic (Li et al., 2021a;b; Pumarola et al., 2021; Xian et al., 2021)
3D scenes.
Unlike prior works on dynamic INRs focusing on modeling a single signal of a 3D scene or video, we aim to learn a generative model over 2D videos.
Also, while prior works focus more on rendering visually appealing 3D scenes, we aim to learn the diverse motion dynamics of videos.
Generative models with INRs.
Following the success of single signals, several works utilize INRs for generative modeling.
One line of works synthesizes the INR weights correspond to the signals via hypernetwork (Skorokhodov et al., 2021;
Anokhin et al., 2021; Chan et al., 2021; Dupont et al., 2021b).
The other line of works controls the generated signals via input condition, i.e ., concatenate the latent code corresponding to the signal to the input coordinates (Schwarz et al., 2020; Kosiorek et al., 2021).
In particular, Skorokhodov et al.
(2021) and Anokhin et al.
(2021) have demonstrated the effectiveness of INR-based generative models for image synthesis.
We extend the applicability of INR-based GANs for video generation by incorporating temporal dynamics.
D YNAMICS - AWARE IMPLICIT GENERATIVE ADVERSARIAL NETWORK
The goal of video generation is to learn the model distribution p G ( v ) to match with the data distribution p data ( v ) , where a video v is deﬁned as a continuous function of images i t for time t ∈ R , and an image i t is a function of spatial coordinates ( x, y ) ∈ R 2 .
To this end, most prior works considered videos as the discretized sequence of images, i.e ., 3D grid of RGB values of size H × W × T , and generated videos with discrete decoders such as convolutional (Tian et al., 2021) or autoregressive
[Figure]
𝒛 "
Motion weight 𝒘 #
Content weight of 1 st layer 𝒘 $ , 𝒘 %
Content weight of upper layers
Frequencies 𝜎 $ ,𝜎 % ≫ 𝜎 #
Video INR 𝜎 $ 𝑥 𝜎 % 𝑦 𝜎 # 𝑡
R G B (a) Generator 2D grid { 𝑥,𝑦 }
Time 𝑡 !
Video INR 2D grid { 𝑥,𝑦 }
Time 𝑡 " 𝐷 !
𝐷 "
Real/Fake
Real/Fake Δ𝑡 ≔ 𝑡 !
− 𝑡 "
Image 𝒊 # !
RGB 𝑥𝑦𝑡 RGB 𝑥𝑦𝑡
Image 𝒊 # " = (b) Discriminator (Yan et al., 2021) networks.
However, the discretization limits video generation hard to scale due to the cubic complexity of generated videos and ignores the continuous dynamics.
Our key idea is to directly model the videos as continuous signals using implicit neural representations (INRs; Sitzmann et al.
; Tancik et al.
Speciﬁcally, we propose dynamics-aware implicit generative adversarial network (DIGAN), an INR-based generative adversarial network (GAN; Goodfellow et al.
for video generation.
Inspired by the success of INR-based GANs for image synthesis (Skorokhodov et al., 2021; Anokhin et al., 2021), DIGAN extends the implicit image GANs for video generation by incorporating temporal dynamics.
We brieﬂy review INRs and
INR-based GANs in Section 3.1 and then introduce our method DIGAN in Section 3.2.
3. G ENERATIVE MODELING WITH IMPLICIT NEURAL REPRESENTATIONS
Consider a signal v ( · ) :
R m → R n of a coordinate mapping to the corresponding signal values, e.g ., videos as v ( x, y, t ) =
( r, g, b ) with m, n = 3 , where ( x, y, t ) are space-time coordinates and ( r, g, b ) are RGB values.
Without loss of generality, we assume that the range of coordinates for signals are [0 , 1] , e.g ., [0 , 1] 3 for videos.
INR aims to directly model the signal with a neural network v φ ( · ) :
R m → R n parametrized by φ , e.g ., using a multi-layer perceptron (MLP).
Recently, Tancik et al.
and Sitzmann et al.
(2020) found that sinusoidal activations with a high-frequency input σx , i.e ., sin( σx ) for σ (cid:29) 1 , signiﬁcantly improve the modeling of complex signals like 3D scenes (Mildenhall et al., 2020) when applied on the ﬁrst layer (or entire layers).
Here, one can decode the
INR v φ ( · ) as a standard discrete grid interpretation of signals by computing the values of pre-deﬁned grid of input coordinates, e.g ., { ( i − 1 H − 1 , j − 1 W − 1 , k − 1 T − 1 ) } ⊂ [0 , 1] 3 for videos of size H × W × T .
Leveraging the power of INRs, several works utilize them for generative models, i.e ., the generator
G ( · ) maps a latent z ∼ p ( z ) from a given prior distribution p ( z ) to an INR parameter φ = G ( z ) that corresponds to the generated signal v φ ( · ) ( e.g ., Chan et al.
Remark that INR-based generative models synthesize the function of coordinates, unlike previous approaches that directly predict the outputs of such a function, e.g ., 2D grid of RGB values for image synthesis.
Thus,
INR-based generative models only need to synthesize a ﬁxed size parameter φ , which reduces the complexity of the generated outputs for complicated signals.
It is particularly important for higherdimensional signals than 2D images, e.g ., video synthesis of grids require cubic complexity.
Speciﬁcally, Skorokhodov et al.
(2021) and Anokhin et al.
(2021) use GAN for training INR-based (or implicit) model for image generation.
GAN trains a discriminator D ( · ) to distinguish the real and generated samples, while generator G ( · ) aims to fool the discriminator.
By gradually improving their abilities from two-player game, the generator converges to the data distribution, i.e ., p G ≈ p data , where p G is a distribution of G ( z ) for z ∼ p ( z ) (Mescheder et al., 2018).
INR-based image GANs resemble the common GAN architectures such as StyleGAN2 (Karras et al., 2020b) but with a different generator to synthesize INRs.
In particular, INR-based image GANs employ conventional convolutional discriminators to identify the fake images decoded from INRs.
Remarkably, these approaches have shown comparable results with state-of-the-art GANs while meriting nice properties, e.g ., inter- and extra-polation and fast inference (Skorokhodov et al., 2021).
[Figure]
3. I NCORPORATING TEMPORAL DYNAMICS FOR IMPLICIT GAN S
Our method, DIGAN, is an INR-based video GAN that incorporates the temporal dynamics into the INRs.
Recall that the video INRs only differ from the image INRs by an extra time coordinate, i.e ., an input becomes a 3D coordinate ( x, y, t ) from a 2D coordinate ( x, y ) ; thus, one can utilize the implicit image GANs for video generation by only expanding the input dimension of INRs by one (for the time coordinate).
However, we found that a careful generator and discriminator design notably improves the generation quality and training efﬁciency.
We provide the overall illustration of DIGAN in Figure 2, and explain the details of DIGAN in the remaining section.
Generator.
A na ¨ ıve extension of implicit image GANs for synthesizing video INRs v φ ( x, y, t ) is to utilize them intactly, only modifying the ﬁrst layer of INRs to handle the extra time coordinate (we assume that the INRs follow the MLP structure).
This approach synthesizes the entire INR parameter φ at once, which overlooks the difference in space and time of videos, e.g ., the smooth change of frames over the time direction.
To tackle this issue, we ﬁrst evoke that the spatial and temporal terms of video INRs can be easily decomposed.
Speciﬁcally, let the output of the ﬁrst layer (without sinusoidal activations) of video INRs be σ x w x x + σ y w y y + σ t w t t + b , where w x , w y , w t , b are weights and biases of the ﬁrst layer and σ x , σ y , σ t > 0 are frequencies of coordinates x, y, t .
Here, note that only the term σ t w t t differs from the image INRs; one may interpret the outputs (of the ﬁrst layer) of video INRs as a continuous trajectory over time t from the initial frame (at t = 0 ) determined by the content parameter φ I :=
φ \ { w t } .
Inspired by this space-time decomposition view, we incorporate the temporal dynamics into the motion parameter w t .
To be speciﬁc, we propose three components to improve the motion parameter w t considering the temporal behaviors.
First, we use a smaller time-frequency σ t than space-frequencies σ x , σ y since the video frames change relatively slowly over time compared to the spatial variations covering diverse objects in images.
Accordingly, such a small frequency encourages the videos to be coherent over time.
Second, we sample a latent vector for motion diversity z M ∼ p M
( z M ) in addition to the original content (image) latent
vector z I ∼ p
I ( z I ) .
Here, the content parameter φ I =
G I
( z I ) is generated as the prior implicit image GANs, but the motion parameter w t = G M
( z I , z M ) is conditioned on both content and motion vectors; we provide the content vector since possible motions of a video often depend on the content.
Finally, we apply a non-linear mapping f M ( · ) on top of the motion features at time t to give more freedoms to motions, i.e ., f M ( w t t ) .
These simple modiﬁcations further improves the generation quality (Section 4.1) while creates diverse motions (Section 4.2). [Table] ± [Table]
Discriminator.
As the generated videos should be natural in both images and motions, prior works on video GANs are commonly equipped with two discriminators: an image discriminator D I and a motion (or video) discriminator D M
(Clark et al., 2019; Tian et al., 2021).
3 For image discriminator, one can utilize the well-known 2D convolutional architectures from the image GANs.
However, the motion discriminator needs an additional design, e.g ., 3D convolutional networks, where inputs are a sequence of images ( e.g ., the entire videos).
The 3D convolutional motion discriminators are the main computational bottleneck of video GANs, and designing an efﬁcient video discriminator has thus been widely investigated (Saito et al., 2020; Kahembwe & Ramamoorthy, 2020).
Instead of the expensive 3D architectures, we propose an efﬁcient 2D convolutional video discriminator utilizing the property of INRs.
We emphasize that the video INRs can efﬁciently generate two frames of arbitrary times t 1 , t 2 without generating the entire sequence, e.g ., as the autoregressive models.
Utilizing this unique property of INRs, we adopt the image discriminator to distinguish the triplet consists of a pair of images and their time difference ( i t 1 , i t 2 , ∆ t ) for ∆ t :=
t 1 − t 2 , by expanding the input channel from 3 to 7.
Thus, our proposed discriminator can learn the dynamics without observing long sequences (Section 4.1).
Moreover, we note that the two frames of INRs are highly correlated; the discriminator can focus on the artifacts in the motion.
E XPERIMENTS
We present the setups and main video generation results in Section 4.1.
We then exhibit the intriguing properties of DIGAN in Section 4.2.
Finally, we conduct ablation studies in Section 4.3.
4. V IDEO GENERATION
Model.
We implement DIGAN upon the INR-GAN (Skorokhodov et al., 2021) architecture, an INRbased GAN for image generation.
The content generator G I is identical to the INR-GAN generator, but the motion generator G M is added.
We set the spatial frequencies σ x = σ y = √ 10 following the original conﬁguration of INR-GAN, but we found that using a smaller value for the temporal frequency σ t performs better; we use σ t = 0 .
25 for all experiments since it worked well in our early experiments.
We use the same discriminator of INR-GAN for both image and motion discriminators but only differ for the input channels: 3 and 7.
See Appendix A.1 for details.
[Table]
[Figure] [Figure]
D
R e a l
G e n e r a t e d
[Table]
Datasets and evaluation.
We conduct the experiments on UCF-101 (Soomro et al., 2012), Tai-ChiHD (TaiChi; Siarohin et al.
, Sky Time-lapse (Sky; Xiong et al.
and a food class subset of Kinetics-600 (Kinetics-food; Carreira et al.
datasets.
All models are trained onframe videos of 128 × 128 resolution unless otherwise speciﬁed.
Speciﬁcally, we use the consecutive 16 frames for UCF-101, Sky, and Kinetics-food, but stride 4 ( i.e ., skip 3 frames after the chosen frame) for TaiChi to make motion more dynamic.
Following prior works, we report the Inception score (IS; Salimans et al.
, Fr ´ echet video distance (FVD; Unterthiner et al.
(2018)), and kernel video distance (KVD; Unterthiner et al.
We average 10 runs for main results and 5 runs for analysis with standard deviations.
See Appendix A.2 for more details.
Baselines.
We mainly compare DIGAN with prior works on UCF-101, a commonly used benchmark dataset for video generation.
Speciﬁcally, we consider VGAN (Vondrick et al., 2016), TGAN (Saito et al., 2017), MoCoGAN (Tulyakov et al., 2018), ProgressiveVGAN (Acharya et al., 2018), LDVDGAN (Kahembwe & Ramamoorthy, 2020), VideoGPT (Yan et al., 2021), TGANv2 (Saito et al., 2020), DVD-GAN (Clark et al., 2019), and MoCoGAN-HD (Tian et al., 2021), where the values are collected from the references.
For other experiments, we compare DIGAN with the state-of-the-art method, MoCoGAN-HD, tested using the ofﬁcial code.
See Appendix B for details.
Main results.
Figure 3 and Table 1 present the qualitative and quantitative video generation results of DIGAN, respectively.
DIGAN can model various video distributions, including unimodal videos like Sky and TaiChi and multimodal videos like UCF-101 and Kinetics-food.
In particular, Figurepresents that DIGAN produces reasonably good videos for challenging multimodal videos.
Also,
Table 1 exhibits that DIGAN signiﬁcantly outperforms the prior work on all datasets, e.g ., improves the FVD of MoCoGAN-HD from 833 to 577 (+30.7%) on UCF-101.
We remark that the Fr´echet inception distance (FID; Heusel et al.
(2017)), a metric for image quality, of DIGAN is similar to the MoCoGAN-HD.
Thus, the FVD gains of DIGAN come from better dynamics modeling.
[Figure]
t
i
[Figure]
n
M o t i o n
[Figure]
i ff e r
[Figure] [Figure]
[Table]
4. I NTRIGUING PROPERTIES
Long video generation.
The primary advantage of DIGAN is an effective generation of long and high-quality videos, leveraging the compact representations of INRs.
We also remark that DIGAN can be efﬁciently trained on the long videos, as the motion discriminator of DIGAN only handles a pair of images instead of long image sequences.
To verify the efﬁcacy of DIGAN, we train a model using 128 frame videos of 128 × 128 resolution from the TaiChi dataset.
Figure 1 shows that DIGAN produces a long and natural motion with reasonable visual quality.
To our best knowledge, we are the ﬁrst to report 128 frame videos of this quality.
See Appendix F for more results.
Time interpolation and extrapolation.
DIGAN can easily interpolate ( i.e ., ﬁll in interim frames) or extrapolate ( i.e ., create out-of-frame videos) videos over time by controlling input coordinates.
The videos inter- or extra-polated by DIGAN are more natural than those from discrete generative models, as INRs model videos continuously.
Table 2 shows that DIGAN outperforms MoCoGAN-HD on all considered inter- and extra-polation scenarios.
In particular, DIGAN is remarkably effective for time extrapolation as INRs regularize the videos smoothly follow the scene ﬂows deﬁned by previous dynamics.
Figure 4 shows that DIGAN can even extrapolate to 4 × longer frames while
MoCoGAN-HD fails on the Sky dataset.
See Appendix G for more results.
Non-autoregressive generation.
DIGAN can generate samples of arbitrary time: it enables DIGAN to infer the past or interim frames from future frames or parallelly compute the entire video at once.
It is impossible for many prior video generation approaches that sample the next frame conditioned on the previous frames in an autoregressive manner.
Figure 5 visualizes the forward and backward prediction results of DIGAN on the TaiChi dataset, conditioned on the initial frame indicated by the yellow box.
Here, we ﬁnd the content and motion latent codes by projecting the initial frames of t = { 6 , 7 , 8 } and predict the frames of t ∈ { 3 , . . .
DIGAN well predicts both past and future motions, e.g ., slowly folding arms.
DIGAN can also infer the interim frames from past and future frames, as shown in Appendix E.
On the other hand, Table 3 shows the generation speed of DIGAN is much faster than its competitors, VideoGPT and MoCoGAN-HD.
Different from prior works, we remark that DIGAN can compute multiple frames in parallel; generation can be N times further faster under N number of GPUs. For more analysis about the efﬁciency, see Appendix I. [Figure] [Figure]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure] Diverse motion sampling.
DIGAN can generate diverse videos from the initial frame by controlling the motion vectors.
Figure 6 shows the videos from two random motion vectors on the Sky dataset.
Note that the shape of clouds moves differently, but the tree in the left below stays.
The freedom of variations conditioned on the initial frame depends on datasets, as discussed in Appendix D.
Space interpolation and extrapolation.
DIGAN can also inter- and extra-polate videos in the space directions.
Figure 7 and Table 4 shows the qualitative and quantitative results for space interpolation ( i e ., upsampling) on the TaiChi dataset.
DIGAN produces 4 × higher resolution videos without ad-hoc training tricks, outperforming na ¨ ıve heuristics such as bicubic interpolation.
Figure 8 visualizes the space extrapolation ( i.e ., zoom-out) results on various datasets.
DIGAN creates out-of-box scenes while preserving temporal consistency.
See Appendix H for more results.
INR weight interpolation.
Interpolation of INR parameter sampled from DIGAN produces semantically meaningful videos.
Figure 9 visualizes the videos from the linearly interpolated INR parameters on the TaiChi dataset.
The videos smoothly vary over interpolation, e.g ., cloth color changes from white to blue.
It is not obvious as the INR weights lie in a structured, high-dimensional space.
4. A BLATION STUDY
We conduct the ablation studies on the components of DIGAN.
Table 5 shows that all the proposed generator components: smaller frequency σ t , motion vector z M , and non-linear MLP mapping f M ( · ) contribute to the generation performance measured by FVD scores on the UCF-101 dataset.
We note that the motion vector z M and MLP f M ( · ) remarkably affect FVD when applied solely, but the allcombination result is saturated.
On the other hand, Figure 10 veriﬁes that the motion discriminator considers the time difference ∆ t of a given pair of images.
Speciﬁcally, we provide a far image pair ( i 0 , i 1 ) , i.e ., the ﬁrst and the last frames, with the time difference ∆ t .
The motion discriminator thinks the triplet ( i 0 , i 1 , ∆ t ) is fake if ∆ t ≈ 0 and real if ∆ t ≈ 1 , as the real difference is ∆ t = 1 .
Namely, the discriminator leverages ∆ t to identify whether the input triplets are real or fake.
C ONCLUSION
We proposed DIGAN, an implicit neural representation (INR)-based generative adversarial network (GAN) for video generation, incorporating the temporal dynamics of videos.
Extensive experiments veriﬁed the superiority of DIGAN with multiple intriguing properties.
We believe our work would guide various directions in video generation and INR research in the future.
E THICS STATEMENT
Video generation has potential threats of creating videos for unethical purposes, e.g ., fake propaganda videos of politicians or sexual videos of any individuals.
The generated videos, often called
DeepFake, have arisen as an important social problem (Westerlund, 2019).
To tackle the issue, there have been enormous efforts in detecting fake videos ( e.g ., Guera & Delp (2018)).
Here, we claim that the generation and detection techniques should be developed in parallel, rather than prohibiting the generation research itself.
This is because the advance of technology is inevitable, and such prohibition only promotes the technology hide in the dark, making them hard to pick out.
In this respect, the generative adversarial network (GAN) is a neat solution as it naturally trains both generator and discriminator (or detector).
Notably, the discriminator trained by GAN can effectively detect the fake samples created by other generative models (Wang et al., 2020).
Our proposed video generation method, DIGAN, is also built upon the GAN framework.
In particular, we introduced a discriminator that identiﬁes the fake videos without observing long sequences.
We believe our proposed discriminator would be a step towards designing an efﬁcient DeepFake detector.
R EPRODUCIBILITY STATEMENT
We describe the implementation details of the model in Appendix A.1, and details of the datasets and evaluation in Appendix A.2.
We also provide our code in the supplementary material.
A I MPLEMENTATION DETAILS A. M ODEL DETAILS
Following the original conﬁguration of INR-GAN (Skorokhodov et al., 2021), we set the same spatial frequencies σ x = σ y = √ 10 and StyleGAN2 (Karras et al., 2020b) discriminator.
We use a small temporal frequency σ t = 0 .
25 to encourage temporal coherence.
Here, we remark that the number of frames used in most experiments (16) is much smaller than the image resolution (128), which also affects choosing the σ t .
In this respect, choosing a larger σ t (but still smaller than σ x , σ y ) can boost the performance for longer videos, e.g ., we use σ t = 0 .
5 for training on 128 frame videos.
Like in INR-GAN, we also use a progressive multi-layer perception (MLP) with a factorized multiplicative modulation layers as implicit neural representation architecture.
For non-linear mapping f M ( · ) , we use the 2-layer MLP with leaky ReLU activation and, no bias are applied for motion vectors.
We also apply DiffAug (Zhao et al., 2020) to mitigate overﬁtting from the limited number of videos like in Tian et al.
Speciﬁcally, we use all augmentations proposed from DiffAug except CutOut (DeVries & Taylor, 2017), and use the same augmentation to the same video.
All other hyperparameters are identical to StyleGAN2 except for R 1 regularization coefﬁcient γ : we use γ = 1 in all experiments.
With these setups, all the experiments are processed with 4 NVIDIA
V100 32GB GPUs where it takes at most ∼ 4.4 days to complete.
We note that standard generative adversarial networks (GANs; Goodfellow et al.
and implicit
GANs are in a dual relation as the former samples the input latent while the latter samples the network parameters, which are combined to compute the ﬁnal outputs.
However, the generator of implicit GAN resembles the one of StyleGAN2 in practice since StyleGAN2 injects the input latent to the intermediate layers through the mapping network and modulates the weights with them.
A. D ATASET AND EVALUATION DETAILS
Datasets.
In what follows, we describe datasets that we used for the evaluation of our method.
All videos are ﬁrst pre-processed to video clips of consecutive 16 frames, unless otherwise speciﬁed.
• UCF-101 (Soomro et al., 2012) is a 101-class video action dataset total of 13,320 videos with 320 × 240 resolution.
Each video clip is center-cropped to 240 × 240 and resized into 128 ×resolution.
We conducted two different experiments for a fair comparison: training the model with the train split of 9,357 videos or with all 13,320 videos without the split (following the setup of prior state-of-the-art baselines (Clark et al., 2019; Tian et al., 2021)).
• Tai-Chi-HD
(Siarohin et al., 2019) is a video dataset total of 280 long videos of people doing Tai-Chi.
We use the ofﬁcial link for downloading and cropping the dataset as the rectangle videos of 128 × 128 resolution.
4 We use 16 frame video clips of stride 4 ( i.e ., skip 3 frames after the chosen frame) for dynamic motion.
We use all data without the split on training.
We exclude the following videos due to the broken link: 8xSkbMUpegs , RCiy2FYViEg , iFMbu9-Mejc , ceoe2fz648U , 6jHyn4z0KLk , VMSqvTE90hk , xmwGBXYofEE , Dn0mNZmAh2k , VhprHat04dk , KYdyIdusD0g , EaEZVfhn07o , L745tFFmCQ , ytT4iU7h-A8 , 5ujMzSyHO 8 , JdiIQg47Wc4 , aAwbJ9MO91I , and XRyc2kiTlM .
• Sky Time-lapse (Xiong et al., 2018) is a collection of sky time-lapse total of 5,000 videos.
We use the same data pre-processing following the ofﬁcial link.
We use the train split for training the model and test split for the evaluation, following the setups in prior works.
• Kinetics-600 (Carreira et al., 2018) is a large-scale 600-class video action dataset, consists of a total of 495,547 videos.
We sub-sampled a food subclass in the dataset to train the model, where we follow the list of such a subclass from Weissenborn et al.
(2020), namely: (baking , barbequing , breading , cooking , cutting , pancake , vegetables , meat , cake , sandwich , pizza , sushi , tea , peeling , fruit , eggs , and salad .
We use train split for the model training and use the validation set for the evaluation.
Note that we only use these classes, different from Weissenborn et al.
(2020) to train the model with the whole dataset.
Evaluation metrics.
We follow the prior setups for evaluation for a fair comparison.
We use the
C3D network (Tran et al., 2015) pre-trained on Sports-1M (Karpathy et al., 2014) and ﬁne-tuned on UCF-101 datasets for the Inception score (IS; Salimans et al.
We use the ofﬁcial TGAN (Saito et al., 2017) implementation for computing IS: the score is evaluated over 10,000 generated videos.
6 We note this metric does not use the Inception network (Szegedy et al., 2016) have been used in evaluating IS in image generation, but only the formula for evaluation is identical.
For Fr´echet video distance (FVD; Unterthiner et al.
(2018)), and kernel video distance (KVD; Unterthiner et al.
, we use the C3D network trained on Kinetics-400 (Kay et al., 2017).
All evaluations are done by averaging 10 runs of scores computed from 2,048 sampled real and generated videos, following the setup in TGANv2 (Saito et al., 2020).
Forward and backward prediction.
For the forward and backward prediction in Figure 5, one should project the given frame into the latent code.
To this end, we follow StyleGAN2 projection procedure Karras et al.
(2020a)
and optimize for 20,000 iteration.
Generation time.
To fairly compare the generation time with the baselines in Table 3, we utilize the same machine and stop other processes.
We used Intel(R)
Xeon(R) CPU E5-2630 v4 @
2.20GHz and a Titan XP GPU for the measurement.
Space extrapolation.
We generate the video by 2D grid of range [ − 0 .
B B ASELINES
In this section, we explain video generation baselines we used for evaluating DIGAN at a high level.
• VGAN (Vondrick et al., 2016) extends image generative adversarial network (GAN; (Goodfellow et al., 2014)) by replacing 2D spatial architecture with 3D spatio-temporal architecture.
• TGAN
(Saito et al., 2017) separates the spatial (image) generator and temporal (latent dynamics) build on Wasserstein GAN (Arjovsky et al., 2017) model for image generation.
MoCoGAN (Tulyakov et al., 2018) proposes GAN to disentangle videos by the content of single latent and motion of stochastic latent trajectory and generate these two components.
• ProgressiveVGAN (Acharya et al., 2018) progressively generates videos both in spatial and temporal directions, like pregressive image generation in ProgressiveGAN (Karras et al., 2017).
• LDVD-GAN (Kahembwe & Ramamoorthy, 2020) proposes a efﬁcient video discriminator for training video GANs by considering its kernel dimension to be low.
• VideoGPT
(Yan et al., 2021) compresses videos as a sequence of discrete latent vectors via vector-quantized variational auto-encoder (van den Oord et al., 2017), then trains autoregressive Transformer model (Vaswani et al., 2017) with these latent sequences.
• TGANv2 (Saito et al., 2020) designs a new video GAN of computationally efﬁcient generator and discriminator by proposing sub-modules on each component.
• DVD-GAN
(Clark et al., 2019) uses spatial and temporal discriminators, where the input of the temporal discriminator is spatially down-sampled to reduce the computational bottleneck.
• MoCoGAN-HD (Tian et al., 2021) leverages a pre-trained image generator to additionally train a motion generator on the image latent space to synthesize the video with those two generators.
C A DDITIONAL RELATED WORK
Video prediction.
A related but distinct area, video prediction, aims to forecast future video frames from early frames (Srivastava et al., 2015; Finn et al., 2016; Denton & Birodkar, 2017; Babaeizadeh et al., 2018; Denton & Fergus, 2018; Lee et al., 2018; Villegas et al., 2019; Kumar et al., 2020;
Franceschi et al., 2020; Luc et al., 2020; Lee et al., 2021).
By restricting the distribution conditioned on initial frames, video prediction models generally show better visual quality than video generation (Babaeizadeh et al., 2021).
Also, video prediction models employ image-to-image architectures as they only need to modify the initial frames regarding the motion, while video generation models use latent-to-image architectures.
While our primary focus is video generation, one can also utilize our model for video prediction by projecting initial frames into the corresponding latent codes.
Video synthesis with motion and content decomposition.
For better video generation or prediction with controllability, several works have proposed methods to decompose the motion and contents.
Speciﬁcally, they model videos with a content vector and a sequence of motion vectors from different subspaces, incorporating distinct motion and content encoders for a prediction (Villegas et al., 2017;
Hsieh et al., 2018) or generators for a generation (Tulyakov et al., 2018; Tian et al., 2021; Munoz et al., 2021).
Similarly, we utilize two generators, but the motion is represented as a single vector of the weight of implicit neural representations rather than a sequence of latents.
D L ATENT DYNAMICS OF MOTION FEATURES
[Figure]
Figure 11 visualizes the latent dynamics of motion features on various datasets.
Note that the freedom of motion features vary for different datasets: UCF-101 allows high-level (resolution 32) and low-level (resolution 128) variations, Sky only allows low-level variations, and TaiChi does not permit the variations much.
Intuitively, the next move of TaiChi is mostly determined by the prior frames: the movement follows the pre-deﬁned gestures.
Sky permits some variations as shown in
Figure 6, but the wind direction determines the global motion.
UCF-101 has the largest freedom of motion as the dataset contains videos of various actions and diverse objects.
E I NTERMEDIATE SCENE PREDICTION
R e a l
G e n e r a t e d
R e a l
G e n e r a t e d
R e a l
G e n e r a t e d
We show that our proposed method, DIGAN, can predict the intermediate frames between the two frames of the different time steps.
To this end, we ﬁnd the inverse content and motion latent codes by projecting two given frames into the generator (we follow the StyleGAN2 projection procedure
Karras et al. (2020b) and optimize 20,000 iteration for the projection).
As shown in Figure 12,
DIGAN can well predict the intermediate dynamics even the given frames are somewhat far apart.
This result implies that DIGAN indeed has learned the dynamics of the ground truth distribution.
F M ORE EXAMPLES FOR LONG VIDEO GENERATION
[Figure]
G M ORE EXAMPLES FOR TIME EXTRAPOLATION
[Figure]
H M ORE EXAMPLES FOR SPACE EXTRAPOLATION
[Figure] [Figure]
I T RAINING EFFICIENCY OF DIGAN
In addition to the computational efﬁciency on inference (in Table 3), we provide the analysis of our method on efﬁciencies of diverse aspects, including computation, energy, memory, and time efﬁciency.
Here, we mainly compare our method with MoCoGAN-HD (Tian et al., 2021), which is an energy-efﬁcient ( i.e ., shorter GPU days for training) and state-of-the-art method on the generation quality.
All experiments are performed under the same machine (NVIDIA V100 32GB GPUs).
• Computational efﬁciency.
We report ﬂoating point operations per second (FLOPS) 7 of video GAN generators (following Liu et al.
The DIGAN generator requires 147.9 GFLOPS to generate a 16 frame video of 128 × 128 resolution, 4.6 times smaller than 682.3 GFLOPS of the MoCoGAN-HD generator.
• Energy efﬁciency on training (GPU days for training).
The FVD value of DIGAN trained on UCF-101 for 8 GPU days achieves 689 ± 24, which is better than 838 of MoCoGAN-HD trained for 16 GPU days.
This implies that DIGAN is (at least 2 times) more energy-efﬁcient than MoCoGAN-HD.
• Memory efﬁciency on training.
We report the memory size per GPU to load 16 frame videos of 128 × 128 resolution of batch size 4 for training video GANs.
DIGAN allocates 9.7GB memory, 2.9 times smaller than 28.0GB of MoCoGAN-HD.
• Time efﬁciency on inference.
Per each GPU, DIGAN discriminators can (forward) compute 195.6 video clips/sec, 3.2 times larger than 60.3 video clips/sec of MoCoGAN-HD discriminators with 128 frame videos of 128 × 128 resolution and a batch size of 16.
It conﬁrms that both the generator and discriminator of our method are much more time-efﬁcient; recall that we already demonstrated that our generator
is ∼ 2.3 times faster per each GPU in Table 3.
In summary, DIGAN is more computation-efﬁcient ( > 4.6 times), energy-efﬁcient on training ( >times), memory-efﬁcient on training ( > 2.9 times), and time-efﬁcient on inference ( > 3.2 times) under each machine setup above.
Such gaps stem from the fact that our method utilizes: (a) INR-based generator without time-consuming autoregressive modeling and (b) 2D convolutional discriminator rather than computationally heavy 3D convolutional discriminator.
Finally, we emphasize that DIGAN can synthesize multiple frames of a video in parallel (which is impossible under prior autoregressive or convolutional network-based methods), resulting in N times improved time-efﬁciency to generate a single video under N number of GPUs.
It can be particularly important for synthesizing extremely long videos; since it requires tremendous time if utilizing prior methods that are impossible to compute frames in parallel.
Like this, the superiority of computational efﬁciency of DIGAN can be further dramatically improved in the presence of multiple GPUs.