Original: T OWARDS S MOOTH V IDEO C OMPOSITION
Revised: T OWARDS S MOOTH V IDEO C OMPOSITION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Video generation, with the purpose of producing a sequence of frames, requires synthesizing consistent and persistent dynamic contents over time.
Revised: Video generation, with the purpose of producing a sequence of frames, requires synthesizing consistent and persistent dynamic contents over time.

Original: This work investigates how to model the temporal relations for composing a video with arbitrary number of frames, from a few to even infinite, using generative adversarial networks (GANs).
Revised: This work investigates how to model the temporal relations for composing a video with arbitrary number of frames, from a few to even infinite, using generative adversarial networks (GANs).

Original: First, towards composing adjacent frames, we show that the alias-free operation for single image generation, together with adequately pre-learned knowledge, bring a smooth frame transition without harming the perframe quality.
Revised: First, towards composing adjacent frames, we show that the alias-free operation for single image generation, together with adequately pre-learned knowledge, bring a smooth frame transition without harming the perframe quality.

Original: Second, through incorporating a temporal shift module (TSM), which is originally designed for video understanding, into the discriminator, we manage to advance the generator in synthesizing more reasonable dynamics.
Revised: Second, through incorporating a temporal shift module (TSM), which is originally designed for video understanding, into the discriminator, we manage to advance the generator in synthesizing more reasonable dynamics.

Original: Third, we develop a novel B-Spline based motion representation to ensure the temporal smoothness, and hence achieve infinite-length video generation, going beyond the frame number used in training.
Revised: Third, we develop a novel B-Spline based motion representation to ensure the temporal smoothness, and hence achieve infinite-length video generation, going beyond the frame number used in training.

Original: We evaluate our approach on a range of datasets and show substantial improvements over baselines on video generation.
Revised: We evaluate our approach on a range of datasets and show substantial improvements over baselines on video generation.

Original: Code and models will be made publicly available.
Revised: Code and models are publicly available at https://genforce.github. io/StyleSV .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Synthesizing images using a generative adversarial network (GAN) (Goodfellow et al., 2014;
Revised: Synthesizing images using a generative adversarial network (GAN) (Goodfellow et al., 2014;

Original: Radford et al., 2016; Karras et al., 2019; 2020b; 2021; 2018) usually requires to compose diverse visual concepts with fine details of a single object and plausible spatial arrangement of different objects.
Revised: Radford et al., 2016; Karras et al., 2019; 2020b; 2021; 2018) usually requires to compose diverse visual concepts with fine details of a single object and plausible spatial arrangement of different objects.

Original: Recent advances in GANs have enabled many appealing applications such as customized editing (Goetschalckx et al., 2019;
Revised: Recent advances in GANs have enabled many appealing applications such as customized editing (Goetschalckx et al., 2019;

Original: Shen et al., 2020; Jahanian et al., 2020;
Revised: Shen et al., 2020; Jahanian et al., 2020;

Original: Yang et al., 2021) and animation (Qiu et al., 2022; Alaluf et al., 2022).
Revised: Yang et al., 2021) and animation (Qiu et al., 2022; Alaluf et al., 2022).

Original: However, employing GANs for video generation remains challenging considering the additional requirement on the temporal dimension.
Revised: However, employing GANs for video generation remains challenging considering the additional requirement on the temporal dimension.

Original: In fact, a video is not simply a stack of images.
Revised: In fact, a video is not simply a stack of images.

Original: Instead, the contents in video frames should have a smooth transition over time, and the video may last arbitrarily long.
Revised: Instead, the contents in video frames should have a smooth transition over time, and the video may last arbitrarily long.

Original: Thus, compared to image synthesis, the crux of video synthesis lies in modeling the temporal relations across frames.
Revised: Thus, compared to image synthesis, the crux of video synthesis lies in modeling the temporal relations across frames.

Original: We argue that the temporal relations fall into three folds regarding the time scale.
Revised: We argue that the temporal relations fall into three folds regarding the time scale.

Original: First, when looking at a transient dynamic, we would focus more on the subtle change between neighbor frames and expect decent local motions, such as facial muscle movement and cloud drifting.
Revised: First, when looking at a transient dynamic, we would focus more on the subtle change between neighbor frames and expect decent local motions, such as facial muscle movement and cloud drifting.

Original: Along with the duration getting longer, say a segment, more contents within the frame may vary.
Revised: Along with the duration getting longer, say a segment, more contents within the frame may vary.

Original: Under such a case, learning a consistent global motion is vital.
Revised: Under such a case, learning a consistent global motion is vital.

Original: For example, in a video of first-view driving, trees and buildings along the street should move backward together with the car running forward.
Revised: For example, in a video of first-view driving, trees and buildings along the street should move backward together with the car running forward.

Original: Finally, for those extremely long videos, the objects inside are not immutable.
Revised: Finally, for those extremely long videos, the objects inside are not immutable.

Original: It therefore requires the motion to be generalizable along the time axis in a continuous and rational sense.
Revised: It therefore requires the motion to be generalizable along the time axis in a continuous and rational sense.

Original: This work targets smooth video composition through modeling multi-scale temporal relations with
Revised: This work targets smooth video composition through modeling multi-scale temporal relations with

Original: GANs.
Revised: GANs.

Original: First, we confirm that, same as in image synthesis, the texture sticking problem ( i.e. , some visual concepts are bound to their coordinates) also exists in video generation, interrupting the smooth flow of frame contents.
Revised: First, we confirm that, same as in image synthesis, the texture sticking problem ( i.e. , some visual concepts are bound to their coordinates) also exists in video generation, interrupting the smooth flow of frame contents.

Original: To tackle this obstacle, we borrow the alias-free technique (Karras et al., 2021) from single image generation and preserve the frame quality via appropriate pretraining.
Revised: To tackle this obstacle, we borrow the alias-free technique (Karras et al., 2021) from single image generation and preserve the frame quality via appropriate pretraining.

Original: Then, to assist the generator in producing reasonable dynamics, we introduce a temporal shift module (TSM)
Revised: Then, to assist the generator in producing reasonable dynamics, we introduce a temporal shift module (TSM)

Original: (Lin et al., 2019) into the discriminator as an inductive bias.
Revised: (Lin et al., 2019) into the discriminator as an inductive bias.

Original: That way, the discriminator could capture more information from the temporal perspective for real/fake classification, serving as a better guidance to the generator.
Revised: That way, the discriminator could capture more information from the temporal perspective for real/fake classification, serving as a better guidance to the generator.

Original: Furthermore, we observe that the motion representation in previous work (Skorokhodov et al., 2022) suffers from undesired content jittering (see Sec.
Revised: Furthermore, we observe that the motion representation in previous work (Skorokhodov et al., 2022) suffers from undesired content jittering (see Sec.

Original: 2.4 for details) for super-long video generation.
Revised: 2.4 for details) for super-long video generation.

Original: We identify the cause of such a phenomenon as the first-order discontinuity when interpolating motion embeddings.
Revised: We identify the cause of such a phenomenon as the first-order discontinuity when interpolating motion embeddings.

Original: Towards this problem, we design a novel B-Spline based motion representation that could soundly and continuously generalize over time.
Revised: Towards this problem, we design a novel B-Spline based motion representation that could soundly and continuously generalize over time.

Original: A low-rank strategy is further proposed to alleviate the issue that frame contents may repeat cyclically.
Revised: A low-rank strategy is further proposed to alleviate the issue that frame contents may repeat cyclically.

Original: We evaluate our approach on various video generation benchmarks, includingself-collected YouTube driving dataset, SkyTimelapse (Xiong et al., 2018), Taichi-HD (Siarohin et al., 2019b) and observe consistent and substantial improvements over existing alternatives.
Revised: We evaluate our approach on various video generation benchmarks, including YouTube driving dataset (Zhang et al., 2022), SkyTimelapse (Xiong et al., 2018), Taichi-HD (Siarohin et al., 2019b) and observe consistent and substantial improvements over existing alternatives.

Original: Given its simplicity and efficiency, our approach sets up a simple yet strong baseline for the task of video generation.
Revised: Given its simplicity and efficiency, our approach sets up a simple yet strong baseline for the task of video generation.

Original:  M ETHOD
Revised:  M ETHOD

Original: We introduce the improvements made on the prior art StyleGAN-V (Skorokhodov et al., 2022) to set a new baseline for video generation.
Revised: We introduce the improvements made on the prior art StyleGAN-V (Skorokhodov et al., 2022) to set a new baseline for video generation.

Original: We first introduce the default configuration ( Config-A ) in
Revised: We first introduce the default configuration ( Config-A ) in

Original: Sec.
Revised: Sec.

Original: We then make a comprehensive overhaul on it.
Revised: We then make a comprehensive overhaul on it.

Original: Concretely, in Sec.
Revised: Concretely, in Sec.

Original: 2.2 we confirm that the alias-free technique ( Config-B ) in single image generation, together with adequately pre-learned knowledge ( Config-C ), could result in a smooth transition of two adjacent frames.
Revised: 2.2 we confirm that the alias-free technique ( Config-B ) in single image generation, together with adequately pre-learned knowledge ( Config-C ), could result in a smooth transition of two adjacent frames.

Original: Sec.
Revised: Sec.

Original: 2.3 shows that when the temporal information is explicitly modeled into the discriminator through the temporal shift module (Lin et al., 2019) ( Config-D ), the generator can produce significantly better dynamic content across frames.
Revised: 2.3 shows that when the temporal information is explicitly modeled into the discriminator through the temporal shift module (Lin et al., 2019) ( Config-D ), the generator can produce significantly better dynamic content across frames.

Original: Although prior arts could already generate arbitrarily long videos, the cyclic jittering is observed as time goes by.
Revised: Although prior arts could already generate arbitrarily long videos, the cyclic jittering is observed as time goes by.

Original: We therefore propose a B-Spline based motion representation ( Config-E ) to ensure the continuity, which together with a low-rank temporal modulation ( ConfigF ) could produce much more realistic and natural long videos in Sec. 2.4.
Revised: We therefore propose a B-Spline based motion representation ( Config-E ) to ensure the continuity, which together with a low-rank temporal modulation ( ConfigF ) could produce much more realistic and natural long videos in Sec. 2.4.

Original: 2. P RELIMINARY
Revised: 2. P RELIMINARY

Original: StyleGAN-V (Skorokhodov et al., 2022) introduces continuous motion representations and a holistic discriminator for video generation.
Revised: StyleGAN-V (Skorokhodov et al., 2022) introduces continuous motion representations and a holistic discriminator for video generation.

Original: Specifically, continuous frames I t could be obtained by feeding continuous t into a generator G ( · ) :
Revised: Specifically, continuous frames I t could be obtained by feeding continuous t into a generator G ( · ) :

Original: [Equation]
Revised: [Equation]

Original: where u and v t denote the content code and continuous motion representation.
Revised: where u and v t denote the content code and continuous motion representation.

Original: Concretely, content code is sampled from a standard  distribution while the motion representation v t consists of two embeddings: time positional embedding v pet and interpolated motion embedding v met .
Revised: Concretely, content code is sampled from a standard gaussian distribution while the motion representation v t consists of two embeddings: time positional embedding v pet and interpolated motion embedding v met .

Original: To obtain the time positional embedding v pe t , we first randomly sample N A codes as a set of discrete time anchors
Revised: To obtain the time positional embedding v pet , we first randomly sample N A codes as a set of discrete time anchors

Original: A i , i ∈ [0 , · · · , N A − 1] that share equal interval (256 frames in practice).
Revised: A i , i ∈ [0 , · · · , N A − 1] that share equal interval (256 frames in practice).

Original: Convolutional operation with 1D kernel is then applied on anchor A i for temporal modeling, producing the corresponding features a i with timestamp t i .
Revised: Convolutional operation with 1D kernel is then applied on anchor A i for temporal modeling, producing the corresponding features a i with timestamp t i .

Original: For an arbitrary continuous t , its corresponding interval is first found with the nearest left and right anchor feature a l and a l +1 , at time t l and t l +1 , so that t l ≤ t < t l +1 , l ∈ [0 , · · · , N A − 2] .
Revised: For an arbitrary continuous t , its corresponding interval is first found with the nearest left and right anchor feature a l and a l +1 , at time t l and t l +1 , so that t l ≤ t < t l +1 , l ∈ [0 , · · · , N A − 2] .

Original: For time positional embedding v pet ,a wave function is derived with learned frequency β , phase γ and amplitude α from the left anchor
Revised: For time positional embedding v pet ,

Original: Config-E and Config-F alleviate the  when 
Revised: Config-E and Config-F alleviate the discontinuity when motion

Original: For 
Revised: For all

Revised: , a is

Original: [Table]
Revised: [Table]

Original:  feature a l :
Revised: a wave function is derived with learned frequency β , phase γ and amplitude α from the left anchor feature a l :

Original: [Equation]
Revised: [Equation]

Original: where M α , M β , M γ are learnable MLPs.
Revised: where M α , M β , M γ are learnable MLPs.

Original: The interpolated motion embedding v met could be obtained through linear interpolation between anchor feature a l and a l +1 based on t :
Revised: The interpolated motion embedding v met could be obtained through linear interpolation between anchor feature a l and a l +1 based on t :

Original: [Equation]
Revised: [Equation]

Original: The final motion representation v t isthus the sum of these two terms:
Revised: The final motion representation v t is the sum of these two terms:

Original: [Equation]
Revised: [Equation]

Original: Additionally, a sparse sampling strategy is proposed for efficient training.
Revised: Additionally, a sparse sampling strategy is proposed for efficient training.

Original: Namely, N t discrete time steps t i and real video frames I i are sampled respectively.
Revised: Namely, N t discrete time steps t i and real video frames I i are sampled respectively.

Original: By feeding N t discrete time steps and one content code into generator, multiple synthesis I ′ i
Revised: By feeding N t discrete time steps and one content code into generator, multiple synthesis I ′ i

Original: =
Revised: =

Original: G ( u, v t i ) are obtained.
Revised: G ( u, v t i ) are obtained.

Original: In order to distinguish real video frames from synthesized ones, a discriminator D ( · ) extracts feature y i for the individual frame, and then a fusion function Θ would perform the temporal aggregation over all frames:
Revised: In order to distinguish real video frames from synthesized ones, a discriminator D ( · ) extracts feature y i for the individual frame, and then a fusion function Θ would perform the temporal aggregation over all frames:

Original: [Equation]
Revised: [Equation]

Original: The final discriminative logit l is computed by a MLP M conditioned on time steps:
Revised: The final discriminative logit l is computed by a MLP M conditioned on time steps:

Original: [Equation]
Revised: [Equation]

Original: In the following context, we would make a comprehensive overhaul on it, leading to a new simple yet strong approach for video generation.
Revised: In the following context, we would make a comprehensive overhaul on it, leading to a new simple yet strong approach for video generation.

Original: 2. A LIAS - FREE OPERATIONS AND PRE - LEARNED KNOWLEDGE
Revised: 2. A LIAS - FREE OPERATIONS AND PRE - LEARNED KNOWLEDGE

Original: When composing a few frames, they usually share similar visual concepts and micro motion gradually occurs.
Revised: When composing a few frames, they usually share similar visual concepts and micro motion gradually occurs.

Original: Therefore, the smooth transition between frames tends to make the synthesis much more realistic.
Revised: Therefore, the smooth transition between frames tends to make the synthesis much more realistic.

Original: However, by examining the synthesized frames of StyleGAN-V, we find that texture sticking (Karras et al., 2021) exists.
Revised: However, by examining the synthesized frames of StyleGAN-V, we find that texture sticking (Karras et al., 2021) exists.

Original: To be specific, we track the pixels at certain coordinates as video moves on and the brush effect in Fig.
Revised: To be specific, we track the pixels at certain coordinates as video moves on and the brush effect in Fig.

Original: 2a indicates that these pixels actually move little.
Revised: 2a indicates that these pixels actually move little.

Original: Thus, the texture sticks to fixed coordinates.
Revised: Thus, the texture sticks to fixed coordinates

Original: Alias-free operations (Config-B)
Revised: Alias-free operations (Config-B).

Original: To overcome the texture sticking issue, we followthe StyleGAN3 (Karras et al., 2021) to leverage the alias-free technique in the generator, as one could remove the unhealthy dependency on absolute pixel coordinates through decent signal processing.
Revised: To overcome the texture sticking issue, we follow StyleGAN3 (Karras et al., 2021) to leverage the alias-free technique in the generator, as one could remove the unhealthy dependency on absolute pixel coordinates through decent signal processing.

Original: As is shown in Fig.
Revised: As is shown in Fig.

Original: 2b, brush effect disappear i.e., texture sticking is significantly alleviated, with the aid of alias-free technique.
Revised: 2b, brush effect disappear, i.e., texture sticking is significantly alleviated, with the aid of alias-free technique.

Revised: [Figure]

Original: However, as reported in Tab. 1, using StyleGAN3 achieves lower performance in FID and FVD than before.
Revised: However, as reported in Tab. 1, using StyleGAN3 achieves lower performance in FID and FVD than before.

Original: One possible reason is that the training receipt is directly inherited from the original
Revised: One possible reason is that the training receipt is directly inherited from the original

Original: [Figure]

Original: StyleGAN-V without any modification.
Revised: StyleGAN-V without any modification.

Original: Nevertheless, given that it may impair FID and FVD to some extent, we still take StyleGAN3 in for sake of its alias-free benefit.
Revised: Nevertheless, given that it may impair FID and FVD to some extent, we still take StyleGAN3 in for sake of its alias-free benefit.

Original: Pre-learned knowledge (Config-C).
Revised: Pre-learned knowledge (Config-C).

Original: As simply replacing the generator backbone with StyleGAN can deteriorate the synthesis, we propose to make the best of pre-learned knowledge to catch up with the performance.
Revised: As simply replacing the generator backbone with StyleGAN can deteriorate the synthesis, we propose to make the best of pre-learned knowledge to catch up with the performance.

Original: As specified above, the key challenge for video generation is to model temporal relations between frames.
Revised: As specified above, the key challenge for video generation is to model temporal relations between frames.

Original: However, this target is entangled with the urge to model single image distribution.
Revised: However, this target is entangled with the urge to model single image distribution.

Original: We confirm that pre-training at image level and then fine-tuning in video level can decouple these two targets well.
Revised: We confirm that pre-training at image level and then fine-tuning in video level can decouple these two targets well.

Original: As discovered in (Karras et al., 2019; Yang et al., 2021),
Revised: As discovered in (Karras et al., 2019; Yang et al., 2021),

Original: GAN’s deep layers mainly control details, for example coloring, detailed refinements, and etc.
Revised: GAN’s deep layers mainly control details, for example coloring, detailed refinements, and etc.

Original: We hypothesize that these parts’ knowledge can be pre-learned and re-used in the video generation learning process.
Revised: We hypothesize that these parts’ knowledge can be pre-learned and re-used in the video generation learning process.

Original: To be specific, we set N t as 1 in Eq.
Revised: To be specific, we set N t as 1 in Eq.

Original: (5), and train the whole network from scratch as the image pre-training stage.
Revised: (5), and train the whole network from scratch as the image pre-training stage.

Original: Then deep layers’ weights are loaded in the video fine-tuning stage.
Revised: Then deep layers’ weights are loaded in the video fine-tuning stage.

Original: With pretrained deep layers’ weights, now the optimization process can focus on early layers to learn to model sequential changes in structures which show natural movements.
Revised: With pretrained deep layers’ weights, now the optimization process can focus on early layers to learn to model sequential changes in structures which show natural movements.

Original: As shown in Tab. 1, after adopting Config-C , the generated quality gets improved compared to Config-B .
Revised: As shown in Tab. 1, after adopting Config-C , the generated quality gets improved compared to Config-B .

Original: Now, with the aliasfree technique, we achieve smooth frame transition without sacrificing much per-frame quality.
Revised: Now, with the aliasfree technique, we achieve smooth frame transition without sacrificing much per-frame quality.

Original: 2. E XPLICIT T EMPORAL R EASONING IN D ISCRIMINATOR
Revised: 2. E XPLICIT T EMPORAL R EASONING IN D ISCRIMINATOR

Original: In an adversarial context, it is essential to have a strong discriminator, assuring the sufficient training of the generator.
Revised: In an adversarial context, it is essential to have a strong discriminator, assuring the sufficient training of the generator.

Original: In video generation task, the discriminator thus has to model the temporal relations of multiple frames to distinguish unnatural movements from real ones.
Revised: In video generation task, the discriminator thus has to model the temporal relations of multiple frames to distinguish unnatural movements from real ones.

Original: However, a simple concatenation operation ⊕ is adopted before:
Revised: However, a simple concatenation operation ⊕ is adopted before:

Original: y =
Revised: y =

Original: ⊕ i
Revised: ⊕ i

Original: y
Revised: y

Original: i , where y i denotes a single frame’s feature extracted by the discriminator and y denotes the feature after temporal fusion.
Revised: i , where y i denotes a single frame’s feature extracted by the discriminator and y denotes the feature after temporal fusion.

Original: Temporal modeling (Config-D).
Revised: Temporal modeling (Config-D).

Original: We therefore introduce an explicit temporal modeling approach i.e., temporal shift module (TSM)  that shows its superiority in standard video understanding area and, most importantly, introduces zero overhead.
Revised: We therefore introduce an explicit temporal modeling approach, i.e., temporal shift module (TSM) (Lin et al., 2019) that shows its superiority in standard video understanding area and, most importantly, introduces zero overhead.

Original: Concretely, at each convolution layer before temporal fusion, we have features from multiple frames: y ni ∈ R H × W × C , i = 0 , · · · , N t − 1 , where n is the layer index, H and W are the feature map resolution and C is the feature channel number.
Revised: Concretely, at each convolution layer before temporal fusion, we have features from multiple frames: y ni ∈ R H × W × C , i = 0 , · · · , N t − 1 , where n is the layer index, H and W are the feature map resolution and C is the feature channel number.

Original: Then, TSM performs channel-wise swapping between adjacent frames:
Revised: Then, TSM performs channel-wise swapping between adjacent frames:

Original: [Equation]
Revised: [Equation]

Original: where ⊕ 3 ( · ) is a concatenation operation, and ( x, y ) are arbitrary coordinates.
Revised: where ⊕ 3 ( · ) is a concatenation operation, and ( x, y ) are arbitrary coordinates.

Original: In this way, one quarter of the channels of a single frame after temporal fusion contain information from  adjacent frames.
Revised: In this way, one quarter of the channels of a single frame after temporal fusion contain information from [Figure] adjacent frames.

Original: Follow-up convolution kernels in deeper layers can perform efficient temporal reasoning based on this mixed representation.
Revised: Follow-up convolution kernels in deeper layers can perform efficient temporal reasoning based on this mixed representation.

Original: In our ablation study, while FID gets worse slightly[Figure] compared to Config-C , adopting explicit temporal modeling ( Config-D ) improves FVD 16 and
Revised: In our ablation study, while FID gets worse slightly compared to Config-C , adopting explicit temporal modeling ( Config-D ) improves FVD 16 and

Original: FVD 128 significantly, yielding over 100% improvement on YouTube Driving dataset.
Revised: FVD 128 significantly, yielding over 100% improvement on YouTube Driving dataset.

Original: 2. T OWARDS U NLIMITED V IDEO G ENERATION
Revised: 2. T OWARDS U NLIMITED V IDEO G ENERATION

Original: Considering that the main metrics i.e., FID, FVD 16 and FVD 128 used before merely measuring the synthesis of relatively short videos ( e.g., 128 frames usually cover around 6 seconds), we further investigate whether current configuration is able to produce arbitrarily long videos.
Revised: Considering that the main metrics i.e., FID, FVD 16 and FVD 128 used before merely measuring the synthesis of relatively short videos ( e.g., 128 frames usually cover around 6 seconds), we further investigate whether current configuration is able to produce arbitrarily long videos.

Original: Note that, by saying arbitrarily long, we mention infinite length along time axis, but not infinite content.
Revised: Note that, by saying arbitrarily long, we mention infinite length along time axis, but not infinite content.

Original: Ideally, we could easily generate infinite frames by continuously sampling time t .
Revised: Ideally, we could easily generate infinite frames by continuously sampling time t .

Original: The generator is supposed to produce the corresponding synthesis.
Revised: The generator is supposed to produce the corresponding synthesis.

Original: However, as rollouting the video, a conspicuous jittering phenomenon occurs periodically every 10 seconds.
Revised: However, as rollouting the video, a conspicuous jittering phenomenon occurs periodically every 10 seconds.

Original: Concretely, as demonstrated in Fig.
Revised: Concretely, as demonstrated in Fig.

Original: 3, the crossroad moves forward at the beginning, and then suddenly goes backwards.
Revised: 3, the crossroad moves forward at the beginning, and then suddenly goes backwards.

Original: Discontinuity of motion embeddings.
Revised: Discontinuity of motion embeddings.

Original: As mentioned in Sec. 2.1, motion embedding v t contains time positional embedding v pet and interpolated motion embedding v met .
Revised: As mentioned in Sec. 2.1, motion embedding v t contains time positional embedding v pet and interpolated motion embedding v met .

Original: In particular, v pet could be obtained by a learned wave function on the left nearest anchor feature a l while the motion embedding v met derives from the linear interpolation between the left and right anchor feature a l and a l +1 .
Revised: In particular, v pet could be obtained by a learned wave function on the left nearest anchor feature a l while the motion embedding v met derives from the linear interpolation between the left and right anchor feature a l and a l +1 .

Original: This interpolation would result in the first-order discontinuity for both learned wave function and linear interpolation when getting through multiple anchors, as shown in Fig. 4a.
Revised: This interpolation would result in the first-order discontinuity for both learned wave function and linear interpolation when getting through multiple anchors, as shown in Fig. 4a.

Original: Moreover, T-SNE (Van der
Revised: Moreover, T-SNE (Van der

Original: Maaten & Hinton, 2008) is applied to investigate the motion embedding given a long period.
Revised: Maaten & Hinton, 2008) is applied to investigate the motion embedding given a long period.

Original: Fig.
Revised: Fig.

Original: 5a confirms that such discontinuity would cause the drastic change i.e., jittering content.
Revised: 5a confirms that such discontinuity would cause the drastic change, i.e., jittering content.

Original: B-Spline based motion representations (Config-E).
Revised: B-Spline based motion representations (Config-E).

Original: With an assumption of the discontinuity of motion embeddings, we design a basic spline (B-Spline) based motion representations that could guarantee first-order numerical smoothness.
Revised: With an assumption of the discontinuity of motion embeddings, we design a basic spline (B-Spline) based motion representations that could guarantee first-order numerical smoothness.

Original: To be specific, B-Spline of order n is a piece-wise polynomial function of degree n − 1 in a given variable.
Revised: To be specific, B-Spline of order n is a piece-wise polynomial function of degree n − 1 in a given variable.

Original: It is widely used in computer-aided design and computer graphics for curve fitting, controlling a smooth curve with calculated weights w i on several control points.
Revised: It is widely used in computer-aided design and computer graphics for curve fitting, controlling a smooth curve with calculated weights w i on several control points.

Original: In particular, the number of control points is determined by the order of the
Revised: In particular, the number of control points is determined by the order of the

Original: B-Spline.
Revised: B-Spline.

Original: With a pre-defined knot sequence { t i } , a B-Spline with any order can be defined by means of the Cox–de Boor recursion formula (de Boor, 1971).
Revised: With a pre-defined knot sequence { t i } , a B-Spline with any order can be defined by means of the Cox–de Boor recursion formula (de Boor, 1971).

Original: The first-order B-Spline is then defined by:
Revised: The first-order B-Spline is then defined by:

Original: [Equation]
Revised: [Equation]

Revised: if

Revised: [Equation]

Revised: ≤

Revised: [Equation]

Revised: <

Original:  Here i is the interval index.
Revised: [Equation] [Equation] otherwise. Here i is the interval index.

Original: B-spline of higher orders is further defined recursively: 
Revised: B-spline of higher orders is further defined recursively: B

Original: [Equation]
Revised: [Equation]

Original: ) where kis the order and
Revised: k

Original: [Equation]
Revised: [Equation]

Original: We treat each anchor feature a i as the control point and use Bi,k ( t ) as its corresponding weight given time t . Hence, B-Spline based anchor feature is defined as:
Revised: [Equation] [Equation] B

Original: [Equation]
Revised: [Equation]

Revised: +1 ,k ( t ) , (9) where k is the order and ω

Revised: i,k ( t ) :

Revised: = (cid:40) t − t i t i + k − t i t i + k ̸

Revised: = t i , otherwise.

Original: [Figure]
Revised: [Figure]

Original: 𝑤
Revised: 𝑤

Original: [Figure]
Revised: [Figure]

Original: 𝑤
Revised: 𝑤𝑤

Original: [Figure]
Revised: [Figure]

Revised: We treat each anchor feature a i as the control point and use B i,k ( t ) as its corresponding weight given time t .

Revised: Hence, B-Spline based anchor feature is defined as:

Revised: [Equation]

Original: We further use ˆ a ( t ) instead of discretized a i to calculate time positional embedding v pe t and interpolated motion embedding v met as defined in Eq.
Revised: We further use ˆ a ( t ) instead of discretized a i to calculate time positional embedding v pet and interpolated motion embedding v met as defined in Eq.

Original: (2) and Eq.
Revised: (2) and Eq.

Original: Fig.
Revised: Fig.

Original: 4b suggests the continuity property of B-Spline.
Revised: 4b suggests the continuity property of B-Spline.

Original: Meanwhile, through T-SNE again, the B-Spline based motion representations become much smoother in Fig.
Revised: Meanwhile, through T-SNE again, the B-Spline based motion representations become much smoother in Fig.

Original: 5b.
Revised: 5b.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, the crossroad gradually approaches the observer without any sudden jittering.
Revised: 3, the crossroad gradually approaches the observer without any sudden jittering.

Original: Low-rank temporal modulation (Config-F).
Revised: Low-rank temporal modulation (Config-F).

Original: After adopting B-Spline based motion embedding, the jittering phenomenon gets erased.
Revised: After adopting B-Spline based motion embedding, the jittering phenomenon gets erased.

Original: However, we find that similar content periodically appears as time goes by (see Fig.
Revised: However, we find that similar content periodically appears as time goes by (see Fig.

Original: A6a).
Revised: A6a).

Original: This phenomenon implies that motion embedding represents visual concepts in an ill manner.
Revised: This phenomenon implies that motion embedding represents visual concepts in an ill manner.

Original: We thus hypothesize that the new motion embedding might be endowed with a stronger capacity to represent various visual concepts.
Revised: We thus hypothesize that the new motion embedding might be endowed with a stronger capacity to represent various visual concepts.

Original: Inspired by recent works (Bau et al., 2020; Wang et al., 2022a), we suppress the representation capacity through the low-rank strategy.
Revised: Inspired by recent works (Bau et al., 2020; Wang et al., 2022a), we suppress the representation capacity through the low-rank strategy.

Original: Original StyleGAN-based architecture incorporates styles into convolutional kernels via modulation trick (Karras et al., 2020a).
Revised: Original StyleGAN-based architecture incorporates styles into convolutional kernels via modulation trick (Karras et al., 2020a).

Original: When generating videos, a similar technique is also applied with small modifications.
Revised: When generating videos, a similar technique is also applied with small modifications.

Original: Content embedding u together with the motion embedding v t would be fed into an affine layer M , generating the style code to modulate the original weight W :
Revised: Content embedding u together with the motion embedding v t would be fed into an affine layer M , generating the style code to modulate the original weight W :

Original: [Equation]
Revised: [Equation]

Original: (12) where ⊕ stands for the concatenation operation.
Revised: where ⊕ stands for the concatenation operation.

Original: That is, motion and content embedding could equally contribute to the final style to change the visual concepts of frames.
Revised: That is, motion and content embedding could equally contribute to the final style to change the visual concepts of frames.

Original: In order to suppress the representational capacity of motion embeddings, we first separate the motion and content codes and then replace the kernel regarding the motion code with a low-rank one: 
Revised: In order to suppress the representational capacity of motion embeddings, we first separate the motion and content codes and then replace the kernel regarding the motion code with a low-rank one: [Equation] , where

Original: [Equation]
Revised: [Equation]

Original: where U, V is tall matrix that guarantees W mo is a low-rank matrix.
Revised: where U, V is tall matrix that guarantees W mo is a low-rank matrix.

Original: With such a low-rank strategy, the capacity of motion embedding is suppressed.
Revised: With such a low-rank strategy, the capacity of motion embedding is suppressed.

Original: The repeating issue is alleviated in Fig.
Revised: The repeating issue is alleviated in Fig.

Original: A6b.
Revised: A6b.

Original: In conclusion, to generate a short video (less than 10 seconds), Config-D is the best choice.
Revised: In conclusion, to generate a short video (less than 10 seconds), Config-D is the best choice.

Original: In contrast, one can use Config-F for long video generation.
Revised: In contrast, one can use Config-F for long video generation.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our method on multiple datasets and compare it against prior arts.
Revised: We evaluate our method on multiple datasets and compare it against prior arts.

Original: Additionally, some good properties are also given to support the effectiveness of our approach.
Revised: Additionally, some good properties are also given to support the effectiveness of our approach.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: Akin to previous literature, Frechet Inception Distance (FID) (Heusel et al., 2017) and Frechet Video Distance (FVD) (Unterthiner et al., 2018) serve as the quantitative metrics  to evaluate the synthesis for image and video respectively.
Revised: Akin to previous literature, Frechet Inception Distance (FID) (Heusel et al., 2017) and Frechet Video Distance (FVD) (Unterthiner et al., 2018) serve as the quantitative metrics [Figure] [Table] to evaluate the synthesis for image and video respectively.

Original: In terms of FVD, two temporal spans (covering consecutive 16 and 128 frames) are chosen to measure the synthesized videos.
Revised: In terms of FVD [Table] two temporal spans (covering consecutive 16 and 128 frames) are chosen to measure the synthesized videos.

Original: Benchmarks.
Revised: Benchmarks.

Original: In Sec. 2, we have already conducted comprehensive ablation studies on SkyTimelapse (Xiong et al., 2018),
Revised: In Sec. 2, we have already conducted comprehensive ablation studies on SkyTimelapse (Xiong et al., 2018),

Original: Taichi-HD (Siarohin et al., 2019b), and YouTube Driving dataset. To be [Figure]
Revised: Taichi-HD (Siarohin et al., 2019b), and YouTube Driving dataset (Zhang et al., 2022).

Original: [Table] specific, SkyTimelapse contains over two thousand videos which in average last twenty seconds.
Revised: To be specific, SkyTimelapse contains over two thousand videos which in average last twenty seconds.

Original: Various types of scenes (daytime, nightfall [Table] and aurora) are included.
Revised: Various types of scenes (daytime, nightfall, and aurora) are included.

Original: Similar to StyleGANV (Skorokhodov et al., 2022), we resize the videos of SkyTimelapse to 256 × 256 resolution.
Revised: Similar to StyleGAN-V (Skorokhodov et al., 2022), we resize the videos of SkyTimelapse to 256 ×resolution.

Original: Taichi-HD contains over three thousand videos recording a person performing Tai-chi.
Revised: Taichi-HD contains over three thousand videos recording a person performing Taichi.

Original: We also resize the videos to 256 × 256 resolution.
Revised: We also resize the videos to 256 × 256 resolution.

Original: Moreover, we collect a new driving dataset by crawling videos from YouTube. The dataset  consists of 134 first-view driving videos with a total length of over 120 hours, covering up to 68 cities, showing various conditions, from different weather, different regions, to diverse scene types.
Revised: YouTube Driving dataset (Zhang et al., 2022) consists of 134 first-view driving videos with a total length of over 120 hours, covering up to 68 cities, showing various conditions, from different weather, different regions, to diverse scene types.

Original: The driving scenes have tight geometrical constraints, with most of the objects (vehicles, buildings) following the rigid body constraint.
Revised: The driving scenes have tight geometrical constraints, with most of the objects (vehicles, buildings) following the rigid body constraint.

Original: We resize the videos to 180 × 320 resolution.
Revised: We resize the videos to 180 × 320 resolution.

Original: A part of videos with countryside views in 320 × 640 resolution is also chosen to benchmark high resolution video generation.
Revised: A part of videos with countryside views in 320 × 640 resolution is also chosen to benchmark high resolution video generation.

Original: For training, we resize them to 256 × 256 and 512 × 512 resolution respectively.
Revised: For training, we resize them to 256 × 256 and 512 × 512 resolution respectively.

Original: We believe this dataset poses new challenges for video generation.

Original: Additionally, we compare our approach against previous methods including MoCoGAN (Tulyakov et al., 2018) and its StyleGAN2 based variant, MoCoGAN-HD (Tian et al., 2021), VideoGPT (Yan et al., 2021), DIGAN (Yu et al., 2022), and StyleGAN-V (Skorokhodov et al., 2022).
Revised: Additionally, we compare our approach against previous methods including MoCoGAN (Tulyakov et al., 2018) and its StyleGAN2 based variant, MoCoGAN-HD (Tian et al., 2021), VideoGPT (Yan et al., 2021), DIGAN (Yu et al., 2022), and StyleGAN-V (Skorokhodov et al., 2022).

Original: Training.
Revised: Training.

Original: We follow the training receipt of StyleGAN-V and train models on a server with 8 A
Revised: We follow the training receipt of StyleGAN-V and train models on a server with 8 A

Original: GPUs.
Revised: GPUs.

Original: In terms of various methods and datasets, we grid search the R 1 regularization weight, whose details are available in Appendix.
Revised: In terms of various methods and datasets, we grid search the R 1 regularization weight, whose details are available in Appendix.

Original: In particular, performances of MoCoGAN and its StyleGAN based variant, MoCoGAN-HD, VideoGPT, and DIGAN are directly borrowed from StyleGAN-V. 
Revised: In particular, performances of MoCoGAN and its StyleGAN based variant, MoCoGAN-HD, VideoGPT, and DIGAN are directly borrowed from StyleGAN-V. [Table] [Table]

Original: 3. M AIN R ESULTS
Revised: 3. M AIN R ESULTS

Original: As shown in Tab. 2, our method significantly outperforms existing baselines on SkyTimelapse in terms of FVD 16 and FVD 128 .
Revised: As shown in Tab. [Table] our method significantly outperforms existing baselines on SkyTimelapse in terms of FVD 16 and FVD 128 .

Revised: On Taichi-HD

Original: On Taichi-HD, our method achieves consistent improvement in terms of FID, FVD 16 , and FVD 128 as reported in Tab.3.
Revised: [Table] our method achieves consistent improvement in terms of FID, FVD [Table] and FVD 128 as reported in Tab.

Original: We also compare our method with StyleGAN-V in challenging YouTube Driving dataset.
Revised: We also compare our method with StyleGAN-V in challenging YouTube Driving dataset.

Original: at both 256 × 256 and 512 × 512 resolution.
Revised: at both 256 × 256 and 512 × 512 resolution.

Original: For 256 × [Table] [Table] resolution in Tab.
Revised: For 256 ×resolution in Tab.

Original: [Table]
Revised: [Table]

Original: our method achieves better performance in both FID, FVD 16 , and FVD 128 .
Revised: our method achieves better performance in both FID, FVD 16 , and FVD 128 .

Original: For 512 × 512 resolution, StyleGAN-V has a comparable FID score but much worse FVD score.
Revised: For 512 × 512 resolution, StyleGAN-V has a comparable FID score but much worse FVD score.

Original: For both two resolutions on YouTube driving, we yield a strong performance of less than 50% FVD score than the prior art.
Revised: For both two resolutions on YouTube driving, we yield a strong performance of less than 50% FVD score than the prior art.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, our method achieves an appealing visual effect, with smooth frame transition.
Revised: 1, our method achieves an appealing visual effect, with smooth frame transition.

Original: 3. P ROPERTIES
Revised: 3. P ROPERTIES

Original: Good translation equivariance.
Revised: Good translation equivariance.

Original: We adopt alias-free technique from image generation for a smooth frame transition.
Revised: We adopt alias-free technique from image generation for a smooth frame transition.

Original: This design thus allows for geometric manipulation of the video with strong translation equivariance using a human-specified translation matrix.
Revised: This design thus allows for geometric manipulation of the video with strong translation equivariance using a human-specified translation matrix.

Original: We showcase several examples in Fig.
Revised: We showcase several examples in Fig.

Original: 6 by feeding different translation matrix to the video generator.
Revised: 6 by feeding different translation matrix to the video generator.

Original: It is clear that videos can be successfully translated [Table] and translation equivariance is well maintained over time.
Revised: It is clear that videos can be successfully translated, and translation equivariance is well maintained over time.

Original: This equivariance property improves human controllability over the video generation process.
Revised: This equivariance property improves human controllability over the video generation process.

Original: [Table]
Revised: [Table]

Original: No jittering phenomenon.
Revised: No jittering phenomenon.

Original: As mentioned in Sec. 2.4, discontinuity exists in the motion representation in the prior method, around which jittering phenomenon appears in generated videos.
Revised: As mentioned in Sec. 2.4, discontinuity exists in the motion representation in the prior method, around which jittering phenomenon appears in generated videos.

Original: It is caused by discretized anchor features with fixed interval lengths (256 frames in practice).
Revised: It is caused by discretized anchor features with fixed interval lengths (256 frames in practice).

Original: Our method develops a novel B-Spline based motion representation to ensure the temporal smoothness over the anchorfeatures.
Revised: Our method develops a novel B-Spline based motion representation to ensure the temporal smoothness over the anchor

Original:  To quantify the smoothness of generated videos, we sample 16 frames across the anchor (eight frames before and behind the anchor) and calculate the FID 16 metric (which we name ”FID 16 -anchor”) in YouTube Driving dataset.
Revised: [Figure] features. To quantify the smoothness of generated videos, we sample 16 frames across the anchor (eight frames before and behind the anchor) and calculate the FID 16 metric (which we name ”FID 16 -anchor”) in YouTube Driving dataset.

Original: As reported in Tab. 5, Config-D suffers from severe performance drop. While Config-F with B-Spline based motion representation maintains the move left move [Figure] move
Revised: As reported in Tab. 5, Config-D suffers from severe performance drop, while Config-F with B-Spline based motion representation maintains the performance.

Original: [Figure]

Original: performance, indicating that our method can generate long videos with smooth transitions without performance degrading.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Synthesizing videos from latent space.
Revised: Synthesizing videos from latent space.

Original: Most of the video generation frameworks are built upon
Revised: Most of the video generation frameworks are built upon

Original: GANs (Goodfellow et al., 2014; Brock et al., 2019; Karras et al., 2019; 2020b; 2021; 2018; Radford et al., 2016), owing to their flexibility as well as strong performance in image generation.
Revised: GANs (Goodfellow et al., 2014; Brock et al., 2019; Karras et al., 2019; 2020b; 2021; 2018; Radford et al., 2016), owing to their flexibility as well as strong performance in image generation.

Original: Prior arts (Tian et al., 2021;
Revised: Prior arts (Tian et al., 2021;

Original: Saito et al., 2017; Tulyakov et al., 2018; Fox et al., 2021; Skorokhodov et al., 2022;
Revised: Saito et al., 2017; Tulyakov et al., 2018; Fox et al., 2021; Skorokhodov et al., 2022;

Original: Yu et al., 2022; Vondrick et al., 2016;
Revised: Yu et al., 2022; Vondrick et al., 2016;

Original: Saito et al., 2020; Clark et al., 2019; Wang et al., 2022b) adopt image generators to synthesize frames from a time-coherent sequence of latent codes generated by recurrent networks.
Revised: Saito et al., 2020; Clark et al., 2019; Wang et al., 2022b) adopt image generators to synthesize frames from a time-coherent sequence of latent codes generated by recurrent networks.

Original: To explicitly disentangle the motion from the content, MoCoGAN (Tulyakov et al., 2018) and TGAN (Saito et al., 2017) employ a motion code and a content code as input noise for the generator, serving as a common strategy for the later works.
Revised: To explicitly disentangle the motion from the content, MoCoGAN (Tulyakov et al., 2018) and TGAN (Saito et al., 2017) employ a motion code and a content code as input noise for the generator, serving as a common strategy for the later works.

Original: StyleGAN-V (Skorokhodov et al., 2022), which we use as a strong baseline, and DIGAN (Yu et al., 2022)
Revised: StyleGAN-V (Skorokhodov et al., 2022), which we use as a strong baseline, and DIGAN (Yu et al., 2022)

Original: both use implicit neural-based representations for continuous video synthesis.
Revised: both use implicit neural-based representations for continuous video synthesis.

Original: Brooks et al.
Revised: Brooks et al.

Original: (2022) leverages a multiresolution training strategy to prioritize the time axis and ensure long-term consistency.
Revised: (2022) leverages a multiresolution training strategy to prioritize the time axis and ensure long-term consistency.

Original: Another line of works (Yan et al., 2021; Kalchbrenner et al., 2017; Weissenborn et al., 2019; Rakhimov et al., 2020; Walker et al., 2021) use autoregressive models to achieve video generation.
Revised: Another line of works (Yan et al., 2021; Kalchbrenner et al., 2017; Weissenborn et al., 2019; Rakhimov et al., 2020; Walker et al., 2021) use autoregressive models to achieve video generation.

Original: Kalchbrenner et al.
Revised: Kalchbrenner et al.

Original: (2017) use a PixelCNN (Van den Oord et al., 2016) decoder to synthesize the next frame pixel by pixel in an autoregressive manner.
Revised: (2017) use a PixelCNN (Van den Oord et al., 2016) decoder to synthesize the next frame pixel by pixel in an autoregressive manner.

Original: VideoGPT (Yan et al., 2021) built upon VQ-VAE adopts an autoregressive model to decompose videos into a sequence of tokens.
Revised: VideoGPT (Yan et al., 2021) built upon VQ-VAE adopts an autoregressive model to decompose videos into a sequence of tokens.

Original: TATS-base (Ge et al., 2022) similarly leverages 3D-VQGAN to decompose videos into tokens and further uses transformers to model the relationship between frame tokens.
Revised: TATS-base (Ge et al., 2022) similarly leverages 3D-VQGAN to decompose videos into tokens and further uses transformers to model the relationship between frame tokens.

Original: Additionally, some works using diffusion-based models also present promising results on video generation.
Revised: Additionally, some works using diffusion-based models also present promising results on video generation.

Original: Video diffusion (Ho et al., 2022) models entire videos using a 3D video architecture while Yang et al.
Revised: Video diffusion (Ho et al., 2022) models entire videos using a 3D video architecture while Yang et al.

Original: (2022) uses an image diffusion model to synthesize frames with a recurrent network.
Revised: (2022) uses an image diffusion model to synthesize frames with a recurrent network.

Original: Synthesizing videos with guided conditions.
Revised: Synthesizing videos with guided conditions.

Original: A close line of research is video prediction, aiming to generate full videos from the observation of previous frames (Babaeizadeh et al., 2017; Kumar et al., 2019; Lee et al., 2018; Luc et al., 2020;
Revised: A close line of research is video prediction, aiming to generate full videos from the observation of previous frames (Babaeizadeh et al., 2017; Kumar et al., 2019; Lee et al., 2018; Luc et al., 2020;

Original: Nash et al., 2022; Finn et al., 2016) or a given conditions (Kim et al., 2020; 2021; Ha & Schmidhuber, 2018).
Revised: Nash et al., 2022; Finn et al., 2016) or a given conditions (Kim et al., 2020; 2021; Ha & Schmidhuber, 2018).

Original: These works typically employ the reconstruction losses to make the future frames predictable for the model.
Revised: These works typically employ the reconstruction losses to make the future frames predictable for the model.

Original: Video translation (Pan et al., 2019; Yang et al., 2018; Wang et al., 2018; Chan et al., 2019; Siarohin et al., 2019a; Ren et al., 2020; Siarohin et al., 2019b) is also a paradigm for video synthesis, which translates the given segmentation masks and keypoints into videos.
Revised: Video translation (Pan et al., 2019; Yang et al., 2018; Wang et al., 2018; Chan et al., 2019; Siarohin et al., 2019a; Ren et al., 2020; Siarohin et al., 2019b) is also a paradigm for video synthesis, which translates the given segmentation masks and keypoints into videos.

Original: Some (Li et al., 2022; Liu et al., 2021; Ren & Wang, 2022) focus on synthesizing videos of indoor and outdoor scenes for given camera trajectories, which need to model the 3D scene explicitly.
Revised: Some (Li et al., 2022; Liu et al., 2021; Ren & Wang, 2022) focus on synthesizing videos of indoor and outdoor scenes for given camera trajectories, which need to model the 3D scene explicitly.

Original: Besides, Ho et al.
Revised: Besides, Ho et al.

Original: ; Hong et al.
Revised: ; Hong et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: (2021a;b) explore text-to-video generation with diffusion-based models and transformers.
Revised: (2021a;b) explore text-to-video generation with diffusion-based models and transformers.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we set up a simple yet effective framework for video generation task.
Revised: In this work, we set up a simple yet effective framework for video generation task.

Original: By introducing the alias-free operations together with pre-learned knowledge, per-frame quality is improved.
Revised: By introducing the alias-free operations together with pre-learned knowledge, per-frame quality is improved.

Original: Explicit temporal modeling in discriminator enhances the dynamic synthesis.
Revised: Explicit temporal modeling in discriminator enhances the dynamic synthesis.

Original: In addition, a new
Revised: In addition, a new

Original: B-Spline based motion representation  is proposed to enable smooth infinite frame generation.
Revised: B-Spline based motion representation with low-rank temporal modulation is proposed to improve infinite frame generation.

Original: Experimental results show substantial improvementsover baseline approach on different datasets.
Revised: Experimental results show substantial improvements on different datasets.

Original: Discussion.
Revised: Discussion.

Original: Our approach has several limitations.
Revised: Our approach has several limitations.

Original: Concretely, the current framework follows prior arts using two latent codes to represent content and motion respectively.
Revised: Our current framework uses two latent codes to represent content and motion, but they often entangle with each other.

Original: However, in our experiments, it is suggested that they usually entangle with each other, especially for the driving scenes which hardly define the content and motion strictly.

Original: Moreover, the frame quality is still far from satisfactory.
Revised: Moreover, the frame quality is still far from satisfactory.

Original: The object shape is not consistent across consecutive frames, producing apparent artifacts.
Revised: The object shape is not consistent across consecutive frames, producing apparent artifacts.

Original: This could be further improved by introducing structural priors.
Revised: This could be further improved by introducing structural priors.

Original: Last, although we could generate infinite videos by gradually changing content code, direct control of novel content over time is lacking.
Revised: Last, although we could generate infinite videos by gradually changing content code, direct control of novel content over time is lacking.

Original: Ge et al. (2022) used explicit planning in the latent space to achieve the controllability of the synthesis, especially for the infinite video generation. Similar ideas couldbe adopted in our framework to improve the quality of generated long videos.
Revised: Explicit planning in the latent space, as used by Ge et al. (2022), could improve the quality of generated long videos.

Original: A PPENDIX
Revised: A PPENDIX

Original: This appendix is organized as follows.
Revised: This appendix is organized as follows.

Original: appendix A and appendix B present the implementation and dataset details respectively, followed by appendix D showing more visual results.
Revised: appendix A and appendix B present the implementation and dataset details respectively, followed by appendix D showing more visual results.

Original: A I MPLEMENTATION D ETAILS .
Revised: A I MPLEMENTATION D ETAILS .

Original: Our method is developed based on the official implementation of StyleGAN-V (Skorokhodov et al., 2022).
Revised: Our method is developed based on the official implementation of StyleGAN-V (Skorokhodov et al., 2022).

Original: We adopt hyper-parameters, the optimizer, the loss function, and the training script to ensure a fair comparison.
Revised: We adopt hyper-parameters, the optimizer, the loss function, and the training script to ensure a fair comparison.

Original: Notably, the total number of seen images for sufficient data training is 25 million regardless of datasets.
Revised: Notably, the total number of seen images for sufficient data training is 25 million regardless of datasets.

Original: We evaluate once after training on every 2.5 million images, and report the result with highest FVD 16 score.
Revised: We evaluate once after training on every 2.5 million images, and report the result with highest FVD 16 score.

Original: Due to statistics variance of different datasets, we search training regularization term, i.e. R 1 value, for each method and dataset.
Revised: Due to statistics variance of different datasets, we search training regularization term, i.e. R 1 value, for each method and dataset.

Original: Empirically, we find that a smaller R 1 value ( e.g., 0.25) works well for pretraining stage ( Config-C ).
Revised: Empirically, we find that a smaller R 1 value ( e.g., 0.25) works well for pretraining stage ( Config-C ).

Original: While a larger R 1 value ( e.g., 4) better suits to video generation learning.
Revised: While a larger R 1 value ( e.g., 4) better suits to video generation learning.

Original: B D ATASET D ETAILS .
Revised: B D ATASET D ETAILS .

Original: In this section, We introduce the datasets we use, briefly analyze their characteristics and challenges brought for video generation, and show sampled real video frames.
Revised: In this section, We introduce the datasets we use, briefly analyze their characteristics and challenges brought for video generation, and show sampled real video frames.

Original: SkyTimelapse.
Revised: SkyTimelapse.

Original: SkyTimelapse (Xiong et al., 2018) is a time-lapse dataset collected from the Internet showing dynamic sky scenes, such as the cloudy sky with moving clouds, and the starry sky with moving stars.
Revised: SkyTimelapse (Xiong et al., 2018) is a time-lapse dataset collected from the Internet showing dynamic sky scenes, such as the cloudy sky with moving clouds, and the starry sky with moving stars.

Original: It contains various conditions, for instance, different weather conditions (daytime, nightfall, dawn, starry night and aurora), different kinds of foreground object (cloudy sky, starry sky, and sunny sky), and different motion patterns of the sky.
Revised: It contains various conditions, for instance, different weather conditions (daytime, nightfall, dawn, starry night and aurora), different kinds of foreground object (cloudy sky, starry sky, and sunny sky), and different motion patterns of the sky.

Original: It also contains some other objects like trees, mountains, and buildings which further improve the visual diversity.
Revised: It also contains some other objects like trees, mountains, and buildings which further improve the visual diversity.

Original: However, since clouds are fluids, there are very few constraints on movement.
Revised: However, since clouds are fluids, there are very few constraints on movement.

Original: Minor disturbances and deformations can make a generated video look realistic.
Revised: Minor disturbances and deformations can make a generated video look realistic.

Original: The whole dataset contains over two thousand videos which in average last twenty seconds.
Revised: The whole dataset contains over two thousand videos which in average last twenty seconds.

Original: We resize the videos to 256 × 256 following prior works.
Revised: We resize the videos to 256 × 256 following prior works.

Original: We sample several video clips in Fig.
Revised: We sample several video clips in Fig.

Original: A1.
Revised: A1.

Original: YouTube Driving.
Revised: YouTube Driving.

Original: We collect YouTube Driving Dataset  from YouTube which contains a massive amount of real-world driving frames with various conditions, from different weather, different regions, to diverse scene types.
Revised: YouTube Driving Dataset (Zhang et al., 2022) is crawled from YouTube which contains a massive amount of real-world driving frames with various conditions, from different weather, different regions, to diverse scene types.

Original: To be specific, 134 videos with a total length of over 120 hours are collected, covering up to 68 cities.
Revised: To be specific, 134 videos with a total length of over 120 hours are collected, covering up to 68 cities.

Original: Fig.
Revised: Fig.

Original: A2 shows sampled frames.
Revised: A2 shows sampled frames.

Original: The videos are resized to 256 × 256 resolution for training.
Revised: The videos are resized to 256 × 256 resolution for training.

Original: The driving scenesshown in our collected frames have tighter geometrical constraints, with most of the objects (vehicles, buildings) following the rigid body constraint.
Revised: The driving scenes have tighter geometrical constraints, with most of the objects (vehicles, buildings) following the rigid body constraint.

Original: The 3D structure of the whole scene should be maintained without any deformation.
Revised: The 3D structure of the whole scene should be maintained without any deformation.

Original: Besides, different types of objects show different motion patterns.
Revised: Besides, different types of objects show different motion patterns.

Original: Roads and buildings recede at a reasonable speed opposite to that of ego car, while other cars show lane changes, stops, accelerations.
Revised: Roads and buildings recede at a reasonable speed opposite to that of ego car, while other cars show lane changes, stops, accelerations.

Original: We believe YouTube Driving dataset is challenging for video generation, hoping that this dataset will lead to new questions that will move the video generation field forward.

Original: Countryside.
Revised: Countryside.

Original: We select a part of videos with countryside scenes from YouTube Driving dataset and resize them to 512 × 512  for training.
Revised: We select a part of videos with countryside scenes from YouTube Driving dataset and resize them to 512 × 512 resolution for training.

Revised: We use this dataset to benchmark the performance of generating high-resolution videos.

Original: Real samples are presented in Fig.
Revised: Real samples are presented in Fig.

Original: A3, where we benchmark the performance of generating high-resolution videos.
Revised: A3.

Original: Taichi-HD.
Revised: Taichi-HD.

Original: Taichi-HD (Siarohin et al., 2019b) is a collection of YouTube videos recording a single person performing Tai-chi.
Revised: Taichi-HD (Siarohin et al., 2019b) is a collection of YouTube videos recording a single person performing Tai-chi.

Original: It contains 3049 training videos in total, with various foreground objects (people with different styles of clothing) and diverse background.
Revised: It contains 3049 training videos in total, with various foreground objects (people with different styles of clothing) and diverse background.

Original: It requires realistic human motion generation, which contains non-rigid motion modeling, which poses a significant challenge for video generation.
Revised: It requires realistic human motion generation, which contains non-rigid motion modeling, which poses a significant challenge for video generation.

Original: [Table]
Revised: [Table]

Original: C U SER S TUDY
Revised: C U SER S TUDY

Original: To further prove our method’s efficacy, we conduct user study over baseline ( Config-A ), ConfigD , and Config-F on YouTube
Revised: To further prove our method’s efficacy, we conduct user study over baseline ( Config-A ), ConfigD , and Config-F on YouTube

Original: Driving dataset and Taichi-HD dataset.
Revised: Driving dataset and Taichi-HD dataset.

Original: We evaluate from three perspectives:
Revised: We evaluate from three perspectives:

Original: Single Image Quality.
Revised: Single Image Quality.

Original: We provide user with a collection of images randomly sampled from generated videos.
Revised: We provide user with a collection of images randomly sampled from generated videos.

Original: Short Video Quality.
Revised: Short Video Quality.

Original: We provide user with a collection of generated videos with consecutiveframes (up to about 5 seconds).
Revised: We provide user with a collection of generated videos with consecutiveframes (up to about 5 seconds).

Original: Long Video Quality.
Revised: Long Video Quality.

Original: We provide user with a collection of generated videos with consecutiveframes (up to 20 seconds).
Revised: We provide user with a collection of generated videos with consecutiveframes (up to 20 seconds)

Original: As shown in Tab. A1, in terms of single image quality, users prefer Config-D over other settings in both YouTube Driving and Taichi-HD dataset.
Revised: As shown in Tab. A1, in terms of single image quality, users prefer Config-D over other settings in both YouTube Driving and Taichi-HD dataset.

Original: The same trend occurs in short video quality assessment.
Revised: The same trend occurs in short video quality assessment.

Original: With pre-learned knowledge and a stronger discriminator with temporal shift module (TSM), Config-D reaps a significantly higher number of votes.
Revised: With pre-learned knowledge and a stronger discriminator with temporal shift module (TSM), Config-D reaps a significantly higher number of votes.

Original: However, for long video generation,
Revised: However, for long video generation,

Original: Config-F becomes the best choice for both YouTube Driving and Taichi-HD dataset.
Revised: Config-F becomes the best choice for both YouTube Driving and Taichi-HD dataset.

Original: Unnatural jittering and discontinuous motion can drastically affect the users’ video viewing experience.
Revised: Unnatural jittering and discontinuous motion can drastically affect the users’ video viewing experience.

Original: With
Revised: With

Original: B-Spline based motion embedding and low rank time modulation, Config-F is undoubtedly the best option for long video generation.
Revised: B-Spline based motion embedding and low rank time modulation, Config-F is undoubtedly the best option for long video generation.

Original: D M ORE V ISUAL R ESULTS .
Revised: D M ORE V ISUAL R ESULTS .

Original: Texture Sticking phenomenon.
Revised: Texture Sticking phenomenon.

Original: Texture sticking exists in videos generated by prior methods.
Revised: Texture sticking exists in videos generated by prior methods.

Original: Our method leverage the alias-free technique in the generator to overcome this issue.
Revised: Our method leverage the alias-free technique in the generator to overcome this issue.

Original: Fig.
Revised: Fig.

Original: A4 shows more comparison between our method and StyleGAN-V (Skorokhodov et al., 2022).
Revised: A4 shows more comparison between our method and StyleGAN-V (Skorokhodov et al., 2022).

Original: Jittering phenomenon.
Revised: Jittering phenomenon.

Original: Motion embedding designed in previous method is not first-order continuous, leading to jittering phenomenon when generating long videos.
Revised: Motion embedding designed in previous method is not first-order continuous, leading to jittering phenomenon when generating long videos.

Original: We develop B-Spline based motion embedding with smooth property which alleviates the problem.
Revised: We develop B-Spline based motion embedding with smooth property which alleviates the problem.

Original: We show examples comparing video sequences generated by our method with StyleGAN-V (Skorokhodov et al., 2022) in Fig.
Revised: We show examples comparing video sequences generated by our method with StyleGAN-V (Skorokhodov et al., 2022) in Fig.

Original: A5.
Revised: A5.

Original: Our videos are smooth, without abnormal movements.
Revised: Our videos are smooth, without abnormal movements.

Original: In videos generated by
Revised: In videos generated by

Original: StyleGAN-V (Skorokhodov et al., 2022), street lights and zebra crossings may suddenly change their direction of movement.
Revised: StyleGAN-V (Skorokhodov et al., 2022), street lights and zebra crossings may suddenly change their direction of movement.

Original: Repeating contents.
Revised: Repeating contents.

Original: With B-Spline based motion embedding, Config-E can compose smooth long videos.
Revised: With B-Spline based motion embedding, Config-E can compose smooth long videos.

Original: However, similar contents appear periodically as shown in Fig.
Revised: However, similar contents appear periodically as shown in Fig.

Original: A6a.
Revised: A6a.

Original: B-Spline based motion embedding is endowed with so strong capacity that it represents content and motion concepts simultaneously rather than motion concepts only.
Revised: B-Spline based motion embedding is endowed with so strong capacity that it represents content and motion concepts simultaneously rather than motion concepts only.

Original: We suppress the representation capacity through the low-rank strategy.
Revised: We suppress the representation capacity through the low-rank strategy.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: A6b, Config-F with low rank time modulation can generate progressively forward contents without the repeating phenomenon.
Revised: A6b, Config-F with low rank time modulation can generate progressively forward contents without the repeating phenomenon.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]