Original: R EVISITING H IERARCHICAL A PPROACH FOR P ERSISTENT L ONG -T ERM V IDEO P REDICTION
Revised: R EVISITING H IERARCHICAL A PPROACH FOR P ERSISTENT L ONG -T ERM V IDEO P REDICTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning to predict the long-term future of video frames is notoriously challenging due to the inherent ambiguities in a distant future and  dramatic ampliﬁcation of  prediction error over time.
Revised: Learning to predict the long-term future of video frames is notoriously challenging due to the inherent ambiguities in the distant future and the dramatic ampliﬁcation of the prediction error through time.

Original: Despite the recent advances in the literature, existing approaches are limited to moderately short-term prediction (less than a few seconds), while extrapolating it to a longer future quickly leads to destruction in structure and content.
Revised: Despite the recent advances in the literature, existing approaches are limited to moderately short-term prediction (less than a few seconds), while extrapolating it to a longer future quickly leads to destruction in structure and content.

Original: In this work, we revisitthe hierarchical models in video prediction.
Revised: In this work, we revisit hierarchical models in video prediction.

Original: Our method generates future frames by ﬁrst estimating a sequence of dense semantic structures and subsequently translating the estimated structures to pixels by video-to-video translation model.
Revised: Our method generates future frames by ﬁrst estimating a sequence of dense semantic structures and subsequently translating the estimated structures to pixels by video-to-video translation.

Original: Despite the simplicity, we show that modeling structures and their dynamics in  categorical structure space with  stochastic sequential estimator leads to surprisingly successful long-term prediction.
Revised: Despite its simplicity, we show that modeling structures and their dynamics in a categorical structure space with a stochastic sequential estimator leads to surprisingly successful long-term prediction.

Original: We evaluate our method on two challenging video prediction scenarios, car driving and human dancing , and demonstrate that it can generate complicated scene structures and motions over a very long time horizon ( i.e., thousands frames), setting a new standard of video prediction with orders of magnitude longer prediction time than existing approaches.
Revised: We evaluate our method on three challenging video prediction datasets involving car driving and human dancing , and demonstrate that it can generate complicated scene structures and motions over a very long time horizon ( i.e., thousands frames), setting a new standard of video prediction with orders of magnitude longer prediction time than existing approaches.

Original: Video results are avaiable at https://bit.ly/2EyDSem.
Revised: Full videos are available at https://bit.ly/2EyDSem.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Video prediction aims to generate future frames conditioned on a short video clip.
Revised: Video prediction aims to generate future frames conditioned on a short video clip.

Original: It has received much attention in recent years as forecasting the future of visual sequence improves the planning for model-based reinforcement learning (Finn et al., 2016; Hafner et al., 2019; Ha & Schmidhuber, 2018) and has immediate applications (Hoai & Torre, 2013; Ryoo, 2011; Lan et al., 2014).
Revised: It has received much attention in recent years as forecasting the future of visual sequence is critical in improving the planning for model-based reinforcement learning (Finn et al., 2016; Hafner et al., 2019; Ha & Schmidhuber, 2018), forecasting future event (Hoai & Torre, 2013), action (Lan et al., 2014), and activity (Lan et al., 2014; Ryoo, 2011).

Original:  Many previous approaches have formulated video prediction as  conditionalimage generation problem by recursively synthesizing future frames using Generative Adversarial Networks (GANs) (Lee et al., 2018; Vondrick et al., 2016;Tulyakov et al., 2018) or Variational Autoencoders(VAEs) (Denton & Fergus, 2018;
Revised: To make it truly beneﬁcial for these applications, video prediction should be capable of forecasting long-term future. Many previous approaches have formulated video prediction as a conditional generation task by recursively synthesizing future frames conditioned on the previous frames (Vondrick et al., 2016;

Original:  Babaeizadeh et al., 2018; Castrejon et al., 2019; Villegas et al., 2019).
Revised: Tulyakov et al., 2018; Denton & Fergus, 2018; Babaeizadeh et al., 2018; Castrejon et al., 2019; Villegas et al., 2019).

Original: These approaches have made meaningful progress inpredicting a short-term future. However, those methods havenot been successful in synthesizing convincing long-term future due to the complexity of spatio-temporal variations in high-dimensional data, anddue to the difﬁculty of extrapolating from short sequences to much longer future.
Revised: Despite their success in short-term forecasting, however, none of these approaches have been successful in synthesizing convincing long-term future, due to the challenges in modeling complex dynamics and extrapolating from short sequences to much longer future.

Original: As the prediction errorsare easily propagated and ampliﬁed through time, the quality of the predicted frames quickly degrades over time.
Revised: As the prediction errors easily accumulates and ampliﬁes through time, the quality of the predicted frames quickly degrades over time.

Original: One way to reduce the error propagation is to extrapolate in a low dimensional structure space instead of directly estimating pixel-level dynamics in a video.
Revised: One way to reduce the error propagation is to extrapolate in a low dimensional structure space instead of directly estimating pixel-level dynamics in a video.

Original: Therefore, many hierarchical modeling approaches are proposed (Villegas et al., 2017b; Wichers et al., 2018;
Revised: Therefore, many hierarchical modeling approaches are proposed (Villegas et al., 2017b; Wichers et al., 2018;

Original: Liang et al., 2017; Yan et al., 2018; Walker et al., 2017; Kim et al., 2019).
Revised: Liang et al., 2017; Yan et al., 2018; Walker et al., 2017; Kim et al., 2019).

Original: These approaches ﬁrst generate a sequence using  low-dimensional structure representation, and subsequently generate appearance conditioned on the predicted structures.
Revised: These approaches ﬁrst generate a sequence using a low-dimensional structure representation, and subsequently generate appearance conditioned on the predicted structures.

Original: Hierarchical approaches are potentially promising for long-term prediction since learning structure-aware dynamics allows the model to generate semantically accurate motion and content in the future.
Revised: Hierarchical approaches are potentially promising for long-term prediction since learning structure-aware dynamics allows the model to generate semantically accurate motion and content in the future.

Original: However, previous approaches often employed too speciﬁc and incomprehensive structures such as human body joints (Villegas et al., 2017b; Yan et al., 2018;
Revised: However, previous approaches often employed too speciﬁc and incomprehensive structures such as human body joints (Villegas et al., 2017b; Yan et al., 2018;

Original: Yang et al., 2018; Walker et al., 2017; Kim et al., 2019) or face landmarks (Yan et al., 2018;
Revised: Yang et al., 2018; Walker et al., 2017; Kim et al., 2019) or face landmarks (Yan et al., 2018;

Original: Yang et al., 2018).
Revised: Yang et al., 2018).

Original: Moreover, theyalso made oversimpliﬁed assumptions on the future by using  deterministic loss or assuming homogeneous content.
Revised: Moreover, they made oversimpliﬁed assumptions of the future by using a deterministic loss or assuming homogeneous content.

Original: We therefore argue that the beneﬁt ofthe hierarchical model has been underestimated and its impact on long-term video prediction has not been properly demonstrated.
Revised: We therefore argue that the beneﬁt of hierarchical model has been underestimated and their impact on long-term video prediction has not been properly demonstrated.

Original: In this paper, we propose a hierarchical model with  general structure representation (i.e., dense semantic label map ) for long-term video prediction with complex scenes.
Revised: In this paper, we propose a hierarchical model with a general structure representation (i.e., dense semantic label map ) for long-term video prediction with complex scenes.

Original: We abstract the scene as 
Revised: We abstract the scene as categorical labels for each pixel, and predict the motion and content change in this label space

Original: [Figure]
Revised: [Figure]

Original: categorical labels for each pixel and then predict the motion and content change in this label space using variational sequence models.
Revised: using variational sequence models.

Original: Given the context frames and the predicted label maps, we then generate the textures by translating the sequence of label maps to actual RGB frames.
Revised: the context frames and the predicted label maps, we then generate the textures by translating the sequence of label maps to the RGB frames.

Original: As dense label map is generic and universal, we can learn comprehensive scene dynamics including foreground object motion and global camera motion.
Revised: As dense label maps are generic and universal, we can learn comprehensive scene dynamics from object motion to even dramatic scene change.

Original: Moreover, we can also capture the multi-modal dynamics in the dense labelspace with thestochastic prediction of the variational sequence model.
Revised: We can also capture the multi-modal dynamics in the dense label the prediction of the variational sequence model.

Original: Consequently, we are able to generate  long term  videos from driving scene to human dancing, including motion of multiple objects and even an evolution of thecontent in a distant future. Our experiment results demonstrate that the proposed hierarchical model can generate surprisingly long-term future, which are semantically accurate and capturing complex motions. We also show that thepredicted frame quality ispreserved through time, which enables persistent future prediction virtually near inﬁnite
Revised: that we can generate a surprisingly long term future of videos from driving scenes to human the complex motion of multiple objects and even an evolution of the in a future the is persistent future prediction virtually near prediction wise the without ground

Original:  M ETHOD
Revised:  M ETHOD

Original: Given the context frames x 1: C = { x 1 , x 2 , ..., x C } , our goal is to synthesize the future frames x C +1: T = { x C +1 , x C +2 , ..., x T } up to an arbitrary long-term future T .
Revised: Given the context frames x 1: C = { x 1 , x 2 , ..., x C } , our goal is to synthesize the future frames x C +1: T = { x C +1 , x C +2 , ..., x T } up to an arbitrary long-term future T .

Original: Let s t ∈ R W × H × N denotes a dense label map of the frame x t deﬁned over N categories, which is inferred by the pre-trained semantic segmentation model 1 .
Revised: Let s t ∈ R W × H × N denote a dense label map of the frame x t deﬁned over N categories, which is inferred by the pre-trained semantic segmentation model 1 .

Original: Then given the context frames x 1: C and the label maps s 1: C , our hierarchical framework synthesizes the future frames ˆ x C +1: T by  following steps.
Revised: Then given the context frames x 1: C and the label maps s 1: C , our hierarchical framework synthesizes the future frames ˆ x C +1: T by the following steps.

Original: • The structure generator takes the context label maps as inputs, and produces a sequence of the future label maps by ˆ s C +1: T ∼ G struct ( s 1: C , z 1: T ) , where z t denotes the latent variable encoding  stochasticity of the structure.
Revised: • The structure generator takes the context label maps as input and produces a sequence of the future label maps by ˆ s C +1: T ∼ G struct ( s 1: C , z 1: T ) , where z t denotes the latent variable encoding the stochasticity of the structure.

Original: • Given the context frames and the predicted structures, the image generator produces the RGB frames by ˆ x C +1: T ∼ G image ( x 1: C , { s 1: C , ˆ s C
Revised: • Given the context frames and the predicted structures, the image generator produces the RGB frames by ˆ x C +1: T ∼ G image ( x 1: C , { s 1: C , ˆ s C

Original: +1: T } ) .
Revised: +1: T } ) .

Original: Figure 1 illustrates the overall pipeline.
Revised: Figure 1 illustrates the overall pipeline.

Original: Note that there are various factors beyond motion that cause spatio-temporal variations in the label maps, such as the emergence of new objects, partial observability, or even the dramaticchanges of entire scene  by the global camera motion ( e.g. , panning).
Revised: Note that there are various factors beyond motion that make spatio-temporal variations in the label maps, such as the emergence of new objects, partial observability, or even the dramatic scene changes by the global camera motion ( e.g. , panning).

Original: By learning to model these dynamics in the semantic labels with  stochastic sequence estimator, and conditioning the video generation with the estimated labels, the proposed hierarchical model can synthesizethe convincing frames into the very long-term future. 
Revised: By learning to model these dynamics in the semantic labels with the stochastic sequence estimator and conditioning the video generation with the estimated labels, the proposed hierarchical model can synthesize convincing frames into the very long-term future. Below, we describe each component.

Original: 2. S EQUENTIAL D ENSE S TRUCTURE G ENERATOR
Revised: 2. S EQUENTIAL D ENSE S TRUCTURE G ENERATOR

Original: Our structure generator models the dynamics inthe label maps by p ( s ≤ T | z ≤ T ) =
Revised: Our structure generator models the dynamics in label maps by p ( s ≤ T | z ≤ T ) =

Original: (cid:81)
Revised: (cid:81)

Original: Tt =1 p ( s t | s <t , z ≤ t ) .
Revised: Tt =1 p ( s t | s <t , z ≤ t ) .

Original: We employ the sequence model based on VAE (Denton & Fergus, 2018) since (1) it provides a probabilistic framework to handle stochastic variations of the structure and (2) it can easily incorporatethe discrete sequence ( i.e., label maps).
Revised: We employ the sequence model based on VAE (Denton & Fergus, 2018) since (1) it provides a probabilistic framework to handle stochastic variations of the structure and (2) it can easily incorporate discrete sequences ( i.e., label maps).

Original: Speciﬁcally, we optimize the following variational lower-bound of the sequence:
Revised: Speciﬁcally, we optimize the following variational lower-bound of the sequence:

Original: [Equation]
Revised: [Equation]

Original: where q φ ( z ≤ T | s ≤ T ) is the approximated posterior distribution and p ψ ( z t | s <t ) is the prior distribution.
Revised: where q φ ( z ≤ T | s ≤ T ) is the approximated posterior distribution and p ψ ( z t | s <t ) is the prior distribution.

Original: At each step, it learns to represent and reconstruct each label map s t through CNN-based encoder f enc and decoder f dec , respectively, while the temporal dependency of the label maps is modeled stochastically by the two LSTMs as follows:
Revised: At each step, it learns to represent and reconstruct each label map s t through CNN-based encoder f enc and decoder f dec , respectively, while the temporal dependency of the label maps is modeled stochastically by the two LSTMs as follows:

Original: [Equation]
Revised: [Equation]

Original: where h t = f enc ( s t) , where h t − 1 = f enc ( s t −
Revised: where h t = f enc ( s t

Original: [Equation]
Revised: [Equation]

Revised: where h t − 1 = f enc ( s t −

Original: [Equation]
Revised: [Equation]

Original: where LSTM θ and LSTM φ respectively approximate the generative and the posterior distributions recurrently up to the time step t .
Revised: where LSTM θ and LSTM φ respectively approximate the generative and the posterior distributions recurrently up to the time step t .

Original: Unlike Denton & Fergus (2018); Villegas et al.
Revised: Unlike Denton & Fergus (2018); Villegas et al.

Original: (2019) that exploits skip-connection from the last observed context frame during both training and testing time, we skip hidden representations of the encoder to the decoder at every time step during testing to handle long-term dynamics in structure. 
Revised: (2019) that exploit skip-connection from the last observed context frame during both training and testing time, we skip hidden representations of the encoder to the decoder at every time step during testing to handle longterm dynamics in structure. We also apply aggressive dilated convolutions to encode global motion context.

Original:  During training, we applythe teacher-forcing by feeding ground-truth label maps and sampling the latent z t from the posterior distribution N ( µ φ ( t ) , σ φ ( t ))
Revised: Please see Appendix C.1 for more detail. During training, we apply teacher-forcing by feeding ground-truth label maps and sampling the latent z t from the posterior distribution N ( µ φ ( t ) , σ φ ( t ))

Original: =
Revised: =

Original: LSTM φ ( f enc ( s t )) .
Revised: LSTM φ ( f enc ( s t )) .

Original: During inference, we recursively generate label maps by (1) sampling z t from the prior distribution N ( µ ψ ( t ) , σ ψ ( t )) =
Revised: During inference, we recursively generate label maps by (1) sampling z t from the prior distribution N ( µ ψ ( t ) , σ ψ ( t )) =

Original: LSTM ψ ( f enc (ˆ s t − 1 )) , (2) producing the frame ˆ s t through  decoder, and (3) discretizing the predicted label map by taking pixel-wise maximum.
Revised: LSTM ψ ( f enc (ˆ s t − 1 )) , (2) producing the frame ˆ s t through the decoder, and (3) discretizing the predicted label map by taking pixel-wise maximum.

Original: Such discretization provides additional merits for being more robust against error propagation than continuous structures such as 2D keypoints (Villegas et al., 2017b; Yan et al., 2018; Yang et al., 2018) or optical ﬂow (Walker et al., 2017).
Revised: Such discretization provides additional merits for being more robust against error propagation than continuous structures such as 2D keypoints (Villegas et al., 2017b; Yan et al., 2018; Yang et al., 2018) or optical ﬂow (Walker et al., 2017).

Original: Extension to object boundary prediction When the pre-trained instance-wise segmentation model is available, we can optionally extend the structure generator to jointly predict the object boundary maps.
Revised: Extension to object boundary prediction When the pre-trained instance-wise segmentation model is available, we can optionally extend the structure generator to jointly predict the object boundary maps.

Original: Such structure can add a notion of object instance, and is useful to improve the image generator in sequences with many occluding objects.
Revised: Such structure can add a notion of object instance, and is useful to improve the image generator in sequences with many occluding objects.

Original: Let e t ∈ { 0 , 1 } W × H denotes the object boundary map at frame x t .
Revised: Let e t ∈ { 0 , 1 } W × H denotes the object boundary map at frame x t .

Original: Then we train the denoising autoencoder ˆ e t =
Revised: Then we train the denoising autoencoder ˆ e t =

Original: G edge ( s t ) that produces a boundary map given a label map for each frame 1 .
Revised: G edge ( s t ) that produces a boundary map given a label map for each frame 1 .

Original: We train G edge using  conditional GAN objective to maximize the joint probability between the label and boundary maps p ( s t , e t ) .
Revised: We train G edge using the conditional GAN objective to maximize the joint probability between the label and boundary maps p ( s t , e t ) .

Original: The boundary and label maps are then combined as the output of the structure generator ˜ s t =
Revised: The boundary and label maps are then combined as the output of the structure generator ˜ s t =

Original: [ˆ s t , ˆ e t ] , and is used as an input to the image generator described below.
Revised: [ˆ s t , ˆ e t ] , and is used as an input to the image generator described below.

Original: See Appendix B.2 for implementation details.
Revised: See Appendix C.2 for implementation details.

Original: 2. S TRUCTURE -C ONDITIONAL P IXEL S EQUENCE G ENERATOR
Revised: 2. S TRUCTURE -C ONDITIONAL P IXEL S EQUENCE G ENERATOR

Original: Given a sequence of the structures and the context frames, the image generator learns to model  conditional distribution of the RGB frames by p ( x ≤ T | s ≤ T ) =
Revised: Given a sequence of the structures and the context frames, the image generator learns to model the conditional distribution of the RGB frames by p ( x ≤ T | s ≤ T ) =

Original: (cid:81)
Revised: (cid:81)

Original: Tt =1 p ( x t | x <t , s ≤ t ) .
Revised: Tt =1 p ( x t | x <t , s ≤ t ) .

Original: We formulate this task as a video-to-video translation problem, and employ the state-of-the-art conditional video generation model (Wang et al., 2018).
Revised: We formulate this task as a video-to-video translation problem, and employ a state-of-the-art conditional video generation model (Wang et al., 2018).

Original: Speciﬁcally, the video synthesis network F in Wang et al.
Revised: Speciﬁcally, the video synthesis network F in Wang et al.

Original: (2018) consists of three main components; the generator H , occlusion mask predictor M , and optical ﬂow estimator W , which are combined to generate each frame ˆ x t by the following operation:
Revised: (2018) consists of three main components; the generator H , occlusion mask predictor M , and optical ﬂow estimator W , which are combined to generate each frame ˆ x t by the following operation:

Original: [Equation]
Revised: [Equation]

Original: where ˆ w t − 1 = W (ˆ x t − τ : t − 1 , s t − τ : t ) is the warped previous frame ˆ x t − 1 using the estimated optical ﬂow, h t
Revised: where ˆ w t − 1 = W (ˆ x t − τ : t − 1 , s t − τ : t ) is the warped previous frame ˆ x t − 1 using the estimated optical ﬂow, h t

Original: = H (ˆ x t − τ : t − 1 , s t − τ : t ) is the hallucinated frame at t , and m t =
Revised: = H (ˆ x t − τ : t − 1 , s t − τ : t ) is the hallucinated frame at t , and m t =

Original: M (ˆ x t − τ : t − 1 , s t − τ : t ) is the soft occlusion mask blending ˆ w t − 1 and h t .
Revised: M (ˆ x t − τ : t − 1 , s t − τ : t ) is the soft occlusion mask blending ˆ w t − 1 and h t .

Original: Unlike models synthesizing future frames by transforming the context frames x C (Villegas et al., 2019; 2017b; Yan et al., 2018; Tulyakov et al., 2018), this model is appropriate to synthesize  long-term future since it can handle both transformation of the existing objects (via ˆ w t − 1 ) and synthesis of the emerging objects (via h t ).
Revised: Unlike models synthesizing future frames by transforming the context frames x C (Villegas et al., 2019; 2017b; Yan et al., 2018; Tulyakov et al., 2018), this model is appropriate to synthesize the long-term future since it can handle both transformation of the existing objects (via ˆ w t − 1 ) and synthesis of the emerging objects (via h t ).

Original: To ensure both frame-level and video-level generation quality, the video synthesis network F is trained against a conditional image discriminator D I and a conditional video discriminator D V through adversarial learning by
Revised: To ensure both frame-level and video-level generation quality, the video synthesis network F is trained against a conditional image discriminator D I and a conditional video discriminator D V through adversarial learning by

Original: [Equation]
Revised: [Equation]

Original: where L I ( F, D I ) and L V ( F, D V ) are frame-level and video-level adversarial losses, respectively.
Revised: where L I ( F, D I ) and L V ( F, D V ) are frame-level and video-level adversarial losses, respectively.

Original: For efﬁcient training, we follow (Wang et al., 2018) to adopt sampling operators
Revised: For efﬁcient training, we follow (Wang et al., 2018) to adopt sampling operators

Original: φ I ( x ≤ T , s ≤ T ) =
Revised: φ I ( x ≤ T , s ≤ T ) =

Original: ( x i , s i ) and φ V ( w <T , x ≤ T , s ≤ T ) =
Revised: ( x i , s i ) and φ V ( w <T , x ≤ T , s ≤ T ) =

Original: ( w i
Revised: ( w i

Original: [Equation]
Revised: [Equation]

Original: 2: i [Equation] τ (cid:48) , x i
Revised: 2: i − τ (cid:48) , x i

Original: [Equation]
Revised: [Equation]

Original: 1: i
Revised: 1: i

Original: [Equation]
Revised: [Equation]

Original: τ (cid:48) , s i
Revised: τ (cid:48) , s i

Original: [Equation]
Revised: [Equation]

Original: 1: i [Equation] τ (cid:48) ) for frame and video-level ad- versarial learning objectives, respectively, where i is an integer sampled from U (1 , T ) and U ( τ (cid:48) + 1 , T + 1) for frame sampling operator φ I and video sampling operator φ V , respectively.
Revised: 1: i − τ (cid:48) ) for frame and video-level ad- versarial learning objectives, respectively, where i is an integer sampled from U (1 , T ) and U ( τ (cid:48) + 1 , T + 1) for frame sampling operator φ I and video sampling operator φ V , respectively.

Original: Then the ﬁnal learning objective is formulated by
Revised: Then the ﬁnal learning objective is formulated by

Original: [Equation]
Revised: [Equation]

Original: During training, the model is trained to estimate a frame sequence using the ground-truth structures, while we use the predicted structures by the structure generator duringthe testing. 
Revised: During training, the model is trained to estimate a frame sequence using the ground-truth structures, while we use the predicted structures by the structure generator during testing. See Appendix C. for implementation details.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The task of predicting future video frames has been extensively studied over the past few years (Villegas et al., 2019; Denton & Fergus, 2018; Lee et al., 2018; Babaeizadeh et al., 2018; Vondrick et al., 2016;
Revised: The task of predicting future video frames has been extensively studied over the past few years (Villegas et al., 2019; Denton & Fergus, 2018; Lee et al., 2018; Babaeizadeh et al., 2018; Vondrick et al., 2016;

Original: Finn et al., 2016; Clark et al., 2019;
Revised: Finn et al., 2016; Clark et al., 2019;

Original: Castrejon et al., 2019).
Revised: Castrejon et al., 2019).

Original: Early approaches focus on modeling simple and deterministic dynamics using regression loss (Srivastava et al., 2015; Ranzato et al., 2014) or predictive coding (Lotter et al., 2017).
Revised: Early approaches focus on modeling simple and deterministic dynamics using regression loss (Srivastava et al., 2015; Ranzato et al., 2014) or predictive coding (Lotter et al., 2017).

Original: However, such deterministic models may not be appropriate for modeling stochastic variations in real-world videos.
Revised: However, such deterministic models may not be appropriate for modeling stochastic variations in real-world videos.

Original: Recently, deep generative models have been employed to model dynamics in complex videos.
Revised: Recently, deep generative models have been employed to model dynamics in complex videos.

Original: Babaeizadeh et al.
Revised: Babaeizadeh et al.

Original: (2018) proposed a variational approach for modeling stochasticity in a sequence.
Revised: (2018) proposed a variational approach for modeling stochasticity in a sequence.

Original: Denton & Fergus (2018) incorporated more ﬂexible frame-wise inference models.
Revised: Denton & Fergus (2018) incorporated more ﬂexible frame-wise inference models.

Original: The prediction quality of the variational models has been further improved by employing rich structure in latent variables (Castrejon et al., 2019), maximally increasing the network parameters (Villegas et al., 2019), or incorporating adversarial loss (Vondrick et al., 2016;
Revised: The prediction quality of the variational models has been further improved by employing rich structure in latent variables (Castrejon et al., 2019), maximally increasing the network parameters (Villegas et al., 2019), or incorporating adversarial loss (Vondrick et al., 2016;

Original: Clark et al., 2019; Tulyakov et al., 2018; Villegas et al., 2017a).
Revised: Clark et al., 2019; Tulyakov et al., 2018; Villegas et al., 2017a).

Original: Despite their success, these approaches still fail to generate long-term videos and the common artifacts were losing object structures, switching object categories, etc. .
Revised: Despite their success, these approaches still fail to generate long-term videos and the common artifacts were losing object structures, switching object categories, etc .

Original: Our approach addresses these issues by explicitly predicting dynamics in a low-dimensional label map using a hierarchical model.
Revised: Our approach addresses these issues by explicitly predicting dynamics in a low-dimensional label map using a hierarchical model.

Original: Hierarchical models studied in the past for video prediction were usually in speciﬁc video domains (Villegas et al., 2017b; Wichers et al., 2018;
Revised: Hierarchical models studied in the past for video prediction were usually in speciﬁc video domains (Villegas et al., 2017b; Wichers et al., 2018;

Original: Liang et al., 2017; Yan et al., 2018; Walker et al., 2017; Minderer et al., 2019).
Revised: Liang et al., 2017; Yan et al., 2018; Walker et al., 2017; Minderer et al., 2019).

Original: Villegas et al.
Revised: Villegas et al.

Original: (2017b) employed LSTMs to predict human body joints and visual analogy making to create textures, which is extended by Wichers et al.
Revised: (2017b) employed LSTMs to predict human body joints and visual analogy making to create textures, which is extended by Wichers et al.

Original: (2018) and Minderer et al.
Revised: (2018) and Minderer et al.

Original: to unsupervised approaches.
Revised: to unsupervised approaches.

Original: Other approaches employed sequential VAE (Yan et al., 2018) or GAN (Yang et al., 2018) to predict the human body posture.
Revised: Other approaches employed sequential VAE (Yan et al., 2018) or GAN (Yang et al., 2018) to predict the human body posture.

Original: However, these approaches are designed speciﬁcally for certain objects, and evaluated under simple videos containing only a single moving object.
Revised: However, these approaches are designed speciﬁcally for certain objects, and evaluated under simple videos containing only a single moving object.

Original: Also, they mostly utilized deterministic model to learn structure-level dynamics.
Revised: Also, they mostly utilized deterministic model to learn structure-level dynamics.

Original: These simpliﬁed assumptions on video content and dynamics limit their application to real-world videos.
Revised: These simpliﬁed assumptions on video content and dynamics limit their application to real-world videos.

Original: We propose to resolve these limitations by using dense semantic label maps as universal representation and stochastic sequential estimator for modeling video dynamics.
Revised: We propose to resolve these limitations by using dense semantic label maps as universal representation and stochastic sequential estimator for modeling video dynamics.

Original: E XPERIMENTS

Original: In this section, we present our results on three challenging video datasets.

Original: Full videos are available at anonymous website: https://bit.ly/2EyDSem

Original:   4. E VALUATION M ETRICS
Revised:  E XPERIMENTS 4. E VALUATION M ETRICS

Original: Below we describe the evaluation metrics used in the experiment.
Revised: Below we describe the evaluation metrics used in the experiment.

Original: Unless otherwise speciﬁed, we use 64 × 64 images for all quantitative evaluation for fair comparison to the existing works.
Revised: Unless otherwise speciﬁed, we use 64 × 64 images for all quantitative evaluation for fair comparison to the existing works.

Original:  Short-term prediction We employ three conventional metrics in literature to evaluate the prediction performance on short-term videos ( i.e., less than 50 frames): VGG cosine similarity (CSIM) measuring the frame-wise perceptual similarity using VGG features, mean Intersection-over-Union (mIoU) measuring the structure-level similarity using the label maps extracted by the pre-trained segmentation network (Zhu et al., 2019) and Fr´echet Video Distance (FVD) (Unterthiner et al., 2018) measuring a Fr´echet distance between the ground-truth videos and the generated ones in a video representation space.
Revised: Full videos are available on the anonymous website: https://bit.ly/2EyDSem. Short-term prediction We employ three conventional metrics in literature to evaluate the prediction performance on short-term videos ( i.e., less than 50 frames): VGG cosine similarity (CSIM) measuring the frame-wise perceptual similarity using VGG features, mean Intersection-over-Union (mIoU) measuring the structure-level similarity using the label maps extracted by the pre-trained segmentation network (Zhu et al., 2019) and Fr´echet Video Distance (FVD) (Unterthiner et al., 2018) measuring a Fr´echet distance between the ground-truth videos and the generated ones in a video representation space.

Original: Detailed evaluation protocols are described in Appendix A.4.
Revised: Detailed evaluation protocols are described in Appendix A.

Original: Long-term prediction Compared to short-term prediction, evaluating prediction quality of arbitrary long-term video is challenging for a number of reasons.
Revised: Long-term prediction Compared to short-term prediction, evaluating prediction quality of arbitrary long-term video is challenging for a number of reasons.

Original: First, the ground-truth videos are seldom available at this scale, making it impossible to adopt metrics based onthe frame-wise comparison ( e.g., , CSIM, mIoU).
Revised: First, the ground-truth videos are seldom available at this scale, making it impossible to adopt metrics based on frame-wise comparison ( e.g., , CSIM, mIoU).

Original: Second, the uncertainty of future prediction increases exponentially with time, making it intractable to employ density-based metrics ( e.g., , FVD) as it requires exponentially many samples.
Revised: Second, the uncertainty of future prediction increases exponentially with time, making it intractable to employ density-based metrics ( e.g., , FVD) as it requires exponentially many samples.

Original: Since our goal is to demonstrate video prediction at the scale of hundreds to thousands of frames, we introduce a novel metric based on FVD, which enables evaluation of temporal and frame-level  quality without ground-truth sequences and allows tractable evaluation through time.
Revised: Since our goal is to demonstrate video prediction at the scale of hundreds to thousands of frames, we introduce a novel metric based on FVD, which enables evaluation of temporal and frame-level synthesis quality without ground-truth sequences and allows tractable evaluation through time.

Original: Speciﬁcally, we introduce a shot-level video-quality evaluation metric as
Revised: Speciﬁcally, we introduce a shot-level video-quality evaluation metric as

Original: [Equation]
Revised: [Equation]

Original: which computes a FVD between the ground-truth shots
Revised: which computes a FVD between the ground-truth shots

Original: X ω
Revised: X ω

Original: = { X 1 ω , · · · , X L ω } and the predictions ˆ X t : t + ω − 1 =
Revised: = { X 1 ω , · · · , X Lω } and the predictions ˆ X t : [Equation]

Original:  { ˆ X 1 t : t + ω − 1 , · · · , ˆ X Mt : t + ω − 1 } in a sliding window manner, where L denotes the total number of overlapping shots in  training video and M denotes the number of predicted shots.
Revised: + ω − 1 = { ˆ X 1 t : t + ω − 1 , · · · , ˆ X M t : t + ω − 1 } in a sliding window manner, where L denotes the total number of overlapping shots in the training video and M denotes the number of predicted shots.

Original: The shot-wise FVD then allows us to evaluate the  qualityof generated frames and their motions in a short interval deﬁned by ω through time . 
Revised: The shot-wise FVD evaluates the synthesis quality in a short interval deﬁned by ω through time . We show in the experiment that it is indeed aligned well with human perception.

Original: We also compute Inception Score (Salimans et al., 2016) for frame-level quality evaluation, which does not require ground-truths thus appropriate for longterm evaluation.
Revised: We also compute Inception Score (Salimans et al., 2016) for frame-level quality evaluation, which does not require ground-truths thus appropriate for long-term evaluation.

Original: Please ﬁnd Appendix A.4 for more details.
Revised: Please ﬁnd Appendix A for more details.

Original: Human evaluation We conduct  user study by Amazon Mechnical Turnk (AMT).
Revised: Human evaluation We conduct a user study on Amazon Mechanical Turk (AMT).

Original: To evaluate theprediction quality in different time scales, we presented  videos  predictedup to 5, 300, 450 frames by different methods and ask users to choose the best.
Revised: To evaluate the quality at different time scales, we presented 5 second (50 frames) videos extracted at 1, 250, 400th predicted frames for all methods and counted their chosen ratio as the best.

Original: Please ﬁnd Appendix A.4 for more details.
Revised: More details are in Appendix A.

Original: 4. R ESULTS ON H UMAN D ANCING S EQUENCES
Revised: 4. R ESULTS ON H UMAN D ANCING S EQUENCES

Original: We ﬁrst present our results on human dancing videos collected from  Web (Wang et al., 2018).
Revised: We ﬁrst present our results on human dancing videos collected from the Web (Wang et al., 2018).

Original: Baselines We compare our method withthe two state-of-the-art video prediction models.
Revised: Baselines We compare our method with two state-of-the-art video prediction models.

Original: SVGextend (Villegas et al., 2019) is directly operating on RGB frames via  stochastic estimator, and served as our baseline for  non-hierarchical model.Following Villegas et al.
Revised: SVGextend (Villegas et al., 2019) is directly operating on RGB frames via a stochastic estimator and serves as our baseline for a non-hierarchical model.

Original: (2019), we employ the largest model with maximum parameters for fair comparison.
Revised: Following the paper, we employ the largest model with maximum parameters for fair comparison.

Original: We employ Villegas et al.
Revised: We employ Villegas et al.

Original: (2017b)
Revised: (2017b)

Original: as a hierarchical model designed speciﬁcally for long-term prediction of human motion usingthe pose (Newell et al., 2016).
Revised: as a hierarchical model designed speciﬁcally for long-term prediction of human motion using pose (Newell et al., 2016).

Original: All models are trained to predict 40 future frames given 5 context frames.
Revised: All models are trained to predict 40 future frames given 5 context frames.

Original: [Table]
Revised: [Table]

Original: Short-term prediction results We evaluate the short-term prediction performance by synthesizing 50 future frames given the 5 context frames and comparing them to groundtruths.
Revised: Short-term prediction results We evaluate the short-term prediction performance by synthesizing 50 future frames given the 5 context frames and comparing them to groundtruths.

Original: Table 1 summarizes the quantitative evaluation results (see Figure ?? for qualitative results).
Revised: Table 1 summarizes the quantitative evaluation results (see Figure A for qualitative results).

Original: Even in a short prediction interval, our method generates substantially higher quality samples than the compared methods in terms of modeling appearance (CSIM), structure (mIoU), and motion (FVD).
Revised: Even in a short prediction interval, our method generates substantially higher quality samples than the baselines in terms of modeling appearance (CSIM), structure (mIoU), and motion (FVD).

Original: We observe that it is because the dancing sequences contain rapid and complex variations in dynamics, making prediction task particularly challenging.
Revised: This is because the dancing sequences contain rapid and complex variations in dynamics, making prediction task particularly challenging.

Original: We discuss more detailed analysis across methods in a long-term prediction task below.
Revised: We discuss more detailed analysis across methods in a long-term prediction task below.

Original: Long-term prediction results We evaluate 500 frame prediction results based on shot-wise (FVD) and frame-wise (Inception score).
Revised: Long-term prediction results We evaluate 500 frame prediction results based on shot-wise FVD, frame-wise Inception score, and human evaluation.

Original: Figure 2 and 3 summarize the quantitative and qualitative comparisons, respectively.
Revised: Figure 2 and 3 summarize the quantitative and qualitative comparisons, respectively.

Original: We notice that the SVG-extend fails to model complex dynamics in dancing, as both the Inception score and FVD are constantly getting worse through time.
Revised: We notice that the SVG-extend fails to model complex dynamics in dancing even early in prediction.

Original: It is because the dancing motions involve fast transition and frequent self-occlusions, escalating catastrophic error propagation through time.
Revised: This is because the dancing motions involve fast transition and frequent self-occlusions, resulting in catastrophic error propagation through time.

Original: On the other hand, as shown in the Inception score, (Villegas et al., 2017b) produces much reasonable future frames as it exploits human body joints for generation.
Revised: On the other hand, as shown in the Inception score, Villegas et al. (2017b) produces much reasonable future frames as it exploits human body joints for generation.

Original: However, the deterministic LSTM module is not strong enough to capture complex dancing motions, and ends up generating static postures in the long term that leads to very high FVD scores.
Revised: However, the deterministic LSTM module is not strong enough to capture complex dancing motions, and ends up generating static postures in the long term that leads to very high FVD scores.

Original: In contrast, we observe that our method generates both realistic frames and convincing motions,  context t=1~ t=250~ t=490~
Revised: In contrast, we observe that our method generates both realistic frames and convincing motions, leading to stable Inception and FVD scores [Figure] context t=1~ t=250~ t=490~

Original: small big
Revised: small big

Original: O u r s
Revised: O u r s

Original: [Figure]
Revised: [Figure]

Original: V i ll e g a s e t a l .t=
Revised: V i ll e g a s e t a l .

Original: [Figure]
Revised: [Figure]

Original: O u r s t=
Revised: O u r s t

Original: [Figure]
Revised: [Figure]

Original: 500 t=251~
Revised: t=251~

Original: V i ll e g a s e t a l .
Revised: V i ll e g a s e t a l .

Original: S V G t=1~
Revised: S V G t=1~

Original: S VG - e x t e n d
Revised: S V

Original: [Figure]
Revised: [Figure]

Original: leading to stable Inception and FVD scores in a long-term future.
Revised: - e x t e n d in the long-term future.

Original: Also, contrary to Villegas et al. (2017b) thatproduces a deterministic future, our model can synthesize multiple futures based on the stochastic sequence estimator for structure.
Revised: The human evaluation (Figure 2(c)) also shows the consistent results that our method outperforms all methods from the short- to long-term prediction.

Original: Figure A illustrates that the generated frames by our method are covering diverse and interesting motions. When our method is applied to predict 2500+ frames in 128 × 128 resolution (Figure 4), we observe that it generates future frames that are convincing and involve diverse dance motions without noticeable quality degradation through time. 
Revised: When our method is applied to predict up to 2040 frames in 128 × 128 resolution (see Figure C in the appendix), we observe that it generates future frames that are convincing and involve diverse dance motions without noticeable quality degradation through time. In addition to the synthesis quality, Figure B illustrates that our method can generate diverse and interesting motions with the stochastic structure estimator.

Original: 4. R ESULTS ON KITTI B ENCHMARK
Revised: 4. R ESULTS ON KITTI B ENCHMARK

Original: Baselines In addition to SVG-extend, we employ the future segmentation model (Bhattacharyya et al., 2019)
Revised: Baselines In addition to SVG-extend, we employ the future segmentation model (Bhattacharyya et al., 2019)

Original: (Bayes-WD-SL) predicting the dense label map of future as a strong baseline for hierar- [Figure] chical model.
Revised: (Bayes-WD-SL) that predicts future dense label maps as a strong baseline for hierarchical model.

Original: Since it generates only the label maps, we employ the same image generator with ours to produce RGB frames from the predicted label maps.
Revised: Since it generates only the label maps, we employ the same image generator with ours to produce RGB frames from the predicted label maps.

Original: All models are trained to predict 15 future frames given 5 context frames.
Revised: All models are trained to predict 15 future frames given 5 context frames.

Original: Please refer to the appendix for the detailed architecture settings.
Revised: Please refer to the appendix for the detailed architecture settings.

Original: [Figure]
Revised: [Figure]

Original:  Short-term prediction results We evaluate the short-term prediction performance by synthesizing 50future frames given the 5 context frames and comparing them to groundtruths.
Revised: [Table] Short-term prediction results Similar to the previous section, we evaluate the short-term prediction quality by synthesizing 50 frames given the 5 frames.

Original: Table 2 summarize the quantitativeevaluation results (see Figure C for qualitative results).
Revised: Table 2 summarize the quantitative results (see Figure D for qualitative analysis).

Original: We observe that SVG-extend performs worse in terms of structural accuracy (mIoU) and motion (FVD), as it loses the object structure and generates arbitrary pixel dynamics.
Revised: We observe that SVG-extend performs worse in terms of structural accuracy (mIoU) and motion (FVD), as it loses the object structure and generates arbitrary pixel dynamics.

Original: The Bayes-WD-SL performs better in terms of mIoU, as it conditions the frame prediction with the structure estimation.
Revised: The Bayes-WD-SL performs better in terms of mIoU, as it conditions the frame prediction with the structure estimation.

Original: However, as shown high FVD score, it fails to model temporal variations in the structure, largely due to its feedforward architecture.
Revised: However, as shown high FVD score, it fails to model temporal variations in the structure, largely due to its feedforward architecture.

Revised: This unstructured motion leads to disastrous failure in a long-term prediction, as we will discuss later.

Original: In contrary, our method generates semantically accurate motions and structures, and substantially outperforms the others in all metrics.
Revised: In contrary, our method generates semantically accurate motions and structures, and substantially outperforms the others in all metrics.

Original: C o n t e x t t = 5~ 5 [-3t = 10 ~ 1 
-]
Revised: Long-term prediction results Figure 4 and 5 [-3t = 10 ~ 1 
-]summarize the quantitative and qualitative comparisons, respectively. In all metrics, our method outperforms the other baselines with substantial margins, showing that our method synthesizes both high-quality frames and motion. We notice that [Figure]

Original: [Figure]
Revised: [Figure]

Original:  t = 1 5 00 ~ 1 5 3 5 t = 2 000 ~2 0 

Revised: C o n t e x t t = 5~ 5 3t =000 ~ 1 0 t = 100 ~ 1 5 t =000 ~

Original: [Figure][Table] Long-term prediction results Figure 5 and Figure 6 summarize the quantitative and qualitative comparison results, respectively. In both measures, our method outperforms the other baselines with substantial margins, showing that our method synthesizes both reasonable motions and high-quality frames.
Revised: [Figure]

Original: We notice that SVG-extend simulates arbitrary pixel motions, resulting in relatively constant Inception and shot-wise FVD scores through time.
Revised: SVG-extend simulates arbitrary pixel motions, resulting in relatively constant Inception and shotwise FVD scores through time.

Original: However, these unstructured motions lead to rapid destruction in recognizable concepts in the synthesized frames (Figure 6).
Revised: However, these unstructured motions lead to rapid destruction in recognizable concepts in the synthesized frames (Figure 5).

Original: We also observe similar failure in future segmentation baseline (Bayes-WD-SL), which converges to monotonic label maps due to severe error propagation.
Revised: The hierarchical model (Bayes-WDSL) generates more convincing frames in a short-term as shown in lower shot-wise FVD but fails to extrapolate in long-term. Importantly, such trends in shot-wise FVD (Figure 4(b)) aligns well with human evaluation results (Figure 4(c)), showing that it is appropriate metric for evaluation of long-term prediction.

Original: Interestingly, we observe that the Inception Score of our method increases through time.
Revised: Interestingly, we observe that the Inception Score of our method increases through time.

Original: It is because the long-term prediction by our method often leads to scenes with simple and typical structures, such as a highway, where the image generator can produce more high-quality frames than complicated scenes appearing early in the test videos.
Revised: It is because the long-term prediction by our method often leads to scenes with simple and typical structures, such as a highway, where the image generator can produce more high-quality frames than complicated scenes appearing early in the test videos.

Original: Note that such transition of the scene is still reasonable as it is frequently observed in the training data.
Revised: Note that such transition of the scene is still reasonable as it is frequently observed in the training data.

Original: Nonetheless, we observe that our method generates reasonable sequences through time, and the qualityof the synthesized frames is maintained in a reasonable range even in avery distant future.
Revised: Nonetheless, we observe that our method generates reasonable sequences through time while maintaining its quality in a reasonable range even in a distant future.

Original: Finally, Figure 7 illustrates the prediction results over 2500 future frames.
Revised: Finally, Figure 6 illustrates the prediction results overfuture frames.

Original: We observe that the generated sequences are reasonable in both structure and motion, and capture interesting translations of the scenes through time ( e.g., from suburban to rural areas). 
Revised: We observe that the generated sequences are reasonable in both structure and motion, and capture interesting translations of the scenes through time ( e.g., from suburban to rural areas). [Table]

Original: 4. R ESULTS ON C ITYSCAPES DATASET
Revised: 4. R ESULTS ON C ITYSCAPES DATASET

Original: To further evaluate the quality of structure prediction, we compare our structure generator with existing future segmentation methods that directly predict the segmentation map of the future frames.
Revised: To further evaluate the quality of structure prediction, we compare our structure generator with existing future segmentation methods that directly predict the segmentation map of the future frames.

Original: Baselines We compare our method with S2S (Luc et al., 2017) and F2F (Luc et al., 2018), which are the state-of-the-arts in the future segmentation literature.
Revised: Baselines We compare our method with S2S (Luc et al., 2017) and F2F (Luc et al., 2018), which are the state-of-the-arts in the future segmentation literature.

Original: S2S is a deterministic model based on fully-convolutional network that predicts future semantic label maps in a multi-scale and autoregressive manner.
Revised: S2S is a deterministic model based on fully-convolutional network that predicts future semantic label maps in a multi-scale and autoregressive manner.

Original: F2F is an extension of S2S to future instance segmentation task by predicting the high-level feature maps of Mask R-CNN (He et al., 2017).
Revised: F2F is an extension of S2S to future instance segmentation task by predicting the high-level feature maps of Mask R-CNN (He et al., 2017).

Original: Similar to KITTI, we use the label maps extracted by the pre-trained semantic segmentation network (Zhu et al., 2019) for the training.
Revised: Similar to KITTI, we use the label maps extracted by the pre-trained semantic segmentation network (Zhu et al., 2019) for the training.

Original: Evaluation We follow the standard evaluation protocols in the literature (Luc et al., 2017; 2018) for quantitative and qualitative comparisons.
Revised: Evaluation We follow the standard evaluation protocols in the literature (Luc et al., 2017; 2018) for quantitative and qualitative comparisons.

Original: For each validation sequence, each model is provided with 4 contexts, and produces up to 29th frame.
Revised: For each validation sequence, each model is provided with 4 contexts, and produces up to 29th frame.

Original: Then, we compute mIoUs between ground-truths and predictions at 20th time-step.
Revised: Then, we compute mIoUs between ground-truths and predictions at 20th time-step.

Original: For fair comparisons with the future instance segmentation model ( i.e., F2F), we follow Luc et al.
Revised: For fair comparisons with the future instance segmentation model ( i.e., F2F), we follow Luc et al.

Original: and measure mIoUs only for moving objects.
Revised: and measure mIoUs only for moving objects.

Original: For S2S and
Revised: For S2S and

Original: F2F, we use publicly available pre-trained models provided by the authors.
Revised: F2F, we use publicly available pre-trained models provided by the authors.

Original: Results Table 3 presents the quantitative evaluation results (See Figure E for qualitative results).
Revised: Results Table 3 presents the quantitative evaluation results (See Figure G for qualitative results).

Original: As it shows, the two baselines produce blurry predictions, resulting in inaccurate structures and multiple semantics in a single structure.
Revised: The two baselines produce blurry predictions, resulting in inaccurate structures and mislabelings.

Original: These problems have been widely observed in the literature and  attributed to the inabilityof deterministic models to handle stochasticityin data (Denton & Fergus, 2018).
Revised: These problems have been widely observed in the literature and are attributed to the inability to handle stochasticity (Denton & Fergus, 2018).

Original: On the other hand, our method outperforms the two baselines in all classes as it can handle highly stochastic nature of complex driving scenes.
Revised: Our method outperforms the two baselines in all classes as it can handle highly stochastic nature of complex driving scenes.

Original: Figure F illustrates the  results of our full model including  boundary map and image generator on 256 × 512 resolution videos.
Revised: Figure 7 and Figure H illustrates the long-term prediction results of our full model including the boundary map and image generator on 256 × 512 resolution.

Original: It shows that our method can generate convincing future even in extremely complex and high-resolution videos.
Revised: It shows that our method can generate convincing future even in extremely complex and high-resolution videos.

Original: See Appendix A.3.3 for more detailed discussion.
Revised: See Appendix B.3 for more detailed discussion.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a hierarchical approach for persistent video prediction.
Revised: We proposed a hierarchical approach for persistent video prediction.

Original: We revisit the hierarchical model with stochastic estimation of dense label maps and integration of image generator robust against temporal misprediction in pixels and structures.
Revised: We revisit the hierarchical model with stochastic estimation of dense label maps and integration of image generator robust against temporal misprediction in pixels and structures.

Original: Our experimental results show that a carefully designed hierarchical model can learn to synthesize a very long-term future with convincing structures and dynamics even in complex videos.
Revised: Our experimental results show that a carefully designed hierarchical model can learn to synthesize a very long-term future with convincing structures and dynamics even in complex videos.

Original: By scaling up the video prediction to order-ofmagnitude longer, we believe that our work can help the future research to focus on more challenging problems of learning a long-term dependency, eliminating explicit structure estimation, etc.
Revised: By scaling up the video prediction to order-ofmagnitudes longer, we believe that our work can help the future research to focus on more challenging problems of learning a long-term dependency and eliminating explicit structure estimation.

Original: A PPENDIX A A DDITIONAL EXPERIMENT RESULTS
Revised: A PPENDIX A E VALUATION PROTOCOLS

Revised: Short-term prediction.

Revised: Following the previous works, we evaluate the short-term prediction quality by comparing the ground-truth and predicted frames in test videos.

Revised: We employ three metrics:

Revised: VGG cosine similarity (CSIM), mean Intersection-over-Union (mIoU), and Fr´echet Video Distance (FVD) (Unterthiner et al., 2018).

Revised: The ﬁrst metric measures the frame-wise perceptual similarity using VGG features, while the second one measures the structure-level similarity using the label maps extracted by the pre-trained semantic segmentation network (Zhu et al., 2019).

Revised: To handle stochasticity, we sample 100 predictions for each test clip and report the scores of the best scoring predictions.

Revised: For the video-level evaluation, we adopt FVD that measures a Fr´echet distance between the ground-truth videos and the generated ones in a video representation space.

Revised: We use an Inﬂated 3D Convnet (I3D) (Carreira & Zisserman, 2017) network pre-trained on Kinetics 400 dataset (Carreira & Zisserman, 2017) to obtain representations on videos, and use all 100 predictions to compute

Revised: FVD against ground-truth sequences.

Revised: Long-term prediction.

Revised: To evaluate the shot-wise FVD, we employ the I3D Convnet as the shortterm prediction.

Revised: For the models trained on KITTI dataset, we use a ResNet-18 (He et al., 2016) pre-trained on Places 365 (Zhou et al., 2017) to compute the Inception Score.

Revised: For the models trained on the Human Dancing dataset, we use the Inception V3 (Szegedy et al., 2015) pre-trained on ImageNet (Russakovsky et al., 2015) for measuring the Inception Score.

Revised: Human evaluation.

Revised: We conducted a human evaluation using Amazon Mechanical Turk (AMT).

Revised: For evaluating both the short- and long-term prediction performances, we generated 500 predicted frames for all compared methods (SVG-Extend, Villegas et al.

Revised: (2019), and ours).

Revised: We extracted 5second clips (50 frames encoded in 10 fps) at the 1st, 250th, and 400th frames, and asked users to rank the videos according to perceived quality and realism of the videos.

Revised: On the KITTI dataset, we conducted the evaluation over the 133 unique sequences present in the validation set.

Revised: On the Human Dancing dataset, we conducted evaluation over 97 videos used for validation.

Revised: Responses for each video was aggregated from 5 independent human evaluators for each clip.

Revised: B M ORE D ETAILS OF E XPERIMENT R ESULTS

Original: This section provides additional experiment details and results that could not be accommodated in the main paper due to space restriction.
Revised: This section provides additional experiment details and results that could not be accommodated in the main paper due to space restriction.

Original: Please ﬁnd our anonymous website to assess full videos for qualitative analysis: https://bit.ly/2EyDSem.
Revised: Please ﬁnd our anonymous website to assess full videos for qualitative analysis: https://bit.ly/2EyDSem.

Original: A. H UMAN D ANCING DATASET A.1. D ATASET
Revised: B. H UMAN D ANCING DATASET B . D ATASET

Original: To evaluate if our model can learn complex and highly structured motion, we used videos of human dancing.
Revised: To evaluate if our model can learn complex and highly structured motion, we used videos of human dancing 2 .

Original: We construct this dataset by crawling a set of videos from  web containing a single person covering various dance moves.
Revised: We construct this dataset by crawling a set of videos from the web containing a single person covering various dance moves.

Original: Following the pre-processings used in Wang et al.
Revised: Following the pre-processing used in Wang et al.

Original: (2018), we crop the center of the video and resize it to square frames.
Revised: (2018), we crop the center of the video and resize it to square frames.

Original: We collect approximately 240 videos in total for training.
Revised: We collect approximately 240 videos in total for training.

Original: For structure representation, we use the body parts segmentation obtained by
Revised: For structure representation, we use the body parts segmentation obtained by

Original: DensePose (Alp G ¨ uler et al., 2018).
Revised: DensePose (Alp G¨uler et al., 2018).

Original: We subsample each sequence by a factor of two.
Revised: We sub-sample each sequence by a factor of two.

Original: A.1. I MPLEMENTATION D ETAILS
Revised: B.1. I MPLEMENTATION D ETAILS

Original: SVG-extend (Villegas et al., 2019)
Revised: SVG-extend (Villegas et al., 2019).

Original: We use the modiﬁed version of SVG (Denton & Fergus (2018)) by Villegas et al. (2019), which we refer to as SVG-extend throughout the paper. The two main differences between SVG and SVG-extend are two-fold: (1) SVG-extend adopts the shallower auto-encoder architecture than thatof SVG. (2) SVG-extend replaces Fully-Connected-LSTM with
Revised: We use the same architectural and hyperparameter settings described in Section C.1 except that SVG-extend in this case is trained to model pixel-level RGB

Original: Convolutional-LSTM Shi et al. These two modiﬁcations are to increase the capacity of the architecture. Since the detailed architectural conﬁguration of SVG-extend is not described in Villegas et al. (2019), we summarize our reproduced version of this framework in Table A, B, C, and D. On the 64 × 64 image resolution, we use the same architecture described in the tables, while, on 128 ×image resolution, we simply stack more convolutional blocks (non-linear convolutional layers followed by upsampling or downsampling layers) in the Encoder and Decoder networks so that the shape of h t , z t , g t is kept to R 128 × 8 × 8 agnostic to the image resolutions.
Revised: We videos sequences directly as an unsupervised non-hierarchical baseline, unlike ours that works on semantic segmentation sequences as sequential dense structure generator.

Original: For hyperparameters, we use C = 5 ; we set β = 0 . 0005 ; we use ADAM optimizer Kingma & Ba (2015) with the learning rate of 0 . 0001 and ( β 1 , β 2 ) =

Original:  et al.
Revised: Villegas et al.

Original: (2017b)
Revised: (2017b).

Revised: This method also adopts a hierarchical approach that leverages human keypoints as additional supervision for predicting future video frames.

Revised: Speciﬁcally, this method consists of ﬁve modules, namely (1) the Hourglass network (Newell et al., 2016) that estimates human keypoints from pixel-level images, (2) a single layer deterministic encoder-decoder LSTM withhidden units and tanh activations that predicts a sequence of future human keypoints given a few observed ones, (3) an image generator that synthesizes pixel-level image frames given the predicted and observed keypoints based on visual-analogy scheme, (4) an image discriminator for adversarial learning, and (5) the AlexNet (Krizhevsky et al., 2012) pre-trained on ImageNet (Russakovsky et al., 2015) that is used for perceptual loss (Johnson et al., 2016).

Revised: For training this model on our human dancing dataset, we directly follow the setting used in the paper (Villegas et al., 2017b), and use the publicly available code 3 provided by the authors.

Original: A.1. Q UALITATIVE R ESULTS
Revised: B.1. Q UALITATIVE R ESULTS

Original: Short-term prediction [TODO:add the qualitative results on short-term prediction and discussions here.]
Revised: Short-term prediction. We show in Figure A the qualitative comparisons of short-term predictions across models, which corresponds to the Table 1 in the main paper. As shown in the ﬁgure, SVG-extend starts to fail in producing plausible samples in a very short-term ( t = 8 ∼ ), which can be attributed to its implicit modeling of spatio-temporal variations of structures and appearances.

Revised: Villegas et al.

Original:  Diversity of samples Figure
Revised: (2017b) improves the quality of both structures and appearances as well as extrapolation through time via the explicit modeling of structures followed by the translation of predicted structures into pixel-level frames in a hierarchical manner. However, it still fails to produce realistic pixel-level frames due to the adoption of (1) coarse-grained human landmarks and deterministic model which are insufﬁcient to fully describe the ﬁne details of stochastic and structured motions, and (2) image synthesis performed by transforming the last observed frames into the future, which may not be appropriate when there exist large appearance changes due to self-occlusion and movement. On the other hand, our approach can produce high-ﬁdelity predictions with plausible motions thanks to the stochastic and comprehensive estimation of scene dynamics under the semantic segmentation space. Diversity of samples.

Original: A illustrates the multiple prediction results conditioned on the same context frames.
Revised: Figure B illustrates the multiple prediction results conditioned on the same context frames.

Original: We observe that our model produces diverse and plausible future frames, where such stochasticity mainly stems from the structure generator.
Revised: We observe that our model produces diverse and plausible future frames, where such stochasticity mainly stems from the structure generator.

Original: Long-term prediction results Figure B illustrates the prediction results on a very long-term future (over 2500 frames), which corresponds to the Figure 4 in the main paper. We observe that our method generates convincing frames without particular quality degradation and error propagation over a very long-term future.
Revised: Long-term prediction results. Figure C illustrates the prediction results on a very long-term future (over 2500 frames).We observe that our method generates convincing frames without particular quality degradation and error propagation over a very long-term future.

Original: We also observe that the predicted frames contain interesting dance motions, such as the ones with horizontal translation ( t = 1000 − 1040 ), dynamics on legs ( t = 500 − 540 ) and arms ( t = 2000 − 2040 ), etc .{+etc. t=43 t= t= t= t= t= t= t= t=43 t=1 t=
 t=15 t=22 t=29 t= Ground-Truth+}
Revised: We also observe that the predicted frames contain interesting dance motions, such as the ones with horizontal translation ( t = 1000 ∼ 1040 ), dynamics on legs ( t = 500 ∼ 540 ) and arms ( t = 2000 ∼ 2040 ), {+etc. t=43 t= t= t= t= t= t= t= t=43 t=1 t=
 t=15 t=22 t=29 t= Ground-Truth+}

Original: [Figure]
Revised: SVG-Extend Villegas et al. Ours t= Ground-Truth

Revised: SVG-Extend

Revised: Villegas et al.

Revised: Ours

Revised: Ground-Truth

Revised: Ours

Revised: Ours

Revised: Villegas et al.

Revised: SVG-Extend

Revised: SVG-Extend

Revised: Villegas et al.

Revised: Ground-Truth ㅋㅋ 내가 할게 given the same context frames, we sample 100 predictions for each model and show their best predictions in terms of the cosine similarity metric best

Revised: , we show the semantic label maps predicted by Densepose (Alp G¨uler et al., 2018).

Revised: [Figure]

Original: ··· t
Revised: ··· t=

Original: [Figure]
Revised: [Figure]

Original: t= t= t=
Revised: t= t= t=

Original: [Figure]
Revised: [Figure]

Original: Ground-Truth t=6 t=12 ··· t=18 t= t= t=
Revised: Ground-Truth t=6 t=12 ··· t=18 t=48 t= t=

Original: Sample
Revised: Sample

Original: Sample Sample
Revised: Sample Sample

Original: Ground-Truth
Revised: Ground-Truth

Original: Sample Sample
Revised: Sample Sample

Original: Sample···
Revised: Sample···

Original: [-C o n t e x t t = ~ t = ~ 15t =5~55t = ~ 
 C o n t e x t t =  ~3t = ~ 13t =  ~ 13t =  ~
 t =5~
-]
Revised: [-C o n t e x t t = ~ t = ~ 15t =5~55t = ~ 
 C o n t e x t t =  ~3t = ~ 13t =  ~ 13t =  ~
 t =5~
-][Figure]

Original: C o n t e x t t = ~  t = ~ 15t =5~55t =  ~ [-25t =5~
-]{+13t =  ~0 
+}
Revised: C o n t e x t t = ~ 5 t = ~ 13t =  ~ [-25t =5~
-]{+13t =  ~0 
+}

Original: [Figure]
Revised: [Figure]

Original: t =5~ 

Revised: t =5~

Original: A. KITTI BENCHMARK A.2. D ATASET
Revised: B. KITTI BENCHMARK B.2. D ATASET

Original: An ability to cope with partial observability and high stochasticity is highly important in the video prediction task.
Revised: An ability to cope with partial observability and high stochasticity is highly important in the video prediction task.

Original: To evaluate a model under this criterion, we adopt KITTI dataset (Geiger et al., 2013) consisting of driving scenes captured by a front-view camera in residential and rural areas.
Revised: To evaluate a model under this criterion, we adopt KITTI dataset (Geiger et al., 2013) consisting of driving scenes captured by a front-view camera in residential and rural areas.

Original: Since the segmentation labels are available for only a small subset of frames in this dataset ( frames in total), we extract the labels using the semantic segmentation model (Zhu et al., 2019) pre-trained on still images (Cordts et al., 2016; Geiger et al., 2013) for both training (the entire sequences) and testing (only context frames).
Revised: Since the segmentation labels are available for only a small subset of frames in this dataset ( frames in total), we extract the labels using the semantic segmentation model (Zhu et al., 2019) pre-trained on still images (Cordts et al., 2016; Geiger et al., 2013) for both training (the entire sequences) and testing (only context frames).

Original: We follow the pre-processings and train/test splits used in Lotter et al.
Revised: We follow the pre-processing and train/test splits used in Lotter et al.

Original: (2017), and subsample each sequence by a factor of two.
Revised: (2017), and sub-sample each sequence by a factor of two.

Original: A.2. I MPLEMENTATION D ETAILS
Revised: B.2. I MPLEMENTATION D ETAILS

Original: SVG-extend (Villegas et al., 2019)
Revised: SVG-extend (Villegas et al., 2019).

Original: The architectural details described in A.1.2 still hold in  this  dataset.
Revised: Same as Section B.1.2, we follow the implementation details speciﬁed in Section C.1 for training this model on KITTI Benchmark dataset.

Original: Bayes-WD-SL (Bhattacharyya et al., 2019)
Revised: Bayes-WD-SL (Bhattacharyya et al., 2019).

Revised: This work is a future segmentation method that exploits odometry information as additional supervision and enables stochastic estimation of uncertain futures through the Bayesian approach.

Revised: On top of a ResNet-based deterministic video prediction model (Luc et al., 2017), this work introduces a novel Bayesian formulation to better handle inherent uncertainties when predicting future frames.

Revised: Speciﬁcally, this method introduces (1) weight-dropout (WD) which assumes a ﬁnite set of weights for approximating model space, followed by Bernoulli variational distribution on those ﬁnite set of weights, and (2) synthetic likelihoods (SL) which mitigates the constraint on each model to explain all the data points and thus allows sampled models to be diverse to better deal with the uncertainties.

Revised: For training this model on KITTI Benchmark, we follow the same settings used in the original paper for the experiments on Cityscapes dataset except that we omit odometry supervision for fair comparisons.

Revised: We use the publicly available code.

Original: A.2. Q UALITATIVE R ESULTS
Revised: B.2. Q UALITATIVE R ESULTS

Original: Short-term prediction Figure C illustrates the qualitative comparison for short-term prediction, which correspond to Table 2 in the main paper.
Revised: Short-term prediction. Figure D illustrates the qualitative comparisons for short-term prediction, which correspond to Table 2 in the main paper.

Original: As shown in Figure C, SVG performs very well at the very beginning of the prediction ( t = 1 ∼ 40 ), since it directly exploits the content from the last context frame via temporal skip-connection.
Revised: As shown in Figure D, SVG-extend performs very well at the very beginning of the prediction ( t = 1 ∼ 40 ), since it directly exploits the content from the last context frame via temporal skip-connection.

Original: However, its prediction quality degrades rapidly through time, as it loses the object structure and simulates arbitrary, unstructured pixel-wise motions in a long-term.
Revised: However, its prediction quality degrades rapidly through time, as it loses the object structure and simulates arbitrary, unstructured pixel-wise motions in the long-term.

Original: The future segmentation baseline (Bayes-WD-SL) preserves  much reasonable structure, since its generation is conditioned on the estimated label maps similar to ours.
Revised: The future segmentation baseline (Bayes-WD-SL) preserves a much reasonable structure, since its generation is conditioned on the estimated label maps similar to ours.

Original: However, it fails to model structures ina motion and thus loses such structures at the end.
Revised: However, it fails to model structures in motion and thus loses such structures in the end.

Original: These results demonstrate that both the unsupervised and hierarchical baselines fail to extrapolate to the long-term horizon than the one used during the training.
Revised: These results demonstrate that both the unsupervised and hierarchical baselines fail to extrapolate to the long-term horizon than the one used during the training.

Original: Compared to these baselines, our method generates semantically more accurate motions and structures, and substantially outperforms the others especially in the latter prediction steps.
Revised: Compared to these baselines, our method generates semantically more accurate motions and structures, and substantially outperforms the others especially in the latter prediction steps.

Original: Diversity of samples [TODO:add the qualitative results  and discussions here]
Revised: Diversity of samples. We show diverse prediction results in Figure E, which are conditioned on the same context frames. As can be seen in the ﬁgure, predicted sequences not only are accurate in a near future ( t = 18 ), but also show diverse motion and scene dynamics through time. For example, we can see diverse scene dynamics such as emergence of novel vehicles ( t = 54 ∼ 144 ), as well as transition to the novel scene ( t = 144 ∼ 180 ).

Original: Long-term prediction results Figure D illustrates the long-term prediction results overframes in 25 × 256 resolution, which corresponds tothe Figure 7 in the main paper.
Revised: Long-term prediction results. Figure F illustrates the long-term prediction results over 2, frames in 256 × 256 resolution, which corresponds to Figure 6 in the main paper.

Original: We observe that the generated sequences are reasonable in both structure and motion.
Revised: We observe that the generated sequences are reasonable in both structure and motion.

Original: Interestingly, unrolling the prediction over a very long-term leads to very interesting transitions of the scene, such as translation from suburban ( t < 1035 ) to rural areas ( t > 1500 ), which cannot be captured in a short-term period.
Revised: Interestingly, unrolling the prediction over a very long-term leads to very interesting transitions of the scene, such as translation from suburban ( t < 1035 ) to rural areas ( t > 1500 ), which cannot be captured in a short-term period.

Original: We believe that these results illustrate well why the long-term prediction is important.
Revised: We believe that these results illustrate well why the long-term prediction is important [Figure] t [Figure]

Revised: t= t= t= t= t= t=50 t=

Original: [Figure]
Revised: [Figure]

Original: given the same context frames predictions 
Revised: given the same context frames, we sample 100 predictions for each model and show their best predictions in terms of the cosine similarity metric. (bottom pane) for each best prediction, we show the semantic label maps predicted by pre-trained semantic segmentation network (Zhu et al., 2019). [Figure] [Figure]

Revised: t= t= t

Revised: [Figure]

Revised: [Figure]

Revised: Ground-Truth t=6 t=12 ··· t=18 t=48 t= t

Original: [Figure]
Revised: [Figure]

Original: Context t=500~ t=1000~ t=1500~ t=2000~
Revised: Sample Sample Sample Ground

Revised: [Figure]

Revised: Truth

Revised: Sample

Revised: Sample

Revised: Sample···

Revised: [Figure]

Revised: [Figure]

Revised: Context t=500~ t=1000~ t=1500~ t=2000~

Original: A. C ITYSCAPES DATASET A.3. D ATASET
Revised: B. C ITYSCAPES DATASET B.3. D ATASET

Original: To evaluate the quality of forecasting dense labels by the structure generator, we employ the
Revised: To evaluate the quality of forecasting dense labels by the structure generator, we employ the

Original: Cityscapes (Cordts et al., 2016), a widely-used benchmark in future segmentation tasks.
Revised: Cityscapes (Cordts et al., 2016), a widely-used benchmark in future segmentation tasks.

Original: The dataset consists of 2,975 training, 1,525 testing, and 500 validation sequences where each sequence is 30frame-long and has ground-truth segmentation labels only in the 20th frame.
Revised: The dataset consists of 2,975 training, 1,525 testing, and 500 validation sequences where each sequence is 30frame-long and has ground-truth segmentation labels only in the 20th frame.

Original: Similar to KITTI dataset, we employ the pre-trained segmentation network (Zhu et al., 2019) to generate segmentation labels  for training.
Revised: Similar to KITTI dataset, we employ the pre-trained segmentation network (Zhu et al., 2019) to generate segmentation labels of 128 × 256 image resolution for training our SVG-extend.

Original: Following (Luc et al., 2017; 2018), we subsample frames by a factor of three for each sequence and downsample the frames to 128 × 256 image resolution.
Revised: Following (Luc et al., 2017; 2018), we subsample frames by a factor of three for each sequence and test all models on 128 ×image resolution.

Original: A.3. I MPLEMENTATION D ETAILS
Revised: B.3. I MPLEMENTATION D ETAILS

Original: S2S (Luc et al., 2017)
Revised: S2S (Luc et al., 2017).

Revised: We use the best performing S2S model proposed by the authors, namely S2Sdil, AR, ﬁne-tune , which improves the performance of the single-frame prediction baseline (S2S) by introducing the followings: (1) dilated convolution to enlarge receptive ﬁelds of the model.

Revised: (2) autoregressive ﬁne-tuning that uses the prediction at time t as an input to predict the future frame at time t + 1 during training, which enables the single-frame prediction model to predict an arbitrary number of frames into the future.

Revised: We use a publicly available pre-trained model 4 provided by the authors, which was trained using the predicted semantic segmentation maps by Dilation network (Yu & Koltun, 2016) on 128 × 256 image resolution.

Original: F2F (Luc et al., 2018)
Revised: F2F (Luc et al., 2018).

Revised: This method targets a future instance segmentation task, which extends future segmentation method S2S

Revised: (Luc et al., 2017) by formulating the task of the model as predicting intermediate feature maps of Mask R-CNN (He et al., 2017) on the target future instance segmentation masks.

Revised: We use a publicly available pre-trained model and code 5 provided by the authors, which was ﬁrst trained on MS-COCO dataset (Lin et al., 2014) and then ﬁnetuned on the Cityscapes images of 1024 × 2048 .

Revised: Following (Luc et al., 2017; 2018), we down-sample the predicted future instance segmentation masks to 128 × 256 resolution for the evaluation.

Original: A.3. Q UALITATIVE R ESULTS
Revised: B.3. Q UALITATIVE R ESULTS

Original: Qualitative comparison We present the qualitative comparisons of our method to existing future segmentation models, which corresponds to Table 3 in the main paper.
Revised: Qualitative comparison. We present the qualitative comparisons of our method to existing future segmentation models, which correspond to Table 3 in the main paper.

Original: As shown in the table, our method outperforms the two baselines in all classes, thanks to its ability to handle  highly stochastic nature of complex driving scenes.
Revised: As shown in the table, our method outperforms the two baselines in all classes, thanks to its ability to handle the highly stochastic nature of complex driving scenes.

Original: We also compare qualitative samples of each model in Figure E. As shown in the ﬁgure, our model predicts convincing structures and their temporal dynamics are accurate.
Revised: We also compare qualitative samples of each model in Figure G. As shown in the ﬁgure, our model predicts convincing structures and their temporal dynamics are accurate.

Original: On the other hand, the two baselines produce blurry predictions, resulting in (1) inaccurate structures (S2S at all timestamps and F2F at 17-29th timestamps), and (2) multiple semantics in a single structure (S2S at all timestamps and F2F at 23-29th timestamps).
Revised: On the other hand, the two baselines produce blurry predictions, resulting in (1) inaccurate structures (S2S at all timestamps and F2F at 17-29th timestamps), and (2) multiple semantics in a single structure (S2S at all timestamps and F2F at 23-29th timestamps).

Original: These blurred prediction problems have been widely observed in the literature and attributed to the inability of deterministic models to handle stochasticity in data Denton & Fergus (2018).
Revised: These blurred prediction problems have been widely observed in the literature and attributed to the inability of deterministic models to handle stochasticity in data (Denton & Fergus, 2018). Short-term prediction.

Revised: To evaluate the future segmentation results in Section 4.4, we extractfuture semantic or instance segmentations ahead (14, 17, 20, 23, 26, and 29th) from each model given 4 contexts (2, 5, 8, and 11th).

Revised: Then, following (Luc et al., 2018), we compute per-class and mean IoUs for 8 moving object categories only: person, rider, car, truck bus, train, motorcycle, bicycle .

Revised: Long-term video prediction.

Original:  contextx_t+0 predictionx_t+3 predictionx_t+6 predictionx_t+
Revised: Figure H illustrates the qualitative results of video prediction results on the Cityscapes dataset. As the Cityscapes contains many occluding objects, we employ the extended structure generator with instance boundary prediction to improve the frame prediction quality. We show in Figure H the predicted sequences up to 96 future frames conditioned oncontext frames. As shown in the ﬁgure, our method can easily incorporate the additional structures ( i.e. instance boundary map), and can be generalized to complex and high-resolution sequences contextx_t+0 predictionx_t+3 predictionx_t+6 predictionx_t+

Original: S2S
Revised: S2S

Original: F2F
Revised: F2F

Original: Oracle
Revised: Oracle

Original: Ours
Revised: Ours

Original: Batch 52 / frankfurt_000001_
Revised: Batch 52 / frankfurt_000001_

Original: 14th
Revised: 14th

Original: 17th 20th
Revised: 17th 20th

Original: Oracle [49] S2S Ours F2F
Revised: Oracle [49] S2S Ours F2F

Original: Oracle [49]
Revised: Oracle [49]

Original: S2S
Revised: S2S

Original: Ours
Revised: Ours

Original: F2F 14th 17th 20th
Revised: F2F 14th 17th 20th

Original: Ground-Truth
Revised: Ground-Truth

Original: Ours
Revised: Ours

Original: F2F [30]
Revised: F2F [30]

Original: S2S [27]
Revised: S2S [27]

Original: Oracle [33] 17th 14th 20th 23th
Revised: Oracle [33] 17th 14th 20th 23th

Original: 26th 29th
Revised: 26th 29th

Original: GT
Revised: GT

Original: Ours
Revised: Ours

Original: F2F
Revised: F2F

Original: S2S
Revised: S2S

Original: Oracle [49] 17th 14th 20th 23th
Revised: Oracle [49] 17th 14th 20th 23th

Original: 26th 29thLong-term video prediction Figure F illustrates the qualitative results of video prediction results on Citsycapes dataset. As the Cityscapes contain many occluding objects, we employ the extended structure generator with instance boundary prediction to improve the frame prediction quality. We show inFigure F the predicted sequences up to 96 future frames conditioned on 4 context frames.
Revised: 26th 29th to predict instance-wise structures into the long future without error propagations ( e.g. blurred predictions in the structure space).

Original: As shown in the ﬁgure, our method can easily incorporate the additional structures ( i.e., instance boundary map), and can be generalized to much complex and high-resolution sequences to predict instance-wise structures into the long future without error propagations ( e.g. blurred predictions in the structure space).

Original: On the other hand, we also observe some undesirable biases in the predicted motion dynamics in the structure space, such as static or cyclic motions of the structures (e.g.red dashed boxes in Figure F).
Revised: On the other hand, we also observe some undesirable biases in the predicted motion dynamics in the structure space, such as static or cyclic motions of the structures (e.g.red dashed boxes in Figure H).

Original: We notice that it is because the training sequences in Cityscapes datasets are considerably short ( i.e., , 30 frames), which makes the structure generatorto favor short-term dependencies in the prediction ( e.g., , short-term skip connections).
Revised: We notice that it is because the training sequences in Cityscapes datasets are considerably short ( i.e. 30 frames), which makes the structure generator favor short-term dependencies in the prediction ( e.g. short-term skip connections).

Original: It leads to biases in the model and artifacts in long-term extrapolation, such as static motions in the distant objects ( e.g., , buildings and trees far away from the camera), ﬁxed content ( e.g., , repeatedly generated objects),  etc. , which are not observed in the other datasets with reasonably long training sequences.
Revised: It leads to biases in the model and artifacts in long-term extrapolation, such as static motions in the distant objects ( e.g. buildings and trees far away from the camera), ﬁxed content ( e.g. repeatedly generated objects), and etc. , which are not observed in the other datasets with reasonably long training sequences.

Original: 71th 128th 185th 242th
Revised: 71th 128th 185th 242th

Original: [Figure]
Revised: [Figure]

Original: PredictedFrames

Original: A. E VALUATION PROTOCOLS

Original: Short-term prediction Following the previous works

Original: , we evaluate the short-term prediction quality by comparing the ground-truth and predicted frames in test videos. We employ three metrics: VGG cosine similarity (CSIM), mean Intersection-over-Union (mIoU), and Fr´echet Video Distance (FVD) (Unterthiner et al., 2018). The ﬁrst metric measures the frame-wise perceptual similarity using VGG features, while the second one measures the structure-level similarity using the label maps extracted by the pre-trained semantic segmentation network (Zhu et al., 2019).
Revised: ObservedFrame

Original: To handle stochasticity, we sample 100 predictions for each test clip and report the scores of the best scoring predictions.

Original: For the video-level evaluation, we adopt FVD that measures a Fr´echet distance between the ground-truth videos and the generated ones in a video representation space.

Original: We use a Inﬂated 3D Convnet (I3D) (Carreira & Zisserman, 2017) network pre-trained on Kinetics 400 dataset (Carreira & Zisserman, 2017) to obtain representations on videos, and use all 100 predictions to compute FVD against ground-truth sequences. To evaluate the future segmentation results in Section 4.4,
Revised: PredictedFrames

Original: Long-term prediction To evaluate the shot-wise FVD, we employ the For the models trained on

Original: KITTI dataset, we use a ResNet-18 (He et al., 2016) pre-trained on Places 365 (Zhou et al., 2017) to compute the Inception Score.

Original: For the models trained on the Human Dancing dataset, we use the

Original: Inception V3 (Szegedy et al., 2015) pre-trained on ImageNet (Russakovsky et al., 2015).

Original: B M ODEL ARCHITECTURE
Revised: C M ODEL ARCHITECTURE

Original: In this section, we detail the architectures and hyperparameters used in our experiments.
Revised: In this section, we detail the architectures and hyperparameters used in our experiments.

Original: B. S EQUENTIAL D ENSE S TRUCTURE G ENERATOR
Revised: C. S EQUENTIAL D ENSE S TRUCTURE G ENERATOR

Original: For this model, we use the modiﬁed version of SVG  by Villegas et al.
Revised: We use the modiﬁed version of SVG (Denton & Fergus, 2018) by Villegas et al.

Original: Villegas et al. (2019)  which was originally proposed by Denton et al. Denton & Fergus (2018).
Revised: (2019) as our sequential dense structure generator, which we refer to as SVG-extend throughout the paper.

Original: The two main differences between the original and the modiﬁed version are two-fold: (1) Villegas et al. Villegas et al. (2019) use the shallower auto-encoder architecture than Denton et al. Denton & Fergus (2018).
Revised: The two main differences between SVG and SVG-extend are two-fold: (1) SVG-extend adopts the shallower auto-encoder architecture than that of SVG.

Original: (2) Villegas et al. Villegas et al. (2019) use the Convolutional-LSTM Shi et al.
Revised: (2) SVG-extend replaces Fully-Connected-LSTM with Convolutional-LSTM (Shi et al., 2015).

Original: while Denton et al. Denton & Fergus (2018) uses the Fully-Connected-LSTM. These  modiﬁcations are to increase the capacity of the architecture.
Revised: These two modiﬁcations are to better exploit spatial information contained in 3D feature maps, which is not maintained when inputs are mapped to low-dimensional 1-D image embedding vectors.

Original: Since the detailed architectural conﬁguration of SVG is not described in Villegas et al. (2019), we summarize our reproduced version of this framework in Table A, B, C, and D. For all experiments on the 64 × 64 image resolution, we use the same architecture described in the tables.
Revised: Since the detailed architectural conﬁguration of SVG-extend is not described in (Villegas et al., 2019), we summarize our reproduced version of this framework in Table A, B, C, and D. On the 64 × 64 image resolution, we use the same architecture described in the tables.

Original: For much higher resolutions ( e.g. 128 × 128 and 256 × 256 ), we simply stack  more convolutional blocks (non-linear convolutional layers followed by upsampling or downsampling layers) in the Encoder and Decoder networks so that the shape of h t , z t , g t is kept to R 128 × 8 × 8 agnostic to the image resolution.
Revised: On 128 ×and 256 × 256 image resolutions, we simply stack one and two more convolutional blocks (nonlinear convolutional layers followed by upsampling or downsampling layers) in the Encoder and Decoder networks, respectively, so that the shape of h t , z t , g t is kept to R 128 × 8 × 8 agnostic to the image resolutions.

Original: For hyperparameters, we use C = 5 for KITTI and Human Dancing, and use C = 4 for Cityscapes; we set β = 0 . 0005 ; we use ADAM optimizer Kingma & Ba (2015) with the learning rate of 0 .and ( β 1 , β 2 ) =(0 . 9 , 0 . 999) .
Revised: For hyperparameters, we use C = 5 ; we set β = 0 . 0005 ; we use ADAM optimizer (Kingma & Ba, 2015) with the learning rate of 0 . 0001 and ( β 1 , β 2 ) =

Original: We refer to and modify the publicly available code 1 from Denton et al. Denton & Fergus (2018) to reproduce Villegas et al. Villegas et al.
Revised: C

Original: B. B OUNARY MAP PREDICTION
Revised: B OUNARY MAP PREDICTION

Original: When the pre-trained instance-wise segmentation model is available, we can optionally extend the structure generator to jointly predict the object boundary maps.
Revised: When the pre-trained instance-wise segmentation model is available, we can optionally extend the structure generator to jointly predict the object boundary maps.

Original: Such structure can add a notion of object instance, and is useful to improve the image generator in sequences with many occluding objects.
Revised: Such structures can add a notion of object instance and is useful to improve the image generator in sequences with many occluding objects.

Original: Let e t ∈ { 0 , 1 } W × H denotes the object boundary map at frame x t .
Revised: Let e t ∈ { 0 , 1 } W × H denotes the object boundary map at frame x t .

Original: Then we train the denoising autoencoder G edge to produce  edge map of each frames by
Revised: Then we train the denoising autoencoder G edge to produce an edge map of each frame by

Original: [Equation]
Revised: [Equation]

Original: We employed the encoder-decoder network () as the conditional generator, and train it based on adversarial loss by
Revised: We employed the encoder-decoder network (Nazeri et al., 2019) as the conditional generator, and train it based on adversarial loss by

Original: [Equation]
Revised: [Equation]

Original: (9) where D is the discriminator.
Revised: where D is the discriminator.

Original: Note that the edge generator in Eq.
Revised: Note that the edge generator in Eq.

Original: equation 8 is applied to each frames independently; we observe that adding the predicted boundary map as an input to the structure generator makes the modelto be prune to error propagation.
Revised: 8 is applied to each frame independently; we observe that adding the predicted boundary map as an input to the structure generator makes the model be prune to error propagation.

Original: The boundary and label maps are then combined as the output of the structure generator ˜ s t =
Revised: The boundary and label maps are then combined as the output of the structure generator ˜ s t =

Original: [ˆ s t , ˆ e t ] .
Revised: [ˆ s t , ˆ e t ] .

Original: B. S TRUCTURE -C ONDITIONAL P IXEL S EQUENCE G ENERATOR
Revised: C. S TRUCTURE -C ONDITIONAL P IXEL S EQUENCE G ENERATOR

Original: We use Vid2Vid [39] for our structure-conditional pixel sequence generator.
Revised: We use Vid2Vid (Wang et al., 2018) for our structure-conditional pixel sequence generator.

Original: For Vid2Vid on KITTI dataset of 64 × 64 resolution, we reduce the number of downsampling layers in the generator and the scale parameter of the patch discriminator to 1.
Revised: For Vid2Vid on KITTI dataset of 64 × 64 resolution, we reduce the number of downsampling layers in the generator and the scale parameter of the patch discriminator to 1.

Original: We use τ = 5 for KITTI and Human Dancing and use τ = 4 for Cityscapes.
Revised: We use τ = 5 for KITTI and Human Dancing and use τ = 4 for Cityscapes.

Original: We use τ (cid:48)
Revised: We use τ (cid:48)

Original: = 3 for all experiments.
Revised: = 3 for all experiments.

Original: If not speciﬁed otherwise, we do not modify any hyperparameters and simply follow the default settings for the training.
Revised: If not speciﬁed otherwise, we do not modify any hyperparameters and simply follow the default settings for the training.

Original: We use the publicly available code 2 .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]