Original: R EVISITING H IERARCHICAL A PPROACH FOR P ERSISTENT L ONG -T ERM V IDEO P REDICTION
Revised: R EVISITING H IERARCHICAL A PPROACH FOR P ERSISTENT L ONG -T ERM V IDEO P REDICTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning to predict the long-term future of video frames is notoriously challenging due to the inherent ambiguities in the distant future and the dramatic ampliﬁcation of the prediction error through time.
Revised: Learning to predict the long-term future of video frames is notoriously challenging due to the inherent ambiguities in the distant future and the dramatic ampliﬁcation of the prediction error through time.

Original: Despite the recent advances in the literature, existing approaches are limited to moderately short-term prediction (less than a few seconds), while extrapolating it to a longer future quickly leads to destruction in structure and content.
Revised: Despite the recent advances in the literature, existing approaches are limited to moderately short-term prediction (less than a few seconds), while extrapolating it to a longer future quickly leads to destruction in structure and content.

Original: In this work, we revisit hierarchical models in video prediction.
Revised: In this work, we revisit hierarchical models in video prediction.

Original: Our method generates future frames by ﬁrst estimating a sequence of dense semantic structures and subsequently translating the estimated structures to pixels by video-to-video translation.
Revised: Our method generates future frames by ﬁrst estimating a sequence of dense semantic structures and subsequently translating the estimated structures to pixels by video-to-video translation.

Original: Despite its simplicity, we show that modeling structures and their dynamics in a categorical structure space with a stochastic sequential estimator leads to surprisingly successful long-term prediction.
Revised: Despite its simplicity, we show that modeling structures and their dynamics in a categorical structure space with a stochastic sequential estimator leads to surprisingly successful long-term prediction.

Original: We evaluate our method on three challenging video prediction datasets involving car driving and human dancing , and demonstrate that it can generate complicated scene structures and motions over a very long time horizon ( i.e., thousands frames), setting a new standard of video prediction with orders of magnitude longer prediction time than existing approaches.
Revised: We evaluate our method on three challenging video prediction datasets involving car driving and human dancing , and demonstrate that it can generate complicated scene structures and motions over a very long time horizon ( i.e., thousands frames), setting a new standard of video prediction with orders of magnitude longer prediction time than existing approaches.

Original: Full videos are available at https://bit.ly/2EyDSem.
Revised: Full videos are available at https://bit.ly/2EyDSem.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Video prediction aims to generate future frames conditioned on a short video clip.
Revised: Video prediction aims to generate future frames conditioned on a short video clip.

Original: It has received much attention in recent years as forecasting the future of visual sequence is critical in improving the planning for model-based reinforcement learning
Revised: It has received much attention in recent years as forecasting the future of visual sequence is critical in improving the planning for model-based reinforcement learning

Original: (Finn et al., 2016; Hafner et al., 2019; Ha &
Revised: (Finn et al., 2016; Hafner et al., 2019; Ha &

Original: Schmidhuber, 2018), forecasting future event (Hoai & Torre, 2013), action (Lan et al., 2014), and activity (Lan et al., 2014; Ryoo, 2011).
Revised: Schmidhuber, 2018), forecasting future event (Hoai & Torre, 2013), action (Lan et al., 2014), and activity (Lan et al., 2014; Ryoo, 2011).

Original: To make it truly beneﬁcial for these applications, video prediction should be capable of forecasting long-term future.
Revised: To make it truly beneﬁcial for these applications, video prediction should be capable of forecasting long-term future.

Original: Many previous approaches have formulated video prediction as a conditional generation task by recursively synthesizing future frames conditioned on the previous frames (Vondrick et al., 2016;
Revised: Many previous approaches have formulated video prediction as a conditional generation task by recursively synthesizing future frames conditioned on the previous frames (Vondrick et al., 2016;

Original: Tulyakov et al., 2018; Denton & Fergus, 2018; Babaeizadeh et al., 2018; Castrejon et al., 2019; Villegas et al., 2019).
Revised: Tulyakov et al., 2018; Denton & Fergus, 2018; Babaeizadeh et al., 2018; Castrejon et al., 2019; Villegas et al., 2019).

Original: Despite their success in short-term forecasting, however, none of these approaches have been successful in synthesizing convincing long-term future, due to the challenges in modeling complex dynamics and extrapolating from short sequences to much longer future.
Revised: Despite their success in short-term forecasting, however, none of these approaches have been successful in synthesizing convincing long-term future, due to the challenges in modeling complex dynamics and extrapolating from short sequences to much longer future.

Original: As the prediction errors easily accumulates and ampliﬁes through time, the quality of the predicted frames quickly degrades over time.
Revised: As the prediction errors easily accumulates and ampliﬁes through time, the quality of the predicted frames quickly degrades over time.

Original: One way to reduce the error propagation is to extrapolate in a low dimensional structure space instead of directly estimating pixel-level dynamics in a video.
Revised: One way to reduce the error propagation is to extrapolate in a low dimensional structure space instead of directly estimating pixel-level dynamics in a video.

Original: Therefore, many hierarchical modeling approaches are proposed (Villegas et al., 2017b; Wichers et al., 2018;
Revised: Therefore, many hierarchical modeling approaches are proposed (Villegas et al., 2017b; Wichers et al., 2018;

Original: Liang et al., 2017; Yan et al., 2018; Walker et al., 2017; Kim et al., 2019).
Revised: Liang et al., 2017; Yan et al., 2018; Walker et al., 2017; Kim et al., 2019).

Original: These approaches ﬁrst generate a sequence using a low-dimensional structure representation, and subsequently generate appearance conditioned on the predicted structures.
Revised: These approaches ﬁrst generate a sequence using a low-dimensional structure representation, and subsequently generate appearance conditioned on the predicted structures.

Original: Hierarchical approaches are potentially promising for long-term prediction since learning structure-aware dynamics allows the model to generate semantically accurate motion and content in the future.
Revised: Hierarchical approaches are potentially promising for long-term prediction since learning structure-aware dynamics allows the model to generate semantically accurate motion and content in the future.

Original: However, previous approaches often employed too speciﬁc and incomprehensive structures such as human body joints (Villegas et al., 2017b; Yan et al., 2018;
Revised: However, previous approaches often employed too speciﬁc and incomprehensive structures such as human body joints (Villegas et al., 2017b; Yan et al., 2018;

Original: Yang et al., 2018; Walker et al., 2017; Kim et al., 2019) or face landmarks (Yan et al., 2018;
Revised: Yang et al., 2018; Walker et al., 2017; Kim et al., 2019) or face landmarks (Yan et al., 2018;

Original: Yang et al., 2018).
Revised: Yang et al., 2018).

Original: Moreover, they made oversimpliﬁed assumptions of the future by using a deterministic loss or assuming homogeneous content.
Revised: Moreover, they made oversimpliﬁed assumptions of the future by using a deterministic loss or assuming homogeneous content.

Original: We therefore argue that the beneﬁt of hierarchical model has been underestimated and their impact on long-term video prediction has not been properly demonstrated.
Revised: We therefore argue that the beneﬁt of hierarchical model has been underestimated and their impact on long-term video prediction has not been properly demonstrated.

Original: In this paper, we propose a hierarchical model with a general structure representation (i.e., dense semantic label map ) for long-term video prediction with complex scenes.
Revised: In this paper, we propose a hierarchical model with a general structure representation (i.e., dense semantic label map ) for long-term video prediction with complex scenes.

Original: We abstract the scene as categorical labels for each pixel, and predict the motion and content change in this label space
Revised: We abstract the scene as categorical labels for each pixel, and predict the motion and content change in this label space

Original: [Figure]
Revised: [Figure]

Revised: Section 3.

Revised: [Figure]

Revised: [Figure]

Original: using variational sequence models.
Revised: using variational sequence models.

Original:  the context frames and the predicted label maps, we then generate the textures by translating the sequence of label maps to the RGB frames.
Revised: Given the context frames and the predicted label maps, we then generate the textures by translating the sequence of label maps to the RGB frames.

Original: As dense label maps are generic and universal, we can learn comprehensive scene dynamics from object motion to even dramatic scene change.
Revised: As dense label maps are generic and universal, we can learn comprehensive scene dynamics from object motion to even dramatic scene change.

Original: We can also capture the multi-modal dynamics in the dense label  the  prediction of the variational sequence model.
Revised: We can also capture the multi-modal dynamics in the dense label space with the stochastic prediction of the variational sequence model.

Original:  that we can generate a surprisingly long term future of videos from driving scenes to human  the complex motion of multiple objects and even an evolution of the  in a future the  is  persistent future prediction virtually near prediction wise the  without ground
Revised: Our experiments demonstrate that we can generate a surprisingly long-term future of videos, from driving scenes to human dancing, including the complex motion of multiple objects and even an evolution of the content in a distant future. We also show that the predicted frame quality is preserved through time, which enables persistent future prediction virtually near-inﬁnite time horizon. For scalable evaluation of long-term prediction at this scale, we also propose a novel metric called shot-wise FVD , which enables the evaluation of spatio-temporal prediction quality without ground-truth frames and is consistent with human perception.

Original:  M ETHOD
Revised:  M ETHOD

Original: Given the context frames x 1: C = { x 1 , x 2 , ..., x C } , our goal is to synthesize the future frames x C +1: T = { x C +1 , x C +2 , ..., x T } up to an arbitrary long-term future T .
Revised: Given the context frames x 1: C = { x 1 , x 2 , ..., x C } , our goal is to synthesize the future frames x C +1: T = { x C +1 , x C +2 , ..., x T } up to an arbitrary long-term future T .

Original: Let s t ∈ R W × H × N denote a dense label map of the frame x t deﬁned over N categories, which is inferred by the pre-trained semantic segmentation model 1 .
Revised: Let s t ∈ R W × H × N denote a dense label map of the frame x t deﬁned over N categories, which is inferred by the pre-trained semantic segmentation model 1 .

Original: Then given the context frames x 1: C and the label maps s 1: C , our hierarchical framework synthesizes the future frames ˆ x C +1: T by the following steps.
Revised: Then given the context frames x 1: C and the label maps s 1: C , our hierarchical framework synthesizes the future frames ˆ x C +1: T by the following steps.

Original: • The structure generator takes the context label maps as input and produces a sequence of the future label maps by ˆ s C +1: T ∼ G struct ( s 1: C , z 1: T ) , where z t denotes the latent variable encoding the stochasticity of the structure.
Revised: • The structure generator takes the context label maps as input and produces a sequence of the future label maps by ˆ s C +1: T ∼ G struct ( s 1: C , z 1: T ) , where z t denotes the latent variable encoding the stochasticity of the structure.

Original: • Given the context frames and the predicted structures, the image generator produces the RGB frames by ˆ x C +1: T ∼ G image ( x 1: C , { s 1: C , ˆ s C
Revised: • Given the context frames and the predicted structures, the image generator produces the RGB frames by ˆ x C +1: T ∼ G image ( x 1: C , { s 1: C , ˆ s C

Original: +1: T } ) .
Revised: +1: T } ) .

Original: Figure 1 illustrates the overall pipeline.
Revised: Figure 1 illustrates the overall pipeline.

Original: Note that there are various factors beyond motion that make spatio-temporal variations in the label maps, such as the emergence of new objects, partial observability, or even the dramatic scene changes by the global camera motion ( e.g. , panning).
Revised: Note that there are various factors beyond motion that make spatio-temporal variations in the label maps, such as the emergence of new objects, partial observability, or even the dramatic scene changes by the global camera motion ( e.g. , panning).

Original: By learning to model these dynamics in the semantic labels with the stochastic sequence estimator and conditioning the video generation with the estimated labels, the proposed hierarchical model can synthesize convincing frames into the very long-term future.
Revised: By learning to model these dynamics in the semantic labels with the stochastic sequence estimator and conditioning the video generation with the estimated labels, the proposed hierarchical model can synthesize convincing frames into the very long-term future.

Original: Below, we describe each component.
Revised: Below, we describe each component.

Original: 2. S EQUENTIAL D ENSE S TRUCTURE G ENERATOR
Revised: 2. S EQUENTIAL D ENSE S TRUCTURE G ENERATOR

Original: Our structure generator models the dynamics in label maps by p ( s ≤ T | z ≤ T ) =
Revised: Our structure generator models the dynamics in label maps by p ( s ≤ T | z ≤ T ) =

Original: (cid:81)
Revised: (cid:81)

Original: Tt =1 p ( s t | s <t , z ≤ t ) .
Revised: Tt =1 p ( s t | s <t , z ≤ t ) .

Original: We employ the sequence model based on VAE (Denton & Fergus, 2018) since (1) it provides a probabilistic framework to handle stochastic variations of the structure and (2) it can easily incorporate discrete sequences ( i.e., label maps).
Revised: We employ the sequence model based on VAE proposed by Denton & Fergus (2018) since (1) it provides a probabilistic framework to handle stochastic variations of the structure and (2) it can easily incorporate discrete sequences ( i.e., label maps).

Original: Speciﬁcally, we optimize the following variational lower-bound of the sequence:
Revised: Speciﬁcally, we optimize the following β -VAE objective (Denton & Fergus, 2018; Higgins et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: where q φ ( z ≤ T | s ≤ T ) is the approximated posterior distribution and p ψ ( z t | s <t ) is the prior distribution.
Revised: where q φ ( z ≤ T | s ≤ T ) is the approximated posterior distribution and p ψ ( z t | s <t ) is the prior distribution.

Original: At each step, it learns to represent and reconstruct each label map s t through CNN-based encoder f enc and decoder f dec , respectively, while the temporal dependency of the label maps is modeled stochastically by the two LSTMs as follows:
Revised: At each step, it learns to represent and reconstruct each label map s t through CNN-based encoder f enc and decoder f dec , respectively, while the temporal dependency of the label maps is modeled stochastically by the two LSTMs as follows:

Original: [Equation]
Revised: [Equation]

Original: where h t = f enc ( s t [Equation] where h t − 1 = f enc ( s t − 
Revised: where h t = f enc ( s t ) , where h t − 1 = f enc ( s t − 1 ) ,

Original: [Equation]
Revised: [Equation]

Original: where LSTM θ and LSTM φ respectively approximate the generative and the posterior distributions recurrently up to the time step t .
Revised: where LSTM θ and LSTM φ respectively approximate the generative and the posterior distributions recurrently up to the time step t .

Original: Unlike Denton & Fergus (2018); Villegas et al.
Revised: Unlike Denton & Fergus (2018); Villegas et al.

Original: (2019) that exploit skip-connection from the last observed context frame during both training and testing time, we skip hidden representations of the encoder to the decoder at every time step during testing to handle longterm dynamics in structure.
Revised: (2019) that exploit skip-connection from the last observed context frame during both training and testing time, we skip hidden representations of the encoder to the decoder at every time step during testing to handle longterm dynamics in structure as in (Villegas et al., 2017a; Finn et al., 2016).

Original: We also apply aggressive dilated convolutions to encode global motion context.
Revised: We also apply aggressive dilated convolutions to encode global motion context.

Original: Please see Appendix C.1 for more detail.
Revised: Please see Appendix C.1 for more detail.

Original: During training, we apply teacher-forcing by feeding ground-truth label maps and sampling the latent z t from the posterior distribution N ( µ φ ( t ) , σ φ ( t ))
Revised: During training, we apply teacher-forcing by feeding ground-truth label maps and sampling the latent z t from the posterior distribution N ( µ φ ( t ) , σ φ ( t ))

Original: =
Revised: =

Original: LSTM φ ( f enc ( s t )) .
Revised: LSTM φ ( f enc ( s t )) .

Original: During inference, we recursively generate label maps by (1) sampling z t from the prior distribution N ( µ ψ ( t ) , σ ψ ( t )) =
Revised: During inference, we recursively generate label maps by (1) sampling z t from the prior distribution N ( µ ψ ( t ) , σ ψ ( t )) =

Original: LSTM ψ ( f enc (ˆ s t − 1 )) , (2) producing the frame ˆ s t through the decoder, and (3) discretizing the predicted label map by taking pixel-wise maximum.
Revised: LSTM ψ ( f enc (ˆ s t − 1 )) , (2) producing the frame ˆ s t through the decoder, and (3) discretizing the predicted label map by taking pixel-wise maximum.

Original: Such discretization provides additional merits for being more robust against error propagation than continuous structures such as 2D keypoints (Villegas et al., 2017b; Yan et al., 2018; Yang et al., 2018) or optical ﬂow (Walker et al., 2017).
Revised: Such discretization provides additional merits for being more robust against error propagation than continuous structures such as 2D keypoints (Villegas et al., 2017b; Yan et al., 2018; Yang et al., 2018) or optical ﬂow (Walker et al., 2017).

Original: Extension to object boundary prediction When the pre-trained instance-wise segmentation model is available, we can optionally extend the structure generator to jointly predict the object boundary maps.
Revised: Extension to object boundary prediction When the pre-trained instance-wise segmentation model is available, we can optionally extend the structure generator to jointly predict the object boundary maps.

Original: Such structure can add a notion of object instance, and is useful to improve the image generator in sequences with many occluding objects.
Revised: Such structure can add a notion of object instance, and is useful to improve the image generator in sequences with many occluding objects.

Original: Let e t ∈ { 0 , 1 } W × H denotes the object boundary map at frame x t .
Revised: Let e t ∈ { 0 , 1 } W × H denotes the object boundary map at frame x t .

Original: Then we train the denoising autoencoder ˆ e t =
Revised: Then we train the conditional generator ˆ e t =

Original: G edge ( s t ) that produces a boundary map given a label map for each frame 1 .
Revised: G edge ( s t ) that produces a boundary map given a label map for each frame 1 .

Original: We train G edge using the conditional GAN objective to maximize the joint probability between the  label and  boundary maps p ( s t , e t ) .
Revised: We train G edge using the conditional GAN objective to match the generated distribution to the joint distribution of the real label and real boundary maps p ( s t , e t ) .

Original: The boundary and label maps are then combined as the output of the structure generator ˜ s t =
Revised: The boundary and label maps are then combined as the output of the structure generator ˜ s t =

Original: [ˆ s t , ˆ e t ] , and is used as an input to the image generator described below.
Revised: [ˆ s t , ˆ e t ] , and is used as an input to the image generator described below.

Original: See Appendix C.2 for implementation details.
Revised: See Appendix C. for implementation details.

Original: 2. S TRUCTURE -C ONDITIONAL P IXEL S EQUENCE G ENERATOR
Revised: 2. S TRUCTURE -C ONDITIONAL P IXEL S EQUENCE G ENERATOR

Original: Given a sequence of the structures and the context frames, the image generator learns to model the conditional distribution of the RGB frames by p ( x ≤ T | s ≤ T ) =
Revised: Given a sequence of the structures and the context frames, the image generator learns to model the conditional distribution of the RGB frames by p ( x ≤ T | s ≤ T ) =

Original: (cid:81)
Revised: (cid:81)

Original: Tt =1 p ( x t | x <t , s ≤ t ) .
Revised: Tt =1 p ( x t | x <t , s ≤ t ) .

Original: We formulate this task as a video-to-video translation problem, and employ a state-of-the-art conditional video generation model (Wang et al., 2018).
Revised: We formulate this task as a video-to-video translation problem, and employ a state-of-the-art conditional video generation model (Wang et al., 2018).

Original: Speciﬁcally, the video synthesis network F in Wang et al.
Revised: Speciﬁcally, the video synthesis network F in Wang et al.

Original: (2018) consists of three main components; the generator H , occlusion mask predictor M , and optical ﬂow estimator W , which are combined to generate each frame ˆ x t by the following operation:
Revised: (2018) consists of three main components; the generator H , occlusion mask predictor M , and optical ﬂow estimator W , which are combined to generate each frame ˆ x t by the following operation:

Original: [Equation]
Revised: [Equation]

Original: where ˆ w t − 1 = W (ˆ x t − τ : t − 1 , s t − τ : t ) is the warped previous frame ˆ x t − 1 using the estimated optical ﬂow, h t
Revised: where ˆ w t − 1 = W (ˆ x t − τ : t − 1 , s t − τ : t ) is the warped previous frame ˆ x t − 1 using the estimated optical ﬂow, h t

Original: = H (ˆ x t − τ : t − 1 , s t − τ : t ) is the hallucinated frame at t , and m t =
Revised: = H (ˆ x t − τ : t − 1 , s t − τ : t ) is the hallucinated frame at t , and m t =

Original: M (ˆ x t − τ : t − 1 , s t − τ : t ) is the soft occlusion mask blending ˆ w t − 1 and h t .
Revised: M (ˆ x t − τ : t − 1 , s t − τ : t ) is the soft occlusion mask blending ˆ w t − 1 and h t .

Original: Unlike models synthesizing future frames by transforming the context frames x C (Villegas et al., 2019; 2017b; Yan et al., 2018; Tulyakov et al., 2018), this model is appropriate to synthesize the long-term future since it can handle both transformation of the existing objects (via ˆ w t − 1 ) and synthesis of the emerging objects (via h t ).
Revised: Unlike models synthesizing future frames by transforming the context frames x C (Villegas et al., 2019; 2017b; Yan et al., 2018; Tulyakov et al., 2018), this model is appropriate to synthesize the long-term future since it can handle both transformation of the existing objects (via ˆ w t − 1 ) and synthesis of the emerging objects (via h t ).

Original: To ensure both frame-level and video-level generation quality, the video synthesis network F is trained against a conditional image discriminator D I and a conditional video discriminator D V through adversarial learning by
Revised: To ensure both frame-level and video-level generation quality, the video synthesis network F is trained against a conditional image discriminator D I and a conditional video discriminator D V through adversarial learning by

Original: [Equation]
Revised: [Equation]

Original: where L I ( F, D I ) and L V ( F, D V ) are frame-level and video-level adversarial losses, respectively.
Revised: where L I ( F, D I ) and L V ( F, D V ) are frame-level and video-level adversarial losses, respectively.

Original: For efﬁcient training, we follow (Wang et al., 2018) to adopt sampling operators
Revised: For efﬁcient training, we follow (Wang et al., 2018) to adopt sampling operators

Original: φ I ( x ≤ T , s ≤ T ) =
Revised: φ I ( x ≤ T , s ≤ T ) =

Original: ( x i , s i ) and φ V ( w <T , x ≤ T , s ≤ T ) =
Revised: ( x i , s i ) and φ V ( w <T , x ≤ T , s ≤ T ) =

Original: ( w i
Revised: ( w i

Original: [Equation]
Revised: [Equation]

Original: 2: i − τ (cid:48) , x i
Revised: 2: i − τ (cid:48) , x i

Original: [Equation]
Revised: [Equation]

Original: 1: i
Revised: 1: i

Original: [Equation]
Revised: [Equation]

Original: τ (cid:48) , s i
Revised: τ (cid:48) , s i

Original: [Equation]
Revised: [Equation]

Original: 1: i − τ (cid:48) ) for frame and video-level ad- versarial learning objectives, respectively, where i is an integer sampled from U (1 , T ) and U ( τ (cid:48) + 1 , T + 1) for frame sampling operator φ I and video sampling operator φ V , respectively.
Revised: 1: i − τ (cid:48) ) for frame and video-level ad- versarial learning objectives, respectively, where i is an integer sampled from U (1 , T ) and U ( τ (cid:48) + 1 , T + 1) for frame sampling operator φ I and video sampling operator φ V , respectively.

Original: Then the ﬁnal learning objective is formulated by
Revised: Then the ﬁnal learning objective is formulated by

Original: [Equation]
Revised: [Equation]

Original: During training, the model is trained to estimate a frame sequence using the ground-truth structures, while we use the predicted structures by the structure generator during testing.
Revised: During training, the model is trained to estimate a frame sequence using the ground-truth structures, while we use the predicted structures by the structure generator during testing.

Original: See Appendix C. for implementation details.
Revised: See Appendix C. for implementation details.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The task of predicting future video frames has been extensively studied over the past few years (Villegas et al., 2019; Denton & Fergus, 2018; Lee et al., 2018; Babaeizadeh et al., 2018; Vondrick et al., 2016;
Revised: The task of predicting future video frames has been extensively studied over the past few years (Villegas et al., 2019; Denton & Fergus, 2018; Lee et al., 2018; Babaeizadeh et al., 2018; Vondrick et al., 2016;

Original: Finn et al., 2016; Clark et al., 2019;
Revised: Finn et al., 2016; Clark et al., 2019;

Original: Castrejon et al., 2019).
Revised: Castrejon et al., 2019).

Original: Early approaches focus on modeling simple and deterministic dynamics using regression loss (Srivastava et al., 2015; Ranzato et al., 2014) or predictive coding (Lotter et al., 2017).
Revised: Early approaches focus on modeling simple and deterministic dynamics using regression loss (Srivastava et al., 2015; Ranzato et al., 2014) or predictive coding (Lotter et al., 2017).

Original: However, such deterministic models may not be appropriate for modeling stochastic variations in real-world videos.
Revised: However, such deterministic models may not be appropriate for modeling stochastic variations in real-world videos.

Original: Recently, deep generative models have been employed to model dynamics in complex videos.
Revised: Recently, deep generative models have been employed to model dynamics in complex videos.

Original: Babaeizadeh et al.
Revised: Babaeizadeh et al.

Original: (2018) proposed a variational approach for modeling stochasticity in a sequence.
Revised: (2018) proposed a variational approach for modeling stochasticity in a sequence.

Original: Denton & Fergus (2018) incorporated more ﬂexible frame-wise inference models.
Revised: Denton & Fergus (2018) incorporated more ﬂexible frame-wise inference models.

Original: The prediction quality of the variational models has been further improved by employing rich structure in latent variables (Castrejon et al., 2019), maximally increasing the network parameters (Villegas et al., 2019), or incorporating adversarial loss (Vondrick et al., 2016;
Revised: The prediction quality of the variational models has been further improved by employing rich structure in latent variables (Castrejon et al., 2019), maximally increasing the network parameters (Villegas et al., 2019), or incorporating adversarial loss (Vondrick et al., 2016;

Original: Clark et al., 2019; Tulyakov et al., 2018; Villegas et al., 2017a).
Revised: Clark et al., 2019; Tulyakov et al., 2018; Villegas et al., 2017a).

Original: Despite their success, these approaches still fail to generate long-term videos and the common artifacts were losing object structures, switching object categories, etc .
Revised: Despite their success, these approaches still fail to generate long-term videos and the common artifacts were losing object structures, switching object categories, etc .

Original: Our approach addresses these issues by explicitly predicting dynamics in a low-dimensional label map using a hierarchical model.
Revised: Our approach addresses these issues by explicitly predicting dynamics in a low-dimensional label map using a hierarchical model.

Original: Hierarchical models studied in the past for video prediction were usually in speciﬁc video domains (Villegas et al., 2017b; Wichers et al., 2018;
Revised: Hierarchical models studied in the past for video prediction were usually in speciﬁc video domains (Villegas et al., 2017b; Wichers et al., 2018;

Original: Liang et al., 2017; Yan et al., 2018; Walker et al., 2017; Minderer et al., 2019).
Revised: Liang et al., 2017; Yan et al., 2018; Walker et al., 2017; Minderer et al., 2019).

Original: Villegas et al.
Revised: Villegas et al.

Original: (2017b) employed LSTMs to predict human body joints and visual analogy making to create textures, which is extended by Wichers et al.
Revised: (2017b) employed LSTMs to predict human body joints and visual analogy making to create textures, which is extended by Wichers et al.

Original: (2018) and Minderer et al.
Revised: (2018) and Minderer et al.

Original: to unsupervised approaches.
Revised: to unsupervised approaches.

Original: Other approaches employed sequential VAE (Yan et al., 2018) or GAN (Yang et al., 2018) to predict the human body posture.
Revised: Other approaches employed sequential VAE (Yan et al., 2018) or GAN (Yang et al., 2018) to predict the human body posture.

Original: However, these approaches are designed speciﬁcally for certain objects, and evaluated under simple videos containing only a single moving object.
Revised: However, these approaches are designed speciﬁcally for certain objects, and evaluated under simple videos containing only a single moving object.

Original: Also, they mostly utilized deterministic model to learn structure-level dynamics.
Revised: Also, they mostly utilized deterministic model to learn structure-level dynamics.

Original: These simpliﬁed assumptions on video content and dynamics limit their application to real-world videos.
Revised: These simpliﬁed assumptions on video content and dynamics limit their application to real-world videos.

Original: We propose to resolve these limitations by using dense semantic label maps as universal representation and stochastic sequential estimator for modeling video dynamics.
Revised: We propose to resolve these limitations by using dense semantic label maps as universal representation and stochastic sequential estimator for modeling video dynamics.

Original:  E XPERIMENTS 4. E VALUATION M ETRICS
Revised:  E XPERIMENTS 4. E VALUATION M ETRICS

Original: Below we describe the evaluation metrics used in the experiment.
Revised: Below we describe the evaluation metrics used in the experiment.

Original: Unless otherwise speciﬁed, we use 64 × 64 images for all quantitative evaluation for fair comparison to the existing works.
Revised: Unless otherwise speciﬁed, we use 64 × 64 images for all quantitative evaluation for fair comparison to the existing works.

Original: Full videos are available on the anonymous website: https://bit.ly/2EyDSem.
Revised: Full videos are available on the anonymous website: https://bit.ly/2EyDSem.

Original: Short-term prediction We employ three conventional metrics in literature to evaluate the prediction performance on short-term videos ( i.e., less than 50 frames): VGG cosine similarity (CSIM) measuring the frame-wise perceptual similarity using VGG features, mean Intersection-over-Union (mIoU) measuring the structure-level similarity using the label maps extracted by the pre-trained segmentation network (Zhu et al., 2019) and Fr´echet Video Distance (FVD) (Unterthiner et al., 2018) measuring a Fr´echet distance between the ground-truth videos and the generated ones in a video representation space.
Revised: Short-term prediction We employ three conventional metrics in literature to evaluate the prediction performance on short-term videos ( i.e., less than 50 frames): VGG cosine similarity (CSIM) measuring the frame-wise perceptual similarity using VGG features, mean Intersection-over-Union (mIoU) measuring the structure-level similarity using the label maps extracted by the pre-trained segmentation network (Zhu et al., 2019) and Fr´echet Video Distance (FVD) (Unterthiner et al., 2018) measuring a Fr´echet distance between the ground-truth videos and the generated ones in a video representation space.

Original: Detailed evaluation protocols are described in Appendix A.
Revised: Detailed evaluation protocols are described in Appendix A.

Original: Long-term prediction Compared to short-term prediction, evaluating prediction quality of arbitrary long-term video is challenging for a number of reasons.
Revised: Long-term prediction Compared to short-term prediction, evaluating prediction quality of arbitrary long-term video is challenging for a number of reasons.

Original: First, the ground-truth videos are seldom available at this scale, making it impossible to adopt metrics based on frame-wise comparison ( e.g., , CSIM, mIoU).
Revised: First, the ground-truth videos are seldom available at this scale, making it impossible to adopt metrics based on frame-wise comparison ( e.g., , CSIM, mIoU).

Original: Second, the uncertainty of future prediction increases exponentially with time, making it intractable to employ density-based metrics ( e.g., , FVD) as it requires exponentially many samples.
Revised: Second, the uncertainty of future prediction increases exponentially with time, making it intractable to employ density-based metrics ( e.g., , FVD) as it requires exponentially many samples.

Original: Since our goal is to demonstrate video prediction at the scale of hundreds to thousands of frames, we introduce a novel metric based on FVD, which enables evaluation of temporal and frame-level synthesis quality without ground-truth sequences and allows tractable evaluation through time.
Revised: Since our goal is to demonstrate video prediction at the scale of hundreds to thousands of frames, we introduce a novel metric based on FVD, which enables evaluation of temporal and frame-level synthesis quality without ground-truth sequences and allows tractable evaluation through time.

Original: Speciﬁcally, we introduce a shot-level video-quality evaluation metric as
Revised: Speciﬁcally, we introduce a shot-level video-quality evaluation metric as

Original: [Equation]
Revised: [Equation]

Original: which computes a FVD between the ground-truth shots
Revised: which computes a FVD between the ground-truth shots

Original: X ω
Revised: X ω

Original: = { X 1 ω , · · · , X Lω } and the predictions ˆ X t : [Equation]
Revised: = { X 1 ω , · · · , X Lω } and the predictions ˆ X t : t + ω − 1 =

Original: + ω − 1 = { ˆ X 1 t : t + ω − 1 , · · · , ˆ X M t : t + ω − 1 } in a sliding window manner, where L denotes the total number of overlapping shots in the training video and M denotes the number of predicted shots.
Revised: { ˆ X 1 t : t + ω − 1 , · · · , ˆ X M t : t + ω − 1 } in a sliding window manner, where L denotes the total number of overlapping shots in the training video and M denotes the number of predicted shots.

Original: The shot-wise FVD evaluates the synthesis quality in a short interval deﬁned by ω through time .
Revised: The shot-wise FVD evaluates the synthesis quality in a short interval deﬁned by ω through time .

Original: We show in the experiment that it is indeed aligned well with human perception.
Revised: We show in the experiment that it is indeed aligned well with human perception.

Original: We also compute
Revised: We also compute

Original: Inception Score (Salimans et al., 2016) for frame-level quality evaluation, which does not require ground-truths thus appropriate for long-term evaluation.
Revised: Inception Score (Salimans et al., 2016) for frame-level quality evaluation, which does not require ground-truths thus appropriate for long-term evaluation.

Original: Please ﬁnd Appendix A for more details.
Revised: Please ﬁnd Appendix A for more details.

Original: Human evaluation We conduct a user study on Amazon Mechanical Turk (AMT).
Revised: Human evaluation We conduct a user study on Amazon Mechanical Turk (AMT).

Original: To evaluate the quality at different time scales, we presented 5 second (50 frames) videos extracted at 1, 250, 400th predicted frames for all methods and counted their chosen ratio as the best.
Revised: To evaluate the quality at different time scales, we presented 5 second (50 frames) videos extracted at 1, 250, 400th predicted frames for all methods and counted their chosen ratio as the best.

Original: More details are in Appendix A.
Revised: More details are in Appendix A.

Original: 4. R ESULTS ON H UMAN D ANCING S EQUENCES
Revised: 4. R ESULTS ON H UMAN D ANCING S EQUENCES

Original: We ﬁrst present our results on human dancing videos collected from the Web (Wang et al., 2018).
Revised: We ﬁrst present our results on human dancing videos collected from the Web (Wang et al., 2018).

Original: Baselines We compare our method with two state-of-the-art video prediction models.
Revised: Baselines We compare our method with two state-of-the-art video prediction models.

Original: SVGextend (Villegas et al., 2019) is directly operating on RGB frames via a stochastic estimator and serves as our baseline for a non-hierarchical model.
Revised: SVGextend (Villegas et al., 2019) is directly operating on RGB frames via a stochastic estimator and serves as our baseline for a non-hierarchical model.

Original: Following the paper, we employ the largest model with maximum parameters for fair comparison.
Revised: Following the paper, we employ the largest model with maximum parameters for fair comparison.

Original: We employ Villegas et al.
Revised: We employ Villegas et al.

Original: (2017b)
Revised: (2017b)

Original: as a hierarchical model designed speciﬁcally for long-term prediction of human motion using pose (Newell et al., 2016).
Revised: as a hierarchical model designed speciﬁcally for long-term prediction of human motion using pose (Newell et al., 2016).

Original: All models are trained to predict 40 future frames given 5 context frames.
Revised: All models are trained to predict 40 future frames given 5 context frames.

Original: [Table]
Revised: [Table]

Original: Short-term prediction results We evaluate the short-term prediction performance by synthesizing 50 future frames given the 5 context frames and comparing them to groundtruths.
Revised: Short-term prediction results We evaluate the short-term prediction performance by synthesizing 50 future frames given the 5 context frames and comparing them to groundtruths.

Original: Table 1 summarizes the quantitative evaluation results (see Figure A for qualitative results).
Revised: Table 1 summarizes the quantitative evaluation results (see Figure A for qualitative results).

Original: Even in a short prediction interval, our method generates substantially higher quality samples than the baselines in terms of modeling appearance (CSIM), structure (mIoU), and motion (FVD).
Revised: Even in a short prediction interval, our method generates substantially higher quality samples than the baselines in terms of modeling appearance (CSIM), structure (mIoU), and motion (FVD).

Original: This is because the dancing sequences contain rapid and complex variations in dynamics, making prediction task particularly challenging.
Revised: This is because the dancing sequences contain rapid and complex variations in dynamics, making prediction task particularly challenging.

Original: We discuss more detailed analysis across methods in a long-term prediction task below.
Revised: We discuss more detailed analysis across methods in a long-term prediction task below.

Original: Long-term prediction results We evaluate 500 frame prediction results based on shot-wise FVD, frame-wise Inception score, and human evaluation.
Revised: Long-term prediction results We evaluate 500 frame prediction results based on shot-wise FVD, frame-wise Inception score, and human evaluation.

Original: Figure 2 and 3 summarize the quantitative and qualitative comparisons, respectively.
Revised: Figure 2 and 3 summarize the quantitative and qualitative comparisons, respectively.

Original: We notice that the SVG-extend fails to model complex dynamics in dancing even early in prediction.
Revised: We notice that the SVG-extend fails to model complex dynamics in dancing even early in prediction.

Original: This is because the dancing motions involve fast transition and frequent self-occlusions, resulting in catastrophic error propagation through time.
Revised: This is because the dancing motions involve fast transition and frequent self-occlusions, resulting in catastrophic error propagation through time.

Original: On the other hand, as shown in the Inception score, Villegas et al.
Revised: On the other hand, as shown in the Inception score, Villegas et al.

Original: (2017b) produces much reasonable future frames as it exploits human body joints for generation.
Revised: (2017b) produces much reasonable future frames as it exploits human body joints for generation.

Original: However, the deterministic LSTM module is not strong enough to capture complex dancing motions, and ends up generating static postures in the long term that leads to very high FVD scores.
Revised: However, the deterministic LSTM module is not strong enough to capture complex dancing motions, and ends up generating static postures in the long term that leads to very high FVD scores.

Original: In contrast, we observe that our method generates both realistic frames and convincing motions, leading to stable Inception and FVD scores
Revised: In contrast, we observe that our method generates both realistic frames and convincing motions, leading to stable Inception and FVD scores

Original: [Figure]
Revised: [Figure]

Original: context t=1~ t=250~ t=490~
Revised: context t=1~ t=250~ t=490~

Original: small big
Revised: small big

Original: O u r s
Revised: O u r s

Original: [Figure]
Revised: [Figure]

Original: V i ll e g a s e t a l .
Revised: V i ll e g a s e t a l .

Original: [Figure]
Revised: [Figure]

Original: O u r s t
Revised: O u r s t

Original: [Figure]
Revised: [Figure]

Original: t=251~
Revised: t=251~

Original: V i ll e g a s e t a l .
Revised: V i ll e g a s e t a l .

Original: S V G t=1~
Revised: S V G t=1~

Original: S V
Revised: S V

Original: [Figure]
Revised: [Figure]

Original: - e x t e n d in the long-term future.
Revised: - e x t e n d in the long-term future.

Original: The human evaluation (Figure 2(c)) also shows the consistent results that our method outperforms all methods from the short- to long-term prediction.
Revised: The human evaluation (Figure 2(c)) also shows the consistent results that our method outperforms all methods from the short- to long-term prediction.

Original: When our method is applied to predict up to 2040 frames in 128 × 128 resolution (see Figure C in the appendix), we observe that it generates future frames that are convincing and involve diverse dance motions without noticeable quality degradation through time.
Revised: When our method is applied to predict up to 2040 frames in 128 × 128 resolution (see Figure C in the appendix), we observe that it generates future frames that are convincing and involve diverse dance motions without noticeable quality degradation through time.

Original: In addition to the synthesis quality, Figure B illustrates that our method can generate diverse and interesting motions with the stochastic structure estimator.
Revised: In addition to the synthesis quality, Figure B illustrates that our method can generate diverse and interesting motions with the stochastic structure estimator.

Original: 4. R ESULTS ON KITTI B ENCHMARK
Revised: 4. R ESULTS ON KITTI B ENCHMARK

Original: Baselines In addition to SVG-extend, we employ the future segmentation model (Bhattacharyya et al., 2019)
Revised: Baselines In addition to SVG-extend, we employ the future segmentation model (Bhattacharyya et al., 2019)

Original: (Bayes-WD-SL) that predicts future dense label maps as a strong baseline for hierarchical model.
Revised: (Bayes-WD-SL) that predicts future dense label maps as a strong baseline for hierarchical model.

Original: Since it generates only the label maps, we employ the same image generator with ours to produce RGB frames from the predicted label maps.
Revised: Since it generates only the label maps, we employ the same image generator with ours to produce RGB frames from the predicted label maps.

Original: All models are trained to predict 15 future frames given 5 context frames.
Revised: All models are trained to predict 15 future frames given 5 context frames.

Original: Please refer to the appendix for the detailed architecture settings.
Revised: Please refer to the appendix for the detailed architecture settings.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Short-term prediction results Similar to the previous section, we evaluate the short-term prediction quality by synthesizing 50 frames given the 5 frames.
Revised: Short-term prediction results Similar to the previous section, we evaluate the short-term prediction quality by synthesizing 50 frames given the 5 frames.

Original: Table 2 summarize the quantitative results (see Figure D for qualitative analysis).
Revised: Table 2 summarize the quantitative results (see Figure D for qualitative analysis).

Original: We observe that SVG-extend performs worse in terms of structural accuracy (mIoU) and motion (FVD), as it loses the object structure and generates arbitrary pixel dynamics.
Revised: We observe that SVG-extend performs worse in terms of structural accuracy (mIoU) and motion (FVD), as it loses the object structure and generates arbitrary pixel dynamics.

Original: The Bayes-WD-SL performs better in terms of mIoU, as it conditions the frame prediction with the structure estimation.
Revised: The Bayes-WD-SL performs better in terms of mIoU, as it conditions the frame prediction with the structure estimation.

Original: However, as shown high FVD score, it fails to model temporal variations in the structure, largely due to its feedforward architecture.
Revised: However, as shown high FVD score, it fails to model temporal variations in the structure, largely due to its feedforward architecture.

Original: This unstructured motion leads to disastrous failure in a long-term prediction, as we will discuss later.
Revised: This unstructured motion leads to disastrous failure in a long-term prediction, as we will discuss later.

Original: In contrary, our method generates semantically accurate motions and structures, and substantially outperforms the others in all metrics.
Revised: In contrary, our method generates semantically accurate motions and structures, and substantially outperforms the others in all metrics.

Original: Long-term prediction results Figure 4 and 5 summarize the quantitative and qualitative comparisons, respectively.
Revised: Long-term prediction results Figure 4 and 5 summarize the quantitative and qualitative comparisons, respectively.

Original: In all metrics, our method outperforms the other baselines with substantial margins, showing that our method synthesizes both high-quality frames and motion.
Revised: In all metrics, our method outperforms the other baselines with substantial margins, showing that our method synthesizes both high-quality frames and motion.

Original: We notice that
Revised: We notice that

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: C o n t e x t t = 5~ 5 3t =000 ~ 1 0 t = 100 ~ 1 5 t =000 ~ 

Revised: C o n t e x t t = 5~ 5 3t =000 ~ 1 0 t = 100 ~ 1 5 t =000 ~

Original: [Figure]
Revised: [Figure]

Original: SVG-extend simulates arbitrary pixel motions, resulting in relatively constant Inception and shotwise FVD scores through time.
Revised: SVG-extend simulates arbitrary pixel motions, resulting in relatively constant Inception and shotwise FVD scores through time.

Original: However, these unstructured motions lead to rapid destruction in recognizable concepts in the synthesized frames (Figure 5).
Revised: However, these unstructured motions lead to rapid destruction in recognizable concepts in the synthesized frames (Figure 5).

Original: The hierarchical model (Bayes-WDSL) generates more convincing frames in a short-term as shown in lower shot-wise FVD but fails to extrapolate in long-term.
Revised: The hierarchical model (Bayes-WDSL) generates more convincing frames in a short-term as shown in lower shot-wise FVD but fails to extrapolate in long-term.

Original: Importantly, such trends in shot-wise FVD (Figure 4(b)) aligns well with human evaluation results (Figure 4(c)), showing that it is appropriate metric for evaluation of long-term prediction.
Revised: Importantly, such trends in shot-wise FVD (Figure 4(b)) aligns well with human evaluation results (Figure 4(c)), showing that it is appropriate metric for evaluation of long-term prediction.

Original: Interestingly, we observe that the Inception Score of our method increases through time.
Revised: Interestingly, we observe that the Inception Score of our method increases through time.

Original: It is because the long-term prediction by our method often leads to scenes with simple and typical structures, such as a highway, where the image generator can produce more high-quality frames than complicated scenes appearing early in the test videos.
Revised: It is because the long-term prediction by our method often leads to scenes with simple and typical structures, such as a highway, where the image generator can produce more high-quality frames than complicated scenes appearing early in the test videos.

Original: Note that such transition of the scene is still reasonable as it is frequently observed in the training data.
Revised: Note that such transition of the scene is still reasonable as it is frequently observed in the training data.

Original: Nonetheless, we observe that our method generates reasonable sequences through time while maintaining its quality in a reasonable range even in a distant future.
Revised: Nonetheless, we observe that our method generates reasonable sequences through time while maintaining its quality in a reasonable range even in a distant future.

Original: Finally, Figure 6 illustrates the prediction results overfuture frames.
Revised: Finally, Figure 6 illustrates the prediction results overfuture frames.

Original: We observe that the generated sequences are reasonable in both structure and motion, and capture interesting translations of the scenes through time ( e.g., from suburban to rural areas).
Revised: We observe that the generated sequences are reasonable in both structure and motion, and capture interesting translations of the scenes through time ( e.g., from suburban to rural areas).

Original: [Table]
Revised: [Table]

Original: 4. R ESULTS ON C ITYSCAPES DATASET
Revised: 4. R ESULTS ON C ITYSCAPES DATASET

Original: To further evaluate the quality of structure prediction, we compare our structure generator with existing future segmentation methods that directly predict the segmentation map of the future frames.
Revised: To further evaluate the quality of structure prediction we compare our structure generator with existing future segmentation methods that directly predict the segmentation map of the future frames.

Original: Baselines We compare our method with S2S (Luc et al., 2017) and F2F (Luc et al., 2018), which are the state-of-the-arts in the future segmentation literature.
Revised: Baselines We compare our method with S2S (Luc et al., 2017) and F2F (Luc et al., 2018), which are the state-of-the-arts in the future segmentation literature.

Original: S2S is a deterministic model based on fully-convolutional network that predicts future semantic label maps in a multi-scale and autoregressive manner.
Revised: S2S is a deterministic model based on fully-convolutional network that predicts future semantic label maps in a multi-scale and autoregressive manner.

Original: F2F is an extension of S2S to future instance segmentation task by predicting the high-level feature maps of Mask R-CNN (He et al., 2017).
Revised: F2F is an extension of S2S to future instance segmentation task by predicting the high-level feature maps of Mask R-CNN (He et al., 2017).

Original: Similar to KITTI, we use the label maps extracted by the pre-trained semantic segmentation network (Zhu et al., 2019) for the training.
Revised: Similar to KITTI, we use the label maps extracted by the pre-trained semantic segmentation network (Zhu et al., 2019) for the training.

Original: Evaluation We follow the standard evaluation protocols in the literature (Luc et al., 2017; 2018) for quantitative and qualitative comparisons.
Revised: Evaluation We follow the standard evaluation protocols in the literature (Luc et al., 2017; 2018) for quantitative and qualitative comparisons.

Original: For each validation sequence, each model is provided with 4 contexts, and produces up to 29th frame.
Revised: For each validation sequence, each model is provided with 4 contexts, and produces up to 29th frame.

Original: Then, we compute mIoUs between ground-truths and predictions at 20th time-step.
Revised: Then, we compute mIoUs between ground-truths and predictions at 20th time-step.

Original: For fair comparisons with the future instance segmentation model ( i.e., F2F), we follow Luc et al.
Revised: For fair comparisons with the future instance segmentation model ( i.e., F2F), we follow Luc et al.

Original: and measure mIoUs only for moving objects.
Revised: and measure mIoUs only for moving objects.

Original: For S2S and
Revised: For S2S and

Original: F2F, we use publicly available pre-trained models provided by the authors.
Revised: F2F, we use publicly available pre-trained models provided by the authors.

Original: Results Table 3 presents the quantitative evaluation results (See Figure G for qualitative results).
Revised: Results Table 3 presents the quantitative evaluation results (See Figure H for qualitative results).

Original: The two baselines produce blurry predictions, resulting in inaccurate structures and mislabelings.
Revised: The two baselines produce blurry predictions, resulting in inaccurate structures and mislabelings.

Original: These problems have been widely observed in the literature and are attributed to the inability to handle stochasticity (Denton & Fergus, 2018).
Revised: These problems have been widely observed in the literature and are attributed to the inability to handle stochasticity (Denton & Fergus, 2018).

Original: Our method outperforms the two baselines in all classes as it can handle highly stochastic nature of complex driving scenes.
Revised: Our method outperforms the two baselines in all classes as it can handle highly stochastic nature of complex driving scenes.

Original: Figure 7 and Figure H illustrates the long-term prediction results of our full model including the boundary map and image generator on 256 × 512 resolution.
Revised: Figure 7 and Figure J illustrates the long-term prediction results of our full model including the boundary map and image generator on 256 × 512 resolution.

Original: It shows that our method can generate convincing future even in extremely complex and high-resolution videos.
Revised: It shows that our method can generate convincing future even in extremely complex and high-resolution videos.

Original: See Appendix B.3 for more detailed discussion.  
Revised: See Appendix B.3 for more detailed discussion.  A BLATION S TUDY

Revised: As demonstrated in the previous sections, our proposed hierarchical model succeeds at predicting frames persistently up to an arbitrary long-term future.

Original: C ONCLUSION
Revised: To identify the source of this persistency, we conduct an ablation study to analyze the contribution of components in our hierarchical framework. Speciﬁcally, we note that SVG-extend, which suffers from dramatic error propagation through time and was extensively studied in the context of direct RGB frames prediction task (Villegas et al., 2019), performs surprisingly well when applied to predicting future semantic structures. Therefore, we revisit SVG-extend as a sequential dense structure generator and study where the robustness to the errors comes from when it is adapted to predicting semantic label sequences. Following Villegas

Revised: [Figure]

Revised: [Figure]

Revised: time

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: i n c e p t i o n s c o r e

Revised: CNN

Revised: [Figure]

Revised: Ours

Revised: [Figure]

Revised: time

Revised: [Figure]

Revised: F

Revised: V D

Revised: [Figure]

Revised: Ours et al.

Revised: (2019), we ablate (1) stochastic estimation modules (LSTM φ and LSTM ψ ) and (2) recurrent estimation module (LSTM θ ).

Revised: In addition, we investigate the beneﬁts of (3) discretization of predicted label maps and (4) our hierarchical model design that ﬁrst predicts the structures and motions via structure generator and then translates them into pixel-level RGB frames.

Revised: The latter can naturally prevent the errors in pixel-level RGB prediction propagating through the structure predictions.

Revised: To this end, we construct four ablated baselines from our structure generator (SVG-extend) as follows: (1) LSTM where stochastic estimation modules are removed, (2) CNN where all the stochastic and recurrent estimation modules are removed, (3) SVG soft where the discretization process is removed and the model is modiﬁed to observe and estimate continuous logits of semantic segmentation model (Zhu et al., 2019), and (4) SVG rgb prop where an output of the structure generator at time t is translated to RGB frame by the pixel generator and then further processed by the semantic segmentation model (Zhu et al., 2019) before used as an input at time t

Revised: + 1 .

Revised: We train all the models in KITTI Benchmark on the 64 × 64 resolution and compare them with the inception score and shotwise FVD.

Revised: Figure 8 summarize the quantitative results.

Revised: For qualitative comparisons, please refer to

Revised: Stochastic estimation.

Revised: As shown in the quantitative comparison results, deterministic baselines ( CNN and LSTM ) suffer from drastic degradation of shot-wise FVD compared to the stochastic ones.

Revised: Intuitively, this can be understood with the fact that deterministic models tend to seek the most likely solution and thus are prone to deceptive bad local minima, where, once have fallen to the bad solutions, they can hardly have a chance to recover from them ( e.g. trapped in a loop and recursively generates the same structures and motions as shown in Figure G).

Revised: Recurrent estimation.

Revised: Thanks to the ability to capture temporal dependencies among time steps, we ﬁnd that the model with a recurrent module ( e.g LSTM ) (1) predicts more plausible sequences in a short to mid-term (shot-wise FVD at t < 200 ) and (2) generates more recognizable scenes (inception score), compared to the model without the recurrent module ( CNN ).

Revised: Discretization.

Revised: Exploiting the fact that semantic structures are discrete in nature and inducing that bias to our model can be beneﬁcial for predicting accurate structures.

Revised: This is supported by our results where the model without discretization ( SVG soft ) fails at predicting accurate structures with less recognizable scenes ( e.g. shattered structures in Figure G, lowered inception score compared to ours.), albeit it produces plausible textures with motions ( e.g. steady shot-wise FVD).

Revised: Hierarchical prediction.

Revised: We note that errors induced from pixel-level predictions gradually add up through time and have a signiﬁcant effect on predicting plausible structures and motions (shotwise FVD of SVG rgb prop ), which, along with discretization of semantic label maps, demonstrates our architectural design choice for the hierarchical prediction.

Revised:  C ONCLUSION

Original: We proposed a hierarchical approach for persistent video prediction.
Revised: We proposed a hierarchical approach for persistent video prediction.

Original: We revisit the hierarchical model with stochastic estimation of dense label maps and integration of image generator robust against temporal misprediction in pixels and structures.
Revised: We revisit the hierarchical model with stochastic estimation of dense label maps and integration of image generator robust against temporal misprediction in pixels and structures.

Original: Our experimental results show that a carefully designed hierarchical model can learn to synthesize a very long-term future with convincing structures and dynamics even in complex videos.
Revised: Our experimental results show that a carefully designed hierarchical model can learn to synthesize a very long-term future with convincing structures and dynamics even in complex videos.

Original: By scaling up the video prediction to order-ofmagnitudes longer, we believe that our work can help the future research to focus on more challenging problems of learning a long-term dependency and eliminating explicit structure estimation.
Revised: By scaling up the video prediction to order-ofmagnitudes longer, we believe that our work can help the future research to focus on more challenging problems of learning a long-term dependency and eliminating explicit structure estimation.

Original: A PPENDIX A E VALUATION PROTOCOLS
Revised: A PPENDIX A E VALUATION PROTOCOLS

Original: Short-term prediction.
Revised: Short-term prediction.

Original: Following the previous works, we evaluate the short-term prediction quality by comparing the ground-truth and predicted frames in test videos.
Revised: Following the previous works, we evaluate the short-term prediction quality by comparing the ground-truth and predicted frames in test videos.

Original: We employ three metrics:
Revised: We employ three metrics:

Original: VGG cosine similarity (CSIM), mean Intersection-over-Union (mIoU), and Fr´echet Video Distance (FVD) (Unterthiner et al., 2018).
Revised: VGG cosine similarity (CSIM), mean Intersection-over-Union (mIoU), and Fr´echet Video Distance (FVD) (Unterthiner et al., 2018).

Original: The ﬁrst metric measures the frame-wise perceptual similarity using VGG features, while the second one measures the structure-level similarity using the label maps extracted by the pre-trained semantic segmentation network (Zhu et al., 2019).
Revised: The ﬁrst metric measures the frame-wise perceptual similarity using VGG features, while the second one measures the structure-level similarity using the label maps extracted by the pre-trained semantic segmentation network (Zhu et al., 2019).

Original: To handle stochasticity, we sample 100 predictions for each test clip and report the scores of the best scoring predictions.
Revised: To handle stochasticity, we sample 100 predictions for each test clip and report the scores of the best scoring predictions.

Original: For the video-level evaluation, we adopt FVD that measures a Fr´echet distance between the ground-truth videos and the generated ones in a video representation space.
Revised: For the video-level evaluation, we adopt FVD that measures a Fr´echet distance between the ground-truth videos and the generated ones in a video representation space.

Original: We use an Inﬂated 3D Convnet (I3D) (Carreira & Zisserman, 2017) network pre-trained on Kinetics 400 dataset (Carreira & Zisserman, 2017) to obtain representations on videos, and use all 100 predictions to compute
Revised: We use an Inﬂated 3D Convnet (I3D) (Carreira & Zisserman, 2017) network pre-trained on Kinetics 400 dataset (Carreira & Zisserman, 2017) to obtain representations on videos, and use all 100 predictions to compute

Original: FVD against ground-truth sequences.
Revised: FVD against ground-truth sequences.

Original: Long-term prediction.
Revised: Long-term prediction.

Original: To evaluate the shot-wise FVD, we employ the I3D Convnet as the shortterm prediction.
Revised: To evaluate the shot-wise FVD, we employ the I3D Convnet as the shortterm prediction.

Original: For the models trained on KITTI dataset, we use a ResNet-18 (He et al., 2016) pre-trained on Places 365 (Zhou et al., 2017) to compute the Inception Score.
Revised: For the models trained on KITTI dataset, we use a ResNet-18 (He et al., 2016) pre-trained on Places 365 (Zhou et al., 2017) to compute the Inception Score.

Original: For the models trained on the Human Dancing dataset, we use the Inception V3 (Szegedy et al., 2015) pre-trained on ImageNet (Russakovsky et al., 2015) for measuring the Inception Score.
Revised: For the models trained on the Human Dancing dataset, we use the Inception V3 (Szegedy et al., 2015) pre-trained on ImageNet (Russakovsky et al., 2015) for measuring the Inception Score.

Original: Human evaluation.
Revised: Human evaluation.

Original: We conducted a human evaluation using Amazon Mechanical Turk (AMT).
Revised: We conducted a human evaluation using Amazon Mechanical Turk (AMT).

Original: For evaluating both the short- and long-term prediction performances, we generated 500 predicted frames for all compared methods (SVG-Extend, Villegas et al.
Revised: For evaluating both the short- and long-term prediction performances, we generated 500 predicted frames for all compared methods (SVG-Extend, Villegas et al.

Original: (2019), and ours).
Revised: (2019), and ours).

Original: We extracted 5second clips (50 frames encoded in 10 fps) at the 1st, 250th, and 400th frames, and asked users to rank the videos according to perceived quality and realism of the videos.
Revised: We extracted 5second clips (50 frames encoded in 10 fps) at the 1st, 250th, and 400th frames, and asked users to rank the videos according to perceived quality and realism of the videos.

Original: On the KITTI dataset, we conducted the evaluation over the 133 unique sequences present in the validation set.
Revised: On the KITTI dataset, we conducted the evaluation over the 133 unique sequences present in the validation set.

Original: On the Human Dancing dataset, we conducted evaluation over 97 videos used for validation.
Revised: On the Human Dancing dataset, we conducted evaluation over 97 videos used for validation.

Original: Responses for each video was aggregated from 5 independent human evaluators for each clip.
Revised: Responses for each video was aggregated from 5 independent human evaluators for each clip.

Original: B M ORE D ETAILS OF E XPERIMENT R ESULTS
Revised: B M ORE D ETAILS OF E XPERIMENT R ESULTS

Original: This section provides additional experiment details and results that could not be accommodated in the main paper due to space restriction.
Revised: This section provides additional experiment details and results that could not be accommodated in the main paper due to space restriction.

Original: Please ﬁnd our anonymous website to assess full videos for qualitative analysis: https://bit.ly/2EyDSem.
Revised: Please ﬁnd our anonymous website to assess full videos for qualitative analysis: https://bit.ly/2EyDSem.

Original: B.
Revised: B.

Original: H UMAN D ANCING DATASET B
Revised: H UMAN D ANCING DATASET B

Original: . D ATASET
Revised: . D ATASET

Original: To evaluate if our model can learn complex and highly structured motion, we used videos of human dancing 2 .
Revised: To evaluate if our model can learn complex and highly structured motion, we used videos of human dancing 2 .

Original: We construct this dataset by crawling a set of videos from the web containing a single person covering various dance moves.
Revised: We construct this dataset by crawling a set of videos from the web containing a single person covering various dance moves.

Original: Following the pre-processing used in Wang et al.
Revised: Following the pre-processing used in Wang et al.

Original: (2018), we crop the center of the video and resize it to square frames.
Revised: (2018), we crop the center of the video and resize it to square frames.

Original: We collect approximately 240 videos in total for training.
Revised: We collect approximately 240 videos in total for training.

Original: For structure representation, we use the body parts segmentation obtained by
Revised: For structure representation, we use the body parts segmentation obtained by

Original: DensePose (Alp G¨uler et al., 2018).
Revised: DensePose (Alp G¨uler et al., 2018).

Original: We sub-sample each sequence by a factor of two.
Revised: We sub-sample each sequence by a factor of two.

Original: B.1. I MPLEMENTATION D ETAILS
Revised: B.1. I MPLEMENTATION D ETAILS

Original: SVG-extend (Villegas et al., 2019).
Revised: SVG-extend (Villegas et al., 2019).

Original: We use the same architectural and hyperparameter settings described in Section C.1 except that SVG-extend in this case is trained to model pixel-level RGB
Revised: We use the same architectural and hyperparameter settings described in Section C.1 except that SVG-extend in this case is trained to model pixel-level RGB

Original: We videos sequences directly as an unsupervised non-hierarchical baseline, unlike ours that works on semantic segmentation sequences as sequential dense structure generator.
Revised: We videos sequences directly as an unsupervised non-hierarchical baseline, unlike ours that works on semantic segmentation sequences as sequential dense structure generator.

Original: Villegas et al.
Revised: Villegas et al.

Original: (2017b).
Revised: (2017b).

Original: This method also adopts a hierarchical approach that leverages human keypoints as additional supervision for predicting future video frames.
Revised: This method also adopts a hierarchical approach that leverages human keypoints as additional supervision for predicting future video frames.

Original: Speciﬁcally, this method consists of ﬁve modules, namely (1) the Hourglass network (Newell et al., 2016) that estimates human keypoints from pixel-level images, (2) a single layer deterministic encoder-decoder LSTM withhidden units and tanh activations that predicts a sequence of future human keypoints given a few observed ones, (3) an image generator that synthesizes pixel-level image frames given the predicted and observed keypoints based on visual-analogy scheme, (4) an image discriminator for adversarial learning, and (5) the AlexNet (Krizhevsky et al., 2012) pre-trained on ImageNet (Russakovsky et al., 2015) that is used for perceptual loss (Johnson et al., 2016).
Revised: Speciﬁcally, this method consists of ﬁve modules, namely (1) the Hourglass network (Newell et al., 2016) that estimates human keypoints from pixel-level images, (2) a single layer deterministic encoder-decoder LSTM withhidden units and tanh activations that predicts a sequence of future human keypoints given a few observed ones, (3) an image generator that synthesizes pixel-level image frames given the predicted and observed keypoints based on visual-analogy scheme, (4) an image discriminator for adversarial learning, and (5) the AlexNet (Krizhevsky et al., 2012) pre-trained on ImageNet (Russakovsky et al., 2015) that is used for perceptual loss (Johnson et al., 2016).

Original: For training this model on our human dancing dataset, we directly follow the setting used in the paper (Villegas et al., 2017b), and use the publicly available code 3 provided by the authors.
Revised: For training this model on our human dancing dataset, we directly follow the setting used in the paper (Villegas et al., 2017b), and use the publicly available code 3 provided by the authors.

Original: B.1. Q UALITATIVE R ESULTS
Revised: B.1. Q UALITATIVE R ESULTS

Original: Short-term prediction.
Revised: Short-term prediction.

Original: We show in Figure A the qualitative comparisons of short-term predictions across models, which corresponds to the Table 1 in the main paper.
Revised: We show in Figure A the qualitative comparisons of short-term predictions across models, which corresponds to the Table 1 in the main paper.

Original: As shown in the ﬁgure,
Revised: As shown in the ﬁgure,

Original: SVG-extend starts to fail in producing plausible samples in a very short-term ( t = 8 ∼ ), which can be attributed to its implicit modeling of spatio-temporal variations of structures and appearances.
Revised: SVG-extend starts to fail in producing plausible samples in a very short-term ( t = 8 ∼ ), which can be attributed to its implicit modeling of spatio-temporal variations of structures and appearances.

Original: Villegas et al.
Revised: Villegas et al.

Original: (2017b) improves the quality of both structures and appearances as well as extrapolation through time via the explicit modeling of structures followed by the translation of predicted structures into pixel-level frames in a hierarchical manner.
Revised: (2017b) improves the quality of both structures and appearances as well as extrapolation through time via the explicit modeling of structures followed by the translation of predicted structures into pixel-level frames in a hierarchical manner.

Original: However, it still fails to produce realistic pixel-level frames due to the adoption of (1) coarse-grained human landmarks and deterministic model which are insufﬁcient to fully describe the ﬁne details of stochastic and structured motions, and (2) image synthesis performed by transforming the last observed frames into the future, which may not be appropriate when there exist large appearance changes due to self-occlusion and movement.
Revised: However, it still fails to produce realistic pixel-level frames due to the adoption of (1) coarse-grained human landmarks and deterministic model which are insufﬁcient to fully describe the ﬁne details of stochastic and structured motions, and (2) image synthesis performed by transforming the last observed frames into the future, which may not be appropriate when there exist large appearance changes due to self-occlusion and movement.

Original: On the other hand, our approach can produce high-ﬁdelity predictions with plausible motions thanks to the stochastic and comprehensive estimation of scene dynamics under the semantic segmentation space.
Revised: On the other hand, our approach can produce high-ﬁdelity predictions with plausible motions thanks to the stochastic and comprehensive estimation of scene dynamics under the semantic segmentation space.

Original: Diversity of samples.
Revised: Diversity of samples.

Original: Figure B illustrates the multiple prediction results conditioned on the same context frames.
Revised: Figure B illustrates the multiple prediction results conditioned on the same context frames.

Original: We observe that our model produces diverse and plausible future frames, where such stochasticity mainly stems from the structure generator.
Revised: We observe that our model produces diverse and plausible future frames, where such stochasticity mainly stems from the structure generator.

Original: Long-term prediction results.
Revised: Long-term prediction results.

Original: Figure C illustrates the prediction results on a very long-term future (over 2500 frames).We observe that our method generates convincing frames without particular quality degradation and error propagation over a very long-term future.
Revised: Figure C illustrates the prediction results on a very long-term future (over 2500 frames).We observe that our method generates convincing frames without particular quality degradation and error propagation over a very long-term future.

Original: We also observe that the predicted frames contain interesting dance motions, such as the ones with horizontal translation ( t = 1000 ∼ 1040 ), dynamics on legs ( t = 500 ∼ 540 ) and arms ( t = 2000 ∼ 2040 ), etc.
Revised: We also observe that the predicted frames contain interesting dance motions, such as the ones with horizontal translation ( t = 1000 ∼ 1040 ), dynamics on legs ( t = 500 ∼ 540 ) and arms ( t = 2000 ∼ 2040 ), etc.

Original: t=43 t= t= t= t= t= t= t= t=43 t=1 t=

Revised: t=43 t= t= t= t= t= t= t= t=43 t=1 t=

Original: t=15 t=22 t=29 t=
Revised: t=15 t=22 t=29 t=

Original: Ground-Truth
Revised: Ground-Truth

Original: SVG-Extend
Revised: SVG-Extend

Original: Villegas et al.
Revised: Villegas et al.

Original: Ours t=
Revised: Ours t=

Original: Ground-Truth
Revised: Ground-Truth

Original: SVG-Extend
Revised: SVG-Extend

Original: Villegas et al.
Revised: Villegas et al.

Original: Ours
Revised: Ours

Original: Ground-Truth
Revised: Ground-Truth

Original: Ours
Revised: Ours

Original: Ours
Revised: Ours

Original: Villegas et al.
Revised: Villegas et al.

Original: SVG-Extend SVG-Extend
Revised: SVG-Extend SVG-Extend

Original: Villegas et al.
Revised: Villegas et al.

Original: Ground-Truth ㅋㅋ 내가 할게 given the same context frames, we sample 100 predictions for each model and show their best predictions in terms of the cosine similarity metric best
Revised: Ground-Truth ㅋㅋ 내가 할게 given the same context frames, we sample 100 predictions for each model and show their best predictions in terms of the cosine similarity metric best

Original: , we show the semantic label maps predicted by Densepose (Alp G¨uler et al., 2018).
Revised: , we show the semantic label maps predicted by Densepose (Alp G¨uler et al., 2018).

Original: [Figure]
Revised: [Figure]

Original: ··· t=
Revised: ··· t=

Original: [Figure]
Revised: [Figure]

Original: t= t= t=
Revised: t= t= t=

Original: [Figure]
Revised: [Figure]

Original: Ground-Truth t=6 t=12 ··· t=18 t=48 t= t=
Revised: Ground-Truth t=6 t=12 ··· t=18 t=48 t= t=

Original: Sample
Revised: Sample

Original: Sample Sample
Revised: Sample Sample

Original: Ground-Truth
Revised: Ground-Truth

Original: Sample Sample
Revised: Sample Sample

Original: Sample···
Revised: Sample···

Original: [Figure]
Revised: [Figure]

Original: C o n t e x t t = ~ 5 t = ~ 13t =  ~ 13t =  ~0 

Revised: C o n t e x t t = ~ 5 t = ~ 13t =  ~ 13t =  ~0

Original: [Figure]
Revised: [Figure]

Original: t =5~ 

Revised: t =5~

Original: B. KITTI BENCHMARK B.2. D ATASET
Revised: B. KITTI BENCHMARK B.2. D ATASET

Original: An ability to cope with partial observability and high stochasticity is highly important in the video prediction task.
Revised: An ability to cope with partial observability and high stochasticity is highly important in the video prediction task.

Original: To evaluate a model under this criterion, we adopt KITTI dataset (Geiger et al., 2013) consisting of driving scenes captured by a front-view camera in residential and rural areas.
Revised: To evaluate a model under this criterion, we adopt KITTI dataset (Geiger et al., 2013) consisting of driving scenes captured by a front-view camera in residential and rural areas.

Original: Since the segmentation labels are available for only a small subset of frames in this dataset ( frames in total), we extract the labels using the semantic segmentation model (Zhu et al., 2019) pre-trained on still images (Cordts et al., 2016; Geiger et al., 2013) for both training (the entire sequences) and testing (only context frames).
Revised: Since the segmentation labels are available for only a small subset of frames in this dataset ( frames in total), we extract the labels using the semantic segmentation model (Zhu et al., 2019) pre-trained on still images (Cordts et al., 2016; Geiger et al., 2013) for both training (the entire sequences) and testing (only context frames).

Original: We follow the pre-processing and train/test splits used in Lotter et al.
Revised: We follow the pre-processing and train/test splits used in Lotter et al.

Original: (2017), and sub-sample each sequence by a factor of two.
Revised: (2017), and sub-sample each sequence by a factor of two.

Original: B.2. I MPLEMENTATION D ETAILS
Revised: B.2. I MPLEMENTATION D ETAILS

Original: SVG-extend (Villegas et al., 2019).
Revised: SVG-extend (Villegas et al., 2019).

Original: Same as Section B.1.2, we follow the implementation details speciﬁed in Section C.1 for training this model on KITTI Benchmark dataset.
Revised: Same as Section B.1.2, we follow the implementation details speciﬁed in Section C.1 for training this model on KITTI Benchmark dataset.

Original: Bayes-WD-SL (Bhattacharyya et al., 2019).
Revised: Bayes-WD-SL (Bhattacharyya et al., 2019).

Original: This work is a future segmentation method that exploits odometry information as additional supervision and enables stochastic estimation of uncertain futures through the Bayesian approach.
Revised: This work is a future segmentation method that exploits odometry information as additional supervision and enables stochastic estimation of uncertain futures through the Bayesian approach.

Original: On top of a ResNet-based deterministic video prediction model (Luc et al., 2017), this work introduces a novel Bayesian formulation to better handle inherent uncertainties when predicting future frames.
Revised: On top of a ResNet-based deterministic video prediction model (Luc et al., 2017), this work introduces a novel Bayesian formulation to better handle inherent uncertainties when predicting future frames.

Original: Speciﬁcally, this method introduces (1) weight-dropout (WD) which assumes a ﬁnite set of weights for approximating model space, followed by Bernoulli variational distribution on those ﬁnite set of weights, and (2) synthetic likelihoods (SL) which mitigates the constraint on each model to explain all the data points and thus allows sampled models to be diverse to better deal with the uncertainties.
Revised: Speciﬁcally, this method introduces (1) weight-dropout (WD) which assumes a ﬁnite set of weights for approximating model space, followed by Bernoulli variational distribution on those ﬁnite set of weights, and (2) synthetic likelihoods (SL) which mitigates the constraint on each model to explain all the data points and thus allows sampled models to be diverse to better deal with the uncertainties.

Original: For training this model on KITTI Benchmark, we follow the same settings used in the original paper for the experiments on Cityscapes dataset except that we omit odometry supervision for fair comparisons.
Revised: For training this model on KITTI Benchmark, we follow the same settings used in the original paper for the experiments on Cityscapes dataset except that we omit odometry supervision for fair comparisons.

Original: We use the publicly available code.
Revised: We use the publicly available code.

Original: B.2. Q UALITATIVE R ESULTS
Revised: B.2. Q UALITATIVE R ESULTS

Original: Short-term prediction.
Revised: Short-term prediction.

Original: Figure D illustrates the qualitative comparisons for short-term prediction, which correspond to Table 2 in the main paper.
Revised: Figure D illustrates the qualitative comparisons for short-term prediction, which correspond to Table 2 in the main paper.

Original: As shown in Figure D, SVG-extend performs very well at the very beginning of the prediction ( t = 1 ∼ 40 ), since it directly exploits the content from the last context frame via temporal skip-connection.
Revised: As shown in Figure D, SVG-extend performs very well at the very beginning of the prediction ( t = 1 ∼ 40 ), since it directly exploits the content from the last context frame via temporal skip-connection.

Original: However, its prediction quality degrades rapidly through time, as it loses the object structure and simulates arbitrary, unstructured pixel-wise motions in the long-term.
Revised: However, its prediction quality degrades rapidly through time, as it loses the object structure and simulates arbitrary, unstructured pixel-wise motions in the long-term.

Original: The future segmentation baseline (Bayes-WD-SL) preserves a much reasonable structure, since its generation is conditioned on the estimated label maps similar to ours.
Revised: The future segmentation baseline (Bayes-WD-SL) preserves a much reasonable structure, since its generation is conditioned on the estimated label maps similar to ours.

Original: However, it fails to model structures in motion and thus loses such structures in the end.
Revised: However, it fails to model structures in motion and thus loses such structures in the end.

Original: These results demonstrate that both the unsupervised and hierarchical baselines fail to extrapolate to the long-term horizon than the one used during the training.
Revised: These results demonstrate that both the unsupervised and hierarchical baselines fail to extrapolate to the long-term horizon than the one used during the training.

Original: Compared to these baselines, our method generates semantically more accurate motions and structures, and substantially outperforms the others especially in the latter prediction steps.
Revised: Compared to these baselines, our method generates semantically more accurate motions and structures, and substantially outperforms the others especially in the latter prediction steps.

Original: Diversity of samples.
Revised: Diversity of samples.

Original: We show diverse prediction results in Figure E, which are conditioned on the same context frames.
Revised: We show diverse prediction results in Figure E, which are conditioned on the same context frames.

Original: As can be seen in the ﬁgure, predicted sequences not only are accurate in a near future ( t = 18 ), but also show diverse motion and scene dynamics through time.
Revised: As can be seen in the ﬁgure, predicted sequences not only are accurate in a near future ( t = 18 ), but also show diverse motion and scene dynamics through time.

Original: For example, we can see diverse scene dynamics such as emergence of novel vehicles ( t = 54 ∼ 144 ), as well as transition to the novel scene ( t = 144 ∼ 180 ).
Revised: For example, we can see diverse scene dynamics such as emergence of novel vehicles ( t = 54 ∼ 144 ), as well as transition to the novel scene ( t = 144 ∼ 180 ).

Original: Long-term prediction results.
Revised: Long-term prediction results.

Original: Figure F illustrates the long-term prediction results over 2, frames in 256 × 256 resolution, which corresponds to Figure 6 in the main paper.
Revised: Figure F illustrates the long-term prediction results over 2, frames in 256 × 256 resolution, which corresponds to Figure 6 in the main paper.

Original: We observe that the generated sequences are reasonable in both structure and motion.
Revised: We observe that the generated sequences are reasonable in both structure and motion.

Original: Interestingly, unrolling the prediction over a very long-term leads to very interesting transitions of the scene, such as translation from suburban ( t < 1035 ) to rural areas ( t > 1500 ), which cannot be captured in a short-term period.
Revised: Interestingly, unrolling the prediction over a very long-term leads to very interesting transitions of the scene, such as translation from suburban ( t < 1035 ) to rural areas ( t > 1500 ), which cannot be captured in a short-term period.

Original: We believe that these results illustrate well why the long-term prediction is important
Revised: We believe that these results illustrate well why the long-term prediction is important.

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t= t= t= t= t= t=50 t=
Revised: t= t= t= t= t= t=50 t=

Original: [Figure]
Revised: [Figure]

Original: given the same context frames, we sample 100 predictions for each model and show their best predictions in terms of the cosine similarity metric.
Revised: given the same context frames, we sample 100 predictions for each model and show their best predictions in terms of the cosine similarity metric.

Original: (bottom pane) for each best prediction, we show the semantic label maps predicted by pre-trained semantic segmentation network (Zhu et al., 2019). 
Revised: (bottom pane) for each best prediction, we show the semantic label maps predicted by pre-trained semantic segmentation network (Zhu et al., 2019). [Figure]

Original: [Figure]
Revised: [Figure]

Revised: t

Original: [Figure]
Revised: [Figure]

Original: t=t= t
Revised: t= t

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Ground-Truth t=6 t=12 ··· t=18 t=48 t= t
Revised: Ground-Truth t=6 t=12 ··· t=18 t=48 t= t

Original: [Figure]
Revised: [Figure]

Original: Sample
Revised: Sample

Original: Sample Sample
Revised: Sample Sample

Original: Ground
Revised: Ground

Original: [Figure]
Revised: [Figure]

Original: Truth
Revised: Truth

Original: Sample Sample
Revised: Sample Sample

Original: Sample···
Revised: Sample···

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Context t=500~ t=1000~ t=1500~ t=2000~
Revised: Context t=500~ t=1000~ t=1500~ t=2000~

Revised: [Figure]

Revised: M

Revised: S V G

Revised: S o f t

Revised: S V G r g b p r o p

Original:  B. C ITYSCAPES DATASET B.3. D ATASET
Revised: O u r s B. C ITYSCAPES DATASET B.3. D ATASET

Original: To evaluate the quality of forecasting dense labels by the structure generator, we employ the
Revised: To evaluate the quality of forecasting dense labels by the structure generator, we employ the

Original: Cityscapes (Cordts et al., 2016), a widely-used benchmark in future segmentation tasks.
Revised: Cityscapes (Cordts et al., 2016), a widely-used benchmark in future segmentation tasks.

Original: The dataset consists of 2,975 training, 1,525 testing, and 500 validation sequences where each sequence is 30frame-long and has ground-truth segmentation labels only in the 20th frame.
Revised: The dataset consists of 2,975 training, 1,525 testing, and 500 validation sequences where each sequence is 30frame-long and has ground-truth segmentation labels only in the 20th frame.

Original: Similar to KITTI dataset, we employ the pre-trained segmentation network (Zhu et al., 2019) to generate segmentation labels of 128 × 256 image resolution for training our SVG-extend.
Revised: Similar to KITTI dataset, we employ the pre-trained segmentation network (Zhu et al., 2019) to generate segmentation labels of 128 × 256 image resolution for training our SVG-extend.

Original: Following (Luc et al., 2017; 2018), we subsample frames by a factor of three for each sequence and test all models on 128 ×image resolution.
Revised: Following (Luc et al., 2017; 2018), we subsample frames by a factor of three for each sequence and test all models on 128 ×image resolution.

Original: B.3. I MPLEMENTATION D ETAILS
Revised: B.3. I MPLEMENTATION D ETAILS

Original: S2S (Luc et al., 2017).
Revised: S2S (Luc et al., 2017).

Original: We use the best performing S2S model proposed by the authors, namely S2Sdil, AR, ﬁne-tune , which improves the performance of the single-frame prediction baseline (S2S) by introducing the followings: (1) dilated convolution to enlarge receptive ﬁelds of the model.
Revised: We use the best performing S2S model proposed by the authors, namely S2Sdil, AR, ﬁne-tune , which improves the performance of the single-frame prediction baseline (S2S) by introducing the followings: (1) dilated convolution to enlarge receptive ﬁelds of the model.

Original: (2) autoregressive ﬁne-tuning that uses the prediction at time t as an input to predict the future frame at time t + 1 during training, which enables the single-frame prediction model to predict an arbitrary number of frames into the future.
Revised: (2) autoregressive ﬁne-tuning that uses the prediction at time t as an input to predict the future frame at time t + 1 during training, which enables the single-frame prediction model to predict an arbitrary number of frames into the future.

Original: We use a publicly available pre-trained model 4 provided by the authors, which was trained using the predicted semantic segmentation maps by Dilation network (Yu & Koltun, 2016) on 128 × 256 image resolution.
Revised: We use a publicly available pre-trained model 4 provided by the authors, which was trained using the predicted semantic segmentation maps by Dilation network (Yu & Koltun, 2016) on 128 × 256 image resolution.

Original: F2F (Luc et al., 2018).
Revised: F2F (Luc et al., 2018).

Original: This method targets a future instance segmentation task, which extends future segmentation method S2S
Revised: This method targets a future instance segmentation task, which extends future segmentation method S2S

Original: (Luc et al., 2017) by formulating the task of the model as predicting intermediate feature maps of Mask R-CNN (He et al., 2017) on the target future instance segmentation masks.
Revised: (Luc et al., 2017) by formulating the task of the model as predicting intermediate feature maps of Mask R-CNN (He et al., 2017) on the target future instance segmentation masks.

Original: We use a publicly available pre-trained model and code 5 provided by the authors, which was ﬁrst trained on MS-COCO dataset (Lin et al., 2014) and then ﬁnetuned on the Cityscapes images of 1024 × 2048 .
Revised: We use a publicly available pre-trained model and code 5 provided by the authors, which was ﬁrst trained on MS-COCO dataset (Lin et al., 2014) and then ﬁnetuned on the Cityscapes images of 1024 × 2048 .

Original: Following (Luc et al., 2017; 2018), we down-sample the predicted future instance segmentation masks to 128 × 256 resolution for the evaluation.
Revised: Following (Luc et al., 2017; 2018), we down-sample the predicted future instance segmentation masks to 128 × 256 resolution for the evaluation.

Original: B.3. Q UALITATIVE R ESULTS
Revised: B.3. Q UALITATIVE R ESULTS

Original: Qualitative comparison.
Revised: Qualitative comparison.

Original: We present the qualitative comparisons of our method to existing future segmentation models, which correspond to Table 3 in the main paper.
Revised: We present the qualitative comparisons of our method to existing future segmentation models, which correspond to Table 3 in the main paper.

Original: As shown in the table, our method outperforms the two baselines in all classes, thanks to its ability to handle the highly stochastic nature of complex driving scenes.
Revised: As shown in the table, our method outperforms the two baselines in all classes, thanks to its ability to handle the highly stochastic nature of complex driving scenes.

Original: We also compare qualitative samples of each model in Figure G. As shown in the ﬁgure, our model predicts convincing structures and their temporal dynamics are accurate.
Revised: We also compare qualitative samples of each model in Figure H. As shown in the ﬁgure, our model predicts convincing structures and their temporal dynamics are accurate.

Original: On the other hand, the two baselines produce blurry predictions, resulting in (1) inaccurate structures (S2S at all timestamps and F2F at 17-29th timestamps), and (2) multiple semantics in a single structure (S2S at all timestamps and F2F at 23-29th timestamps).
Revised: On the other hand, the two baselines produce blurry predictions, resulting in (1) inaccurate structures (S2S at all timestamps and F2F at 17-29th timestamps), and (2) multiple semantics in a single structure (S2S at all timestamps and F2F at 23-29th timestamps).

Original: These blurred prediction problems have been widely observed in the literature and attributed to the inability of deterministic models to handle stochasticity in data (Denton & Fergus, 2018).
Revised: These blurred prediction problems have been widely observed in the literature and attributed to the inability of deterministic models to handle stochasticity in data (Denton & Fergus, 2018).

Original: Short-term prediction.
Revised: Short-term prediction.

Original: To evaluate the future segmentation results in Section 4.4, we extractfuture semantic or instance segmentations ahead (14, 17, 20, 23, 26, and 29th) from each model given 4 contexts (2, 5, 8, and 11th).
Revised: To evaluate the future segmentation results in Section 4.4, we extractfuture semantic or instance segmentations ahead (14, 17, 20, 23, 26, and 29th) from each model given 4 contexts (2, 5, 8, and 11th).

Original: Then, following (Luc et al., 2018), we compute per-class and mean IoUs for 8 moving object categories only: person, rider, car, truck bus, train, motorcycle, bicycle .
Revised: Then, following (Luc et al., 2018), we compute per-class and mean IoUs for 8 moving object categories only: person, rider, car, truck bus, train, motorcycle, bicycle .

Original: Long-term video prediction.
Revised: Long-term video prediction.

Original: Figure H illustrates the qualitative results of video prediction results on the Cityscapes dataset.
Revised: Figure J illustrates the qualitative results of video prediction results on the Cityscapes dataset.

Original: As the Cityscapes contains many occluding objects, we employ the extended structure generator with instance boundary prediction to improve the frame prediction quality.
Revised: As the Cityscapes contains many occluding objects, we employ the extended structure generator with instance boundary prediction to improve the frame prediction quality.

Original: We show in Figure H the predicted sequences up to 96 future frames conditioned oncontext frames.
Revised: We show in Figure J the predicted sequences up to 96 future frames conditioned oncontext frames.

Original: As shown in the ﬁgure, our method can easily incorporate the additional structures ( i.e. instance boundary map), and can be generalized to complex and high-resolution sequences contextx_t+0 predictionx_t+3 predictionx_t+6 predictionx_t+
Revised: As shown in the ﬁgure, our method can easily incorporate the additional structures ( i.e. instance boundary map), and can be generalized to complex and high-resolution sequences contextx_t+0 predictionx_t+3 predictionx_t+6 predictionx_t+

Original: S2S
Revised: S2S

Original: F2F
Revised: F2F

Original: Oracle
Revised: Oracle

Original: Ours
Revised: Ours

Original: Batch 52 / frankfurt_000001_
Revised: Batch 52 / frankfurt_000001_

Original: 14th
Revised: 14th

Original: 17th 20th
Revised: 17th 20th

Original: Oracle [49]
Revised: Oracle [49]

Original: S2S
Revised: S2S

Original: Ours
Revised: Ours

Original: F2F
Revised: F2F

Original: Oracle [49]
Revised: Oracle [49]

Original: S2S
Revised: S2S

Original: Ours
Revised: Ours

Original: F2F 14th 17th 20th
Revised: F2F 14th 17th 20th

Original: Ground-Truth
Revised: Ground-Truth

Original: Ours
Revised: Ours

Original: F2F [30]
Revised: F2F [30]

Original: S2S [27]
Revised: S2S [27]

Original: Oracle [33] 17th 14th 20th 23th
Revised: Oracle [33] 17th 14th 20th 23th

Original: 26th 29th
Revised: 26th 29th

Original: GT
Revised: GT

Original: Ours
Revised: Ours

Original: F2F
Revised: F2F

Original: S2S
Revised: S2S

Original: Oracle [49] 17th 14th 20th 23th
Revised: Oracle [49] 17th 14th 20th 23th

Original: 26th 29th to predict instance-wise structures into the long future without error propagations ( e.g. blurred predictions in the structure space).
Revised: 26th 29th to predict instance-wise structures into the long future without error propagations ( e.g. blurred predictions in the structure space).

Original: On the other hand, we also observe some undesirable biases in the predicted motion dynamics in the structure space, such as static or cyclic motions of the structures (e.g.red dashed boxes in Figure H).
Revised: On the other hand, we also observe some undesirable biases in the predicted motion dynamics in the structure space, such as static or cyclic motions of the structures (e.g.red dashed boxes in Figure J).

Original: We notice that it is because the training sequences in Cityscapes datasets are considerably short ( i.e. 30 frames), which makes the structure generator favor short-term dependencies in the prediction ( e.g. short-term skip connections).
Revised: We notice that it is because the training sequences in Cityscapes datasets are considerably short ( i.e. 30 frames), which makes the structure generator favor short-term dependencies in the prediction ( e.g. short-term skip connections).

Original: It leads to biases in the model and artifacts in long-term extrapolation, such as static motions in the distant objects ( e.g. buildings and trees far away from the camera), ﬁxed content ( e.g. repeatedly generated objects), and etc. , which are not observed in the other datasets with reasonably long training sequences.
Revised: It leads to biases in the model and artifacts in long-term extrapolation, such as static motions in the distant objects ( e.g. buildings and trees far away from the camera), ﬁxed content ( e.g. repeatedly generated objects), and etc. , which are not observed in the other datasets with reasonably long training sequences . B.3. T HE EFFECT OF INSTANCE BOUNDARY MAP PREDICTION [Figure] PSNR( ↑ )

Revised: SSIM( ↑ )

Revised: CSIM( ↑ )

Original: 71th 128th 185th 242th
Revised: With G edge 20. Without G edge 19. To analyze the effect of the instance boundary map produced by G edge , we compare the prediction performance of our model with and without G edge . Speciﬁcally, we extract 6 future frames ahead (14, 17, 20, 23, 26, and 29-th) given 4 context frames (1, 4, 7, and 11-th) from the models, and compare frame-wise evaluation results measured by Peak Signal-to-noise ratio (PSNR), Structural Similarity (SSIM), and VGG cosine similarity (CSIM).

Revised: We average the measured scores across the test sequences and timestamps.

Revised: Table A

Original: [Figure]
Revised: and Figure

Original: ObservedFrame
Revised: I show the quantitative and qualitative results, respectively. As shown in the table, we observe marginal improvements in those metrics due to the relatively short length of Cityscapes sequences (up to 30 frames). However, as can be seen in the ﬁgure, we ﬁnd that the generated frames contain more clear instance boundaries with G edge especially in crowded scenes. [Figure]

Revised: [Figure]

Original: PredictedFrames
Revised: PredictedFrames

Original: C M ODEL ARCHITECTURE
Revised: C M ODEL ARCHITECTURE

Original: In this section, we detail the architectures and hyperparameters used in our experiments.
Revised: In this section, we detail the architectures and hyperparameters used in our experiments.

Original: C. S EQUENTIAL D ENSE S TRUCTURE G ENERATOR
Revised: C. S EQUENTIAL D ENSE S TRUCTURE G ENERATOR

Original: We use the modiﬁed version of SVG (Denton & Fergus, 2018) by Villegas et al.
Revised: We use the modiﬁed version of SVG (Denton & Fergus, 2018) by Villegas et al.

Original: (2019) as our sequential dense structure generator, which we refer to as SVG-extend throughout the paper.
Revised: (2019) as our sequential dense structure generator, which we refer to as SVG-extend throughout the paper.

Original: The two main differences between SVG and SVG-extend are two-fold: (1) SVG-extend adopts the shallower auto-encoder architecture than that of SVG.
Revised: The two main differences between SVG and SVG-extend are two-fold: (1) SVG-extend adopts the shallower auto-encoder architecture than that of SVG.

Original: (2) SVG-extend replaces Fully-Connected-LSTM with Convolutional-LSTM (Shi et al., 2015).
Revised: (2) SVG-extend replaces Fully-Connected-LSTM with Convolutional-LSTM (Shi et al., 2015).

Original: These two modiﬁcations are to better exploit spatial information contained in 3D feature maps, which is not maintained when inputs are mapped to low-dimensional 1-D image embedding vectors.
Revised: These two modiﬁcations are to better exploit spatial information contained in 3D feature maps, which is not maintained when inputs are mapped to low-dimensional 1-D image embedding vectors.

Original: Since the detailed architectural conﬁguration of SVG-extend is not described in (Villegas et al., 2019), we summarize our reproduced version of this framework in TableA, B, C,  and D. On the 64 × 64 image resolution, we use the same architecture described in the tables.
Revised: Since the detailed architectural conﬁguration of SVG-extend is not described in (Villegas et al., 2019), we summarize our reproduced version of this framework in Table B, C, D, and E. On the 64 × 64 image resolution, we use the same architecture described in the tables.

Original: On 128 ×and 256 × 256 image resolutions, we simply stack one and two more convolutional blocks (nonlinear convolutional layers followed by upsampling or downsampling layers) in the Encoder and
Revised: On 128 ×and 256 × 256 image resolutions, we simply stack one and two more convolutional blocks (nonlinear convolutional layers followed by upsampling or downsampling layers) in the Encoder and

Original: Decoder networks, respectively, so that the shape of h t , z t , g t is kept to R 128 × 8 × 8 agnostic to the image resolutions.
Revised: Decoder networks, respectively, so that the shape of h t , z t , g t is kept to R 128 × 8 × 8 agnostic to the image resolutions.

Original: For hyperparameters, we use C = 5 ; we set β = 0 . 0005 ; we use ADAM optimizer (Kingma & Ba, 2015) with the learning rate of 0 . 0001 and ( β 1 , β 2 ) =
Revised: For hyperparameters, we use C = 5 ; we set β = 0 . 0005 ; we use ADAM optimizer (Kingma & Ba, 2015) with the learning rate of 0 . 0001 and ( β 1 , β 2 ) =

Original: C
Revised: C

Original: B OUNARY MAP PREDICTION
Revised: B OUNARY MAP PREDICTION

Original: When the pre-trained instance-wise segmentation model is available, we can optionally extend the structure generator to jointly predict the object boundary maps.
Revised: When the pre-trained instance-wise segmentation model is available, we can optionally extend the structure generator to jointly predict the object boundary maps.

Original: Such structures can add a notion of object instance and is useful to improve the image generator in sequences with many occluding objects.
Revised: Such structures can add a notion of object instance and is useful to improve the image generator in sequences with many occluding objects.

Original: Let e t ∈ { 0 , 1 } W × H denotes the object boundary map at frame x t .
Revised: Let e t ∈ { 0 , 1 } W × H denotes the object boundary map at frame x t .

Original: Then we train the denoising autoencoder G edge to produce an edge map of each frame by
Revised: Then we train the denoising autoencoder G edge to produce an edge map of each frame by

Original: [Equation]
Revised: [Equation]

Original: We employed the encoder-decoder network (Nazeri et al., 2019) as the conditional generator, and train it based on adversarial loss by
Revised: We employed the encoder-decoder network (Nazeri et al., 2019) as the conditional generator, and train it based on adversarial loss by

Original: [Equation]
Revised: [Equation]

Original: where D is the discriminator.
Revised: where D is the discriminator.

Original: Note that the edge generator in Eq.
Revised: Note that the edge generator in Eq.

Original: 8 is applied to each frame independently; we observe that adding the predicted boundary map as an input to the structure generator makes the model be prune to error propagation.
Revised: 8 is applied to each frame independently; we observe that adding the predicted boundary map as an input to the structure generator makes the model be prune to error propagation.

Original: The boundary and label maps are then combined as the output of the structure generator ˜ s t =
Revised: The boundary and label maps are then combined as the output of the structure generator ˜ s t =

Original: [ˆ s t , ˆ e t ] .
Revised: [ˆ s t , ˆ e t ] .

Original: C. S TRUCTURE -C ONDITIONAL P IXEL S EQUENCE G ENERATOR
Revised: C. S TRUCTURE -C ONDITIONAL P IXEL S EQUENCE G ENERATOR

Original: We use Vid2Vid (Wang et al., 2018) for our structure-conditional pixel sequence generator.
Revised: We use Vid2Vid (Wang et al., 2018) for our structure-conditional pixel sequence generator.

Original: For
Revised: For

Original: Vid2Vid on KITTI dataset of 64 × 64 resolution, we reduce the number of downsampling layers in the generator and the scale parameter of the patch discriminator to 1.
Revised: Vid2Vid on KITTI dataset of 64 × 64 resolution, we reduce the number of downsampling layers in the generator and the scale parameter of the patch discriminator to 1.

Original: We use τ = 5 for KITTI and
Revised: We use τ = 5 for KITTI and

Original: Human Dancing and use τ = 4 for Cityscapes.
Revised: Human Dancing and use τ = 4 for Cityscapes.

Original: We use τ (cid:48)
Revised: We use τ (cid:48)

Original: = 3 for all experiments.
Revised: = 3 for all experiments.

Original: If not speciﬁed otherwise, we do not modify any hyperparameters and simply follow the default settings for the training.
Revised: If not speciﬁed otherwise, we do not modify any hyperparameters and simply follow the default settings for the training.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]