  Learning to predict the long-term future of video frames is notoriously
challenging due to inherent ambiguities in the distant future and dramatic
amplifications of prediction error through time. Despite the recent advances in
the literature, existing approaches are limited to moderately short-term
prediction (less than a few seconds), while extrapolating it to a longer future
quickly leads to destruction in structure and content. In this work, we revisit
hierarchical models in video prediction. Our method predicts future frames by
first estimating a sequence of semantic structures and subsequently translating
the structures to pixels by video-to-video translation. Despite the simplicity,
we show that modeling structures and their dynamics in the discrete semantic
structure space with a stochastic recurrent estimator leads to surprisingly
successful long-term prediction. We evaluate our method on three challenging
datasets involving car driving and human dancing, and demonstrate that it can
generate complicated scene structures and motions over a very long time horizon
(i.e., thousands frames), setting a new standard of video prediction with
orders of magnitude longer prediction time than existing approaches. Full
videos and codes are available at https://1konny.github.io/HVP/.
