  We propose an architecture and training scheme to predict video frames by
explicitly modeling dis-occlusions and capturing the evolution of semantically
consistent regions in the video. The scene layout (semantic map) and motion
(optical flow) are decomposed into layers, which are predicted and fused with
their context to generate future layouts and motions. The appearance of the
scene is warped from past frames using the predicted motion in co-visible
regions; dis-occluded regions are synthesized with content-aware inpainting
utilizing the predicted scene layout. The result is a predictive model that
explicitly represents objects and learns their class-specific motion, which we
evaluate on video prediction benchmarks.
