  Our goal is to forecast the near future given a set of recent observations.
We think this ability to forecast, i.e., to anticipate, is integral for the
success of autonomous agents which need not only passively analyze an
observation but also must react to it in real-time. Importantly, accurate
forecasting hinges upon the chosen scene decomposition. We think that superior
forecasting can be achieved by decomposing a dynamic scene into individual
'things' and background 'stuff'. Background 'stuff' largely moves because of
camera motion, while foreground 'things' move because of both camera and
individual object motion. Following this decomposition, we introduce panoptic
segmentation forecasting. Panoptic segmentation forecasting opens up a
middle-ground between existing extremes, which either forecast instance
trajectories or predict the appearance of future image frames. To address this
task we develop a two-component model: one component learns the dynamics of the
background stuff by anticipating odometry, the other one anticipates the
dynamics of detected things. We establish a leaderboard for this novel task,
and validate a state-of-the-art model that outperforms available baselines.
