  In this paper, we tackle video panoptic segmentation, a task that requires
assigning semantic classes and track identities to all pixels in a video. To
study this important problem in a setting that requires a continuous
interpretation of sensory data, we present a new benchmark: Segmenting and
Tracking Every Pixel (STEP), encompassing two datasets, KITTI-STEP, and
MOTChallenge-STEP together with a new evaluation metric. Our work is the first
that targets this task in a real-world setting that requires dense
interpretation in both spatial and temporal domains. As the ground-truth for
this task is difficult and expensive to obtain, existing datasets are either
constructed synthetically or only sparsely annotated within short video clips.
By contrast, our datasets contain long video sequences, providing challenging
examples and a test-bed for studying long-term pixel-precise segmentation and
tracking. For measuring the performance, we propose a novel evaluation metric
Segmentation and Tracking Quality (STQ) that fairly balances semantic and
tracking aspects of this task and is suitable for evaluating sequences of
arbitrary length. We will make our datasets, metric, and baselines publicly
available.
