  Tracking by detection, the dominant approach for online multi-object
tracking, alternates between localization and re-identification steps. As a
result, it strongly depends on the quality of instantaneous observations, often
failing when objects are not fully visible. In contrast, tracking in humans is
underlined by the notion of object permanence: once an object is recognized, we
are aware of its physical existence and can approximately localize it even
under full occlusions. In this work, we introduce an end-to-end trainable
approach for joint object detection and tracking that is capable of such
reasoning. We build on top of the recent CenterTrack architecture, which takes
pairs of frames as input, and extend it to videos of arbitrary length. To this
end, we augment the model with a spatio-temporal, recurrent memory module,
allowing it to reason about object locations and identities in the current
frame using all the previous history. It is, however, not obvious how to train
such an approach. We study this question on a new, large-scale, synthetic
dataset for multi-object tracking, which provides ground truth annotations for
invisible objects, and propose several approaches for supervising tracking
behind occlusions. Our model, trained jointly on synthetic and real data,
outperforms the state of the art on KITTI, and MOT17 datasets thanks to its
robustness to occlusions.
