  We introduce the problem of weakly supervised Multi-Object Tracking and
Segmentation, i.e. joint weakly supervised instance segmentation and
multi-object tracking, in which we do not provide any kind of mask annotation.
To address it, we design a novel synergistic training strategy by taking
advantage of multi-task learning, i.e. classification and tracking tasks guide
the training of the unsupervised instance segmentation. For that purpose, we
extract weak foreground localization information, provided by Grad-CAM
heatmaps, to generate a partial ground truth to learn from. Additionally, RGB
image level information is employed to refine the mask prediction at the edges
of the objects. We evaluate our method on KITTI MOTS, the most representative
benchmark for this task, reducing the performance gap on the MOTSP metric
between the fully supervised and weakly supervised approach to just 12% and
12.7% for cars and pedestrians, respectively.
