  The presence of objects that are confusingly similar to the tracked target,
poses a fundamental challenge in appearance-based visual tracking. Such
distractor objects are easily misclassified as the target itself, leading to
eventual tracking failure. While most methods strive to suppress distractors
through more powerful appearance models, we take an alternative approach.
  We propose to keep track of distractor objects in order to continue tracking
the target. To this end, we introduce a learned association network, allowing
us to propagate the identities of all target candidates from frame-to-frame. To
tackle the problem of lacking ground-truth correspondences between distractor
objects in visual tracking, we propose a training strategy that combines
partial annotations with self-supervision. We conduct comprehensive
experimental validation and analysis of our approach on several challenging
datasets. Our tracker sets a new state-of-the-art on six benchmarks, achieving
an AUC score of 67.2% on LaSOT and a +6.1% absolute gain on the OxUvA long-term
dataset.
