  Adversarial attack arises due to the vulnerability of deep neural networks to
perceive input samples injected with imperceptible perturbations. Recently,
adversarial attack has been applied to visual object tracking to evaluate the
robustness of deep trackers. Assuming that the model structures of deep
trackers are known, a variety of white-box attack approaches to visual tracking
have demonstrated promising results. However, the model knowledge about deep
trackers is usually unavailable in real applications. In this paper, we propose
a decision-based black-box attack method for visual object tracking. In
contrast to existing black-box adversarial attack methods that deal with static
images for image classification, we propose IoU attack that sequentially
generates perturbations based on the predicted IoU scores from both current and
historical frames. By decreasing the IoU scores, the proposed attack method
degrades the accuracy of temporal coherent bounding boxes (i.e., object
motions) accordingly. In addition, we transfer the learned perturbations to the
next few frames to initialize temporal motion attack. We validate the proposed
IoU attack on state-of-the-art deep trackers (i.e., detection based,
correlation filter based, and long-term trackers). Extensive experiments on the
benchmark datasets indicate the effectiveness of the proposed IoU attack
method. The source code is available at
https://github.com/VISION-SJTU/IoUattack.
