  Deep neural networks (DNNs) have demonstrated remarkable performance for
various applications, meanwhile, they are widely known to be vulnerable to the
attack of adversarial perturbations. This intriguing phenomenon has attracted
significant attention in machine learning and what might be more surprising to
the community is the existence of universal adversarial perturbations (UAPs),
i.e. a single perturbation to fool the target DNN for most images. The
advantage of UAP is that it can be generated beforehand and then be applied
on-the-fly during the attack. With the focus on UAP against deep classifiers,
this survey summarizes the recent progress on universal adversarial attacks,
discussing the challenges from both the attack and defense sides, as well as
the reason for the existence of UAP. Additionally, universal attacks in a wide
range of applications beyond deep classification are also covered.
