  Do all adversarial examples have the same consequences? An autonomous driving
system misclassifying a pedestrian as a car may induce a far more dangerous --
and even potentially lethal -- behavior than, for instance, a car as a bus. In
order to better tackle this important problematic, we introduce the concept of
hierarchical adversarial robustness. Given a dataset whose classes can be
grouped into coarse-level labels, we define hierarchical adversarial examples
as the ones leading to a misclassification at the coarse level. To improve the
resistance of neural networks to hierarchical attacks, we introduce a
hierarchical adversarially robust (HAR) network design that decomposes a single
classification task into one coarse and multiple fine classification tasks,
before being specifically trained by adversarial defense techniques. As an
alternative to an end-to-end learning approach, we show that HAR significantly
improves the robustness of the network against $\ell_2$ and $\ell_{\infty}$
bounded hierarchical attacks on the CIFAR-10 and CIFAR-100 dataset.
