  Standard Convolutional Neural Networks (CNNs) can be easily fooled by images
with small quasi-imperceptible artificial perturbations. As alternatives to
CNNs, the recently proposed Capsule Networks (CapsNets) are shown to be more
robust to white-box attacks than CNNs under popular attack protocols. Besides,
the class-conditional reconstruction part of CapsNets is also used to detect
adversarial examples. In this work, we investigate the adversarial robustness
of CapsNets, especially how the inner workings of CapsNets change when the
output capsules are attacked. The first observation is that adversarial
examples misled CapsNets by manipulating the votes from primary capsules.
Another observation is the high computational cost, when we directly apply
multi-step attack methods designed for CNNs to attack CapsNets, due to the
computationally expensive routing mechanism. Motivated by these two
observations, we propose a novel vote attack where we attack votes of CapsNets
directly. Our vote attack is not only effective but also efficient by
circumventing the routing process. Furthermore, we integrate our vote attack
into the detection-aware attack paradigm, which can successfully bypass the
class-conditional reconstruction based detection method. Extensive experiments
demonstrate the superior attack performance of our vote attack on CapsNets.
