  Adversarial attacks have rendered high security risks on modern deep learning
systems. Adversarial training can significantly enhance the robustness of
neural network models by suppressing the non-robust features. However, the
models often suffer from significant accuracy loss on clean data. Ensemble
training methods have emerged as promising solutions for defending against
adversarial attacks by diversifying the vulnerabilities among the sub-models,
simultaneously maintaining comparable accuracy as standard training. However,
existing ensemble methods are with poor scalability, owing to the rapid
complexity increase when including more sub-models in the ensemble. Moreover,
in real-world applications, it is difficult to deploy an ensemble with multiple
sub-models, owing to the tight hardware resource budget and latency
requirement. In this work, we propose ensemble-in-one (EIO), a simple but
efficient way to train an ensemble within one random gated network (RGN). EIO
augments the original model by replacing the parameterized layers with
multi-path random gated blocks (RGBs) to construct a RGN. By diversifying the
vulnerability of the numerous paths within the RGN, better robustness can be
achieved. It provides high scalability because the paths within an EIO network
exponentially increase with the network depth. Our experiments demonstrate that
EIO consistently outperforms previous ensemble training methods with even less
computational overhead.
