  Due to limited computational cost and energy consumption, most neural network
models deployed in mobile devices are tiny. However, tiny neural networks are
commonly very vulnerable to attacks. Current research has proved that larger
model size can improve robustness, but little research focuses on how to
enhance the robustness of tiny neural networks. Our work focuses on how to
improve the robustness of tiny neural networks without seriously deteriorating
of clean accuracy under mobile-level resources. To this end, we propose a
multi-objective oneshot network architecture search (NAS) algorithm to obtain
the best trade-off networks in terms of the adversarial accuracy, the clean
accuracy and the model size. Specifically, we design a novel search space based
on new tiny blocks and channels to balance model size and adversarial
performance. Moreover, since the supernet significantly affects the performance
of subnets in our NAS algorithm, we reveal the insights into how the supernet
helps to obtain the best subnet under white-box adversarial attacks.
Concretely, we explore a new adversarial training paradigm by analyzing the
adversarial transferability, the width of the supernet and the difference
between training the subnets from scratch and fine-tuning. Finally, we make a
statistical analysis for the layer-wise combination of certain blocks and
channels on the first non-dominated front, which can serve as a guideline to
design tiny neural network architectures for the resilience of adversarial
perturbations.
