  Recently developed adversarial weight attack, a.k.a. bit-flip attack (BFA),
has shown enormous success in compromising Deep Neural Network (DNN)
performance with an extremely small amount of model parameter perturbation. To
defend against this threat, we propose RA-BNN that adopts a complete binary
(i.e., for both weights and activation) neural network (BNN) to significantly
improve DNN model robustness (defined as the number of bit-flips required to
degrade the accuracy to as low as a random guess). However, such an aggressive
low bit-width model suffers from poor clean (i.e., no attack) inference
accuracy. To counter this, we propose a novel and efficient two-stage network
growing method, named Early-Growth. It selectively grows the channel size of
each BNN layer based on channel-wise binary masks training with Gumbel-Sigmoid
function. Apart from recovering the inference accuracy, our RA-BNN after
growing also shows significantly higher resistance to BFA. Our evaluation of
the CIFAR-10 dataset shows that the proposed RA-BNN can improve the clean model
accuracy by ~2-8 %, compared with a baseline BNN, while simultaneously
improving the resistance to BFA by more than 125 x. Moreover, on ImageNet, with
a sufficiently large (e.g., 5,000) amount of bit-flips, the baseline BNN
accuracy drops to 4.3 % from 51.9 %, while our RA-BNN accuracy only drops to
37.1 % from 60.9 % (9 % clean accuracy improvement).
