  Studying the sensitivity of weight perturbation in neural networks and its
impacts on model performance, including generalization and robustness, is an
active research topic due to its implications on a wide range of machine
learning tasks such as model compression, generalization gap assessment, and
adversarial attacks. In this paper, we provide the first formal analysis for
feed-forward neural networks with non-negative monotone activation functions
against norm-bounded weight perturbations, in terms of the robustness in
pairwise class margin functions and the Rademacher complexity for
generalization. We further design a new theory-driven loss function for
training generalizable and robust neural networks against weight perturbations.
Empirical experiments are conducted to validate our theoretical analysis. Our
results offer fundamental insights for characterizing the generalization and
robustness of neural networks against weight perturbations.
