Deep neural networks are vulnerable to adversarial attacks. Recent studies of adversarial robustness focus on the loss landscape in the parameter space since it is related to optimization  performance. These studies conclude that it is hard to optimize the loss function for adversarial training with respect to parameters because the loss function is not smooth : i.e., its gradient is not Lipschitz continuous. However, this analysis ignores the dependence of adversarial attacks on  parameters. Since adversarial attacks are the worst noise for the models, they should depend on the parameters of the models. In this study , we analyze the smoothness of the loss function of adversarial training for binary linear classification and for general cases considering the dependence . We reveal that the Lipschitz continuity depends on the types of constraints of adversarial attacks in the binary linear classification . Specifically, under the L2 constraints, the adversarial loss is smooth except at zero. We extend the analysis to general cases and prove local smoothness in several cases. Our analysis reveals that the constraint of adversarial examples is a cause of the non-smoothness of adversarial loss . Moreover, we reveal the relation between the flatness of the loss function with respect to input data and the smoothness of the adversarial loss with respect to parameters. Our analysis implies that if we flatten the loss function with respect to input data, the smoothness of adversarial loss tends to decrease .
Deep neural networks are vulnerable to adversarial attacks. Recent studies about adversarial robustness focus on the loss landscape in the parameter space since it is related to optimization and generalization performance. These studies conclude that the difficulty of adversarial training is caused by the non-smoothness of the loss function : i.e., its gradient is not Lipschitz continuous. However, this analysis ignores the dependence of adversarial attacks on model parameters. Since adversarial attacks are optimized for models, they should depend on the parameters . Considering this dependence , we analyze the smoothness of the loss function of adversarial training using the optimal attacks for the model parameter in more detail . We reveal that the constraint of adversarial attacks is one cause of the non-smoothness and that the smoothness depends on the types of the constraints . Specifically, the L_\infty constraint can cause non-smoothness more than the L_2 constraint . Moreover, our analysis implies that if we flatten the loss function with respect to input data , the Lipschitz constant of the gradient of adversarial loss tends to increase. To address the non-smoothness, we show that EntropySGD smoothens the non-smooth loss and improves the performance of adversarial training .
R
of
about
A
and generalization
R
it is hard to optimize the loss function for adversarial training with respect to parameters because the loss function is not smooth
the difficulty of adversarial training is caused by the non-smoothness of the loss function
A
model
R
the worst noise for the
optimized for
R
of the models. In this study
. Considering this dependence
R
for binary linear classification and for general cases considering the dependence
using the optimal attacks for the model parameter in more detail
R
Lipschitz continuity
constraint of adversarial attacks is one cause of the non-smoothness and that the smoothness
R
constraints of adversarial attacks in the binary linear classification
the constraints
R
under the L2 constraints, the adversarial loss is smooth except at zero. We extend the analysis to general cases and prove local smoothness in several cases. Our analysis reveals that the constraint of adversarial examples is a cause of the
the L_\infty constraint can cause
R
of adversarial loss
more than the L_2 constraint
R
we reveal the relation between the flatness of the
our analysis implies that if we flatten the
R
and the smoothness of the adversarial loss with respect to parameters. Our analysis implies that if we flatten the loss function with respect to input data, the smoothness of adversarial loss tends to decrease
, the Lipschitz constant of the gradient of adversarial loss tends to increase. To address the non-smoothness, we show that EntropySGD smoothens the non-smooth loss and improves the performance of adversarial training