Cooperation or Competition: Avoiding Player Domination for Multi-target Robustness by Adaptive Budgets
Abstract
Despite incredible advances, deep learning has been shown to be susceptible to adversarial attacks.
Numerous approaches were proposed to train robust networks both empirically and certifiably.
However, most of them defend against only a single type of attack, while recent work steps forward at defending against multiple attacks.
In this paper, to understand multi-target robustness, we view this problem as a bargaining game in which different players (adversaries) negotiate to reach an agreement on a joint direction of parameter updating.
We identify a phenomenon named player domination in the bargaining game, and show that with this phenomenon, some of the existing max-based approaches such as MAX and
MSD do not converge.
Based on our theoretical results, we design a novel framework that adjusts the budgets of different adversaries to avoid player domination.
Experiments on two benchmarks show that employing the proposed framework to the existing approaches significantly advances multi-target robustness.
Introduction
Machine learning models have been shown to be susceptible to adversarial examples (Szegedy et al., 2014), where human-imperceptible perturbations added to a clean example might arbitrarily change the output of machine learning models.
The adversarial examples are generated by maximizing the loss within a small perturbation region around a clean example, e.g ., ℓ ∞ , ℓ 1 and ℓ 2 balls.
On the other hand, numerous heuristic defenses have been proposed to be robust against adversarial examples, e.g ., distillation (Papernot et al., 2016), logit-pairing (Kannan et al., 2018) and adversarial training (Madry et al., 2018).
However, most of the existing defenses are only robust against one type of attacks (Madry et al., 2018; Raghunathan et al., 2022; Wong & Kolter, 2018;
Engstrom et al., 2018), while they fail to defend against other adversaries.
For example, existing work (Kang et al., 2019b; Maini et al., 2020) showed that robustness in the ℓ p threat model does not necessarily generalize to other ℓ q threat models.
However, for the sake of the safety of machine learning systems, one should target robustness against multiple adversaries simultaneously (Croce & Hein, 2020a).
Recently, various methods (Schott et al., 2019; Tramer & Boneh, 2019; Maini et al., 2020) have been proposed to address this problem.
Multi-target adversarial training, which targets defending against multiple adversarial perturbations, has attracted significant attention: a variational autoencoder-based model (Schott et al., 2019) learns a classifier robust to multiple perturbations; after that, MAX and
AVG strategies, which aggregate different adversaries for adversarial training against multiple threat models, have been shown to enjoy improved performance (Tramer & Boneh, 2019).
To further advance the robustness against multiple adversaries, Maini et al.
(2020) achieved better performance than MAX and AVG by taking the worst-case over all steepest descent directions.
These methods follow a general scheme as the (single-target) adversarial training.
This general scheme for multi-target adversarial training can be seen as an implementation of a cooperative bargaining game (Thomson, 1994).
In this game, different parties have to decide how to maximize the surplus they jointly get.
In the multi-target adversarial training, we view each party as an adversary, and they negotiate to reach an agreed gradient direction that maximizes their utility function.
We can then utilize the results from a game-theoretic perspective to analyze the multi-target adversarial training.
Inspired by the bargaining game for multi-target adversarial training, we first analyze the existing methods, i.e ., MAX (Tramer & Boneh, 2019), MSD (Maini et al., 2020), and AVG (Tramer & Boneh, 2019), and categorize them into two types, MAX-based and AVG-based algorithms.
We identify a phenomenon where one player dominates the bargaining game at any time t , where the agreed gradient at any time t is the same as this player’s gradient.
Following that, we show that MAX-based algorithms, i.e ., MAX and MSD, do not converge.
Based on our theoretical results, we propose a novel mechanism which adaptively adjusts the budget of adversaries to avoid any player dominating the bargaining game.
We show that with our proposed mechanism, the robust accuracy of MAX,
AVG and MSD improves on both the MNIST and CIFAR-10 datasets.
1.1 Cooperative Bargaining Game
Cooperative bargaining game (Thomson, 1994) is a process in which several parties jointly decide how to share a surplus that they can jointly gain.
The multi-target adversarial training can be viewed as a cooperative game in which each target (perturbation) represents a player, whose utility is derived from its gradient, and all the players negotiate to reach an agreed direction.
In the cooperative bargaining game, we have K players with their own utility function u
i : A (cid:83) { d } → R , where A is the set of possible agreements and d is the disagreement point.
The feasible set of utility is defined as S =
{ ( u 1 ( γ ) . . .
, u K ( γ )) : γ ∈ A} .
The goals of players are to maximize their own utility functions.
S is assumed to be convex and compact throughout this paper while there exists
a point γ ∈
A satisfying u
i ( γ ) > u i ( d ) , ∀ i ∈ [ K ] .
We formalize the multi-target adversarial training problem as a bargaining game as follows.
This bargaining game has K players, as we are targeting at multi-target adversarial training.
For each player, they generate a data-dependent perturbation δ k ( x ) , ∀ k ∈
[ K ] to complete the adversarial training, where [ K ] = { 1 , 2 , 3 , · · · , K } .
The possible agreements A are { (cid:80) k ∈
[ K ] γ k = 1 , γ k ≥ 0 , ∀ k ∈
[ K ] } and the disagreement point at 0 , i.e ., staying at the current parameters w .
The utility function for the player k is u k ( γ ) =
g ⊤ k (cid:80) i ∈
[ K ] γ i g i , where g k is the gradient of the player k .
We note that since the agreement set A is compact and convex and the utilities are linear, the set of possible payoffs S is also compact and convex.
Convergence Analysis
We start the section by showing our theoretical results based on two machine learning models.
Furthermore, we design a general framework for the multi-target adversarial training problem to avoid the player domination phenomenon which might lead to the non-convergence of MAX and
MSD in the next section.
All missing proofs are in the appendix.
2.1 Convergence analysis on SVM model
Consider the binary classification setup introduced in Tsipras et al.
(2019), where data ( x , y ) is sampled from a distribution D defined by
[Equation]
where x = [ x 1 , . .
, x d +1 ] ∈ R d +1 , y is a Rademacher random variable, and N ( µ, σ 2 ) is a normal distribution with mean µ and variance σ 2 .
In our setting, p ∈ [0 . 5 , 1] .
x 1 is a robust feature while x 2 , . . .
, x d +1 are non-robust features that are weakly correlated with the label.
Similarly, we set µ to be large enough such that almost any classifier can get a high standard accuracy ( > 99 %), i.e ., µ ≥ 1 / √ d .
We train a linear model f w ( · ) with soft-SVM loss on the data shown above:
[Equation]
where f w ( x ) = w ⊤ x , and γ =
[ γ 1 , γ 2 , γ ∞ ] satisfies (cid:80) i ∈{ 1 , 2 , ∞} γ i = 1 .
With the linearity property of SVM, the closed form of optimal perturbations could be calculated as
[Equation]
Let w t and δ t be the weight vector of classifier and the perturbation at time t , respectively.
The training procedures of AVG, MAX and MSD are illustrated as follows:
Initialize the weights with natural training,
i
e ., minimizing the soft-SVM loss without perturbation
[Equation]
Solve the inner maximization problem using AVG, MAX and MSD, and return the optimal perturbations, i e ., δ t 1 , δ t 2 and δ t ∞ .
Update the weight of the classifier by
[Equation]
where γ t = [1 / 3 , 1 / 3 , 1 / 3] if the algorithm is AVG; γ t ∈ { [1 , 0 , 0] , [0 , 1 , 0] , [0 , 0 , 1] } if the algorithm is MAX or MSD.
Loop Steps 1 and 2 for predefined epochs or until convergence.
We first present the following negative result:
Theorem 1.
Let µ ≥ 4 / √ d , ϵ ∞ ≥ 2 µ , p ≤ 0 . 977 .
If one uses MAX and MSD to train the SVM model and ϵ ∞ ≥ 2 d ϵ 1 and ϵ ∞ ≥ (cid:113) 2 d ϵ 2 , the loss incurred by the ∞ -player ( ∞ -adversary) is larger than that by the 1 -player ( 1 -adversary) and the 2 -player ( 2 -adversary) at any time t , i.e ., ℓ w t ( x + δ ∞ ( x )) ≥ max( ℓ w t ( x + δ 1 ( x )) , ℓ w t ( x + δ 2 ( x ))) , ∀ t, ∀ x .
This theorem shows that under certain conditions, the SVM learning dynamics will be affected by only one player, i.e ., the ∞ -player, during the whole training procedure.
And we further observe that this phenomenon leads to non-convergence of the SVM model as the sign of weights of the model will be flipped.
We define the phenomenon that one player “dominate” the multi-target adversarial training procedure (the training procedure only depends on one player) as follows
Definition 2 (Player dominates the cooperative game) .
If ∃ i ∈ [ k ] such that γ ti = 1 and γ tj = 0 , ∀ j ∈ [ K ] / { i } , ∀ t , then we call that player dominates the bargaining game.
Further, after analyzing the training dynamics of SVM, we notice that when the ∞ -player dominates the bargaining game, and given ϵ ∞ > µ , the SVM model may not converge as the weights for the non-robust features flips over time.
Theorem 3.
Consider Problem equation 1 trained with MAX and MSD.
If ∞ -player dominates the bargaining game and ϵ ∞ > µ , the weights for the non-robust features flips over time, i.e ., sign( w ti )
= − sign( w t − 1 i ) , ∀ i ≥ 2 .
Thus, the training procedure does not converge.
As the ∞ -player dominates this game, the multi-target adversarial training problem reduces to the single-target problem (equation 6).
Further, with Lemma 8, for non-robust feature i , if η ∞ >
η , we have sign( w ti ) =
− sign( w t − 1 i ) .
Therefore, the training procedure does not converge.
Though we only analyze the case when an ∞ -player dominates the bargaining game, we notice that when other players dominate this bargaining game ( i.e ., multi-target adversarial training), with certain conditions, e.g ., ϵ 1 > 4 µ , a similar phenomenon can be observed empirically.
Motivated by the negative results of the SVM model, we next testify a conjecture that player domination may lead to non-convergence of the linear model as well.
2.2 Player Domination Leads to Non-convergence
To testify our conjecture, we introduce a linear model as follows.
The linear model is parameterized by w and optimized by gradient-based algorithms, e.g ., AdaGrad (Duchi et al., 2011) or Adam (Kingma & Ba, 2015).
The parameter at time (epoch) t is denoted by w t .
The loss function of each player is denoted as ℓ k , k ∈ [ K ] , which is L -smooth and µ -strongly convex, and the corresponding gradient is denoted as ∇ w ℓ k ( w t ) or g tk , k ∈ [ K ] , ∀ t .
We assume that for a sequence { w t } t =[1 , ∞ ] generated by any optimization algorithm, the set of the gradient vectors g tk , k ∈ [ K ] at any time t and at any partial limit are linearly independent unless that point is locally optimal.
All loss functions are differentiable and all sub-level sets are bounded.
The learning rate is denoted as η and η < 2 L .
We also assume that the input domain is open and convex.
To generalize our theoretical results, we show that in this linear model, MAX and MSD still may not converge if one player dominates the game.
Theorem 4.
Consider using MAX and MSD to train the linear model described above.
If the bargaining game is dominated by one player during the whole game (see Definition 2), then the loss of all players and the overall loss would increase as time t grows.
That means the training procedure on the above linear model might not converge.
While we have shown that MAX and MSD may not converge under the two models that we study, we notice that AVG provably converges as the loss is decreasing w.r.t the number of epochs.
See the following theorem.
Theorem 5.
Using AVG to train the linear model, the overall loss decrease as time t grows.
This theorem shows that under the same setting, while the loss of each player and the overall loss will be increasing as time grows with MSD and MSD, the overall loss will be decreasing with AVG.
And the key component in the proof is the player domination phenomenon, which directly leads to the increase of loss with MAX and MSD.
That indicates that the training procedure with MAX and MSD might not converge while AVG might converge, which is also a consequence of player domination.
Avoiding Player Domination with Adaptive Budgets
Our theoretical results indicate that when player domination (Def. 2) occurs, MAX and MSD may not converge while the loss of AVG decreases as time grows.
Inspired by this analysis, we design a novel general-purpose algorithm for multi-target adversarial robustness, which adaptively changes the budget of different adversaries.
The resulting AdaptiveBudget method is presented in Algorithm 1.
The core idea of this algorithm is to avoid player domination by adaptively assigning proper attack budgets to different adversaries (players).
Such an assignment is to make no single player achieve significantly better than others.
Concretely, for each batch of data, we first get the adversarial perturbations δ ∞ , δ 1 and δ 2 for the ℓ ∞ , ℓ 1 , and ℓ 2 adversaries.
Then based on the norms of the gradients by forwarding their adversarial examples through our model, the algorithm adaptively adjusts the budgets ϵ for different adversaries to avoid the player domination phenomenon.
Specifically, our Require: Training Epochs E , Training samples ( X , Y ) , adversarial budgets ( ϵ ∞ ,ϵ 1 ,ϵ 2 ) , model f ( · ) , loss function ℓ . 1: for e ∈ [ E ] do 2: for x ,y ∈ ( X , Y ) do 3: g p ← ℓ ′ ( x + δ p ( x )) , δ p ( x ) ← PGD( x ,k,η,ℓ,ϵ p ,ℓ ) , ∀ p ∈ { 1 , 2 , ∞} 4: Get adaptive budgets ˆ ϵ 1 , ˆ ϵ 2 , ˆ ϵ ∞
← AdaptiveBudget ([ g 1 ,g 2 ,g ∞ ] , [ ϵ 1 ,ϵ 2 ,ϵ ∞ ]) ; 5: Adversarial training using MAX, MSD or AVG with budgets (ˆ ϵ 1 , ˆ ϵ 2 , ˆ ϵ ∞ ) ; 6: end for 7: end for 8: Return the classifier f . 9: 10: AdaptiveBudget ( Gradients [ g 1 ,g 2 ,g ∞ ] , Epsilon [ ϵ 1 ,ϵ 2 ,ϵ ∞ ]) : 11: p max ← argmax p ∈{∞ , 1 , 2 } ∥ g p ∥ , p min ← argmin p ∈{∞ , 1 , 2 } ∥ g p ∥ ; 12: p mid ← { 1 , 2 , ∞} / { p max ,p min } ; 13: ϵ p max ← ϵ p max · ∥ gp max ∥
∥ gp mid ∥ , ϵ p min ← ϵ p min · ∥ gp min ∥ ∥ gp mid ∥ ; 14: Return ϵ 1 ,ϵ 2 ,ϵ ∞ . proposed method keeps the budget of the adversary whose norm of gradient is the middle one.
This increases the budget of the adversary whose norm of gradient is the maximum and decreases the budget of the adversary whose norm of gradient is the minimum.
The intuition behind our method is to focus on the hardest task in the current round so that this task might be easier to model in the next round.
After obtaining the adjusted adversarial budgets, the model utilizes MSD, MAX or AVG to approximately solve the inner maximization problem and then updates its parameter with any gradient descent algorithm.
The proposed framework is general and can be applied to all existing multi-target adversarial training algorithms.
The adaptive budget module is employed to break the curse of player domination which might occur when applying MAX and MSD to train a robust model.
In the next section, we corroborate the consistent effectiveness of AdaptiveBudget method with extensive experiments.
Experiments 4.1 Experimental setup and implementation details
Datasets.
We conducted extensive experiments on a synthetic data (Sec. 2.1) to complement our theoretical results and MNIST (LeCun et al., 1998) and CIFAR-10 (Krizhevsky & Hinton, 2009) to show the superiority of our proposed methods over the existing methods of multi-target adversarial training.
Methods.
We train models that defend multiple adversaries using MAX (Tramer & Boneh, 2019), AVG (Tramer & Boneh, 2019) and MSD (Maini et al., 2020).
Attacks used for evaluation.
To fully understand the performance of the defense, we employ PGD adversary and Autoattack (Croce & Hein, 2020b) 1 to test the effectiveness of our method.
We make 10 random restarts for presenting all the results presented later for both MNIST and CIFAR-10.
The budgets for three adversaries, i.e ., ϵ 1 , ϵ 2 , and ϵ ∞ , are the same with the setting at training for both two datasets, while for the number of iterations, we increase them to (100 , 200 , 100) for ( ℓ ∞ , ℓ 2 , ℓ 1 ) on MNIST and (100 , 500 , 100) for ( ℓ ∞ , ℓ 2 , ℓ 1 ) on CIFAR-10.
4.2 Results on MNIST
Here we present results on the MNIST dataset, summarized in Table 1.
Though MNIST has been treated as an “easy” benchmark compared with CIFAR-10 or bigger datasets, e.g ., ImageNet (Deng et al., 2009), we notice that all the single target adversarial training methods, i.e ., ℓ 1 , ℓ 2 , and ℓ ∞ , fail to defend only three attacks, while the best is ℓ ∞ training which defends almost three attacks and outperforms the MAX on the overall Accuracy.
From Table 1, with our proposed method, we notice that the robust accuracy against the ℓ ∞ PGD attack is improved with three methods, i.e ., MAX, MSD, and AVG, using both ℓ 1 and ℓ 2 norms.
The ℓ 1 and ℓ 2 robust accuracy against ℓ 1 and ℓ 2 PGD attacks is improved on MAX by 4 . 3% and 2 .
2% (with adaptive budget using ℓ 1 norm), 2 . 6% and 2 .
3% (with adaptive budget using ℓ 2 norm) as well.
Maini et al.
and Tramer & Boneh (2019) mentioned [Table] “w. adaptive budget” refers to employing our proposed framework which enables adaptive budget to avoid any player dominating the game. “AA” refers to AutoAttack.
“*” means that the results are reproduced from the implementation of Maini et al.
(2020) by the hyperparameters of Maini et al.
“ ℓ 1 (ours)” and “ ℓ 2 (ours)” refers to employing our proposed adaptive budget method w.r.t ℓ 1 and ℓ 2 norms.
[Table]
that there is a trade-off between the robust accuracy against the ℓ ∞ attack and the robust accuracy against the ℓ 1 and ℓ 2 attacks.
Similar observations can be obtained from our experimental results.
For
MSD and AVG, the robust accuracy defending ℓ 1 and ℓ 2 PGD attacks drops a bit, which might be due to that trade-off.
We also observe that the proposed adaptive budget method avoids the player domination phenomenon well as it improves the robust accuracy of MAX by approximately 40% .
The all PGD robust accuracy of the vanilla MAX also shows that the player domination phenomenon hinders MAX from achieving a satisfying robust accuracy for the non-convex scenarios as well.
4.3 Results on CIFAR-
The results are shown in Table 2, and the curve of robust accuracy is shown in Figures 5 and 4.
Due to the limitation of space, we present the most important results in the main paper while leaving the left results in the Appendix.
Main Results.
The results presented in Table 2 show the generalization ability of our proposed method, which improves the robust accuracy of three methods, i.e ., MSD, MAX, and AVG, against ℓ 1 , ℓ 2 , ℓ ∞ , and all attacks by PGD and AutoAttack.
We notice that the robust accuracy of all attacks for PGD and AutoAttack is mainly restricted by how well the model defends the ℓ ∞ attack.
It might be caused by the fact that the radius of ℓ ∞ attack might be too small compared with the radius of ℓ 1 and ℓ 2 attacks such that with the updates by gradient-based algorithms, the gradient of ℓ ∞ adversary is covered by others.
In this way, the model ignores the ℓ ∞ adversary.
And we notice that employing the adaptive budget with either the ℓ 1 or ℓ 2 norm helps models to pay attention to the tasks that are not well learnt as the robust accuracy on ℓ ∞ adversary is relatively improved by the most.
For example, the ℓ ∞ PGD robust accuracy of MAX with the adaptive budget w.r.t the ℓ 1 norm experiences a relative 15 .
01% improvement while there is only a 14 . 03% relative improvement on the ℓ 2 PGD robust accuracy.
Besides, the trade-off between the three attacks is different from the trade-off on MNIST.
On MNIST, the robust accuracy of ℓ 2 adversary is related to that of ℓ 1 adversary while on CIFAR-10, it seems that ℓ 2 robust accuracy is more likely to relate to ℓ ∞ robust accuracy.
Conclusion
In this paper, to achieve the ultimate goal of robustness, i e ., defending any terms of attacks, we first formalized this problem within the scope of the bargaining game, and investigated the convergence property of MAX, MSD, and AVG under two linear cases.
We found that MSD and MSD did not converge theoretically due to a phenomenon named player domination while AVG did not suffer from this.
To avoid player domination in the training of robust models, we designed a novel framework for multi-target adversary training with an adaptive budget method.
Specifically, the adaptive budget method adaptively changed the budget of different attacks to avoid player domination.
Finally, to evaluate the proposed framework, we conducted experiments on two benchmarks, i e ., MNIST and
CIFAR-10.
Experimental results showed that our proposed adaptive budget method improved the robust accuracy on two benchmarks, which complemented our theoretical results and also supported our finding that player domination might interfere with the training of robust models
A Related work
Adversarial Training.
Goodfellow et al.
(2015) has shown that a single, small step in the direction of the sign of the gradient may fool deep learning models for the image classification task.
Later it was extended to a multi-step attack (Kurakin et al., 2017) namely the Basic Iterative Method (now typically referred to as the PGD attack), which significantly improves the success of creating adversarial examples.
After that, different variations of PGD attack (Brendel et al., 2019; Li et al., 2019; Croce & Hein, 2020b) have been proposed to overcome heuristic defenses and create stronger adversaries.
To defend the adversaries, numerous defend methods (Papernot et al., 2016; Kannan et al., 2018; Madry et al., 2018; Zhang et al., 2019b,a, 2020; Wu et al., 2020; Wang et al., 2020; Pang et al., 2020b; Shi et al., 2020; Zhang et al., 2022a) have been discovered.
Among these methods, the most successful defense method is adversarial training (Madry et al., 2018), which formulates the defense problem as a min-max optimization problem and has become one of the few adversarial defenses that is still robust to the following stronger attacks (Carlini & Wagner, 2017; Athalye et al., 2018; Mosbach et al., 2018).
Following that, the empirical robustness (Pang et al., 2020a, 2021; Gao et al., 2022; Zhang et al., 2022b; Wang et al., 2021) has been significantly advanced during the past few decades.
Multi-target Adversarial Training.
Robustness against multiple types of attacks simultaneously relates to our work very closely.
In 2019, Schott et al.
(2019) used multiple variational autoencoders and constructed an architecture named analysis by synthesis for the MNIST dataset.
Their experimental results showing that even for MNIST, it is hard to train a model robust to three different adversaries.
Following that, (Tramer & Boneh, 2019) investigated the theoretical and empirical trade-offs of adversarial robustness when defending against aggregations of multiple adversaries.
Their results showed that the model robust to the ℓ ∞ adversary might not be able to defend other attacks, i.e ., ℓand ℓ 2 attacks, on MNIST.
To alleviate this problem, they designed a augmentation-based method to get ℓ 2 robustness instead of using the ℓ 2 ball.
Later, Croce & Hein (2020a) proposed a provable adversarial defense against all ℓ p norms for p ≥ 1 with regularization methods.
From a greedy search perspective, Maini et al.
(2020) suggested that taking the worst-case over all steepest descent directions helps achieve better performance than MAX and AVG empirically.
Recently, while not studied as a defense method, (Kang et al., 2019a) investigated the transferability of adversarial robustness between models trained against different perturbation models.
B Preliminaries B.1 Problem Formulation
The goal of multi-target adversarial training is to learn a function f
w : X → {− 1 , +1 } that is robust to adversarial examples generated by multiple adversaries 2 , where f is parameterized by w .
The multi-target robust loss of f w is defined as E ( x ,y )
[max δ ∈B ℓ ( f w ( x + δ ) , y )]
where
B = B 1 ( ϵ 1 ) (cid:83) B 2 ( ϵ 2 ) (cid:83) B ∞ ( ϵ ∞ ) and B p ( ϵ ) =
{ δ : ∥ δ ∥ p ≤ ϵ } , and δ is the perturbation.
In deep learning scenarios, Adversarial Training (Madry et al., 2018) is frequently used to train a robust classifier.
Previous multi-target adversarial training work, e.g ., MSD, MAX and AVG, employs the following minimax objective to update the algorithm,
[Equation]
In the deep learning scenario, as this problem is often non-convex and hard to derive the closed-form solution, the above objective is optimized by iteratively optimizing δ and w for several rounds as shown below,
Algorithm gets the optimal perturbation δ t for the t -th round as
[Equation]
Algorithm gets the optimal parameter w t for the t -th round as
[Equation]
[Equation]
4: 5: AVG
Return function
: 9: δ
0 ; 10: for i ∈
[ k ] do 11: δ ip = PGD Step ( x , δ i , η, ℓ p , ϵ p , ℓ ) ; 12: δ i +1 =
argmax δ p ∈{ δ i 1 , δ i 2 , δ i ∞ } loss ( x + δ tp , y ) ; 13: end for 14: Return
To find the approximate optimal perturbation δ and the approximate optimal parameter w under the non-convex scenario, stochastic gradient descent and projected gradient descent (PGD) methods are used to approximately solve the above minimization on w and maximization on δ problems.
Specifically, PGD runs several predefined PGD step PGD step =
Proj Ball p (0 ,ϵ p ) ( δ + η sign( ∇ ℓ ( x + δ , y ))) to approximately find a worst-case adversarial example, where ∇ ℓ ( x + δ , y ) is the gradient of ℓ ( x + δ , y ) .
Tramer & Boneh (2019) first proposed to solve the inner maximization problem of the problem equation 4, i.e ., problem (5), by the MAX (the worst-case perturbation, Algorithm 2) and AVG (augmentation of all perturbations, Algorithm 2) as below
[Equation]
where δ 1 , δ 2 and δ ∞ are obtained by l 1 , l 2 and l ∞ PGD adversaries.
Later, Maini et al.
(2020) designed a “greedy” algorithm named MSD, which solves the inner maximization problem by simultaneously maximizing the worst-case loss overall perturbation models at each projected steepest descent step as shown in Algorithm 2.
And then the inner maximization becomes as follows
[Equation]
C Additional Lemmas
We denote the following single-target adversarial training problem
[Equation]
where p ∈ { 1 , 2 , ∞} is given before the training procedure.
Lemma 6 (lemma D.1 (Tsipras et al., 2019)) .
The optimal solution w ∗ =
( w 1 , ..., w d +1 ) of our optimization problem equation 3 must satisfy w 2 = ...
= w d +1 and sign ( w 2 )
= sign ( µ ) .
Lemma 7 (lemma D.2 (Tsipras et al., 2019)) .
The optimal solution w ∗ =
( w 1 , ..., w d +1 ) of our optimization problem equation 3 must satisfy w 1 ≤ 1 / √ 2 and w 2 = ...
=
w d +1 ≥ 1 / √ 2 d .
Lemma 8.
In the adversarial training framework, for arbitrary step t , if ϵ > µ and
[Equation]
) √ d, 1))]
E [max(0 , 1 + 1 / √ 2(1 + ϵ ) − N (( ϵ − µ ) (cid:113) d 2 , 0 . 5))] ,
[Equation]
the optimal solution w ∗ t = ( w 1 t , ..., w d +1 t ) of our optimization problem must satisfy w 1 t ≤ 1 / √ 2 and w 2 t = ...
=
w d +1 t and | w 2 t | ≥ 1 / √ 2 d .
Moreover, sign ( w it )
= − sign ( w it +1 ) , i ∈
[2 , d + 1]
Proof.
t = 0 , by Theorem 7 the result holds and sign ( w i 0 )
= sign ( µ ) ,
i ∈ [2 , d + 1] .
t = 1 , the perturbed distribution is given by
[Equation]
Assume for the sake of contradiction that w 11 ≥ 1 / √ 2 , by Theorem 6 we have 0 ≥ w
= ...
=
w d + ≥ − 1 / √ 2 d .
Then, with probability at least 1 − p , the first feature predicts the wrong label and without enough weight, the remaining features cannot compensate for it.
Concretely,
[Equation]
We will now show that a solution that assigns zero weight on the first feature ( w 21 = 1 / √ d and w 11 = 0 ), achieves a better margin loss,
[Equation]
Because
[Equation]
we have E [max(0 , 1 − y w ∗ T 1 ( x − δ ∞ ))] ≥ E [max(0 , 1 − y w 1 ( x − δ ∞ ))] , which yields contradiction.
Besides, in this case sign ( w i 1 ) =
sign ( µ − ϵ )
= − sign ( µ ) =
− sign ( w i 0 ) , i ∈ [2 , d + 1] t = 2 , the perturbed distribution is given by
[Equation]
Assume for the sake of contradiction that w 12 ≥ 1 / √ 2 , by Theorem 6 we have 0 ≥ w
= ...
= w d +1≥ − 1 / √ 2 d .
Then, with probability at least 1 − p , the first feature predicts the wrong label and without enough weight, the remaining features cannot compensate for it.
Concretely,
[Equation]
We will now show that a solution that assigns zero weight on the first feature ( w 22 = 1 / √ d and w 12 = 0 ), achieves a better margin loss.
[Equation]
Because
[Equation]
we have E [max(0 , 1 − y w ∗ T 1 ( x − δ ∞ ))] ≥ E [max(0 , 1 − y w 1 ( x − δ ∞ ))] , which yields contradiction.
Besides, in this case sign ( w i 2 ) = sign ( µ + ϵ ) =
sign ( µ ) =
− sign ( w i 1 ) , i ∈ [2 , d + 1]
By induction we can easily derive that w 1 t ≤ 1 / √ 2 , w 2 t = ...
=
w d +1 t , | w 2 t | ≥ 1 / √ 2 d and sign ( w it )
= − sign ( w it +1 ) , i ∈ [2 , d + 1] for all t ≥ 0 .
Lemma 9.
If z ∼ N ( µ, σ 2 ) ,
[Equation]
Lemma 10.
When µ ≥ 4 / √ d , ϵ ≥ 2 µ , and p ≤ 0 .
977 , the optimal solution w ∗ t = ( w 1 t , ..., w d +1 t ) of our optimization problem must satisfy w 1 t ≤ 1 / √ 2 and w 2 t = ...
=
w d +1 t and | w 2 t | ≥ 1 / √ 2 d .
Proof.
Let µ = m/ √ d, m ≥ 4 , ϵ = kµ, k ≥[-
-]
[Equation]
Consider the function h ( a )
=
[Equation]
By numerical simulation we have h ′ ( a ) ≥ 0 , when a ≤ 0 , so h ( a ) is increasing with a when a ≤ 0 , thus − max( E [max( [Equation]
N
[Equation]
√
[Equation]
E [max(0 ,
[Equation]
+
[Equation] √ [Equation] [Equation]
E
[Equation]
max
[Equation]
. 5))] ) ≥ 1 − h ( − 3) = 0 . 9775 > p, by Theorem 8 we have the optimal solution w ∗ t = ( w 1 t , ..., w d +1 t ) of our optimization problem must satisfy w 1 t ≤ 1 / √ 2 and w 2 t = ...
=
w d +1 t and | w 2 t | ≥ 1 / √ 2 d .
Lemma 11.
When w 1 t ≤ 1 / √ 2 and w 2 t = ...
=
w d +1 t and | w 2 t | ≥ 1 / √ 2 d , if ϵ ∞ ≥ 2 d ϵ 1 and ϵ ∞ ≥ (cid:113) 2 d ϵ 2 , ∞ -player dominates 1 -player and 2 -player..
In another word, the training procedure cannot converge.
Proof.
Let ℓ p = 1 − yw ⊤ ( x + δ p ) , we have
[Equation]
Now, we have proved that ∞ -player dominates others and sign( w ti )
= − sign( w t − 1 i ) .
With Lemma 8, we know that at any time t , we have | w it − w it − 1 | ≥ (cid:112) 1 /d, ∀ i ∈
[2 , d + 1] , which means the training procedure cannot converge.
Lemma 12.
MAX and MSD are the same under the SVM scenario.
Proof.
Under the deep learning cases (non-linear and non-convex), MSD follows the steepest direction ( ℓ 1 , ℓ 2 or ℓ ∞ ) in each PGD step to find the perturbation which approximately maximizes the loss function, while MAX uses PGD to find the perturbations empirically and then chooses the perturbation maximizing the loss function.
MSD and MAX are different approaches in deep learning cases (non-linear and non-convex).
On the other side, under the SVM (convex and linear) case, the optimal perturbations with ℓ 1 , ℓand ℓ ∞ constraints have analytical solutions as shown in Eq.
equation 2.
In this way, both MSD and
MAX can directly determine which perturbation maximizes the loss within one step, which means
MSD and MAX are the same under the SVM case.
Standard classification is easy .
Remind that the data consists of a robust feature x 1 , which is strongly related to the label and d non-robust features x i , i ∈ [2 , d + 1] , which are weakly related to the label y .
But with the non-robust features, we can construct a simple linear classifier f that achieves over 99% natural accuracy as
[Equation]
For the natural accuracy, we have
[Equation]
when η ≥ 3 √ d .
Robust classification is not easy .
We have the opposite observation when facing ℓ ∞ adversarial training.
The robust accuracy is shown as
[Equation]
when ϵ ∞ = 2 η .
In the following part of our paper, we show that it is not only difficult to get a fairly good robust accuracy, but also a converged model under the multi-target adversarial training problem.
D Proofs D.1 Proof of Theorem
Proof.
Combining Lemma 8, Lemma 11, and Lemma 12 yields this theorem.
D.1.1 Proof of Theorem
Proof.
As the ∞ -player dominates this game, the multi-target adversarial training problem reduces to the single-target problem equation 6.
Further, with Lemma 8, for non-robust feature i , at any time t , we have sign( w ti ) =
− sign( w t − 1 i ) .
Thus the training procedure does not converge.
D.1.2 Proof of Theorem
Proof.
For the i -th player’s loss (the i -th player dominates the bargaining game at the time t ), as the loss function is µ -strongly convex, we have
[Equation]
For the j -th player’s loss and j ̸ = i , as the loss function is µ -strongly convex, we have
[Equation]
That means at time t , the loss of all player will keep increasing.
And thus, if one player dominate the bargaining game throughout the whole game, the loss of all players and will keep increasing during the whole game, which means the bargaining game might not converge.
D.1.3 Proof of Theorem
Proof.
As the loss function is L -smooth [Equation] ∀ i
[Equation]
we have
[Equation]
Summing the above inequality from i = 1 to K [Equation] we have
[Equation]
The proof is completed
[Equation]
E Extra Experiments
Due to the limitation of space, we put the experimental verification of our negative results and
Figure 4 here.
E.1 Implementation Details
For each algorithm, we employ the default hyper-parameter introduced in their original papers.
We implement all the methods on PyTorch (Paszke et al., 2019) with a single NVIDIA A100 GPU and optimized by the mini-batch SGD with the size of 128 and weight decay.
The raw image is resized to 28 × 28 pixels for MNIST and 32 × 32 pixels for CIFAR-10 as inputs.
Models.
Following Maini et al.
(2020) and Madry et al.
(2018), for MNIST, we use a four-layer convolutional network which consists of two convolutional layers of 32 and 64 5 × 5 filters and 2 units of padding, followed by a fully connected layer with 1024 hidden units, where both convolutional layers are followed by 2 × 2 Max Pooling layers and ReLU activations.
Similarly, following Maini et al.
(2020), for CIFAR-10, we use the pre-activation version of the ResNet18 architecture that consists of nine residual units with two convolutional layers (He et al., 2016).
Attacks used for training.
For MNIST, following Maini et al.
(2020), the setting of three adversaries is shown below.
The ℓ ∞ adversary uses a step size α = 0 .
01 within a radius of ϵ = 0 .
3 foriterations.
The ℓ 2 adversary uses a step size of α = 0 .
1 within a radius of ϵ = 2 .
for 100 iterations, and the ℓ 1 adversary uses a step size of α = 0 .
8 within a radius of ϵ = 10 for 50 iterations.
By default, the attack is run with two restarts, one starting with δ = 0 and another by randomly initializing δ in the allowable perturbation ball.
Similarly, following Maini et al.
(2020), for CIFAR-10, the setting of three adversaries is shown as follows.
The ℓ ∞ adversary used a step size α = 0 .
003 within a radius of ϵ = 0 .
03 for 40 iterations.
The ℓ 2 adversary used a step size α = 0 .
within a radius of ϵ = 0 .for
50 iterations, and the ℓ 1 adversary used a step size α = 1 . 0 with ϵ = 12 for 50 iterations.
Hyperparameter setting and tuning.
We did not tune any hyperparameters as our work is to show a phenomenon and solve it with our proposed adaptivebudget method.
All the hyperparameters are directly copied from Maini et al.
Specifically, on MNIST, for all the models, we used the
Adam (Kingma & Ba, 2015) without weight decay, and used a variation of the learning rate schedule from Smith (2018), which is piecewise linear from 0 to 10 − 3 over the first 6 epochs, and down to 0 over the last 9 epochs.
On CIFAR-10, for all the models, we used the SGD optimizer Robbins &
Monro (1951) with momentum 0 .
9 and weight decay 5 × 10 − 4 .
We used a variation of the learning rate schedule from Smith (2018) to achieve superconvergence in 50 epochs, which is piecewise linear from 0 to 0 .
1 over the first 20 epochs, down to 0 .
005 over the next 20 epochs, and finally back down to 0 in the last 10 epochs.
E.2 Verifying effects of player domination on the SVM case
100 Epoch
T r a i n i n g l o ss     100 Epoch 9601000 # F li p _ w e i g h t s
[Figure] nu m [Figure] [Figure]
100 Epoch
T r a i n i n g l o ss     100 Epoch −0.04 −0.
# F li p _ w e i g h t
[Figure]
Figure 1(a) shows the data of model trained with MAX and MSD
To verify our theoretical results, we conduct experiments and the corresponding results are shown in
Figure 1.
We use a fully connected network (Fully Connected Layer (in= d , out= 1 ), where d = 1000 ).
For the data generation, we set p = 0 .
95 , µ = 4 / √ d , and ϵ 1 = ϵ 2 = ϵ ∞ = 2 µ , and the sample size is 100000 .
We notice that with MAX or MSD (they are equal under the SVM scenario, Lemma 12), the training procedure cannot converge as the training loss is fluctuating while the number of weights whose signs are flipped compared with last epoch is almost 1000.
At the same time, AVG does converge.
That complements our theoretical results (Theorem 1).
Besides, we also conduct experiments verifying the conjecture that when ℓ 1 or ℓ 2 player dominates the bargaining game, the training procedure does not converge as well.
For the case when ℓ 1 dominates, we set ϵ 1 = 4 µ, ϵ 2 = ϵ ∞ = 2 µ , while when ℓ 2 dominates, we set ϵ 2 = 4 µ, ϵ 1 = ϵ ∞ = 2 µ .
We observe exactly the same curves as Figure 1, showing that when ℓ 1 and ℓ 2 dominates, the training procedure with MAX and MSD cannot converge while with AVG, training procedure does converge.
We present the results in Figures 2 and 3.
[Figure]
= 2 µ .
Figure 2(a) shows the data of model trained with MAX and MSD using the SVM model (Sec 2.1) while Figure 2(b) shows the number of model trained with AVG.
100 Epoch
T r a i n i n g l o ss     100 Epoch 9601000 # F li p _ w e i g h t s _ nu m [Figure] and [Figure]
100 Epoch T r a i n i n g l o ss     100 Epoch −0.04 −0.
# F li p _ w e i g h t s _ nu m
[Figure]
Norm choice in adaptive budget.
We use ℓ 1 and ℓ 2 norms for the adaptive budget and the corresponding results are shown in Table 1.
There is no significant difference between the experiments with ℓ 1 and ℓ 2 norms when using our proposed method.
The differences in overall robust accuracy are only 0 .
1% , 1 . 1% , and 1 . 0% on MAX, MSD, and AVG respectively.
The difference in each robust accuracy is also small.
E.3 Results on CIFAR
Similar, for the CIFAR10 dataset, we require that the adapted epsilon are bigger than the half of and smaller that twice of the original epsilon.
Norm choice in adaptive budget.
We notice that the choice of norm in the adaptive budget barely influences the robust accuracy as shown in Table 2.
On both three methods, i.e ., MAX, MSD, and
AVG, our proposed adaptive budget is able to improve the performance with both ℓ 1 and ℓ 2 norms, while the difference between ℓ 1 and ℓ 2 norm is only 0 .
6% and 1 .
4% and 2 .
8% , 1 . 3% and 0 .
9% on MAX, MSD, and AVG against PGD and AutoAttack adversaries.
[Table]
ℓ 1 P
G D
A cc u r a c y
[Table]
P
[Table]
D
[Table]
u r
[Table]
c y
[Table]
ℓ
AA
A cc u r a c y
[Table]
ℓ 1 P
G D
A cc u r a c y
[Table]
ℓ 2 radius 0.2 0.
ℓ 2 P G
D A cc u r a c y 0.00 0.02 0.04 0.06 0.08 ℓ ∞ radius 0.
ℓ ∞ P
G D
A cc u r a c y   60 ℓ 1 radius 0.
ℓ
AA
A cc u r a c y 0.
2.0 ℓ 2 radius 0.
ℓ 2 AA
A cc u r a c y 0.00 0.02 0.04 0.06 0.08 ℓ ∞ radius 0.
ℓ ∞
AA
A cc u r a c y
Robustness curves.
The robustness curves are shown in Figures 5 and 4.
The lines of MAX with either ℓ 1 or ℓ 2 norm-based adaptive budget method are higher than the lines without the adaptive budget method.
The gap between lines with the adaptive budget method and lines without is biggest when the budget of the adversary is small.
Similar observation can be obtained from the line of MSD and AVG in Figure 4.
[Figure]