C OMPOSITE A DVERSARIAL T RAINING FOR M ULTIPLE A DVERSARIAL P ERTURBATIONS AND B EYOND
Anonymous authors
Paper under double-blind review
A BSTRACT
One intriguing property of deep neural networks DNNs) is their vulnerability to adversarial perturbations
Despite the plethora of work on defending against individual perturbation models, improving DNN robustness against the combinations of multiple perturbations is still fairly under-studied
In this paper , we propose composite adversarial training (C AT ), a novel training method that ﬂexibly integrates and optimizes multiple adversarial losses, leading to signiﬁcant robustness improvement with respect to individual perturbations as well as their “compositions”.
Through empirical evaluation on benchmark datasets and models, we show that C AT outperforms existing adversarial training methods by large margins in defending against the compositions of pixel perturbations and spatial transformations, two major classes of adversarial perturbation models, while incurring limited impact on clean inputs
I NTRODUCTION
Despite their state-of-the-art performance in tasks ranging from computer vision (Szegedy et al., 2016) to natural language processing (Seo et al., 2017), deep neural networks (DNNs) are inherently susceptible to adversarial examples (Szegedy et al., 2014), which are maliciously crafted samples to deceive target DNNs.
A ﬂurry of adversarial attacks have been proposed, which craft adversarial examples via either pixel perturbation (Goodfellow et al., 2015b; Moosavi-Dezfooli et al., 2016;
Carlini & Wagner, 2017a) or spatial transformation (Engstrom et al., 2017; Xiao et al., 2018; Alaifari et al., 2019).
To defend against such attacks, a line of work attempts to improve DNN robustness by developing new training and inference strategies (Kurakin et al., 2017; Guo et al., 2018;
Liao et al., 2018; Tram ` er et al., 2018).
Yet, the existing defenses are often circumvented or penetrated by adaptive attacks (Athalye et al., 2018), while adversarial training (Madry et al., 2018; Shafahi et al., 2019) proves to be one state-of-the-art defense that still stands against adaptive attacks.
While most adversarial training methods are primarily designed for individual attacks which are either ﬁxed (Madry et al., 2018) or selected from a pre-deﬁned pool (Tram ` er & Boneh, 2019; Maini et al., 2020), in realistic settings, the adversary is not constrained to individual perturbation models but free to “compose” multiple perturbation models to construct more powerful attacks.
Despite their robustness against individual attacks, the DNNs trained using existing methods often fail to defend against such composite attacks (details in § 2).
Moreover, the existing adversarial training methods focus on pixel perturbation-based attacks (e.g., bounded by (cid:96) p -norm balls), while the research on training DNNs robust against spatial transformation-based attacks is still limited.
To bridge this striking gap, in this paper, we present C AT , a novel adversarial training method able to ﬂexibly integrate and optimize multiple adversarial robustness losses, which leads to DNNs robust with respect to multiple individual perturbation models as well as their “compositions”.
Speciﬁcally,
C AT assumes an attack model that composes multiple perturbations and, while bounded by the overall perturbation budget, optimally allocates the budget to each iteration.
To solve the computational challenges of this formulation, we extend the recent advances on fast projection to (cid:96) p, 1 mixed-norm ball (Liu & Ye, 2010; Sra, 2012; B´ejar et al., 2019) to our setting and signiﬁcantly improve the optimization efﬁciency.
We validate the efﬁcacy of C AT on benchmark datasets and models.
For instance, on MNIST, C AT outperforms alternative adversarial training methods (Tram`er & Boneh, 2019) by over 44% in terms of adversarial accuracy against attacks that combine pixel perturbation and spatial transformation (details in § 4), with comparable clean accuracy and training efﬁciency.
Our contributions can be summarized as follows.
First, we demonstrate that a new class of adversarial attacks, which “compose” multiple perturbations, render most existing adversarial training methods ineffective; then, we propose C AT , the ﬁrst adversarial training method designed for multiple perturbation models as well as their compositions; further, we validate the efﬁcacy of C AT by comparing it against alternative methods on benchmark datasets and DNNs; ﬁnally, we explore the optimization space of composite perturbations, leading to several promising research directions.
F UNDAMENTALS 2. A DVERSARIAL T RAINING
Adversarial training is a class of techniques to train robust DNNs by minimizing the worst-case loss with respect to a given adversarial perturbation model.
Formally, let f θ be a DNN parameterized by θ , (cid:96) the loss function, and D train = { x i , y i } ni =1 the training set.
Then the adversarial training with respect to an (cid:96) p adversary with perturbation magnitude (cid:15) is deﬁned as: [Equation]
[Equation]
where B p ( (cid:15) )
= { δ : (cid:107) δ (cid:107) p ≤ (cid:15) } is the (cid:96) p -norm ball of radius (cid:15) .
Here, the inner maximization problem essentially deﬁnes the target adversarial attack.
For instance, instantiating it as the (cid:96) ∞ projected gradient descent (PGD) attack leads to the well-known PGD adversarial training.
Despite their effectiveness against considered perturbation models (e.g., (cid:96) ∞ perturbation), the existing adversarial training methods often fail to defend against perturbations that they are not designed for (Tram ` er & Boneh, 2019).
Motivated by this, some recent work explores conducting adversarial training with respect to multiple perturbation models simultaneously.
AVG and MAX – Tram ` er & Boneh (2019) propose two methods, A VG and M AX , to aggregate multiple perturbations.
Speciﬁcally, A VG formulates the robustness optimization as:
[Equation]
where A = { 1 , 2 , ∞} .
Compared with Eq. 1, Eq. 2 aggregates multiple adversarial perturbations in the inner loop.
Similarly, instead of averaging multiple perturbations, M AX selects the perturbation resulting in the largest loss:
[Equation]
If A contains only one adversarial perturbation, Eq. 3, Eq. 2, and Eq. 1 are all equivalent.
MSD – While A VG and M AX achieve varying degrees of robustness to the considered perturbations, it is practically difﬁcult to minimize the worst-case loss with respect to the union of perturbations.
To this end, Maini et al.
(2020) propose multiple steepest descent (M SD ) which improves M AX (and A VG ) along two aspects.
First, it selects the largest (or average) perturbation at each inner iteration; Second, it applies the steepest descent instead of the projected gradient descent in generating adversarial inputs.
Formally, M SD formulates the optimization at the t -th iteration as:
[Equation]
5) where Proj C ( · ) is the projection operator onto the convex set C , and v p ( δ ( t ) ) is the steepest descent direction for (cid:96) p perturbation, v p ( δ )
= arg max (cid:107) v (cid:107) p ≤ λ v T ∇ (cid:96) ( f θ ( x + δ ) , y ) , where λ is the step size.
2. C OMPOSITE A DVERSARIAL A TTACK
While the existing adversarial training methods seem effective against individual perturbation models which are either ﬁxed or selected from a pre-deﬁned pool (i.e., the union of perturbations), in natural image airplane car bird cat deer frog truck deer dog horse truthlabel predictedlabel dog frog horse ship truck δ pixel δ ﬂow perturbedimage bird dog dog airplane car a realistic setting, the adversary is able to combine multiple perturbation models to construct more destructive attacks, which we exemplify with a new class of composite adversarial attacks.
Intuitively, the attack constructs an adversarial example by applying a sequence of perturbation models {A i } mi =1 , each A i bounded by an independent perturbation budget (cid:15) i .
Here, we treat A i as an abstract operator
A i ( · , (cid:15) i )
(e.g., pixel perturbation or spatial transformation), which applies the corresponding perturbation (bounded by (cid:15) i ) over the output of its previous perturbation:
[Equation]
We can further generalize the attack to a more ﬂexible setting in which the adversary, while bounded by the overall perturbation budget, is able to optimally allocate the budget to each optimization iteration.
The details of this generalization are discussed in Appendix B. Figure 1 illustrates samples generated by the composite adversarial attack which combines one pixel perturbation (with budget (cid:15) p ) and one spatial transformation (with budget (cid:15) f ).
Given attacks {A i } mi =1 with A i bounded by (cid:15) i , in the composite attack, we re-scale the perturbation budget by a factor of 1 /m (i.e., (cid:15) i /m for A i ), to make the composition of {A i } mi =1 comparable with the union attack. Intuitively, with proper setting of { (cid:15) i } mi =1 , the composition of {A i } mi =1 is strictly stronger than each individual attack as well as their union (detailed proofs in Appendix A).
Figurecompares the robust accuracy of A VG , M AX , and M SD on the unions and compositions of multiple perturbations.
Observe that while effective on the union attacks, the existing methods fail to defend against the composite attacks, with robust accuracy drop as large as 40%. [Figure]
CAT: C OMPOSITE A DVERSARIAL T RAINING
We now present C AT , a new adversarial training method to defend against multiple perturbations as well as their compositions.
3. F ORMULATION
At a high level, C AT adopts the composite adversarial attack as the inner loop of Eq. 1, which generates adversarial examples through a sequence of perturbations:
[Equation]
y ) s.t. (cid:107) δ i (cid:107) ≤ (cid:15) i for i = 1 , . . .
, m (7)
As concrete instances, in the case of composing pixel perturbations (i.e., (cid:96) 1 , (cid:96) 2 , (cid:96) ∞ perturbations) with budget (cid:15) 1 , (cid:15) 2 , and (cid:15) ∞ , respectively, the straightforward composite adversarial examples is produced by x ∗ =
x + δ 1 + δ 2 + δ ∞ , where we omit the clipping operator.
Next we consider composing pixel perturbation and spatial transformation (Xiao et al., 2018; Alaifari et al., 2019) with budget (cid:15) p and (cid:15) f respectively.
We ﬁrst give a brief introduction of spatial transformation.
Instead of directly perturbing the values of pixels, spatial transformation displaces their coordinates.
Formally, the input x is represented as a set of tuples x = { ( u i , v i , b i ) } ni =1 , where ( u i , v i ) are the coordinates of x ’s i -th pixel and b i is its value.
We set f = { (∆ u i , ∆ v i ) } ni =1 (with abuse of notations here), where (∆ u i , ∆ v i ) are the displacement of x ’s i -th pixel.
When we apply f to x to generate the adversarial input x (cid:48) , ( u i , v i ) =
( u (cid:48)
i + ∆ u i , v (cid:48)
i + ∆ v i ) .
Typically, bi-linear interpolation is applied to handle fractional pixel positions (Jaderberg et al., 2015): b (cid:48)
i =
(cid:80) q ∈N ( u i ,v i )
b q (1 − | u
i − u q | )(1 − | v i − v q | ) , where N ( u i , v i ) is the neighboring points of ( u i , v i ) .
Following Alaifari et al.
(2019), we measure the perturbation magnitude as f ’s (cid:96) ∞ -norm: (cid:107) f (cid:107) ∞
=
max { max i | ∆ u i | , max i | ∆ v i |} . Max Max Max
Attack A Attack B i - t h
I t
e r a t i o n
Attack A Attack B i - t h
I t
e r a t i o n
Attack A Attack B i - t h
I t
e r a t i o n
Avg
Attack A Attack B i - t h
I t
e r a t i o n
[Figure]
3. D ISCUSSION
Figure 3 compares different adversarial training methods (A VG , M AX , M SD , and C AT ).
The design of C AT enjoys two major beneﬁts.
First, by deﬁnition, the composite adversarial attack naturally covers the strongest individual attack and the union of these attacks, as demonstrated in the two instantiations above.
Second, C AT generalizes adversarial robustness from individual attacks which are either ﬁxed or selected from a ﬁxed pool to their compositions.
As composite adversarial attacks are by nature stronger than individual attacks, setting the perturbation budget overly large in C AT may cause accuracy degradation with respect to clean inputs.
We propose a variant α -C AT to mitigate this issue.
With 0 < 1 ≤ α as a hyper-parameter, we re-scale the perturbation budget of each component attack A i to α(cid:15)
i during the adversarial training.
C AT -r
One issue with the above α -C AT formulation is the trade-off between attack strength of individual component attack and clean accuracy. With smaller α , we expect a high clean accuracy. However, the component attack might not strong enough to cover the original perturbation size.
For instance, if we take α = 13 , the trained model may have low robust accuracy for each individual attack and hence their union.
On the contrary, a very large α (close to 1) causes signiﬁcant drop in the clean accuracy, make the robust model is not useful.
Our work-round solution for this is to use smaller α to ensure clean accuracy, while we sample component attacks with replacement during adversarial training to enhance the robustness against individual attacks.
In fact, under this implementation, we may sample a multiple stack of the same type attack, which corresponds to that attacker with larger perturbation size.
E MPIRICAL E VALUATION
We empirically evaluate the efﬁcacy of C AT in various settings.
All the experiments are performed on MNIST and CIFAR10 dataset.
Speciﬁcally, our results convey two key messages.
First, C AT trains DNNs robust against both composition perturbations and union perturbations in the (cid:96) p space ( (cid:96) 1 , (cid:96) 2 and (cid:96) ∞ ), suggesting that composite adversarial robustness is a generalization of the adversar- ial robustness with respect to the union of multiple perturbations.
Second, C AT is able to train DNNs robust against both pixel perturbation (e.g., (cid:96) ∞ perturbation) and free-form spatial transformation.
Models and Hyper-parameters – On MNIST, we use the network architecture used by both Maini et al.
(2020) and Madry et al.
(2018), which is a DNN consisting of 4 convolutional layers, followed by 2 fully connected layers.
On CIFAR10, we use a pre-activation version of ResNet32 (He et al., 2016), which is build up with 16 residual blocks, followed by 1 global averaging pool layer, andfully connected layer.
Setting of α – As discussed in § 3.2, to minimize the performance degradation on clean inputs, we re-scale the allowed perturbation magnitude uniformly by α .
To set α properly, we conduct a grid search within { 1 . 0 , 0 . 9 , 0 . 8 , 0 . 7 , 0 . 6 , 0 . 5 } and select the optimal α value by optimizing the trained model’s robust accuracy (under the union threat model) and clean accuracy (with less than 10.0% drop from all baselines under the union threat model).
We implement all the algorithms with PyTorch and run all the experiments on a single Nvidia RTX 6000.
The detailed setting of models and (hyper-)parameters is summarized in Appendix B.
4. R OBUSTNESS IN (cid:96) p S PACE
In the ﬁrst part, we evaluate C AT and baselines on the regular pixel perturbation-based attacks.
Baselines – We compare C AT with M AX - worst-case perturbation, A VG - average-case perturbation (Tram ` er & Boneh, 2019), and M SD (Maini et al., 2020).
Besides, we also include robust DNNs trained with PGD attacks with respect to individual perturbations.
Component Attacks for C AT – We consider 3 commonly used (cid:96) p perturbations, namely (cid:96) ∞ , (cid:96) 2 , and (cid:96) 1 in C AT .
For (cid:96) ∞ attack, we use (cid:96) ∞ PGD attack (Madry et al., 2018).
For (cid:96) 2 attack, we implement an (cid:96) 2 PGD adversary.
For (cid:96) 1 attack, we use the enhanced (cid:96) 1 attack proposed in Maini et al.
Attacks Used for Evaluation – To evaluate the robustness of DNNs trained by C AT and baselines, we consider a collection of representative adversarial attacks.
Individual perturbations – For (cid:96) ∞ attack, we consider both (cid:96) ∞
PGD attack and Fast Gradient Sign attack (Goodfellow et al., 2015a); for (cid:96) 2 attack, we consider (cid:96) 2 PGD, DeepFool (Moosavi-Dezfooli et al., 2016), C&W attack (Carlini & Wagner, 2017b), and Salt&Pepper attack (Rauber et al., 2017); for (cid:96) 1 attack, we use (cid:96) 1 PGD attack.
Combined perturbations – Besides, we consider both unions and compositions of multiple perturbations.
Following Maini et al.
(2020), in the union threat model, the adversary applies all the attacks on the given input and is considered successful if one of the attacks succeeds.
In the composite threat model, we consider a set of composite attacks with A 1 = (cid:96) ∞ -PGD, A 2 = (cid:96) 2 -PGD, and
A 3 = (cid:96) 1 -PGD.
For the composite attacks, we measure the robust accuracy under different settings of re-scale factor α , which is deﬁned similarly in C AT - α .
Results – Table 1 and Table 2 summarize the results with respect to pixel perturbation-based attacks on CIFAR10 and MNIST.
The perturbation budget for each type of (cid:96) p attack is shown in the tables.
To measure models’ performance on clean inputs, we use their accuracy on the test set.
To evaluate the robustness of each model, we measure the robust accuracy of all the models on a random sample of 1,000 test inputs.
The robust accuracy is deﬁned as the fraction of test inputs that are misclassiﬁed initially or are predicted to wrong classes after an attack.
In reporting robust accuracy, we aggregate results of all the attacks from each target norm.
In other word, an input is correctly predicted after attacks for (cid:96) p norm if and only if all the attacks of (cid:96) p norms from above fail for the input.
Similarly, for the union setting, all the attacks except combined perturbations are considered.
On CIFAR10, with a slight degradation of clean accuracy, C AT achieves the same robustness accuracy as M SD under the union of three (cid:96) p perturbations and outperforms all other baselines.
Furthermore, C AT is much more robust against composite adversarial attacks when α = 0 .
5 , outperforming
M SD and other baselines by over 10%.
On MNIST, PGD- (cid:96) ∞ ( P ∞ ) achieves the best performance across all the settings, which however is attributed to the holding gradient masking effects (Tram`er & Boneh, 2019).
In Appendix C, we show the robust accuracy of models in Table 9 on two decision-based attacks, and we conﬁrm the higher attack success rate with black-box models of P ∞ .
Thus, we exclude P ∞ in the following [Table] [Table] discussion.
Across all the other methods, C AT only performs slightly worse than M AX , A VG [Table] and
M SD under the union threat model.
Under the composite threat model ( α = 0 . 5 ) [Table] C AT outperforms baselines in terms of robust accuracy by margins over 20%.
The above results indicate that C AT assumes a strong adversarial attack model during the adversarial training process and produces DNNs with robustness not only against individual perturbations (and their unions) but also against their compositions.
4. R OBUSTNESS AGAINST P IXEL P ERTURBATIONS AND S PATIAL T RANSFORMATIONS
Next we evaluate C AT and baselines against compositions of pixel perturbations and spatial transformations.
Component Attacks – For pixel perturbation, we consider (cid:96) ∞ PGD attack.
For spatial transformation, we consider a projected gradient descent approach.
The key differences between this formulation and ADef (Alaifari et al., 2019) are that 1) PGD is much faster than the DeepFool procedure proposed in ADef and 2) it uses free-form ﬂows instead of the smoothed ﬂows.
Attacks Used for Evaluation – We evaluate C AT and baselines against the above individual perturbations as well as their unions and compositions.
Baselines – Since M SD is only applicable to pixel perturbation-based attacks, we consider A VG and
M AX as the baselines.
Plus, we use two DNNs that are adversarially trained using the above piexl perturbation and spatial transformation respectively, which we refer to as P pixel and P ﬂow .
Results – Table 3 and 4 summarize the results.
We have the following observations.
First, C AT achieves similar (on MNIST) or even better (on CIFAR10) robust accuracy than the baselines under the union threat model.
Second, C AT outperforms all the baselines under the composite threat model by large margins.
For instance, on MNIST, the robust accuracy of all the baselines drops to close to 0 even with α = 0 .
5 (i.e., half of the speciﬁed perturbation budget); in contrast, C AT attains 46% robust accuracy. [Table] [Table] [Figure]
4. R OBUSTNESS AGAINST C OMPOSITE A TTACKS WITH V ARYING α
Thus far, we have assumed the defender and attacker use the same setting of re-scaling factor ( α = 0 . 5 ) in attacking the DNNs.
Next, we evaluate the impact of varying α by the attacker on the robust accuracy.
Figure 4 summarizes the results on CIFAR10 under α = 0.5, 0.8, and 1. (which correspond to stronger attacks).
Observe that as expected, both C AT and baselines experience performance degradation under large α and yet, C AT still consistently outperforms all the baselines by large margins across all the settings.
E XPLORING THE S PACE OF C OMPOSITE P ERTURBATIONS
In this section, we empirically study the critical properties of composite adversarial attacks and C AT .
One particular aspect is the impact of the ordering of component attacks on the adversarial training.
Besides, we consider an even stronger composite attack which, bounded by overall perturbation budget, is able to optimally allocate the budget to each round.
5. O RDERING OF C OMPONENT A TTACKS
We evaluate the impact of the ordering of component attacks under the setting of (cid:96) p perturbations only as well as compositions of pixel perturbations and spatial transformations.
(cid:96) p Perturbations – In this set of experiments, we consider the compositions of (cid:96) ∞ -PGD and (cid:96) 2PGD attacks.
Table 5 and 6 summarize the performance of C AT under the compositions of (cid:96) ∞ and (cid:96) 2 perturbations.
We have the following observations.
First, ﬂipping the two attacks in C AT has little impact on both clean accuracy and robust accuracy of C AT .
Second, observable from the last two columns of each table, the effectiveness of composite adversarial attacks seems also independent of the ordering of component attacks. [Table] [Table] [Table]
[Table] [Table] [Table] Pixel and Spatial Perturbations – Similarly, we empirically evaluate the impact of the ordering of pixel and spatial perturbations.
Here we follow the same setup as § 4 (the same attacks and α for each dataset).
The results are summarized in Table 7.
We observe that on CIFAR10, the ordering has fairly limited impact as in the (cid:96) p case; however, we ﬁnd that ﬁrst applying pixel perturbation and then spatial transformation results in more robust model on MNIST.
We consider the study of the root cause of this interesting phenomenon as our ongoing work.
5. R OBUSTNESS A GAINST F INE - GRAINED C OMPOSITE A DVERSARIES
Finally, we consider an extension of the basic composite adversarial attack (each perturbation is applied only once) to a multiple round setting.
Under this setting, the adversary, while bounded by the overall perturbation budget, is able to optimally allocate the budget to each iteration, leading to even stronger attacks.
Formulation and Solution – We sketch the differences between C AT and K -round C AT here, with full details deferred to the appendix B.
Perturbation Accounting – For each component perturbation
A i , instead of seeking a single δ i , we divide it into K parts δ i,k for k = 1 , . . .
, K .
We measure the overall perturbation cost as the sum of {(cid:107) δ i,k (cid:107) p } Kk =1 , where p is (cid:96) p norm.
In other word, the new constraint is
[Equation] Optimization Solution – This new constraints essentially speciﬁes a (cid:96) 1 ,p mixed-norm ball constraint.
Therefore, we can still solve this new optimization problem with projected gradient descent.
We extend the recent advances (Liu & Ye, 2010; B´ejar et al., 2019) to solve this new problem for the cases of p = 2 and p = ∞ respectively (details in the appendix B).
Results – Figure 5 displays how K -round composite attack impacts the robust accuracy of C AT under the settings of (cid:96) p perturbations only as well as pixel plus spatial perturbations.
As K increases, the robust accuracy decreases by 4% and 7% respectively on MNIST and CIFAR10 under the setting of pixel plus spatial perturbations, while the decrease is much less evident under the setting of (cid:96) p perturbations only, indicating that C AT is fairly robust to K -round composite attacks.
Moreover, by incorporating K -round composite attacks in the adversarial training of C AT , we expect to see further robustness improvement.
C ONCLUSION
While effective against individual perturbation models, existing adversarial defenses often fail to defend against combinations of multiple perturbations.
In this paper, we ﬁrst present a new class of composite attacks that combine multiple perturbations and penetrate the state-of-the-art defenses.
We then propose composite adversarial training (C AT ), a novel training method that improves DNNs robustness not only against individual perturbations but also against their compositions.
Empirical evaluation on benchmark datasets and models shows its promising performance
A P ROOFS
We assume a set of component models {A i } mi =1 , each A i bounded by budget (cid:15) i . Lemma 1. The union of {A i } m i =1 is stronger than each individual attack.
Proof.
(Lemma 1)
The adversarial loss of a given input ( x, y ) with respect to A i is deﬁned as:
[Equation]
where B i ( (cid:15) ) is the feasible set for A i with bound (cid:15) i .
Meanwhile, the adversarial loss with respect to the union of {A i } mi =1 is deﬁned as:
[Equation]
Given that A i is arbitrarily chosen, the union attack is stronger than each component attack.
Therefore, we only need to show that the composite attack is stronger than the union attack.
We prove this by showing that the feasible set for the composite attack is larger than the union attack.
To simplify the discussion, we consider two (cid:96) p attacks A 1 and A ∞ with budget (cid:15) 1 and (cid:15) ∞ respectively, and d is the data dimensionality.
Lemma 2.
If
[Equation]
, the composition of A 1 and A ∞ is strictly stronger than their union.
[Equation]
[Figure]
Proof.
(Lemma 2)
First, it can be veriﬁed that with the constraint, (cid:15) ∞ < (cid:15) 1 < d(cid:15) ∞ , thus the ddimensional hypercubes represented by (cid:15) 1 and (cid:15) ∞ intersect but none of them completely contains the other.
Figure 6 shows the case for d = 2 .
The feasible set B u for the union attack is given by:
[Equation]
10) where B p ( (cid:15) p ) denotes the feasible set for A p ( p = 1 , ∞ ) .
The volume of B u , Vol( B u ) is given by:
[Equation]
In comparison, the volume of the feasible set B c for the composite attack (with budget (cid:15) 1 / 2 and (cid:15) ∞ / 2 for A 1 and A ∞ ) is given by:
[Equation]
Given the constraint, it is trivial to see Vol( B c ) >
Vol( B u ) , indicating that the composite attack entails a larger perturbation space than the union attack.
This result can be generalized to the cases of other (cid:96) p attacks and more than two component attacks.
We present the detailed setting of C AT , baseline methods, as well as the attacks used in § 4. [Table] [Table]
M ODEL T RAINING
We reuse a few pre-trained models from repository provided by previous work.
For the rest of models, their training use the following setups for each dataset.
• MNIST .
The models are optimized with an Adam optimizer (Kingma & Ba, 2015).
The learning rate linearly increases from 0 to 0.001 in the ﬁrst 6 epochs, and then linearly decreases to 0 in the last 9 epochs.
• CIFAR10 .
The models are optimized by a SGD optimizer with momentum of 0.9.
The learning rate linearly increases from 0 to 0.1 in the ﬁrst 20 epochs, then linearly decrease to 0.005 in the next 20 epochs, and it linearly decay to 0 in the last 10 epochs.
Besides, we regularize models with a (cid:96) 2 weight decay of 5 × 10 − 4 .
B. C AT
• (cid:96) p cases .
We run the composite attack during C AT model training with step sizes of 0 .
1 × (cid:15) p for each p ∈ { 1 , 2 , ∞} .
The number of iterations for attacks is 50.
• pixel and spatial perturbations .
We run the composite attack during C AT model training with step sizes of 0 .
1 × (cid:15) p and 0 .
1 × (cid:15) f for pixel and spatial perturbation.
The number of iterations for attacks is 40.
We also present the re-scaling factor α for the models presented in the main text in Table 8.
B. A DVERSARIAL A TTACKS
In evaluation [Table] we run all the attacks except C&W with 5 random restarts.
Below we summarize all the attacks we used in the evaluation and their hyper-parameters.
• Attacks from Foolbox .
We use the following attacks from Foolbox 3.1.1 1 : (cid:96) ∞ PGD, (cid:96)PGD, (cid:96) 1 PGD, Fast Gradient Sign Method, DeepFool, C&W, Salt&Pepper.
We take their default settings from Foolbox in the evaluation.
• Our Implementation of (cid:96) p PGD Attacks .
Plus, we also implement PGD attacks for three (cid:96) p norms ourselves, which achieves higher attack succces rate than the version from the Foolbox.
For (cid:96) ∞ PGD, we run 200 iterations with a step size of 0 .
1 × (cid:15) .
For (cid:96) 2 PGD, we run 500 iterations with a step size of 0 .
05 × (cid:15) .
Our implementation of (cid:96) 1 PGD attack is based upon Section A.1 of (Maini et al., 2020), where we set the the range of number of pixels to modify in each iteration [ k 1 , k 2 ] as: k 1 = 5 and k 2 = 20 .
The number of iterations of this attack is 200, and the step size is set to 0 .
05 × (cid:15) .
• The PGD Spatial Perturbation Attack .
We run this attack with 200 iterations and step size of 0 .
1 × (cid:15) f
. • Composite Adversarial Attacks .
We run this attack with 200 iterations and step sizes of 0 .
1 × (cid:15) for all the component attacks.
B. B ASELINES
• (cid:96) p Perturbations .
We use pre-trained robust models provided by MSD (Maini et al.
, 2020)for all the baseline methods, including P ∞ , P 2 , P 1 , AVG, MAX, and MSD.
• Pixel and Spatial Perturbations .
The adversarial trained models with (cid:96) ∞
PGD is the same as in the (cid:96) p perturbations case.
For P ﬂow , the number of iterations for spatial attack is 50, and the step size is set to 0 .
1 × (cid:15) f .
The same rule applies to AVG and MAX on the both two attacks.
C K -R OUND C OMPOSITE A DVERSARIAL A TTACK
We describe the detailed formulation for the multiple round composite adversarial attack and technical tools to solve this new attack.
C. F ORMULATION
We denote m component attacks as A 1 , . . .
, A m , and whose perturbation sizes are (cid:15) 1 , . .
, (cid:15) m respectively.
Unlike the deﬁnition in § 2, here we represent each attack
A i as A i ( x, δ i ) , which denotes applying perturbation
δ i onto x with the mechanism of A i .
The K -round attack generalizes composite adversarial attack in the following sense, the attack runs in K -round.
At k -th round, it performs a composite attack with perturbations δ k, 1 , . . .
, δ k,m .
The constraint for adversary is the overall magnitude of perturbations he spent for each attack.
Formally,
[Equation]
where x k is the perturbed sample after the k -th round and δ i =
( δ i, 1 , . . .
, δ i,K ) is the concatenation of the perturbation at each iteration.
Observe that by specifying K , Eqn 13 instantiates a spectrum of attacks.
As K approaches inﬁnity, Eqn 13 essentially considers all ﬁnite combinations of ways of allocating the total budgets { (cid:15) i } mi =1 using the m perturbation mechanisms.
[Equation]
Now we present an iterative projected gradient descent algorithm to ﬁnd a solution to Eqn (13).
We use the superscript ( t ) to denote the value of related variables at t -th iteration.
We randomly initialize δ (0) i .
At the t -th iteration, the update rule is deﬁned as:
[Equation] [Equation]
where α is the learning rate and Π S ( · ) is the projection operator of a convex set S .
The computation of Eqn (14) is straightforward.
We focus our discussion on the projection operator in Eqn (15).
Without loss of generality, we omit the subscript i for simplicity.
Let V be a matrix with its k -th row as δ k (for k = 1 , . . .
, K ).
Then the summation [Equation] can be rewritten as
[Equation]
Additional, We suppose the perturbation of i -th component attack is measured with (cid:96) p norm for some p ∈ { 1 , 2 , ∞} , which hold for all the experiments in the paper.
Thus, we reach to the (cid:96) p,mixed-norm of V , denoted as (cid:107) V (cid:107) p, 1 , which is a special case of (cid:96) p,q mixed-norm ball: (cid:107) V (cid:107) p,q = ( (cid:80) mi =1 (cid:107) v i (cid:107) qp ) 1 /q .
Hence, we cast Eqn (15) as the projection operator onto an (cid:96) p, 1 mixed-norm ball.
For p = 1 , the calculation is straightforward since it reduces to the (cid:96) 1 projection operator for the concatenated vectors.
We work with p = 2 and p = ∞ in the next.
C. I MPLEMENTATION FOR p =
We leverage Algorithm 1 in (Sra, 2012) to compute the (cid:96) 2 , ∞ mixed-norm of an input x .
Interested readers could ﬁnd more details in (Sra, 2012).
C. I MPLEMENTATION FOR p = ∞
We leverage the method proposed in (B´ejar et al., 2019), which solves the proximal operator of (cid:96) 1 , ∞ mixed-norm using an active set approach and attains better efﬁciency than previous methods (Quattoni et al., 2009; Gustavo et al., 2018; Chau et al., 2019).
However, as this is a primitive procedure in C AT , which is executed for hundreds of iterations, the basic implementation in (B´ejar et al., 2019) is not scalable for our setting.
We now improve the scalability of (B´ejar et al., 2019) for C AT , with their original algorithm sketched in Algorithm 1.
The method is based on computing the proximal operator of the dual form of (cid:96) ∞ ,mixed-norm ball, (cid:96) ∞ , 1 mixed-norm.
We deﬁne U =
abs( V ) .
Given an initial radius of (cid:96) 1 ball for its row u i , denoted by r , it iteratively improves r by ﬁrst projecting each row with (cid:96) 1 -norm exceeding r to the (cid:96) 1 ball of radius r (line 4 to 8) and then computing a larger r based on the nonzero elements of the projected row (line 9).
In particular, Algorithm 1 uses sorting to solve the (cid:96) 1 -norm projection problem, which is detailed in Algorithm 2.
We state our more efﬁcient implementation based on two key observations here.
Observation 1 – The projection radius r (line 6) in Algorithm 1 for related rows increases at every iteration.
Thus, for each related row v , we face a set of queries with increasing radii r 1 ≤ · · · ≤ r T , where T is the number of queries.
Following the notations of Algorithm 2, let u denote the sorted v in non-decreasing order, and we deﬁne h k = (cid:80) kj =1 u j − ku k .
Note that h is monotonically increasing:
[Equation]
Leveraging this observation, we optimize Algorithm 2 as follows.
At line 2, K is the largest item among 1 , . . .
, N such that r > h K ; we only need to scan h k one pass to ﬁnd the optimal K t for all a t due to that both h and r are increasing.
At line 3, we only need to pre-process the partial sums once for each row before the main loop.
Observation 2 – To update the radius r of the (cid:96) 1 -norm ball in Algorithm 1, we need to access the statistics of line 7 ∼ 9.
A critical observation is that it is not necessary to explicitly compute the projected vector at every iteration to compute the statistics.
Speciﬁcally, line 7 computes the number of non-zero elements J i for the projected i -th row; and line 9 updates r using the sums of the projected rows (cid:80) nj
=1 x i,j and J i .
Combining the previous observation, if K i,t is the optimal K (line 2 of Algorithm 2) for the t -th radius r t and the i -th row, it holds that
[Equation]
and
[Equation]
where τ i,t is calculated based on line 3 of Algorithm 2 for i -th row with radius r t .
Thus, we can avoid computing the intermediate (cid:96) 1 projections.
Algorithm norm: prox λ (cid:107)·(cid:107) 1 ( · ) B ´ ejar et al.
Input: m × n matrix V
Output: X 1 U ← abs
( V ) ; 2 r ← initial radius computed via Lemma 2 and Lemma 3 in B´ejar et al.
(2019) ; 3 do
M ← { i | r < (cid:107) u i (cid:107) 1 } ; foreach i in M do ( u i ) ;
J
{ j | x i,j > 0 } ; while
Table 9 summarizes the results of all the models in Table 2 on two decision-based attacks: (cid:96) 2Pointwise attack and (cid:96) 1 -Pointwise attack (Schott et al., 2019).
One may notice that the inferior performance of P ∞ model on these black-box attacks compared to its performance in Table 2 for white-box attacks.
In summary, we ﬁnd strong gradient masking effects within this model.
(cid:80)
Nn =1 x n = r , and x n ≥ 0 for n = 1 , · · · , N .
Input: y ∈ R N
Output: x 1 u ← sort v in non-increasing order: u 1 ≥ · · · ≥ u N ; 2 K ← max 1 ≤ k ≤ N { k | ( (cid:80) kj =1 u j − r )
/k <
u k } ; 3 τ ← ( (cid:80)
Kk {+=
+} u k − r ) /K ; 4 for n ← 1 , · · · , N do x n ← max( v n − τ, 0) ; P ∞
P
P
MAX
AVG MSD CAT Pointwise ( (cid:96) 2 ) 68.9% 97.3% 98.4% 94.0% 92.9% 95.3% 87.5%
Pointwise ( (cid:96) 1 ) 22.5% 92.7% 93.9% 83.4% 73.0% 86.6% 82.0%
D. V ISUALIZATION OF C OMPOSITE A DVERSARIAL E XAMPLES ON MNIST
Figure 7 complements Figure 1 with adversarial examples against a natural model for MNIST dataset. [Figure] predictedlabel
[Figure]
D. P RELIMINARY R ESULTS FOR CIFAR 100 D ATASETS
Table 10 and Table 11 present performances of baseline models and C AT for CIFAR100 dataset.
Similar to previous results, with slightly reduction in clean accuracy, C AT outperforms MAX and AVG as well as simple PGD adversarial trained models. The MSD is excluded here temporally due to it takes extreme long time to train. We will ﬁll MSD’s result when it is available.
D. C OMPUTATIONAL E FFICIENCY OF C AT
We describe the computational cost of MAX, AVG, and C AT here.
MSD is excluded here due to it requires way more time, as the ofﬁcial implementation.
We consider ﬁlling its result with our implementation later.
All the experiments are with CIFAR10 dataset and of (cid:96) p settings.
The result is in Table 12.
[Table]
[Table]
[Table]