GAT: G ENERATIVE A DVERSARIAL T RAINING FOR A DVERSARIAL E XAMPLE D ETECTION AND R OBUST C LASSIFICATION
A BSTRACT
The vulnerabilities of deep neural networks against adversarial examples have become a significant concern for deploying these models in sensitive domains.
Devising a definitive defense against such attacks is proven to be challenging, and the methods relying on detecting adversarial samples are only valid when the attacker is oblivious to the detection mechanism.
In this paper we propose a principled adversarial example detection method that can withstand norm-constrained white-box attacks.
Inspired by one-versus-the-rest classification, in a K class classification problem, we train K base detectors where the i-th detector is trained to discriminate natural data of class i from adversarial examples perturbed from other classes.
At inference time, we first get the predicted label (say k) of the input, and then use the k-th detector to identify whether the input is a natural sample (of class k) or an adversarial example (perturbed from other classes). We further devise a generative approach to detecting/classifying adversarial examples by interpreting each base detector as an unnormalized density model of the classconditional data.
We provide comprehensive evaluation of the above adversarial example detection/classification methods, and demonstrate their competitive performances and compelling properties.
Code is available at https://github. com/xuwangyin/GAT-Generative-Adversarial-Training
I NTRODUCTION
Deep neural networks have become the staple of modern machine learning pipelines, achieving stateof-the-art performance on extremely difficult tasks in various applications such as computer vision (He et al., 2016),
speech recognition (Amodei et al., 2016), machine translation (Vaswani et al., 2017), robotics (Levine et al., 2016), and biomedical image analysis (Shen et al., 2017).
Despite their outstanding performance, these networks are shown to be vulnerable against various types of adversarial attacks, including evasion attacks (aka, inference or perturbation attacks) (Szegedy et al., 2013; Goodfellow et al., 2014; Carlini & Wagner, 2017b;
Su et al., 2019) and poisoning attacks (Liu et al., 2017; Shafahi et al., 2018).
These vulnerabilities in deep neural networks hinder their deployment in sensitive domains including, but not limited to, health care, finances, autonomous driving, and defense-related applications and have become a major security concern.
Due to the mentioned vulnerabilities, there has been a recent surge toward designing defense mechanisms against adversarial attacks (Gu & Rigazio, 2014; Jin et al., 2015; Papernot et al., 2016b;
Bastani et al., 2016; Madry et al., 2017; Sinha et al., 2018), which has in turn motivated the design of stronger attacks that defeat the proposed defenses (Goodfellow et al., 2014; Kurakin et al.,
2016b;a; Carlini & Wagner, 2017b; Xiao et al., 2018; Athalye et al., 2018; Chen et al., 2018;
He et al., 2018).
Besides, the proposed defenses have been shown to be limited and often not effective and easy to overcome (Athalye et al., 2018).
Alternatively, a large body of work has focused on detection of adversarial examples (Bhagoji et al., 2017; Feinman et al., 2017;
Gong et al., 2017;
Grosse et al., 2017; Metzen et al., 2017; Hendrycks & Gimpel, 2017; Li & Li, 2017; Xu et al., 2017;
Pang et al., 2018; Roth et al., 2019;
Bahat et al., 2019; Ma et al., 2018; Zheng & Hong, 2018; Tian et al., 2018).
While training robust classifiers focuses on maintaining performance in presence of adversarial examples, adversarial detection only cares for detecting such examples.
The majority of the current detection mechanisms focus on non-adaptive threats, for which the attacks are not specifically tuned/tailored to bypass the detection mechanism, and the attacker is oblivious to the detection mechanism.
In fact, Carlini & Wagner (2017a) and Athalye et al.
(2018) showed that the detection methods presented in (Bhagoji et al., 2017; Feinman et al., 2017;
Gong et al., 2017; Grosse et al., 2017; Metzen et al., 2017; Hendrycks & Gimpel, 2017; Li & Li, 2017;
Ma et al., 2018), are significantly less effective than their claims under adaptive attacks.
Overall, current solutions are mostly heuristic approaches that cannot provide performance guarantees to the adversary they considered.
[Figure]
In this paper we present a detection mechanism that can withstand adaptive attacks.
The idea is to partition the input space into subspaces based on the original classifier’s prediction and then perform adversarial/natural sample classification in these subspaces.
The detector (binary classifier) in each subspace is trained to distinguish inclass samples from adversarial examples perturbed from other classes. At inference time, we first use the original classifier to get an input sample’s predicted label ˆ k , and then use the ˆ k -th detector to identify whether the input is a natural sample (of class ˆ k ) or an adversarial example (perturbed from other classes). Figure 1 provides a schematic illustration of the proposed approach.
Our specific contributions are:
• We develop a principled adversarial example detection method that can withstand normconstrained white-box attacks.
Empirically, our best models improve previous state-ofthe-art mean L 2 distortion from 3.68 to 5.65 on MNIST dataset, and from 1.1 to 1.5 on CIFAR10 dataset.
• We study powerful and versatile generative classification models derived from our detection framework and demonstrate their competitive performances over discriminative robust classifiers.
While defense mechanisms based on ordinary adversarial training are vulnerable to rubbish examples, inputs that cause confident predictions of our models have humanunderstandable semantic meanings.
R ELATED WORKS
Adversarial attacks.
Since the pioneering work of Szegedy et al.
(2013), a large body of work has focused on designing algorithms that achieve successful attacks on neural networks (Goodfellow et al., 2014; Moosavi-Dezfooli et al., 2016;
Kurakin et al., 2016b; Chen et al., 2018; Papernot et al., 2016a; Carlini & Wagner, 2017b).
More recently, iterative projected gradient descent (PGD), has been empirically identified as the most effective approach for performing norm constrained attacks, and the attack reasonably approximates the optimal attack (Madry et al., 2017).
Adversarial detection techniques.
The majority of the methods developed for detecting adversarial attacks are based on the following core idea: given a trained K -class classifier, f : R d → { 1 ...K } , and its corresponding natural training samples, D = { x i ∈ R d } Ni =1 , generate a set of adversarially attacked samples D
′ =
{ x ′ j ∈ R d } Mj =1 , and devise a mechanism to discriminate D from D ′ .
For instance, Gong et al.
(2017) use this exact idea and learn a binary classifier to distinguish the natural and adversarially perturbed sets.
Similarly, Grosse et al.
(2017) append a new “attacked” class to the classifier, f , and re-train a secured network that classifies natural images, x ∈ D , into the K classes and all attacked images, x ′ ∈ D ′ , to the ( K + 1) -th class.
In contrast to Gong et al.
; Grosse et al.
(2017), which aim at detecting adversarial examples directly from the image content, Metzen et al.
(2017) trained a binary classifier that receives as input the intermediate layer features extracted from the classifier network f , and distinguished D from D ′ based on such input features.
More importantly, Metzen et al.
(2017) considered the so-called case of adaptive / dynamic adversary and proposed to harden the detector against such attacks using a similar adversarial training approach as in Goodfellow et al.
Unfortunately, the mentioned detection methods are significantly less effective under an adaptive adversary equipped with a strong attack (Carlini & Wagner, 2017a;
Athalye et al., 2018).
A DVERSARIAL D ETECTION M ETHODS 3. I NTEGRATED ADVERSARIAL DETECTION
For a K ( K ≥ 2 ) class classification problem, given a dataset of natural samples D = { x i } Ni =1 , x i ∈ R d , along with labels { y i } Ni =1 , y i ∈ { 1 ...K } , let f : R d → { 1 ...K } be a classifier that is used to do classification on D .
With the labels and predicted labels the dataset respectively forms the partition
D =
(cid:83) D k and D f = (cid:83) D fk , where D k = { x : y
=
k, x ∈ D} , and D fk
= { x : f ( x ) =
k, x ∈ D} .
Let H =
{ h k } Kk =1 , h k : R d → { 0 , 1 } be a set of binary classifiers (detectors), in which h k is trained to discriminate natural samples classified as k , from adversarial samples that fool to be classified as k .
Also, let D ′ be a set of L p norm bounded adversarial examples crafted from D :
D ′
= { x + δ : f ( x + δ ) ̸
= y, f ( x ) =
y, x ∈ D , δ ∈ S} , S = { δ ∈ R d | ∥ δ ∥ p ≤ ϵ } .
Consider the following procedure to determine whether a sample x in D ∪ D ′ is an adversarial example:
First obtain the estimated class label k
= f ( x ) , then use the k -th detector to predict: if h k ( x ) = 1 then categorize x as a natural sample, otherwise categorize it as an adversarial example.
The detection accuracy of the algorithm is given by
[Equation]
where D ′
fk =
{ x : f ( x ) =
k, x ∈ D ′ } .
Hence minimizing the algorithm’s classification error is equivalent to minimizing classification error of individual detectors.
Employing empirical risk minimization, detector k , parameterized by θ k , is trained by
[Equation]
where L is a loss function that measures the distance between h k ’s output and the supplied label (e.g., the binary cross-entropy loss).
In the case of adaptive attacks, when the adversary aims to fool both the classifier and detectors, the accuracy of a naively trained detector could be significantly reduced.
In order to be robust to adaptive attacks, by taking a similar approach as Madry et al.
(2017) we incorporate the attack into the training objective: min θ k ρ ( θ k ) , where ρ ( θ k ) =
E x
[Equation]
where D f \ k = { x : f ( x ) ̸
= k, y ̸
= k, x ∈ D} , and we assume that perturbation budget is large enough such that ∀ x ∈ D f \ k , ∃ δ ∈ S , s.t. f
( x + δ ) =
k .
Now by dropping the f ( x + δ )
=
k constrain we could derive an upper bound for the first loss term:
[Equation]
The detector could instead be trained by minimizing this upper bound using the following unconstrained objective,
[Equation]
[Equation]
Further, we use the fact that when D is used as the training set, f could overfit on D such that
D \ k = { x i : y i ̸
= k } and D k are respectively good approximations of D f \ k and D fk .
This leads to our proposed generative adversarial training (GAT) objective: min θ k ρ [Equation] k ) , where ρ
[Equation]
In a nutshell, each detector is trained using in-class natural samples and detector-adversarial examples crafted from out-class samples.
The inner maximization is solved using the PGD attack (Madry et al., 2017).
We use the binary cross-entropy loss as L and minimize it using gradient-based optimization methods.
3. A G ENERATIVE APPROACH TO ADVERSARIAL EXAMPLE DETECTION / CLASSIFICATION
Our experimental results suggest that a detector trained with objective 5 has a strong generative property: performing adversarial attacks to detector k causes visual features of class k to appear in the attacked data (in some cases, the attacked data become a valid instance of class k ).
Although a similar phenomenon is observed in standard adversarial training (Engstrom et al., 2019; Santurkar et al., 2019; Tsipras et al., 2018), the generative property of the detectors is much stronger than a softmax adversarially robust classier (see Figure 3, Figure 5, and Figure 6).
These results motivate us to interpret the base detector as an unnormalized density model (i.e., an energy-based model (LeCun et al., 2006)) of the class-conditional data. This interpretation allows us to obtain the joint probability of the input and a class category using the Gibbs distribution: p ( x, k ) =
exp( − E θk ( x ))
Z Θ , where E θ k
( x ) = − z
( h k ( x )) , and Z Θ = (cid:80) k (cid:82) exp( − E θ k
( x )) dx is an intractable normalizing constant known as the partition function.
We can then apply the Bayes classification rule to obtain a generative classifier : H ( x ) =
arg max k p ( x, k ) =
arg max k z ( h k ( x )) .
(Without a probabilistic interpretation, the above rule would equivalent to one-versus-the-rest classification.)
In addition, we can base on p ( x, k ) to reject low probability inputs.
We implement the reject option by thresholding ˆ k -th detector’s logit output, where ˆ k is the predicted class.
In the task of adversarial example detection, rejected samples are considered as adversarial examples.
E VALUATION METHODOLOGY 4. R OBUSTNESS TEST
We first test the robustness of individual detectors.
We show that, once a detector is trained with an adequately configured PGD attack, its performance cannot be significantly reduced by an adversary with much stronger configurations.
We note that although the PGD attack has been shown (Madry et al., 2017) to be able to induce robustness in ordinary adversarial training, the test is necessary as it is not clear whether the optimization landscape of the generative objective is the same as its discriminative counterpart.
For instance, we found that the step size used by Madry et al.
(2017) to train their CIFAR10 robust classifier would not induce robustness to our detectors (see Appendix C.2.2).
Furthermore, we use adversarial finetuning on CIFAR10 and ImageNet to speedup detector training.
With the robustness test, we show that PGD attack also introduces robustness within this new training paradigm.
We use AUC (area under the ROC Curve) to measure detection performances.
The metric could be interpreted as the probability that the detector assigns a higher score to a random positive sample than to a random negative example.
While true positive rate and false positive rate are the commonly used metrics for measuring detection performances, they require a detection threshold to be specified.
AUC, however, is an aggregated measurement of detection performance across a range of thresholds, and we found it to be a more stable and reliable metric.
For the k -th detector h k , its AUC is computed on the set { ( x, 0) : x ∈ D ′ f \ k } ∪ { ( x, 1) : x ∈ D fk } , where D ′ f \ k = { arg max x + δ L
( h k ( x + δ ; θ k ) , 0) : x ∈ D f \ k ) } (refer to loss 4).
4. D ETECTION P ERFORMANCE
Having validated the robustness of individual detectors, we evaluate the overall performance of our integrated detection system.
Recalling our detection rule, we first obtain the estimated class label k
= f ( x ) , then use the k -th detector’s logit output z ( h k ( x )) to predict: if z ( h k ( x )) ≥ T k , then x is a natural sample, otherwise it is an adversarially perturbed sample.
For the sake of this evaluation, we use a universal threshold for all the detectors: ∀ k ∈ { 1 ...K } T k = T , and report detection performance at a range of universal thresholds.
In practice, however, the optimal value of each detector’s detection threshold T k should be determined by optimizing a utility function.
We use D = { ( x i , y i ) } Ni =1 to denote the test set that contains natural samples, and D ′
=
{ ( x i + δ i , y i ) } Ni =1 to denote the corresponding perturbed test set.
For a given threshold T , we compute the true positive rate (TPR) on D and false positive rate (FPR) on D ′ .
These two metrics are respectively defined as
[Equation]
6) and
[Equation]
In the FPR definition we use f ( x ) ̸
= y to constrain that only true adversarial examples are counted as false positives.
This constraint is necessary, as we found that for the norm ball constraint we considered in the experiments, not all perturbed samples are adversarial examples that cause misclassification on f .
In order to craft the perturbed dataset D ′ , we consider three attacking scenarios.
Classifier attack.
This attack corresponds to the scenario where the adversary is oblivious to the detection mechanism.
For a given natural sample x and its label y , the perturbed sample x ′ is computed by minimizing the loss, [Equation]
[Equation]
where z ( f ( x ′ )) is the classifier’s logit outputs.
This objective is derived from the CW attack (Carlini & Wagner, 2017b) and used in MadryLab (b;a) to perform untargeted attacks.
Detectors attack.
In this scenario adversarial examples are produced by attacking only the detectors.
We construct a single detection function H by using the i -th detector’s logit output as its i -th logit output: z ( H ( x ))
i =
z ( h i ( x )) .
H is then recognized as a single network, and the perturbed sample x ′ for a given input ( x, y ) is computed by minimizing the loss
[Equation]
According to our detection rule, a low value of the detector’s logit output indicates detection of an adversarial example, thus by minimizing the negative of logit output we make the perturbed example harder to detect.
H could also be fed directly to the CW loss 8 or to the cross-entropy loss, but we found the attack based on the loss in 9 to be significantly more effective.
Combined attack.
With the goal of fooling both the classifier and detectors, perturbed samples are produced by attacking the integrated detection system.
We consider two loss functions for realizing the combined attack.
The first is based on the combined loss function (Carlini & Wagner, 2017a) that has been shown to be effective against existing detection methods.
Given a natural example x and its label y , same as the detectors-attack scenario, we first construct a single detection function
H by aggregating the logit outputs of individual detectors: z ( H ( x ))
i =
z ( h i ( x )) .
We then use the aggregated detector’s largest logit output max k ̸ =
y z ( H ( x ))
k (low value of this quantity indicates detection of an adversarial example) and the classifier logit outputs z ( f ( x )) to construct a surrogate classifier g , with its logit outputs being
[Equation]
A perturbed example x ′ is then computed by minimizing the loss function
[Equation]
In practice we observe that the optimization of this loss tends to stuck at the point where max i ̸
= y z ( f ( x ′ ))
i keeps changing signs while max j ̸ = y z ( H ( x ))
j stays as a large negative number (which indicates detection).
To derive a more effective attack we consider a simple combination of loss 8 and loss 9:
[Equation]
The objective is straightforward: if x ′ is not yet an adversarial example on f , optimize it for that goal; otherwise optimize it for fooling the aggregated detector.
We mention briefly here that we perform the same performance analysis of the generative detection method as detailed in Section 3.2, by computing TPR on D and FPR on D ′ , where D ′ is computed using loss 9, loss 8 and the cross-entropy loss.
4. R OBUST C LASSIFICATION P ERFORMANCE
Integrated classification.
In addition to the generative classifier proposed in Section 3.2, we introduce another classification scheme that comes with a reject option.
The scheme is based on a combination of the naive classifier f and the detectors: for a given input x and its prediction label k
= f ( x ) , if z ( h k ( x )) <
T , then x is rejected, otherwise it’s classified as k .
We respectively use loss 12 and 9 to attack the integrated classifier and the generative classifier.
Performance metric.
In the context of robust classification, the performance of a robust classifier is measured using standard accuracy and robust accuracy — accuracies respectively computed on the natural dataset and perturbed dataset.
On the natural dataset D = { ( x i , y i ) }
Ni =1 , we compute the accuracy as the fraction of samples that are correctly classified ( f ( x ) =
y ) and at the same time not rejected ( z ( h k ( x )) ≥
[Equation] [Equation]
On the perturbed dataset D ′ = { ( x i + δ i , y i ) }
Ni =
we compute the error as the fraction of samples that are misclassified ( f ( x ) ̸ = y ) and at the same time not rejected:
[Equation]
Note that in this case the error is no longer a complement of the accuracy.
For a classification system with a reject option, any perturbed samples that are rejected should be considered as properly handled, regardless of whether they are misclassified.
Thus on the perturbed dataset, the error, which is the fraction of misclassified and not rejected samples, is a more proper notion of such system’s performance.
For a standard softmax robust classifier that comes without an reject option, its perturbed set error is computed as the complement of its accuracy on the perturbed set.
E XPERIMENTS 5. MNIST
Using different p -norm and maximum perturbation ϵ constrains we trained four detection systems (each has 10 base detectors), with training and validation adversarial examples optimized using PGD attacks of different steps and step sizes (see Table 6).
At each step of PGD attack we use the Adam optimizer to perform gradient descent, both for L 2 and L ∞ constrained scenarios.
Appendix A. provides more training details.
Robustness results.
The robustness test results in Table 1 confirm that the base detectors trained with objective 5 are able to withstand much stronger PGD attacks, for both L 2 and L ∞ scenarios.
In Table 8 we present the robustness test result using normalized steepest descent based attack.
We also performed other types of tests, including cross-norm test, cross-perturbation test, and random restart test.
The results of these tests, along with the complete robustness test results on L ∞ ϵ = 0 .and
L ∞ ϵ = 0 .
5 trained detectors, are presented in Appendix C.1.
[Table]
[Table]
Detection results.
Figure 2a shows that generative detection outperforms integrated detection (compare their performances under their respective most effective attacks).
Combined attack, as expected, is the most effective attack against integrated detection.
Attack based on loss 9 is the most effective attack against generative detection (see Figure 7).
Notably, the red curve that overlaps the y-axis indicates that integrated detection is able to perfectly detect adversarial examples crafted by attacking only the classifier (using objective 8)
[Table] In Table 2 we list the performances of the generative detection method along with the state-of-the-art detection method as identified by Carlini & Wagner (2017a).
Our method, apart from being able to provide performance guarantee, outperforms the state-of-the-art method by large margins.
(Please refer to Appendix B for a description of how we compute the mean L 2 distortions.)
Classification results.
In Figure 2b we plot the robust classification performance of our classification methods and a state-of-the-art discriminative robust classifier (Madry et al., 2017).
Our methods provide reject options that allow the user to find a balance between standard accuracy and robust error by adjusting the rejection threshold.
We observe that a stronger attack ( ϵ = 0 . 4 ) breaks the robust classifier (as indicated by the right red cross), while the generative classifier still exhibits robustness, even though both systems are trained with L ∞ ϵ = 0 .
3 constrain
[Figure]
Figure 3 shows perturbed samples produced by performing targeted attacks against the generative classifier and robust classifier.
The generative classifier’s perturbation samples have distinguishable visible features of the target class, indicating that the base detectors, from which the generative classifier is built, have learned the class conditional distributions, and the perturbations have to change the semantics for a successful attack.
In contrast, perturbations introduced by attacking the robust classifier are not interpretable, even though they could cause high logit output of the target classes (see Figure 8 for the logit outputs distribution).
Following this path and with a larger perturbation limit, it is straightforward to generate completely unrecognizable inputs that fool the discriminative robust classifier.
[Figure]
5. CIFAR
On CIFAR10 we train the base detectors using L ∞ ϵ = 8 constrain PGD attack of steps 40 and step size 0.5.
Note that the scale of ϵ and step size here is 0-255 (rather than 0-1 as in the case of
MNIST).
The robust classifier (Madry et al., 2017) that we compare with is trained with the same
L ∞ ϵ = 8 constraint but with a different step size (see Appendix C.2.2 for a discussion of the effects of step size).
Appendix A.2 provides the
Robustness results.
Table 3 shows that the base detector models can withstand attacks that are significantly stronger than the training attack.
In Appendix C.2.1 we report random restart test results, cross-norm and cross-perturbation test results, and robustness test result for L 2 based models.
[Table]
Detection results.
Consistent with the MNIST results, in Figure 4a combined attack is the most effective method against integrated detection.
Generative detection outperforms integrated detection when the detection threshold is low (i [Table] e., when TPR is high).
In this figure we use loss 9 to attack generative detection, and in Figure 9 we show that this loss is more effective than cross-entropy loss and CW loss.
In Table 4 our method outperforms the state-of-the-art adversarial detection method [Table] [Table] [Figure]
Classification results.
Contrary to MNIST’s result [Figure] we did not observe a dramatic decrease in the robust classifier’s performance when we increase the perturbation limit to ϵ = 12 (Figure 4b).
Integrated classification can reach the standard accuracy of a regular classifier [Figure] but at the cost of significantly increased error on the perturbed set.
Figure 5 shows some perturbed samples produced by attacking the generative classifier and robust classifier.
While these two classifiers have similar errors on the perturbed set, samples produced by attacking the generative classifier have more visible features of the targets, suggesting that the adversary has to change more semantic to cause the same error.
Figures 6 and 10 demonstrate that hard to recognize images are able to cause high logit outputs of the robust classifier.
Such examples highlight a major defect of the defense mechanisms based on ordinary adversarial training: they could be easily fooled by unrecognizable inputs (Nguyen et al., 2015; Goodfellow et al., 2014; Schott et al., 2018).
In contrast, samples that cause high logit outputs of the generative classifier all have clear semantic meaning.
In Figure 13 we present image synthesis results using L ∞ ϵ = 16 constrained detectors.
In Appendix D we provide Gaussian noise attack results and a discussion about the interpretability of the generative classification approach.
[Figure]
5. I MAGE N ET
On ImageNet we show GAT induces detection robustness and supports the learning of class conditional distributions.
Our experiment is based on Restricted ImageNet (Tsipras et al., 2018), a subset of ImageNet that has its reorganized into customized categories.
dog consists of images of different breeds collected from ImageNet class from 151 to 268.
We trained a dog class detector by finetuning a pre-trained ResNet50 (He et al., 2016) model.
The dog category covers a range of ImageNet classes, with each one having its logit output.
We use the subnetwork defined by the logit output of class 151 as the detector (in principle logit output of other classes in the range should also work).
Due to computational resource constraints, we only validated the robustness of a L ∞ ϵ = 0 .
02 trained detector (trained with PGD attack of steps 40 and step size 0.001), and we present the result in Table 5.
(On Restricted ImageNet in the case of L ∞ scenario Tsipras et al.
(2018) only demonstrates the robustness of a ϵ = 0 .
005 constrained model).
Please refer to
Appendix C.3 for more results on adversarial example generation and image synthesis.
[Table]
C ONCLUSION
We studied the problem of adversarial example detection under the robust optimization framework and proposed a novel detection method based on input space partitioning.
Our formulation leads to a new generative modeling technique which we called generative adversarial training (GAT).
GAT’s capability to learn class conditional distributions further gives rise to generative detection/classification approaches that show competitive performance and improved interpretability.
In particular, our generative classification method is more resistant to “rubbish examples”, a major threat to even the most successful defense mechanisms.
High computational cost (see Appendix E for more discussion) is a major drawback of our methods, and in the future, we will explore the idea of shared computation between detectors.
A T RAINING D ETAILS A. MNIST TRAINING
We use 50K samples from the original training set for training and the rest 10K samples for validation and report test performances based on the epoch-saved checkpoint that gives the best validation performance.
All base detectors are trained using a network consisting of two max-pooled convolutional layers each with 32 and 64 filters and a fully connected layer of size 1024: same as the one used in Madry et al.
At each iteration we sample a batch of 320 samples from which in-class samples are used as positive samples and out-of-class samples are used for crafting adversarial examples that will be used as negative samples.
To balance positive and negative examples at each batch, we resample the out-of-class set to have same number of samples as in-class set.
All base detectors are trained for 100 epochs
[Table]
A. CIFAR10 TRAINING
We train CIFAR10 base detectors using a ResNet model (same as the one used by Madry et al.
(2017); MadryLab (a)).
To speedup training, we take advantage of a natural trained classifier: the subnetwork of f that defines the output logit z
( f ( · ))
k is essentially a “detector”, that would output high values for samples of class k , and low values for others.
The base detector is then trained by finetuning the subnetwork using objective 5.
The pretrained classifier has a test accuracy of 95.01% (fetched from MadryLab (a)).
At each iteration of training we sample a batch of 300 samples, from which in-class samples are used as positive samples, while an equal number of out-of-class samples are used for crafting adversarial examples.
Adversarial examples for training L 2 and L ∞ models are both optimized using normalized steepest descent based PGD attacks (MadryLab, b).
We report results based on the best performances on the CIFAR10 test set (thus don’t claim generalization performance of the proposed method).
B C OMPUTING MEAN L 2 DISTANCE
We first find the detection threshold T with which the detection system has 0.95 TPR.
We construct a new loss function by adding a weighted loss term that measures perturbation size to objective
[Equation]
We then use unconstrained PGD attack to optimize L ( x ′ ) .
We use binary search to find the optimal c , where in each bsearch attempt if x ′ is a false positive ( max i z
( H ( x ′ ))
i ̸ = y and max i ̸ =
y z ( H ( x ′ ))
i > T ) we consider the current c as effective and continue with a larger c .
The configurations for performing binary search and PGD attack are detailed in Table 7.
The c upper bound is established such that with this upper bound, no samples except those that are inherently misclassified by the generative classifier, could be perturbed as a false positive.
With these settings, our MNIST L ∞ ϵ = 0 . 3 and L ∞ ϵ = 0 .
5 generative detection models respective reached 1.0 FPR and 0.9455 FPR, and CIFAR10 generative model reached 0.9995 FPR.
Due to the technical and computational difficulties of finding the optimal c in loss 15 using binary search, performances based on mean L 2 distortion are not precise, and we encourage future work to measure detection performances based on norm constrained attacks.
[Table]
C M ORE EXPERIMENTAL RESULTS C. M ORE MNIST RESULTS
[Table]
[Table] [Table]
[Table] [Table]
[Figure]
u [Figure] p
[Figure]
s i t i v e r a t
e
o n
[Figure]
I S
[Figure]
s
[Figure] [Figure]
C. M ORE CIFAR10 RESULTS C.2. M ORE ROBUSTNESS TEST RESULTS
[Table]
[Table]
[Table] [Table]
= 8 . 0 trained base detectors under L ∞ ϵ = 8 . 0 constrained PGD attack of steps 40 and step size 0.5. Base detector k
[Table]
k
[Table]
AUC
[Table]
[Figure] [Figure]
C.2. T RAINING STEP SIZE AND ROBUSTNESS
We found training with adversarial examples optimized with a sufficiently small step size to be essential for detection robustness.
In table 18 we tested two L ∞ ϵ = 2 . 0 base detectors respectively trained with 0.5 and 1.0 step size.
The step size 1.0 model is not robust when tested with a much smaller step size.
We observe that when training the step size 1.0 model, training set adv AUC reached 1.0 in less than one hundred iterations, but test set natural AUC plummeted to around 0.
and couldn’t recover thereafter.
(Please refer to Figure 11 for the definitions of adv AUC and nat
AUC.)
This suggests that naturally occurring data, and perturbed data produced using a large step size, which are essentially noise images, live in two easily separable spaces, and training on such data is not beneficial for model performance on natural data or adversarial examples.
[Table]
E FFECTS OF PERTURBATION LIMIT
To study the effects of perturbation limit on generative adversarial training, we analysis the training dynamics of one L ∞ ϵ = 2 . 0 constrained and one L ∞ ϵ = 8 . 0 constrained base detector.
In Figure 11 we show the training and testing history of these two models.
The ϵ = 2 . 0 model history shows that by adversarial finetuning the model reaches robustness in just a few thousands of iterations, and the performance on natural samples is preserved (test natural AUC begins at 0.9971, and ends at 0.9981).
Adversarial finetuning on the ϵ = 8 . 0 model didn’t converge after an extended 20K iterations of training.
The gap between train adv AUC and test adv AUC of the ϵ = 8 . 0 model is more pronounced, and we observed a decrease of test natural AUC from 0.9971 to 0.9909.
These results suggest that training with larger perturbation limit is more time and resource consuming, and could lead to performance decrease on natural samples.
The benefit is that the detector is pushed to a better approximation of the target data distribution.
As an illustration, in Figure 12, perturbations generated by attacking the naturally trained classifier (corresponds to 0 perturbation limit) don’t have clear semantics, while perturbed samples of the L ∞ ϵ = 8 . 0 model are completely recognizable.
[Figure]
O r i g i na l
N a t u r a l t r a
i ned c
l a ss i f
i e r de t e c t
o r de t e c t o r de t e c t o r
C. M ORE I MAGE N ET RESULTS
[Figure] [Figure]
D G AUSSIAN NOISE ATTACK AND MODEL INTERPRETABILITY
In this section we use Gaussian noise attack experiment to motivate a comparative analysis of the interpretabilities of our generative classification approach and discriminative robust classification approach (Madry et al., 2017).
We first discuss how these two approaches determine the posterior class probabilities.
For the discriminative classifier, the posterior probabilities are computed from the logit outputs of the classifier using the softmax function p ( k | x ) =
[Equation]
x )) j ) .
For the generative classifier, the posterior probabilities are computed in two steps: in the first, we train the base detectors, which is the process of solving the inference problem of determining the joint probability p ( x, k ) , and in the second, we use Bayes rule to compute the posterior probability p ( k | x )
= p ( x,
[Equation]
x
[Equation]
cidentally, the formulas for computing the posterior probabilities take the same form.
But in our approach, the exponential of the logit output of a detector (i.e., exp( z ( h k ( x ))) ) has a clear probabilistic interpretation: it’s the unnormalized joint probability of the input and the corresponding class category.
We use Gaussian noise attack to demonstrate that this probabilistic interpretation is consistent with visual perception.
We start from a Gaussian noise image, and gradually perturb it to cause higher and higher logit outputs.
This is implemented by targeted PGD attack against logit outputs of these two classification models.
The resulting images in Figure 18 show that, in our model, the logit output increase direction, i.e. the join probability increase direction, indicates the class semantic changing direction; while for the discriminative robust model, the perturbed image computed by increasing logit outputs are not as clearly interpretable.
In particular, the perturbed images that cause high logit outputs of the robust classifiers are not recognizable.
In summary, as a generative classification approach that explicitly models class conditional distributions, our system offers a probabilistic view of the decision making process of the classification problem; adversarial attacks that rely on imperceptible or uninterpretable noises are not effective against such a system.
E C OMPUTATIONAL COST ISSUE
In this section we provide an analysis of the computational cost of our generative classification approach.
In terms of memory requirements, if we assume the softmax classifier (i.e., the discriminative robust classifier) and the detectors use the same architecture (i.e., only defer in the final layer)
then the detector based generative classifier is approximately K times more expensive than the Kclass softmax classifier.
This also means that the computational graph of the generative classifier is K times larger than the softmax classifier.
Indeed, in the CIFAR10 task, on our Quadro M 24GB GPU (TensorFlow 1.13.1), the inference speed of the generative classifier is roughly ten times slower than the softmax classifier.
We next benchmark the training speed of these two types of classifiers.
The generative classifier has K logit outputs, with each one defined by the logit output of a detector.
Same with the softmax classifier, except that the K outputs share the parameters in the convolutional base.
Now consider ordinary adversarial training on the softmax classifier and generative adversarial training on the generative classifier.
To train the softmax classifier, we use batches of N samples.
For the generative classifier, we train each detector with batches of 2 × M samples ( M positive samples and M negative samples).
At each iteration, we need to respectively compute N and M × K adversarial examples for these two classifiers.
Now we test the speed of the following two scenarios:
1) compute the gradient w.r.t.
to N samples on a single computational graph, and 2) compute the gradient w.r.t to M × K samples on K computational graphs, with each graph working on M samples.
We assume that in scenario 2 all the computational graphs are loaded to GPUs, and thus their computations are in parallel.
In our CIFAR10 experiment, we used batches consisting of 30 positive samples and 30 negative samples to train each ResNet50 based detectors.
In Madry et al.
(2017), the softmax classifier was trained with batches of 128 samples.
In this case, K = 10 , M = 30 , and N = 128 .
On our GPU, scenario 1 took 683 ms ± 6.76 ms per loop, while scenario 2 took 1.85 s ± 42.7 ms per loop.
In this case, we could expect generative adversarial training to be about 2.7 times slower than ordinary adversarial training, if not considering parameter gradient computation.
In practice, large batch size is almost always preferred.
And our method won’t compare as favorably if we choose to use one.
F D ENSITY ESTIMATION ON SYNTHETIC DATASETS
While ordinary discriminative training only learns a good decision boundary, GAT is able to learn the underlying density functions that generate the training data.
Results on 1D (Figure 19) and 2D benchmark datasets (Figure 20) show that through properly configured generative adversarial training, detectors’ output recover target density functions.
[Figure]