GAT: G ENERATIVE A DVERSARIAL T RAINING FOR A DVERSARIAL E XAMPLE D ETECTION AND R OBUST C LASSIFICATION
A BSTRACT
The vulnerabilities of deep neural networks against adversarial examples have become a significant concern for deploying these models in sensitive domains.
Devising a definitive defense against such attacks is proven to be challenging, and the methods relying on detecting adversarial samples are only valid when the attacker is oblivious to the detection mechanism.
In this paper we propose a principled adversarial example detection method that can withstand normconstrained white-box attacks.
Inspired by one-versus-the-rest classification, in a K class classification problem, we train K binary classifiers where the i-th binary classifier is used to distinguish between clean data of class i and adversarially perturbed samples of other classes.
At test time, we first use a trained classifier to get the predicted label (say k) of the input, and then use the k-th binary classifier to determine whether the input is a clean sample (of class k) or an adversarially perturbed example (of other classes).
We further devise a generative approach to detecting/classifying adversarial examples by interpreting each binary classifier as an unnormalized density model of the class-conditional data.
We provide comprehensive evaluation of the above adversarial example detection/classification methods, and demonstrate their competitive performances and compelling properties.
Code is available at https://github.com/ xuwangyin/GAT-Generative-Adversarial-Training 1 .
I NTRODUCTION
Deep neural networks have become the staple of modern machine learning pipelines, achieving stateof-the-art performance on extremely difficult tasks in various applications such as computer vision (He et al., 2016),
speech recognition (Amodei et al., 2016), machine translation (Vaswani et al., 2017), robotics (Levine et al., 2016), and biomedical image analysis (Shen et al., 2017).
Despite their outstanding performance, these networks are shown to be vulnerable against various types of adversarial attacks, including evasion attacks (aka, inference or perturbation attacks) (Szegedy et al., 2013; Goodfellow et al., 2014; Carlini & Wagner, 2017b;
Su et al., 2019) and poisoning attacks (Liu et al., 2017; Shafahi et al., 2018).
These vulnerabilities in deep neural networks hinder their deployment in sensitive domains including, but not limited to, health care, finances, autonomous driving, and defense-related applications and have become a major security concern.
Due to the mentioned vulnerabilities, there has been a recent surge toward designing defense mechanisms against adversarial attacks (Gu & Rigazio, 2014; Jin et al., 2015; Papernot et al., 2016b;
Bastani et al., 2016; Madry et al., 2017; Sinha et al., 2018), which has in turn motivated the design of stronger attacks that defeat the proposed defenses (Goodfellow et al., 2014; Kurakin et al., 2016b;a; Carlini & Wagner, 2017b; Xiao et al., 2018; Athalye et al., 2018; Chen et al., 2018;
He et al., 2018).
Besides, the proposed defenses have been shown to be limited and often not effective and easy to overcome (Athalye et al., 2018).
Alternatively, a large body of work has focused on detection of adversarial examples (Bhagoji et al., 2017; Feinman et al., 2017;
Gong et al., 2017;
Grosse et al., 2017; Metzen et al., 2017; Hendrycks & Gimpel, 2017; Li & Li, 2017; Xu et al., 2017;
Pang et al., 2018; Roth et al., 2019;
Bahat et al., 2019; Ma et al., 2018; Zheng & Hong, 2018; Tian et al., 2018).
While training robust classifiers focuses on maintaining performance in presence of adversarial examples, adversarial detection only cares for detecting such examples.
The majority of the current detection mechanisms focus on non-adaptive threats, for which the attacks are not specifically tuned/tailored to bypass the detection mechanism, and the attacker is oblivious to the detection mechanism.
In fact, Carlini & Wagner (2017a) and Athalye et al.
(2018) showed that the detection methods presented in (Bhagoji et al., 2017; Feinman et al., 2017;
Gong et al., 2017; Grosse et al., 2017; Metzen et al., 2017; Hendrycks & Gimpel, 2017; Li & Li, 2017;
Ma et al., 2018), are significantly less effective than their claims under adaptive attacks.
Overall, current solutions are mostly heuristic approaches that cannot provide performance guarantees.
[Figure]
of the proposed adversarial example detection mechanism.
In this paper we propose a detection mechanism that can withstand adaptive attacks.
The idea is to partition the input space into subspaces based on the original classifier’s decision boundary, and then perform clean/adversarial example classification the subspaces.
The binary classifier in each subspace is trained to distinguish in-class samples from adversarially perturbed samples of other classes.
At inference time, we first use the original classifier to get an input sample’s predicted label ˆ k , and then use the ˆ k -th binary classifier to identify whether the input is a clean sample (of class ˆ k ) or an adversarially perturbed sample (of other classes).
Fig.
1 provides a schematic illustration of the proposed approach.
Our specific contributions are: (1) We develop a principled adversarial example detection method that can withstand adaptive attacks.
Empirically, our best models improve previous state-of-the-art mean L 2 distortion from 3.68 to 5.65 on MNIST dataset, and from 1.1 to 1.5 on CIFAR10 dataset.
(2) We study powerful and versatile generative classification models derived from our detection framework and demonstrate their competitive performances over discriminative robust classifiers.
While discriminative robust classifiers are vulnerable to rubbish examples, inputs that have confident predictions under our models have interpretable features.
R ELATED WORKS
Adversarial attacks Since the pioneering work of Szegedy et al.
(2013), a large body of work has focused on designing algorithms that achieve successful attacks on neural networks (Goodfellow et al., 2014; Moosavi-Dezfooli et al., 2016;
Kurakin et al., 2016b; Chen et al., 2018; Papernot et al., 2016a; Carlini & Wagner, 2017b).
More recently, iterative projected gradient descent (PGD), has been empirically identified as the most effective approach for performing norm constrained attacks, and the attack reasonably approximates the optimal attack (Madry et al., 2017).
Adversarial example detection
The majority of the methods developed for detecting adversarial attacks are based on the following core idea: given a trained K -class classifier, f : R d → { 1 ...K } , and its corresponding clean training samples,
D = { x i ∈ R d } Ni =1 , generate a set of adversarially attacked samples D
′ =
{ x ′ j ∈ R d } Mj =1 , and devise a mechanism to discriminate D from D ′ .
For instance, Gong et al.
(2017) use this exact idea and learn a binary classifier to distinguish the clean and adversarially perturbed sets.
Similarly, Grosse et al.
(2017) append a new “attacked” class to the classifier, f , and re-train a secured network that classifies clean images, x ∈ D , into the K classes and all attacked images, x ′ ∈ D ′ , to the ( K + 1) -th class.
In contrast to Gong et al.
; Grosse et al.
(2017), which aim at detecting adversarial examples directly from the image content, Metzen et al.
(2017) trained a binary classifier that receives as input the intermediate layer features extracted from the classifier network f , and distinguished D from D ′ based on such input features.
More importantly, Metzen et al.
(2017) considered the so-called case of adaptive / dynamic adversary and proposed to harden the detector against such attacks using a similar adversarial training approach as in Goodfellow et al.
Unfortunately, the mentioned detection methods are significantly less effective under an adaptive adversary equipped with a strong attack (Carlini & Wagner, 2017a;
Athalye et al., 2018).
P ROPOSED A PPROACH TO D ETECTING A DVERSARIAL E XAMPLES
The proposed approach to detecting adversarial examples is based on the following simple idea.
Assume there is an input sample x , and it is predicted as ˆ k by the classifier f , then x is either a true sample of class ˆ k
(assuming no misclassification) or an adversarially perturbed sample of other classes.
To determine which is the case we can use a binary classifier that is specifically trained to distinguish between clean samples of class ˆ k and adversarially perturbed samples of other classes.
Because ˆ k can be any one of the K class, we need to train a total of K binary classifier in order to have a complete solution.
Fig.
2 provides a schematic illustration of the above detection idea.
We next provide a mathematical justification and more details about how to train the detection model.
In a K ( K ≥ 2 ) class classification problem, given a dataset of clean samples D = { x i } Ni =1 , x i ∈ R d , along with labels { y i } Ni =1 , y i ∈ { 1 , ..., K } , let f : R d → { 1 , ..., K } be a classifier on D , and D ′ be a set of p -norm bounded adversarial examples computed from D : D ′
= { x + δ : f ( x + δ ) ̸
= y, f ( x ) =
y, x ∈ D , δ ∈ S} , S = { δ ∈ R d | ∥ δ ∥ p ≤ ϵ } .
We use D fk
= { x : f ( x ) =
k, x ∈ D} and D ′
fk =
{ x : f ( x ) =
k, x ∈ D ′ } to respectively denote the clean samples and adversarial examples which are predicted by f as class k .
Let H =
{ d k } K k =1 , where d k : R d → [0 , 1] is a binary classifier trained to distinguish between samples from D fk (assigned as class 1) and samples from D ′ fk (assigned as class 0).
We use d k ( x ) to model p ( x ∈ D fk | x ) and predict x ∈ D fk when d k ( x ) > 12 and x ∈ D
′ fk when d k ( x ) ≤ 12 .
Consider the following procedure to determine whether a sample x is an adversarial example (i.e., whether it comes from D or D ′ ):
First obtain the estimated class label ˆ k = f ( x ) , then use the ˆ k -th binary classifier to predict: if d ˆ k ( x ) >
= 12 then categorize x as a clean sample, otherwise categorize it as an adversarial example.
This algorithm can be viewed as a binary classifier, and its accuracy is given by
[Equation]
Crucially, because the errors of individual binary classifiers are independent, maximizing Eq.
(1) is equivalent to optimizing the performances of individual binary classifiers.
d k solves the binary classification problem of distinguishing between samples from D fk and samples from D ′ fk , and therefore can be trained with a binary classification objective:
[Equation]
where L is a loss function that measures the discrepancy between d k ’s output and the supplied label (e.g., the negative log likelihood loss).
In order to harden d k against adaptive attacks, we follow Madry et al.
and incorporate the adversary into the training objective: min
[Equation]
where
[Equation]
where D f \ k = { x : f ( x ) ̸
= k, y ̸
= k, x ∈ D} , and we assume that ϵ is large enough such that ∀ x ∈ D f \ k , ∃ δ ∈ S , s.t. f
( x + δ ) =
k .
The equality constraint f ( x + δ )
=
k in Eq.
(3) complicates the inner maximization.
We observe that by dropping this constrain we have the following upper bound of the first loss term:
[Equation]
Because we are minimizing L ( d k ( x + δ ; θ k ) , 0) , we can instead minimizing this upper bound, which gives us the unconstrained objective
[Equation]
We can further simply this objective by using the fact that when D is used as the training set, f can overfit on D such that D \ k
= { x i : y i ̸
= k } and D k are respectively good approximations of D f \ k and D fk :
[Equation]
In words, each binary classifier is trained using clean samples of a particular class and adversarial examples (with respect to d k ) created from samples of other class.
The inner maximization is solved using the PGD attack (Madry et al., 2017).
We use the negative log likelihood loss as L and minimize it using gradient-based optimization methods.
From a classification point of view, we can reformulate the above detection algorithm as a classifier that has a rejection option: given input x and its prediction label ˆ k = f
( x ) , if d ˆ k ( x ) < T , then x is rejected, otherwise it’s classified as ˆ k .
We will refer to this classification model as an integrated classifier .
3. A G ENERATIVE APPROACH TO ADVERSARIAL EXAMPLE DETECTION / CLASSIFICATION
The proposed approach makes use of a trained classifier f to get the predicted label, but f is not strictly necessary: we can use H = { d k } Kk =1 in place of f to do classification.
We can interpret H as an one-versus-the-rest (OVR) classifier.
In a K class classification problem, a
OVR classifier consists of K binary classifiers, with each one trained to solve a two-class problem of separating samples in a particular class from samples not in that class.
H differs from a traditional
OVR classifier in that d k is trained to distinguish between samples in class k and adversarially perturbed samples of other classes, but because the loss on adversarial inputs is an upper bound of the loss on clean samples, the binary classifier should also be able to separate samples of class k from clean samples of other classes.
When H is viewed as an OVR classifier, the classification rule is
[Equation]
We can also interpret H as a generative classifier .
Our experiments show that d k has a strong generative property: performing adversarial attacks on d k causes visual features of class k to appear in the attacked data (in some cases, the attacked data become a valid instance of class k ).
Although a similar phenomenon is observed in standard adversarial training (Tsipras et al., 2018; Engstrom et al., 2019; Santurkar et al., 2019), the generative property of our model seems to be much stronger than that of a softmax adversarially robust classier (Fig. 4, Fig. 6, and Fig. 7).
These results motivate us to reinterpret d k as an unnormalized density model (i.e., an energy-based model (LeCun et al., 2006)) of the class- k data.
This interpretation allows us to obtain the class-conditional probability of an input by:
[Equation]
where E k ( x ) =
− z d k ( x ) , with z d k being the logit output of d k , and
[Equation]
is an intractable normalizing constant known as the partition function.
We can then apply the Bayes classification rule to obtain a generative classifier :
[Equation]
where we have assumed all partition functions Z k , k = 1 , ..., K and class priors [Equation] k [Equation]
[Equation] , K to be equal. Because we explicitly model p ( x, k ) , we can use this quantity to reject low probability inputs which can be any samples that do not belong to class k .
In this work we focus on the scenario where low probability inputs are adversarially perturbed samples of other classes and the rejected samples are considered as adversarial examples.
Because d k ( x ) is computed by applying the logistic sigmoid function to z d k ( x ) , and the logistic sigmoid function is a monotonically increasing function of its argument, the generative classifier (Eq.
(9)) is equivalent to the OVR classifier (Eq.
In the following sections, we will use integrated detection to refer to the original detection approach where we make use an extra classifier f , and generative detection to refer to this alternative approach where we first use the generative classifier Eq.
(9) to get the predicted label ˆ k of an input x , and then use d ˆ k to determine whether x is adversarial input.
E VALUATION METHODOLOGY 4. R OBUSTNESS TEST
We first validate the robustness of individual binary classifiers by following the standard methodology for robustness testing: we train the binary classifier with PGD attack configured with a particular combination of step-size and number of steps, and then test the binary classifier’s performance under
PGD attacks configured with different combinations of step-sizes and number of steps.
We use AUC (area under the ROC Curve) as the detection performance metric.
AUC is an aggregated measurement of detection performance across a range of thresholds, and can be interpreted as the probability that the binary classifier assigns a higher score to a random positive sample than to a random negative example.
For a given d k , the AUC is computed on the set { ( x, 0) : x ∈ D ′ f \ k } ∪ { ( x, 1) : x ∈ D fk } , where D ′ f \ k = { x + arg max δ L ( d k ( x + δ ; θ k ) , 0) : x ∈ D f \ k ) } .
4. A DVERSARIAL E XAMPLE D ETECTION P ERFORMANCE
Having validated the robustness of individual binary classifier, we evaluate the overall performance of the proposed approach to detecting adversarial examples.
According to the detection algorithm, we first obtain the predicted label ˆ k = f ( x ) , and then use the ˆ k -th binary classifier’s logit output to predict: if z d ˆ k ( x ) ≥ T , then x is a clean sample, otherwise it is an adversarially perturbed sample.
We use D = { ( x i , y i ) } N i =1 to denote the test set that contains clean samples, and D ′
=
{ ( x i + δ i , y i ) } Ni =1 to denote the corresponding perturbed test set.
For a given T , we compute the true positive rate (TPR) on D and false positive rate (FPR) on D ′
(here, clean samples are in the positive class).
These two metrics are respectively defined as
[Equation]
and
[Equation]
We observe that for the norm ball constraint we considered in the experiments, not all perturbed samples can cause misclassification on f , so we use f ( x ) ̸
= y in the FPR definition
to constrain that only adversarial inputs that actually cause misclassification can be counted as false positives.
Given a clean sample x and its groundtruth label y , we consider three approaches to creating the corresponding adversarial example x ′ .
Here we will focus on untargeted attacks.
Classifier attack This attack corresponds to the scenario where the adversary is oblivious to the detection mechanism.
Inspired by the CW attack (Carlini & Wagner, 2017b), the adversarial example x ′ is computed by minimizing,
[Equation]
where z f ( x ′ ) is the classifier’s logit outputs.
Detector attack In this scenario adversarial examples are produced by attacking only the detector.
We first construct a detection function H by aggregating the logit outputs of individual binary classifiers:
[Equation]
The adversarial example x ′ is then computed by minimizing
[Equation]
According to our detection rule, a low value of a binary classifier’s logit output indicates the detection of an adversarial example, and therefore by minimizing the negative of the logit output we make the adversarial input harder to detect.
H can also be used with the CW loss Eq.
(12) or the cross-entropy loss, but we find the attack based on Eq.
(14) to be most effective.
Combined attack The combined attack is an adaptive method that considers both the classifier and the detector.
We consider two loss functions for the combined attack.
The first is based on the adaptive attack of Carlini & Wagner (2017a) which has been shown to be effective against existing detection methods.
We first construct a new detection function H with Eq.
(13) and then use H ’s largest logit output max k ̸ =
y z H ( x ) k
(low value of this quantity indicates detection of an adversarial example) and the classifier logit outputs z f ( x ) to construct a new classifier g :
[Equation]
The adversarial example x ′ is then computed by minimizing the loss function
[Equation]
In practice we observe that the optimization of Eq.
(16) tends to stuck at the point where max i ̸
= y
z f ( x ′ )
i keeps changing signs while max j ̸ = y
z H ( x ) j staying as a large negative number (which indicates detection).
In light of the above issues we derive a more effective attack by combining Eq.
(12) and Eq.
[Equation]
In words, if x ′ is not yet an adversarial example on f (case 1), optimize it for that goal, otherwise optimize it for evading the detection (case 2).
We note that the above three attacks are for the original detection approach (i.e., integrated detection).
The generative detection approach (Section 3.1) does not make use of f and we use Eq.
(14) to create adversarial examples for generative detection.
4. R OBUST C LASSIFICATION P ERFORMANCE
In robust classification, the accuracy of a softmax robust classifier is evaluated on the original test test (the standard accuracy ) and the adversarially perturbed test set (the robust accuracy ).
Because the generative classifier comes with the reject option, we use slightly different metrics.
On the clean test dataset D = { ( x i , y i ) } Ni =1 , we define the standard accuracy as the fraction of samples that are correctly classified ( f ( x ) =
y ) and at the same time not rejected ( z d ˆ k ( x ) ≥ T ):
[Equation]
In the adversarially perturbed test dataset D ′ = { ( x i + δ ∗ i , y i ) } Ni [-=
-]=1 , we will consider a data sample as properly handled when it is rejected ( z d ˆ k ( x ) < T ), regardless of whether it causes misclassification.
In this way, only misclassified ( f ( x ) ̸ = y ) and unrejected ( z d ˆ k ( x ) ≥ T ) samples are counted as errors:
[Equation]
To compare different classifiers under the same metrics, we compute the error of a softmax robust classifier g on D ′ as
[Equation] We respectively use Eq. (17) and Eq.
(14) to compute the D ′ for the integrated classifier and the generative classifier.
E XPERIMENTS 5. MNIST
We train four detection models (each consists of ten binary classifiers) by using different combinations of p -norm and perturbation limit ϵ (Table 5).
The adversarial examples used for training and validation are computed using PGD attacks of different steps and step sizes (Table 5).
At each step of PGD attack we use the Adam optimizer to perform gradient descent, both for L 2 -based and
L ∞ -based attacks.
Appendix A.1 provides more training details.
Robustness results Table 1 and Table 7 show that d 0 and d 1 are able to withstand PGD attacks configured with different steps and step-size, for both L 2 -based and L ∞ -based attacks.
The binary classifiers also exhibit robustness when the attack uses p -norm or perturbation limit that are different from those used for training the model (Table 8).
The models are also robust when the attacks use multiple random restarts (Table 9). [Table]
[Table]
Detection results Fig.
3a shows the performances of integrated detection and generative detection under different attacks.
Combined attack with Eq.
(17) is the most effective attack against integrated detection, and is much more effective than the combined attack with Eq.
Overall, generative detection outperforms integrated detection when they are evaluated under their respective most effective attack.
It is also interesting to note that when the adversarial examples are created by attacking only the classifier (Eq.
(12)), integrated detection is able to perfectly detect these adversarial examples (see the red curve that overlaps the y-axis).
Given that generative detection is the most effective approach among the proposed approaches, we compare it with state-of-the-art detection methods (Carlini & Wagner, 2017a).
Table 2 shows that generative detection outperforms the state-of-the-art method by large margins.
Appendix B provides details about how the mean L 2 distortions are computed.
Classification results Figure 3b shows the standard and robust classification performances of the proposed classifiers and a state-of-the-art softmax robust classifier (Madry et al., 2017).
Our models provide the reject option that allows the user to find a balance between standard accuracy and robust error by adjusting the rejection threshold.
We observe that a stronger attack ( ϵ = 0 . 4 ) breaks the softmax robust classifier (as indicated by the right red cross), while the generative classifier still exhibits robustness, even though both models are trained with the L ∞ ϵ = 0 .
3 constraint.
[Figure]
Figure 4 shows perturbed samples produced by performing targeted attacks against the generative classifier and softmax robust classifier.
The generative classifier’s perturbation samples have distinguishable visible features of the target class, indicating that individual binary classifiers have learned the class conditional distributions, and the perturbations have to change the semantics for a successful attack.
In contrast, perturbations introduced by attacking the softmax robust classifier are not interpretable, even though they can cause high logit output of the target classes (see Figure 9 for the logit outputs distribution).
5. CIFAR
On CIFAR10 we train a single detection model that consists of ten binary classifiers using L ∞ ϵ
=constrain PGD attack of steps 40 and step size 0.5 (note that the scale of ϵ and step size here is 0-255, as opposed to 0-1 as in the case of MNIST).
The softmax robust classifier (Madry et al., 2017) that we compare with is also trained with L ∞ ϵ = 8 constraint but with a different step size (Appendix C.2.
provides a discussion on the effects of step size).
Appendix A.2 provides the training details.
Robustness results Table 3 shows that d 1 and d 2 can withstand PGD attacks configured with different steps and step-size.
In Appendix C.2.1 we report random restart test results, cross-norm and cross-perturbation test results, and robustness test result for L 2 based models.
Detection results Consistent with the MNIST result, Fig.
5 shows that combined attack with
Eq.
(17) is the most effective attack against integrated detection, and generative detection similarly outperforms integrated detection.
Table 4 shows that generative detection outperforms the state-of-the-art adversarial detection method.
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Classification results Contrary to MNIST’s result, we did not observe a dramatic decrease in the softmax robust classifier’s performance when we increase the perturbation limit to ϵ = 12 (Fig. 5b).
Integrated classification can reach the standard accuracy of a regular classifier, but at the cost of significantly increased error on the perturbed set.
Fig.
6 shows some perturbed samples produced by attacking the generative classifier and robust classifier.
While these two classifiers have similar errors on the perturbed set, samples produced by attacking the generative classifier have more visible features of the attacked classes, suggesting that the adversary needs to change more semantic to cause the same error.
Fig.
7 and Fig.
11 demonstrate that unrecognizable images are able to cause high logit outputs of the softmax robust classifier.
This phenomenon highlights a major defect of the softmax robust classifier: they can be easily fooled by unrecognizable inputs (Nguyen et al., 2015; Goodfellow et al., 2014;
Schott et al., 2018).
In contrast, samples that cause high logit outputs of the generative classifier all have clear semantic meaning.
In Figure 14 we present image synthesis results using
L ∞ ϵ = 16 constrained detectors.
In Appendix D we provide Gaussian noise attack results and a discussion about the interpretability of the generative classification approach.
[Figure] [Figure]
C ONCLUSION
We studied the problem of adversarial example detection under the robust optimization framework and proposed a novel detection method that can withstand adaptive attacks.
Our formulation leads to a new generative modeling technique which we called generative adversarial training (GAT).
GAT’s capability to learn class conditional distributions further gives rise to generative detection/classification approaches that show competitive performance and improved interpretability.
A T RAINING D ETAILS A. MNIST TRAINING
We use 50K samples from the original training set for training and the remaining 10K samples for validation and report the test performance based on the checkpoint which has the best validation performance.
All binary classifiers are trained for 100 epochs, where in each iteration we sample 32 in-class samples as the positive samples and 32 out-class samples to create adversarial examples which will be used as negative samples.
All binary classifier models use a neural network consisting of two convolutional layers each with 32 and 64 filters, and a fully connected layer of size 1024; more details of the network architecture can be found in Madry et al.
[Table]
A. CIFAR10 TRAINING
We train CIFAR10 binary classifiers using a ResNet model (same as the one used by Madry et al.
(2017); MadryLab (a)).
To speedup training, we take advantage of a clean trained classifier: the subnetwork of f that defines the output logit z f ( · ) k is essentially a “binary classifier” that would output high values for samples of class k , and low values for others.
The binary classifier is then trained by finetuning the subnetwork using objective 5.
The pretrained classifier has a test accuracy of 95.01% (fetched from MadryLab (a)).
At each iteration of training we sample a batch of 300 samples, from which in-class samples are used as positive samples, while an equal number of out-of-class samples are used for crafting adversarial examples.
Adversarial examples for training L 2 and L ∞ models are both optimized using normalized steepest descent based PGD attacks (MadryLab, b).
We report results based on the best performances on the CIFAR10 test set (thus don’t claim generalization performance of the proposed method).
B C OMPUTING MEAN L 2 DISTANCE
We first find the detection threshold T with which the detection system has 0.95 TPR.
We construct a new loss function by adding a weighted loss term that measures perturbation size to objective
[Equation]
We then use unconstrained PGD attack to optimize L ( x ′ ) .
We use binary search to find the optimal c , where in each bsearch attempt if x ′ is a false positive ( max i z H ( x ′ )
i ̸
= y and max i ̸
= y z H ( x ′ )
i > T ) we consider the current c as effective and continue with a larger c .
The configurations for performing binary search and PGD attack are detailed in Table 6.
The c upper bound is established such that with this upper bound, no samples except those that are inherently misclassified by the generative classifier, could be perturbed as a false positive.
With these settings, our MNIST L ∞ ϵ = 0 .
3 and L ∞ ϵ = 0 .
5 generative detection models respective reached 1.0 FPR and 0.9455 FPR, and CIFAR generative model reached 0.9995 FPR.
[Table]
C M ORE EXPERIMENTAL RESULTS C. M ORE MNIST RESULTS
[Table]
[Table] [Table]
[Table]
dddddddd
[Figure]
i
[Figure]
[Figure]
C. M ORE CIFAR10 RESULTS C.2. M ORE ROBUSTNESS TEST RESULTS
[Table]
[Table] [Table]
[Table] [Table]
[Figure]
i
[Figure] [Figure]
C.2. T RAINING STEP SIZE AND ROBUSTNESS
We found training with adversarial examples optimized with a sufficiently small step size to be critical for model robustness.
In table 17 we tested two L ∞ ϵ = 2 .
binary classifiers respectively trained with 0.5 and 1.0 step size.
The step size 1.0 model is not robust when tested with a much smaller step size.
We observe that when training the step size 1.0 model, training set adv AUC reached 1.0 in less than one hundred iterations, but test set clean AUC plummeted to around 0.
and couldn’t recover thereafter.
(Please refer to Figure 12 for the definitions of adv AUC and nat
AUC.)
[Table]
E FFECTS OF PERTURBATION LIMIT
To study the effects of perturbation limit, we analyze the training dynamics of one L ∞ ϵ = 2 .constrained and one L ∞ ϵ = 8 .
constrained binary classifiers.
In Figure 12 we show the training and testing history of these two models.
The ϵ = 2 . 0 model history shows that by adversarial finetuning the model reaches robustness in just a few thousands of iterations, and the performance on clean samples is preserved (test clean AUC begins at 0.9971, and ends at 0.9981).
Adversarial finetuning on the ϵ = 8 . 0 model didn’t converge after an extended 20K iterations of training.
The gap between train adv AUC and test adv AUC of the ϵ = 8 . 0 model is more pronounced, and we observed a decrease of test clean AUC from 0.9971 to 0.9909.
These results suggest that training with larger perturbation limit is more time and resource consuming, and could lead to performance decrease on clean samples.
The benefit is that the detector is pushed to a better approximation of the target data distribution.
As an illustration, in Figure 13, perturbations generated by attacking the naturally trained classifier (corresponds to 0 perturbation limit) don’t have clear semantics, while perturbed samples of the L ∞ ϵ = 8 . 0 model are completely recognizable.
[Figure]
O r i g i na l
N a t u r a l t r a
i ned c
l a ss i f
i e r de t e c t
o r de t e c t o r de t e c t o r
C. I MAGE N ET RESULTS
On ImageNet we show GAT induces detection robustness and supports the learning of class conditional distributions.
Our experiment is based on Restricted ImageNet (Tsipras et al., 2018), a subset of ImageNet that has its samples reorganized into customized categories.
The dog category consists of images of different breeds collected from ImageNet class from 151 to 268.
We trained a dog class detector by finetuning a pre-trained ResNet50 (He et al., 2016) model.
The dog category covers a range of ImageNet classes, with each one having its logit output.
We use the subnetwork defined by the logit output of class 151 as the detector (in principle logit output of other classes in the range should also work).
Due to computational resource constraints, we only validated the robustness of a L ∞ ϵ = 0 .
02 trained detector (trained with PGD attack of steps 40 and step size 0.001), and we present the result in Table 18.
(On Restricted ImageNet in the case of L ∞ scenario Tsipras et al.
(2018) only demonstrates the robustness of a ϵ = 0 .
005 constrained model).
Please refer to
Appendix C.3 for more results on adversarial example generation and image synthesis.
[Table]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
Note than large perturbation models didn’t reach robustness.
Please refer to Santurkar et al.
for the detail about how the class conditional Gaussian is estimated.
[Figure]
D G AUSSIAN NOISE ATTACK AND MODEL INTERPRETABILITY
In this section we use Gaussian noise attack experiment to motivate a comparative analysis of the interpretabilities of our generative classification approach and discriminative robust classification approach (Madry et al., 2017).
We first discuss how these two approaches determine the posterior class probabilities.
For the discriminative classifier, the posterior probabilities are computed from the logit outputs of the classifier using the softmax function p ( k | x ) =
[Equation]
( z f ( x ) j ) .
For the generative classifier, the posterior probabilities are computed in two steps: in the first, we train the base detectors, which is the process of solving the inference problem of determining the joint probability p ( x, k ) , and in the second, we use Bayes rule to compute the posterior probability p ( k | x [Equation] =
[Equation]
Coincidentally, the formulas for computing the posterior probabilities take the same form.
But in our approach, the exponential of the logit output of a detector (i.e., exp( z d k ( x )) ) has a clear probabilistic interpretation: it’s the unnormalized joint probability of the input and the corresponding class category.
We use Gaussian noise attack to demonstrate that this probabilistic interpretation is consistent with visual perception.
We start from a Gaussian noise image, and gradually perturb it to cause higher and higher logit outputs.
This is implemented by targeted PGD attack against logit outputs of these two classification models.
The resulting images in Figure 19 show that, in our model, the logit output increase direction, i.e. the join probability increase direction, indicates the class semantic changing direction; while for the discriminative robust model, the perturbed image computed by increasing logit outputs are not as clearly interpretable.
In particular, the perturbed images that cause high logit outputs of the softmax robust classifiers are not recognizable.
In summary, as a generative classification approach that explicitly models class conditional distributions, our system offers a probabilistic view of the decision making process of the classification problem; adversarial attacks that rely on imperceptible or uninterpretable noises are not effective against such a system.
E C OMPUTATIONAL COST ISSUE
In this section we provide an analysis of the computational cost of our generative classification approach.
In terms of memory requirements, if we assume the softmax classifier (i.e., the discriminative softmax robust classifier) and the detectors use the same architecture (i.e., only defer in the final layer)
then the detector based generative classifier is approximately K times more expensive than the K -class softmax classifier.
This also means that the computational graph of the generative classifier is K times larger than the softmax classifier.
Indeed, in the CIFAR10 task, on our Quadro
M6000 24GB GPU (TensorFlow 1.13.1), the inference speed of the generative classifier is roughly ten times slower than the softmax classifier.
We next benchmark the training speed of these two types of classifiers.
The generative classifier has K logit outputs, with each one defined by the logit output of a detector.
Same with the softmax classifier, except that the K outputs share the parameters in the convolutional base.
Now consider ordinary adversarial training on the softmax classifier and generative adversarial training on the generative classifier.
To train the softmax classifier, we use batches of N samples.
For the generative classifier, we train each detector with batches of 2 × M samples ( M positive samples and M negative samples).
At each iteration, we need to respectively compute N and M × K adversarial examples for these two classifiers.
Now we test the speed of the following two scenarios:
1) compute the gradient w.r.t.
to N samples on a single computational graph, and 2) compute the gradient w.r.t to M × K samples on K computational graphs, with each graph working on M samples.
We assume that in scenario 2 all the computational graphs are loaded to GPUs, and thus their computations are in parallel.
In our CIFAR10 experiment, we used batches consisting of 30 positive samples and 30 negative samples to train each ResNet50 binary classifiers.
In Madry et al.
(2017), the softmax classifier was trained with batches of 128 samples.
In this case, K = 10 , M = 30 , and N = 128 .
On our GPU, scenario 1 took 683 ms ± 6.76 ms per loop, while scenario 2 took 1.85 s ± 42.7 ms per loop.
In this case, we could expect generative adversarial training to be about 2.7 times slower than ordinary adversarial training, if not considering parameter gradient computation.
In practice, large batch size is almost always preferred.
And our method won’t compare as favorably if we choose to use one.
F D ENSITY ESTIMATION ON SYNTHETIC DATASETS
While ordinary discriminative training only learns a good decision boundary, GAT is able to learn the underlying density functions that generate the training data.
Results on 1D (Figure 20) and 2D benchmark datasets (Figure 21) show that through properly configured generative adversarial training, detectors’ output recover target density functions.
[Figure]