E VALUATING THE R OBUSTNESS OF N EURAL N ET -
WORKS
: A
N
E
XTREME
V
ALUE T HEORY A PPROACH
Anonymous authors
Paper under double-blind review
A BSTRACT
The robustness of neural networks to adversarial examples has received great attention due to security implications.
Despite various attack approaches to crafting visually imperceptible adversarial examples, little has been developed towards a comprehensive measure of robustness.
In this paper, we provide theoretical justiﬁcation for converting robustness analysis into a local Lipschitz constant estimation problem and propose to use the Extreme Value Theory for efﬁcient evaluation.
Our analysis yields a novel robustness metric called CLEVER, which is short for
C ross L ipschitz E xtreme V alue for n E twork R obustness.
The proposed CLEVER score is attack-agnostic and is computationally feasible for large neural networks.
Experimental results on various networks, including ResNet, Inception-v3 and
MobileNet, show that (i) CLEVER is aligned with the robustness indication measured by the (cid:96) 2 and (cid:96) ∞ norms of adversarial examples from powerful attacks, and (ii) defended networks using defensive distillation or bounded ReLU indeed give better CLEVER scores.
To the best of our knowledge, CLEVER is the ﬁrst attack-independent robustness metric that can be applied to any neural network classiﬁer.
I NTRODUCTION
Recent studies have highlighted the lack of robustness in state-of-the-art neural network models, e.g., a visually imperceptible adversarial image can be easily crafted to mislead a well-trained network (Szegedy et al., 2013; Goodfellow et al., 2015).
Even worse, researchers have identiﬁed that these adversarial examples are not only valid in the digital space but also plausible in the physical world (Kurakin et al., 2016a; Evtimov et al., 2017).
The vulnerability to adversarial examples call into question safety-critical applications and services deployed by neural networks, including autonomous driving systems and malware detection protocols, among others.
In the literature, studying adversarial examples of neural networks has twofold purposes: (i) security implications : devising effective attack algorithms for crafting adversarial examples; and (ii) robustness analysis : evaluating the intrinsic model robustness to adversarial perturbations to normal examples.
Although in principle the means of tackling these two problems are expected to be independent, that is, the evaluation of a neural network’s intrinsic robustness should be agnostic to attack methods, and vice versa, existing approaches extensively use different attack results as a measure of robustness of a target neural network.
Speciﬁcally, given a set of normal examples, the attack success rate and distortion of the corresponding adversarial examples crafted from a particular attack algorithm are treated as robustness metrics.
Consequently, the network robustness is entangled with the attack algorithms used for evaluation and the analysis is limited by the attack capabilities.
More importantly, the dependency between robustness evaluation and attack approaches can cause biased analysis.
For example, adversarial training is a commonly used technique for improving the robustness of a neural network; it can be accomplished by generating adversarial examples and retraining the network parameters with corrected labels.
However, while such an adversarially trained network is made robust to attacks used to craft adversarial examples for training, it can still be vulnerable to unseen attacks.
Motivated by the evaluation criterion for assessing the quality of text and image generation that is completely independent of the underlying generative processes, such as the BLEU score for texts (Papineni et al., 2002) and the INCEPTION score for images (Salimans et al., 2016), we aim to propose a comprehensive and attack-agnostic robustness metric for neural networks.
Stemming from a perturbation analysis of an arbitrary neural network classiﬁer, we derive an universal lower bound on the minimal distortion required to craft an adversarial example from an original one, where the lower bound applies to any attack algorithm and any (cid:96) p norm for p ≥ 1 .
We show that this lower bound associates with the maximum norm of the local gradients with respect to the original example, and therefore robustness evaluation becomes an estimation problem of the local
Lipschitz constant.
To efﬁciently and reliably estimate the local Lipschitz constant, we propose to use extreme value theory (De Haan & Ferreira, 2007) for robustness evaluation.
In this context, the extreme value corresponds to the local Lipschitz constant of our interest, which can be inferred by a set of independently and identically sampled local gradients.
With the aid of extreme value theory, we propose a robustness metric called CLEVER, which is short for C ross L ipschitz E xtreme V alue for n E twork R obustness.
We note that CLEVER is an attack-independent robustness metric that applies to any neural network classiﬁers.
In contrast, the robustness metric proposed in (Hein &
Andriushchenko, 2017), albeit attack-agnostic, only applies to a neural network classiﬁer with one hidden layer.
We highlight the main contributions of this paper as follows:
• We propose a novel robustness metric called CLEVER, which is short for C ross L ipschitz E xtreme V alue for n E twork R obustness.
To the best of our knowledge, CLEVER is the ﬁrst robustness metric that is attack-independent and can be applied to any arbitrary neural network classiﬁer.
• The proposed CLEVER score is well supported by our theoretical analysis on formal robustness guarantees and the use of extreme value theory.
Our robustness analysis extends the results in (Hein & Andriushchenko, 2017) from continuously differentiable functions to functions with ﬁnite non-differentiable points.
• We corroborate the effectiveness of CLEVER by conducting experiments on state-of-the-
art models for ImageNet, including ResNet (He et al., 2016), Inception-v3 (Szegedy et al., 2016) and MobileNet (Howard et al., 2017).
We also use CLEVER to investigate defended networks against adversarial examples, including the use of defensive distillation (Papernot et al., 2016) and bounded ReLU (Zantedeschi et al., 2017).
Experimental results show that our CLEVER score well aligns with the attack-speciﬁc robustness indicated by the (cid:96) 2 and (cid:96)
∞ distortions of adversarial examples.
The remainder of the paper is organized as follows.
Section 2 provides the background and related work on adversarial attack, defense, and theoretical robustness guarantees.
In Section 3, we present a formal robustness analysis for classiﬁcation models.
In Section 4, we propose a novel way to accurately estimate cross Lipschitz constants and introduce the CLEVER score.
We present empirical results on various networks in Section 5, and summarize the paper in Section 6.
B ACKGROUND AND R ELATED WORK 2. A TTACKING N EURAL N ETWORKS USING A DVERSARIAL E XAMPLES
One of the most popular formulations found in literature for crafting adversarial examples to mislead a neural network is to formulate it as a minimization problem, where the variable δ ∈ R d to be optimized refers to the perturbation to the original example, and the objective function takes into account unsuccessful adversarial perturbations as well as a speciﬁc norm on δ for assuring similarity.
For instance, the success of adversarial examples can be evaluated by their cross-entropy loss (Szegedy et al., 2013; Goodfellow et al., 2015) or model prediction (Carlini & Wagner, 2017b).
The norm constraint on δ can be implemented in a clipping manner (Kurakin et al., 2016b) or treated as a penalty function (Carlini & Wagner, 2017b).
The (cid:96) p norm of δ , deﬁned as (cid:107) δ (cid:107) p =
( (cid:80) di =1 | δ i | p ) 1 /p for any p ≥ 1 , is often used for crafting adversarial examples.
In particular, when p = ∞ , (cid:107) δ (cid:107) ∞ = max i ∈{ 1 ,...,d } | δ i | measures the maximal variation among all dimensions in δ .
When p = 2 , (cid:107) δ (cid:107) 2 becomes the Euclidean norm of δ .
When p = 1 , (cid:107) δ (cid:107)
= (cid:80) pi =
1 | δ i | measures the total variation of δ .
The state-of-the-art attack methods for (cid:96) ∞ , (cid:96) 2 and (cid:96) 1 norms are the iterative fast gradient sign method (I-FGSM)
(Goodfellow et al., 2015; Kurakin et al., 2016b), Carlini and
Wagner’s attack (CW attack) (Carlini & Wagner, 2017b), and elastic-net attacks to deep neural networks (EAD) (Chen et al., 2017a), respectively.
These attacks fall into the category of white-box attacks since the network model is assumed to be transparent to an attacker.
Adversarial examples can also be crafted from a black-box network model using an ensemble approach (Liu et al., 2016), training a substitute model (Papernot et al., 2017), or employing zeroth-order optimization based attacks (Chen et al., 2017b).
2. E XISTING D EFENSE M ETHODS
Since the discovery of vulnerability to adversarial examples (Szegedy et al., 2013), various defense methods have been proposed to improve the robustness of neural networks.
The rationale for defense is to make a neural network more resilient to adversarial perturbations, while ensuring the resulting defended model still attains similar test accuracy as the original undefended network.
Papernot et al.
proposed defensive distillation (Papernot et al., 2016), which uses the distillation technique (Hinton et al., 2015) and a modiﬁed softmax function at the ﬁnal layer to retrain the parameters of a neural network with the prediction probabilities (i.e., soft labels) from the original network.
Zantedeschi et al.
showed that by changing the ReLU function to a bounded ReLU function, a neural network can be made more resilient.
Another popular defense approach is adversarial training , which generates and augments adversarial examples with the original training data during the network training stage.
On MNIST, the adversarially trained model proposed by Madry et al.
(Madry et al., 2017) can successfully defend a majority of adversarial examples at the price of increased network capacity.
In addition to network modiﬁcation and adversarial training, detection methods such as feature squeezing (Xu et al., 2017) can also be used to identify adversarial examples.
However, the CW attack is shown to be able to bypass 10 different detection methods (Carlini & Wagner, 2017a).
In this paper, we focus on evaluating the intrinsic robustness of a neural network model to adversarial examples.
The effect of detection methods is beyond our scope.
T HEORETICAL R OBUSTNESS G UARANTEES FOR N EURAL N ETWORKS
(Szegedy et al., 2013) compute global Lipschitz constant for each layer and use their product to explain the robustness issue in neural networks.
However, using global Lipschitz constant can be impractical when the resulting bound on the distortion is loose.
(Hein & Andriushchenko, 2017) gave a robustness lower bound using a local Lipschitz continuous condition and derived a closedform bound for a multi-layer perceptron (MLP) with a single hidden layer and softplus activation.
However, the analysis cannot be extended to a neural network with more than one hidden layer.
(Wang et al., 2016) utilized terminologies from topology to study robustness.
However, no robustness bounds or estimates were provided for neural networks.
On the other hand, works done by (Ehlers, 2017; Katz et al., 2017a;b; Huang et al., 2017) focus on formally verifying the viability of certain properties in neural networks for any possible input, and transform this formal veriﬁcation problem into satisﬁability modulo theory (SMT) and integer linear programming (ILP) problems.
However, this veriﬁcation approach comes with high computational complexity and is only plausible for very small networks.
F ORMAL R OBUSTNESS A NALYSIS FOR A C LASSIFIER
In this section, we formally deﬁne the notion of adversarial examples, minimum (cid:96) p distortions, and lower/upper bounds.
Under a very mild assumption on Lipschitz continuity of the classiﬁer function, we obtain formal robustness guarantees against adversarial perturbations, i.e. the lower bound of minimum (cid:96) p distortion.
For quick reference, the important notations introduced are summarized in
Table 1.
Deﬁnition 3.1 (perturbed example and adversarial example) .
Let x 0 ∈ R
d be an input vector of a K -class classiﬁer function f : R d → R K and the prediction is given as f ( x 0 )
=
argmax 1 ≤ i ≤ K f i ( x 0 ) .
Given x 0 , we say x a is a perturbed example of x 0 with noise δ ∈ R d and (cid:96) p -distortion ∆ p if x a = x 0 + δ and ∆ p = (cid:107) δ (cid:107) p .
An adversarial example is a perturbed example where we can ﬁnd some x a or δ that can attack the classiﬁer successfully.
For un-targeted attack, a successful attack is to ﬁnd a x a such that f ( x a ) (cid:54) =
f
( x 0 ) .
For targeted attack, a target class t ( t (cid:54) =
f
( x 0 ) ) is provided and a successful attack satisﬁes f ( x a ) =
t .
[Table]
Deﬁnition 3.2 (minimum distortion of x 0 ) .
Given an input vector x 0 of a classiﬁer function f , the minimum (cid:96) p distortion of x 0 , denoted as ∆ p, min , is deﬁned as the smallest ∆ p of its adversarial examples.
Deﬁnition 3.3 (lower bound of minimum distortion, β L ≤ ∆ p, min ) .
Suppose ∆ p, min is the minimum distortion of x 0 .
A lower bound of ∆ p, min , denoted by β L , is deﬁned such that any perturbed example of x 0 with (cid:107) δ (cid:107) p ≤ β L cannot be successful.
Deﬁnition 3.4 (upper bound of minimum distortion, β U ≥ ∆ p, min ) .
Suppose ∆ p, min is the minimum distortion of x 0 .
An upper bound of ∆ p, min , denoted by β U , is deﬁned such that there exists an adversarial example of x 0 with (cid:107) δ (cid:107) p ≥ β U .
The lower and upper bounds are instance speciﬁc because they depend on the input x 0 .
Ideally, we would like the bounds to be as close to the minimum distortion of x 0 as possible, meaning that β L is as large as possible and β U is as small as possible, as they reﬂect the robustness of a classiﬁer.
Below we show how to derive a formal robustness guarantee of a classiﬁer with Lipschitz continuity assumption.
Speciﬁcally, our analysis obtains a lower bound of (cid:96) p minimum distortion β L = min j (cid:54) =
c f c ( x 0 ) − f j ( x 0 ) L q .
Lemma 3.1 (Lipschitz continuity (Paulavi ˇ cius & ˇZilinskas, 2006)) .
Let S ⊂ R d be a convex bounded closed set and let h ( x ) :
S → R be a continuously differentiable function on an open set containing S .
Then, h ( x ) is a Lipschitz function with Lipschitz constant L q if the following inequality holds for any x , y ∈ S :
[Equation]
where L q = max {(cid:107)∇ h ( x ) (cid:107) q : x ∈ S } , ∇ h ( x ) = ( ∂h∂x 1 , · · · , ∂h∂x d ) (cid:62) is the gradient of h ( x ) , and 1 p + 1 q = 1 , 1 ≤ p, q ≤ ∞ .
Given Lemma 3.1, we then provide a formal guarantee to the lower bound β L .
Theorem 3.2 (Formal guarantee on lower bound β L ) .
Let x 0 ∈ R d and f : R d → R K be a multiclass classiﬁer with continuously differentiable components
f i and let c
= argmax 1 ≤ i ≤ K f i ( x 0 ) be the class which f predicts for x 0 .
For all δ ∈ R d with
[Equation]
c = argmax 1 ≤ i ≤ K f i ( x 0 + δ ) holds with 1 p + 1 q = 1 , 1 ≤ p, q ≤ ∞ and L q is Lipschitz constant for the function g :
=
f c − f j in (cid:96) p norm.
In other words, β L = min j (cid:54) =
c f c ( x 0 ) − f j ( x 0 )
L q is a lower bound of minimum distortion.
Intuitions behind Theorem 3.2 is shown in Figure 1, as a one-dimensional example.
The function value g ( x ) near point x 0 is inside the double-cone formed by two lines with slopes equal to ±
L q , where L q is the (local) Lipschitz constant of g ( x ) .
When g ( x ) is decreased to 0, an adversarial example is found.
The minimal change δ to decrease g ( x ) to 0, in the worst case where g ( x ) follows the boundary of the double-cone, is g ( x 0 )
L q .
The complete proof is deferred to Appendix A.
We make a few remarks about the theorem.
Remark 1.
Because L q is the Lipschitz constant of the function involving cross terms:
f c ( x ) − f j ( x ) , we also call it cross Lipschitz constant following (Hein & Andriushchenko, 2017).
Remark 2.
An upper bound β U on the minimum distortion ∆ p, min can be easily obtained by any successfully attack, since ∆ p, min is guaranteed to be less than the norm of an adversarial example.
Corollary 3.2.1.
1 Let L q,x 0 be local Lipschitz constant of function g :=
f c − f j at x 0 over some ﬁxed ball B p ( x 0 , R ) :
= { x ∈ R d | (cid:107) x − x 0 (cid:107) p ≤ R } and let δ ∈ B p ( 0 , R ) .
By Theorem 3.2, we get the bound in (Hein & Andriushchenko, 2017):
[Equation]
Remark 3.
The analysis in (Hein & Andriushchenko, 2017) implicitly assumes Lipschitz continuity on f i because they require f i to be continuously differentiable.
Alternatively, here we provide a simple alternative derivation without using Mean Value Theorem and Holder’s Inequality.
To cover non-differentiable functions (a typical property of neural networks owing to ReLU), we consider the following extension.
Lemma 3.3.
2 Let S ⊂
R
d be a convex bounded closed set and let h ( x ) :
S → R be an absolute continuous function and the derivative exists at all but a ﬁnite number of points (denoted as Z ).
Then equation (1) holds with L q = sup x ∈ S \ Z {(cid:107)∇ f ( x ) (cid:107) q } , and we can obtain the same conclusion as Theorem 3.2 and Corollary 3.2.1.
E STIMATING CROSS L IPSCHITZ CONSTANT VIA E XTREME V ALUE T HEORY
In Theorem 3.2, we show that the lower bound of minimum distortion of x 0 is associated with two terms, g ( x 0 ) , where g ( x ) =
f c ( x ) − f j ( x ) , and the cross Lipschitz constant L q,x 0 , which is deﬁned as max x ∈ B p ( x 0 ,R ) (cid:107)∇ g ( x ) (cid:107) q .
Note that g ( x 0 ) is readily available at the output of a classiﬁer.
Although ∇ g ( x ) can be easily computed via back propagation, the calculation of cross
Lipschitz constant is more involved as it requires computing the maximum value of (cid:107)∇ g ( x ) (cid:107) q in a ball.
If we can exhaustively list all the x ∈
B p ( x 0 , R ) , then we can get the exact value max x ∈ B p ( x 0 ,R ) (cid:107)∇ g ( x ) (cid:107) q .
However, it is impossible to do this exhaustive search because we have large dimension d in image classiﬁers, where d = 784 for MNIST, d = 3072 for CIFAR, and d = 150528 for ImageNet.
Instead of exhaustive search, one intuitive approach is to perform sampling on x and take the maximum value of (cid:107)∇ g ( x ( i ) )
(cid:107) q , where x ( i ) are the samples we generated.
The problem with this approach is that we might need a signiﬁcant amount of samples to obtain a good estimate of max (cid:107)∇ g ( x ) (cid:107) q
and we don’t know how good our estimate is compared to the true value.
Fortunately, Extreme Value Theory tells us that the maximum value of random variables can only follow one of the three extreme value distributions, which is useful for us to estimate max (cid:107)∇ g ( x ) (cid:107) q with only a tractable number of samples.
1 proof to Appendix Bproof deferred to Appendix C
Below we ﬁrst give some intuitions on how (cid:107)∇ g ( x ) (cid:107) q can be regarded as a random variable via our sampling approach, and then derive the cumulative density function for a simple one-hidden layer neural network in Theorem 4.1.
In Section 4.2, we discuss Extreme Value Theory and how it can be applied to estimate max (cid:107)∇ g ( x ) (cid:107) q .
4. S AMPLING ON THE D ISTRIBUTION OF G RADIENT N ORM
We generate samples x ( i ) over a ﬁxed ball B p ( x 0 , R ) uniformly and independently, where x 0 is the input image vector of a classiﬁer.
This way, (cid:107)∇ g ( x ) (cid:107) q can be regarded as a random variable
Y with cumulative density function (CDF) F Y
( y ) , which depends on the network architecture.
We derive the CDF for a one-hidden-layer neural network in Theorem 4.1, whose proof is deferred to
Appendix D.
Theorem 4.1 ( F Y ( y ) of one-hidden-layer neural network) .
Consider a neural network f : R d → R K with input x 0 ∈ R d , a hidden layer with U hidden neurons, and rectiﬁed linear unit (ReLU) activation function.
If we sample uniformly in a ball B p ( x 0 , R ) , then the cumulative distribution function of (cid:107)∇ g ( x ) (cid:107) q , denoted as F Y ( y ) , is piece-wise linear with at most M = (cid:80) di
=0 (cid:0) Ui (cid:1) pieces, where g ( x ) =
f c ( x ) − f j ( x ) for some given c and j , and 1 p + 1 q = 1 , 1 ≤ p, q ≤ ∞ .
Figure 2 illustrates Theorem 4.1 with d = 2 , q = 2 and U = 3 .
The three hyperplanes w i x + b i =divide the space into seven regions (with different colors).
The red dash line encloses the ball
B 2 ( x 0 , R 1 ) and the blue dash line encloses a larger ball B 2 ( x 0 , R 2 ) .
If we draw samples uniformly within the balls, the probability of (cid:107)∇ g ( x ) (cid:107) 2 = y is proportional to the intersected volumes of the ball and the regions with (cid:107)∇ g ( x ) (cid:107) 2 = y .
E XTREME V ALUE T HEORY AND THE CLEVER SCORES
Suppose we have n samples {(cid:107)∇ g ( x ( i ) )
(cid:107) q } , and denote them as a sequence of independent and identically distributed (iid) random variables Y 1 , Y 2 , · · · , Y n with CDF F Y ( y ) .
The CDF of M n = max { Y 1 , · · · , Y n } is F nY ( y ) , which is called the limit distribution of F Y ( y ) .
The Extreme Value
Theory in Theorem 4.2 says that F nY ( y ) , if exists, can only be one of the three family of extreme value distributions - the Gumbel class, the Fr ´ echet class and the Reverse Weibull class.
Lemma 4.2 (Fisher-Tippett-Gnedenko (De Haan & Ferreira, 2007)) .
If there exists a sequence of pairs of real numbers ( a n , b n ) such that a n > 0 and lim n →∞ F nY ( a n y + b n ) =
G ( y ) , where G is a non-degenerate distribution function, then G belongs to either the Gumbel class (Type I), the
Fr´echet class (Type II) or the Reverse Weibull class (Type III) with their CDFs as follows:
[Equation]
Weibull
[Equation]
where a ∈ R , b > 0 and c > 0 are the location, scale and shape parameters, respectively.
Lemma 4.2 implies that the maximum values of the samples follow one of the three families of distributions.
We are particularly interested in the Reverse Weibull class, as its CDF has a ﬁnite right end-point a , revealing the upper limit of the samples, also known as the extreme value .
In our case, this is the unknown local cross Lipschitz constant L q, x
we would like to estimate.
Here, we describe how to estimate L q, x 0 with Reverse Weibull class.
We generate N s samples of x ( i ) uniformly from B p ( x 0 , R ) in each batch with a total of N b batches, compute (cid:107)∇ g ( x ( i ) )
(cid:107) q , and store the maximum values of each batch into a set S .
We then perform maximum likelihood estimation on the parameters of the Reverse Weibull distribution with S , and the estimated location parameter, ˆ a , is used as an estimate of L q, x 0 .
The ﬂow of computing CLEVER score for targeted attacks is summarized in Algorithm 1.
CLEVER also applies to un-targeted attacks by computing CLEVER scores over all possible targets and take the minimum of them.
Input: a K -class neural network f ( x ) , data example x 0 with predicted class c , target class j , batch size N b , number of samples per batch N s , perturbation norm p , max perturbation R
Result: CLEVER Score µ ∈ R + for target class j 1 S ← {∅} , g ( x ) ← f c ( x ) − f j ( x ) .
2 for i ← 1 to N b do for k ← 1 to N s do randomly select a point x
( i,k ) ∈
B p ( x 0 , R ) compute
b ik ← (cid:107)∇ g ( x ( i,k ) )
(cid:107) p via back propagation end
S ← S ∪ { max k { b ik }} 8 end 9 ˆ a = location parameter of maximum likelihood estimation of reverse Weibull distribution on S 10 µ ← min( g ( x 0 )
ˆ a , R )
E XPERIMENTAL R ESULTS 5. N ETWORKS AND P ARAMETER S ETUP
We conduct experiments on CIFAR-10 (CIFAR for short), MNIST, and ImageNet data sets.
For the former two smaller data sets CIFAR and MNIST, we evaluate CLEVER scores on four relatively small networks: a single hidden layer MLP with softplus activation (with the same parameters as in (Hein & Andriushchenko, 2017)), a 7-layer AlexNet-like CNN (with the same parameters as in (Carlini & Wagner, 2017b)), and the 7-layer CNN with defensive distillation (Papernot et al., 2016) (DD) and bounded ReLU (Zantedeschi et al., 2017) (BReLU) defense techniques employed.
For ImageNet data set, we use three popular deep network architectures: a 50-layer Residual Network (He et al., 2016) (ResNet-50), Inception-v3 (Szegedy et al., 2016) and MobileNet (Howard et al., 2017).
They were chosen for the following reasons: (i) they all yield (close to) state-of-theart performance among equal-sized networks; and (ii) their architectures are signiﬁcantly different with unique building blocks, i.e., residual block in ResNet, inception module in Inception net, and depthwise separable convolution in MobileNet.
Therefore, they should be appropriate architectures to test our robustness metric.
For MobileNet, we set the width multiplier to 1.0, which achieves a 70 . 6% accuracy on ImageNet.
For all the three networks, we used the pretrained model from
TF-slim library 3 .
In all our experiments, we set the sampling parameters
N b = 500 , N s = 1024 and R = 5 .
We use 500 test-set images for CIFAR and MNIST and use 100 test-set images for ImageNet.
For each image, we evaluate its CLEVER score for three targeted attack classes: a random target class, a least likely class (the lowest probability class when predicting the original example), and the top-2 class (which is usually the easiest target to attack).
We only conduct experiments on targeted attack since it is strictly harder than un-targeted attack.
5. F ITTING G RADIENT N ORM SAMPLES WITH R EVERSE W EIBULL CLASS
We ﬁt the cross Lipschitz constant samples in S (see Algorithm 1) with Reverse Weibull class distribution to obtain the maximum likelihood estimate of the location parameter ˆ a , scale parameter ˆ b and shape parameter ˆ c , as introduced in Lemma 4.2.
To validate that Reverse Weibull distribution is a good ﬁt to the empirical distribution of the cross Lipschitz constant samples, we conduct
Kolmogorov-Smirov Goodness-of-Fit test (a.k.a.
K-S test) to calculate the KS statistics and pvalues.
Tested on CIFAR-MLP, MNIST-CNN, and ImageNet-MobileNet and as displayed on the top of each plot in Figure 3, the resulting high p-values and small KS scores empirically validate the use of Reverse Weibull distribution as an underlying distribution of the cross Lipschitz constant samples.
Therefore, its location parameter ˆ a (i.e., the extreme value) can be used to calculate the
CLEVER score.
Figure 3 plots the probability distribution function of the cross Lipschitz constant
[Figure]
The estimated MLE parameters, p-values and the KS scores are also speciﬁed accordingly.
5. C OMPARING CLEVER S CORE WITH A TTACK - INDUCED N ETWORK R OBUSTNESS
We apply the state-of-the-art white-box attack methods, iterative fast gradient sign method (I-FGSM)
(Goodfellow et al., 2015; Kurakin et al., 2016b) and Carlini and Wagner’s attack (CW attack) (Carlini & Wagner, 2017b), to ﬁnd adversarial examples for 11 networks, including 4 networks trained on CIFAR, 4 networks trained on MNIST, and 3 networks trained on ImageNet.
For CW attack, we run 1000 iterations for ImageNet and CIFAR, and 2000 iterations for MNIST, as MNIST has shown to be more difﬁcult to attack (Chen et al., 2017a).
Attack learning rate is individually tuned for each model: 0.001 for Inception-v3 and ResNet-50, 0.0005 for MobileNet and 0.01 for all other networks.
For I-FGSM, we run 50 iterations and choose the optimal (cid:15) ∈ { 0 . 01 , 0 . 025 , 0 . 05 , 0 . 1 , 0 . 3 , 0 .
81 . 0 } to achieve the smallest (cid:96) ∞ distortion for each individual image.
For defensively distilled (DD) networks, 50 iterations of I-FGSM are not sufﬁcient; we use 250 iterations for CIFAR-DD anditerations for MNIST-DD to get a 100% success rate.
For the problem to be non-trivial, images that are classiﬁed incorrectly are skipped.
For comparison, we compute the CLEVER scores for the same set of images and attack targets.
To the best of our knowledge, CLEVER is the ﬁrst attackindependent robustness score that is capable of handling the large networks studied in this paper, so we directly compare it with the attack-induced distortion metrics in our study
[Table] [Table]
We report that the attack success rates for all the networks are 100%, and thus the average distortion of adversarial examples can indicate the attack-speciﬁc robustness of each network.
Table 2 shows the average (cid:96) 2 and (cid:96) ∞ distortions for each attack algorithm and attack target for all data sets, as well as the corresponding average CLEVER scores for (cid:96) 2 and (cid:96) ∞ norms.
We evaluate the effectiveness of our CLEVER score by comparing the upper bound β U (found by attacks) and CLEVER score, where CLEVER serves as an estimated lower bound, β L .
As expected, CLEVER is smaller than the
[Table]
[Figure]
distortions of adversarial images in most cases.
More importantly, since CLEVER is independent of attack algorithms, the reported CLEVER scores can roughly indicate the distortion of the best possible attack in terms of a speciﬁc (cid:96) p distortion.
The average (cid:96) 2 distortion found by CW attack is close to the (cid:96) 2 CLEVER score, indicating it is a strong (cid:96) 2 attack.
In addition, when a defense mechanism (Defensive Distillation or Bounded ReLU) is used, the corresponding CLEVER scores are consistently increased, indicating that the network is indeed made more resilient to adversarial perturbations.
Our CLEVER score can also be used as a security checkpoint for unseen attacks.
For example, if there is an substantial gap in distortion between the CLEVER score and the considered attack algorithms, it may suggest the existence of a more effective attack that can close the gap.
Since CLEVER score is derived from an estimation of the robustness lower bound, we further verify the viability of CLEVER per each example, i.e., whether it is usually smaller than the upper bound found by attacks.
Table 3 shows the percentage of inaccurate estimations where the CLEVER score is larger than the distortion of adversarial examples found by CW and I-FGSM attacks in three
ImageNet networks.
We found that CLEVER score provides an accurate estimation for most of the examples.
For MobileNet and Resnet-50, our CLEVER score is a strict lower bound of these two attacks for more than 93% of tested examples.
For Inception-v3, the condition of strict lower bound is worse (still more than 78%), but we found that in these cases the attack distortion only differs from our CLEVER score by a fairly small amount.
For the purpose of visual illustration,
Figure 4 shows the scatter plots of our CLEVER score and the (cid:96) 2 distortion of CW attack for all tested examples on ImageNet.
It can be observed thatmost of the adversarial examples are close to the corresponding CLEVER scores, which signiﬁes the near-optimality of CW attack in terms of (cid:96)distortion, as CLEVER sufﬁces for an estimated capacity of the best possible attack.
C ONCLUSION
In this paper, we propose the CLEVER score, a novel and genetic metric to evaluate the robustness of a target neural network classiﬁer to adversarial examples.
Compared to the existing robustness evaluation approaches, our metric has the following advantages: (i) attack-agnostic; (ii) applicable to any neural network classiﬁers; (iii) comes with strong theoretical guarantees; and (iv) is computationally feasible for large neural networks.
Our extensive experiments show that the CLEVER score well matches the practical robustness indication of a wide range of natural and defended networks.
A PPENDIX A: P ROOF OF T HEOREM 3.
Proof.
According to Lemma 3.1, the assumption that g is Lipschitz continuous with Lipschitz constant L q gives
[Equation]
Let x = x
+ δ
and y =
x 0 in (4), we get
[Equation]
, which can be rearranged into the following form
[Equation]
In other words, the assumption of Lipschitz continuity on g gives bounds on g ( x 0 + δ ) in terms of g ( x 0 ) , (cid:107) δ (cid:107) p and L q , which will be useful in deriving the bounds on (cid:107) δ (cid:107) p in below.
First, if g ( x 0 ) − L q (cid:107) δ (cid:107) p > 0 , then the bounds in (5) give
[Equation]
By the deﬁnition of g , the above equation means that if (cid:107) δ (cid:107) p is less than f c ( x 0 ) − f j ( x 0 ) L q , then the classiﬁer decision on x 0 + δ will never change from class c to class j :
[Equation]
On the other hand, if there exists a δ such that g ( x 0 + δ ) < 0 , then the bounds in (5) give
[Equation]
Again, by the deﬁnition of g , the above equation means that if the classiﬁer decision can be changed to class j , we need (cid:107) δ (cid:107) p to be at least larger than a positive constant f c ( x 0 ) − f j ( x 0 )
L q
[Equation]
We should be careful about the above statement - it is possible that there doesn’t exist a δ at all achieving f c ( x 0 + δ ) − f j ( x 0 + δ ) , for example, if the functions satisfy f c ( x ) >
f i ( x ) , ∀ x .
However, (7) says that if the classiﬁer decision of x 0 + δ can be changed, then the (cid:96) p norm of this δ has to be at least larger than f c ( x 0 )
− f j ( x 0 ) L q .
Finally, to achieve c = argmax 1 ≤ i ≤ K f i ( x 0 + δ ) , we take the minimum of the bound on (cid:107) δ (cid:107) p in (6) over j (cid:54) =
c .
I.e. if
[Equation]
the classiﬁer decision can never be changed and the attack will never succeed.
A PPENDIX B: P ROOF OF C OROLLARY 3.2.
Proof.
By Lemma 3.1 and let g = f c − f j , we get L q,x 0 = max y ∈ B p ( x 0 ,R ) (cid:107)∇ g ( y ) (cid:107) q = max y ∈ B p ( x 0 ,R )
(cid:107)∇ f j ( y ) − ∇ f c ( y ) (cid:107) q , which then gives the bound in Theorem 2.1 of (Hein & Andriushchenko, 2017).
A PPENDIX C: P ROOF OF L EMMA 3.
Proof.
For any x, y , let { ( c i x +(1 − c i ) y
k − 1 i =1 be the non-differentiable points between in line ( x, y ) and 0 =
c 0 < c 1 < · · · < c k − 1 < c k = 1 .
Let z i
=
( c i x + (1 − c i ) y ) for all i = 0 , . . .
, k .
Then, using the fundamental theorem of calculus on each interval separately, there exists
¯ z i ∈
( z i , z i − 1 ) for each i such that
[Equation]
A PPENDIX D: P ROOF OF T HEOREM 4.
Proof.
The j th output of a one-hidden-layer neural network can be written as
[Equation]
where σ ( z ) = max( z, 0) is ReLU activation function, W and V are the weight matrices of the ﬁrst and second layer respectively, and w r is the r th row of W .
Thus, we can compute g ( x ) and (cid:107)∇ g ( x ) (cid:107) q below:
[Equation]
and
[Equation]
where I ( z ) is an univariate indicator function:
[Equation]
As illustrated in Figure 2, the hyperplanes
w r x + b r = 0 , r ∈ { 1 , . .
, U } divide the d dimensional spaces R d into different regions, with the interior of each region satisfying different inequality constraints, e.g. w r + x + b r + > 0 and w r − x + b r − < 0 .
Given x , we can identify which region it belongs to by checking the sign of w r x + b r for each r .
Notice that the gradient norm is the same for all the points in the same region, i.e. for any x 1 , x 2 satisfying I ( w r x 1 + b r )
=
I ( w r x 2 + b r ) ∀ r , we have (cid:107)∇ g ( x 1 ) (cid:107) q
=
(cid:107)∇ g ( x 2 ) (cid:107) q .
Since there can be at most M = (cid:80) di =0 (cid:0) Ui (cid:1) different regions for a d -dimensional space with U hyperplanes, (cid:107)∇ g ( x ) (cid:107) q can take at most M different values.
Therefore, if we perform uniform sampling in a ball B p ( x 0 , R ) centered at x 0 with radius R and denote (cid:107)∇ g ( x ) (cid:107) q as a random variable Y , the probability distribution of Y is discrete and its CDF is piece-wise constant with at most M pieces.
Without loss of generality, assume there are M 0 ≤ M distinct values for Y and denote them as m (1) , m (2) , . . .
, m ( M 0 ) in an increasing order, the CDF of
Y , denoted as F Y ( y ) , is the following:
[Equation]
where F Y ( m (0) )
=
with m (0) < m (1) , V d ( E ) is the volume of E in a d dimensional space.