P ROVABLY R OBUST A DVERSARIAL E XAMPLES
Anonymous authors
Paper under double-blind review
A BSTRACT
We introduce the concept of provably robust adversarial examples – connected input regions constructed from standard adversarial examples which are guaranteed to be provably robust to a set of perturbations (such as changes in pixel intensity and geometric transformations).
We present a novel method for generating these regions in a scalable manner which works by iteratively reﬁning the region initially obtained via sampling until certiﬁcation with state-of-the-art methods is achieved.
At each step, a novel optimization procedure is applied to maximize the region’s volume under the constraint that the resulting convex relaxation implies a chosen bound on the certiﬁcation objective.
Our experimental evaluation shows the effectiveness of the method: it successfully ﬁnds large provably robust regions including ones containing ≈ 10 573 adversarial examples for pixel intensity and ≈ 10 599 for geometric transformations.
We verify empirically that our robust adversarial examples are signiﬁcantly more effective against state-of-the-art defenses based on randomized smoothing than the individual attacks used to construct the regions.
I NTRODUCTION
Deep neural networks (DNNs) are vulnerable to adversarial attacks: small input perturbations that cause misclassiﬁcation (Szegedy et al., 2013).
This has caused an increased interest in investigating powerful attacks (Goodfellow et al., 2015; Carlini & Wagner, 2017; Madry et al., 2018;
Andriushchenko et al., 2019; Zheng et al., 2019; Wang et al., 2019; Croce & Hein, 2019; Tramèr et al., 2020).
An important limitation of existing attack methods is that they only produce a single concrete adversarial example and their effect can often be mitigated with existing defenses (Madry et al., 2018;
Hu et al., 2019; Sen et al., 2020; Xiao et al., 2020; Pang et al., 2020; Lécuyer et al., 2019; Cohen et al., 2019;
Salman et al., 2019; Fischer et al., 2020; Li et al., 2020).
The effectiveness of these attacks can be improved by robustifying them: to consolidate the individual examples to produce a large symbolic region guaranteed to only contain adversarial examples.
This Work: Provably Robust Adversarial Examples We present the concept of a provably robust adversarial example – input region capturing a very large set (e.g., > 10 100 ) of points guaranteed to be adversarial.
We then introduce a novel algorithm for generating such regions and apply it to the setting of pixel intensity changes and geometric transformations.
Our work relates to prior approaches on generating robust adversarial examples (Athalye et al., 2018; Qin et al., 2019) but differs in a crucial point: our regions are guaranteed to be adversarial while prior approaches are empirical and offer no such guarantees.
Main Contributions
Our key contributions are:
• The concept of a provably robust adversarial example – a connected input region capturing a very large set of points, generated by a set of perturbations, guaranteed to be adversarial.
• A novel scalable method for synthesizing provably robust adversarial examples, based on iterative reﬁnement and certiﬁcation techniques.
• A thorough evaluation of our method demonstrating it can generate provable regions containing ≈ 10 573 concrete adversarial points for pixel intensity changes, in ≈ 2 minutes, and ≈ 10 599 concrete points for geometric transformations, in ≈ 20 minutes, on a challenging CIFAR10 network.
We also demonstrate that our robust adversarial examples are signiﬁcantly more effective against state-of-the-art defenses based on randomized smoothing than the individual attacks used to construct the regions.
B ACKGROUND
We now discuss the background necessary for the remainder of the paper.
We consider a neural network f : R n 0 → R n l with l layers, n 0 input neurons and n l output classes.
The network classiﬁes an input x to class y ( x ) with the largest corresponding output value, i.e., y ( x ) = argmax i [ f ( x )] i .
Note for brevity we will omit the argument to y when it is clear from the context.
2. N EURAL N ETWORK C ERTIFICATION
In this work, we rely on state-of-the-art neural network certiﬁcation methods based on convex relaxations to prove that the adversarial examples produced by our algorithm are robust.
These certiﬁcation methods take a convex input region I ⊂ R n 0 and prove that every point in I is classiﬁed as target label y t by f .
They propagate the set I through the layers of the network, producing a convex region that covers all possible values of output neurons (Gehr et al., 2018).
Robustness follows by proving that, for all combinations of output neuron values in this region, the output neuron corresponding to class y t has a larger value than the one corresponding to any other class
y (cid:54) =
y t .
Commonly, one proves this property by computing a function L y : R n 0 → R for each label y
(cid:54) =
y t , such that, for all x ∈ I , we have L y ( x ) ≤ [ f ( x )]
y t − [ f ( x )]
y .
For each L y , one computes min x ∈I L y ( x ) to obtain a global lower bound that is true for all x ∈ I .
If we obtain positive bounds for all y (cid:54) =
y t , robustness is proven.
To simplify notation, we will say that the certiﬁcation objective
L ( x ) is the function L y ( x ) with the smallest minimum value on I .
We will call its corresponding minimum value the certiﬁcation error .
We require L y ( x ) to be a linear function of x .
This requirement is consistent with many popular certiﬁcation algorithms based on convex relaxation, such as CROWN (Zhang et al., 2018), DeepZ (Singh et al., 2018a), and DeepPoly (Singh et al., 2019).
Without loss of generality, for the rest of this paper, we will treat DeepPoly as our preferred certiﬁcation method.
2. G EOMETRIC C ERTIFICATION
DeepPoly operates over speciﬁcations based on linear constraints over input pixels for veriﬁcation. These constraints are straightforward to provide for simple pixel intensity transformations such as adversarial patches (Chiang et al., 2020) and L ∞ (Carlini & Wagner, 2017) perturbations that provide a closed-form formula for the input region. However, the geometric transformations do not yield such linear regions.
To prove the robustness of our generated examples to geometric transformations, we rely on DeepG (Balunovi ´ c et al., 2019) which, given a range of geometric transformation parameters, creates an overapproximation of the set of input images generated by geometric perturbations.
DeepG then levarages DeepPoly to certify the input image region.
When generating our geometric robust examples, we work directly in the geometric parameter space and, thus, our input region I and the inputs to our certiﬁcation objective L ( x ) are also in geometric space.
Despite this change, as our approach is agnostic to the choice of the veriﬁer, in the remainder of the paper we will assume the certiﬁcation is done using DeepPoly and not DeepG, unless otherwise stated.
2. R ANDOMIZED S MOOTHING
Randomized smoothing (Lécuyer et al., 2019; Cohen et al., 2019) is a provable defense mechanism against adversarial attacks.
For a chosen standard deviation σ and neural network f as deﬁned above, randomized smoothing computes a smoothed classiﬁer g based on f , such that g ( x ) = argmax c
P ( y ( x + (cid:15) )
=
c ) with random Gaussian noise (cid:15) ∼ N (0 , σ 2 I ) .
This construction of g allows
Cohen et al.
(2019) to introduce the procedure C ERTIFY that provides probabilistic guarantees on the robustness of g around a point x :
Proposition 1.
(From Cohen et al.
With probability at least 1 − α over the randomness in
C ERTIFY , if C ERTIFY returns a class y and a radius R (i.e does not abstain), then g predicts y within radius R around x : g ( x + δ ) =
y , for all (cid:107) δ (cid:107) 2 < R .
We deﬁne adversarial attacks on smoothed classiﬁers as follows:
Deﬁnition 1 (Adversarial attack on smoothed classiﬁers) .
For a ﬁxed σ , α and an adversarial distance R (cid:48) ∈ R > 0 , we call ˜ x ∈ R n 0 an adversarial attack on the smoothed classiﬁer g at the point x ∈ R n 0 , if (cid:107) ˜ x − x (cid:107) 2 < R (cid:48) and g ( x ) (cid:54) = g (˜ x ) .
[Figure]
Similarly to generating adversarial attacks on the network f , we need to balance the adversarial distance R (cid:48) on g . If too big — the problem becomes trivial; if too small — no attacks exist.
We outline the exact procedure we use to heuristically select R (cid:48) in Appendix A.6.
Using the above deﬁnition, we deﬁne the strength of an attack ˜ x as follows:
Deﬁnition 2 (Strength of adversarial attack on smoothed classiﬁers) .
We measure the strength of an attack ˜ x in terms of R adv – the radius around ˜ x , whose L 2 ball is certiﬁed to be the same adversarial class as ˜ x by g using C ERTIFY for a chosen σ and α .
Intuitively, this deﬁnition states that for points ˜ x for which R adv is bigger, the smoothed classiﬁer is less conﬁdent about predicting the correct class, since more adversarial examples are sampled in this region and therefore, the attack on g is stronger.
We use this measure in Section 5 to compare the effectiveness of our adversarial examples to examples obtained by PGD on g .
O VERVIEW
Existing methods for generating robust adversarial examples focus on achieving empirical robustness (Qin et al., 2019; Athalye et al., 2018).
In contrast, we consider provably robust adversarial examples, deﬁned below:
Deﬁnition 3 (Provably Robust Adversarial Example) .
We deﬁne provably robust adversarial example to be any large connected neural network input region, deﬁned by a set of perturbations of an image, that can be formally proven to only contain adversarial examples.
In this section, we outline how we generate such regions.
The technical details are given in Section 4.
To generate a provably robust adversarial example, ideally, we would like to directly maximize the input region’s size, under the constraint that it only contains adversarial examples.
This leads to multiple challenges: Small changes of the parametrization of the input region (e.g., as a hyperbox) can lead to large changes of the certiﬁcation objective, necessitating a small learning rate for optimization algorithms based on gradient descent.
At the same time, the optimizer would have to solve a full forward veriﬁcation problem in each optimization step, which is slow and impractical.
Additionally, according to Balunovic & Vechev (2020), bad initialization of the robust region might further cause convergence to local minima, resulting in small robust regions.
We now provide an overview of how we generate robust adversarial regions while alleviating these problems.
Our method for generating robust examples in the shape of a hyperbox is shown in Figure 1 and assumes an algorithm A that generates adversarial examples and a neural network veriﬁer V .
We require V to provide the certiﬁcation objective L ( x ) as a linear function of the network’s input neurons (whose values can be drawn from the input hyperbox), as described in Section 2.
We split our algorithm into two steps, described in Section 3.1 and Section 3.2, to address the challenges outlined above.
An optional ﬁnal step, illustrated in Figure 2 and outlined in Appendix D, demonstrates the extension of our algorithm to generate polyhedral examples.
3. C OMPUTING AN O VERAPPROXIMATING R EGION
In the ﬁrst step, we compute an overapproximation hyperbox O by ﬁtting a hyperbox around samples obtained from A .
O is represented as a dashed blue rectangle in Figure 1.
Intuitively, we use O to restrict the search space for adversarial regions to a part of the input space that A can attack.
3. C OMPUTING AN U NDERAPPROXIMATING R EGION
In the second step, we compute the hyperbox robust adversarial example U , represented as a dotted violet rectangle in Figure 1.
We will refer to U as underapproximation hyperbox to distinguish it from the polyhedral robust example P generated in Appendix D. U is obtained by iteratively shrinking O until arriving at a provably adversarial region.
We denote the hyperbox created at iteration i of the shrinking process with U i , where U
= O .
At each iteration i , we compute the certiﬁcation objective L i − 1 ( x ) of the region U i − 1 computed in the previous iteration.
If the objective is proven to be positive, we assign U to U i − 1 , as in Step 4 in Figure 1, and return.
Otherwise, we solve an optimization problem, seeking the largest hyperbox U i ⊆
U i − 1 under the constraint that min x ∈U i L i − 1 ( x ) ≥ − p
i − 1 , for a chosen bound p i − 1 ≥ 0 .
We describe the optimization problem involved in the shrinking procedure in the next paragraph.
In Section 4, we will present and motivate our choice of p i − 1 .
An important property of our choice of p i − 1 is that it becomes smaller over time and thus forces the certiﬁcation objective to increase over time until it becomes positive.
We note that optimizing L ( x ) this way is simpler than directly optimizing for U by differentiating through the convex relaxation of the network.
In particular, our method does not require propagating a gradient through the convex relaxation of the network and the convex relaxation is updated only once per iteration.
Additionally, it allows our algorithm to work with convex relaxation procedures that are not differentiable, such as the DeepG method employed in this paper.
The resulting hyperbox U is a provable adversarial region.
Shrinking U i −
There are multiple ways to shrink U i − 1 so that the condition min x ∈U i L i − 1 ( x ) ≥ − p i − 1 is satisﬁed.
However, we aim to obtain the largest possible underapproximation box U , so we make the greedy choice to shrink U i − 1 in a way that results in the largest possible new box U i .
This may not be the globally optimal choice across all iterations of the algorithm, which is harder to compute, but our choice is locally optimal.
One possible solution to the shrinking problem is to use a scheme that we call uniform shrinking, where all lower bounds [ l i − 1 ] j and upper bounds [ u i − 1 ] j in each dimension j of the hyperbox U i −are changed by the same amount.
The optimal amount for this uniform shrinking scheme can then be selected using a binary search.
While this gives an efﬁcient shrinking algorithm, in practice we observe that uniform shrinking produces small regions.
This is expected, as it may force all dimensions of the hyperbox to be shrunk by a large amount even though it might be enough to shrink only along some subset of the dimensions.
In contrast, our shrinking scheme adjusts each [ l i − 1 ] j and [ u i − 1 ] j individually.
While this shrinking scheme is more powerful, optimizing the size of U i with respect to its lower and upper bounds [ l i ] j and [ u i ] j becomes more complex.
We approach the problem differently in the high dimensional input setting related to L ∞ pixel intensity changes and the low dimensional input setting related to geometric changes.
Shrinking for L ∞
Transformations For the high-dimensional setting of L ∞ pixel transformations, we choose to maximize (cid:80) n 0 j =1 [ w i ] j , where [ w i ]
j = [ u i ] j − [ l i ] j denotes the width of the j th dimension of U i .
The linear objective allows for solutions where a small number of dimensions are shrunk towidth in favour of allowing most input dimensions to retain a big width.
As a result, in our computed region U i there are few pixels that can only take a single value while most other pixles can take a big range of values.
Since, (cid:80) n 0 i =
j [ w i ] j is a linear function of [ l i − 1 ] j and [ u i − 1 ] j , the optimization can be posed as a Linear Program (LP) (Schrijver, 1998) and be efﬁciently solved with an LP solver to obtain optimal values for [ u i ] j and [ l i ] j w.r.t the objective, as we show in Appendix A.3.
Shrinking for Geometric Transformations For the low-dimensional setting of geometric transformations, we maximize the logarithm of the volume of U i .
This objective encourages roughly equally-sized ranges for the dimensions of U i .
We solve the optimization problem using PGD, which can lead to merely locally-optimal solutions for the volume of U i .
We, therefore, rely on multiple initializations of PGD and choose the best result.
Appendix A.4 describes the process in detail.
3. P ROVABLY R OBUST P OLYHEDRAL A DVERSARIAL E XAMPLES
We remark that our provably robust adversarial examples are not restricted to hyperbox shapes.
Appendix A describes an optional ﬁnal step of our algorithm that generates provably robust polyhedral adversarial examples P from U and O .
P is iteratively cut from O , such that P ⊇ U is ensured.
Therefore, P represents a bigger region, but takes additional time to compute.
The computation of P , requires a V that accumulates imprecision only at activation functions, such as ReLU or sigmoid, in order to work.
This property is violated by DeepG’s handling of the geometric transformations.
For this reason, we only apply it in the L ∞ -transformation setting.
We refer interested readers to
Appendix A for information on how we generate polyhedral adversarial examples.
O UR F RAMEWORK
We now present our algorithm in more formal terms.
We already discussed in detail how to compute
O in Section 3.
We use Algorithm 1 to compute U .
It requires a neural network f : R n 0 → R n l with l layers, adversarial target class y t , an overapproximation hyperbox O , a speed/precision trade-off parameter c ∈ [0 , 1] , a maximum number of iterations u it and a neural network certiﬁcation method V ( f, U i − 1 , y t ) that takes a neural network f , a hyperbox U i − 1 and a target y t and returns a linear certiﬁcation objective L i − 1 ( x ) .
1: func G ENERATE _
U NDER A PPROX ( f, O , V , y t , c, u it ) 2:
U
= O 3: for i ∈ { 1 , 2 , . . .
, u it } do 4:
L i − 1 ( x ) , e i −
=
V
( f, U i 1 , y t ) 5: if e i then 6: return U i − 1 7: end if 8: p i
= e c 9: if L ∞ _ transformation then 10:
U i =
Shrink_LP( U i − 1 , L i − 1 ( x ) , p i − 1 ) 11: else 12:
U i =
Shrink
PGD( U i
1 , L i
1 ( x ) , p i 1 ) 13: end if 14: end for 15: return FailedToConverge 16: end func
As described in Section 3.2, the algorithm generates a sequence of hyperboxes U 0 ⊇ U 1 ⊇ . . .
⊇ U u it with U
=
O and returns the ﬁrst hyperbox from the sequence proven to be adversarial by V .
At each iteration, the algorithm attempts to certify the hyperbox from the previous iteration U i − 1 to be adversarial by computing the certiﬁcation error
e i −(Line 4) and checking if it is positive (Line 5).
If successful, U i − 1 is returned.
Otherwise, we use L i − 1 ( x ) generated by our attempt at verifying
U i − 1 (Line 4) and generate the constraint min x ∈U i L i − 1 ( x ) ≥ − p
i −based on a parameter p i − 1 ≥ 0 .
We note that min x ∈U i L i − 1 ( x ) is a function of the hyperbox U i , parametrized by its lower and upper bounds.
In order to shrink U i − 1 , we optimize the lower and upper bounds of U i explicitly, such that the constraint holds true for the bounds.
The next section speciﬁes and motivates our choice of p i − 1 .
Next we discuss how we choose p i − 1 .
At each iteration i , we use the constraint min x ∈U i L i − 1 ( x ) ≥ − p
i − 1 to shrink U i −
(Line 9 – Line 13) by calling either the S HRINK _LP or S HRINK _PGD optimization procedures.
S HRINK _LP and S HRINK _PGD are detailed in Appendix A.3 and Appendix A.4, respectively.
To enforce progress of the algorithm, we require the constraint to remove non-zero volume from U i − 1 .
Therefore, the values of p i − 1 for which p
i − 1 ≥ p max i − 1 , with p max i − 1 = − min x ∈U i − 1 L i − 1 ( x ) = − e i − 1 are invalid because: min x ∈U i L i − 1 ( x ) ≥
min x ∈U i − 1 L i − 1 ( x ) ≥ − p i − 1 .
We show the hyperplane L i − 1 ( x ) =
− p max i − 1 denoted as Upper in Step 3 in Figure 1.
We note that Upper always touches U i − 1 by construction.
Decreasing the value of p i − 1 , increases the volume removed from U i − 1 , as demonstrated by the other two hyperplanes in Step 3 in Figure 1.
Additionally, we require p i − 1 ≥
as any value of p i − 1 = 0 creates U i that is provably adversarial.
Choosing a negative value for p i − 1 generates smaller provably adversarial U i than with p
i − 1 = 0 .
We denote the hyperplane corresponding to p i − 1 = 0 in Step 3 in Figure 1 denoted Provable .
We use p i − 1 ∈
[0 , p max i − 1 ) .
[Table]
We note that while p i − 1 = 0 is guaranteed to generate a provable region U i , it is often too greedy.
This is due to the fact that with an imprecise V , L i − 1 ( x ) itself is also imprecise and, therefore a more precise certiﬁcation method may certify U i as adversarial for p i − 1 > 0 .
Further, as the region U i −shrinks, the precision of V increases and vice versa.
Thus V might be capable of certifying U i , for p i − 1 > 0 , since on the next iteration of the algorithm the more precise certiﬁcation objective L i ( x ) will be used for certifying the robustness of the smaller U i .
We choose p i − 1 heuristically, by setting p i − 1 =
p max i − 1 · c , for a chosen parameter c ∈ [0 , 1) .
This is depicted in Line 8 in Algorithm 1.
The closer c is to 1 , the bigger the chance that the modiﬁed region will not verify, but, also, the bigger the region we produce and vice versa.
Thus, c balances the precision and the speed of convergence of the algorithm.
We empirically verify the effect of c in Appendix B.1.
The hyperplane resulting from c = 0 .
65 is denoted as Chosen in Step 3 in Figure 1.
We further note that, when the upper and provable hyperplanes, at a particular iteration get closer than a threshold t we set p i − 1 = 0 to allow for immediate termination of the algorithm.
This speeds up the convergence of the algorithm in the ﬁnal stages, when the precision loss of V is not too big.
E XPERIMENTAL E VALUATION
We now evaluate the effectiveness of our approach on realistic networks.
We implemented our framework in Python and used Tensorﬂow (Abadi et al., 2015) for generating PGD attacks.
We use
Gurobi 9.0 (Gurobi Optimization, LLC, 2020) for solving LP instances.
We rely on ERAN (Singh et al., 2018b) for its DeepPoly and DeepG implementations.
We ran all our experiments on a 2.8 GHz 16 core Intel(R)
Xeon(R) Gold 6242 with 64 GB RAM.
Neural Networks We use MNIST (LeCun et al., 1998) and CIFAR
(Krizhevsky, 2009) based networks.
Table 4 in Appendix A.1 shows the sizes and types of different neural networks.
All networks in our L ∞ experiments except the CIFAR10 ConvBig are not adversarially trained.
In our experiments, normally trained networks were more sensitive to geometric perturbations than L ∞ .
Therefore, we use DiffAI-trained (Mirman et al., 2018) networks which tend to be less sensitive against geometric transformations.
Our deepest network is the MNIST 8 × 200 fully-connected network (FFN) which has 8 layers with 200 neurons each and another layer with 10 neurons.
Our largest network is the CIFAR10 ConvBig network with 6 layers and ≈ 62 K neurons.
This network is among the largest benchmarks that existing veriﬁers can handle in a scalable and precise manner.
We compare the methods on the ﬁrst 100 test images of the datasets, a standard practice in the veriﬁcation literature (Balunovi´c et al., 2019; Singh et al., 2019), while ﬁltering the wrongly-classiﬁed ones.
For each experiment, we tuned the value of c to balance runtime and size.
5. A DVERSARIAL E XAMPLES R OBUST TO I NTENSITY C HANGES
Table 1 summarizes our results on generating examples robust to intensity changes, whose precise deﬁnition is in Appendix A.2.
Further, Appendix E.2 shows images of examples obtained in the experiment.
[-X 370 S 2 X 1 X 1 X
 MNISTC ONV [Table]-]We compare the uniform shrinking described in Section 3, used as a baseline, against our method for generating robust hyperbox and polyhedral adversarial examples.
[-[Table] [Table] 0,1,0,1)    435 S 4 X 2 X 441 S 2 X 1 X
 [Table]-]In all experiments, we compute O using examples collected within an L ∞ ball around a test image with the radius (cid:15) speciﬁed in Table 1.
The values of (cid:15) are chosen such that the attack has a non-trivial probability ( >but < 1 ) of success.
[-X
-]
[-[Table] 1829 S 5 X 2 X 1369 S 2 X 1 X10 599 
 [Table] [Table]-]We use 2500 adversarial examples each from the Frank-Wolfe optimization algorithm (Frank & Wolfe, 1956), PGD with step 0 .
1 (cid:15) and 0 .
01 (cid:15) , respectively.
For the PGD examples,
[Table]
[Table] [Table] [Table]
CIFARC [Table] )    1829 S [Table]
1369 S
[Table]
>
[Table]
599 <
[Table]
)    1489 S
[Table]
954 S
[Table]
>
[Table]
[Table]
[Table]
[Table]
we use output diversiﬁcation (Tashiro et al., 2020) with 5 iterations.
In all experiments, we initialize c = 0 .
99 and multiply it by a factor of 0 .
99 at each iteration of Algorithm 1.
For the hyperbox adversarial examples obtained by both our method and the baseline, we calculate the size of the example in terms of the number of concrete adversarial examples it contains and we report the median of all regions under the Size columns in Table 1.
For the polyhedral adversarial examples, we report the median number of concrete adversarial examples contained within an overapproximated hyperbox around the polyhedral region in the SizeO column.
We note that an underapproximation of the number of concrete adversarial examples in the polyhedral region is the same as those in the hyperbox adversarial examples obtained by our method.
We also report the average runtime of all algorithms on the attackable regions.
We note that for MNIST ConvBig and 8 × 200 , the regions obtained for O produced huge certiﬁcation error > 1000 , which prevented our underapproximation algorithm U to converge on these networks.
To alleviate the problem, we used uniform shrinking to lower the error to under 100 .
We then used the obtained box in place of O .
We conjecture the reason for this issue is the huge amount of imprecision accumulated in the networks due to their depth and width and note that the uniform shrinking on these networks alone performs worse.
For each network, in Table 1, we report the number of distinct pairs of an image and an adversarial target on which A succeeds (column #Reg), as well as the number of unique images on which A succeeds (column #Img). We note that #Reg > #Img.
We further report the number of test images that are correctly classiﬁed by the networks (column #Corr).
The table details the number of regions on which the different methods succeed (column #VerReg), that is, they ﬁnd a robust adversarial example containing > 1000 concrete adversarial examples.
As can be seen in Table 1, our framework creates symbolic regions that consistently contain more concrete adversarial examples (up to 10 573 images on CIFAR10 ConvBig ) than the baseline while being faster.
Our method also succeeds more often, failing only for a single region due to the corresponding hyperbox O also containing < 1000 concrete examples versus 53 failures for the baseline.
Overall, our method, unlike the baselines, generates robust examples containing large number of examples in almost all cases in which the image is attackable, in a few minutes.
5. A DVERSARIAL E XAMPLES R OBUST TO G EOMETRIC C HANGES
Next, we demonstrate the results of our algorithm for generating examples robust to geometric transformations, whose precise deﬁnition is in Appendix A.2.
Our method here relies on DeepG for robustness certiﬁcation, as mentioned in Section 2.
To increase its precision, DeepG selectively employs input space splitting, where each split is veriﬁed individually.
This strategy is effective due to the low dimensionality of the input space (3 to 4 geometric parameters in our experiments).
To this end, we choose to split our method’s initial region O , instead, where all splits are individually shrunken and their sizes are summed.
We compare our method against a baseline based on a more aggressive uniform splitting of O that does not use our shrinking procedure.
Since the cost and precision of veriﬁcation increase with splits, to allow for fair comparison we select the number of baseline splits, so the time taken by the baseline is similar or more than for our method.
Table 2 summarizes the results of our method and the baseline for the MNIST and CIFAR10 datasets.
Further, Appendix E.3 shows visualizations of the adversarial examples obtained in the experiment. [Table]
The Transform column in the table represents the set of common geometric transformations on the images.
Here, R ( x ) signiﬁes a rotation of up to ± x degrees; Sc ( x ) , scaling of up to ± x % ; Sh ( m ) , shearing with a factor up to ± m % ; B ( γ, β ) , changes in contrast between ± γ % and brightness between ± β % ; and T ( x l , x r , y d , y u ) , translation of up to x l pixels left, x r pixels right, y d pixels down and y u pixels up. The selected combinations of transformations contain ≤ 4 parameters, as is standard for assessing adversarial robustness against geometrical perturbations (Balunovi´c et al., 2019).
Unlike DeepG, we chose only combinations of ≥ 3 parameters, as they are more challenging.
The chosen combinations ensure all transformations from DeepG are present.
Similar to L ∞ , we chose the parameter bounds such that the attack algorithm has a non-trivial probability of success.
In all experiments, we compute O using random sampling attacks which are the state-of-the-art for geometric transformations (Engstrom et al., 2019).
For the 3 dimensional experiments, we rely on 15000 random samples, while for the 4D conﬁgurations we use 50000 .
For all experiments, we use c = 0 .
65 , except for the 4 dimensional CIFAR10 experiment, where we use c = 0 .
We rely on the polyhedra created by DeepG that binds the values of the image pixels to estimate the number of concrete adversarial examples contained within our robust examples.
In particular, underapproximation and overapproximation hyperboxes are computed for each image, where the underapproximation hyperbox is based on (Gopinath et al., 2019) and the overapproximation relies on computing the extreme values of each pixel.
The median number of concrete examples are depicted in the Over and Under columns in Table 2.
For all experiments, the baseline failed to generate regions with > 1000 examples on most images and, thus, we omit its computed size in Table 2 to save space.
We note that on the few images where it does not, the size generated by our method is always more.
We report the number of concrete examples, the number of splits, the average runtime on attackable regions, and the number of successfully veriﬁed regions for both the baseline and our method in the #Splits, Time, and #VerReg columns.
Table 2 demonstrates that in a similar or smaller amount of time, our algorithm is capable of generating robust adversarial examples for most regions O , while the baseline fails for most.
Further, our regions are non-trivial as they contain more than 10concrete adversarial attacks.
We note that the runtimes for generating robust examples in this setting is higher than in Table 1.
This is due to both the inefﬁciency of the attack and the time required for the robustness certiﬁcation of a single region.
Due to the ﬂexibility and the generality of our method, we believe it can be easily adapted to more efﬁcient attack and certiﬁcation methods in the future.
5. C OMPARISON WITH L IU ET AL . (2019)
In this section, we extend the method proposed in Liu et al. (2019) to generate provably robust hyperbox regions against pixel intensity changes and compare the resulting algorithm to our method. As the code provided in Liu et al.
(2019) only applies to fully connected networks, we applied both their approach and our method on a new 8x200 MNIST fully connected network with the same architecture as the one used in Table 1.
To facilitate simpler comparison, the network was trained using the code provided in Liu et al.
We initialized both methods with the same overapproximation hyperbox O and compare the shrinking phase.
With similar runtime ( ≈ 200 s), out of 84 possible regions, our method and Liu et al.
(2019) created 84 and 76 regions with median underapproximation volume of 10 184 and 10 46 , respectively.
We point out that our method is capable of creating polyhedral regions and handle the geometric perturbations, while Liu et al.
(2019) cannot.
5. R OBUSTNESS OF A DVERSARIAL E XAMPLES TO R ANDOMIZED S MOOTHING
Next, we show that our adversarial examples robust to intensity changes are signiﬁcantly more effective against state-of-the-art defenses based on randomized smoothing compared to the concrete attacks by A generated for our examples, as well as the baseline from Table 1.
We note that our setting is different than Salman et al.
(2019) which directly attacks the smoothed classiﬁer.
Instead, we create examples with our method on the original network f with A , based on the L 2 PGD attack and evaluate them on g .
For all methods, we calculate the quantity R adv , introduced in Deﬁnition 2, that measures the adversarial attack strength on g .
For our method and the baseline we select ˜ x in Deﬁnition 2 as the middle point of the underapproximation box projected back to the L 2 ball, with center x and radius the adversarial distance R (cid:48) .
For the concrete attacks, ˜ x is simply the attack itself. We exclude images whose adversarial distance R (cid:48) is more than 33% bigger than the certiﬁed radius R of x , as we consider them robust attacks. We experiment with networks f with the same architectures as in Table 1.
We supply additional details on the experimental setup in Appendix A.6.
In Table 3, we show the ratio between the adversarial strengths R adv of all other methods and our method averaged across the ﬁrst 100 test images. The last two rows in Table 3 depict the mean and the 95% percentile of the strength ratio across the attacks generated by A .
We note the smaller the ratio in Table 3 is, the weaker the attacker is with respect to our method. We observe that our examples produce attacks on the smoothed classiﬁer g with average strength ranging from 2 × to 6 × bigger compared to the alternatives. We conjecture, this is caused by our examples being chosen such that the regions around them have comparatively higher concrete example density.
We note the baseline also largely performs better than the mean individual attack, further supporting our claim.
5. C OMPARISON WITH E MPIRICALLY R OBUST E XAMPLES
We compare our provably robust adversarial examples to empirically robust adversarial examples (Athalye et al., 2018) – large input regions, deﬁned by a set of perturbations of an image, that are empirically veriﬁed to be adversarial.
The empirical veriﬁcation is based on calculating the expectation over transformations (EoT) of the obtained region deﬁned in Athalye et al.
We consider a region to be an empirically robust adversarial example if it has an EoT ≥ 90% .
We demonstrate that EoT is a limited measure, as it is not uniform.
That is, empirically robust adversarial examples can exhibit high EoT - e.g 95% , while specially chosen subregions incur very low EoT scores.
In Appendix A.5, we discuss our extension to Athalye et al.
(2018) used in this experiment.
We execute the experiment on the ﬁrst MNIST ConvBig transformation in Table 2.
We note that the algorithm from Athalye et al.
(2018) relies on sampling and does not scale to high dimensional L ∞ perturbations.
The algorithm produces 24 different regions, signiﬁcantly less than our 44 provable regions.
The 24 empirically robust adversarial examples produced achieve average EoT of 95 .
To demonstrate the non-uniformness of the empirical robustness, for each region, we considered a subregion of the example, with dimensions of size 10% of the original example, placed at the corner of the example closest to the original image.
We calculated the EoT of these subregions and found that 14 / 24 regions had an EoT of < 50% , with 9 having an EoT of 0% .
This demonstrates that the fundamental assumption that the EoT of a region is roughly uniform throughout does not hold and instead it is possible and even common in practice that a signiﬁcant part of an empirically robust example is much less robust than the remaining region.
Further, this often happens close to the original input image, thus providing the misleading impression that an empirically robust adversarial example is a much stronger adversarial attack than it actually is.
This misleading behaviour suggests that provably robust adversarial examples, which cannot exhibit such behaviour by construction, provide a more reliable method of constructing adversarial examples invariant to intensity changes and geometric transformations.
We note that a limiting factor to the practical adoption of our method remains scalability to real world networks and datasets such as
ImageNet, due to the reliance on certiﬁcation techniques that do not scale there yet.
We note that our method is general and efﬁcient enough to directly beneﬁt from any future advancement in veriﬁcation.
C ONCLUSION
We introduced the concept of provably robust adversarial examples – large input regions guaranteed to be adversarial to a set of perturbations.
We presented a scalable method for synthesizing such examples, generic enough to handle multiple perturbations types including pixel intensity changes and geometric transformations.
We demonstrated the effectiveness of our method by showing it creates more and larger regions than baselines, in similar or less time, including regions containing ≈ 10 573 adversarial examples for pixel intensity and ≈ 10 599 for geometric transformations.
A I MPLEMENTATION D ETAILS A. N ETWORKS
Table 4 shows the architectures of the networks on which we evaluate.
A. A DVERSARIAL EXAMPLE DEFINITIONS
In this section, we provide precise deﬁnitions of our adversarial examples robust to pixel intensity changes and geometric changes, used in Table 1 and Table 2.
A.2. A DVERSARIAL EXAMPLES ROBUST TO L ∞ PIXEL INTENSITY CHANGES
Deﬁnition 4.
For an image x ∈ R n 0 , a neural network f , adversarial budget (cid:15) ∈ R , and adversarial class y t (cid:54) =
f
( x ) , a convex region
I ⊆ R n 0 is called an adversarial example around x robust to pixel intensity changes, iff for all points ¯ x ∈ I
it is satisﬁed that (cid:107) ¯ x − x (cid:107) ∞ ≤ (cid:15) and y t
= f
(¯ x ) .
A.2.
A DVERSARIAL EXAMPLES ROBUST TO GEOMETRIC CHANGES
Deﬁnition 5.
Let T : R n 0 × R t → R n 0 be a geometric perturbation that takes an image x ∈ R nand a vector p ∈ R t of geometric parameters, bounded within the hyperbox [ p l , p u ] , given by the lower and upper bound vectors p l , p u ∈ R t , and produces geometrically transformed version of the image, denoted ¯ x , that should be classiﬁed the same as x .
For an image x ∈ R n 0 , a neural network f , and adversarial class y t (cid:54) =
f ( x ) , a convex region
I ⊆
R t is called an adversarial example around x robust to the geometric perturbation T , iff for all points ¯ p ∈
I it is satisﬁed that ¯ p ∈ [ p l , p u ] and y t
= f ( T ( x, ¯ p )) .
We note that this deﬁnition allows for T to be a complex composition of simple geometric transformations such as rotations and translations, as it is the case in Table 2.
In this case, p is a vector of the parameters of all transformations.
We further note that while Deﬁnition 5 deﬁnes I in terms of the geometric parameters p , one can also look at I , as the region in image space obtained by propagating all possible ¯ p through T .
DeepG Balunovi´c et al.
(2019) overapproximates the later region in image space with a polyhedra to produce the polyhedral region ¯ I in image space.
In Table 2, we report the underapproximation and overapproximation sizes of ¯
I in terms of number of concrete images it contains.
A. D ETAILED DESCRIPTION OF S HRINK _LP
In this section we describe in detail the function S HRINK _LP, used to shrink U i − 1 to U i when generating examples robust to pixel intensity changes, based on the parameters L i − 1 ( x ) and p i − 1 .
S HRINK _
LP deﬁnes two sets of LP variables for the lower bounds l i ∈ R n 0 and the upper bounds u i ∈ R n 0 of U i .
As described in Section 3.2, the LP optimizes the sum of widths of the dimensions of
U i , i.e., (cid:80) n 0 j =1 ([ u i ] j − [ l i ] j ) .
Recall that L i − 1 ( x ) is a linear function of x represented as L i − 1 ( x ) = a · x + b .
As described in Section 3.2, the LP we deﬁne needs to enforce min x ∈U i L i − 1 ( x ) ≥ − p
i − 1 .
We rewrite min x ∈U i L i − 1 ( x ) as:
[Equation]
Algorithm
1: func PGD_P ROJECT ( l i , u i , l i − 1 , u i − 1 , hp , a, b, p i − 1 ) 2: l i =
Clip( l i , l i − 1 , u i − 1 ) 3: u i =
Clip( u i , l i , u i − 1 ) 4: val = hp( l i , u i )
− b 5: if val ≥ − p i − 1 then 6: return l i , u i 7: end if 8: [ l i ] a> 0 = − [ l i ] a> 0 · ( p i − 1 − b ) val 9: [ u i ] a< 0 = − [ u i ] a< 0 · ( p i − 1 − b ) val 10: l i =
Clip( l i , l i − 1 , u i − 1 ) 11: u i =
Clip( u i , l i , u i − 1 ) RETURN l i , u i 12: end func
Since hp ( l i , u i ) is a linear function of l i and u i , we add the constraint hp
( l i , u i ) ≥ − p
i − 1 to the LP: l i , u i = argmax (cid:80) n 0 j =1 ([ u i ] j − [ l i ] j ) s.t. hp
( l i , u
[Equation]
[Equation]
p
[Equation]
[Equation]
[Equation]
≤ [ l
[Equation]
] j ≤ [ u i ]
[Equation]
≤ [ u i − 1 ] j for all j .
where l i − 1 , u i − 1 ∈ R n 0 represent the lower and upper bounds of U i − 1 , the second constraint enforces
U i ⊆
U i − 1 and l i , u i ∈ R n 0 denotes the LP solution.
Finally, S HRINK _LP constructs U i =
[ l i , u i ] . A D ETAILED DESCRIPTION OF S HRINK _PGD In this section we describe in detail the function S HRINK _PGD, used to shrink U i − 1 to U i for generating regions robust to geometric transformations, based on the parameters L i − 1 ( x ) and p i − 1 .
S HRINK _PGD relies on PGD to optimize the log volume of U i with respect to its lower and upper bounds, as described in §sec:under.
The optimization problem can be written as:
[Equation]
where we use the same notations as in the previous section.
Since we rely on PGD, we need to deﬁne how to project values of l i and u i that violate the constraints in equation 1.
Algorithm 2 demonstrates the heuristic we use for the projection.
Algorithm 2 slightly abuses the notations above, and treats l i and u i , as the concrete values of l i and u i to be projected, instead of treating them as variables.
It ﬁrst clips l i and u i to enforce U i ⊆
U i −
(Line 2—Line 3).
It then checks whether hp ( l i , u i ) ≥ − p
i − 1 is satisﬁed and if so returns (Line 4—Line 7).
Otherwise, all elements of l i corresponding to positive values of a , [ l i ] a> 0 , and all elements of u i corresponding to negative values of a , [ u i ] a< 0 , are adjusted by a factor of − p
i − 1 − b val , where val = hp
( l i , u i ) − b .
This enforces hp ( l i , u i ) ≥ − p
i −(Line 8—Line 9).
In rare cases the second projection can result in values of l i and u i violating
U i ⊆
U i − 1 .
Our solution is to clip again (Line 10—Line 11).
After the last clip, in the rare case mentioned above the returned values for l i and u i violate hp ( l i , u i ) ≥ − p
i − 1 .
However, we did not ﬁnd this to be a problem in practice, as it occurs rarely and one can continue as if choosing a different value for c in Algorithm 1.
A. D ETAILED DESCRIPTION OF ADAPTED E O T USED IN EXPERIMENTS IN S ECTION 5.
In this section we discuss the extension of Athalye et al.
we use to produce the empirically robust adversarial examples for the experiment in §sec:eotcomp.
We deﬁne an optimization procedure, where we seek a hyperbox geometric region U emp ⊆ R n 0 , deﬁned in terms of the lower bound l emp ∈ R n 0 and the upper bound u emp ∈ R n 0 vectors.
We seek to ﬁnd U emp within the set of allowed geometric transformations I ⊆ R n 0 , such that U emp is an empirically robust adversarial example [Table] calculated rely on differentiable gradients through geometric transformations we
Neural networks In Section 5.4, we experiment with networks f with the same architectures as in Table 4. Following the standard practices for training networks used with smoothing, outlined in
Cohen et al. (2019), we trained the networks on images with added Gaussian noise with a standard deviation of 0 . 15 . Selecting α and σ In the experiments in Section 5.4, we need to select α and σ to compute R adv using Deﬁnition 2.
For all methods we use α = 0 .
005 , as suggested by Cohen et al.
We select the σ value that produces the biggest adversarial strength R adv for our method and use it in all methods.
We note that tuning σ for the individual attacks generated by A is very computationally intensive and therefore we avoid it.
Selecting R (cid:48)
In this paragraph, we detail how the adversarial distance R (cid:48) is chosen.
We choose it heuristically, for each individual x .
For each x , we select R (cid:48) by searching for the smallest adversarial distance on f , where at least 10% of 500 attacks with A on f succeed but exclude images x , whose
R (cid:48) is more than 33% bigger than the certiﬁed radius R of g .
This procedure allows us to select the adversarial distance R (cid:48) , that is neither too small, nor too big, which is important to avoid making the problem trivial, as outlined in Section 2.3.
In particular, for this choice of R (cid:48) the smoothed classiﬁer g is likely to be attackable, since f contains enough attacks for a high density input adversarial region to exist.
On top of that, the exclusion of images x with too big adversarial distance R (cid:48) experimentally allowed us to exclude trivial attacks, for which x is attackable for most classes on g .
B A DDITIONAL EXPERIMENTS B. D EPENDENCE OF O UR M ETHOD ON THE V ALUE OF c
In this section, we examine the effect of the parameter c introduced in Algorithm 1 on the size of our output.
We show how the number concrete adversarial examples contained in our regions robust to geometric transformations vary, as a function of c .
Table 5 depicts the results on the MNIST ConvBig scaling and translation experiment, originally depicted in Table 2.
Columns Time and #It in Tabledepict the average time and iterations taken by Algorithm 1 to converge.
The results correspond to our intuition for c : the higher the value of c is, the greater the volume produced by our method, even allowing the algorithm to succeed for two additional images.
However, this increases runtime as the average number of iterations increases.
C A NALYSIS OF A LGORITHMC. S OUNDNESS OF A LGORITHM
Theorem 1. If Algorithm 1 converges, it returns a provably robust region U . Proof. Algorithm 1 can only complete its execution using Line 6 or Line 15.
Since we assume the algorithm successfully converged, returning at Line 15 is impossible.
Therefore, the algorithm must have returned the region U i − 1 at Line 6 and, therefore, e i − 1 must be non-negative (otherwise we would not have taken the right branch at Line 5). However, e i − 1 is obtained by executing the veriﬁer V on U i − 1 at Line 4. By the deﬁnition of our veriﬁer V , it returns non-negative e i − 1 only if U i − 1 is provably robust. This ﬁnishes our proof.
In this section, we demonstrate that Algorithm 1 converges exponentially fast under a suitable assumption for the monotonicity of the neural network certiﬁcation method V . To this end, we introduce the following deﬁnition of monotonicity of V , that we will leverage for the rest of the section. Deﬁnition 6.
Let V ( f, I , y t ) be a neural network certiﬁcation method that returns the certiﬁcation objective L I ( x ) for an input region I and target y t on the neural network f . We call V objectivemonotonic if for all f and input regions X and Y with X ⊆ Y the property: [Equation] is satisﬁed.
Intuitively, Deﬁnition 6 states that a veriﬁer is objective-monotonic if the certiﬁcation objective produced for a smaller region X is always higher (closer to veriﬁcation) than the certiﬁcation objective produced for a bigger region Y , when both of them are optimized over the smaller region
X . We note that most practical neural network certiﬁcation methods, including DeepPoly, are not objective-monotonic — that is one can ﬁnd triples f , X and Y violating the deﬁnitions above. However, in practice we ﬁnd that for all practical neural network certiﬁcation methods the deﬁnitions hold for most triples. This is expected, as for violating triples we know that the overapproximation generated for X needs to be larger than the one generated for Y .
Intuitively, if that was the case for a large portion of the triples, the certiﬁcation method will not be tight. Next, we leverage this deﬁtion to demonstrate that our algorithm converges exponentially fast under the idealized condition that the certiﬁcation method used is objective-monotonic. Theorem 2. For objective-monotonic certiﬁcation methods V in Algorithm 1, the certiﬁcation error at the i -th iteration e i converges exponentially fast towards 0 (towards certiﬁcation).
Proof. To prove Theorem 2, we will leverage the deﬁnition of objective-monotonic certiﬁcation method above to bind the certiﬁcation error between consequitive iterations of Algorithm 1. We will then use the bound to derive a bound on the certiﬁcation error e i in terms of the intial certiﬁcation error e 0 .
Finally, we will use the bound to show the exponential convergence. Since U i ⊆ U i − 1 , we can apply Deﬁnition 6 to get the inequality: [Equation]
By construction of U i , we further know: [Equation] Combining both inequalities and noting that e i = min x ∈U i L
i ( x ) , we get: [Equation] i [Equation]
[Equation]
[Figure] [Equation] [Figure] Since equation 2 holds for all i , we can apply it recursively on its right-hand side to get:
[Equation] We recall that e i is negative unless U i veriﬁes. Therefore by equation 3, e i converges exponentially fast towards 0 , regardless of e 0 .
D P ROVABLY R OBUST P OLYHEDRA A DVERSARIAL E XAMPLES
In this section, we show an extension to our method that allows us to generate provably robust adversarial examples represented as a general polyhedron shapes P in the neural network input space.
The approach relies on the underapproximation and overapproximation boxes U and O , presented in the main body, to guide the search of P and is demonstrated in Figure 2.
The results of the method presented in this section were shown in Table 1 in Section 5.1.
D. B ACKGROUND
Our method for generating polyhedral robust adversarial examples requires more restricted form for V , compared to the algorithms presented in the main body.
In this subsection we discuss the relevant background and the additional requirements we pose on V and introduce the relevant notation for the rest of this section.
For this section, we will focus on ReLU-based neural networks.
We note that our concepts can easily be extended to networks with other activation functions such as sigmoid or tanh.
We consider neural network with l layers f 1 , . . .
, f l .
The ﬁrst l − 1 layers are of the form
f i ( x ) = max(0 , A i · x + b i ) for i ∈ { 1 , . . .
, l − 1 } and the ﬁnal layer is of the form f l ( x ) =
A l · x + b l .
The neural network f is the composition of all layers: f
= f l ◦ · · · ◦ f 1 .
Let n i be the number of neurons in layer i .
If z ai,j is the input to the j -th ReLU activation in layer
i and z ri,j is the corresponding ReLU output, we can represent the neural network as the following system of constraints:
For i ∈ { 1 , . . .
, l } and j ∈ { 1 , . . . , n i } ,
[Equation]
Here, z r 0 ,j describes the j -th of n 0 input activations.
We call the two types of constraints afﬁne constraints and ReLU constraints, respectively.
We additionally constrain our inputs z r 0 to be points z ai,j z ri,j u i,j l i,j z r i , j = λ i , j · z a i , j + µ i , j (b) z ai,j z ri,j u i,j l i,j z r i , j = λ i , j · z a i , j + µ i , j z r i , j = z a i , j (c) z ai,j z ri,j u i,j l i,j z r i , j = λ i , j · z a i , j + µ i , j z r
i , j =
z a i , j [Figure] of a polyhedron P ⊆ R n 0 . An adversarial class y t has the highest score on all inputs in P if min( z al,y t − z al,y ) > 0 with respect to all constraints for all y ∈ { 1 , . . .
, n l } \ { y t } .
The goal in section will be to ﬁnd a large polyhedron P for which we can verify that this is true.
Linear approximations of ReLU As reasoning about neural network constraints directly is often intractable, given bounds l i,j ≤ z ai,j ≤ u i,j , the triangle approximation (Ehlers, 2017) relaxes the
ReLU constraints such that all involved inequalities are linear:
[Equation]
Figure 3 (a) visualizes the triangle approximation.
Here, λ i,j = u i,j / ( u i,j − l i,j ) and µ i,j = − l i,j · u i,j / ( u i,j − l i,j ) are selected such that this set of constraints describes the convex hull of the
ReLU constraints in the ( z ai,j , z ri,j ) -plane and the given bounds.
The DeepPoly approximation (Singh et al., 2019) further relaxes the triangle approximation by keeping only one of the lower bounds on each variable z ri,j .
It picks either z ri,j ≥ z ai,j or z ri,j ≥ 0 , whichever minimizes the area of the resulting triangle in the ( z ai,j , z ri,j ) -plane.
Figure 3 (b) and (c) show the two convex relaxations used in DeepPoly.
We can obtain l i,j and u i,j by optimizing each component of z ai,j according to the previously determined constraints for layers 0 , . .
, i .
We note that in the special case where u i,j ≤ 0 or l i,j ≥ 0 , no approximation is needed and z ri,j is directly assigned 0 or z ai,j , respectively.
Neurons for which the approximation is needed are called undecided .
Computing neuron bounds Most veriﬁcation algorithms based on convex relaxation start with a chosen input region for which they calculate the bounds l i,j and u i,j of z ai,j in layer-by-layer fashion, where the constraints for previous layers are used to derive the bounds for layer i .
In addition to
DeepPoly and the triangle approximation, this is also true for other commonly used convex relaxation algorithms such as CROWN (Zhang et al., 2018) and DeepZ (Singh et al., 2018a).
Next, we brieﬂy describe how the bounds are computed for a particular layer in DeepPoly.
We focus on DeepPoly since it is the particular convex relaxation algorithm we chose to employ in this section.
For all neurons, DeepPoly stores a single linear lower and upper bound inequality.
These inequalities bound the neuron’s value in terms of the values of neurons from previous layers in the network.
For
ReLU neurons, these are the inequalities described above.
For afﬁne neurons, these are simply the afﬁne transformations, where the afﬁne equality is converted to a pair of inequalities.
To arrive at the bounds l i,j and u i,j of z ai,j , DeepPoly repeatedly backsubstitutes variables in the linear inequalities of z ai,j with their corresponding linear inequalities from previous layers.
The process continues until we arrive at linear inequalities L i,j ( z r 0 ) ≤ z a i,j ≤ U i,j ( z r 0 ) , with L i,j ( x )
= a i,j · x + b i,j and U i,j ( x ) = a i,j · x + b i,j denoting linear functions in terms of input variables x ∈ R n 0 with coefﬁcients a i,j , a i,j ∈ R n 0 and b i,j , b i,j ∈ R .
The ﬁnal bounds l i,j and u i,j are obtained by optimizing the respective functions
L i,j ( x ) and U i,j ( x ) over the input region.
We note that our algorithm does not depend on details of DeepPoly’s backsubstitution algorithm:
It only requires the existence of linear functions L i,j ( x ) and U i,j ( x ) bounding neuron values z ai,j in terms of input activations.
This makes it compatible with other popular veriﬁcation algorithms based on convex relaxation, such as CROWN (Zhang et al., 2018) and DeepZ (Singh et al., 2018a).
Certiﬁcation Objective To certify the robustness of an input region, V needs to certify that min( z al,y t − z al,y ) > 0 for all y ∈ { 1 , . . .
, n l } \ { y t } .
To calculate lower bounds on those minima, a common trick is to introduce an afﬁne layer that computes the objectives z al,y t − z al,y for each y .
We will refer to this layer as objective layer.
To obtain lower bounds on our objectives, we compute lower bounds of the corresponding neuron activations in the objective layer with the usual backsubstitution procedure.
[-P 
-]
For the rest of the section we will denote the backsubstituted linear lower bounds on our objectives in terms of input activations for a given target y t with L l +1 ,y ( x ) =
a l +1 ,y · x + b l +1 ,y for each y .
Finally, to verify that min( z a l,y t − z a l,y ) > 0 , it sufﬁces to show that L l +1 ,y ( x ) > 0 for each point x in the input region.
We note that the function denoted L l +1 ,y ( x ) in this section corresponds to the function denoted L y ( x ) in the main body.
D. C OMPUTING P
The algorithm for computing P is presented in Algorithm 3.
It takes as input:
– The neural network f , as described in Section D.1.
– The underapproximation hyperbox U .
– The overapproximation hyperbox O .
– A neural network certiﬁcation method V ( f, P it − 1 , i, j ) that takes a neural network f , a polyhedron P it − 1 at iteration it − 1 , the layer number i and the neuron number j and returns the linear functions
L i,j ( x ) and U i,j ( x ) and the corresponding concrete bounds l i,j and u i,
j (See Section D.1).
– The adversarial attack’s target class y t .
– The maximum number p it of iterations used to generate the polyhedron P .
The algorithm computes P , such that U ⊆ P ⊆
O is satisﬁed.
We motivate this choice by noting that we know that U is certiﬁably robust and it is therefore a lower bound on P .
The algorithm generates a sequence of polyhedra P 0 ⊇ P 1 ⊇ . . .
⊇ P p with P
= O .
At each iteration, it computes two sets of half-space constraints – hs o (Line 4) and hs b (Line 5), corresponding to the neurons in the objective and the afﬁne layers of the network.
We detail how these constraints are computed in the next two paragraphs.
We add these constraints to the polyhedron from the last iteration P it − 1 to obtain P
it
(Line 6).
The algorithm terminates when P it − 1 is certiﬁed to be robust (Line 7 – Line 9) and P it −is returned.
The result of our algorithm is represented by the red region denoted as P in Step 3 in
Figure 2.
We note that, since P
= O , P ⊆
O is guaranteed by construction, as each subsequent iteration of the algorithm shrinks the polyhedron further.
However, U ⊆ P depends on the choice of the half-space constraints.
Generating Certiﬁcation Objective Constraints Algorithm 4 details the procedure we use to collect the half-space constraints corresponding to the objective layer of the network.
To obtain constraints from V , we collect L l +1 ,y ( x ) ≥ 0 for all output classes y ∈ { 1 , . .
, n l } \ { y t } (Line 5).
We know that adding these hyperplanes to our polyhedron P it − 1 will result in a certiﬁably robust region P it , however the region will not obey U ⊆ P it .
Therefore, we adjust the bias of constraints, so that the hyperplanes L l +1 ,y ( x ) = 0 do not intersect U (Line 8).
This will be described in more detail below.
All resulting bias-adjusted hyperplanes are collected in the set hs o (Line 9) and returned.
We note that if l j > 0 , the half-spaces L l +1 ,y ( x ) ≥ 0 are trivial.
That is { L l +1 ,y ( x ) ≥ 0 } ∩
P
it −
=
P it − 1 , and thus they are ﬁltered out of hs
o (Line 6).
In addition to detecting the trivial constraints, we use l j to check if P it − 1 is certiﬁably robust and return the result alongside hs o in Line 11.
1: func G EN _P OLY ( f, U , O , V , y t , p it ) 2: {+P 
+} = O 3: for it ∈ { 1 , 2 , . . .
, p it } do 4: hs o , Ver = Gen_Obj_Planes( f, U , P it − 1 , V , y t ) 5: hs b = Gen_Bound_Planes( f, U , P it − 1 , V ) 6:
P it =
Add_Planes( P it − 1 , hs b ∪ hs o ) 7: if Ver then 8: return P it − 1 9: end if 10: end for 11: return FailedToConverge 12: end func
1: func G EN _O BJ _P LANES ( f, U , P it − 1 , V , y t ) 2: hs o =
Ver =
True 4: for j ∈ { 1 , . .
. , n l } \ { y t } do 5:
L l +1 ,j ( x ) , _ , l j , _ = V ( f, P it − 1 , l + 1 , j ) 6: if l j ≤ 0 then 7:
Ver = False 8: ˜
L l +1 ,j ( x ) =
Adjust_Bias( U , L l +1 ,j ( x ) ) 9: hs o +=
[ ˜ L l +1 ,j ( x ) ≥ 0 ] 10: end if 11: end for 12: return hs o , Ver 13: end func
Issues Related to Using Only Objective Constraints In our experiments, we found that only adding bias-adjusted objective constraints leads to slow convergence of our algorithm.
In particular, we found that the ﬁrst few iterations of the algorithm result in a substantial improvement in the certiﬁcation objective.
However, after this initial fast progress, we observe that the change in the certiﬁcation objective becomes very small.
The reason for this is that the generated constraints from consecutive executions of V are very similar.
We conjecture that the underlying reason is that the constraints no longer change the input shape enough to substantially change the convex approximation of the network.
This is in contrast to what we observe when computing the underapproximation hyperbox region U .
The difference between the two settings comes from the number of added constraints per iteration.
For U , usually many lower and upper bounds are adjusted each iteration, while the objective constraints added are much less, especially for the pixel intensity changes experiment, where the input is very high-dimensional.
Generating Constraints from Afﬁne Layers To facilitate faster convergence, we generate additional constraints for all undecided neurons in the network.
We generate two constraints, one with respect to the lower bound of the neuron L i,j ( x ) ≥ 0 and one with respect to the upper bound
U i,j ( x ) ≤ 0 .
Algorithm 5 describes the process.
L i,j ( x ) and U i,j ( x ) are computed by V (Line 5) alongside their concrete bounds l i,j and u i,j .
The concrete bounds are used to check which neurons are undecided (Line 6).
The generated constraints have the effect of making the neurons decided and, while also enforcing substantial change in the convex relaxation of the neural network.
Note that the newly obtained constraints are bias-adjusted similar to the objective constraints described above (Line 7–Line 8).
We point out that the bias adjustment described below takes care of the problem that L i,j ( x )
≥and
U i,j ( x ) ≤ 0 cannot be simultaneously true.
Empirically, these hyperplanes serve an important role for improving the convergence rate of our algorithm.
Algorithm 5 collects them in the set hs b (Line 9) and returns them to be added to P it − 1 .
Hyperplane bias adjustment So far, we have discussed how constraints are generated but not how we adjust their biases.
The bias adjustment algorithm is given in Algorithm 6. The bias adjustment
1: func G EN _B OUND _P LANES ( f, U , P it − 1 , V ) 2: hs b = [] 3: for i ∈ { 1 , 2 , . . .
, l − 1 } do 4: for j ∈ { 1 , 2 , . . .
, n i } do 5:
L i,j ( x ) , U i,j ( x ) , l i,j , u i,j = V ( f, P it − 1 , i, j ) 6: if l i,j ≤ 0 and u i,j ≥ 0 then 7: ˜ L i,j ( x ) = Adjust_Bias( U , L i,j ( x ) ) 8: ˜ U i,j ( x ) = Adjust_Bias( U , − U
i,j ( x ) ) 9: hs b +=
[ ˜ L i,j ( x ) ≥ 0 , ˜
U
i,j ( x ) ≥ 0 ] 10: end if 11: end for 12: end for 13: return hs b 14: end func
1: func A DJUST _B IAS ( U , h ( x ) ) 2: h min = min x ∈U ( h ( x )) 3: b adj = min( 0 , h min ) 4: ˜ h
( x ) =
h ( x ) − b adj 5: return ˜ h
( x ) 6: end func for a lower bound constraint h ( x ) ≥ 0 enforces that the intersection of the constraint and the given underapproximation hyperbox U is U .
It does so by computing h min =
min x ∈U ( h ( x ))
(Line 2) and changing the given constraint to h ( x ) ≥ h min (Line 4).
The optimization h min = min x ∈U ( h ( x )) is analytically solvable, as demonstrated Section A.3.
Note that our algorithm adjusts only the constraints with negative h min (Line 3), since constraints with positive h min already satisfy the property.
The following lemma shows the soundness of our bias-adjustment algorithm:
Lemma 3.
The intersection of the bias-adjusted half-space constraint ˜ h
( x ) ≥ 0 deﬁned in Algorithm 6 with U is U .
Proof.
By the construction of h min , for all points x ∈ U we have h ( x ) ≥ h min .
We distinguish two cases for the value of h min — non-negative and negative.
If h min ≥ 0 , the bias adjustment b adj is assigned to 0 and ˜ h
( x ) =
h ( x ) .
Since h ( x ) ≥ h min for all points x ∈ U ,
˜ h
( x ) = h ( x ) ≥ h min ≥ 0 for all points x ∈ U .
Therefore, there does not exist point in U for which the bias-adjusted half-space constraint ˜
h ( x ) ≥ 0 is violated.
If h min < 0 , the bias adjustment b adj is assigned to h min and ˜ h ( x )
= h ( x ) − h min .
Therefore, ˜ h ( x ) = h ( x ) − h min ≥ h min − h min = 0 for all points x ∈ U .
Analogously to the other case, ˜ h ( x ) ≥ 0 is not violated by the points in U .
The bias adjustment for upper bound constraints h ( x ) ≥ 0 is achieved by calling Algorithm 6 with − h ( x ) , as in Line 8 in Algorithm 5.
Geometrically, the bias adjustment is represented at Step 2 in
Figure 2.
Adapting DeepPoly to Polyhedral Input Regions
Out of the box DeepPoly only handles certiﬁcation of hyperbox input regions.
This is because in order to obtain the concrete bounds l i,j = min x ∈I L i,j ( x ) and u i,j
= max x ∈I U i,j ( x ) for an input region I ⊆ R n
it uses the analytical solution of the optimization problems discussed in §sec:lpshrink.
In order to use DeepPoly for polyhedral region I , we solve the optimization problem using an Linear Program (LP).
This is computationally expensive – it involves solving two LP instances per neuron.
For large networks, this can be prohibitive.
To alleviate this issue, in the ﬁrst iteration of Algorithm 3, we remember which neurons are undecided.
The computation of the undecided neurons in the ﬁrst iteration can be done using regular DeepPoly, as P
= O .
In future iterations we solve the LP instances only for the undecided neurons, while using the P 0 bounds for the decided neurons.
This optimization is sound, since we have P it ⊆
P it −by construction.
This improves the performance substantially – for most certiﬁably robust input regions in ReLU-based networks we have just a few undecided neurons.
Proving that P constraints U and is contained within O
In this paragraph, we present a proof for the desired property of Algorithm 3 that P constraints U and is contained within O .
Theorem 4.
For all polyhedral regions P generated by Algorithm 3, U ⊆ P ⊆
O holds.
Proof.
Here we brieﬂy outline the proof of Theorem 4.
The proof is divided into two parts.
We ﬁrst prove that
P ⊆ O
and then U ⊆ P .
To prove P ⊆
O we leverage the fact that the polyhedra P is constructed from the overapproximation box O by intersecting it with the half-space constraints hb ib and hb io .
Since all constraints only reduce the volume of the polyhedra, it follows by induion that P i ⊆
O for all i .
Therefore, P ⊆ O .
To prove U ⊆ P we note that the generated half-space constraints hb b and hb o are chosen such that hb b ∩ U
= U and hb o ∩ U =
U (See Lemma 3).
Applying this property by induction over the generated half-space constraints, it follows that U
i ⊆ P for all i .
Therefore, U ⊆ P
E V ISUALISATION OF R OBUST A DVERSARIAL E XAMPLES E. D ISCUSSION
Figure 4 shows an adversarial region provably robust to intensity changes containing 10 284 adversarial images produced by our method on MNIST ConvBig.
The original image is of the digit 5 , and all images in our region are classiﬁed as 3 by the network.
The colorbar in Figure 4 quantiﬁes the number of values each pixel can take in our inferred region.
The yellow and violet colors represent the two extremes.
The intensity of the yellow-colored pixels can vary the most, thus these pixels contribute to more adversarial examples.
The intensity of the purple-colored varies the least, thus the adversarial examples in our region are sensitive to the intensity values of these pixels.
In our region, the intensity of most background pixels on the edges of the image can vary a lot, as these are green.
Violet and green color are more evenly distributed among pixels closer to the foreground (part of the digit "5").
Further, the intensity of several pixels in the foreground can also vary signiﬁcantly.
We note that all yellow pixels occur in the foreground.
In summary, our region can capture examples that can be generated by signiﬁcant variations in the intensities of several background and foreground pixels.
We supply visualizations for all of the experiments described in Section 5.1 and 5.2 in Appendix E.
E. A DVERSARIAL E XAMPLES R OBUST TO I NTENSITY C HANGES
In Figure 5 – Figure 8 we visualise adversarial examples provably robust to pixel intensity changes for the different networks in Table 1.
For all ﬁgures, the colorbar on the right-hand side quantiﬁes the number of values each pixel can take in our inferred region.
The yellow and violet colors represent the two extremes.
The intensity of the yellow-colored pixels can vary the most, thus these pixels contribute more to the adversarial examples.
In Figure 5 we show adversarial examples for the MNIST ConvBig network.
The 3 sub-ﬁgures represent the digits 9 , 5 , and 3 , but the images in our regions are classiﬁed as 4 , 3 , and 8 , respectively.
Our regions are of size 10 220 , 10 284 , and 10 262 , respectively.
[Figure] In Figure 5 we show adversarial examples for the MNIST ConvSmall network.
The 3 sub-ﬁgures represent the digits 2 , 9 , and 9 , but the images in our regions are classiﬁed as 3 , 5 , and 4 , respectively.
Our regions are of size 10 520 , 10 652 , and 10 708 , respectively.
[Figure] In Figure 7 we show adversarial examples for the MNIST 8 × 200 network.
The 3 sub-ﬁgures represent the digits 5 , 9 , and 2 , but the images in our regions are classiﬁed as 8 , 8 , and 3 , respectively.
Our regions are of size 10 148 , 10 128 , and 10 157 , respectively.
[Figure]
In Figure 8 we show adversarial examples for the CIFAR10 ConvSmall network.
The 3 sub-ﬁgures represent a boat, a bird, and another bird, but the images in our regions are classiﬁed as an airplane , a frog, and a dog, respectively.
Our regions are of size 10 626 , 10 563 , and 10 608 , respectively.
[Figure]
E. A DVERSARIAL E XAMPLES R OBUST TO G EOMETRIC C HANGES
In Figure 9 – Figure 11 we visualise adversarial examples provably robust to geometric perturbations for the different networks in Table 2.
For all ﬁgures, the colorbar on the right-hand side quantiﬁes the number of values each pixel can take in our inferred region.
The yellow and violet colors represent the two extremes.
The intensity of the yellow-colored pixels can vary the most, thus these pixels contribute more to the adversarial examples.
For all ﬁgures, the subﬁgures represent an adversarial example for each of the rows in Table 2.
In Figure 9 we show adversarial examples for the MNIST ConvBig network.
The 3 sub-ﬁgures represent the digits 6 , 1 , and 3 , but the images in our regions are classiﬁed as 5 , 8 , and 7 , respectively.
Our regions have underapproximations of size 10 159 , 10 183 , and 10 13 , respectively.
[Figure]
In Figure 10 we show adversarial examples for the MNIST ConvSmall network.
The 3 sub-ﬁgures represent the digits 6 , 1 , and 3 , but the images in our regions are classiﬁed as 5 , 8 , and 7 , respectively.
Our regions are of size 10 115 , 10 82 , and 10 63 , respectively.
[Figure]
In Figure 11 we show adversarial examples for the CIFAR10 ConvSmall network.
The 3 subﬁgures represent a dog, a truck, and a frog but the images in our regions are classiﬁed as a frog, an automobile, and a deer, respectively.
Our regions are of size 10 1422 , 10 1270 , and 10 1505 , respectively.
[Figure]
Subﬁgures (a), (b) and (c) correspond to the ﬁrst, second and third geometric perturbations for the
CIFAR10 ConvSmall network in Table 2, respectively.