E VALUATING R OBUSTNESS OF N EURAL N ETWORKS WITH M IXED I NTEGER P ROGRAMMING
A BSTRACT
Neural networks trained only to optimize for training accuracy can often be fooled by adversarial examples — slightly perturbed inputs misclassiﬁed with high conﬁdence.
Veriﬁcation of networks enables us to gauge their vulnerability to such adversarial examples.
We formulate veriﬁcation of piecewise-linear neural networks as a mixed integer program.
On a representative task of ﬁnding minimum adversarial distortions, our veriﬁer is two to three orders of magnitude quicker than the state-of-the-art.
We achieve this computational speedup via tight formulations for non-linearities, as well as a novel presolve algorithm that makes full use of all information available.
The computational speedup allows us to verify properties on convolutional and residual networks with over 100,000 ReLUs — several orders of magnitude more than networks previously veriﬁed by any complete veriﬁer.
In particular, we determine for the ﬁrst time the exact adversarial accuracy of an MNIST classiﬁer to perturbations with bounded l ∞ norm (cid:15) = 0 .
1 : for this classiﬁer, we ﬁnd an adversarial example for 4.38% of samples, and a certiﬁcate of robustness to norm-bounded perturbations for the remainder.
Across all robust training procedures and network architectures considered, and for both the MNIST and CIFAR-10 datasets, we are able to certify more samples than the state-of-the-art and ﬁnd more adversarial examples than a strong ﬁrst-order attack.
I NTRODUCTION
Neural networks trained only to optimize for training accuracy have been shown to be vulnerable to adversarial examples : perturbed inputs that are very similar to some regular input but for which the output is radically different (Szegedy et al., 2014).
There is now a large body of work proposing defense methods to produce classiﬁers that are more robust to adversarial examples.
However, as long as a defense is evaluated only via heuristic attacks (such as the Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015) or Carlini & Wagner (2017b)’s attack (CW)), we have no guarantee that the defense actually increases the robustness of the classiﬁer produced.
Defense methods thought to be successful when published have often later been found to be vulnerable to a new class of attacks.
For instance, multiple defense methods are defeated in Carlini & Wagner (2017a) by constructing defense-speciﬁc loss functions and in Athalye et al.
(2018) by overcoming obfuscated gradients.
Fortunately, we can evaluate robustness to adversarial examples in a principled fashion.
One option is to determine (for each test input) the minimum distance to the closest adversarial example, which we call the minimum adversarial distortion (Carlini et al., 2017).
Alternatively, we can determine the adversarial test accuracy (Bastani et al., 2016), which is the proportion of the test set for which no perturbation in some bounded class causes a misclassiﬁcation.
An increase in the mean minimum adversarial distortion or in the adversarial test accuracy indicates an improvement in robustness.
We present an efﬁcient implementation of a mixed-integer linear programming (MILP) veriﬁer for properties of piecewise-linear feed-forward neural networks.
Our tight formulation for nonlinearities and our novel presolve algorithm combine to minimize the number of binary variables in the MILP problem and dramatically improve its numerical conditioning.
Optimizations in our MILP implementation improve performance by several orders of magnitude when compared to a na¨ıve
MILP implementation, and we are two to three orders of magnitude faster than the state-of-the-art
Satisﬁability Modulo Theories (SMT)
based veriﬁer, Reluplex (Katz et al., 2017)
We make the following key contributions:
• We demonstrate that, despite considering the full combinatorial nature of the network, our veriﬁer can succeed at evaluating the robustness of larger neural networks, including those with convolutional and residual layers.
• We identify why we can succeed on larger neural networks with hundreds of thousands of units.
First, a large fraction of the ReLUs can be shown to be either always active or always inactive over the bounded input domain.
Second, since the predicted label is determined by the unit in the ﬁnal layer with the maximum activation, proving that a unit never has the maximum activation over all bounded perturbations eliminates it from consideration.
We exploit both phenomena, reducing the overall number of non-linearities considered.
• We determine for the ﬁrst time the exact adversarial accuracy for MNIST classiﬁers to perturbations with bounded l ∞ norm (cid:15) .
We are also able to certify more samples than the state-of-the-art and ﬁnd more adversarial examples across MNIST and CIFAR-10 classiﬁers with different architectures trained with a variety of robust training procedures.
Our code is available at https://github.com/vtjeng/MIPVerify.jl .
R ELATED W ORK
Our work relates most closely to other work on veriﬁcation of piecewise-linear neural networks;
Bunel et al.
(2018) provides a good overview of the ﬁeld.
We categorize veriﬁcation procedures as complete or incomplete .
To understand the difference between these two types of procedures, we consider the example of evaluating adversarial accuracy.
As in Kolter & Wong (2017), we call the exact set of all ﬁnal-layer activations that can be achieved by applying a bounded perturbation to the input the adversarial polytope .
Incomplete veriﬁers reason over an outer approximation of the adversarial polytope.
As a result, when using incomplete veriﬁers, the answer to some queries about the adversarial polytope may not be decidable.
In particular, incomplete veriﬁers can only certify robustness for a fraction of robust input; the status for the remaining input is undetermined.
In contrast, complete veriﬁers reason over the exact adversarial polytope.
Given sufﬁcient time, a complete veriﬁer can provide a deﬁnite answer to any query about the adversarial polytope.
In the context of adversarial accuracy, complete veriﬁers will obtain a valid adversarial example or a certiﬁcate of robustness for every input.
When a time limit is set, complete veriﬁers behave like incomplete veriﬁers, and resolve only a fraction of queries.
However, complete veriﬁers do allow users to answer a larger fraction of queries by extending the set time limit.
Incomplete veriﬁers for evaluating network robustness employ a range of techniques, including duality (Dvijotham et al., 2018; Kolter & Wong, 2017; Raghunathan et al., 2018), layer-by-layer approximations of the adversarial polytope (Xiang et al., 2018), discretizing the search space (Huang et al., 2017), abstract interpretation (Gehr et al., 2018), bounding the local Lipschitz constant (Weng et al., 2018), or bounding the activation of the ReLU with linear functions (Weng et al., 2018).
Complete veriﬁers typically employ either MILP solvers as we do (Cheng et al., 2017; Dutta et al., 2018; Fischetti & Jo, 2018; Lomuscio & Maganti, 2017) or SMT solvers (Carlini et al., 2017; Ehlers, 2017; Katz et al., 2017; Scheibler et al., 2015).
Our approach improves upon existing MILP-based approaches with a tighter formulation for non-linearities and a novel presolve algorithm that makes full use of all information available, leading to solve times several orders of magnitude faster than a na¨ıvely implemented MILP-based approach.
When comparing our approach to the state-of-the-art
SMT-based approach (Reluplex) on the task of ﬁnding minimum adversarial distortions, we ﬁnd that our veriﬁer is two to three orders of magnitude faster.
Crucially, these improvements in performance allow our veriﬁer to verify a network with over 100,000 units — several orders of magnitude larger than the largest MNIST classiﬁer previously veriﬁed with a complete veriﬁer.
A complementary line of research to veriﬁcation is in robust training procedures that train networks designed to be robust to bounded perturbations.
Robust training aims to minimize the “worst-case loss” for each example — that is, the maximum loss over all bounded perturbations of that example (Kolter & Wong, 2017).
Since calculating the exact worst-case loss can be computationally costly, robust training procedures typically minimize an estimate of the worst-case loss: either a lower bound as is the case for adversarial training (Goodfellow et al., 2015), or an upper bound as is the case for certiﬁed training approaches (Hein & Andriushchenko, 2017; Kolter & Wong, 2017; Raghunathan et al., 2018).
Complete veriﬁers such as ours can augment robust training procedures by resolving the status of input for which heuristic attacks cannot ﬁnd an adversarial example and incomplete veriﬁers cannot guarantee robustness, enabling more accurate comparisons between different training procedures.
B ACKGROUND AND N OTATION
We denote a neural network by a function f ( · ; θ ) : R m → R n parameterized by a (ﬁxed) vector of weights θ .
For a classiﬁer, the output layer has a neuron for each target class.
Veriﬁcation as solving an MILP .
The general problem of veriﬁcation is to determine whether some property P on the output of a neural network holds for all input in a bounded input domain C ⊆ R m .
For the veriﬁcation problem to be expressible as solving an MILP, P must be expressible as the conjunction or disjunction of linear properties P i,j over some set of polyhedra C i , where
C = ∪C i .
In addition, f ( · ) must be composed of piecewise-linear layers.
This is not a particularly restrictive requirement: piecewise-linear layers include linear transformations (such as fully-connected, convolution, and average-pooling layers) and layers that use piecewise-linear functions (such as ReLU or maximum-pooling layers).
We provide details on how to express these piecewise-linear functions in the MILP framework in Section 4.1.
The “shortcut connections” used in architectures such as
ResNet (He et al., 2016) are also linear, and batch normalization (Ioffe & Szegedy, 2015) or dropout (Srivastava et al., 2014) are linear transformations at evaluation time (Bunel et al., 2018).
F ORMULATING R OBUSTNESS E VALUATION OF C LASSIFIERS AS AN MILP
Evaluating Adversarial Accuracy.
Let G ( x ) denote the region in the input domain corresponding to all allowable perturbations of a particular input x .
In general, perturbed inputs must also remain in the domain of valid inputs X valid .
For example, for normalized images with pixel values ranging from 0 to 1, X valid =
[0 , 1] m .
As in Madry et al.
, we say that a neural network is robust to perturbations on x if the predicted probability of the true label λ
( x ) exceeds that of every other label for all perturbations:
[Equation]
Equivalently, the network is robust to perturbations on x if and only if Equation 2 is infeasible for x (cid:48) .
[Equation]
where f i ( · ) is the i th output of the network.
For conciseness, we call x robust with respect to the network if f ( · ) is robust to perturbations on x .
If x is not robust, we call any x (cid:48) satisfying the constraints a valid adversarial example to x .
The adversarial accuracy of a network is the fraction of the test set that is robust; the adversarial error is the complement of the adversarial accuracy.
As long as G ( x ) ∩ X valid can be expressed as the union of a set of polyhedra, the feasibility problem can be expressed as an MILP.
The four robust training procedures we consider (Kolter & Wong, 2017; Wong et al., 2018; Madry et al., 2018; Raghunathan et al., 2018) are designed to be robust to perturbations with bounded l ∞ norm, and the l ∞ -ball of radius (cid:15) around each input x can be succinctly represented by the set of linear constraints
G ( x ) =
{ x (cid:48) | ∀ i : − (cid:15) ≤ ( x − x (cid:48) )
i ≤ (cid:15) } .
Evaluating Mean Minimum Adversarial Distortion.
Let d ( · , · ) denote a distance metric that measures the perceptual similarity between two input images.
The minimum adversarial distortion under d for input x with true label λ ( x ) corresponds to the solution to the optimization:
[Equation]
subject to
[Equation]
We can target the attack to generate an adversarial example that is classiﬁed in one of a set of target labels T by replacing Equation 4 with argmax i ( f i ( x (cid:48) ))
∈ T .
The most prevalent distance metrics in the literature for generating adversarial examples are the l(Carlini & Wagner, 2017b; Chen et al., 2018), l 2 (Szegedy et al., 2014), and l ∞ (Goodfellow et al., 2015; Papernot et al., 2016) norms.
All three can be expressed in the objective without adding any additional integer variables to the model (Boyd & Vandenberghe, 2004); details are in Appendix A.3.
4. F ORMULATING P IECEWISE - LINEAR F UNCTIONS IN THE MILP FRAMEWORK
Tight formulations of the ReLU and maximum functions are critical to good performance of the
MILP solver; we thus present these formulations in detail with accompanying proofs.
Formulating ReLU Let y = max( x, 0) , and l ≤ x ≤ u .
There are three possibilities for the phase of the ReLU.
If u ≤ 0 , we have y ≡ 0 .
We say that such a unit is stably inactive .
Similarly, if l ≥ 0 , we have y ≡ x .
We say that such a unit is stably active .
Otherwise, the unit is unstable .
For unstable units, we introduce an indicator decision variable a = 1 x ≥ 0 .
As we prove in Appendix A.1, y = max( x, 0) is equivalent to the set of linear and integer constraints in Equation 6.
[Equation]
Formulating the Maximum Function Let y = max( x 1 , x 2 , . . .
, x m ) , and l i ≤ x i ≤ u i .
Proposition 1 .
Let l max (cid:44) max( l 1 , l 2 , . . .
, l m ) .
We can eliminate from consideration all x i where u i ≤ l max , since we know that y ≥ l max ≥ u i ≥ x i .
We introduce an indicator decision variable a i for each of our input variables, where a i = 1 = ⇒ y
= x i .
Furthermore, we deﬁne u max, − i (cid:44) max j (cid:54) =
i ( u j ) .
As we prove in Appendix A.2, the constraint y = max( x 1 , x 2 , . . .
, x m ) is equivalent to the set of linear and integer constraints in Equation 7.
m (cid:94)
i
[Equation]
4. P ROGRESSIVE B OUNDS T IGHTENING
We previously assumed that we had some element-wise bounds on the inputs to non-linearities.
In practice, we have to carry out a presolve step to determine these bounds.
Determining tight bounds is critical for problem tractability: tight bounds strengthen the problem formulation and thus improve solve times (Vielma, 2015).
For instance, if we can prove that the phase of a ReLU is stable, we can avoid introducing a binary variable.
More generally, loose bounds on input to some unit will propagate downstream, leading to units in later layers having looser bounds.
We used two procedures to determine bounds: I NTERVAL A RITHMETIC ( IA ), also used in Cheng et al.
(2017); Dutta et al.
(2018), and the slower but tighter L INEAR P ROGRAMMING ( LP ) approach.
Implementation details are in Appendix B.
Since faster procedures achieve efﬁciency by compromising on tightness of bounds, we face a tradeoff between higher build times (to determine tighter bounds to inputs to non-linearities), and higher solve times (to solve the main MILP problem in Equation 2 or Equation 3-5).
While a degree of compromise is inevitable, our knowledge of the non-linearities used in our network allows us to reduce average build times without affecting the strength of the problem formulation.
The key observation is that, for piecewise-linear non-linearities, there are thresholds beyond which further reﬁning a bound will not improve the problem formulation.
With this in mind, we adopt a progressive bounds tightening approach: we begin by determining coarse bounds using fast procedures and only spend time reﬁning bounds using procedures with higher computational complexity if doing so could provide additional information to improve the problem formulation.
4 Pseudocode demonstrating how to efﬁciently determine bounds for the tightest possible formulations for the
ReLU and maximum function is provided below and in Appendix C respectively.
G ET B OUNDS F OR R E LU ( x, fs )
The process of progressive bounds tightening is naturally extensible to more procedures.
Kolter &
Wong (2017); Wong et al.
; Dvijotham et al.
; Weng et al.
each discuss procedures to determine bounds with computational complexity and tightness intermediate between IA and LP .
Using one of these procedures in addition to IA and LP has the potential to further reduce build times.
E XPERIMENTS
Dataset.
All experiments are carried out on classiﬁers for the MNIST dataset of handwritten digits or the CIFAR-10 dataset of color images.
Architectures.
We conduct experiments on a range of feed-forward networks.
In all cases, ReLUs follow each layer except the output layer.
MLP - m × [ n ] refers to a multilayer perceptron with m hidden layers and n units per hidden layer.
We further abbreviate MLP - 1 × [500] and MLP - 2 × [200] as
MLP A and MLP B respectively.
CNN A and CNN B refer to the small and large ConvNet architectures in
Wong et al.
CNN A has two convolutional layers (stride length 2) with 16 and 32 ﬁlters (size 4 × 4 ) respectively, followed by a fully-connected layer with 100 units.
CNN B has four convolutional layers with 32, 32, 64, and 64 ﬁlters, followed by two fully-connected layers with 512 units.
RES refers to the ResNet architecture used in Wong et al.
(2018), with 9 convolutional layers in four blocks, followed by two fully-connected layers with 4096 and 1000 units respectively.
Training Methods.
We conduct experiments on networks trained with a regular loss function and networks trained to be robust.
Networks trained to be robust are identiﬁed by a preﬁx corresponding to the method used to approximate the worst-case loss: LP d 5 when the dual of a linear program is used, as in Kolter & Wong (2017); SDP d when the dual of a semideﬁnite relaxation is used, as in
Raghunathan et al.
(2018); and Adv
when adversarial examples generated via Projected Gradient
Descent (PGD) are used, as in Madry et al.
Full details on each network are in Appendix D.1.
Experimental Setup.
We run experiments on a modest 8 CPUs@2.20 GHz with 8GB of RAM.
Appendix D.2 provides additional details about the computational environment.
Maximum build effort is LP .
Unless otherwise noted, we report a timeout if solve time for some input exceeds 1200s.
5. P ERFORMANCE C OMPARISONS 5.1. C OMPARISONS TO OTHER MILP- BASED C OMPLETE V ERIFIERS
Our MILP approach implements three key optimizations: we use progressive tightening , make use of the information provided by the restricted input domain G ( x ) , and use asymmetric bounds in the ReLU formulation in Equation 6.
None of the four other MILP-based complete veriﬁers implement progressive tightening or use the restricted input domain , and only Fischetti & Jo (2018) uses asymmetric bounds .
Since none of the four veriﬁers have publicly available code, we use ablation tests to provide an idea of the difference in performance between our veriﬁer and these existing ones.
When removing progressive tightening , we directly use LP rather than doing IA ﬁrst.
When removing using restricted input domain , we determine bounds under the assumption that our perturbed input could be anywhere in the full input domain X valid , imposing the constraint x (cid:
) ∈ G ( x ) only after all bounds are determined.
Finally, when removing using asymmetric bounds , we replace l and u in
Equation 6 with − M and M respectively, where M (cid:44) max( − l, u ) , as is done in Cheng et al.
Dutta et al.
; Lomuscio & Maganti (2017).
We carry out experiments on an MNIST classiﬁer; results are reported in Table 1.
[Table] [Table]
[Table] [Table]
The ablation tests demonstrate that each optimization is critical to the performance of our veriﬁer
In terms of performance comparisons we expect our veriﬁer to have a runtime several orders of magnitude faster than any of the three veriﬁers not using asymmetric bounds .
While Fischetti & Jo (2018) do use asymmetric bounds , they do not use information from the restricted input domain ; we thus expect our veriﬁer to have a runtime at least an order of magnitude faster than theirs
5.1. C OMPARISONS TO O THER C OMPLETE AND I NCOMPLETE V ERIFIERS
We also compared our veriﬁer to other veriﬁers on the task of ﬁnding minimum targeted adversarial distortions for MNIST test samples.
Veriﬁers included for comparison are 1) Reluplex (Katz et al., 2017), a complete veriﬁer also able to ﬁnd the true minimum distortion; and 2) LP 6 , Fast-Lip ,
Fast-Lin (Weng et al., 2018), and LP-full (Kolter & Wong, 2017), incomplete veriﬁers that provide a certiﬁed lower bound on the minimum distortion.
Veriﬁcation Times, vis-`a-vis the state-of-the-art SMT-based complete veriﬁer Reluplex .
Figure 1 presents average veriﬁcation times per sample.
All solves for our method were run to completion.
On the l ∞ norm, we improve on the speed of Reluplex by two to three orders of magnitude
Minimum Targeted Adversarial Distortions, vis-`a-vis incomplete veriﬁers.
Figure 2 compares lower bounds from the incomplete veriﬁers to the exact value we obtain.
The gap between the best lower bound and the true minimum adversarial distortion is signiﬁcant even on these small networks.
This corroborates the observation in Raghunathan et al.
(2018) that incomplete veriﬁers provide weak bounds if the network they are applied to is not optimized for that veriﬁer.
For example, under the l ∞ norm, the best certiﬁed lower bound is less than half of the true minimum distortion.
In context: a network robust to perturbations with l ∞ norm-bound (cid:15) = 0 .
1 would only be veriﬁable to (cid:15) = 0 .
5. D ETERMINING A DVERSARIAL A CCURACY OF MNIST AND CIFAR-10 C LASSIFIERS
We use our veriﬁer to determine the adversarial accuracy of classiﬁers trained by a range of robust training procedures on the MNIST and CIFAR-10 datasets.
Table 2 presents the test error and estimates of the adversarial error for these classiﬁers.
For MNIST, we veriﬁed a range of networks trained to be robust to attacks with bounded l ∞ norm (cid:15) = 0 .
1 , as well as networks trained to be robust to larger attacks of (cid:15) = 0 .
2 , 0 . 3 and 0 .
Lower bounds on the adversarial error are proven by providing adversarial examples for input that is not robust.
We compare the number of samples for which we successfully ﬁnd adversarial examples to the number for PGD, a strong ﬁrst-order attack.
Upper bounds on the adversarial error are proven by providing certiﬁcates of robustness for input that is robust.
We compare our upper bounds to the previous state-of-the-art for each network.
While performance depends on the training method and architecture, we improve on both the lower and upper bounds for every network tested.
8 For lower bounds, we successfully ﬁnd an adversarial example for every test sample that PGD ﬁnds an adversarial example for.
In addition, we observe that PGD ‘misses’ some valid adversarial examples: it fails to ﬁnd these adversarial examples even though they are within the norm bounds.
As the last three rows of Table 2 show, PGD misses for a larger fraction of test samples when (cid:15) is larger.
We also found that PGD is far more likely to miss for some test sample if the minimum adversarial distortion for that sample is close to (cid:15) ; this observation is discussed in more depth in Appendix G. For upper bounds, we improve on the bound on adversarial error even when the upper bound on the worst-case loss — which is used to generate the certiﬁcate of robustness — is explicitly optimized for during training (as is the case for LP d and SDP d training).
Our method also scales well to the more complex CIFAR-10 dataset and the larger LP d - RES network (which has 107,496 units), with the solver reaching the time limit for only 0.31% of samples.
Most importantly, we are able to determine the exact adversarial accuracy for Adv- MLP B and
LP d - CNN A for all (cid:15) tested, ﬁnding either a certiﬁcate of robustness or an adversarial example for every test sample.
For Adv- MLP B and LP d - CNN A , running our veriﬁer over the full test set takes approximately 10 hours on 8 CPUs — the same order of magnitude as the time to train each network on a single GPU.
Better still, veriﬁcation of individual samples is fully parallelizable — so veriﬁcation time can be reduced with more computational power.
[Table]
5.2. O BSERVATIONS ON D ETERMINANTS OF V ERIFICATION T IME
All else being equal
[Table]
we might expect veriﬁcation time to be correlated to the total number of ReLUs, since the solver may need to explore both possibilities for the phase of each ReLU.
However, there is clearly more at play: even though LP d - CNN A and Adv- CNN A have identical architectures, veriﬁcation time for Adv- CNN
A is two orders of magnitude higher.
[Table]
The key lies in the restricted input domain G ( x ) for each test sample x .
When input is restricted to
G ( x ) , we can prove that many ReLUs are stable (with respect to G ).
Furthermore, we can eliminate some labels from consideration by proving that the upper bound on the output neuron corresponding to that label is lower than the lower bound for some other output neuron.
As the results in Tableshow, a signiﬁcant number of ReLUs can be proven to be stable, and a signiﬁcant number of labels can be eliminated from consideration.
Rather than being correlated to the total number of ReLUs, solve times are instead more strongly correlated to the number of ReLUs that are not provably stable, as well as the number of labels that cannot be eliminated from consideration.
D ISCUSSION
This paper presents an efﬁcient complete veriﬁer for piecewise-linear neural networks.
While we have focused on evaluating networks on the class of perturbations they are designed to be robust to, deﬁning a class of perturbations that generates images perceptually similar to the original remains an important direction of research.
Our veriﬁer is able to handle new classes of perturbations (such as convolutions applied to the original image) as long as the set of perturbed images is a union of polytopes in the input space.
We close with ideas on improving veriﬁcation of neural networks.
First, our improvements can be combined with other optimizations in solving MILPs.
For example, Bunel et al.
(2018) discusses splitting on the input domain , producing two sub-MILPs where the input in each sub-MILP is restricted to be from a half of the input domain.
Splitting on the input domain could be particularly useful where the split selected tightens bounds sufﬁciently to signiﬁcantly reduce the number of unstable ReLUs that need to be considered in each sub-MILP.
Second, as previously discussed, taking advantage of locally stable ReLUs speeds up veriﬁcation; network veriﬁability could be improved during training via a regularizer that increases the number of locally stable ReLUs.
Finally, we observed (see Appendix H) that sparsifying weights promotes veriﬁability.
Adopting a principled sparsiﬁcation approach (for example, l 1 regularization during training, or pruning and retraining (Han et al., 2016)) could potentially further increase veriﬁability without compromising on the true adversarial accuracy.
A CKNOWLEDGMENTS
This work was supported by Lockheed Martin Corporation under award number RPP2016-002 and the NSF Graduate Research Fellowship under grant number 1122374.
We would like to thank
Eric Wong, Aditi Raghunathan, Jonathan Uesato, Huan Zhang and Tsui-Wei Weng for sharing the networks veriﬁed in this paper, and Guy Katz, Nicholas Carlini and Matthias Hein for discussing their results.
A F ORMULATING THE MILP MODEL A. F ORMULATING R E LU IN AN MILP M ODEL
We reproduce our formulation for the ReLU below.
[Equation]
We consider two cases.
Recall that a is the indicator variable a = 1 x ≥ 0 .
When a = 0 , the constraints in Equation 10 and 11 are binding, and together imply
that y =
The other two constraints are not binding, since Equation 9 is no stricter than Equation
when x < 0 , while Equation 8 is no stricter than Equation 10 since x − l ≥
We thus have a = 0 = ⇒ y
= 0 .
When a = 1 , the constraints in Equation 8 and 9 are binding, and together imply that
y
= x .
The other two constraints are not binding, since Equation 11 is no stricter than Equation 9 when x > 0 , while Equation 10 is no stricter than Equation 8 since x ≤ u .
We thus have a = 1 = ⇒ y
= x .
This formulation for rectiﬁed linearities is sharp (Vielma, 2015) if we have no further information about x .
This is the case since relaxing the integrality constraint on a leads to ( x, y ) being restricted to an area that is the convex hull of y = max( x, 0) .
However, if x is an afﬁne expression x = w T z + b , the formulation is no longer sharp, and we can add more constraints using bounds we have on z to improve the problem formulation.
A. F ORMULATING THE M AXIMUM F UNCTION IN AN MILP M ODEL
We reproduce our formulation for the maximum function below.
[Equation] [Equation]
[Equation] [Equation]
Equation 15 ensures that exactly one of the a i is 1.
It thus sufﬁces to consider the value of a i for a single variable.
When a i = 1 , Equations 13 and 14 are binding, and together imply that
y
= x i .
We thus have a i = 1 = ⇒ y
= x i .
When a i = 0 , we simply need to show that the constraints involving x i are never binding regardless of the values of x 1 , x 2 , . .
, x m .
Equation 14 is not binding since a i = 0 implies x i is not the (unique) maximum value.
Furthermore, we have chosen the coefﬁcient of a i such that Equation 13 is not binding, since x i + u max, − i − l i ≥ u max, − i ≥ y .
This completes our proof.
A. E XPRESSING l p NORMS AS THE O BJECTIVE OF AN MIP M ODEL A.3. l
When d ( x (cid:48) , x ) = (cid:107) x (cid:48) − x (cid:107)
, we introduce the auxiliary variable δ , which bounds the elementwise absolute value from above: δ j ≥ x (cid:48)
j − x j , δ j ≥ x j − x (cid:48)
j .
The optimization in Equation 3-5 is
equivalent to
[Equation]
A.3. l ∞
When d ( x (cid:48) , x ) = (cid:107) x (cid:48) − x (cid:107) ∞ , we introduce the auxiliary variable (cid:15) , which bounds the l ∞ norm from above: (cid:15) ≥ x (cid:48)
j − x j , (cid:15) ≥ x j − x (cid:48)
j .
The optimization in Equation 3-5 is equivalent to
[Equation]
22) subject to argmax
[Equation] [Equation]
[Equation] [Equation]
A.3. l
When d ( x (cid:48) , x ) = (cid:107) x (cid:48) − x (cid:107) 2 , the objective becomes quadratic, and we have to use a Mixed Integer
Quadratic Program (MIQP) solver.
However, no auxiliary variables are required: the optimization in
Equation 3-5 is simply equivalent to
[Equation] [Equation]
subject to
[Equation]
[Equation]
B D ETERMINING T IGHT B OUNDS ON D ECISION V ARIABLES
Our framework for determining bounds on decision variables is to view the neural network as a computation graph G .
Directed edges point from function input to output, and vertices represent variables.
Source vertices in G correspond to the input of the network, and sink vertices in G correspond to the output of the network.
The computation graph begins with deﬁned bounds on the input variables (based on the input domain ( G ( x ) ∩ X valid ) ), and is augmented with bounds on intermediate variables as we determine them.
The computation graph is acyclic for the feed-forward networks we consider.
Since the networks we consider are piecewise-linear, any subgraph of G can be expressed as an
MILP, with constraints derived from 1) input-output relationships along edges and 2) bounds on the values of the source nodes in the subgraph.
Integer constraints are added whenever edges describe a non-linear relationship.
We focus on computing an upper bound on some variable v ; computing lower bounds follows a similar process.
All the information required to determine the best possible bounds on v is contained in the subtree of G rooted at v , G v .
(Other variables that are not ancestors of v in the computation graph cannot affect its value.)
Maximizing the value of v in the MILP M v corresponding to G v gives the optimal upper bound on v .
We can reduce computation time in two ways.
Firstly, we can prune some edges and vertices of G v .
Speciﬁcally, we select a set of variables with existing bounds V I that we assume to be independent (that is, we assume that they each can take on any value independent of the value of the other variables in V I ).
We remove all in-edges to vertices in V I , and eliminate variables without children, resulting in the smaller computation graph G v,V I .
Maximizing the value of v in the MILP M v,V I corresponding to G v,V I gives a valid upper bound on v that is optimal if the independence assumption holds.
We can also reduce computation time by relaxing some of the integer constraints in M v to obtain a
MILP with fewer integer variables M (cid:48) v .
Relaxing an integer constraint corresponds to replacing the relevant non-linear relationship with its convex relaxation.
Again, the objective value returned by maximizing the value of v over M (cid:48) v may not be the optimal upper bound, but is guaranteed to be a valid bound.
B. FULL
FULL considers the full subtree G v and does not relax any integer constraints.
The upper and lower bound on v is determined by maximizing and minimizing the value of v in M v respectively.
FULL is also used in Cheng et al.
and Fischetti & Jo (2018).
If solves proceed to optimality, FULL is guaranteed to ﬁnd the optimal bounds on the value of a single variable v .
The trade-off is that, for deeper layers, using FULL can be relatively inefﬁcient, since solve times in the worst case are exponential in the number of binary variables in M v .
Nevertheless, contrary to what is asserted in Cheng et al.
, we can terminate solves early and still obtain useful bounds.
For example, to determine an upper bound on v , we set the objective of
M v to be to maximize the value of v .
As the solve process proceeds, we obtain progressively better certiﬁed upper bounds on the maximum value of v .
We can thus terminate the solve process and extract the best upper bound found at any time, using this upper bound as a valid (but possibly loose) bound on the value of v .
B. L INEAR P ROGRAMMING ( LP )
LP considers the full subtree G v but relaxes all integer constraints.
This results in the optimization problem becoming a linear program that can be solved more efﬁciently.
LP represents a good middle ground between the optimality of FULL and the performance of IA .
B. I NTERVAL A RITHMETIC ( IA )
IA selects
[Table]
to be the parents of v .
In other words, bounds on v are determined solely by considering the bounds on the variables in the previous layer.
We note that this is simply interval arithmetic
Moore et al.
Consider the example of computing bounds on the variable ˆ z i
=
W i
z i − 1 + b i , where l z i − 1 ≤
z
i − 1 ≤ u z i − 1 .
We have
[Equation] [Equation]
IA is efﬁcient (since it only involves matrix operations for our applications).
However, for deeper layers, using interval arithmetic can lead to overly conservative bounds.
C P ROGRESSIVE B OUNDS T IGHTENING
G ET B OUNDS F OR M AX ﬁnds the tightest bounds required for specifying the constraint
y = max( xs ) .
Using the observation in Proposition 1, we stop tightening the bounds on a variable if its maximum possible value is lower than the minimum value of some other variable.
G ET B OUNDS F OR M AX returns a tuple containing the set of elements in xs that can still take on the maximum value, as well as a dictionary of upper and lower bounds.
G ET B OUNDS F OR M AX ( xs , fs )
D A DDITIONAL E XPERIMENTAL D ETAILS D. N ETWORKS U SED
The source of the weights for each of the networks we present results for in the paper are provided below.
• MNIST classiﬁers not designed to be robust: – MLP - 2 × [20] and MLP - 3 × [20] are the MNIST classiﬁers in Weng et al.
(2018), and can be found at https://github.com/huanzhang12/ CertifiedReLURobustness .
• MNIST classiﬁers designed for robustness to perturbations with l ∞ norm-bound (cid:15) = 0 .
1 : – LP d - CNN B is the large MNIST classiﬁer for (cid:15)
= 0 .
1 in Wong et al.
(2018), and can be found at https://github.com/locuslab/convex_adversarial/ blob/master/models_scaled/mnist_large_0_1.pth .
– LP d - CNN A is the MNIST classiﬁer in Kolter & Wong (2017), and can be found at https://github.com/locuslab/convex_adversarial/ blob/master/models/mnist.pth .
– Adv- CNN A was trained with adversarial examples generated by PGD.
PGD attacks were carried out with l ∞ norm-bound (cid:15) = 0 .
1 , 8 steps per sample, and a step size of 0 . 334 .
An l 1 regularization term was added to the objective with a weight of 0 .on
the ﬁrst convolution layer and 0 .
for the remaining layers.
Adv- MLP - 2 × [200] was trained with adversarial examples generated by PGD.
PGD attacks were carried out with with l ∞ norm-bound (cid:15) = 0 .
15 , 200 steps per sample, and a step size of 0 . 1 .
An l 1 regularization term was added to the objective with a weight of 0 .
003 on the ﬁrst layer and 0 for the remaining layers.
– SDP d - MLP - 1 × [500] is the classiﬁer in Raghunathan et al.
• MNIST classiﬁers designed for robustness to perturbations with l ∞ norm-bound (cid:15) = 0 .
2 , 0 . 3 , 0 . 4 : – LP d - CNN A was trained with the code available at https://github.com/ locuslab/convex_adversarial at commit 4e9377f .
Parameters selected were batch size=20, starting epsilon=0.01, epochs=200, seed=0 .
– LP d - CNN B is the large MNIST classiﬁer for (cid:15)
= 0 .
3 in Wong et al.
(2018), and can be found at https://github.
• CIFAR-10 classiﬁers designed for robustness to perturbations with l ∞ norm-bound (cid:15) =– LP d - CNN A is the small CIFAR classiﬁer in Wong et al.
(2018), courtesy of the authors.
• CIFAR-10 classiﬁers designed for robustness to perturbations with l ∞ norm-bound (cid:15) =
– LP d - RES is the resnet CIFAR classiﬁer in Wong et al.
(2018), and can be found at https://github.com/locuslab/convex_adversarial/ blob/master/models_scaled/cifar_resnet_8px.pth .
D. C OMPUTATIONAL E NVIRONMENT
We construct the MILP models in Julia (Bezanson et al., 2017) using JuMP (Dunning et al., 2017), with the model solved by the commercial solver Gurobi 7.5.2 (Gurobi Optimization, 2017).
All experiments were run on a KVM virtual machine with 8 virtual CPUs running on shared hardware, with Intel(R) Xeon(R) CPU E5-2630 v4 @
2.20GHz processors, and 8GB of RAM.
E P ERFORMANCE OF V ERIFIER WITH O THER MILP S OLVERS
To give a sense for how our veriﬁer performs with other solvers, we ran a comparison with the Cbc (Forrest et al., 2018) and GLPK (Makhorin, 2012) solvers, two open-source MILP solvers.
[Table]
When we use GLPK as the solver, our performance is signiﬁcantly worse than when using Gurobi, with the solver timing out on almost 4% of samples.
While we time out on some samples with Cbc, our veriﬁer still provides a lower bound better than PGD and an upper bound signiﬁcantly better than the state-of-the-art for this network.
Overall, veriﬁer performance is affected by the underlying MILP solver used, but we are still able to improve on existing bounds using an open-source solver.
F A DDITIONAL S OLVE S TATISTICS F. N ODES E XPLORED
Table 5 presents solve statistics on nodes explored to supplement the results reported in Table 2.
If the solver explores zero nodes for a particular sample, it proved that the sample was robust (or found an adversarial example) without branching on any binary variables.
This occurs when the bounds we ﬁnd during the presolve step are sufﬁciently tight.
We note that this occurs for over 95% of samples for LP d - CNN A for (cid:15)
= 0 .
F. D ETERMINANTS OF V ERIFICATION T IME
Table 6 presents additional information on the determinants of veriﬁcation time for networks we omit in Table 3.
[Table]
- classiﬁers to perturbations with l ∞ norm-bound
[Table]
G W HICH A DVERSARIAL E XAMPLES ARE M ISSED BY PGD?
PGD succeeds in ﬁnding an adversarial example if and only if the starting point for the gradient descent is in the basin of attraction of some adversarial example.
Since PGD initializes the gradient descent with a randomly chosen starting point within G ( x ) ∩ X valid , the success rate (with a random start) corresponds to the fraction of G ( x ) ∩
X valid that is in the basin of attraction of adversarial
Intuitively, the success rate of PGD should be inversely related to the magnitude of the minimum adversarial distortion ˆ δ : if ˆ δ is small, we expect more of G ( x ) ∩ X valid to correspond to adversarial examples, and thus the union of the basins of attraction of the adversarial examples is likely to be larger.
We investigate here whether our intuition is substantiated.
To obtain the best possible empirical estimate of the success rate of PGD for each sample, we would need to re-run PGD initialized with multiple different randomly chosen starting points within
G ( x ) ∩ X valid .
However, since we are simply interested in the relationship between success rate and minimum adversarial distortion, we obtained a coarser estimate by binning the samples based on their minimum adversarial distortion, and then calculating the fraction of samples in each bin for which PGD with a single randomly chosen starting point succeeds at ﬁnding an adversarial example.
Figure 3 plots this relationship for four networks using the CNN A architecture and trained with the same training method LP d but optimized for attacks of different size.
Three features are clearly discernible:
• PGD is very successful at ﬁnding adversarial examples when the magnitude of the minimum adversarial distortion, ˆ δ , is small.
• The success rate of PGD declines signiﬁcantly for all networks as ˆ δ approaches (cid:15) .
• For a given value of ˆ δ , and two networks a and b trained to be robust to attacks with l ∞ norm-bound (cid:15) a and (cid:15) b respectively (where (cid:15) a < (cid:15) b ), PGD is consistently more successful at attacking the network trained to be robust to smaller attacks, a , as long as ˆ δ (cid:28) (cid:15) a .
The sharp decline in the success rate of PGD as ˆ δ approaches (cid:15) is particularly interesting, especially since it is suggests a pathway to generating networks that appear robust when subject to PGD attacks of bounded l ∞ norm but are in fact vulnerable to such bounded attacks: we simply train the network to maximize the total number of adversarial examples with minimum adversarial distortion close to (cid:15) .
H S PARSIFICATION AND V ERIFIABILITY
When verifying the robustness of SDP d - MLP A , we observed that a signiﬁcant proportion of kernel weights were close to zero.
Many of these tiny weights are unlikely to be contributing signiﬁcantly to the ﬁnal classiﬁcation of any input image.
Having said that, setting these tiny weights to zero could potentially reduce veriﬁcation time, by 1) reducing the size of the MILP formulation, and by 2) ameliorating numerical issues caused by the large range of numerical coefﬁcients in the network (Gurobi, 2017).
We generated sparse versions of the original network to study the impact of sparseness on solve times.
Our heuristic sparsiﬁcation algorithm is as follows: for each fully-connected layer i , we set a fraction f
i of the weights with smallest absolute value in the kernel to 0, and rescale the rest of the weights such that the l 1 norm of the kernel remains the same.
9 Note that MLP A consists of only two layers: one hidden layer (layer 1 ) and one output layer (layer 2 ).
[Table]
Table 7 summarizes the results of verifying sparse versions of SDP d - MLP A ; the ﬁrst row presents results for the original network
[Table]
and the subsequent rows present results when more and more of the kernel weights are set to zero.
When comparing the ﬁrst and last rows
[Table]
we observe an improvement in both mean time and fraction timed out by an order of magnitude.
As expected
[Table]
sparsifying weights increases the test error
[Table]
but the impact is not signiﬁcant until f 1 exceeds 0.8.
We also ﬁnd that sparsiﬁcation signiﬁcantly improves our upper bound on adversarial error — to a point: the upper bound on adversarial error for f 1 = 0 .is higher than that for f 1 = 0 .
8 , likely because the true adversarial error has increased signiﬁcantly.
or test errors.
Starting with a network that is robust, we have demonstrated that a simple sparsiﬁcation approach can already generate a sparsiﬁed network with an upper bound on adversarial error signiﬁcantly lower than the best upper bound that can be determined for the original network.
Adopting a more principled sparsiﬁcation approach could achieve the same improvement in veriﬁability but without compromising on the true adversarial error as much.
I R OBUST T RAINING AND R E LU STABILITY
Networks that are designed to be robust need to balance two competing objectives.
Locally, they need to be robust to small perturbations to the input.
However, they also need to retain sufﬁcient global expressiveness to maintain a low test error.
For the networks in Table 3, even though each robust training approach estimates the worst-case error very differently, all approaches lead to a signiﬁcant fraction of the ReLUs in the network being provably stable with respect to perturbations with bounded l ∞ norm.
In other words, for the input domain G ( x ) consisting of all bounded perturbations of the sample x , we can show that, for many
ReLUs, the input to the unit is always positive (and thus the output is linear in the input) or always negative (and thus the output is always zero).
As discussed in the main text, we believe that the need for the network to be robust to perturbations in G drives more ReLUs to be provably stable with respect to G .
To better understand how networks can retain global expressiveness even as many ReLUs are provably stable with respect to perturbations with bounded l ∞ norm (cid:15) , we study how the number of ReLUs that are provably stable changes as we vary the size of G ( x ) by changing the maximum allowable l ∞ norm of perturbations.
The results are presented in Figure 4.
As expected, the number of ReLUs that cannot be proven to be stable increases as the maximum allowable l ∞ norm of perturbations increases.
More interestingly, LP d - CNN A is very sensitive to the (cid:15)
= 0 .
1 threshold, with a sharp increase in the number of ReLUs that cannot be proven to be stable when the maximum allowable l ∞ norm of perturbations increases beyond 0.102.
An increase of the same abruptness is not seen for the other two networks.
[Figure]
The results reported in Table 3 are marked by a dotted line.
As we increase the maximum allowable l ∞ norm of perturbations, the number of ReLUs that cannot be proven to be stable increases across all networks (as expected), but LP d - CNN A is far more sensitive to the (cid:15)
=
threshold