General Cutting Planes for Bound-Propagation-Based Neural Network Verification
Anonymous Author(s)
AffiliationAddress email
Abstract
Bound propagation methods, when combined with branch and bound, are amongthe most effective methods to formally verify properties of deep neural networkssuch as correctness, robustness, and safety.
However, existing works cannot handlethe general form of cutting plane constraints widely accepted in traditional solvers,which are crucial for strengthening verifiers with tightened convex relaxations.
Inthis paper, we generalize the bound propagation procedure to allow the additionof arbitrary cutting plane constraints, including those involving relaxed integervariables that do not appear in existing bound propagation formulations.
Ourgeneralized bound propagation method, GCP-CROWN, opens up the opportunityto apply g eneral c utting p lane methods for neural network verification while benefiting from the efficiency and GPU acceleration of bound propagation methods.
As a case study, we investigate the use of cutting planes generated by off-the-shelfmixed integer programming (MIP) solver.
We find that MIP solvers can generatehigh-quality cutting planes for strengthening bound-propagation-based verifiersusing our new formulation.
Since the branching-focused bound propagation procedure and the cutting-plane-focused MIP solver can run in parallel utilizing differenttypes of hardware (GPUs and CPUs), their combination can quickly explore alarge number of branches with strong cutting planes, leading to strong verificationperformance.
Experiments demonstrate that our method is the first verifier that cancompletely solve the oval20 benchmark, and can verify twice as many instances onthe oval21 benchmark compared to the best tool in VNN-COMP 2021, and alsonoticeably outperforms state-of-the-art verifiers on a wide range of benchmarks.
Introduction
Neural network (NN) verification aims to formally prove or disprove certain properties (e g., correctness and safety properties) of a NN under a certain set of inputs.
These methods can provideworst-case performance guarantees of a NN, and have been applied to mission-critical applications thatinvolve neural networks, such as automatic aircraft control [31, 4], learning-enabled cyber-physicalsystems [54], and NN based algorithms in an operating system [51].
The NN verification problem is generally NP-complete [30].
For piece-wise linear networks, it canbe encoded as a mixed integer programming (MIP) [53] problem with the non-linear ReLU neuronsdescribed by binary variables.
Thus, fundamentally, the NN verification problem can be solved usingthe branch and bound (BaB) [10] method similar to generic MIP solvers, by branching some binaryvariables and relaxing the rest into a convex problem such as linear programming (LP) to obtainbounds on the objective.
Although early neural network verifiers relied on off-the-shelf CPU-based
LP solvers [36, 9] for bounding in BaB, LP solvers do not scale well to large NNs.
Thus, many recent verifiers are instead based on efficient and GPU-accelerated algorithms customized to NNverification, such as bound propagation methods [60, 57], Lagrangian decomposition methods [8, 17] and others [16, 11].
Bound propagation methods, presented in a few different formulations [58,18, 56, 61, 50, 25], empower state-of-the-art NN verifiers such as α,β -CROWN [61, 60, 57] and
VeriNet [3], and can achieve two to three orders of magnitudes speedup compared to solving the NNverification problem using an off-the-shelf solver directly [57], especially on large networks.
Despite the success of existing NN verifiers, we experimentally find that state-of-the-art NN verifiersmay timeout on certain hard instances which a generic MIP solver can solve relatively quickly,sometimes even without branching.
Compared to an MIP solver, a crucial factor missing in mostscalable NN verifiers is the ability to efficiently generate and solve general cutting planes (or “cuts”).
In generic MIP solvers, cutting planes are essential to strengthen the convex relaxation, so thatmuch less branching is required.
Advanced cutting planes are among the most important factorsin modern MIP solvers [6]; they can strengthen the convex relaxation without removing any validinteger solution from the MIP formulation.
In the setting of NN verification, cutting planes reflectscomplex intra-layer and inter-layer dependencies between multiple neurons, which cannot be easilycaptured by existing bound propagation methods with single neuron relaxations [45].
This motivatesus to seek the combination of efficient bound propagation method with effective cutting planes tofurther increase the power of NN verifiers.
A few key factors make the inclusion of general cutting planes in NN verifiers quite challenging.
First,existing efficient bound propagation frameworks such as CROWN [61] and β -CROWN [57] cannotsolve general cutting plane constraints that may involve variables across different layers in the MIPformulation.
Particularly, these frameworks do not explicitly include the integer variables in the MIPformulation that are crucial when encoding many classical strong cutting planes, such as Gomorycuts and mixed integer rounding (MIR) cuts.
Furthermore, although some existing works [47, 52, 40]enhanced the basic convex relaxation used in NN verification (such as the Planet relaxation [19]),these enhanced relaxations involve only one or a few neurons in a single layer or two adjacent layers,and are not general enough.
In addition, an LP solver is often required to handle these additionalcutting plane constraints [40], for which the efficient and GPU-accelerated bound propagation cannotbe used, so the use of these tighter relaxations may not always bring improvements.
In this paper, we achieve major progress in using general cutting planes in bound propagation based
NN verifiers.
To mitigate the challenge of efficiently solving general cuts, our first contribution isto generalize existing bound propagation methods to their most general form, enabling constraintsinvolving variables from neurons of any layer as well as integer variables that encode the status of a
ReLU neuron.
This allows us to consider any cuts during bound propagation without relying on a slow
LP solver, and opens up the opportunity for using advanced cutting plane techniques efficiently forthe NN verification problem.
Our second contribution involves combining a cutting-plane-focused,off-the-shelf MIP solver with our GPU-accelerated, branching-focused bound propagation methodcapable of handling general cuts.
We entirely disable branching in the MIP solver and use it only forgenerating high quality cutting planes not restricting to neurons within adjacent layers.
Although an
MIP solver often cannot verify large neural networks, we find that they can generate high qualitycutting planes within a short time, significantly helping bound propagation to achieve better bounds.
Our experiments show that general cutting planes can bring significant improvements to NN verifiers:we are the first verifier that completely solves all instances in the oval20 benchmark, with anaverage time of only 13 seconds per instance; on the even harder oval21 benchmark in VNNCOMP 2021 [3], we can verify twice as many instances compared to the competition winner.
Wealso outperform existing state-of-the-art bound propagation based methods including those usingmulti-neuron relaxations (a limited form of cutting planes).
Background
The NN verification problem We consider the verification problem for an L -layer ReLU-based
Neural Network (NN) with inputs ˆ x (0) :
= x ∈ R d 0 , weights W ( i ) ∈
R d i × d i − 1 , and biasesb ( i ) ∈
R d i ( i ∈ { 1 , · · · , L } ).
We can get the NN outputs f ( x ) = x ( L ) ∈ R d L by sequentiallypropagating the input x through affine layers with x ( i ) =
W ( i ) ˆ x
( i − 1)
+ b ( i ) and ReLU layer withˆ x
( i ) =
ReLU( x ( i ) ) .
We also let scalars ˆ x ( i ) j and x ( i ) j denote the post-activation and pre-activation,respectively, of j -th ReLU neuron in i -th layer.
Throughout the paper, we use bold symbols to denotevectors (e.g., x ( i ) ) and regular symbols to denote scalars (e.g., x ( i )
j is the j -th element of x ( i ) ).
Weuse the shorthand [ N ] to denote { 1 , · · · , N } , and W ( i ) : ,j is the j -th column of W ( i ) .
Commonly, the input x is bounded within a perturbation set C (such as an ℓ p norm ball) and theverification specification defines a property of the output f ( x ) that should hold for any x ∈ C ,e.g., whether the true label’s logit f y ( x ) will be always larger than another label’s logit f j ( x ) , (i.e.,checking if f y ( x ) − f j ( x ) is always positive).
Since we can append the verification specification (suchas a linear function on neural network output) as an additional layer of the network, canonically, the
NN verification problem requires one to solve the following one-dimensional ( d L = 1 ) optimizationobjective on f ( x ) :
[Equation]
with the relevant property defined to be proven if the optimal solution f ∗ ≥ 0 .
Throughout this work,we consider the ℓ ∞ norm ball C :=
{ x : ∥ x − x 0 ∥ ∞ ≤ ϵ } where x 0 is a predefined constant (e.g., aclean input image), although it is possible to extend to other norms or specifications [43, 59].
The MIP and LP formulation for NN verification
The mixed integer programming (MIP)formulation is the root of many NN verification algorithms.
This formulation uses binary variables
zto encode the non-linear ReLU neurons to make the non-convex optimization problem (1) tractable.
Additionally, we assume that we know sound pre-activation bounds l ( i ) ≤ x ( i ) ≤ u ( i ) for x ∈ Cwhich can be obtained via cheap bound propagation methods such as IBP [23] or CROWN [61].
Then ReLU neurons for each layer i can be classified into three classes [58], namely “active” ( I +( i ) ),“inactive” ( I − ( i ) ) and “unstable” ( I ( i ) ) neurons, respectively:
[Equation]
Based on the definition of ReLU, activate and inactive neurons are linear functions, so only unstableneurons require binary encoding.
The MIP formulation of (1) is:
[Equation]
Since a MIP problem is slow or intractable to solve, it is commonly relaxed as a When the integervariables are relaxed to continuous ones, we obtain the LP relaxation of NN verification problem:
[Equation]
The ReLU constraints involving z is often projected out, leading to the well-known Planet relaxationused in many NN verifiers, replacing (6), (7) and (8) with a single constraint to get an equivalent LP:
[Equation]
[Equation] [Equation]
Due to the relaxations, the objective of the LP formulation is always a lower bound of the MIPformulation: f ∗ LP ≤ f ∗ .
A verifier using this formulation is incomplete: if f ∗ LP ≥ 0 , then f ∗ ≥ 0and the property is verified; otherwise, we cannot conclude the sign of f ∗ so the verifier must return“unknown”.
Branch and bound can be used to improve the lower bound and achieve completeness [10,57].
However, in this paper, we work on an orthogonal direction of strengthening the LP formulationby adding cutting planes to obtain larger bounds.
Bound propagation methods Instead of solving the LP formulation directly using a LP solver,bound propagation methods aims to quickly give a lower bound for f ∗ LP .
For example, CROWN [61]and β -CROWN [57] propagate a sound linear lower bound backwards for f L ( x ) with respect to eachintermediate layer.
For example, suppose we know
[Equation]
With i =
L − 1 the above is trivially hold with a ( L − 1) =
W ( L ) , c
( L − 1)
=
b ( L ) .
In boundpropagation methods, a propagation rule propagates an inequality (13) through a previous layerˆ x
( i ) :
=
ReLU( W ( i ) ˆ x ( i − 1)
+ b ( i ) ) to obtain a sound inequality with respect to ˆ x
( i − 1) :
[Equation]
Here a ( i − 1) , c ( i − 1) can be calculated in close-form via a ( i ) , c ( i ) , W ( i ) , b ( i ) , l ( i ) and u ( i ) such thatthe bound still holds (see Lemma 1 in [57]).
Applying the procedure repeatedly will eventually reachthe input layer:
[Equation]
The minimization on linear layer can be solved easily when C is a ℓ p norm ball to obtain a valid lowerbound of f ∗ .
Since the bounds propagate layer-by-layer, this process can be implemented efficientlyon GPUs [59] without relying on a slow LP solver, which greatly improves the scalability and solvingtime.
Additionally, it is often used to obtain intermediate layer bounds l ( i ) and and u ( i ) required forthe MIP formulation (6)(7), by treating each x ( i ) j as the output neuron.
The bound propagation rulecan either be derived in primal space [61], dual space [58] or abstract interpretations [50].
In Sec.3.1,we will discuss the our bound propagation procedure with general cutting plane constraints.
Neural Network Verification with General Cutting Planes
3.1 GCP-CROWN: General Cutting Planes in Bound Propagation
In this section, we generalize existing bound propagation method to handle general cutting planeconstraints.
Our goal is to derive a bound propagation rule similar to CROWN and β -CROWNdiscussed in Section 2, however considering additional constraints among any variables within the
LP relaxation.
To achieve this, we first derive the dual problem of the LP; inspired by the dualformulation, we derive the bound prorogation rule in a layer by layer manner that takes all cuttingplane constraints into consideration.
The derivation process is inspired by [58, 45] and [57].
LP relaxation with cutting planes.
In this section, we derive the bound propagation procedureunder the presence of general cutting plane constraints.
A cutting plane is a constraint involving anyvariables x ( i ) (pre-activation), ˆ x ( i ) (post-activation), z ( i ) (ReLU indicators) from any layer i :
[Equation]
Here h ( i ) , g ( i ) and q ( i ) are coefficients for this cut constraint.
The difference between a validcutting plane and an arbitrary constraint is that a valid cutting plane should not remove any validinteger solution from the MIP formulation.
Our new bound propagation procedure can work for anyconstraints, although in this work we focus on studying the impacts of cutting planes.
When there are
N cutting planes, we write them in a matrix form:
[Equation]
(15) where H ( i ) , G ( i ) , Q ( i
) ∈
R N × d i .
The LP relaxation with all cutting planes is:
[Equation]
Since an additional constraint is added, f ∗ LP-cut ≥ f ∗ LP
and we get closer to f ∗ .
Unlike the original LPwhere each constraint only contains variables from two consecutive layers, our general cutting planeconstraint may involve any variable from any layer in a single constraint.
The dual problem with cutting planes We first show the dual problem for the above LP.
The dualproblem we consider here is different from existing works in two ways: first, we have constraintswith integer variables to support potential cutting planes on z .
Additionally, we have cutting planesconstraints that may involve variables in any layer, so the dual in previous works such as [58] cannotbe directly reused.
Our dual problem is given below (derivation details in Appendix A):
[Equation]
s
[Equation]
L
= − 1; and for each i ∈ [
L
−
[Equation]
and for each j ∈ I ( i ) the two equalities below hold:
[Equation]
Instead of solving the dual problem exactly, we use it to obtain a lower bound of f ∗ LP-cut .
Intuitively,due to the definition of due problem, any valid setting of dual variables leads to a lower bound off ∗ LP-cut .
Informally, starting from ν ( L ) = − 1 , by applying the constraints in this dual formulation,we can compute ν
( L − 1) , ν ( L − 2) , · · · until ν (1) .
The final objective is a function of ν ( i ) , i ∈
[ N ] andother dual variables.
Precisely, our GCP-CROWN bound propagation procedure with general cuttingplane constraint is presented in the theorem below (proof in Appendix A):
Theorem 3.1 (Bound propagation with general cutting planes) .
Given any optimizable parameters0 ≤ α ( i ) j ≤ 1 and β ≥ 0 , f ∗ LP-cut is lower bounded by the following objective function:
[Equation]
where variables ν
( i ) are obtained by propagating ν ( L ) =
− 1 throughout all i ∈
[ L − 1] :
[Equation]
Here ˆ ν
( i ) j , π ( i ) j ∗ and h ( i ) j
( β ) are defined for each unstable neuron j ∈
I ( i ) as:
[Equation]
i
[Equation] [Equation]
Based on Theorem 3.1, to obtain an lower bound of f ∗ LP-cut , we start with any valid setting of0 ≤ α ≤ 1 and β ≥ 0 and ν ( L ) = − 1 .
According to the bound propagation rule, we can compute each ν ( i ) , i ∈
[ L − 1] , in a layer by layer manner.
Then objective g ( α , β ) can be evaluated based onall ν ( i ) to give an lower bound of f ∗ LP-cut .
Since any valid setting of α and β lead to a valid lowerbound, we can optimize α and β using gradient ascent in a similar manner as in [60, 57] to tightenthis lower bound.
The entire procedure can also run on GPU for great acceleration.
Connection to Convex Outer Adversarial Polytope
In convex outer adversarial polytope [58],a bound propagation rule was developed in a similar manner in the dual space without consideringcutting plane constraints, and is a special case of ours.
We denote their bound propagation objectivefunction as g WK which also contains optimizable parameters α WK .
Proposition 3.2.
Given the same input x , perturbation set C , network weights, and N cutting planeconstraints,
[Equation]
Proof.
In Theorem 3.1, when all β are set to 0, then π ( i ) j
[Equation]
we recover exactly the same bound propagation equations as in [58].
However, since we allow theaddition of cutting plane methods and we can maximize over the additional parameter β , the objectivegiven by our bound propagation is always at least as good as g KW .
Connection to CROWN-like bound propagation methods CROWN [61] and α -CROWN
[60]use the same bound propagation rule as [58] so Proposition 3.2 also applies, although they werederived from primal space without explicitly formulating the problem as a LP.
Salman et al.
[45]showed that many other bound propagation methods [50, 55] are equivalent to or weaker than [58].
Recently, Wang et al.
[57] extends CROWN to β -CROWN to handle split constraints (e.g., x ( i ) j ≥ 0 ).
It can be seen a special case as GCP-CROWN where all H , G and Q matrices are zeros except:
H ( i ) j,j = 1 ,j ∈
I − ( i ) for x ( i ) j ≤ 0 split ;
H ( i ) j,j = − 1 ,j ∈
I
+( i ) for x ( i ) j ≥ 0 split
In addition, [20] encodes multi-neuron relaxations using sparse H ( i ) and G ( i ) and each cut containsa small number of neurons involving x ( i ) and ˆ x ( i ) for the same layer i .
Wang et al.
[57] derivedbound propagation rules from both the dual LP and the primal space with a Lagrangian without LP.
However, in our case, it is not intuitive to derive bound propagation without LP due to the potentialcutting planes on relaxed integer variables z , which do not appear without the explicit LP formulation.
Furthermore, although we derived cutting planes for bound propagation methods, it is technicallyalso possible to derive them using other bounding frameworks such as Lagrangian decomposition [8].
3.2 Branch-and-bound with GCP-CROWN and MIP Solver Generated Cuts
[Figure]
To build a complete NN verifier, we follow the popular
[Figure]
branch-and-bound (BaB) procedure [10, 9] in state-of-the-
[Figure]
art NN verifiers with GPU accelerated bound propagationmethod [8, 60, 16, 57], and our GCP-CROWN is used asthe bounding procedure in BaB.
We refer the readers to Ap-
[Figure]
pendix B for a more detailed background on branch-andbound.
Having the efficient bound propagation procedurewith general cutting plane constraints, we now need to finda good set of general cutting planes H ( i ) , G ( i ) , Q ( i ) to accelerate NN verification.
Since GCP-CROWN can adoptany cutting planes, to fully exploit its power, we proposeto use off-the-shelf MIP solvers to generate cutting planesand create an NN verifier combining GPU-acceleratedbound propagation with strong cuts generated by a MIPsolver.
We make the bound propagation on GPU andthe MIP solver on CPU run in parallel with cuts addedon-the-fly, so the original strong performance of boundpropagation-based NN verifier will never be affected bya potentially slow MIP solver.
This allows us to make fulluse of available computing resource (GPU + CPU).
The architecture of our verifier is shown in Fig 1.
MIP solvers for cutting plane generation Generic MIP solvers such as cplex [28] and gurobi [24]also apply a branch-and-bound strategy, conceptually similar to state-of-the-art NN verifiers.
Theyoften tend to be slower than specialized NN verifiers because MIP solvers rely on slower boundingprocedures (e.g., Simplex or barrier method) and cannot apply an GPU-accelerated method such asbound propagation or Lagrangian decomposition.
However, we still find that MIP solvers are a strongbaseline when combined with tight intermediate layer bounds .
For example, in the oval21 benchmarkin Table 2, α -CROWN+MIP (MIP solver combined with tight intermediate layer bound computed byα -CROWN) is able to solve 4 more instances compared to all other tools in the competition.
Ourinvestigation found that α -CROWN+MIP explores much less branches than other state-of-the-artbranch-and-bound based NN verifiers, however before branching starts, the MIP solver produces veryeffective cutting planes that can often verify an instance with little branching.
MIP solvers is ableto discover sophisticated cutting planes involving several NN layers reflecting complex correlationsamong neurons, while existing NN verifiers only exploit specific forms of constraints within samelayer or between adjacent layers [52, 47, 40].
This motivates us to combine the strong cutting planesgenerated by MIP solvers with GPU-accelerated bound-propagation-based BaB.
In this work, we use cplex as the MIP solver for cutting plane generation since gurobi cannotexport cuts.
We entirely disable all branching features in cplex and make it focus on finding cuttingplanes only.
These cutting planes are generated only for the root node of the BaB search tree so theyare sound for any subdomains with neuron splits in BaB. We conduct branching using our generalizedbound propagation procedure in Section (3.1) with the cutting planes generated by cplex .
Fully-parallelized NN verifier design We design our verifier as shown in Figure 1.
After parsingthe NN under verification, we launch a separate process to encode the NN verification problem as
MIP problem and start multiple MIP solvers, one for each target label to be verified.
At the same time,the main NN verifier process executes branch-and-bound without waiting for the MIP solver process.
In each iteration of branch and bound, we query the MIP solving processes and fetch any newlygenerated cutting planes.
If any cutting planes are produced, they are added as H ( i ) , G ( i ) , Q ( i ) in
GCP-CROWN and tighten the bounds for subsequent branching and bounding.
If no cutting planesare produces, GCP-CROWN reduces to β -CROWN [57].
Since our verifier is based on strengtheningthe bounds in β -CROWN with sound cutting planes, we are also sound and complete.
Adjustments to existing branch and bound method.
We implement GCP-CROWN into theα,β -CROWN verifier [61, 59, 57], the winning verifier in VNN-COMP 2021 [3], as the backbonefor BaB with bound propagation.
To better exploit the power of cutting planes under a fully paralleland asynchronous design, we made a key changes to the BaB procedure.
When the number of
BaB subdomains are greater than batch size, we rank the subdomains by their lower bounds andchoose the easiest domains with largest lower bounds first to verify with GCP-CROWN, unlike mostexisting verifiers which solve the worst domains first.
We use such a order because the MIP solvergenerates cutting planes incrementally.
Solving these easier subdomains tend to require no or fewercutting planes, so we solve them at earlier stages where cutting planes have not been generated or arerelatively weak.
On the other hand, if we split worst subdomains first, the number of subdomains willgrow quickly, and it can take a long time to verify these domains when stronger cuts become availablelater.
Under a similar rationale, when verifying a multi-class model and BaB needs to verify eachtarget class label one by one, we start BaB from the easiest label first ( α -CROWN bound closest to0), allowing the MIP solver to run longer for harder labels, generating stronger cuts for harder labels.
Experiments
We now evaluate our verifier, GCP-CROWN with MIP cuts, on a few popular verification benchmarks.
Since our verifier uses a MIP solver in our pipeline, we also include a new baseline, α -CROWN+MIP,which uses gurobi as the MIP solver with the tightest possible intermediate layer bounds fromα -CROWN [59].
We use the same branch and bound algorithm as in β -CROWN
and we use filteredsmart branching (FSB) [16] as the branching heuristic in all experiments.
Without cutting planes,
GCP-CROWN becomes vanilla β -CROWN as we share the same code base as β -CROWN.
Weinclude detailed setup for our experiments in Appendix C.
Results on the oval20 benchmark in VNN-COMP 2020.
oval20 is a popular benchmarkconsistently used in huge amount of NN verifiers and it perfectly reflects the progress of NN verifiers.
[Table]
0.00 ERAN ∗ [47, 48, 50, 49] 805.
- 0.00 β -CROWN [57] 118.
0.00 α -CROWN+MIP† 335.
[Figure]
with MIP cuts 13.
papers.
[Figure]
We include literature results for many baselines in Table 1.
We are the only verifier that can completelysolve all three models without any timeout.
Our average runtime is significantly lower compared tothe time of baselines because we have no timeout (counted as 3600s), and our slowest instance onlytakes about a few minutes while easy ones only take a few seconds, as shown in Figure 2.
Additionally,we often use less number of branches compared to the state-of-the-art verifier, β -CROWN, since ourstrong cutting planes help us to eliminate many hard to solve subdomains in BaB. Furthermore, wehighlight that α -CROWN+MIP also achieves a low timeout rate, although it is much slower than ourbound propagation based approach combined with cuts from a MIP solver.
Results on VNN-COMP 2021 benchmarks.
Among the eight scoring benchmarks in VNNCOMP 2021 [3], only two ( oval21 and cifar10-resnet ) are most suitable for the evaluation ofthis work.
Among other benchmarks, acasxu and nn4sys have low input dimensionality and requireinput space branching rather than ReLU branching; verivital , mnistfc , and eran benchmarksconsist of small MLP networks that can be solved directly by MIP solvers; marabou contains mostlyadversarial examples, making it a good benchmark for falsifiers rather than verifiers.
We present ourresults in Table 2.
Besides results from 6 VNN-COMP participants, we also include two additionalbaselines, α -CROWN+MIP (same as in Table 1), and MN-BaB [20], a recently proposed branch andbound framework with multi-neuron relaxations [39], which can be viewed as a restricted form ofcutting planes.
On the oval21 benchmark, OVAL, VeriNet and α,β -CROWN are the best performingtools, verified 11 out of 27 instances, while we can verify twice more instances (22 out of 27) onthis benchmark.
On the cifar10-resnet benchmark, our verifier also solves the most number ofinstances and achieves the lowest average time.
In fact, α -CROWN+MIP is also a strong baseline,solving 4 more instances than all competition participants, showing the importance of strong cuttingplanes.
Our GCP-CROWN with MIP cuts combines the benefits of fast bound propagation on GPUwith the strong cutting planes generated by a MIP solver and achieves the best performance.
Results on SDP-FO benchmarks.
We further evaluate our method on the SDP-FO benchmarksin [15, 57].
This benchmark contains 7 mostly adversarially trained MNIST and CIFAR models with200 instances each, which are hard for many existing verifiers.
Beyond the baselines reported in [57],we also include two additional baselines, α -CROWN+MIP (same as in Table 1) and MN-BaB [20](same as in Table 2).
Table 3 shows that our method improves the percentage of verified images(“verified accuracy”) on all models compared to state-of-the-art verifiers, further closing the gapbetween verified accuracy and empirical robust accuracy obtained by PGD attack (reported as “upperbound” in Table 3).
[Table]
[Table] [Table]
Related Work
Cutting plane method is a classic technique to strengthen the convex relaxation of an integer programming problem.
Generic cutting planes such as Gomory’s cut [21, 22], Chvátal–Gomory cut [12],implied bound cut [27], lift-and-project [35], reformulation-linearization techniques [46] and mixedinteger rounding cuts [41, 37] can be applied to almost any LP relaxed problems, and problem specificcutting planes such as Knapsack cut [14], Flow-cover cut [42] and Clique cut [29] require specificproblem structures.
Modern MIP solvers typically uses a branch-and-cut strategy, which tends togenerate a large number of cuts before starting the next iteration of branching, and solve the LPrelaxation of the MIP problem with cutting planes with an exact method such as the Simplex method.
Our GCP-CROWN is a specialized solver for the NN verification problem, which can quickly obtaina lower bound of the LP relaxation with cutting planes specially for the NN verification problem.
The verification of piece-wise linear NNs can be formulated as a MIP problem, so early works [30, 31,53] solve an integer or combinatorial formulation directly.
For efficiency reasons, most recent worksuse a convex relaxation such as linear relaxation [19, 58] or semidefinite relaxation [44, 15].
Salmanet al.
[45] discussed the limitation of many convex relaxation based NN verification algorithmsand coined the term “convex relaxation barrier”, specifically for the popular single-neuron “Planet”relaxation [19].
Several works developed novel techniques to break this barrier.
[47] added constraintsthat depends on the aggregation of multiple neurons, and these constraints were passed to a LP solver.[40] enhanced the multi-neuron formulation of [47] to obtain tighter relaxations.
[1] studied strongerconvex relaxations for a ReLU neuron after an affine layer, and [52] constructed efficient algorithmsbased on this relaxation for incomplete verification.
[16] extended the formulation in [1] to a dualform and combined it with branch and bound to achieve completeness.
[7] proposed specialized cutsconsidering neuron dependencies and solve them using a MIP solver.
[20] combined the multi-neuronrelaxation [40] with branch and bound.
Although these works can be seen as a special form of cuttingplanes, they mostly focused on enhancing the relaxation for several neurons within a single layeror two adjacent layers.
GCP-CROWN can efficiently handle general cutting plane constraints withneurons from any layers in a bound propagation manner, and the cutting planes we find from a MIPsolver can be seen as tighter convex relaxations encoding multi-neuron and multi-layer correlations.
Conclusion
In this paper, we propose GCP-CROWN, an efficient and GPU-accelerated bound propagation methodfor NN verification capable of handling any cutting plane constraints.
We combine GCP-CROWNwith branch and bound and high quality cutting planes generated by a MIP solver to tighten theconvex relaxation for NN verification.
The combination of fast bound propagation and strong cuttingplanes lead to state-of-the-art verification performance on multiple benchmarks.
Our work opens upa great opportunity for studying more efficient and powerful cutting planes for NN verification.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] Code will be
available when the final version of the this paper is released.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[No] Not applicable
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] We only use public data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[No] We only use public data.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[-The ∥·∥ 1 comes from minimizing over x 0 with the constraint x 0 − ϵ ≤ x ≤ x 0 + ϵ , and the ReLU( · )term comes from minimizing over z ( i ) j with the constraint 
-]
[N/A]
[-Thislower bound ( − 3 . 
-]
[-The CIFAR-
-]
[Table]