General Cutting Planes for Bound-Propagation-Based Neural Network Verification
Anonymous Author(s)
AffiliationAddress email
Abstract
Bound propagation methods, when combined with branch and bound, are amongthe most effective methods to formally verify properties of deep neural networkssuch as correctness, robustness, and safety.
However, existing works cannot handlethe general form of cutting plane constraints widely accepted in traditional solvers,which are crucial for strengthening verifiers with tightened convex relaxations.
Inthis paper, we generalize the bound propagation procedure to allow the additionof arbitrary cutting plane constraints, including those involving relaxed integervariables that do not appear in existing bound propagation formulations.
Ourgeneralized bound propagation method, GCP-CROWN, opens up the opportunityto apply g eneral c utting p lane methods for neural network verification while benefiting from the efficiency and GPU acceleration of bound propagation methods.
As a case study, we investigate the use of cutting planes generated by off-the-shelfmixed integer programming (MIP) solver.
We find that MIP solvers can generatehigh-quality cutting planes for strengthening bound-propagation-based verifiersusing our new formulation.
Since the branching-focused bound propagation procedure and the cutting-plane-focused MIP solver can run in parallel utilizing differenttypes of hardware (GPUs and CPUs), their combination can quickly explore alarge number of branches with strong cutting planes, leading to strong verificationperformance.
Experiments demonstrate that our method is the first verifier that cancompletely solve the oval20 benchmark, and can verify twice as many instances onthe oval21 benchmark compared to the best tool in VNN-COMP 2021, and alsonoticeably outperforms state-of-the-art verifiers on a wide range of benchmarks.
Introduction
Neural network (NN) verification aims to formally prove or disprove certain properties (e g., correctness and safety properties) of a NN under a certain set of inputs.
These methods can provideworst-case performance guarantees of a NN, and have been applied to mission-critical applications thatinvolve neural networks, such as automatic aircraft control [28, 4], learning-enabled cyber-physicalsystems [49], and NN based algorithms in an operating system [46].
The NN verification problem is generally NP-complete [27].
For piece-wise linear networks, itcan be encoded as a mixed integer programming (MIP) [48] problem with the non-linear ReLUneurons described by binary variables.
Thus, fundamentally, the NN verification problem can besolved using branch and bound (BaB) [8] method similar to generic MIP solvers, by branchingsome binary variables and relaxing the rest into a convex problem such as linear programming(LP) to obtain bounds on the objective.
Although early neural network verifiers relied on off-theshelf CPU-based LP solvers [31, 7] for bounding in BaB, LP solvers do not scale well to large NNs.
Thus, many recent verifiers are instead based on efficient and GPU-accelerated algorithmscustomized to NN verification [6, 55, 13, 52, 9].
Bound propagation methods, presented in a few different formulations [53, 15, 51, 56, 45, 22], empower state-of-the-art NN verifiers such as α,βCROWN [56, 55, 52] and VeriNet [3], and can achieve two to three orders of magnitudes speedupcompared to solving the NN verification problem using an off-the-shelf solver directly [52], especiallyon large networks.
Despite the success of existing NN verifiers, we experimentally find that state-of-the-art NN verifiersmay timeout on certain hard instances which a generic MIP solver can solve relatively quickly,sometimes even without branching.
Compared to an MIP solver, a crucial factor missing in mostscalable NN verifiers is the ability to efficiently generate and solve general cutting planes (or “cuts”).
In generic MIP solvers, cutting planes are essential to strengthen the convex relaxation, so thatmuch less branching is required.
Advanced cutting planes are among the most important factorsin modern MIP solvers [5]; they can strengthen the convex relaxation without removing any validinteger solution from the MIP formulation.
In the setting of NN verification, cutting planes reflectscomplex intra-layer and inter-layer dependencies between multiple neurons, which cannot be easilycaptured by existing bound propagation methods with single neuron relaxations [40].
This motivatesus to seek the combination of efficient bound propagation method with effective cutting planes tofurther increase the power of NN verifiers.
A few key factors make the inclusion of general cutting planes in NN verifiers quite challenging.
First,existing efficient bound propagation frameworks such as CROWN [56] and β -CROWN [52] cannotsolve general cutting plane constraints that may involve variables across different layers in the MIPformulation.
Particularly, these frameworks do not explicitly include the integer variables in the MIPformulation that are crucial when encoding many classical strong cutting planes, such as Gomorycuts and mixed integer rounding (MIR) cuts.
Furthermore, although some existing works [42, 47, 35]enhanced the basic convex relaxation used in NN verification (such as the Planet relaxation [16]),these enhanced relaxations involve only one or a few neurons in a single layer or two adjacent layers,and are not general enough.
In addition, an LP solver is often required to handle these additionalcutting plane constraints [35], for which the efficient and GPU-accelerated bound propagation cannotbe used, so the use of these tighter relaxations may not always bring improvements.
In this paper, we achieve major progress in using general cutting planes in bound propagation based
NN verifiers.
To mitigate the challenge of efficiently solving general cuts, our first contribution isto generalize existing bound propagation methods to their most general form, enabling constraintsinvolving variables from neurons of any layer as well as integer variables that encode the status of a
ReLU neuron.
This allows us to consider any cuts during bound propagation without relying on a slow
LP solver, and opens up the opportunity for using advanced cutting plane techniques efficiently forthe NN verification problem.
Our second contribution involves combining a cutting-plane-focused,off-the-shelf MIP solver with our GPU-accelerated, branching-focused bound propagation methodcapable of handling general cuts.
We entirely disable branching in the MIP solver and use it only forgenerating high quality cutting planes not restricting to neurons within adjacent layers.
Although an
MIP solver often cannot verify large neural networks, we find that they can generate high qualitycutting planes within a short time, significantly helping bound propagation to achieve better bounds.
Our experiments show that general cutting planes can bring significant improvements to NN verifiers:we are the first verifier that completely solves all instances in the oval20 benchmark, with anaverage time of only 13 seconds per instance; on the even harder oval21 benchmark in VNNCOMP 2021 [3], we can verify twice as many instances compared to the competition winner.
Wealso outperform existing state-of-the-art bound propagation based methods including those usingmulti-neuron relaxations (a limited form of cutting planes).
Background
The NN verification problem We consider the verification problem for an L -layer ReLU-based
Neural Network (NN) with inputs ˆ x (0) :
= x ∈ R d 0 , weights W ( i ) ∈
R d i × d i − 1 , and biasesb ( i ) ∈
R d i ( i ∈ { 1 , · · · , L } ).
We can get the NN outputs f ( x ) = x ( L ) ∈ R d L by sequentiallypropagating the input x through affine layers with x ( i ) =
W ( i ) ˆ x
( i − 1)
+ b ( i ) and ReLU layer withˆ x
( i ) =
ReLU( x ( i ) ) .
We also let scalars ˆ x ( i ) j and x ( i ) j denote the post-activation and pre-activation,respectively, of j -th ReLU neuron in i -th layer.
Throughout the paper, we use bold symbols to denotevectors (e.g., x ( i ) ) and regular symbols to denote scalars (e.g., x ( i )
j is the j -th element of x ( i ) ).
Weuse the shorthand [ N ] to denote { 1 , · · · , N } , and W ( i ) : ,j is the j -th column of W ( i ) .
Commonly, the input x is bounded within a perturbation set C (such as an ℓ p norm ball) and theverification specification defines a property of the output f ( x ) that should hold for any x ∈ C ,e.g., whether the true label’s logit f y ( x ) will be always larger than another label’s logit f j ( x ) , (i.e.,checking if f y ( x ) − f j ( x ) is always positive).
Since we can append the verification specification (suchas a linear function on neural network output) as an additional layer of the network, canonically, the
NN verification problem requires one to solve the following one-dimensional ( d L = 1 ) optimizationobjective on f ( x ) :
[Equation]
with the relevant property defined to be proven if the optimal solution f ∗ ≥ 0 .
Throughout this work,we consider the ℓ ∞ norm ball C :=
{ x : ∥ x − x 0 ∥ ∞ ≤ ϵ } where x 0 is a predefined constant (e.g., aclean input image), although it is possible to extend to other norms or specifications [38, 54].
The MIP and LP formulation for NN verification
The mixed integer programming (MIP)formulation is the root of many NN verification algorithms.
This formulation uses binary variables
zto encode the non-linear ReLU neurons to make the non-convex optimization problem (1) tractable.
Additionally, we assume that we know sound pre-activation bounds l ( i ) ≤ x ( i ) ≤ u ( i ) for x ∈ Cwhich can be obtained via cheap bound propagation methods such as IBP [20] or CROWN [56].
Then ReLU neurons for each layer i can be classified into three classes [53], namely “active” ( I +( i ) ),“inactive” ( I − ( i ) ) and “unstable” ( I ( i ) ) neurons, respectively:
[Equation]
Based on the definition of ReLU, activate and inactive neurons are linear functions, so only unstableneurons require binary encoding.
The MIP formulation of (1) is:
[Equation]
Since a MIP problem is slow or intractable to solve, it is commonly relaxed as a When the integervariables are relaxed to continuous ones, we obtain the LP relaxation of NN verification problem:
[Equation]
The ReLU constraints involving z is often projected out, leading to the well-known Planet relaxationused in many NN verifiers, replacing (6), (7) and (8) with a single constraint to get an equivalent LP:
[Equation]
[Equation]
Due to the relaxations, the objective of the LP formulation is always a lower bound of the MIPformulation: f ∗ LP ≤ f ∗ .
A verifier using this formulation is incomplete: if f ∗ LP ≥ 0 , then f ∗ ≥ 0and the property is verified; otherwise, we cannot conclude the sign of f ∗ so the verifier must return“unknown”.
Branch and bound can be used to improve the lower bound and achieve completeness [8,52].
However, in this paper, we work on an orthogonal direction of strengthening the LP formulationby adding cutting planes to obtain larger bounds.
Bound propagation methods Instead of solving the LP formulation directly using a LP solver,bound propagation methods aims to quickly give a lower bound for f ∗ LP .
For example, CROWN [56]and β -CROWN [52] propagate a sound linear lower bound backwards for f L ( x ) with respect to eachintermediate layer.
For example, suppose we know
[Equation]
With i =
L − 1 the above is trivially hold with a ( L − 1) =
W ( L ) , c
( L − 1)
=
b ( L ) .
In boundpropagation methods, a propagation rule propagates an inequality (13) through a previous layerˆ x
( i ) :
=
ReLU( W ( i ) ˆ x ( i − 1)
+ b ( i ) ) to obtain a sound inequality with respect to ˆ x
( i − 1) :
[Equation]
Here a ( i − 1) , c ( i − 1) can be calculated in close-form via a ( i ) , c ( i ) , W ( i ) , b ( i ) , l ( i ) and u ( i ) such thatthe bound still holds (see Lemma 1 in [52]).
Applying the procedure repeatedly will eventually reachthe input layer:
[Equation]
The minimization on linear layer can be solved easily when C is a ℓ p norm ball to obtain a valid lowerbound of f ∗ .
Since the bounds propagate layer-by-layer, this process can be implemented efficientlyon GPUs [54] without relying on a slow LP solver, which greatly improves the scalability and solvingtime.
Additionally, it is often used to obtain intermediate layer bounds l ( i ) and and u ( i ) required forthe MIP formulation (6)(7), by treating each x ( i ) j as the output neuron.
The bound propagation rulecan either be derived
1,we will discuss the our bound propagation procedure with general cutting plane constraints.
Neural Network Verification with General Cutting Planes
3.1 GCP-CROWN: General Cutting Planes in Bound Propagation
In this section, we generalize existing bound propagation method to handle general cutting planeconstraints.
Our goal is to derive a bound propagation rule similar to those in Section 2, howeverconsidering additional constraints among any variables within the LP relaxation.
To achieve this,we first derive the dual problem of the LP; inspired by the dual formulation, we derive the boundprorogation rule in a layer by layer manner that takes all cutting plane constraints into consideration.
LP relaxation with cutting planes.
In this section, we derive the bound propagation procedureunder the presence of general cutting plane constraints.
A cutting plane is a constraint involving anyvariables x ( i ) (pre-activation), ˆ x ( i ) (post-activation), z ( i ) (ReLU indicators) from any layer i :
[Equation]
Here h ( i ) , g ( i ) and q ( i ) are coefficients for this cut constraint.
The difference between a validcutting plane and an arbitrary constraint is that a valid cutting plane should not remove any validinteger solution from the MIP formulation.
Our new bound propagation procedure can work for anyconstraints, although in this work we focus on studying the impacts of cutting planes.
When there are
N cutting planes, we write them in a matrix form:
[Equation]
(15) where H ( i ) , G ( i ) , Q ( i
) ∈
R N × d i .
The LP relaxation with all cutting planes is:
[Equation]
t.
[Equation]
[Equation]
[Equation]
[Equation] [Equation] [Equation] Since an additional constraint is added, f ∗ LP-cut ≥ f ∗ LP
and we get closer to f ∗ .
Unlike the original LPwhere each constraint only contains variables from two consecutive layers, our general cutting planeconstraint may involve any variable from any layer in a single constraint.
The dual problem with cutting planes We first show the dual problem for the above LP.
The dualproblem we consider here is different from existing works in two ways: first, we have constraintswith integer variables to support potential cutting planes on z .
Additionally, we have cutting planesconstraints that may involve variables in any layer, so the dual in previous works such as [53] cannotbe directly reused.
Our dual problem is given below (derivation details in Appendix A):
[Equation]
s
[Equation]
L
= − 1; and for each i ∈
[ L − 1] :
[Equation]
and for each j ∈ I ( i ) the two equalities below hold:
[Equation]
Instead of solving the dual problem exactly, we use it to obtain a lower bound of f ∗ LP-cut .
Intuitively,due to the definition of due problem, any valid setting of dual variables leads to a lower bound off ∗ LP-cut .
Informally, starting from ν ( L ) = − 1 , by applying the constraints in this dual formulation,we can compute ν
( L − 1) , ν ( L − 2) , · · · until ν (1) .
The final objective is a function of ν ( i ) , i ∈
[ N ] andother dual variables.
Precisely, our GCP-CROWN bound propagation procedure with general cuttingplane constraint is presented in the theorem below (proof in Appendix A):
Theorem 3.1 (Bound propagation with general cutting planes) .
Given any optimizable parameters0 ≤ α ( i ) j ≤ 1 and β ≥ 0 , f ∗ LP-cut is lower bounded by the following objective function:
[Equation]
where variables ν
( i ) are obtained by propagating ν ( L ) =
− 1 throughout all i ∈
[ L − 1] :
[Equation]
Here ˆ ν
( i ) j , π ( i ) j ∗ and h ( i ) j
( β ) are defined for each unstable neuron j ∈
I ( i ) as:
[Equation]
[Equation]
Based on Theorem 3.1, to obtain an lower bound of f ∗ LP-cut , we start with any valid setting of0 ≤ α ≤ 1 and β ≥ 0 and ν ( L ) = − 1 .
According to the bound propagation rule, we can compute each ν ( i ) , i ∈
[ L − 1] , in a layer by layer manner.
Then objective g ( α , β ) can be evaluated based onall ν ( i ) to give an lower bound of f ∗ LP-cut .
Since any valid setting of α and β lead to a valid lowerbound, we can optimize α and β using gradient ascent in a similar manner as in [55, 52] to tightenthis lower bound.
The entire procedure can also run on GPU for great acceleration.
Connection to Convex Outer Adversarial Polytope
In convex outer adversarial polytope [53],a bound propagation rule was developed in a similar manner in the dual space without consideringcutting plane constraints, and is a special case of ours.
We denote their bound propagation objectivefunction as g WK which also contains optimizable parameters α WK .
Proposition 3.2.
Given the same input x , perturbation set C , network weights, and N cutting planeconstraints,
[Equation]
Proof.
In Theorem 3.1, when all β are set to 0, then π ( i ) j
[Equation]
and h ( i ) j ( β ) =
π ( i ) j ∗ l ( i ) j ,we recover exactly the same bound propagation equations as in [53].
However, since we allow theaddition of cutting plane methods and we can maximize over the additional parameter β , the objectivegiven by our bound propagation is always at least as good as g KW .
Connection to CROWN-like bound propagation methods CROWN [56] and α -CROWN
[55]use the same bound propagation rule as [53] so Proposition 3.2 also applies, although they werederived from primal space without explicitly formulating the problem as a LP.
Salman et al.
[40]showed that many other bound propagation methods [45, 50] are equivalent to or weaker than [53].
Recently, Wang et al.
[52] extends CROWN to β -CROWN to handle split constraints (e.g., x ( i ) j ≥ 0 ).
It can be seen a special case as GCP-CROWN where all H , G and Q matrices are zeros except:
H ( i ) j,j = 1 ,j ∈
I − ( i ) for x ( i ) j ≤ 0 split ;
H ( i ) j,j = − 1 ,j ∈
I
+( i ) for x ( i ) j ≥ 0 split
Wang et al.
[52] derived bound propagation rules from both the dual LP and the primal space with a Lagrangian without LP.
However, in our case, it is not intuitive to derive bound propagation without LP due to the potential cutting planes on relaxed integer variables z , which do not appear without theexplicit LP formulation.
3.2 Branch-and-bound with GCP-CROWN and MIP Solver Generated Cuts
[Figure]
To build a complete NN verifier, we follow the popular
[Figure]
branch-and-bound (BaB) procedure [8, 7] in state-of-the- art NN verifiers with GPU accelerated bound propagation
[Figure]
method [6, 55, 13, 52], and our GCP-CROWN is used asthe bounding procedure in BaB. We refer the readers to Appendix B for a more detailed background on branch-andbound. Having the efficient bound propagation procedurewith general cutting plane constraints, we now need to find
[Figure]
a good set of general cutting planes H ( i ) , G ( i ) , Q ( i ) to accelerate NN verification.
Since GCP-CROWN can adoptany cutting planes, to fully exploit its power, we proposeto use off-the-shelf MIP solvers to generate cutting planesand create an NN verifier combining GPU-acceleratedbound propagation with strong cuts generated by a MIPsolver.
We make the bound propagation on GPU andthe MIP solver on CPU run in parallel with cuts addedon-the-fly, so the original strong performance of boundpropagation-based NN verifier will never be affected bya potentially slow MIP solver.
This also allows us to makefull use of available computing resource (GPU + CPU).
The architecture of our verifier is presentedin Figure 1.
MIP solvers for cutting plane generation Generic MIP solvers such as cplex [25] andgurobi [21] also apply a branch-and-bound strategy, conceptually similar to state-of-the-art NNverifiers.
They often tend to be slower than specialized NN verifiers because MIP solvers rely onslower bounding procedures (e.g., Simplex or barrier method) and cannot apply an efficient boundpropagation procedure on GPUs.
However, we still find that MIP solvers are a strong baselinewhen combined with tight intermediate layer bounds .
For example, in the oval21 benchmark in Table 2, α -CROWN+MIP (MIP solver combined with tight intermediate layer bound computed byα -CROWN) is able to solve 4 more instances compared to all other tools in the competition.
Ourinvestigation found that α -CROWN+MIP explores much less branches than other state-of-the-artbranch-and-bound based NN verifiers, however before branching starts, the MIP solver produces veryeffective cutting planes that can often verify an instance with little branching.
MIP solvers is ableto discover sophisticated cutting planes involving several NN layers reflecting complex correlationsamong neurons, while existing NN verifiers only exploit specific forms of constraints within samelayer or between adjacent layers [47, 42, 35].
This motivates us to combine the strong cutting planesgenerated by MIP solvers with GPU-accelerated bound-propagation-based BaB.
In this work, we use cplex as the MIP solver for cutting plane generation since gurobi cannotexport cuts.
We entirely disable all branching features in cplex and make it focus on finding cuttingplanes only.
These cutting planes are generated only for the root node of the BaB search tree so theyare sound for any subdomains with neuron splits in BaB. We conduct branching using our generalizedbound propagation procedure in Section (3.1) with the cutting planes generated by cplex .
Fully-parallelized NN verifier design We design our verifier as shown in Figure 1.
After parsingthe NN under verification, we launch a separate process to encode the NN verification problem as
MIP problem and start multiple MIP solvers, one for each target label to be verified.
At the same time,the main NN verifier process executes branch-and-bound without waiting for the MIP solver process.
In each iteration of branch and bound, we query the MIP solving processes and fetch any newlygenerated cutting planes.
If any cutting planes are produced, they are added as H ( i ) , G ( i ) , Q ( i ) in
GCP-CROWN and tighten the bounds for subsequent branching and bounding.
If no cutting planesare produces, GCP-CROWN reduces to β -CROWN [52].
Since our verifier is based on strengtheningthe bounds in β -CROWN with sound cutting planes, we are also sound and complete.
Adjustments to existing branch and bound method.
We implement GCP-CROWN into theα,β -CROWN verifier [56, 54, 52], the winning verifier in VNN-COMP 2021 [3], as the backbonefor BaB with bound propagation.
To better exploit the power of cutting planes under a fully paralleland asynchronous design, we made a key changes to the BaB procedure.
When the number of
BaB subdomains are greater than batch size, we rank the subdomains by their lower bounds andchoose the easiest domains with largest lower bounds first to verify with GCP-CROWN, unlike mostexisting verifiers which solve the worst domains first.
We use such a order because the MIP solvergenerates cutting planes incrementally.
Solving these easier subdomains tend to require no or fewercutting planes, so we solve them at earlier stages where cutting planes have not been generated or arerelatively weak.
On the other hand, if we split worst subdomains first, the number of subdomains willgrow quickly, and it can take a long time to verify these domains when stronger cuts become availablelater.
Under a similar rationale, when verifying a multi-class model and BaB needs to verify eachtarget class label one by one, we start BaB from the easiest label first ( α -CROWN bound closest to0), allowing the MIP solver to run longer for harder labels, generating stronger cuts for harder labels.
Experiments
In this section, we evaluate our verifier, GCP-CROWN with MIP cuts, on a few popular verificationbenchmarks.
Since our verifier uses a MIP solver in our pipeline, we also include a new baseline,α -CROWN+MIP, which uses gurobi as the MIP solver with the tightest possible intermediate layerbounds from α -CROWN [54].
We include detailed setup for our experiments in Appendix C.
Results on the oval20 benchmark in VNN-COMP 2020.
oval20 is a popular benchmarkconsistently used in huge amount of NN verifiers and it perfectly reflects the progress of NN verifiers.
We include literature results for many baselines in Table 1.
We are the only verifier that can completelysolve all three models without any timeout.
Our average runtime is significantly lower compared tothe time of baselines because we have no timeout (counted as 3600s), and our slowest instance only
[Table]
0.00 ERAN ∗ [42, 43, 45, 44] 805.
- 0.00 β -CROWN [52] 118.
0.00 α -CROWN+MIP† 335.
[Figure]
with MIP cuts 13.
[Figure]
takes about a few minutes while easy ones only take a few seconds, as shown in Figure 2.
Additionally,we often use less number of branches compared to the state-of-the-art verifier, β -CROWN, since ourstrong cutting planes help us to eliminate many hard to solve subdomains in BaB. Furthermore, wehighlight that α -CROWN+MIP also achieves a low timeout rate, although it is much slower than ourbound propagation based approach combined with cuts from a MIP solver.
Results on VNN-COMP 2021 benchmarks.
Among the eight scoring benchmarks in VNNCOMP 2021 [3], only two ( oval21 and cifar10-resnet ) are most suitable for the evaluation ofthis work.
Among other benchmarks, acasxu and nn4sys have low input dimensionality and requireinput space branching rather than ReLU branching; verivital , mnistfc , and eran benchmarksconsist of small MLP networks that can be solved directly by MIP solvers; marabou contains mostlyadversarial examples, making it a good benchmark for falsifiers rather than verifiers.
We present ourresults in Table 2.
Among results from 6 VNN-COMP participants, we also include two additionalbaselines, α -CROWN+MIP (same as in Table 1), and MN-BaB [17], a recently proposed branch andbound framework with multi-neuron relaxations [34], which can be viewed as a restricted form ofcutting planes.
On the oval21 benchmark, OVAL, VeriNet and α,β -CROWN are the best performingtools, verified 11 out of 27 instances, while we can verify twice more instances (22 out of 27) onthis benchmark.
On the cifar10-resnet benchmark, our verifier also solves the most number ofinstances and achieves the lowest average time.
In fact, α -CROWN+MIP is also a strong baseline,solving 4 more instances than all competition participants, showing the importance of strong cuttingplanes.
Our GCP-CROWN with MIP cuts combines the benefits of fast bound propagation on GPUwith the strong cutting planes generated by a MIP solver and achieves the best performance.
Results on SDP-FO benchmarks.
We further evaluate our method on the SDP-FO benchmarksin [12, 52].
This benchmark contains 7 mostly adversarially trained MNIST and CIFAR models with200 instances each, which are hard for many existing verifiers.
Beyond the baselines reported in [52],we also include two additional baselines, α -CROWN+MIP (same as in Table 1) and MN-BaB [17](same as in Table 2).
Table 3 shows that our method improves the percentage of verified images(“verified accuracy”) on all models compared to state-of-the-art verifiers, further closing the gapbetween verified accuracy and empirical robust accuracy obtained by PGD attack (reported as “upperbound” in Table 2).
[Table]
[Table]
Related Work
Cutting plane method is a classic technique to strengthen the convex relaxation of an integer programming problem.
Generic cutting planes such as Gomory’s cut [18, 19], Chvátal–Gomory cut [10],implied bound cut [24], lift-and-project [30], reformulation-linearization techniques [41] and mixedinteger rounding cuts [36, 32] can be applied to almost any LP relaxed problems, and problem specificcutting planes such as Knapsack cut [11], Flow-cover cut [37] and Clique cut [26] require specificproblem structures.
Modern MIP solvers typically uses a branch-and-cut strategy, which tends togenerate a large number of cuts before starting the next iteration of branching, and solve the LPrelaxation of the MIP problem with cutting planes with an exact method such as the Simplex method.
Our GCP-CROWN is a specialized solver for the NN verification problem, which can quickly obtaina lower bound of the LP relaxation with cutting planes specially for the NN verification problem.
The verification of piece-wise linear NNs can be formulated as a MIP problem, so early works [27, 28,48] solve an integer or combinatorial formulation directly.
For efficiency reasons, most recent worksuse a convex relaxation such as linear relaxation [16, 53] or semidefinite relaxation [39, 12].
Salmanet al.
[40] discussed the limitation of many convex relaxation based NN verification algorithmsand coined the term “convex relaxation barrier”, specifically for the popular single-neuron “Planet”relaxation [16].
Several works developed novel techniques to break this barrier.
[42] added constraintsthat depends on the aggregation of multiple neurons, and these constraints were passed to a LP solver.[35] enhanced the multi-neuron formulation of [42] to obtain tighter relaxations.
[1] studied strongerconvex relaxations for a ReLU neuron after an affine layer, and [47] constructed efficient algorithmsbased on this relaxation for incomplete verification.
[13] further extended the formulation in [1] toa dual form and combined it with branch and bound to achieve completeness.
[17] combined themulti-neuron relaxation [35] with branch and bound.
Although these works can be seen as a specialform of cutting planes, they mostly focused on enhancing the relaxation for several neurons within asingle layer or two adjacent layers.
Our GCP-CROWN can efficiently handle general cutting planeconstraints with neurons from any layers in a bound propagation manner, and the cutting planeswe used from a MIP solver can be seen as tighter convex relaxations encoding multi-neuron andmulti-layer correlations.
Conclusion
In this paper, we propose GCP-CROWN, an efficient and GPU-accelerated bound propagation methodfor NN verification capable of handling any cutting plane constraints.
We combine GCP-CROWNwith branch and bound and high quality cutting planes generated by a MIP solver to tighten theconvex relaxation for NN verification.
The combination of fast bound propagation and strong cuttingplanes lead to state-of-the-art verification performance on multiple benchmarks.
Our work opens upa great opportunity for studying more efficient and powerful cutting planes for NN verification.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[No] Code will be
available when the final version of the this paper is released.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No] Not applicable
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] We only use public data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] We only use public data.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
A The dual problem with cutting planes
In this section we derive the dual formulation for neural network verification problem with generalcutting planes (or arbitrarily added linear constraints across any layers).
Our derivation is based onthe linearly relaxed MIP formulation with original binary variables z intact, to allow us to add cutsalso to the integer variable (the mostly commonly used triangle relaxation does not have z ).
Wefirst write the full LP relaxation with arbitrary cutting planes, as well as their corresponding dualvariables:
[Equation]
The Lagrangian function can be constructed as:
[Equation]
[Equation]
Here µ , τ , γ , π are shorthands for all dual variables in each layer and each neuron.
Note that forsome constraints, their dual variables are not created because they are trivial to handle in the stepsteps.
Rearrange the equation and swap the min and max (strong duality) gives us:
[Equation]
[Equation]
Here W ( i +1) : ,j denotes the j -th column of W ( i +1) .
Note that for the term involving j ∈
I +( i ) wehave replaced ˆ x ( i ) with x ( i ) to obtain the above equation.
For the the term x ( i ) j , j ∈ I +( i ) it is always0 so does not appear.
Solving the inner minimization gives us the dual formulation:
[Equation]
, i
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
Before we give the bound propagation procedure, we first give a technical lemma:
Lemma A.1.
Given u ≥ 0 , l ≤ 0 , π ≥ 0 , γ ≥ 0 , and π + γ = C , and define the function:
[Equation]
Then
[Equation]
where the optimal values for π and γ are:
[Equation]
Proof.
Case 1: when uγ + lπ + q ≥ 0 , the objective becomes:
[Equation]
Since q is a constant and the object only involves a non-negative variable γ with non-positivecoefficient − u , γ needs to be as smaller as possible as long as the constraint is satisfied.
Substituteπ =
C − γ into the constraint uγ + lπ + q ≥ 0 ,
[Equation]
Considering γ is within [0 , C ] , the optimal γ and π are:
[Equation]
Case 2: when uγ + lπ + q ≤ 0 , the objective becomes:
[Equation]
Since q is a constant and the object only involves a non-negative variable π with non-positivecoefficient l , π needs to be as smaller as possible as long as the constraint is satisfied.
Substituteγ =
C − π into the constraint uγ + lπ + q ≤ 0 ,
[Equation]
The optimal π ∗ and γ ∗ are the same as in case 1.
The optimal objective can be obtained by substituteπ ∗ and γ ∗ into g ( π, γ ) .
The objective depends on q because when q is too large ( q ≥ − lC ) or toosmall ( q ≤ − uC ) , π ∗ and γ ∗ are fixed at either 0 or C .
Theorem A.2 (Bound propagation with general cutting planes) .
Given the following bound propagation rule on ν with optimizable parameter 0 ≤ α ( i ) j ≤ 1 and β ≥ 0 :
[Equation]
Then f ∗ LP-cut is lower bounded by the following objective with any valid 0 ≤ α ≤ 1 and β ≥ 0 :
[Equation]
where h ( i ) j
( β ) is defined as:
[Equation]
Proof.
Given (22), for j ∈ I ( i ) , observing that the upper and lower bounds of ReLU relaxationsfor a single neuron cannot be tight simultaneously [53], π ( i ) j + γ ( i ) j and µ ( i ) j
+ τ ( i ) j cannot be bothnon-zero 1 .
Thus, we have
[Equation]
To avoid clutter, we define ˆ ν ( i ) j :=
ν ( i +1)
⊤ W
( i +1) : ,j − β ⊤ G ( i ) : ,j , and we define [ · ] + := max(0 , · ) and[ · ] − := min(0 , · ) .
To derived the proposed bound propagation rule, in (17), we must eliminate variable γ ( i ) j and π ( i ) j .
Ignoring terms related to ν , we observe that we can optimize the term π ( i ) j l ( i ) j − ReLU( u ( i ) j γ ( i )
j +l ( i )
j π ( i ) j − β ⊤ Q
( i ) : ,j ) for each j for I ( i ) individually.
We seek the optimal solution for the followoptimization problem on function h :
[Equation]
Note that we optimize over π ( i ) j and γ ( i ) j here and treat β as a constant.
Applying Lemma A.1 withπ =
π ( i ) j , γ = γ ( i ) j , u = u ( i ) j , l
[Equation]
[Equation]
⊤ Q ( i ) : ,j , C =
[ˆ ν ( i ) j ]
+ we obtain the optimal objectivefor (25):
[Equation]
Substitute this solution of π ( i ) j into (21), and also observe that due to (24), τ ( i ) j is a variable between0 and (cid:104)
ˆ ν
( i ) j (cid:105) − , so we can rewrite (21) as:
[Equation]
Here 0 ≤ α ( i ) j ≤ 1 is an optimizable parameter that can be updated via its gradient during boundpropagation, and any 0 ≤ α ( i )
j ≤ 1 produces valid lower bounds.
All above substitutions replaceeach dual variable π , γ and µ to a valid setting in the dual formulation, so the final objective g ( α , β )is always a sound lower bound of f ∗ LP-cut .
, γ ( i ) j
= 0 , π ( i )
j = µ ( i ) j
= 0 or π ( i ) j ̸ = 0 , µ ( i ) j ̸
0 , τ ( i ) j γ ( i )
j
This situation may happen when u ( i ) j or l ( i ) j is exactly tight and the solution x ( i ) j is at the intersection of one lower and upper linear equatlities.
Practically, this can be avoided by adding a small δ to u ( i ) j or l ( i ) j , and in most scenarios u ( i )
j are l ( i )
j are loose bounds so this situation will not occur.
The same argument is
B More technical details and background of GCP-CROWN with MIP cutsC More details on experiments