P IECEWISE L INEAR N EURAL N ETWORK V ERIFICATION : A COMPARATIVE STUDY
Anonymous authors
Paper under double-blind review
A BSTRACT
The success of Deep Learning and its potential use in many important safetycritical applications has motivated research on formal veriﬁcation of Neural Network (NN) models.
Despite the reputation of learned NN models to behave as black boxes and the theoretical hardness of proving their properties, researchers have been successful in verifying some classes of models by exploiting their piecewise linear structure.
Unfortunately, most of these approaches test their algorithms without comparison with other approaches.
As a result, the pros and cons of the different algorithms are not well understood.
Motivated by the need to accelerate progress in this very important area, we investigate the trade-offs of a number of different approaches based on Mixed Integer Programming, Satisﬁability Modulo
Theory, as well as a novel method based on the Branch-and-Bound framework.
We also propose a new data set of benchmarks, in addition to a collection of previously released testcases that can be used to compare existing methods.
Our analysis not only allows a comparison to be made between different strategies, the comparison of results from different solvers also revealed implementation bugs in published methods.
We expect that the availability of our benchmark and the analysis of the different approaches will allow researchers to develop and evaluate promising approaches for making progress on this important topic.
I NTRODUCTION
Despite their success in a wide variety of applications, Deep Neural Networks have seen limited adoption in safety-critical settings.
The main explanation for this lies in their reputation for being black-boxes whose behaviour can not be predicted.
Current approaches to evaluate the quality of trained models mostly rely on testing using held-out data sets.
However, as Edsger W. Dijkstra said (Buxton & Randell, 1970), “testing shows the presence, not the absence of bugs”.
If deep learning models are to be deployed to applications such as autonomous driving cars, we need to be able to enforce and verify safety-critical behaviours.
A particularly illustrative instance of the limit of our understanding of the behaviour of learned models lies in the existence of adversarial examples (Szegedy et al., 2014): small perturbations, imperceptible to the human eye, are capable of signiﬁcantly modifying the predictions generated by a network, despite it performing well on its test set.
Several methods have been proposed (Gu &
Rigazio, 2015; Goodfellow et al., 2015) to make networks more robust to those perturbed inputs.
It is however not clear if those methods are successful at reducing the number of adversarial examples or if they are just capable of reducing the number of adversarial examples that currently known methods can generate.
The only way to know this is to measure the size of the region around training samples guaranteed to not contain adversarial examples.
Some researchers have indeed tried to use formal methods to obtain guarantees like the one mentioned above.
To the best of our knowledge, Zakrzewski (2001) was the ﬁrst to propose a method to verify simple, one hidden layer neural networks, but only recently (Katz et al., 2017a;
Narodytska et al., 2017; Cheng et al., 2017c) were researchers able to work with non-trivial models by taking advantage of the structure of ReLU-based networks.
Even then, these works are not scalable to the large networks encountered in most real world problems.
A major roadblock in the area has been the lack of any analysis of the success and failure modes of the proposed approaches.
To remedy this problem, we gather a data set of test cases based on existing literature and parametrically explore the space of possible architectures.
We use it to evaluate different published approaches, implementing them ourselves where no public version was available, and generate the ﬁrst experimental comparison of published tools.
In addition, we introduce a general formalism for the problem, showing possible directions for improvement, as well as a new method showing signiﬁcantly better performance on practical scenarios.
Additionally, our comparison also revealed bugs in some publicly available NN veriﬁcation software, made evident by contradictions in the results of the various methods.
1. P ROBLEM SPECIFICATION
We now specify the problem of formal veriﬁcation of neural networks.
Given a network that implements a function y = f
( x ) , a bounded input domain C and a property P , we want to prove that
[Equation]
A toy-example of such a problem is given in Figure 1.
On the domain C = [ − 2; 2] × [ − 2; 2] , we want to prove that the output y of the one hidden-layer network, always satisfy the property
P ( y ) (cid:44) [ y > − 5] .
We will use this as a running example to explain the methods used for comparison in our experiments.
In this paper, we are going to focus on Piecewise-Linear Neural Networks (PL-NN), that is, networks for which we can decompose C into a set of polyhedra C i such that C =
∪ i C i , and the restriction of f to C
i is a linear function for all i .
While this prevents us from including networks that use activation functions such as sigmoid or tanh, PL-NNs allow the use of linear transformations such as fullyconnected or convolutional layers, pooling units such as MaxPooling and activation functions such as ReLUs and its various extensions such as Leaky ReLU or PReLU (He et al., 2015).
In other words, PL-NNs represent the majority of networks used in practice.
Note that layers such as BatchNormalization also preserve piecewise linearity at test-time.
The properties that we are going to consider are Boolean formulas over linear inequalities.
Although the formulas in (1) deﬁne the property to be a function of the ouptut y , we have no loss of generality: any property involving more variables could be expressed as a function over the output of a different network, including all additional variables in its output.
As the problem of PL-NN veriﬁcation has been shown to be NP-complete using a reduction from 3-SAT (Katz et al., 2017a), it is unlikely that any polynomial time algorithm will exist.
Therefore, experimental comparison remains the only approach possible to evaluate the relative advantages of different methods, which we propose to do.
R ELATED WORKS
We start by brieﬂy presenting related work that we do not include in our comparison as they are not capable of performing general veriﬁcation of problem of the form of Problem 1.
Zakrzewski (2001) and Hein & Andriushchenko (2017) propose methods based on the second derivatives of the functions expressed by the networks to guarantee that the output of the network doesn’t change too much around points.
However, this requires the additional assumption that all layers of the networks are twice differentiable, which PL-NN don’t satisfy.
At the other end of the spectrum, Narodytska et al.
(2017) and Cheng et al.
(2017c) use SAT solvers to propose veriﬁcation methods for the specialised case of Binarized Neural Networks (Hubara et al., 2016).
These methods, however, don’t translate to PL-NNs.
In addition to exact formal methods, some other approaches were proposed that don’t provide complete veriﬁcation.
Pulina & Tacchella (2010) proposed a method for veriﬁcation of networks with sigmoid activation function.
The approximation to the non-linearities means their method can not always return a decision for certain problems.
Bastani et al.
(2016) studied PL-NN in the context of obtaining robustness guarantees against adversarial examples.
For scalability reasons, they added additional assumptions, limiting the domain considered to the set of points sharing the same activation pattern than a reference point. This effectively circumvents all of the non-linearities of the network.
Xiang et al.
(2017) and Huang et al.
(2017) both rely on discretisation and perform layer by layer analysis to obtain guarantees over the output, but Huang et al.
(2017) require the user to specify a family of possible changes and Xiang et al.
(2017) over-approximate the output space x 1 [-2, 2] x 2 [-2, 2] a b  - - y - -
Prove that y > −
Network.
We attempt to prove the property that the network output is always greater thanStep xxa in a out b in b out y
Fix linear constraints     -
{+-5 Fix a ReLU      -        -
 Fix linear constraints - - -    - . . .+} of each layer, leading to the existence of undecidable properties, making both inadequate for our comparison.
V ERIFICATION M ETHODS
The methods we involve in our comparison all leverage the piecewise-linear structure of PL-NN to make the problem more tractable.
We shall use the network of Figure 1 throughout as an illustrative example.
All the methods we compare follow the same general principle: given a property to prove, they attempt to discover a counterexample that would make the property false.
This is accomplished by deﬁning a set of variables corresponding to the inputs, hidden units and output of the network, as well as the set of constraints that a counterexample would satisfy.
For the network of Figure 1, the variables would be { x 1 , x 2 , a in , a out , b in , b out , y } and the set of constraints would be:
[Equation]
Here, a in is the input value to hidden unit a , while a out is the value after the ReLU.
Any point satisfying all the above constraints would be a counterexample to the property, as it would imply that it is possible to drive the output to -5 or less.
However, if this problem is unsatisﬁable, no counterexample can exist, which implies that the property is True.
We want to emphasize that our requirements go beyond saying that no counterexample could be found: it is necessary to prove that no counter-examples exists.
The difﬁculty stems from the non-linear constraints of (2c).
We will now explain how each method tackles this problem.
3. M IXED I NTEGER P ROGRAMMING E NCODING
A possible way to eliminate the non-linearities is to encode them with the help of binary variables, which transform the PL-NN veriﬁcation problem into a Mixed Integer Program (MIP).
Lomuscio &
Maganti (2017) and Cheng et al.
(2017b) advocate the use of big-M encoding to achieve this.
For example, the non-linearities of equation (2c) are replaced by
[Equation]
where M a is a sufﬁciently large value.
The binary variable δ a indicates which phase the ReLU is in: if δ a = 0 , the ReLU is blocked and a out = 0 , else the ReLU is passing and a out
= a in .
The problem remains difﬁcult due to the integrality constraint on δ a .
We provide more details in Appendix A on how to handle MaxPooling units similarly.
This approach has some advantages.
As the ﬁnal problem to solve ends up being a MIP, imposing integrality constraints on the inputs comes at no additional cost.
This can prove useful if some input features to the network are known to necessarily be integers (Cheng et al., 2017a).
For other methods, imposing these integer constraints would not be possible: either the proof would be attempted on the relaxed version of the networks or it would have to be done for all possible combinations of integer inputs.
As the solving of MIP is NP-hard, the performance of those methods is going to be dependent both on the quality of the solver used and of the encoding.
Cheng et al.
(2017b) proposes several methods to obtain a good encoding by picking appropriate values of M, in order for the quality of the linear relaxation of the MIP problem to be as high as possible.
However, in the end, the problem is still resolved by a general purpose MIP solver and the question remains open whether a solver not knowing more about the speciﬁc structure of the problem can be efﬁcient on challenging benchmarks.
3. R ELUPLEX
Katz et al.
(2017a) presents a procedure named Reluplex to verify properties of Neural Network containing linear functions and ReLU activation unit, functioning as an SMT solver using the splittingon-demand framework (Barrett et al., 2006).
The principle of Reluplex is to always maintain an assignment to all of the variables, even if some of the constraints are violated.
Starting from an initial assignment, it attempts to ﬁx some violated constraints at each step.
It prioritises ﬁxing linear constraints ((2a) and (2b) in our illustrative example) using a simplex algorithm, even if it leads to violated ReLU constraints (2c).
This can be seen in step 1 and 3 of the process shown in Table 1.
If no solution to the problem containing only linear constraints exists, this shows that the counterexample search is unsatisﬁable.
Otherwise, all linear constraints are ﬁxed and Reluplex attempts to ﬁx one violated ReLU at a time, such as in step 2 of Table 1 (ﬁxing the ReLU b ), potentially leading to newly violated linear constraints.
This process is not guaranteed to converge, so to guarantee progress, non-linearities getting ﬁxed too often are split into two cases.
This generates two new sub-problems, each involving an additional linear constraint instead of the linear one.
The ﬁrst one solves the problem where a out = 0 , the second one where a out = a in .
In the worst setting, the problem will be split completely over all possible combinations of activation patterns, at which point the sub-problems are simple LPs.
3. P LANET
Ehlers (2017a) also proposed an approach based on splitting the problems over the possible phase of the activation function.
Unlike Reluplex, the proposed tool, named Planet, operates by attempting to ﬁnd an assignment to the phase of the non-linearities.
Reusing the notation introduced in Section 3.1, it assigns a value of True or False to each δ a variable, verifying at each step the feasibility of the partial assignment.
As opposed to Reluplex, this has the advantage of being easily extended to networks containing MaxPooling units.
In order to detect incoherent assignments (such as both a and b being in the greater than zero region of the ReLU in the example of Figure 1) faster, the author proposes a global linear approximation to a neural network.
In addition to the existing linear constraints ((2a) and (2b)), the non linear constraints are approximated by sets of linear constraints representing the convex hull of the nonlinearities.
ReLUs are replaced by the set of equations:
[Equation]
where a out and a in are respectively the output and input of the ReLU, and u in and l in are pre-computed upper and lower bounds of the ReLUs input.
This feasible domain is illustrated in Figure 2a.
MaxPooling units are replaced by the set of constraints:
[Equation]
where in i are the inputs to the MaxPooling unit and l in i their lower bounds.
A one dimensional cut of this is represented in Figure 2b.
As a consequence, the whole network is approximated by an LP that can be efﬁciently queried to detect infeasibility or automatically deducing implied assignments to other δ i variables.
The values of the lower bounds and upper bounds necessary to deﬁne the constraints are built iteratively by optimizing the corresponding variable, based on the constraints imposed by the previous layers.
Additional heuristics to make infeasibility detection and implied phase inference faster are described in the original paper (Ehlers, 2017a). [Figure]
[Figure] [Figure] out Both Reluplex and Planet rely on the splitting mechanism over ReLUs to guarantee progress but their use of it is fundamentally different: while Reluplex (Katz et al., 2017a) drives the search for satisﬁability using a simplex algorithm and splits the ReLU lazily to unlock cases, Planet (Ehlers, 2017a) drives the whole search by eagerly making splits using a SAT solver and making deductions based on those.
Reluplex always maintains an assignment to all variables even though it doesn’t respect all constraints until the end; Planet only maintains assignments to the phase of the nonlinearities.
As those two approaches have never been compared on common benchmarks, it is hard to identify which is the most promising one or the speciﬁc cases in which one method outperforms the other, even though they rely on similar principles.
3. B RANCH AND B OUND OPTIMIZATION FOR VERIFICATION
Veriﬁcation as optimization We now present a different way of approaching the Neural Network veriﬁcation problem.
The whole satisﬁability problem will be transformed into an optimization problem where the decision will be obtained by checking the sign of the minimum.
We will show how any Boolean formula on linear inequalities can be encoded as additional layers at the end of the network.
Assume that the property is a simple inequality: P ( y ) (cid:44)
c T y ≥ b .
In that case, it is sufﬁcient to add to the network a ﬁnal fully connected layer with one output, with weight of c and a bias of − b .
If the global minimum of this network is positive, it indicates that for all y that the original network can output, we have c T y − b ≥
= ⇒ c T y ≥ b , and as a consequence the property is True.
On the other hand, if the global minimum is negative, then it provides a counter-example to the property.
Clauses speciﬁed using OR (denoted by (cid:87) ) can be encoded by using a MaxPooling unit; if the property is P ( y ) = (cid:87)
i (cid:2)
c
Ti y ≥ b i (cid:3) , this is equivalent to max i (cid:0)
c
Ti y − b i (cid:1) ≥ 0 .
Clauses speciﬁed using
AND (denoted by (cid:86) ) can be encoded similarly: the property P ( y )
= (cid:86) i (cid:2)
c
Ti y ≥ b i (cid:3) is equivalent to min i (cid:0) c
Ti y − b i (cid:1) ≥
⇐⇒ − (cid:0) max i (cid:0) − c
Ti y + b i (cid:1)(cid:1) ≥
As a result, we can formulate any Boolean formula over linear inequalities on the output of the network as a sequence of additional layers, and the veriﬁcation problem would be reduced to a global minimization problem.
Aside from some speciﬁc class of NN (Amos et al., 2017), this remains a hard problem.
The advantage is one of formalism, allowing us to prove complex properties ,containing several OR clauses, with a single procedure rather than having to decompose the desired property into separate queries as was done in previous work (Katz et al., 2017a).
Finding the exact global minimum, while not necessary for veriﬁcation, will have the advantage of generating a value.
If this value is positive, it will correspond to the margin by which the property is satisﬁed.
When estimating robustness to adversarial examples, existing methods choose to perform a binary search over the maximum radius guaranteeing the absence of adversarial examples.
The optimization process would be a more appropriate formalism here.
Branch and Bound for Optimization Optimization algorithms such as Stochastic Gradient Descent which are the usual workhorses of Deep Learning are not appropriate for this minimization problem.
Despite being capable of converging to good local minima, they have no way of guaranteeing whether or not a minima is global.
We now present an approach to tackle this problem, based on the Branch-and-Bound method.
Algorithm 1 describes the generic form of the Branch-and-Bound method.
The input domain will be repeatedly split into sub-domains (line 7), over which we will compute lower and upper bounds of the minimum of the output (lines 9-10).
The best upper-bound found so far will serve as a candidate for the global minimum.
Any domain whose lower bound is greater than the current global upper bound can be pruned away as it necessarily won’t contain the global minimum (line 13, lines 15-17).
By iteratively splitting the domains, we will be able to compute tighter lower bounds.
We keep track of the global lower bound on the minimum by taking the minimum over the lower bounds of all sub-domains (line 19).
When the global upper bound and the global lower bound differ by less than a small scalar (cid:15) (line 5), we consider that we have converged.
1: function B A B( net , domain , (cid:15) )
(cid:46)
Global minimum of net on domain to an (cid:15) accuracy 2: global ub ← inf 3: global lb ← − inf 4: domains ← [( global lb , domain )] 5: while global ub − global lb > (cid:15) do 6: ( , dom ) ← pick out ( domains ) 7: [ dom 1 , . .
, dom s ] ← split ( dom ) 8: for i = 1 . . .
s do 9: dom ub ← compute upper bound ( net , dom i ) 10: dom
lb ← compute lower bound ( net , dom i ) 11: if dom ub < global ub then (cid:46)
If found a better upper bound 12: global ub ← dom ub 13: domains ← prune domains ( domains , global ub ) (cid:46) Remove domains with dom lb ≥ global ub 14: end if 15: if dom lb < global ub then 16: domains ← domains + [( dom lb , dom i )] 17: end if 18: end for 19: global lb ← min { lb | ( lb , dom ) ∈ domains } 20: end while 21: return global ub 22: end function
The description of the veriﬁcation problem as optimization and the pseudo-code of Algorithm 1 are generic and would apply to veriﬁcation problems beyond the speciﬁc case of PL-NN.
To obtain a practical algorithm, it is necessary to specify the elementary functions:
• pick out ( domains ) :
Select one of the domains to branch on.
Several heuristics are possible, based on the bounds previously computed on the domains or based on the size of the domains.
• split ( domain ) : Takes as argument a domain and returns a partition of domains [ dom 1 , . .
, dom s ] such that (cid:83) i dom i =
domain
and that ( dom i ∩ dom j ) = ∅ , ∀ i (cid:54) =
j .
Choosing the split function will deﬁne the “shape” of the domains that we are operating on, potentially making the computation of the bounds harder or easier.
• compute { lower , upper } bound ( net , domain ) :
Compute a (lower / upper) bound of the minimum of net over the feasible domain domain .
We want the lower bound to be as high as possible (so that this whole domain can be pruned easily) and the upper bound to be as low as possible (so that we can use it to prune out other regions of the search space).
If we have several approaches to compute bounds, we can employ all of them at once and only keep the tightest.
In our experiments, we use the result of minimising the variable corresponding to the output of the network, subject to the constraints of the linear approximation introduced by Ehlers (2017a) as lower bound and simple random sampling as upper bound.
As the procedure is generic, any improved algorithm leading to faster or tighter bounds would directly translate to improvements to the veriﬁcation process.
In practice, it is not necessary to run Algorithm 1 to convergence for veriﬁcation.
If a negative global upper bound is found, the corresponding input is a valid counter-example.
Similarly, as soon as the global lower bound goes above zero, we know that the property is veriﬁed.
Now equipped with an understanding of the workings of each method, we are now ready to compare them.
S OLVER AND P ROPERTIES 4. D ATASET
As each method implements a different strategy and the worst-case analysis always indicate exponential runtimes, these approaches can only be compared experimentally.
We attempt to perform veriﬁcation on three data sets of veriﬁcation properties and report the comparison results.
The dimensions of all of the problems are given in Table
[Table]
[Table]
of
[Table] [Table]
units
[Table]
4. M ETHODS
Reluplex , based on the version released by the authors (Katz et al., 2017b).
The tool is implemented in C++ and relies on a modiﬁed version of the GLPK library to deal with the Simplex algorithm.
Note that as the tool doesn’t support MaxPooling units out of the box, we automatically convert the
[-10 − 10 − 10 − 10 − 10 − 10 − 10 −
-]
[-set 10 − 10 − 10 − 10 − 10 − 10 
-]
[-Relative area − −175000 − −125000 − −75000 − −
-]
MaxPooling layers into a series of linear layers with ReLU activations.
To do so, we decompose the element-wise maximum into a series of pairwise maximum
[Equation]
and decompose the pairwise maximums as sum of ReLUs:
[Equation]
where l x 2 is a pre-computed lower bound of the value that x 2 can take.
We initially observed some errors due to numerical accuracy appearing on the CollisionDetection dataset which we reported to the authors.
They provided us guidance on setting tolerance hyperparameters appropriately, reducing the number of incorrectly classiﬁed properties.
We report results for this set of parameters.
Planet , based on the version released by the author (Ehlers, 2017b).
The tool is implemented in
C++, using GLPK to solve linear programs and a modiﬁed version of MiniSat to drive the search.
We wrote some software to convert in both directions between the input format of both Reluplex and Planet.
We discovered some memory issues on the original implementation, which we reported to the author and used the ﬁxed version for all experiments.
MIP , that consists on encoding the satisﬁability problem as a Mixed Integer Program, using the “big
M” encoding of the non-linearity.
The exact encoding of MaxPooling and ReLU can be found in
Appendix A.
This method is similar to the one implemented in (Cheng et al., 2017b; Lomuscio &
Maganti, 2017) but due to the lack of availability of open-sourced methods, we reimplemented the approach in Python, using the Gurobi MIP solver.
To choose the value of M for each ReLU, we made use of the linear approximation of Planet.
This leads to better values of M than the dataﬂow analysis discussed by Cheng et al.
(2017b).
Even when compared to their other proposed heuristic {+10 − 10 − 10 − 10 − 10 − 10 − 10 −
 Relative area 32. L o w e r b o un d+}
[Figure]
{+10 − 10 − 10 − 10 − 10 − 10 
+}
{+Relative area − −175000 − −125000 − −75000 − −
+}
L o w e r b o un d
ReApproximatingFixedApproximation
[Figure]
for the selection of M , it has several advantages: it encompasses information from all layers of the network rather than from only a few layers; it solves simple LPs rather than MIPs.
The default settings of Gurobi were used.
We also include an alternate version, MIPtune , where for each dataset, we started by running the Gurobi Tuning tool for a day on one property chosen at random in the dataset so as to pick hyperparameters.
Those hyperparameters were then used for all the properties of the dataset.
BaB , based on the method described in Section 3.4.
Our pick out strategy chooses the domain that currently has the smallest lower bound.
We split the domain in half across its longest edge to generate two new smaller domains as our split method.
We generate upper bounds on the minimum by randomly sampling points on the considered domain, and minimise the linear approximation of the network proposed by (Ehlers, 2017a) as a lower bound.
Our implementation is in Python and uses Gurobi to solve LPs.
We also include results for a reluBaB variant, where the split method is based on the phase of the non-linearities, similarly to Planet but without using a SAT solver to drive the search.
Note that as opposed to the approach taken by Ehlers (2017a) of building a single approximation of the network, we rebuild a new approximation for each sub-domain.
This is motivated by the observation shown in Figure 3 which demonstrate the signiﬁcant improvements it brings, especially for deep networks.
In addition, the branch and bounds methods explictly search for upper bounds as well, which has the potential to improve the time at which counterexamples are exhibited.
4. E VALUATION C RITERIA
For each of the data sets, we compare the different methods using the same protocol.
Each run is done with a timeout of two hours, and a maximum allowed memory usage of 20GB, on a single core of a machine with an i7-5930K CPU and 32GB of RAM.
We will release all code and data necessary to replicate our analysis. Success rate corresponds to the proportion of properties for which the solver returns the correct answer before timing out or being terminated for using too much memory. We compare the performances of each method and present the results separately for the cases where the properties are True or False.
SAT means that the satisﬁability problem for a counter-example was satisﬁable, implying that the property was False.
In this case, the runtime corresponds to the time it took before exhibiting a counter-example to the property.
On the other hand, UNSAT corresponds to the time it took to prove that the problem was infeasible and that no counterexamples to the property could exist.
As methods returning SAT exhibit a counterexample, disagreement between solvers can be easily resolved by evaluating the network over the counterexample and checking the property on the output.
We use this criterion to establish a ground truth for each property.
We reported bugs to the original authors of each method every time such a disagreement was detected.
In the case of a timeout, the runtime for the method is counted as the maximum allowed time (7200 seconds), even though the actual runtime would be worse.
As a result, the average runtime for methods with low success rate would be worse in practice than reported here.
After ﬁxing the
Planet solver, no Out of Memory error was encountered.
To give more insights into the relative performance of each solver, we count the Number of wins , which corresponds to how many times a solver was the fastest to solve a property.
When the relative difference between the runtime of two solvers is less than 1%, we don’t count any win.
Having now presented the software tools we use, we now report their performance on each data set.
A NALYSIS 5. A IRBORNE C OLLISION A VOIDANCE S YSTEM
The Airborne Collision Avoidance System ( ACAS ) data set, as released by Katz et al. (2017a) is a neural network based advisory system recommending horizontal manoeuvres for an aircraft in order to avoid collisions, based on sensor measurements.
Each of the ﬁve possible manoeuvres is assigned a score by the neural network and the action with the minimum score is chosen.
In the case of complex property to prove, such as “NoAction (CoC) is the minimal score”, their counter-example search is implemented as a series of four satisﬁability problems: “Does there exist a point where the score for { WeakLeft, WeakRight, StrongLeft, StrongRight } is less than the score for NoAction?”. While our approach discussed in Section 3.4 would be able to merge all of these satisﬁability problems into a single one, we use the original strategy of the authors for a fair comparison [Table]
of the methods and are therefore not present in the SAT/UNSAT breakdowns. Reluplex and BaB/reluBaB boast comparable success rate but the BaBs ﬁnish signiﬁcantly faster, especially for SAT.
Results for the ACAS data are shown in Table 3.
Our proposed method using Branch and Bound performs the best along all criteria.
Compared to the second best method, Reluplex, it is more than an order of magnitude faster at exhibiting counter-examples and more than twice as fast at proving the correctness of True properties.
The variants performing splitting on the non-linearity phase, reluBaB, solves more properties but is on average slightly slower. On this data set where the networks are deep, the lack of updates to the linear approximation make the approximation really poor (see Figure 3), which explains Planet not doing well. The BaB methods are the only ones performing better on SAT problems than on UNSAT ones.
We postulate that this is due to the relatively small dimensionality of the input, which makes random testing capable of easily getting good coverage and discovering counterexamples quickly so the simple upper bound method we employed is good enough to ﬁnd counterexamples.
The tuning of the MIP solver doesn’t improve the performance of the method signiﬁcantly.
Although the performance difference is signiﬁcant on the property that was used for the tuning (40 times faster), the hyperparameters don’t generalize well to other properties; the tuned version being faster than the default one in only half the testcases.
This behaviour is also observed for the other datasets.
5. C OLLISION D ETECTION
In the CollisionDetection data set, as released by the authors of Planet (Ehlers, 2017a), the network attempts to predict whether two vehicles with parameterized trajectories are going to collide.
A total of 500 properties are extracted from problems arising from a binary search to identify the size of the region around training examples in which the prediction of the network doesn’t change.
Planet is the best performing method on the set of benchmarks that accompanied the release of the tool, even though it was the worst performing one on ACAS.
Table 4 shows it being the fastest method for most of the properties.
Conversely, BaB / reluBaB becomes the worst performing [Table] method, especially on True properties.
It is however important to note that all solvers ﬁnished signiﬁcantly below the timeout limit, indicating that the data set isn’t extremely challenging.
On this data set, Reluplex classiﬁed one False properties as unsatisﬁable.
We evaluated the counterexamples returned by other solvers using Reluplex’s code and conﬁrmed that they were valid counterexamples. Using other settings of tolerance parameters, this property would be appropriately classiﬁed but these settings lead to other properties being misjudged.
5. PCAMNIST
To get a better understanding of what factors inﬂuence the performance of various methods, we propose a novel PCAMNIST data set, which we generated to have control over different hyperparameters.
The networks takes k features as input, corresponding to the ﬁrst k eigenvectors of a
Principal Component Analysis of the digits from the MNIST data set. We train the network to predict whether the represented numbers are odd or even. Obtained accuracies are given in Appendix B.
For each network, we attempt to prove whether or not there exists an input for which the score assigned to the odd class is greater than the score of the even class plus a large conﬁdence.
We heuristically determine what the largest such conﬁdence can be by running gradient descent with random restarts for a long period of time. Knowing this maximum conﬁdence, we can choose the value of the conﬁdence differential we want to prove so as to have the property be true (or false) by a margin of our choosing.
Table 5 gives the overall results over the data set but given that all properties have different parameters, global summaries may not be an adequate representation.
It is however notable that Reluplex which was one of the best performing method on ACAS is here one of the worst.
We also present the result in Figure 4, where we generate a plot showing the evolution of runtimes for each of the architectural parameters of the networks.
This allows us to assess the impact of each factor on the performance of the solvers.
For all graphs of Figure 4, the trend for all the methods are similar, which seems to indicate that hard properties are intrinsically hard and not just hard for a speciﬁc solver. This doesn’t mean that [Table] [Figure]
all solvers have identical performance proﬁles. For example, in the easiest settings where all the solvers are succesful at solving a property, Planet is always the fastest solver, indicating that in case of easy problems, it is the most appropriate. On the other hand, MIP can be considered the more robust solver as it performs the best on hard tasks when most of the other solvers fail [Figure]
Figure 4a shows an expected trend: the largest the number of inputs, the harder the problem is. Similarly, Figure 4b shows unsurprisingly that wider networks require more time to solve, which can be explained by the fact that they have more non-linearities. The impact of the margin, as shown in Figure 4c is also clear: properties that are clearly True or clearly False are easiest to prove, while properties that are on the limit are signiﬁcantly harder.
Note that as opposed to the other solvers,
Planet has a lot more difﬁculties to ﬁnd counterexamples, even when the property is False by a signiﬁcant margin. Figure 4d presents some intriguing results: after a certain point, increasing the depth of the network makes the problem easier. Note however than this is only considering the impact of the network depth once we have ﬁxed the value of the margin and not the impact of the network depth by itself.
C ONCLUSION
The improvement of formal veriﬁcation of Neural Networks represents an important challenge to be tackled before learned models can be used in safety critical applications.
The lack of a shared set of benchmarks between researchers makes it hard to evaluate progress or estimate promising research directions.
We gathered test cases from existing literature, proposed new ones, and evaluated the performance of published methods, which allowed us to surface issues in published tools and offer an informed view of the status of the ﬁeld.
While no clear winner emerged from all the data sets, the importance of testing methods on various datasets appears clearly from the fact that an approach performing the best on its dataset of choice can also be the worst performing on another one so any improvement claimed should be demonstrated on a variety of tasks.
In addition, we proposed a conceptually simple method that offered competitive performance on the challenging data sets.
Our method is sufﬁciently general to be easily improvable, as any better lower bound will directly translate to faster veriﬁcation.
A MIP ENCODING DETAILS
We now present the encoding used to convert the non linearity of a neural network into inequality constraints involving binary variables.
In the case of a ReLU activation
[Equation]
we can replace this equation containing the max non-linearity by
[Equation]
where M a is a value such that M a is an upper-bound of a in and − M a is a lower-bound of a in .
For MaxPooling units, we can either make the choice to decompose the MaxPooling into a series of
ReLU and use their linear encoding.
Another solution is to encode the MaxPooling directly.
The constraint out
[Equation]
(10) can be replaced by out ≥
[Equation]
where M is an upper-bound on all in i and lb i are lower-bounds on each in i .
B PCAMNIST ACCURACIES
[Table]