P IECEWISE L INEAR N EURAL N ETWORK V ERIFICATION : A COMPARATIVE STUDY
Anonymous authors
Paper under double-blind review
A BSTRACT
The success of Deep Learning and its potential use in many important safetycritical applications has motivated research on formal veriﬁcation of Neural Network (NN) models.
Despite the reputation of learned NN models to behave as black boxes and the theoretical hardness of proving their properties, researchers have been successful in verifying some classes of models by exploiting their piecewise linear structure.
Unfortunately, most of these approaches test their algorithms on their own models and do not compare with other approaches.
As a result, the advantages and downsides of the different algorithms are not well understood.
Motivated by the need of accelerating progress in this very important area, we investigate the trade-offs of a number of different approaches based on Mixed
Integer Programming, Satisﬁability Modulo Theory, as well as a novel method based on the Branch-and-Bound framework.
We also propose a new data set of benchmarks, in addition to a collection of previously released testcases that can be used to compare existing methods.
Our analysis not only allowed a comparison to be made between different strategies, the comparison of results from different solvers also revealed implementation bugs in published methods.
We expect that the availability of our benchmark and the analysis of the different approaches will allow researchers to develop and evaluate promising approaches for making progress on this important topic.
I NTRODUCTION
Despite their success in a wide variety of applications, Deep Neural Networks have seen limited adoption in safety-critical settings.
The main explanation for this lies in their reputation for being black-boxes whose behaviour can not be predicted.
Current approaches to evaluate the quality of trained models mostly rely on testing using held-out data sets.
However, as Edsger W. Dijkstra said (Buxton & Randell, 1970), “testing shows the presence, not the absence of bugs”.
If deep learning models are to be deployed to applications such as autonomous driving cars, we need to be able to enforce and verify safety-critical behaviours.
A particularly illustrative instance of the limit of our understanding of the behaviour of learned models lies in the existence of adversarial examples (Szegedy et al., 2014): small perturbations, imperceptible to the human eye, are capable of signiﬁcantly modifying the predictions generated by a network, despite it performing well on its test set.
Several methods have been proposed (Gu &
Rigazio, 2015; Goodfellow et al., 2015) to make networks more robust to those perturbed inputs.
It is however not clear if those methods are successful at reducing the number of adversarial examples or if they are just capable of reducing the number of adversarial examples that currently known methods can generate.
The only way to know this is to measure the size of the region around training samples guaranteed to not contain adversarial examples.
Some researchers have indeed tried to use formal methods to obtain guarantees like the one mentioned above.
To the best of our knowledge, Zakrzewski (2001) was the ﬁrst to propose a method to verify simple, one hidden layer neural networks, but only recently (Katz et al., 2017a;
Narodytska et al., 2017; Cheng et al., 2017c) were researchers able to work with non-trivial models by taking advantage of the structure of ReLU-based networks.
Even then, these works are not scalable to the large networks encountered in most real world problems.
A major roadblock in the area has been the lack of any analysis of the success and failure modes of the proposed approaches.
To remedy this problem, we gather a data set of test cases based on existing literature and parametrically explore the space of possible architectures.
We use it to evaluate different published approaches, implementing them ourselves where no public version was available, and generate the ﬁrst experimental comparison of published tools.
In addition, we introduce a general formalism for the problem, showing possible directions for improvement, as well as a new method showing signiﬁcantly better performance on practical scenarios.
Additionally, our comparison also revealed bugs in some publicly available NN veriﬁcation software, made evident by contradictions in the results of the various methods.
1. P ROBLEM SPECIFICATION
We now specify the problem of formal veriﬁcation of neural networks.
Given a network that implements a function y = f
( x ) , a bounded input domain C and a property P , we want to prove that
[Equation]
A toy-example of such a problem is given in Figure 1.
On the domain C = [ − 2; 2] × [ − 2; 2] , we want to prove that the output y of the one hidden-layer network, always satisfy the property
P ( y ) (cid:44) [ y > − 5] .
We will use this as a running example to explain the methods used for comparison in our experiments.
In this paper, we are going to focus on Piecewise-Linear Neural Networks (PL-NN), that is, networks for which we can decompose C into a set of polyhedra C i such that C =
∪ i C i , and the restriction of f to C
i is a linear function for all i .
While this prevents us from including networks that use activation functions such as sigmoid or tanh, PL-NNs allow the use of linear transformations such as fullyconnected or convolutional layers, pooling units such as MaxPooling and activation functions such as ReLUs and its various extensions such as Leaky ReLU or PReLU (He et al., 2015).
In other words, PL-NNs represent the majority of networks used in practice.
Note that as the networks to be veriﬁed are in their test-time state, layers such as Batch-Normalization still preserve piecewise linearity.
The properties that we are going to consider are Boolean formulas over linear inequalities.
Although the formulas in (1) deﬁne the property to be a function of the ouptut y , we have no loss of generality: any property involving more variables could be expressed as a function over the output of a different network, including all additional variables in its output.
The formalism introduced in (1) imposes the property to be deﬁned only over the output.
It is however not leading to any loss of generality as it would be possible to modify the network and make it “pass-through” inputs or hidden units so as to make them part of the output of the network.
As the problem of PL-NN veriﬁcation has been shown to be NP-complete using a reduction from 3-SAT (Katz et al., 2017a), it is unlikely that any polynomial time algorithm will exist.
Therefore, experimental comparison remains the only approach possible to evaluate the relative advantages of different methods, which we propose to do.
R ELATED WORKS
We start by brieﬂy presenting related work that we don’t include in our comparison as they aren’t capable of performing general veriﬁcation of problem of the form of Problem 1.
Zakrzewski (2001) and Hein & Andriushchenko (2017) propose methods based on the second derivatives of the functions expressed by the networks to guarantee that the output of the network doesn’t change too much around points.
However, this requires the additional assumption that all layers of the networks are twice differentiable, which PL-NN don’t satisfy.
At the other end of the spectrum, Narodytska et al.
(2017) and Cheng et al.
(2017c) use SAT solvers to propose veriﬁcation methods for the specialised case of Binarized Neural Networks (Hubara et al., 2016).
These methods, however, don’t translate to PL-NNs.
In addition to exact formal methods, some other approaches were proposed that don’t provide complete veriﬁcation.
Pulina & Tacchella (2010) proposed a method for veriﬁcation of networks with sigmoid activation function.
The necessary approximation to the smooth non-linearities lead to the incapability of their method to return a decision for certain problems.
Bastani et al.
(2016) studied x 1 [-2, 2] x 2 [-2, 2] a b  - - y - -
Prove that y > −
Network.
We attempt to prove the property that the network output is always greater thanStep xxa in a out b in b out y
[Table]
constraints     -
[Table] [Table]
[Table] [Table]
PL-NN in the context of obtaining robustness guarantees against adversarial examples
but added additional assumptions for scalability reasons, limiting the domain considered to the set of points sharing the same activation pattern than a reference point, effectively circumventing all of the nonlinearities of the network.
Xiang et al.
(2017) and Huang et al.
(2017) both rely on discretisation and perform layer by layer analysis to obtain guarantees over the output, but Huang et al.
(2017) require the user to specify a family of possible changes and Xiang et al.
(2017) over-approximates the output space of each layer, leading to the existence of undecidable properties, making both inadequate for our comparison.
V ERIFICATION M ETHODS
The methods we involve in our comparison all leverage the piecewise-linear structure of PL-NN to make the problem more tractable.
We use the network of Figure 1 throughout as an illustrative example.
All the methods we compare follow the same general principle: given a property to prove, they attempt to discover a counterexample that would make the property false.
This is accomplished by deﬁning a set of variables corresponding to the inputs, hidden units and output of the network, as well as the set of constraints that a counterexample would satisfy.
For the network of Figure 1, the variables would be { x 1 , x 2 , a in , a out , b in , b out , y } and the set of constraints would be:
[Equation]
Here, a in is the input value to hidden unit a , while a out is the value after the ReLU.
Any point satisfying all the above constraints would be a counterexample to the property, as it would imply that it is possible to drive the output to -5 or less.
However, if this problem is unsatisﬁable, no counterexample can exist, which implies that the property is True.
We want to emphasize that our requirements go beyond saying that no counterexample could be found: it is necessary to prove that no counter-examples exists.
The difﬁculty stems from the non-linear constraints of (2c).
We will now explain how each method tackles this problem.
3. M IXED I NTEGER P ROGRAMMING E NCODING
A possible way to eliminate the non-linearities is to encode them with the help of binary variables, which transform the PL-NN veriﬁcation problem into a Mixed Integer Program (MIP).
Lomuscio &
Maganti (2017) and Cheng et al.
(2017b) advocate the use of big-M encoding to achieve this.
For example, the non-linearities of equation (2c) are replaced by
[Equation]
where M a is a sufﬁciently large value.
The binary variable δ a indicates which phase the ReLU is in: if δ a = 0 , the ReLU is blocked and a out = 0 , else the ReLU is passing and a out
= a in .
The problem remains difﬁcult due to the integrality constraint on δ a .
We provide more details in Appendix A on how to handle MaxPooling units similarly.
This approach has some advantages.
As the ﬁnal problem to solve ends up being a MIP, imposing integrality constraints on the inputs comes at no additional cost.
This can prove useful if some input features to the network are known to necessarily be integers (Cheng et al., 2017a).
For other methods, imposing these integer constraints would not be possible: either the proof would be attempted on the relaxed version of the networks or it would have to be done for all possible combination of integer inputs.
As the solving of MIP is NP-hard, the performance of those methods is going to be dependent both on the quality of the solver used and of the encoding.
Cheng et al.
(2017b) proposes several methods to obtain a good encoding by picking appropriate values of M, in order for the quality of the linear relaxation of the MIP problem to be as high as possible.
However, in the end, the problem is still resolved by a general purpose MIP solver and the question remains open whether a solver not knowing more about the speciﬁc structure of the problem can be efﬁcient on challenging benchmarks.
3. R ELUPLEX
Katz et al.
(2017a) presents a procedure named Reluplex to verify properties of Neural Network containing linear functions and ReLU activation unit, functioning as an SMT solver using the splittingon-demand framework (Barrett et al., 2006).
The principle of Reluplex is to always maintain an assignment to all of the variables, even if some of the constraints are violated.
Starting from an initial assignment, it attempts to ﬁx some violated constraints at each step.
It prioritises ﬁxing linear constraints ((2a) and (2b) in our illustrative example) using a simplex algorithm, even if it leads to violated ReLU constraints (2c).
This can be seen in step 1 and 3 of the process shown in Table 1.
If no solution to the problem containing only linear constraints exists, this shows that the counterexample search is unsatisﬁable.
Otherwise, all linear constraints are ﬁxed and Reluplex attempts to ﬁx one violated ReLU at a time, such as in step 2 of Table 1 (ﬁxing the ReLU b ), potentially leading to newly violated linear constraints.
This process not being guaranteed to converge, non-linearities getting split too often are split into two cases.
This generates two new sub-problems, each involving an additional linear constraint instead of the linear one.
The ﬁrst one solves the problem where a out = 0 , the second one where a out = a in .
In the worst setting, the problem will be split completely over all possible combinations of activation patterns, at which point the sub-problems are simple LPs.
3. P LANET
Ehlers (2017a) also proposed an approach based on splitting the problems over the possible phase of the activation function.
Unlike Reluplex, the proposed tool, named Planet, operates by attempting to ﬁnd an assignment to the phase of the non-linearities.
Reusing the notation introduced in Section 3.1, it assigns a value of True or False to each δ a variable, verifying at each step the feasibility of the partial assignment.
As opposed to Reluplex, this has the advantage of being easily extended to networks containing MaxPooling units.
In order to detect incoherent assignments (such as both a and b being in the greater than zero region of the ReLU in the example of Figure 1) faster, the author proposes a global linear approximation to a neural network.
In addition to the existing linear constraints ((2a) and (2b)), the non linear constraints are approximated by sets of linear constraints representing the convex hull of the nonlinearities.
ReLUs are replaced by the set of equations:
[Equation]
where a out and a in are respectively the output and input of the ReLU, and ub in and lb in are precomputed upper and lower bounds of the ReLUs input.
This feasible domain is illustrated in Figure 2a.
MaxPooling units are replaced by the set of constraints:
[Equation]
where in i are the inputs to the MaxPooling unit and lb in i their lower bounds.
A one dimensional cut of this is represented in Figure 2b.
As a consequence, the whole network is approximated by an LP
[Figure] [Figure]
out
[Figure]
that can be efﬁciently queried to detect infeasibility or automatically deducing implied assignments to other δ
i variables.
The values of the lower bounds and upper bounds necessary to deﬁne the constraints are built iteratively by optimizing the corresponding variable, based on the constraints imposed by the previous layers.
Additional heuristics to make infeasibility detection and implied phase inference faster are described in the original paper (Ehlers, 2017a).
Both Reluplex and Planet rely on the splitting mechanism over ReLUs to guarantee progress but their use of it is fundamentally different: while Reluplex (Katz et al., 2017a) drives the search for satisﬁability using a simplex algorithm and splits the ReLU lazily to unlock cases, Planet (Ehlers, 2017a) drives the whole search by eagerly making splits using a SAT solver and making deductions based on those.
Reluplex always maintains an assignment to all variables even though it doesn’t respect all constraints until the end; Planet only maintains assignments to the phase of the nonlinearities.
As those two approaches have never been compared on common benchmarks, it is hard to identify which is the most promising one or the speciﬁc cases in which one method outperforms the other, even though they rely on similar principles.
3. B RANCH AND B OUND OPTIMIZATION FOR VERIFICATION
Veriﬁcation as optimization We now present a different way of approaching the Neural Network veriﬁcation problem.
The whole satisﬁability problem will be transformed into an optimization problem where the decision will be obtained by checking the sign of the minimum.
We will show how any Boolean formula on linear inequalities can be encoded as additional layers at the end of the network.
Assume that the property is a simple inequality: P ( y ) (cid:44)
c T y ≥ b .
In that case, it is sufﬁcient to add to the network a ﬁnal fully connected layer with one output, with weight of c and a bias of − b .
If the global minimum of this network is positive, it indicates that for all y that the original network can output, we have c T y − b ≥
= ⇒ c T y ≥ b , and as a consequence the property is True.
On the other hand, if the global minimum is negative, then it provides a counter-example to the property.
Clauses speciﬁed using OR (denoted by (cid:87) ) can be encoded by using a MaxPooling unit; if the property is P ( y ) = (cid:87)
i (cid:2)
c
Ti y ≥ b i (cid:3) , this is equivalent to max i (cid:0)
c
Ti y − b i (cid:1) ≥ 0 .
Clauses speciﬁed using AND (denoted by (cid:86) ) can be encoded similarly: the property P ( y )
= (cid:86) i (cid:2)
c
Ti y ≥ b i (cid:3) is equivalent to min i (cid:0) c T i y
− b i (cid:1) ≥
⇐⇒ − (cid:0) max i (cid:0) − c T i y
+ b i (cid:1)(cid:1) ≥
As a result, we can formulate any Boolean formula over linear inequalities on the output of the network as a sequence of additional layers, and the veriﬁcation problem would be reduced to a global minimization problem.
Aside from some speciﬁc class of NN (Amos et al., 2017), this remains a hard problem.
The advantage is one of formalism, allowing us to prove the desired properties with a single procedure rather than decomposing the desired property into separate queries as was done in previous work (Katz et al., 2017a).
Finding the exact global minimum, while not necessary for veriﬁcation, will have the advantage of generating a value.
If this value is positive, it will correspond to the margin by which the property is satisﬁed.
When estimating robustness to adversarial examples, existing methods choose to perform a binary search over the maximum radius guaranteeing the absence of adversarial examples.
The optimization process would be a more appropriate formalism here.
Branch and Bound for Optimization Optimization algorithms such as Stochastic Gradient Descent which are the usual workhorses of Deep Learning are not appropriate for this minimization problem.
Despite being capable of converging to good local minima, they have no way of guaran- teeing whether or not a minima is global.
We now present an approach to tackle this problem, based on the Branch-and-Bound method.
Algorithm 1 describes the generic form of the Branch-and-Bound method.
The input domain will be repeatedly split into sub-domains (line 7), over which we will compute lower and upper bounds of the minimum of the output (lines 9-10).
The best upper-bound found so far will serve as a candidate for the global minimum.
Any domain whose lower bound is greater than the current global upper bound can be pruned away as it necessarily won’t contain the global minimum (line 13, lines 15-17).
By iteratively splitting the domains, we will be able to compute tighter lower bounds.
We keep track of the global lower bound on the minimum by taking the minimum over the lower bounds of all sub-domains (line 19).
When the global upper bound and the global lower bound differ by less than a small scalar (cid:15) (line 5), we consider that we have converged.
1: function B A B( net , domain , (cid:15) )
(cid:46)
Global minimum of net on domain to an (cid:15) accuracy 2: global ub ← inf 3: global lb ← − inf 4: domains ← [( global lb , domain )] 5: while global ub − global lb > (cid:15) do 6: ( , dom ) ← pick out ( domains ) 7: [ dom 1 , . .
, dom s ] ← split ( dom ) 8: for i = 1 . . .
s do 9: dom ub ← compute upper bound ( net , dom i ) 10: dom
lb ← compute lower bound ( net , dom i ) 11: if dom ub < global ub then (cid:46)
If found a better upper bound 12: global ub ← dom ub 13: domains ← prune domains ( domains , global ub ) (cid:46) Remove domains with dom lb ≥ global ub 14: end if 15: if dom lb < global ub then 16: domains ← domains dom dom i )] 17: end if 18: end for 19: global lb
} 20: end while 21: return global ub 22: end function
The description of the veriﬁcation problem as optimization and the pseudo-code of Algorithm 1 are generic and would apply to veriﬁcation problems beyond the speciﬁc case of PL-NN.
To obtain a practical algorithm, it is necessary to specify the elementary functions:
• pick out ( domains ) :
Select one of the domains to branch on.
Several heuristics are possible, based on the bounds previously computed on the domains or based on the size of the domains.
• split ( domain ) : Takes as argument a domain and returns a partition of domains [ dom 1 , . .
, dom s ] such that (cid:83) i dom i =
domain
and that ( dom i ∩ dom j ) = ∅ , ∀ i (cid:54) =
j .
Choosing the split function will deﬁne the “shape” of the domains that we are operating on, potentially making the computation of the bounds harder or easier.
• compute { lower , upper } bound ( net , domain ) :
Compute a (lower / upper) bound of the minimum of net over the feasible domain domain .
We want the lower bound to be as high as possible (so that this whole domain can be pruned easily) and the upper bound to be as low as possible (so that we can use it to prune out other regions of the search space).
If we have several approaches to compute bounds, we can employ all of them at once and only keep the tightest.
In our experiments, we use the result of minimising the variable corresponding to the output of the network, subject to the constraints of the linear approximation introduced by Ehlers (2017a) as lower bound and simple random sampling
[Table] [Table]
[Table] [Table]
{ 2, 4, 5 } layers of 2* { 5, 10, 25 } hidden units, 1 output, with a bias of { 0.01, 1, 10 } as upper bound.
As the procedure is generic, any improved algorithm leading to faster or tighter bounds would directly translate to improvements to the veriﬁcation process.
In practice, it is not necessary to run Algorithm 1 to convergence for veriﬁcation.
If a negative global upper bound is found, the corresponding input is a valid counter-example.
Similarly, as soon as the global lower bound goes above zero, we know that the property is veriﬁed.
Now equipped with an understanding of the workings of each method, we are now ready to compare them.
S OLVER AND P ROPERTIES
As each method implements a different strategy and the worst-case analysis always indicate exponential runtimes, these approaches can only be compared experimentally.
We attempt to perform veriﬁcation on three data sets of veriﬁcation properties and report the comparison results.
The dimensions of all of the problems are given in Table 2.
For each of the data sets, we compare four methods using the same protocol.
Each run is done with a timeout of two hours, and a maximum allowed memory usage of 20GB, on a single core of a machine with an i7-5930K CPU and 32GB of RAM.
We will release all code and data necessary to replicate our analysis.
Success rate corresponds to the proportion of properties for which the solver returns the correct answer before timing out or being terminated for using too much memory.
We compare the performances of each method and present the results separately for the cases where the properties are
True or False.
SAT means that the satisﬁability problem for a counter-example was satisﬁable, implying that the property was False.
In this case, the runtime corresponds to the time it took before exhibiting a counter-example to the property.
On the other hand, UNSAT corresponds to the time it took to prove that the problem was infeasible and that no counterexamples to the property could exist.
As methods returning SAT exhibit a counterexample, disagreement between solvers can be easily resolved by evaluating the network over the counterexample and checking the property on the output.
We use this criterion to establish a ground truth for each property, except for Table 5 where we know that the result is UNSAT by construction.
We reported bugs to the original authors of each method every time such a disagreement was detected.
In the case of a timeout, the runtime for the method is counted as the maximum allowed time ( s), even though the actual runtime would be worse.
As a result, the average runtime for methods with low success rate would be worse in practice than reported here.
After ﬁxing the Planet solver, no Out of Memory error was encountered.
To give more insights into the relative performance of each solver, we count the Number of wins , which corresponds to how many times a solver was the fastest to solve a property.
When the relative difference between the runtime of two solvers is less than 1%, we don’t count any win.
Reluplex , based on the version released by the authors (Katz et al., 2017b).
The tool is implemented in C++ and relies on a modiﬁed version of the GLPK library to deal with the Simplex algorithm.
Note that as the tool doesn’t support MaxPooling units out of the box, we automatically convert the
10 − 10 − 10 − 10 − 10 − 10 − 10 −
Relative area 32.
L o w e r b o un d
[Figure]
set 10 − 10 − 10 − 10 − 10 − 10
Relative area − −175000 − −125000 − −75000 − −
L o w e r b o un d
ReApproximatingFixedApproximation
[Figure]
MaxPooling layers into a series of linear layers with ReLU activations.
To do so, we decompose the element-wise maximum into a series of pairwise maximum
[Equation]
and decompose the pairwise maximums as sum of ReLUs:
[Equation]
where lb x 2 is a pre-computed lower bound of the value that x 2 can take.
Planet , based on the version released by the author (Ehlers, 2017b).
The tool is implemented in
C++, using GLPK to solve linear programs and a modiﬁed version of MiniSat to drive the search.
We wrote some software to convert in both directions between the input format of both Reluplex and Planet.
We discovered some memory issues on the original implementation, which we reported to the author and used the ﬁxed version for all experiments.
MIP , that consists on encoding the satisﬁability problem as a Mixed Integer Program, using the “big
M” encoding of the non-linearity.
The exact encoding of MaxPooling and ReLU can be found in
Appendix A.
This method is similar to the one implemented in (Cheng et al., 2017b; Lomuscio &
Maganti, 2017) but due to the lack of availability of open-sourced methods, we reimplemented the approach in Python, using the Gurobi MIP solver.
To choose the value of M for each ReLU, we made use of the linear approximation of Planet.
This leads to better values of M than the dataﬂow analysis discussed by Cheng et al.
(2017b).
Even when compared to their other proposed heuristic for the selection of M , it has several advantages: it encompasses information from all layers of the network rather than from only a few layers; it solves simple LPs rather than MIPs.
BaB , based on the method described in Section 3.4.
Our pick out strategy chooses the domain that currently has the smallest lower bound.
We split the domain in half across its longest edge to generate two new smaller domains as our split method.
We generate upper bounds on the minimum by randomly sampling points on the considered domain, and minimise the linear approximation of the network proposed by (Ehlers, 2017a) as a lower bound.
Our implementation is in Python and uses
Gurobi to solve LPs.
Note that as opposed to the approach taken by Ehlers (2017a) of building a single approximation of the network, we rebuild a new approximation for each sub-domain.
This is motivated by the observation shown in Figure 3 which demonstrate the signiﬁcant improvements it brings, especially for deep networks.
Having now presented the software tools we use, we know report their performance on each of our data set.
A NALYSIS 5. A IRBORNE C OLLISION A VOIDANCE S YSTEM
The Airborne Collision Avoidance System ( ACAS ) data set, as released by Katz et al. (2017a) is a neural network based advisory system recommending horizontal manoeuvres for aircraft in order
to
[Table] [Table]
avoid collisions, based on sensor measurements.
Each of the ﬁve possible manoeuvres is assigned a score by the neural network and the action with the minimum score is chosen.
In the case of complex property to prove, such as “NoAction (CoC) is the minimal score”, their counter-example search is implemented as a series of four satisﬁability problems: “Does there exist a point where the score for { WeakLeft, WeakRight, StrongLeft, StrongRight } is less than the score for NoAction?” while our approach discussed in Section 3.4 would be able to merge all of these satisﬁability problems into a single one.
For our experiments, we use the original strategy of the authors.
Results for the ACAS data are shown in Table 3.
Our proposed method using Branch and Bound performs the best along all criteria.
Compared to the second best method, Reluplex, it is more than an order of magnitude faster at exhibiting counter-examples and more than twice as fast at proving the correctness of True properties.
On this data set which is the one with the largest number of hidden units, Planet doesn’t manage to exhibit any counterexample to a property.
We hypothesise that with such a high number of non-linearities, the strategy of eager assignments to phase of Planet is disadvantaged as opposed to the more lazy approach of Reluplex and BaB. All methods but BaB perform better on UNSAT problem than on SAT ones.
We postulate that this is due to the relatively small dimensionality of the input, which makes random testing capable of easily getting good coverage and discovering counterexamples quickly.
5. C OLLISION D ETECTION
In the CollisionDetection data set, as released by the authors of Planet, the network attempts to predict whether two vehicles with parameterized trajectories are going to collide.
A total ofproperties are extracted from problems arising from a binary search to identify the size of the region around training examples in which the prediction of the network doesn’t change.
Planet is the best performing method on the set of benchmarks that accompanied the release of the tool.
Table 4 shows it being the fastest method for most of the properties.
Conversely, BaB becomes the worst performing method, especially on True properties.
It is however important to note that all solvers ﬁnished signiﬁcantly below the timeout limit, indicating that the data set isn’t extremely challenging.
On this data set, Reluplex classiﬁed three False properties as unsatisﬁable.
We evaluated the counterexamples returned by other solvers using Reluplex’s code and conﬁrmed that they were valid counterexamples.
We reported the issue to the original authors of Reluplex who identiﬁed numerical instabilities as the reason for the discrepancy.
5. T WIN S TREAM
To get a better understanding of what factors inﬂuence the performance of various methods, we propose a novel TwinStream data set, which we generated to have control over different hyper-
[Table]
parameters.
The networks contain two separate streams, where each of the streams has the same architecture, weights, and inputs.
The ﬁnal layer of the network computes the difference between the outputs of the two streams, and add a positive bias term, which we will refer to as the margin.
As a result, the output is always equal to the value of the ﬁnal bias.
On each of those networks, we attempt to prove the true property that the output of the network is positive.
We generate networks with random weights using Glorot initialisation (Glorot & Bengio, 2010), and vary the depth, number of hidden units in each of the stream, number of inputs, and the value of the margin.
Note that as opposed to the other two data sets, the weights aren’t the result of an optimization process and therefore may not be representative of real use-cases.
For 16 out of the 27 networks having a margin of 0.01 and one of the networks having a margin of 1, Reluplex returns a SAT result although the properties are True by construction.
Evaluating the returned counterexamples using Reluplex itself conﬁrms that they are not valid counterexamples, but caused by numerical issues.
We also reported these issues to the authors of Reluplex.
Surprisingly, the generic MIP solver and the Planet solver perform best on this data set.
A possible explanation is that due to the heavy structure present in the problem (each non-linearity having a twin exactly equal to it in the other stream), methods explicitly representing the decision over the phase of the non-linearity inherently have the advantage.
We generate a plot showing the evolution of runtimes for each of the architectural parameters of
TwinStream networks.
Each curve represented corresponds to a ﬁxed conﬁguration of all other parameters.
This allows us to assess the impact of each factor on the performance of solvers.
While MIP (Figure 4b) and Planet (Figure 4c) are relatively insensitive to changes in the number of inputs in the network, Branch-and-Bound (Figure 4a) incurs signiﬁcantly higher run-times as the number of inputs to the network rises.
This can be explained by the fact that MIP and Planet branch over the hidden units whose number is kept constant here, while BaB branches over the dimensions of the input.
On the other hand, when studying the impact of the network depth, MIP and Planet (Figure 5b and 5c respectively) are much more sensible to the higher number of layers, unlike BaB (Figure 5a) who surprisingly performs better on deeper networks.
The analysis of Figure 6 doesn’t reveal any surprising insights: higher number of non-linearities in each layer corresponds to more complex problem, which directly translates to longer runtime for all solvers.
Figure 7 reveals that the expected runtime of Branch-and-Bound is more sensible to the amount by which a property is True, compared to solvers that reason directly other the phases of non-linearities.
This is natural as in the case of BaB, properties with a higher margin corresponds simply to cases where it is possible to stop the counter-example search procedure faster.
C ONCLUSION
The improvement of formal veriﬁcation of Neural Networks represents an important challenge to be tackled before learned models can be used in safety critical applications.
The lack of a shared set of benchmarks between researchers makes it hard to evaluate progress or estimate promising research directions.
We gathered test cases from existing literature, proposed new ones, and evaluated the performance of published methods, which allowed us to surface issues in published tools and offer an informed view of the status of the ﬁeld.
In addition, we proposed a conceptually simple method that offers competitive performance on the most realistic data set available.
Our method is sufﬁciently general to be easily improvable, as any better lower bound will directly translate to faster veriﬁcation.
[Figure] [Figure]
[Figure] [Figure]
[Figure]
A MIP ENCODING DETAILS
We now present the encoding used to convert the non linearity of a neural network into inequality constraints involving binary variables.
In the case of a ReLU activation
[Equation]
we can replace this equation containing the max non-linearity by
[Equation]
where M a is a value such that M a is an upper-bound of a in and − M a is a lower-bound of a in .
For MaxPooling units, we can either make the choice to decompose the MaxPooling into a series of
ReLU and use their linear encoding.
Another solution is to encode the MaxPooling directly.
The constraint
[Equation]
can be replaced by
[Equation]
where M is an upper-bound on all in i and lb i are lower-bounds on each in i .