An Efﬁcient Framework for Computing Tight Lipschitz Constants of Neural Networks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Lipschitz constants are connected to many properties of neural networks, such asrobustness, fairness, and generalization.
Existing methods for computing Lipschitzconstants either are computationally inefﬁcient or produce loose upper bounds.
In this paper, we develop an efﬁcient framework for computing the ` 1 local
Lipschitz constant of a neural network by tightly upper bounding the norm of
Clarke Jacobian.
We view the computation for Clarke Jacobian by chain rule asa higher-order backward computational graph.
On this graph, we adopt boundpropagation with linear relaxation to obtain provable bounds, and we derive tightlinear relaxations for speciﬁc nonlinearities in Clarke Jacobian that previous workscould not tightly bound.
We further tighten our bounds by Branch-and-Bound(BaB) when time budget allows.
Experiments show that on tiny models, ourmethod produces comparable bounds compared to exact methods that cannot scaleto slightly larger models; on larger models, our method efﬁciently produces tighterresults than existing relaxed or naive methods, and our method scales to muchlarger practical models that previous works could not handle.
We also demonstratea potential application of our method for provable monotonicity analysis.
Introduction
Lipschitz constants are important for characterizing many properties of neural networks, includingrobustness [35, 15, 37, 47, 17], fairness [10, 19], generalization [2] and explanation [12].
In order tomore precisely characterize the local behavior of a network, we consider local Lipschitz constantswhich are only required to hold for a small local region.
Intuitively they characterize how fast theoutput of the network changes between any two input within the region.
It is challenging to exactly and efﬁciently compute local Lipschitz constants.
Naive approaches suchas computing the product of the induced norm for all layers cannot capture local information andtypically produce vacuous bound that are not useful in practice.
On the other hand, Jordan & Dimakis[18] used mixed-integer programming (MIP) to compute exact local Lipschitz constants for small
ReLU networks (e.g., up to a few hundred neurons), but solving a MIP is often too inefﬁcient and thisapproach cannot scale to slightly larger networks.
Another line of works focus on computing a relaxedupper bound for better scalability compared to MIP.
Bound propagation methods [43, 46, 33] whichcan compute guaranteed lower or upper bounds for neural network veriﬁcation have been appliedto bound the Jacobian [42, 47], which can produce local Lipschitz constant. Bound propagationcomputes the linear relationship between layers, where nonlinearities in networks are relaxed intolinear bounds, e.g., nonlinear activations can be bounded by two lines.
However, when bounding thegradient of ReLU activation in the chain rule for computing Jacobian, in nontrivial cases, existingbound propagation methods [42, 47] degenerate to loose interval bounds [16, 28, 14] that only use bounding lines with zero slopes.
Moreover, most of these works only used small toy models andcannot feasibly handle larger practical networks.
In this paper, we aim to efﬁciently compute relatively tight local Lipschitz constants.
We formulatethe problem as upper bounding the norm of the Clarke Jacobian by the chain rule; and we viewthe computation for this norm as a higher-order backward computational graph augmented to theoriginal forward graph of the network, which allows us to develop a scalable and ﬂexible frameworkfor practical neural networks.
Our contributions can be summarized as follows:
First, we generalize existing bound propagationmethods originally used for neural network veriﬁcation, to a higher-order backward computationalgraph for bounding Clarke Jacobian.
Second, on the backward graph, we propose tight linearrelaxations for Clarke gradients in the chain rule, while previous works degenerate to loose intervalbounds in nontrivial cases.
When time budget allows, we further tighten our bounds via Branchand-Bound, for a trade-off between tightness and time cost.
Lastly, our experiments show that ourmethod efﬁciently produces tightest local Lipschitz constants compared to other relaxed methods,and is much more efﬁcient than the exact MIP method.
Moreover, while previous works commonlyused small toy models, our method can scale to much larger models including practical convolutionalneural networks (CNN) that previous works could not handle.
We also demonstrate an application ofour method for provably analyzing the monotonicity of neural networks.
Related Work2.1 Computing Lipschitz Constants
For neural networks, a loose global Lipschitz constant (upper bound) can be computed by theproduct of layer-wise induced norms [35].
Based on computing this product, Virmaux & Scaman[38] considered the effect of activation and approximated upper bounds which, however, are notguaranteed; Gouk et al. [13] used a power method to handle convolutional layers.
LipSDP [11] usedsemideﬁnite programming (SDP) to compute tighter bounds.
Global Lipschitz constants can be muchlooser than local Lipschitz constants as they have to hold even for distinct input points, and theycannot characterize the local behavior of neural networks.
Thus we focus on local Lipschitz constantsin this paper.
To compute local Lipschitz constants, LipMIP [18] used mixed integer programming (MIP) tocompute exact results, which is costly and cannot scale to slightly larger models.
LipOpt [22] usedpolynomial optimization, but their method is still inefﬁcient and limited to smooth activations, notincluding the widely used ReLU.
Avant & Morgansen [1] derived layer-wise analytical bounds butonly on a simpliﬁed deﬁnition for Lipschitz constants, which cannot characterize Lipschitzness in theentire local region.
Several other works used methods essentially based on interval bound propagation [16, 28, 14] whichcomputes an interval of lower and upper bounds for each element in the network or the chain rule for
Jacobian.
Among them, RecurJac [47], improved from FastLip [42], has a recursive procedure totighten bounds in some special cases but still uses interval bounds in nontrivial cases; and a concurrentwork [23] handled non-smooth input perturbations.
However, interval bounds are known to be veryloose [48].
Bhowmick et al.
[3] combined interval bounds with branch-and-bound to compute exactlocal Lipschitz constants, which is inefﬁcient due to loose interval bounds.
In contrast, we proposetight linear relaxation for computing local Lipschitz constants with bound propagation.
2.2 Certiﬁed Bounds for Neural Networks
Computing sound local Lipschitz constants is also closely related to neural network veriﬁcationwhich computes certiﬁed output bounds.
Some works computed the exact bounds by satisﬁabilitymodulo theory [20] or mixed integer linear programming [36].
For more efﬁcient veriﬁcation, boundpropagation with convex relaxations has been proposed [43, 40, 46, 42, 33, 30].
In addition, Branchand-Bound can be used to compute tighter certiﬁed bounds Katz et al. Bunel et al.
[ et al. [ et al et al. [ branching activations or input bounding
Background
3.1 ReLU Network
Suppose f ( x ) is a neural network given input x 2 R d .
For the simplicity of presentation, we mainly assume the neural network is feedforward with ReLU activation throughout the paper, but our method can be applied to general network architectures and activations as will be discussed in Section 4.
Suppose the network has n layers, it takes input h 0 ( x ) =
x and then computes
[Equation]
where i 2 [ n ] means 1  i  n , z i ( x ) is the pre-activation output of the i -th linear layer with weight
W i and bias b i , and h i ( x ) is the output after activation  ( · ) .
This formulation also supports CNNssince convolutional layers are also linear.
3.2 Lipschitz Constant
The ( ↵ ,  ) -Lipschitz constant of a network f ( x ) over an open set
X 2 R d is deﬁned as:
[Equation]
If f is smooth and ( ↵ ,  ) -Lipschitz continuous over X , the Lipschitz constant can be computed byupper bounding the norm of Jacobian, i.e., L ( ↵ ,  ) ( f, X ) =
sup x 2 X kr f
( x ) k ↵ ,  , where kr f ( x ) k ↵ , is the matrix norm induced by vector norms, i.e., kr f ( x ) k ↵ ,  = sup k z k ↵ =1 kr f ( x ) z k  .
Sinceneural networks with non-smooth ReLU activation are non-smooth functions, we consider Clarke Jacobian (generalized Jacobian) [7] instead, which is deﬁned as the convex hull of lim i !
1 r f ( x i )for any sequence { x i }
i =1 such that every f ( x i ) is differentiable at x i .
We denote the Clarke Jacobian 100 at x as @ f ( x ) , and then:
[Equation]
Global Lipschitz constant, which considers the supremum over X = R d , needs to guarantee Eq.
(1)even for distant x 1 , x 2 , and thus it can be loose and cannot capture the local behavior of the network.
We focus on local Lipschitz constants, where X is a small local region.
Speciﬁcally, we consider
X =
B p ( x 0 , ✏ ) :
= { x : k x  x 0 k p  ✏ } which is a small ` p -ball around x 0 , and then the local 105 Lipschitz constant only needs to hold for input near x 0 , where x 0 can be data points from cleandatasets. When ✏ is large enough to cover any valid input, local Lipschitz constants will becomeglobal Lipschitz constants.
3.3 Backward Bound Propagation
To bound the Clarke Jacobian, we will use bound propagation which is originally used to compute thecertiﬁed output bounds of neural networks.
We adopt a backward bound propagation [46, 43, 33, 44]which typically propagates the linear relationship between the output layer to be bounded and allthe previous layers it depends on.
Suppose we want to compute certiﬁed bounds for output layerh n ( x ) w.r.t.
all x from a small domain B p ( x 0 , ✏ ) .
Starting from the output layer, we take A n = I ,and then h n ( x ) =
A n h n ( x ) .
For every i 2 [ n ] , suppose h
i
x ) can be bounded by linear functionsw.r.t.
h i  1 ( x ) parameterized by P
i , P i , q i , q i (the bounds elementwise hold): P
[Equation]
Then for every i = n, n  1 , · · · , 1 in order, A i h i ( x ) can be recursively bounded by substituting 117 h i ( x ) with Eq.
[Equation]
where [ · ] + stands for taking positive elements from the matrix or vector, and vice versa for [ · ]  .
A i  1 119 stands for the coefﬁcients of the linear relationship propagated from layer i to layer i  1 , and
c i is a 120 bias term produced at this layer.
By this recursive procedure in a backward manner and accumulating c i i 2 [ n ]) , eventually bounds are propagated to the input layer as h n ( x )  A 0 x +
P ni =1 c i , which 122 directly represents the linear relationship between the output layer and input.
We can eliminatex from A 0 x to obtain ﬁnal bounds by considering the speciﬁc domain of x .
In particular, whenx 2 B p ( x 0 , ✏ ) ( p  1) , for every j -th dimension of h n ( x ) , we can use the fact that [ A 0 ] j, : x  125 [ A 0 ] j, : x 0 + ✏ k [ A 0 ] j, : k q , where [ A 0 ] j, : stands for the j -th row in A 0 , q =1 / (1  1 /p ) , and q -normis the dual-norm of p -norm.
All the bounds here provably hold for any x 2 B p ( x 0 , ✏ ) , and lowerbounds can also be obtained by a similar bound propagation.
Parameters in Eq.
(3) can be obtained by considering the relation between h i ( x ) and h i  1 ( x ) .
Wecan ﬁrst obtain bounds l i  z i ( x )  u i using bound propagation by viewing z i ( x ) as the output layer.
Then the linear relaxation bounds h
i ( x ) =
( z i ( x )) with linear functions of z i ( x ) :
[Equation]
For every neuron j , if [ l i ] j  0 or [ u i ] j  0 ,  ( z i ( x ))
=
z i ( x ) or  ( z i ( x ))
=
respectively isalready linear, and then we can simply take [ s i ]
j [ z i ( x )]
j +[ t i ] j
=
[ s i ] j [ z i ( x )]
j +[ t i ] j =  ( z i ( x )) .
Otherwise when [ l i ] j < 0 < [ u i ] j , as illustrated in Figure 1, the upper bound can be the line passing
ReLU activation at [ z i ( x )]
j = [ l i ] j and [ z i ( x )]
j = [ u i ] j respectively, i.e., [ s i ] j [ z i ( x )]
j + [ t i ] j =  ([ u i ] j )   ([ l i ] j )
[ u i ] j  [ l i ] j  ( z i ( x )  [ l i ] j ) +  ([ l i ] j ) ; the lower bound can be any line with a slope between0 and 1 and a bias of 0, i.e., 0  [ s i ] j  1 , [ t i ] j = 0 , where [ s i ]
j can be viewed as a parameteroptimized with an objective of tightening output bounds [27, 45, 29].
To satisfy Eq.
(3), we can take:
[Equation]
Methodology
We present our method in this section.
We ﬁrst present a novel view to formulate the local Lipschitzconstant computation as a higher-order backward computational graph, and use bound propagation toobtain the bounds of the Clarke Jacobian.
We discuss our methods for tightly bounding non-linearitiesin Clarke Jacobian, including the the norm of Clarke Jacobian and the Clarke gradient of activation.
Finally, we use Branch-and-Bound to further tighten the bounds if time budget allows.
4.1 Clarke Jacobian for ReLU Networks
In ReLU networks, the ReLU activation for a single neuron is deﬁned as  ( x ) = max { x, 0 } .
Itis non-smooth at x =0 .
When computing the Lipschitz constant, we consider Clarke gradient [7]:when x < 0 , @ ( x ) = { 0 } ; when x > 0 , @ ( x ) = { 1 } ; and when x = 0 , @ ( x ) =
For the i -thlayer (1  i<n ) , we use a diagonal matrix  i ( x ) to denote the Clarke gradient of ReLU activationfor all the neurons in the layer, i.e.,  i ( x ) [-
-]=
diag { @ ([ z i ( x )] j ) } , where @ ([ z i ( x )] j ) is the Clarkegradient of the j -th neuron.
Chain rule can be used to compute the Clarke Jacobian for the whole ReLU network, where Clarke gradient is used for ReLU activations.
Denote J i ( x ) = @ f ( x ) @
h i  1 ( x ) as the Clarke Jacobian w.r.t.
the ( i  1) -th layer in the chain rule, and we have
[Equation]
[Table]
The backward graph relies on the pre-activation values ( z 1 ( x ) , · · · , z n  1 ( x ) ) in the forward pass to compute the Clarke gradient of activations.
Linear relaxation for two categories of nonlinearities in the backward graph (nodes in purple), norm ( k J 1 ( x ) k ) and applying Clarke gradient (  0 ) are detailed in Figure 2, Figure 4 and Figure 5.
We also perform branch-and-bound (BaB) to tighten the bounds, where we branch on pre-activation nodes in the forward graph (nodes in green).
and ﬁnally J 1 ( x ) 2 @ f ( x ) is the Clarke Jacobian of the network.
As we want to compute the Lipschitzconstant, by Eq. (2), we upper bound k J 1 ( x ) k ↵ ,  w.r.t. all x 2 X . Given the forward pass computation of the original network, we can formulate a computational graphfor k J 1 ( x ) k ↵ ,  in a backward pass accordingly as shown in Figure 3.
We augment the originalforward graph with the backward graph, and the backward graph has a dependency on the outputof the forward graph due to  1 ,  2 , · · · ,  n  1 .
We build the augmented graph by traversing the 159 forward graph, similar to automatic differentiation for neural network training.
The computationalgraph formulation for Clarke Jacobian allows our method to be easily generalized to different networkarchitectures, with a bound propagation framework for general neural networks [44], as long as thelinear relaxation for all the involved nonlinearities is well derived.
We use the bound propagation framework described in Section 3.3 to upper bound k J 1 ( x ) k ↵ ,  .
Onthe backward graph, when bounds are propagated to J i ( x ) , we use e
A i to denote the coefﬁcientmatrix, and we use e c i to denote the bias term (in contrast to A i and c i in Eq.
and then wehave: 8 i 2 [ n ] , k J 1 ( x ) k ↵ ,   e
A i J i ( x ) +
P i  1 j = e c j .
As intermediate bounds for z i ( x ) are neededfor bounding the forward graph, we also need intermediate bounds L i  J i ( x )  U i ( 8 x 2 X ) for 168 the backward graph.
And there are particularly two categories of nonlinearities to be tackled onthe backward graph: 1) Norm in k J 1 ( x ) k ↵ ,  ; 2)
J i +1 ( x )  i ( 8 i 2 [ n  1]) in the chain rule as Eq.
170 (7), which applies the Clarke gradient of ReLU  i ( x ) on the later layer’s Clarke Jacobian.
Wehandle them in Section 4.2 and Section 4.3 respectively.
Then bounds can be propagated startingat k J 1 ( x ) k ↵ ,  , to J 1 ( x ) , J 2 ( x ) , · · · , J n ( x ) in order.
Bounds are eventually propagated to J n ( x ) ask J 1 ( x ) k ↵ ,   e
A n J n ( x ) +
P i  1 j = e c j , where J n ( x ) can be substituted by J n ( x ) =
W n (Eq.
(7)),which yields ﬁnal bounds on k J 1 ( x ) k ↵ ,  .
4.2 Norm of Clarke Jacobian
Norm of Clarke Jacobian is the ﬁrst nonlinearity we would encounter during backward boundpropagation, as it is the last node in the backward graph (see Figure 3).
In this paper, we mainlyconsider ` 1 -norm with ↵ =
= 1 for simplicity. When f ( x ) is a K -way classiﬁer and f ( x ) 2 R k ,we have J 1 ( x ) 2 R K ⇥ d , and its norm is computed as k J 1 ( x ) k 1 = max
 k  K
P dj =
[ J 1 ( x )] kj   ,where K rows in J 1 ( x ) can be bounded separately, and after that we can aggregate results on K rowsby taking the max .
And sometimes we may only care about one particular target class.
Thus, for thesimplicity of the following analysis, we assume K = 1 as we can handle one row in J 1 ( x ) at eachtime, but they can still be batched in implementation.
Then Clarke Jacobian J i ( x ) ( i 2 [ n ]) and itsbounds L i , U i can be viewed as row vectors, and we will use [ · ] j to denote the j -th element.
By atight linear relaxation for | · | as shown in Figure 2 (only the upper bound is needed), we have thefollowing bound for k J 1 ( x ) k 1 :
Proposition 1.
Suppose 8 j
[ d ] , [ L 1 ] j  [ J 1 ( x )] j  [ U 1 ] j ( 8 x 2 X ) , the following bound holds: [Equation]
e [Equation] e c 0 , where [ e
A 1 ] j =
[ U 1 ] j
[ L 1 ] j  [ U 1 ] j  [ L 1 ] j , [ e c 0 ] j =
[ e
A 1 ] j [ L 1 ] j +  [ L 1 ] j  .
We prove it in Appendix C.1, and this propagates bounds from the norm of Clarke Jacobian to J 1 ( x ) .
4.3 Clarke Gradient of the Activation
For each layer i 2 [ n  1] in the chain rule for computing the Clarke Jacobian as Eq.
J i +1 ( x )  i ( x )is involved in addition to ﬁxed weight matrix W i .
i ( x ) is determined by pre-activation value
z i ( x )from the forward graph.
For the j -th neuron in the layer, [ J i +1 ( x )  i ( x )]
j =
[ J i +1 ( x )]
j [  i ( x )] jj ,when [ u i ]
j < 0 or [ l i ] j > 0 , we have [  i ( x )]
jj =0 and [  i ( x )]
jj =1 respectively ﬁxed, and thenfor these special cases [ J i +1 ( x )  i ( x )]
j is linear w.r.t.
[ J i +1 ( x )]
j .
Otherwise, [ J i +1 ( x )  i ( x )] j isnonlinear since both [ J i +1 ( x )]
j and [  ( x )]
jj may vary given different x 2 X .
We then focus onrelaxing the nonlinearity [ J i +1 ( x )]
j [  i ( x )]
jj given [  i ( x )]
jj = [0 , 1] .
in layer i , when [  i ( x )]
jj = [0 , 1] is uncertain, we plot [ J i +1 ( x )  i ( x )]
j as a group of functions w.r.t.
[ J i +1 ( x )]
j (violet area).
In this example,  1 . 5  [ J i +1 ( x )] j  1 . 5 .
Our linear relaxation (dashed blue lines) is much tighter than interval bounds (dashed green lines) by Zhang et al.
[Figure]
each Clarke gradient.
It is the internal computation for boxes in Figure 3.
Two ReLU nodes are inserted as per Proposition 2.
In bound propagation, the  0 node takes intermediate bounds from z i ( x ) to decide whether bounds are directly propagated from J i +1 ( x ) (no relaxation needed), or indirectly via  ( J i +1 ( x )) and   (  J i +1 ( x )) (with relaxation), for each neuron respectively.
As illustrated in Figure 4, the solid blue line and solid red line are the exact upper and lower boundrespectively for [ J i +1 (
x )  i ( x )] j , which are piecewise linear w.r.t.
[ J i +1 ( x )]
j .
The most nontrivialcase appears when [ L i +1 ( x )]
j < 0 <
[ U i +1 ( x )]
j , where exact bounds for [ J i +1 ( x )  i ( x )]
j arenonlinear given the input region.
We relax the exact bounds into linear bounds that can be handledby bound propagation with linear relaxation.
Previous works such as RecurJac [47] have not tightlybounded this linearity in the nontrivial case and commonly degenerate to interval bounds (dashedgreen lines with zero slopes in the ﬁgure, which are loose bounds).
In contrast, we propose a tight linear relaxation (dashed blue and red lines in Figure 4).
We observethat the exact upper bound is essentially a ReLU function of J i +1 ( x ) (solid blue line in Figure 4),and the exact lower bound can be viewed as a ReLU function ﬂipped both horizontally and vertically(solid red line in Figure 4).
Formally, we have:
Proposition 2.
For the j -th neuron in layer i ( i 2 [ n  1]) , if [  i ( x )]
jj = [0 , 1] , we have:
[Equation]
which can be rewritten using ReLU activation:
[Equation]
Its correctnesscan be easily veriﬁed by considering the sign of [ J i +1 ( x )]
j as shown in Appendix C.2.
We then decompose [ J i +1 ( x )  i ( x )]
j into two ReLU activations (Eq.
(9)), as illustrated in Figure 5.
ReLU can be further relaxed by Eq.
(5), as derivied in Appendix C.3, which yields:
Proposition 3.
For the j -th neuron in the i -th layer ( i 2 [ n  1] ), given [  i ( x )]
jj = [0 , 1] and[ L i +1 ]
j  [ J i +1 ( x )]
j  [ U i +1 ]
j , we have the following relaxation:[
[Equation]
where [
[Equation]
In the following theorem, we show that relaxation in Proposition 3 is the tightest linear relaxation:
[-+
-]Theorem 4.
For the j -th neuron in the i -th layer ( i 2 [ [-e s i +
 j =-]n  1] ), and any s, t, s, t satisfying
[Equation]
we have [Equation]
s i
+1 ] j , [ e t i +1 ] j ) {+= ) 
+} [ L i +1 ]
j <J < [ U i +1 ] j , sJ + t < [ e s i +1 ]
j J + [ e t i +1 ] j , ( s, t ) 6 =
([ e s i
+1 ]
j , [ e t i +1 ] {+j ) = ) 
+}
[ L i +1 ]
j <J <
[ U i +1 ]
j , sJ + t >
[ e s i
+1 ]
j J + [ e t i +1 ] j , where [ e s i
+1 ]
j , [ e t i +1 ]
j , [ e s i
+1 ]
j , [ e t i +1 ]
j are by Proposition 3.
This theorem states that our proposed linear relaxation is provably tighter than any other valid linearrelaxation. We prove this theorem in Appendix C.4.
4.4 Branch and Bound
Formulating the backward computational graph also enables the use of Branch-andBound (BaB) to further tighten the bounds if time budget allows.
We denote C={ ( l 1 , u 1 ) , ( l 2 , u 2 ) , · · · ( l n  1 , u n  1 ) } as the domain of all pre-activation bounds of ReLU neurons 229 before BaB, not including ReLU from Eq.
We use
C to denote a pool of domains that we currentlyhave, and initially C =
{C 0 } .
BaB aims to recursively split domains in C into smaller subdomains
C 1 , C 2 , · · · , where C 0 = C
S
C
S · · · ) , and bounds for each domain can be computed respectively 232 using bound propagation, since linear relaxation can generally be tighter with smaller domains.
At each iteration, we take B domains in C with loosest bounds, where B is the batch size for BaB(we take all if there are fewer domains).
For each domains, we choose one layer
ˆ i and one ReLUneuron ˆ j in layer
ˆ i on the forward graph, such that the neuron has uncertain Clarke gradient for thisdomain, i.e., [ l ˆ i ]
ˆ j  0  [ u ˆ i ] ˆ j .
We branch this domain into two subdomains with intermediate bounds([ l ˆ i ] ˆ j ,  ˜ ✏ ) and (˜ ✏ , [ l ˆ i ] ˆ j ) respectively, where ˜ ✏ is a sufﬁciently small value (e.g., 10  9 ).
Thereby, the Clarke gradient for ReLU becomes certain in each subdomain (0 and 1 respectively) with otherneurons unchanged.
We compute the bounds for the new subdomains, and we repeat this processuntil reaching a time limit or there is no domain with uncertain Clarke gradient to branch.
We use a heuristic score to decide which neuron to branch for a given domain, by estimating thepotential improvement to the bounds after the branching [5].
Here for branching a ReLU neuronon the forward graph, we estimate the potential improvement on the bounds of the backward graph. In particular, suppose neuron ˆ j in layer
ˆ i is branched, the gap between the upper and lower boundin the liner relaxation is eliminated, which can tighten the ﬁnal bound.
We estimate the area of thegap before branching (the gap between the blue dashed line and the red dashed line in Figure 4)as 12 ([ U ˆ i +1 ]
ˆ j  [ L ˆ i +1 ] ˆ j ) 2 , multiplied by the coefﬁcient [ e
A ˆ i ] ˆ j from the bound propagation as theheuristic score.
We thereby branch neurons with highest scores.
Experiments
In our experiments, we focus on evaluating the tightness of local Lipschitz bounds and the efﬁciencyof computation.
There are three parts: 1) We compare the tightness and efﬁciency on relativelysmall models and synthetic data, to accommodate slow baselines; 2) We also use practical imagedatasets with larger models.
3) Finally, we demonstrate an application of our method on analyzingthe monotonicity of neural networks.
Additional experimental details are provided in Appendix B.
Baselines We compare our methods to the following baselines: NaiveUB upper bounds Lipschitzconstants simply by multiplying the induced norm of each layer, which can scale to arbitrarily largemodels but the bound is often vacuous; LipMIP [18] computes exact local Lipschitz constants using
MIP, which compute the exact (tightest) bound but can only work for very small models.
LipSDP [11]computes upper bounds of ` 2 local Lipschitz constants using SDP, and we convert their ` 2 resultsinto upper bounds for the ` 1 -case by a p d factor.
LipBaB [3] uses loose interval bounds withbranch-and-bound.
RecurJac [47] uses bound propagation with interval bounds in nontrivial casesbut has tightened bounds for some special cases.
All the baselines do not support CNN in theirimplementation, and we need to convert CNN models to equivalent MLP models for baselines. [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
5.1 Comparison on a Synthetic Dataset
We follow Jordan & Dimakis [18] and train several small models on a synthetic dataset.
We comparedifferent methods on models with varying width as shown in Table 1, and we also show results withvarying depth and ✏ in Appendix A.1.
Since some of these models are already large for LipMIP, weset a time limit of 1000 seconds, and we report the upper bound it returns at timeout.
For our BaB,we set a timeout of 60 seconds and stop further branching at timeout.
Without BaB, our method is very efﬁcient while it already outperforms baselines on tightness exceptfor LipMIP on the smallest MLP.
Compared to LipMIP, for the smallest MLP on which LipMIP doesnot timeout, our result without BaB is very close to their exact result; and for other relatively largermodels, LipMIP cannot produce exact results within the timeout and our result is much tighter thanthe upper bounds by LipMIP.
And LipSDP fails to yield results within the timeout on many models.
Although LipBaB has BaB to tighten bounds, their results are still loose within the time budget,since their BaB is based on loose interval bounds and is thus inefﬁcient.
RecurJac is a relativelyefﬁcient baseline but still returns looser bounds compared to our method.
Our BaB further improvesthe tightness of our bounds, and our method with BaB consistently produces tightest results within areasonable time budget, compared to all the baselines.
5.2 Comparison on Image Datasets
[Table]
To further evaluate the scalability of different methods to relatively large models, we then conductexperiments on image datasets including MNIST [25], CIFAR-10 [21], and TinyImagenet [24].
Weuse a 3-layer MLP and a 4-layer CNN on MNIST, and we use 4-layer and 6-layer CNN models on
CIFAR-10 and TinyImageNet.
All the models here are too large for LipSDP; LipMIP and LipBaBcan only handle the MLP model on MNIST, and they cannot return any valid upper bound for the
[Table]
[Table]
CNN models within a reasonable time budget; RecurJac can handle the CNN model on MNIST, butother CNN models on CIFAR-10 and TinyImageNet are still too large for RecurJac after converted toequivalent MLP models.
In contrast, our formulation for the backward computational graph allowsus to utilize the efﬁcient bound propagation for convolutional layers from Xu et al.
We do notenable BaB to save time cost on these large datasets.
We show results in Table 2 and Table 3.
Ourresults are much tighter compared to the baselines even if we do not enable BaB, and BaB can furthertighten results.
More importantly, our method can scale to larger CNN models on CIFAR-10 and TinyImageNet that previous works could not handle.
We present additional results in Appendix.
5.3 An Application on Monotonicity Analysis
We further use an application to demonstrate that our proposed method can be used to check themonotonicity of neural networks.
Given a network, we aim to check whether the network’s output isa monotonic function w.r.t.
each input feature.
Monotonicity can be preferred properties in somereal-world scenarios [32, 8, 26, 34].
In this experiment, we adopt the Adult dataset [4], with a binaryclassiﬁcation task about predicting income level.
There are only several continuous features, and weaim to check the monotonicity of income level w.r.t.
age, education number, capital gain, capitalloss, and hours per week.
We train an MLP model on this dataset and verify its monotonicity on theﬁrst 100 test examples.
Signs of Clarke Jacobian can be used to check monotonicity [31], and weachieve this by checking the bounds of Clarke Jacobian.
For each continuous feature j we concern,we set an input domain X j where only this particular feature can be varied between the minimumand maximum possible values of this feature in the dataset, while we keep other features ﬁxed.
And then we obtain bounds on the Clarke Jacobian of the output class for high income level, as
L 1  J 1 ( x )  U 1 ( 8 x 2 X j ) .
Then if [ L 1 ] j > 0 , the predicted conﬁdence on high income level 308 is monotonically increasing w.r.t.
feature j , and vice versa.
We count the percentage of examplesthat satisfy each type of monotonicity respectively.
We show results in Table 4.
It suggests that thepredicted conﬁdence on high income level tends to be monotonically increasing w.r.t.
age, educationnumber, and hours per week, which is reasonable.
Results also suggest that the models learn littlemonotonicity on capital gain or loss features.
Conclusion
In this paper, we propose an efﬁcient framework to compute tight local Lipschitz constants forneural networks by upper bounding the norm of Clarke Jacobian.
We model the computation for the
Clarke Jacobian as a backward computational graph.
On the backward graph, we conduct boundpropagation and propose tight linear relaxation for nonlinearities in Clarke Jacobian.
We furthertighten bounds with branch-and-bound when time budget allows.
Experiments show that our methodefﬁciently produces much tighter results compared to existing relaxed methods and can scale to largerconvolutional networks on which previous works cannot handle.
However, it remains challengingto handle even more large-scale models such as those for ImageNet [9], and it will be interestingfor future work to further extend the scalability.
Lastly, we also use our method to analyze themonotonicity of neural networks as a potential application.