Traversing the Local Polytopes of ReLU Neural Networks
Abstract
Although neural networks (NNs) with ReLU activation func- tions have found success in a wide range of applications, their adoption in risk-sensitive settings has been limited by the concerns on robustness and interpretability.
Previous works to examine robustness and to improve interpretability par- tially exploited the piecewise linear function form of ReLU
NNs.
In this paper, we explore the unique topological struc- ture that ReLU NNs create in the input space, identifying the adjacency among the partitioned local polytopes and develop- ing a traversing algorithm based on this adjacency.
Our polytope traversing algorithm can be adapted to a wide range of applications related to robustness and interpretability.
As the traversing algorithm explicitly visits all local polytopes, it re- turns a clear and full picture of the network behavior within the traversed region.
The time and space complexity of the traversing algorithm is determined by the number of a ReLU
NN’s partitioning hyperplanes passing through the traversing region.
Introduction & Related Work
Neural networks with rectiﬁed linear unit activation functions (ReLU NNs) are arguably the most popular type of neural networks in deep learning.
This type of network enjoys many appealing properties including better performance than NNs with sigmoid activation (Glorot, Bordes, and Bengio 2011), universal approximation ability (Arora et al.
Lu et al.
2017; Montufar et al.
SchmidtHieber 2020), and fast training speed via scalable algorithms such as stochastic gradient descent (SGD) and its variants (Zou et al. 2020).
Despite their strong predictive power
ReLU NNs have seen limited adoption in risk-sensitive settings (Bunel et al.
These settings require the model to make robust predictions against potential adversarial noise in the input (Athalye et al. 2018;
Carlini and Wagner 2017; Goodfellow,
Shlens and Szegedy 2014; Szegedy et al. 2014).
The alignment between model behavior and human intuition is also desirable (Liu et al.
2019): prior knowledge such as monotonicity may be incorporated into model design and training (Daniels and Velikova 2010; Gupta et al. 2019; Liu et al.
2020; Sharma and Wehrheim 2020); users and auditors of the model may require a certain degree of explanations of the model predictions (Gopinath et al. 2019;
Chu et al.
The requirements in risk-sensitive settings has motivated a great amount of research on verifying certain properties of ReLU NNs.
These works often exploit the piecewise linear function form of ReLU NNs.
In Bastani et al.
the robustness of a network is veriﬁed in very small input region via linear programming (LP).
To consider the nonlinearity of ReLU activation functions, Ehlers (2017)
; Katz et al.
; Pulina and Tacchella (2010, 2012) formulated the robustness veriﬁcation problem as a satisﬁability modulo theories (SMT) problem.
A more popular way to model
ReLU nonlinearality is to introduce a binary variable representing the on-off patterns of ReLU neurons.
Property veriﬁcation can then be solved using mixed-integer programming (MIP) (Anderson et al.
2020; Fischetti and Jo 2017;
Liu et al.
2020; Tjeng, Xiao, and Tedrake 2018; Weng et al.
The piecewise linear functional form of ReLU NNs also creates distinct topological structures in the input space.
Previous studies have shown that a ReLU NN partitions the input space into convex polytopes and has one linear model associated with each polytope (Montufar et al. 2014;
Serra, Tjandraatmadja, and Ramalingam 2018; Croce, Andriushchenko, and Hein 2019; Robinson, Rasheed, and San 2019; Sudjianto et al. 2020; Yang et al. 2020).
Each polytope can be coded by a binary activation code, which reﬂects the on-off patterns of the ReLU neurons.
The number of local polytopes is often used as a measure of the model’s expressivity (Hanin and Rolnick 2019; Lu et al. 2017).
Built upon this framework, multiple studies (Sudjianto et al. 2020;
Yang, Zhang, and Sudjianto 2020; Zhao et al. 2021) tried to explain the behavior of ReLU NNs and to improve their interpretability.
They viewed ReLU NN as a collection of linear models.
However, the relationship among the local polytopes and their linear models was not fully investigated.
When the network’s behavior within some speciﬁc region in the input space is of interest, one can collect all the local polytopes overlapped with the region to conduct analysis.
The methods to collect these polytopes can be categorized into top-down and bottom-up approaches.
The top-down approaches in Xiang, Tran, and Johnson (2017); Yang et al.
(2020) pass the entire region of interest through a ReLU
NN and calculate how the hyperplanes corresponding to the neurons partition the region into local polytopes.
The major drawback of the top-down approach is that the analysis must start after the computationally expensive forward passing is fully ﬁnished.
One the contrary, the bottom-up approaches start from a point of interest inside the region, moving from one local polytope to another while running the analysis, and can be stopped at any time.
Croce and Hein (2018); Croce, Rauber, and Hein (2020) achieved the movement among polytopes by generating a sequence of samples in the input space using randomized local search.
Although being computationally simple, this sample-based method does not guarantee covering all polytopes inside the region of interest.
The most recent work and also the closest to ours is Vincent and Schwager (2021), where polytope boundaries and adjacency are identiﬁed using LP, and the traversing is done directly on the polytopes.
Inthispaper, we explore the topological relationship among the local polytopes created by ReLU NNs.
We propose algorithms to identify the adjacency among these polytopes, based on which we develop traversing algorithms to visit all polytopes within a bounded region in the input space.
Compared with (Vincent and Schwager 2021), our polytope traversing algorithm exploits ReLU NNs’ hierarchical partitioning of the input space to reduce computational overhead and achieve signiﬁcant acceleration for discovering adjacent polytopes.
The thoroughness of our traversing algorithm is proved.
The algorithm is versatile and can be adapted to many applications related to model robustness and interpretability.
Although we focus on ReLU
NN with fully connected layers through out this paper, our polytope traversing algorithm can be naturally extended to other piecewise linear networks such as those containing convolutional and maxpooling layers.
The rest of this paper is organized as follows:
Sectionreviews how polytopes are created by ReLU NNs.
Section 3 introduces two related concepts: the boundaries of a polytope and the adjacency among the polytopes.
Our polytope traversing algorithm is described in Section 4.
Section 5 discusses several applications that our traversing algorithm can be adapted to.
The paper is concluded in Section 6.
The Local Polytopes in ReLU NNs 2. The case of one hidden layer
A ReLU NN partitions the input space R P into several polytopes and forms a linear model within each polytope.
To see this, we ﬁrst consider a simple NN with one hidden layer of
M neurons.
It takes an input x ∈ R P and outputs o ∈ R Q by calculating:
[Equation]
where σ ( x ) [Equation] =
[Equation]
x [Equation] [Equation] For problems with a binary or categorical target variable (i.e. binary or multi-class classiﬁcation), a sigmoid or softmax layer is added after o respectively to convert the convert the NN outputs to proper probabilistic predictions.
The ReLU activation function σ ( · ) inserts non-linearity into the model by checking a set of linear inequalities: w Tm x + b m ≥ 0 , m = 1 , 2 , . .
, M , where w Tm is the m th row of matrix W and b m is the m th element of b .
Each neuron in the hidden layer cre- ates a partitioning hyperplane in the input space with the linear equation w Tm x + b m = 0 .
The areas on two sides of the hyper- plane are two halfspaces .
The entire input space is, therefore, partitioned by these M hyperplanes.
We deﬁne a local polytope as a set containing all points that fall on the same side of each and every hyperplane.
The polytope encoding function (2) uses an element- wise indicator function 1 ( · ) to create a unique binary code c for each polytope.
Since the m th neuron is called “ON” for some x if w Tm x + b m ≥ 0 , the code c also represents the on-off pattern of the neurons.
Using the results of this encoding function, we can express each polytope as an intersection of M halfspaces as in [Equation] 3), where the binary code c controls the directions of the inequalities.
[Equation]
Figure 1.(b) shows an example of ReLU NN trained on a two-dimensional synthetic dataset (plotted in Figure 1.(a)).
The bounded input space is [ − 1 , 1]
and the target variable is binary.
The network has one hidden layer of 20 neurons.
The partition- ing hyperplanes associated with these neurons are plotted as the blue dashed lines.
They form in total 91 local polytopes within the bounded input space.
For a given x , if w Tm x + b m ≥ 0 , the ReLU neuron turns on and passes through the value.
Otherwise, the neuron is off and suppresses the value to zero.
Therefore, if we know the m th neuron is off, we can mask the corresponding w m and b m by zeros and create ˜ W c and ˜ b c that satisfy (5).
The non-linear operation, therefore, can be replaced by the a locally linear operation after zero- masking.
Because each local polytope R c has a unique neuron activation pattern encoded by c , the zero-masking process in (4) is also unique for each polytope.
Here, 1 is a vector of 1s of length p and ⊗ denotes element-wise product.
[Equation]
Within each polytope, as the non-linearity is taken out by the zero-masking process, the input x and output o have a linear relationship:
[Equation]
where ˆ
[Equation]
The linear model associated with polytope R c has the weight matrix ˆ W o c and the bias vector ˆ b o c .
The ReLU NN is now represented by a collection of linear models, each deﬁned on a local polytope R c .
In Figure 1.(b), we represent the linear model in each local polytopes as a red solid line indicating ( ˆ w o c ) T x + ˆ b o c = 0 .
In this binary response case, the two sides of this line have the opposite class prediction.
We only plot the line if it passes through its corresponding polytope.
For other polytopes, the entire polytopes fall on one side of their corresponding class-separating lines and the predicted class is the same within the whole polytope.
The red lines all together form the decision boundary of the ReLU NN and are continuous when passing through one polytope to another.
This is a direct result of ReLU NN being a continuous model.
2. The case of multiple layers
We can generalize the results to ReLU NNs with multiple hid- den layers.
A ReLU NN with L hidden layers hierarchically partitions the input space and is locally linear in each and every
level- L polytope .
Each level- L polytope R L has a unique binary code level
W l x + ˆ b l = 0 , l =
L can be calculated recursively level by level, using the zero masking procedure
[Equation]
We emphasis that ˜ W l , ˜
b l , ˆ W l +1 , and ˆ b l +1 depend on all polytope code up to level l : c 1 c 2 . . .
c l .
The subscription c is dropped to simplify the notations.
At each level l , the encoding function C l ( · ) and the polytope R l expressed as an intersection of (cid:80) lt =1 M t halfspaces can be written recursively as: [Equation] [Equation]
[Equation]
Finally, the linear model in a level- L polytope is:
[Equation]
where ˆ W o = W o ˜ W L , ˆ b
[Equation]
= W o ˜
[Equation]
In many cases, some of the 5 hyperplanes are outside the levelpolytope and, therefore, not creating a new sub-partition.
The hy- perplanes do create new partitions are plotted as the orange dashed lines. straight but are continuous when passing through one polytope to another which is also a result of ReLU being In to tal, this As
Polytope Boundaries and Adjacency
Beyond viewing ReLU NNs as a collection of linear models de- ﬁned on local polytopes, we explore the topological relationship among these polytopes.
A key concept is the boundaries of each polytope.
As shown in (13), each level- l polytope R c with corresponding binary code c = c 1 c 2 . . .
c l is an intersection of (cid:80) lt =1 M t halfspaces induced by a set of inequality constraints.
Two situations can rise among these inequalities.
First, an arbitrary c may lead to conﬂicting inequalities and makes R c an empty set.
This situation can be common when the number of neurons is much larger than the dimension of the input space.
Second, there can be redundant inequalities which means removing them does not affect set R c .
We now show that the non-redundant inequalities are closely related to the boundaries of a polytope.
Deﬁnition 3.1 Let R contains all x ∈ R P that satisfy M linear inequalities: R = { x | g 1 ( x ) ≤ 0 , g 2 ( x ) ≤ 0 , . .
. , g M ( x ) ≤ 0 } .
Assume that R (cid:54) = ∅ .
Let ˜ R contains all x ’s that satisfy M − 1 linear inequalities:
˜ R = { x | g 1 ( x ) ≤ 0 , . .
, g m − 1 ( x ) ≤ 0 , g m +1 ( x ) ≤ 0 , . .
g
M ( x ) ≤ 0 } .
Then the inequality g m ( x ) ≤ 0 is a redundant inequality with respect to (w.r.t.)
R if R =
˜
R .
With the redundant inequality deﬁned above, the following lemma provides an algorithm to identify them.
The proof of this lemma is in the Appendix.
Lemma 3.1 Given a set R = { x | g 1 ( x ) ≤ 0 , . .
. , g M ( x ) ≤ 0 } (cid:54) = ∅ , then g m ( x ) is a redundant inequality if the new set formed by ﬂipping this inequality is empty: ˆ R = { x | g 1 ( x ) ≤ 0 , . .
. , g m ( x ) ≥ 0 ,
[Equation]
We can now deﬁne the boundaries of a polytope formed by a set of linear inequalities using a similar procedure in Lemma3.1.
The concept of polytope boundaries also leads to the deﬁnition of adja- cency.
Intuitively, we can move from one polytope to its adjacent polytope by crossing a boundary.
Deﬁnition 3.2 Given a non-empty set formed by M linear in- equalities:
R = { x | g 1 ( x ) ≤ 0 , . .
. , g M ( x ) ≤ 0 } (cid:54) = ∅ , then the hyperplane g m ( x ) = 0 is a boundary of R if the new set formed by ﬂipping the corresponding inequality is non-empty: ˆ R = { x | g 1 ( x ) ≤ 0 , . .
. , g m ( x ) ≥ 0 , . .
. , g M ( x ) ≤ 0 } (cid:54) = ∅ .
Polytope ˆ R is called one-adjacent to R .
Since for each polytope the directions of its linear inequalities are reﬂected by the binary code, two one-adjacent polytopes must have their code differ by one bit.
Figure 2.(a) demonstrates the ad- jacency among the local polytopes.
The ReLU NN is the same as in Figure 1.(b).
Using the procedure in Deﬁnition 3.2, 4 out of the 20 partitioning hyperplanes are identiﬁed each one
Aswehave shown in the Section 2.2, ReLU NNs create poly- topes level by level.
We follow the same hierarchy to deﬁne the polytope adjacency.
Assume two non-empty level- l polytopes, R and ˆ R , are inside the same level- ( l − 1) polytope, which means their corresponding code c
= c 1 c 2 . . .
c l and ˆ c = c 1 c 2 . . .
ˆ c l only differs at level- l .
We say that polytope ˆ R is a level- l one-adjacent neighbor of R if ˆ c l and c l only differs in one bit.
The condition that c
= c 1 c 2 . . .
c l and ˆ c = c 1 c 2 . . .
ˆ c l only differs at level- l is important.
In this way, the two linear inequalities associated with each pair of bits in c and ˆ c have the same coefﬁcients, and the difference in c l and ˆ c l only changes the direction of the linear inequality.
On the other hand, if the two codes differ at a level l (cid:48) < l , then according to the recursive calculation in (8) and (9), the codes starting from level l (cid:48)
+ 1 will correspond to linear inequalities of different coefﬁcients, leaving our Deﬁnition 3.2 of adjacency not applicable.
Figure 2.(b) demonstrates the hierarchical adjacency among the local polytopes.
The ReLU NN is the same as in Figure 1.(c).
Level-1 polytopes (1 , · ) and (2 , · ) are both (level-1) one-adjacent to (0 , · ) .
Within the level-1 polytope (0 , · ) , Level-2 polytopes (0 , 0) and (0 , 1) are (level-2) one-adjacent to each other.
Similarly, we can identify the level-2 adjacency of the other two pairs (1 , 0) − (1 , 1) and (2 , 0) − (2 , 1) .
Note that in the plot, even thought one can move from polytope (2 , 1) to (0 , 1) by crossing one partitioning hyperplane, we do not deﬁne these two polytopes as adja- cent, as they lie into two different level-1 polytopes.
Polytope Traversing 4. The case of one hidden layer
The adjacency deﬁned in the previous section provides us an order to traverse the local polytopes: starting from an initial polytope R , visiting its all one-adjacent neighbors, then visiting all the neigh- bors’ neighbors and so on.
This algorithm can be viewed as breath-ﬁrst search (BFS) on a polytope graph .
To create this graph, we turn each polytope created by the ReLU NN into a node.
An edge is added between each pair of polytopes that are one-adjacent to each other.
The BFS algorithm uses a queue to keep track the traversing progress.
At the beginning of traversing, the initial polytope is added to an empty queue and is marked as visited afterwards.
In each iteration, we pop out the ﬁrst polytope from the queue and identify all of its one-adjacent neighbors.
Among these identiﬁed polytopes, we add those that have not been visited to the back of the queue and mark them as visited.
The iteration stops when the queue is empty.
The key component of the polytope traversing algorithm is to identify a polytope’s one-adjacent neighbors.
For a polytope R c coded by c of M bits, there are at most M one-adjacent neigh- bors with codes corresponding to ﬂipping one of the bits in c .
Each valid one-adjacent neighbor must be non-empty and can be reached by crossing a boundary.
Therefore, we can check each linear inequality in (3) and determine whether it is a boundary or re- dundant.
Some techniques of identifying redundant inequalities are summarized in Telgen (1983).
By ﬂipping the bits corresponding to the identiﬁed boundaries, we obtain the codes of the one-adjacent polytopes.
Equivalently, we can identify the one-adjacent neighbors by go- ing through all M candidate codes and selecting those corresponding to non-empty sets.
Checking the feasibility of a set constrained by a set of linear inequalities is often referred to as the “PhaseI Problem” of LP and can be solved efﬁciently by modern LP solvers.
During BFS iterations, we can hash the checked codes to avoid checking them repetitively.
The BFS-based polytope travers- ing algorithm is summarized in Algorithm 1.
We now state the correctness of this algorithm with its proof in Appendix.
Theorem 4.1 Given a ReLU NN with one hidden layer of M neu- rons as speciﬁed in (1), Algorithm 1 covers all non-empty local polytopes created by the neural network.
That is, for all x ∈ R P , there exists one R c as deﬁned in (3) such that x ∈ R c and c ∈ S R , where S
R is the result returned by Algorithm 1.
Algorithm 1 visits all the local polytopes created by a ReLU NN within R P .
The time complexity is exponential to the number of neurons, as all 2 M possible activation patterns are checked once in the worst-case scenario.
The space complexity is also exponen- tial to the number of neurons as we hash all the checked activation patterns.
Furthermore, for each activation pattern, we solve a phase-I problem of LP with M inequalities in R P .
Traversing all local polytopes in R P , therefore, becomes intractable for neural networks with a large number of neurons.
Fortunately, traversing in R P is usually undesirable.
Firstly, a neural network may run into extrapolation issues for points out- side the sample distribution.
The polytopes far away from the areas covered by the samples are often considered unreliable.
Secondly, many real-life applications, to be discussed in Section 5, only require traversing within small bounded regions to examine the local behavior of a model.
In the next section, we introduce a technique to improve the efﬁciency when traversing within a bounded region.
4. Polytope traversing within a bounded region
We ﬁrst consider a region with each dimension bounded indepen- dently: l j ≤ x j ≤ u j , j = 1 , 2 , . . .
, P .
These 2 × P linear inequalities creates a hypercube denoted as B .
During the BFS-based polytope traversing, we repetitively ﬂip the direction of one of the
M inequalities to identify the one-adjacent neighbors.
When the bounded region is small, it is likely that only a small number of the M hyperplanes cut through the hypercube.
For the other hyper- planes, the entire hypercube falls onto only one side.
Flipping to the other sides of these hyperplanes would leave the bounded region.
Therefore, at the very beginning of polytope traversing, we can run through the M hyperplanes to identify those cutting through the hypercube.
Then in each neighbor identifying step, we only ﬂip these hyperplanes.
To identify the hyperplanes cutting through the hypercube, we denote the two sides of a hyperplane H and ¯ H :
H = { x | w Tm x + b m ≤ 0 } and ¯ H =
{ x | w Tm x + b m ≥ 0 } .
If neither H ∩ B nor ˆ
H ∩ B is empty, we say the hyperplane w Tm x + b m = 0 cuts through B . H ∩ B and ˆ H ∩ B are both constrained by 2 × P +inequalities, checking their feasibility can again be formulated as a [Table]
Check if R ˆ c
= { x | ( − 1)
ˆ c k (cid:0)
w Tk x + b k (cid:1) ≤ 0 , k
= 1 , 2 . . .
, M } is empty using LP.
Add c to c
14: if then 15:
Append ˆ c to the end of the Q .
Add c to S
17: Return phase-I problem of LP.
We name this technique hyperplane pre- screening and summarize it in algorithm 2.
Hyperplane pre-screening effectively reduces the complexity from O (cid:0) 2 M (cid:1) to O (cid:16)
2 |T | (cid:17) , where |T | is the number of hyperplanes cutting through the hypercube.
The number 2 |T | corresponds to the worst-case scenario.
Since the BFS-based traversing only checks non-empty polytopes and their potential one-adjacent neighbors, the number of activation patterns actually checked can be less than 2 |T | .
In general, the fewer hyperplanes go through B the faster polytope traversing ﬁnishes.
Figure 2.(a) shows traversing the 8 local polytopes within the bounded region.
The ReLU NN is the same as in Figure 1.(b).
Require hyperplanes w Tm x + b m ≤ 0 , m = 1 , 2 , . . .
, M .
Require:
A bounded B e.g { x | l j ≤ x j ≤ u j , j = 1 , 2 , . . . , The lines marked in red are the hyperplanes cutting through the bounded region and are identiﬁed by the pre-screening algorithm.
The evolution of the BFS queue is shown in Figure 2.(c).
The gray arrows show the traversing order.
The colored arrows at the bottom indicate the one-adjacent neighbors added to the queue.
When polytope No.0 is popped from the queue, its one-adjacent neighbors, No.1, 2, 3, and 4, are added to the queue.
Next, when polytope No.1 is popped, its one-adjacent neighbors, No.5 and 6, are added.
Polytope No.0, although as a one-adjacent neighbor to No.1, is ignored since it has been visited.
Similarly, when polytope No.2 is popped, only one of its one-adjacent neighbors, No. 7, is added, since all others have been visited (including those in the queue).
The algorithm ﬁnished after popping Polytope No.8 as no new polytopes can be added and the queue is empty.
All 8 local polytopes in the bounded region are traversed.
Because B is bounded by a set of linear inequalities, the correct- ness of BFS-based polytope traversing as stated in Theorem 4.1 can be easily extended to this bounded traversing case.
It can be proved by showing that for any two non-empty polytopes overlapped with
B , we can move from one to another by repetitively ﬁnding a one- adjacent neighbor within B .
We emphasis that the correctness of
BFS-based polytope traversing can be proved for any traversing region bounded by a set of linear inequalities.
This realization is critical to generalize our results to the case of ReLU NNs with multiple hidden layers.
Furthermore, as any closed convex set can be represented as the intersection of a set of (possibly inﬁnite) half- spaces, the correctness of BFS-based polytope traversing is true for any closed convex B .
4. Hierarchical polytope traversing in the case of multiple hidden layers
The BFS-based polytope traversing algorithm can be generalized to ReLU NNs with multiple hidden layers.
In section 2.2, we described how a ReLU NN with L hidden layers hierarchically par- tition the input space into polytopes of L different level.
Then in section3, we showed the adjacency of level- l polytopes is condi- tioned on all of them belonging to the same level- ( l − 1) polytope.
Therefore, to traverse all level- L polytopes, we need to traverse all level- ( L − 1) polytopes and within each of them traversing the sub-polytopes by following the one-adjacent neighbors.
The procedure above leads us to a recursive traversing scheme.
Assume a ReLU NN with L hidden layers and a closed convex traversing region B .
Starting from a sample x ∈ B , we traverse all level-1 polytopes using the BFS-based algorithm.
Inside each level-1 polytope, we traverse all the contained level-2 polytopes, so on and so forth until we reach the level-L polytopes.
As shown in (13), each level- l polytope is constrained by (cid:80) lt =1 M t linear inequalities, the way to identify level- l one-adjacent neighbors is largely the same as what we have described in Section 4.1.
Two level- l one-adjacent neighbors must have the same (cid:80) l − 1 t =1 M t linear inequalities corresponding to c 1 c 2 . . .
c l − 1 , and have one of the last M l inequalities differ in direction, so there are M l cases to check.
We can use hyperplane pre-screening at each level of travers- ing.
When traversing the level- l polytopes within in a level- ( l − 1) polytope R l − 1 , we update the bounded traversing region by taking the intersection of R l − 1 and B .
We then screen the M l partitioning hyperplanes and only select those passing through this update traversing region.
The BFS-based hierarchical polytope traversing algorithm is summarized in Algorithm 3.
The correctness of this algorithm can be proved based on the results in Section 4.2, which guarantees the thoroughness of traversing the level- l polytopes within in any level- ( l − 1) polytope.
Then the overall thoroughness is guaranteed because each level of traversing is thorough.
We state the result in the following theorem.
Theorem 4.2 Given a ReLU NN with L hidden layers and a closed convex traversing region B .
Algorithm 3 covers all
non-empty level- L polytopes created by the neural network that overlap with B .
That is, for all x ∈ B , there exists one R c as deﬁned in (13) such that x ∈ R c and c ∈ S R , where S R is the result returned by Algorithm 3.
Figure 2.(b) shows traversing the 6 local polytopes within the bounded region.
The ReLU NN is the same as in Figure 1.(c).
The evolution of the hierarchical BFS queue is shown in Figure 2.(d).
The level-1 BFS queue is shown vertically while the level-2 BFS queue is shown horizontally.
Starting from level-1 polytope (0 , · ) , the algorithm traverses the two level-2 polytopes inside it (line 10 in Algorithm 3).
It then identiﬁes the two (level-1) one-adjacent neighbors of (0 , · ) : (1 , · ) and (2 , · ) .
Every time a level-1 polytope is identiﬁed, the algorithm goes into it to traverse all the level-2 polytopes inside (line 36).
At the end of the recursive call, alllocal polytopes in the bounded region are traversed.
Require: A ReLU NN with L hidden layers.
region topes.
2:3: function HIERARCHICAL TRAVERSE( x , l ) 4:
Initialize an empty queue Q l for BFS at level l .
Initialize an empty set S l c to store all checked level- l codes.
Calculate x ’s initial polytope code c recursively using (12).
if l ==
L then 8:
Add c to S R 9: else 10:
HIERARCHICAL TRAVERSE( x , l +1) 11:
if l > 1 then 12:
Get the level- ( l − 1) polytope code speciﬁed by the front segment of c : c 1: l − 1 = c 1 c 2 . . .
c
l − 1 .
Use c 1: l − 1 to get the level- ( l − 1) polytope R l − 1 c as in (13).
14: else 15:
R 0 c
=
R P 16:
Form the new traversing region B l − 1 = B ∩ R l − 1 c .
Append the code segment c l to the end of the Q l .
Add the code segment c l to S c .
Get the M l hyperplanes associated with c l .
Pre-Screen the hyperplanes associated with c l using Algorithm 2 with bounded region B l − 1 .
Collect the pre-screening results T .
22: while Q l is not empty do 23:
Pop the ﬁrst element in the front of BFS queue: c l =
Q l . pop () .
24: for m ∈ T do 25:
Create a candidate polytope code ˆ c l by ﬂipping one bit in c l : ˆ c lm = 1 − c lm and ˆ c lk = c lk ∀ k
(cid:54)
= m .
26: if ˆ c l / ∈ S c then 27:
Get set R ˆ c
= { x | ( − 1) ˆ c k (cid:16) (cid:104)
ˆ w lk , x (cid:105) + ˆ b lk (cid:17) ≤ 0 , k
= 1 , 2 . .
, M l } 28:
Check if R ˆ c B l 1 is empty using LP.
Add ˆ c l to S c .
30: if R ˆ c ∩ B l − 1 (cid:54) = ∅ then 31:
Append ˆ c l to the end of the Q l .
32: if l ==
L then 33:
Add ˆ c = c 1 c 2 . . .
ˆ c l to S R 34: else 35:
Find a point ˆ x ∈ R ˆ c ∩ B l −36:
HIERARCHICAL TRAVERSE( l
37:38: HIERARCHICAL TRAVERSE( x ,1) 39: Return S R .
The Applications of Polytope Traversing
The biggest advantage of the polytope traversing algorithm is its ability to be adapted to solve many different problems of practical interest.
Problems such as local adversarial attacks, searching for counterfactual samples, and local monotonicity veriﬁcation can be solved easily when the model is linear.
As we have shown in Sections 2.2, the local model within each level- L polytope created by a ReLU NN is indeed linear.
By aggregating the results from the linear model in each polytope, our algorithm provides a way to analyze the behavior within the entire traversing region.
The aforementioned applications have been traditionally solved using MIP (Anderson et al. 2020;
Fischetti and Jo 2017; Liu et al.
2020; Tjeng, Xiao, and Tedrake 2018; Weng et al.
Our algo- rithms based on polytope traversing have several advantages.
First, our method exploits the topological structure created by ReLU NNs and fully explains the model behavior in small neighborhoods.
For the 2 M cases created by a ReLU NN with M neurons, MIP eliminates the searching branches using branch-and-bound.
Our method, on the other hand, eliminates the searching branches by checking the feasibility of the local polytopes and their adjacency.
Since a small traversing region often covers a limited number of polytopes, our algorithm has short running time when solving local problems.
Second, since our algorithm explicitly identiﬁes and visits all the polytopes, the ﬁnal results contain not only the optimal solution but also the whole picture of the model behavior, providing explainability to the often-so-called black-box model.
Last but probably the most important, our algorithm is highly versatile and ﬂexible.
Within each local polytope, the model is linear, which is often the simplest type of model to work with.
Any analysis that one runs on a linear model can be transplanted here and wrapped inside the polytope traversing algorithm.
Conclusion
We explored the unique topological structure that ReLU NNs cre- ate in the input space; identiﬁed the adjacency among the partitioned local polytopes; developed a traversing algorithm based on this adjacency; and proved the thoroughness of polytope traversing.
Our polytope traversing algorithm could be extended to other piecewise linear networks such as those containing convolutional or maxpooling layers.
Acknowledgments
The authors would like to thank Lin Dong, Linwei Hu, Rahul Singh, and Han Wang from Wells Fargo, and Sihan Zeng from
Georgia Institute of Technology for their valuable inputs and feed- back on this project.
Appendix 8. Proof of Lemma 3.
Lemma 8.
Given a set R = { x | g 1 ( x ) ≤ 0 , . .
. , g M ( x ) ≤ 0 } (cid:54) = ∅ , then g m ( x ) is a redundant inequality if the new set formed by ﬂipping this inequality is empty: ˆ R = { x | g 1 ( x ) ≤ 0 , . .
. , g m ( x ) ≥ 0 , . .
. , g M ( x ) ≤ 0 } = ∅ .
Proof: Let ˜ R be the set formed by removing inequality g m ( x ) ≤ 0 : ˜ R = { x | g 1 ( x ) ≤ 0 , . . .
, g m − 1 ( x ) ≤ 0 , g m +1 ( x ) ≤ 0 , . .
. , g M ( x ) ≤ 0 } .
Then ˜ R
=
R ∪ ˆ R .
If ˆ R = ∅ , then R = ˜
R and the inequality g m ( x ) ≤ 0 satisﬁes Deﬁnition 3.1.
(cid:3)
Note the other direction of Lemma 3.1 may not hold.
One exam- ple is when identical inequalities appear in the set: both inequalities in R = { x | g 1 ( x ) ≤ 0 , g 2 ( x ) ≤ 0 } are redundant by deﬁnition if g 1 ( · ) =
g 2 ( · ) .
However, the procedure in Lemma3.1 will not identify them as redundant.
8. Proof of Theorem 4.
Theorem 8.2 Given a ReLU NN with one hidden layer of M neu- rons as speciﬁed in (1), Algorithm 1 covers all non-empty local polytopes created by the neural network.
That is, for all x ∈ R P , there exists one R c as deﬁned in (3) such that x ∈ R c and c ∈ S R , where S
R is the result returned by Algorithm 1.
Proof: Since each partitioning hyperplane divide R P into two halfspaces, all 2 M activation patterns encoded by c covers the entire input space.
We construct a graph with 2 M nodes, each representing a possible polytope code.
Some the nodes may correspond to an empty set due to conﬂicting inequalities.
For each pair of nonempty polytope that are one-adjacent to each other, we add an edge to their corresponding nodes.
What left to prove is that any pair of non-empty polytopes are connected.
W.l.o.g.
assume two nodes with code c and ˆ c that differ only in the ﬁrst K bits.
Also assume the polytopes R c and R ˆ c are both non-empty.
We will show that there must exist a non-empty polytope R ˜ c that is one-adjacent to R c with code ˜ c different from ˆ c in one of the ﬁrst K bits.
As a result, ˜ c is now one bit closer to ˆ c .
We prove the claim above by contradiction.
Assuming claim is not true, we ﬂip any one of the ﬁrst K bits in R c , and the corresponding polytope R ˜ c
k must be empty.
By Deﬁnition 3.1, the inequality ( − 1) c m (cid:0) w Tm x + b m (cid:1) ≤ 0 , m = 1 , 2 , . . .
, K must all be redundant, which means they can be removed from the set of constraints (Telgen 1982, 1983):
[Equation]
The derived relationship in (15) plus the assumption that all R ˜ c k must be empty lead to the conclusion that R ˆ c = ∅ , which contradict with the non-empty assumption.
Therefore, for any two non-empty polytopes R c and R ˆ c , we can create a path from R c to R ˆ c by iteratively ﬁnding an intermediate polytope whose code is one bit closer to ˆ c .
Since the polytope graph covers all input space and all non-empty polytopes are connected, BFS guarantees the thoroughness of traversing.
[Equation]