N EURAL N ETWORK A PPROXIMATION BASED ON H AUSDORFF DISTANCE OF T ROPICAL Z ONOTOPES
Anonymous authors
Paper under double-blind review
A BSTRACT
In this work we theoretically contribute to neural network approximation by providing a novel tropical geometrical viewpoint to structured neural network compression.
In particular, we show that the approximation error over a bounded sphere of two neural networks with ReLU activations and one hidden layer depends on the Hausdorff distance of the respective zonotopes.
This theorem comes as a ﬁrst step towards a purely geometrical interpretation of neural network approximation.
Based on this theoretical contribution, we propose geometrical methods that employ the K-means algorithm to compress the fully connected parts of ReLU activated deep neural networks.
We analyze the error bounds of our algorithms theoretically based on our approximation theorem and evaluate them empirically on neural network compression.
Our experiments follow a proof-of-concept strategy and indicate that our geometrical tools achieve improved performance over relevant tropical geometry techniques and can be competitive against non-tropical methods.
I NTRODUCTION
Tropical geometry (Maclagan & Sturmfels, 2015) is a mathematical ﬁeld based on algebraic geometry and strongly linked to polyhedral and combinatorial geometry.
It is built upon the tropical semiring which originally refers to the min-plus semiring ( R min , ∧ , +) , but may also refer to the max-plus semiring (Cuninghame-Green, 2012; Butkovi ˇ c, 2010).
In our work, we follow the convention of the max-plus semiring ( R max , ∨ , +) which replaces the classical operations of addition and multiplication by max and sum respectively.
These operations turn polynomials into piecewise linear functions and, thus, having direct applications in neural networks.
Tropical mathematics have a wide range of applications including game theory (Akian et al., 2012), dynamical systems on weighted lattices (Maragos, 2017), ﬁnite state transducers (Theodosis &
Maragos, 2018; 2019) and convex regression (Maragos & Theodosis, 2020; Tsilivis et al., 2021).
Recently, there has been remarkable theoretical impact of tropical geometry in the study of neural networks and machine learning (Maragos et al., 2021).
Zhang et al.
(2018) prove the equivalence of ReLU activated neural networks with tropical rational mappings.
Furthermore, they show that zonotopes may serve as a computational mean for applications such as computing the bound shown in (Montúfar et al., 2014) regarding the number of the network’s linear regions.
In a similar context,
Charisopoulos & Maragos (2018) compute an upper bound to the number of linear regions of convolutional and MaxOut layers and propose a randomized algorithm for linear region counting.
Moreover, Charisopoulos & Maragos (2017) study the morphological perceptron tropically and propose a training algorithm using the convex-concave procedure.
Also, Dimitriadis & Maragos (2021) examine the training, pruning and shape properties of Morphological neural networks.
Pruning (Blalock et al., 2020) or, generally, compressing neural networks gained interest in recent years due to the surprising capability of reducing the size of a neural network without compromising performance.
As tropical geometry explains the mathematical structure of neural networks, pruning may be revisited tropically.
Indeed, Alfarra et al.
(2020) study the lottery ticket hypothesis (Frankle & Carbin, 2018) from the tropical geometrical perspective and propose an unstructured compression algorithm based sparsifying the zonotope matrices of the network.
Further, Smyrnis et al.
(2020) propose a novel tropical division algorithm that applies to neural network minimization.
A generalization of this method applies to multiclass networks (Smyrnis & Maragos, 2020).
Contributions In our work, we contribute to structured neural network approximation from the mathematical viewpoint of tropical geometry:
• We establish a novel bound on the approximation error of two neural networks with ReLU activations and one hidden layer based on tropical geometry. To prove this we bound the error of two tropical polynomials via the Hausdorff distance of their respective Extended Newton Polytopes, which for the neural network case have the structure of zonotopes.
• Motivated by the geometrical bound, we derive two geometrical neural network compression methods that are based on zonotope reduction and employ K-means algorithm.
Our algorithms are applied on the fully connected layers of ReLU activated neural networks.
• Our geometrical algorithms are analyzed both theoretically and experimentally.
The theoretical evaluation is based on the bounding theorem of neural network approximation error by the Hausdorff distance of the zonotopes.
On the experimental part, we examine the performance of our algorithms on retaining the accuracy of convolutional neural networks when applying compression on their fully connected layers.
Proofs, illustrations and experimental speciﬁcations are included in the supplementary material.
B ACKGROUND ON T ROPICAL G EOMETRY
We study tropical geometry from the viewpoint of the max-plus semiring ( R max , ∨ , +) which is deﬁned as the set R max
= R ∪ {−∞} equipped with two operations ( ∨ , + ).
Operation ∨ stands for max and + stands for sum.
In max-plus algebra we deﬁne polynomials in the following way.
Tropical polynomials A tropical polynomial f in d variables x = ( x 1 , x 2 , ..., x d ) T is deﬁned as the function
[Equation]
(1) where A is a ﬁnite collection of vectors in R d and c a is the corresponding monomial coefﬁcient in R max
=
R ∪ {−∞} .
The set of such polynomials constitutes the semiring R max [ x ] of tropical polynomials.
Note that each term a T x + c a corresponds to a hyperplane in R d .
We thus call the vectors
a ∈
A the slopes of the tropical polynomial, and c a the respective bias .
We allow slopes to take any value in R d and rather than Z d ≥ 0 , as it is normally the case for regular polynomials.
These polynomials are also referred to as signomials (Dufﬁn & Peterson, 1973) in the literature.
Polytopes Polytopes have been studied extensively (Ziegler, 2012; Grünbaum, 2013) and occur as a geometric tool for ﬁelds such as linear programming and optimization.
They also have an important role in the analysis of neural networks.
For instance, Zhang et al.
; Charisopoulos & Maragos (2018) show that linear regions of neural networks correspond to vertices of polytopes.
Thus, the counting of linear regions reduces to a combinatorial geometry problem.
In what follows, we explore this connection of tropical geometry with polytopes.
Consider the tropical polynomial deﬁned in (1).
The Newton Polytope associated to f ( x ) is deﬁned as the convex hull of the slopes of the polynomial
[Equation]
Furthermore, the Extended Newton Polytope of f ( x ) is deﬁned as the convex hull of the slopes of the polynomial extended in the last dimension by the corresponding bias coefﬁcient.
[Equation]
The following proposition computes the Extended Newton Polytope that occurs when a tropical operation is applied between two tropical polynomials.
It will allow us to compute the polytope representation corresponding to a neural network’s hidden layer.
Proposition 1.
(Zhang et al., 2018; Charisopoulos & Maragos, 2018) Let f, g ∈ R max [ x ] be two tropical polynomials .
Then for the Extended Newton Polytopes it is true that
[Equation]
ENewt ( f )
ENewt ( g )
ENewt ( f ∨ g )
ENewt ( f + g )
UF ( ENewt ( f ∨ g )) (0 , 0 , 0) (2 , 1 , 1) (1 , 0 , 0) (0 , 1 , 0) (0 , 0 , 1) (0 , 0 , 0) (2 , 1 , 1) (1 , 0 , 0) (0 , 1 , 0) (0 , 0 , 1) (1 , 0 , 0) (0 , 1 , 0) (0 , 0 , 1) (3 , 1 , 1) (2 , 2 , 1) (2 , 1 , 2)
Here ⊕ denotes Minkowski addition.
In particular, for two sets A, B ⊆ R d it is deﬁned as
[Equation]
Corollary 1.
This result can be generalized to any ﬁnite set of polynomials using induction.
Example 1.
Let f, g be two tropical polynomials in 2 variables, such that
[Equation]
The tropical operations applied to these polynomials give
[Equation]
Fig. 1 illustrates the Extended Newton Polytopes of the original and the computed polynomials.
The Extended Newton Polytope provides a geometrical representation of a tropical polynomial.
In addition, it may be used to compute the values that the polynomial attains, as Proposition 2 indicates.
Proposition 2.
(Charisopoulos & Maragos, 2018) Let f ∈ R max [ x ] be a tropical polynomial in d variables.
Let UF ( ENewt ( f )) be the points in the upper envelope of ENewt ( f ) , where upward direction is taken regarding the last dimension of R d +1 .
Then for each a ∈
A there exists a linear region of f on which f ( x ) = a T x + c a if and only if ( a T , c a ) is a vertex of UF ( ENewt ( f )) .
Example 2.
Using the polynomials from Example 1 we compute a reduced representation for f ∨ g .
[Equation]
Indeed, the remaining terms correspond to the vertices of UF ( ENewt ( f ∨ g )) shown in Fig.
2. T ROPICAL G EOMETRY OF N EURAL N ETWORKS
Tropical geometry has the capability of expressing the mathematical structure of ReLU activated neural networks.
We review some of the basic properties governing neural networks and introduce notation that will be used in our analysis.
For this purpose, consider the ReLU activated neural network of Fig.
2 with one hidden layer.
Network tropical equations The network consists of an input layer x
= ( x 1 , ..., x d ) , a hidden layer f
= ( f 1 , ..., f n ) with ReLU activations, an output layer v = ( v 1 , ..., v m ) and two linear layers deﬁned by the matrices A, C respectively.
As illustrated in Fig.
2 we have A i, : = (cid:0) a Ti , b i (cid:1) for the ﬁrst linear layer and C j, : =
( c j 1 , c j 2 , ..., c jn ) for the second linear layer, as we ignore its biases.
Furthermore, the output of the i − th component of the hidden layer f is computed as
[Equation]
We deduce that each f i is a tropical polynomial with two terms.
It therefore follows that ENewt ( f i ) is a linear segment in R d
+1 .
The components of the output layer may be computed as
[Equation]
Tropical rational functions Functions p j , q j are both linear combinations of { f i } with positive coefﬁcients, which implies that they are tropical polynomials.
We conclude that every output node v i can be written as a difference of two tropical polynomials, which is deﬁned as a tropical rational function .
This indicates that the output layer of the neural network of Fig. 2 is equivalent to a tropical rational mapping .
In fact, this result holds for deeper networks, in general, as demonstrated by Theorem 1.
Theorem 1.
(Zhang et al., 2018) Consider a ReLU activated deep neural network as a mapping
F : R d → R m .
Then F is equivalent to a tropical rational mapping.
xx...
x k . ..
x d ff... f i . ..
f n vv... v j .. .
v m a
i 1 a ia
i k a i d c j 1 c jc ji c j n bbb i b n
It is not known whether a tropical rational function r ( x ) admits an efﬁcient geometric representation that determines its values { r ( x ) } for x ∈ R d , as it holds for tropical polynomials with their polytopes in Proposition 2.
For this reason, we choose to work separately on the polytopes of the tropical polynomials p j , q j .
Zonotopes Zonotopes are deﬁned as the Minkowski sum of a ﬁnite set of line segments.
They are a special case of polytopes that occur as a building block for our network.
These geometrical structures provide a representation of the polynomials p j , q j in (3) that further allows us to build our compression algorithms.
We use the notation P j , Q j for the Extended Newton Polytopes of tropical polynomials p j , q j , respectively.
Notice from (3) that for each component v j of the output p j , q j are written as linear combinations of tropical polynomials that correspond to linear segments.
Thus P j and Q j are zonotopes.
We call P j the positive zonotope, corresponding to the positive polynomial p j and Q j the negative one.
Zonotope Generators Each neuron of the hidden layer represents geometrically a line segment contributing to the positive or negative zonotope.
We thus call these line segments generators of the zonotope.
The generators further receive the characterization positive or negative depending on the zonotope they contribute to.
It is intuitive to expect that a zonotope gets more complex as its number of generators increases.
In fact, each vertex of the zonotope can be computed as the sum of vertices of the generators, where we choose a vertex from each generating line segment, either 0 or c ji (cid:0) a Ti , b i (cid:1) .
We summarize the above with the following extension of (Charisopoulos & Maragos, 2018).
Proposition 3.
P j , Q j are zonotopes in R d +1 .
For each vertex v of P j there exists a subset of indices
I + of { 1 , 2 , ..., n } with c ji > 0 , ∀ i ∈ I + such that
v
=
(cid:80) i ∈ I
+ c ji (cid:0) a T i , b i (cid:1) .
Similarly, a vertex u of
Q j can be written as u = (cid:80) i ∈
I − c ji (cid:0) a Ti , b i (cid:1) where I − corresponds to c ji < 0 , ∀ i ∈ I − .
A PPROXIMATION OF T ROPICAL P OLYNOMIALS
In this section we present the core theorem of our contribution that bounds the error of the approximation of two neural networks following the architecture of Fig. 2.
To achieve this we ﬁrst need to derive a bound on the error of approximating a simpler functional structure, namely tropical polynomials.
The motivation behind the geometrical bounding of the error of the polynomials is Proposition 2.
It indicates that a polynomial’s values are determined at each point of the input space by the upper envelope of its Extended Newton Polytope.
Therefore, it is expected that two tropical polynomials with approximately equal Extended Newton Polytopes, should attain similar values.
We formally prove this intuitive result and the bound of the error depends on how close the corresponding Extended Newton Polytopes are. The metric we use to deﬁne this notion is the Hausdorff distance .
Hausdorff distance The distance of a point u ∈ R d from the ﬁnite set V ⊂ R d is denoted by either dist ( u , V ) or dist ( V , u ) and computed as min {(cid:107)
u − v (cid:107) , v ∈ V} which is the Euclidean distance of u from its closest point v ∈ V .
The Hausdorff distance H ( V , U ) of two ﬁnite point sets V , U ⊂ R d is deﬁned as
[Equation]
Let P, ˜
P be two polytopes with their vertex sets denoted by V P , V ˜ P respectively.
We deﬁne the
Hausdorff distance H (cid:16) P, ˜ P (cid:17) of the two polytopes as the Hausdorff distance of their respective vertex sets V P , V ˜ P .
Namely,
[Equation]
Clearly, the Hausdorff distance is a metric of how close two polytopes are to each other.
Indeed, it becomes zero when the two polytopes coincide.
According to this metric, the following bound on the error of tropical polynomial approximation is derived.
Proposition 4.
Let p, ˜ p ∈ R max [ x ] be two tropical polynomials and let P =
ENewt ( p ) , ˜ P =
ENewt (˜ p ) .
Then,
[Equation]
where B = { x ∈ R d : (cid:107) x (cid:107) ≤ r } is the hypersphere of radius r , and ρ = √ r 2 + 1 .
The above proposition enable us to handle the more general case of neural networks with one hidden layer, that are equivalent with tropical rational mappings.
Applying repeatedly Proposition 4 to each tropical polynomial corresponding to the networks, we get the following bound.
Theorem 2.
Let v, ˜ v ∈ R max [ x ] be two neural networks with architecture as in Fig.
With ˜ P j , ˜ Q j we denote the positive and negative zonotopes of ˜ v .
The following bound holds. [Equation] Remark 1. The reason we choose to compute the error of the approximation on a bounded hypersphere B is twofold.
Firstly, the unbounded error of linear terms always diverges to inﬁnity and, secondly, in practice the working subspace of our dataset is usually bounded, e.g. images.
N EURAL N ETWORK C OMPRESSION A LGORITHMS
Compression problem formulation The tropical geometry theorem on the approximation of neural networks enables us to derive compression algorithms for ReLU activated neural networks.
Suppose that we want to compress the neural network of Fig. 2 by reducing the number of neurons in the hidden layer, from n to K .
Let us assume that the output of the compressed network is the tropical rational map ˜ v =
(˜ v 1 , ..., ˜ v m ) .
Its j − th component may be written as ˜ v j ( x ) = ˜ p
j ( x ) − ˜ q j ( x ) where using Proposition 3 the zonotopes of ˜ p j , ˜ q j are generated by ˜ c ji (˜ a Ti , ˜
b i ) , ∀ i .
The generators need to be chosen in such a way that ˜ v j ( x ) ≈ v j ( x ) for all x ∈ B .
Due to Theorem 2 it sufﬁces to ﬁnd generators such that the resulting zonotopes have H (cid:16) P j , ˜ P j (cid:17) , H (cid:16) Q j , ˜ Q j (cid:17) as small as possible ∀ j .
We thus formulated neural network compression as a geometrical zonotope approximation problem.
Our approaches Approximating a zonotope with fewer generators is a problem known as zonotope order reduction (Kopetzki et al., 2017).
In our case we approach this problem by manipulating the zonotope generators c ji (cid:0) a Ti , b i (cid:1) , ∀ i, j 1 .
Each of the algorithms presented will create a subset of altered generators that approximate the original zonotopes.
Ideally, we require the approximation to hold simultaneously for all positive and negative zonotopes of each output component v j .
However, this is not always possible, as in the case of multiclass neural networks, and it necessarily leads to heuristic manipulation.
Our ﬁrst attempt to tackle this problem applies the K-means algorithm to the xxfffffffv c 1 c 2 cc 4 cc 6 cbbbbbbb 7 { a ji } i ∈ (1 , 2) ,j ∈ (1 ,..., 7) (a) Original network.
c 1 ( a T 1 ,b 1 ) c 2 ( a T 2 ,b 2 ) c 3 ( a T 3 ,b 3 ) c 4 ( a T 4 ,b 4 ) c 5 ( a T 5 ,b 5 ) c 6 ( a T 6 ,b 6 ) c
( a T 7 ,b 7 ) (
[Figure]
Original
[Figure]
˜ c 1 (˜ a T 1 , ˜ b 1 ) ˜ c
(˜ a T 2 , ˜ b 2 ) ˜ c 4 (˜ a T 4 , ˜ b 4 ) ≡ c 7 ( a T 7 ,b 7 ) ˜ c
3 (˜ a T 3 , ˜ b 3 ) (c)
Resulting zonotopes.
xx˜ f˜ f˜ f˜ fv + + − − ˜ c 1 ˜ b˜ c 2 ˜ b˜ c 3 ˜ b˜ c 4 ˜ b 4 { ˜ c
j ˜ a ji } i ∈ (1 , 2) ,j ∈ (1 ,..., 4) (d) Minimized network.
positive and negative generators, separately. This method is restricted on applying to single output neural networks.
Our second approach further develops this technique to multiclass neural networks. Speciﬁcally, it utilizes K-means on the vectors associated with the Neural Paths passing from a node in the hidden layer, as we deﬁne later.
The algorithms we present refer to the neural network of Fig. 2 with one hidden layer, but we may repeatedly apply them to compress deeper networks.
4. Z ONOTOPE A PPROXIMATION
Zonotope K-means The ﬁrst compression approach is geometrical and utilizes K-means, which is inherently a vector compression algorithm.
Consider the case of a single output neural network, e.g as in Fig. 2 but with m = 1 , and wish to reduce its hidden layer size from n to K neurons.
We use the notation c
i , i = 1 , ..., n for weights of the second linear layer, connecting the hidden layer with the output node.
Algorithm 1 is presented and an example of its execution can be found in Fig.
Algorithm 1:
means Compression Split generators into positive (cid:8) c i (cid:0) a Ti , b i (cid:1) :
c i > 0 (cid:9) and negative (cid:8) c i (cid:0) a Ti , b i (cid:1) :
c i < 0 (cid:9) .
Apply K-means for K 2 centers, separately for both sets of generators and receive (cid:110)
˜
c i (cid:16) ˜ a Ti , ˜
b i (cid:17) :
˜ c i > 0 (cid:111) , (cid:110) ˜
c i (cid:16) ˜ a Ti , ˜
b i (cid:17) :
˜ c i < 0 (cid:111) as output.
Construct the ﬁnal weights.
For the ﬁrst linear layer, the weights and the bias which correspond to the i − th neuron become the vector ˜ c i (cid:16) ˜ a Ti ,
˜
b i (cid:17) .
The weights of the second linear layer are set to 1 for every hidden layer neuron where ˜ c i (cid:16) ˜ a Ti ,
˜ b i (cid:17) occurs from positive generators and − 1 , elsewhere.
Proposition 5.
Zonotope K-means produces a compressed neural network with output ˜ v satisfying
[Equation]
where K is the total number of centers used in both K-means, δ max is the largest distance from a point to its corresponding cluster center and N max is the maximum cardinality of a cluster.
The above proposition provides an upper bound for the approximation error of resulting network of
Zonotope K-means from the original one.
This bound increases as we use fewer cluster centers in
K-means.
If we use K ≈ n centers the error approaches 0 , because then δ max → 0 , N max → 1 , and if
K ≈
the error gets a ﬁxed value depending on the magnitude of the weights of the linear layers.
4. M ULTIPLE Z ONOTOPE A PPROXIMATION
The exact positive and negative zonotope approximation performed by Zonotope K-means algorithm has a main disadvantage: it can only be used in single output neural networks.
Indeed, suppose that we want to employ the preceeding algorithm to approach the zonotopes of each output in a multiclass neural network.
That would require 2 m separate executions of K-means which are not necessarily consistent.
For instance, it is possible to have c j 1 i > 0 and c j 2 i < 0 for some output components v j 1 , v j 2 .
That means that in the compression procedure of v j 1 , the i − th neuron belongs to the positive generators set, while for v j 2 , it belongs to the negative one.
This makes the two compressions incompatible.
Moreover, the drawback of restricting to single output only allow us to compress the ﬁnal ReLU layer and not any preceeding ones.
Algorithm 2: Neural Path K-means Compression
Apply K-means for K centers to the vectors (cid:0) a T i , b i , C T : ,i (cid:1) , i = 1 , ..., n , and get the centers (cid:16) ˜a Ti ,
˜ b i , ˜ C T : ,i (cid:17) , i = 1 , ..., K . 2.
Construct the ﬁnal weights.
For the ﬁrst linear layer matrix the i − th row becomes (cid:16) ˜ a Ti , ˜
b i (cid:17) , while for the second linear layer matrix, the i − th column becomes ˜ C
( i ) .
Neural Path K-means To overcome this obstacle we apply a simultaneous approximation of the zonotopes.
The method is called Neural Path K-means and directly applies K-means to the vectors of the weights (cid:0) a Ti , b i , c 1 i , ..., c mi (cid:1) associated to each neuron i of the hidden layer.
The name of the algorithm emanates from the fact that the vector associated to each neuron consists of the weights of all the neural network paths passing from this neuron.
The procedure is presented in Algorithm 2.
Null Generators Neural Path K-means does not apply compression directly to each zonotope of the network, but is rather a heuristic approach for this task.
Precisely, if we focus on the set of generators of the zonotopes of output j , Neural Path K-means might mix positive and negative generators together in the same cluster.
For instance, suppose (cid:16) ˜ a Tk , ˜ b k , ˜ C T : ,k (cid:17) is the cluster center corresponding to vectors (cid:0) a Ti , b i , C T : ,i (cid:1) for i ∈
I .
Then regarding output j , it is not necessary that ∀ i ∈
I all c ji have the same sign.
Thus, the compressed positive zonotope ˜ P j might contain generators of the original negative zonotope Q j and vice versa.
We will call generators c ji (cid:0) a Ti , b i (cid:1) contributing to opposite zonotopes, null generators .
Proposition 6.
Neural Path K-means produces a compressed neural network with output ˜ v satisfying
[Equation]
where K is the number of K-means clusters, δ max the maximum distance from any point to its corresponding cluster center, N max , N min the maximum and minimum cardinality respectively of a cluster and N j the set of null generators with respect to output j .
The performance of Neural Path K-means is evaluated with Proposition 6.
The result we deduce is analogous to Zonotope K-means.
The approximation error is eliminated as the number K of centers approaches n .
This holds because for K = n we get δ max = 0 , N max = 1 and N j = ∅ , ∀ j ∈ [ m ] .
However, as K is reduced, the upper bound reaches a value depending on the magnitude of the weights of the linear layers together with weights corresponding to null generators.
[Table] [Table]
E XPERIMENTS
We conduct experiments on compressing the linear layers of convolutional neural networks, while leaving the convolutional ﬁlters unchanged.
Our experiments work as a proof-of-concept indicating that our theoretical claims indeed hold in practice.
In particular, the heart of our contribution lies in presenting novel tropical geometrical background for neural network compression.
Moreover, our techniques motivate further research in the area, which will allow them to become more competitive.
Our methods compress the linear layers of the network layer by layer.
They perform a functional approximation of the original network and thus they are applicable for both classiﬁcation and regression tasks.
To compare them with other techniques in the literature we choose methods with similar structure, i.e. structured pruning techniques without re-training.
For example, Alfarra et al.
(2020) propose a compression algorithm based on the sparsiﬁcation of the matrices representing the zonotopes and worked as a great intuition for our work.
However, their method is unstructured and incompatible for comparison.
The methods we choose to compare are two tropical methods for single-output (Smyrnis et al., 2020) and multi-output (Smyrnis & Maragos, 2020) networks, Random
Structured, L 1 and a modiﬁcation of ThiNet (Luo et al., 2017) adapted to linear layers.
Smyrnis et al.
(2020); Smyrnis & Maragos (2020) propose a novel tropical division framework that aims on the reduction of zonotope vertices.
Random method prunes neurons according to uniform probability, while L 1 prunes those with the lowest value of L 1 norm of their weights.
Also, ThiNet uses a greedy criterion for discarding the neurons that have the smallest contribution to the output of the network.
MNIST Dataset, Pairs 3-5 and 4-9 The ﬁrst experiment is performed on the binary classiﬁcation tasks of pairs 3 / 5 and 4 / 9 of the MNIST dataset and so we can utilize both of our proposed methods.
In Table 1, we compare our methods with a tropical geometrical approach of Smyrnis et al.
Their method is based on a tropical division framework for network minimization.
For fair comparison, we use the same CNN with two fully connected layers and hidden layer of size 1000 .
According to
Table 1, our techniques seem to have similar performance.
They retain the accuracy of the network while reducing its size.
Moreover, in Table 2 we include experimental computation of the theoretical bounds provided by Proposition 5, 6.
We notice that the bounds decrease as the remaining weights get less.
The behaviour of the bounds was expected to be incremental because the less weights we use, the compression gets worse and the error becomes larger.
However, the opposite holds which means that the bounds are tighter for higher pruning rates.
It is also important to mention that the bounds become 0 when we keep all the weights, as expected.
MNIST and Fashion-MNIST Datasets
For the second experiment we employ MNIST and
Fashion-MNIST datasets.
The corresponding classiﬁcation is multiclass and thus Neural Path
K-means may only be applied.
In Table 3, we compare it with the multiclass tropical method of
[Table]
Smyrnis & Maragos (2020) using the same CNN architecture they do.
Furthermore, in plots 4a, 4b we compare Neural Path K-means with ThiNet and baseline pruning methods by compressing LeNet5 (LeCun et al., 1998).
To get a better idea of how our method performs in deeper architectures we provide plots 4c,4d that illustrate the performance of compressing a deep neural network with layers of size 28 ∗ 28 , 512 , 256 , 128 and 10 , which we refer to as deepNN.
The compression is executed on all hidden layers beginning from the input and heading to the output.
From Table 3, we deduce that our method performs better than (Smyrnis & Maragos, 2020).
Also, it achieves higher accuracy scores and experience lower variance as shown in plots 4a-4d.
Neural Path K-means, overall, seems to have good performance, even competitive to ThiNet.
Its worst performance occurs on low percentages of remaining weights.
An explanation for this is that K-means provides a high-quality compression as long as the number of centers is not less than the number of "real" clusters.
[Table]
[Figure]
CIFAR Dataset We conduct our ﬁnal experiment on CIFAR datasets using CIFAR-VGG (Blalock et al., 2020) and an altered version of AlexNet adapted for CIFAR.
The resulting plots are shown in
Fig. 4e-4h.
We deduce that Neural Path K-means retains a good performance on larger datasets.
In particular, in most cases it has slightly better accuracy an lower deviation than the baselines, but has worse behaviour when keeping almost zero weights.
C ONCLUSIONS AND F UTURE W ORK
We presented a novel theorem on the bounding of the approximation error of two neural networks with one hidden layer.
This theorem occurs from the bounding of two tropical polynomials via the Hausdorff distance of their Extended Newton Polytopes.
Our theorem can be used to derive geometrical compression algorithms for the fully connected part of ReLU activated deep neural networks, with application to convolutional layers being an ongoing work.
A P ROOFS FOR THE SECTION "B ACKGROUND ON T ROPICAL G EOMETRY " A. P ROOF OF P ROPOSITION
Proof.
The ﬁrst argument follows from the fact that both p j , q j are linear combinations of tropical polynomials consisting of two terms.
Indeed, we compute
[Equation]
Each ENewt (cid:0) max( c ji a Ti x + c ji b i , 0) (cid:1) is a line segment with endpoints 0 and (cid:0)
c ji a Ti , c ji b i (cid:1) =
c ji (cid:0) a Ti , b i (cid:1) .
Therefore P j is written as the Minkowski sum of line segments, which is a zonotope by deﬁnition.
Similarly Q j is a zonotope.
Furthermore, from the deﬁnition of the Minkowski sum, each point v ∈ P j may be written as (cid:80) c ji > 0 v i , where each v
i is a point in the segment ENewt (cid:0) max( c ji a T
i x + c ji b i , 0) (cid:1) .
A vertex of
P j can only occur if v i is an extreme point of ENewt (cid:0) max( c ji a Ti x + c ji b i , 0) (cid:1) for every i which is equivalent to either v i = 0 or v i
=
c ji (cid:0) a Ti , b i (cid:1) .
This means that every vertex of P j corresponds to a subset I + ⊆
[ n ] of indices i with c ji > 0 , for which we choose v i =
c ji (cid:0) a Ti , b i (cid:1) and for the rest it holds v i = 0 .
Thus,
[Equation]
=
[Equation]
In the same way we derive the analogous result for the negative zonotope Q j .
Corollary 2.
The geometric result concerning the structure of zonotopes can be extended to maxpooling layers.
For instance, a max-pooling layer of size 2 × 2 corresponds to a polytope that is constructed as the Minkowski sum of pyramids which could stand as generalized case of zonotope.
B P ROOFS FOR THE SECTION "A PPROXIMATION OF T ROPICAL P OLYNOMIALS " B. P ROOF OF P ROPOSITION
Proof.
Consider a point x ∈ B and assume that p ( x ) = a T x + b, ˜ p ( x )
=
c
T x + d .
Then,
[Equation]
where (cid:0) u T , v (cid:1) may be any vertex of ˜ P .
Similarly, we derive
[Equation]
for any vertex (cid:0) r T , s (cid:1) of P .
Therefore, we may select the vertices (cid:0) u T , v (cid:1) ∈
˜ P, (cid:0) r T , s (cid:1) ∈ P so that their respective distances from (cid:0) a T , b (cid:1) and (cid:0) c T , d (cid:1) , respectively, are minimized.
Choosing them in such a way gives
[Equation]
In similar manner we deduce
[Equation]
Notice, that for the relations (5) and (6) we used Cauchy-Schwartz inequality
[Equation]
Inequality (5) holds at any point x ∈ B for some vertex (cid:0) a T , b (cid:1) ∈ P , therefore
[Equation]
for all x ∈ B .
Similarly
[Equation]
we derive
[Equation]
Combining (7) and (8) gives
[Equation]
Hence, from the deﬁnition of the Hausdorff distance of two polytopes we derive the desired upper bound
[Equation]
Remark 2.
Note that with similar proof one may replace the Hausdorff distance of the two polytopes by the Hausdorff distance of their upper envelopes.
This makes our theorem an exact generalization of Proposition 2.
However, this format is difﬁcult to use in practice, because it is computationally harder to determine the vertices of the upper envelope.
B. P ROOF OF T HEOREM
Proof.
Notice that we may write
[Equation]
Thus from from Proposition 4 we derive
[Equation] [Table] C P ROOFS AND F IGURES FOR THE SECTION "N EURAL N ETWORK C OMPRESSION A LGORITHMS " C. I LLUSTRATION OF Z ONOTOPE K- MEANS
Below we present a larger version of Fig.
5 demonstrating the execution of Zonotope K-means .
xxfffffffv c 1 cccc 5 ccbbbbbbb{ a ji } i ∈ (1 , 2) ,j ∈ (1 ,..., 7) (a) Original network.
xx˜ f˜ f˜ f˜ fv + + − − ˜ c 1 ˜ b˜ c 2 ˜ b˜ c 3 ˜ b˜ c 4 ˜ b 4 { ˜ c
j ˜ a ji } i ∈ (1 , 2) ,j ∈ (1 ,..., 4) (b) Minimized network.
c 1 ( a T 1 ,b 1 ) c 2 ( a T 2 ,b 2 ) c 3 ( a T 3 ,b 3 ) c 4 ( a T 4 ,b 4 ) c 5 ( a T 5 ,b 5 ) c 6 ( a T 6 ,b 6 ) c
( a T 7 ,b 7 ) (c) Original zonotopes ˜ c
(˜ a T 1 , ˜ b 1 ) ˜
c 2 (˜ a T 2 , ˜ b 2 )
˜ c 4 (˜ a T 4 , ˜ b 4 ) ≡ c 7 ( a T 7 ,b 7 ) ˜ c
3 (˜ a T 3 , ˜ b 3 ) (
[Figure]
of 3 , 4 .
Similarly, the approximation ˜ Q of Q is colored in green and deﬁned by the generators
˜ c
i (cid:16) ˜ a Ti , ˜ b i (cid:17) , i = 3 , 4 that stand as representative centers for { 5 , 6 } and 7 respectively.
C. P ROOF OF P ROPOSITION
Proof.
We remind that for the output functions it holds
[Equation]
From triangle inequality we deduce
[Equation]
Prop 4 bounds | p ( x ) −
˜ p ( x ) | and | q ( x ) − ˜ q ( x ) | are bounded by H (cid:16) P, ˜ P (cid:17) and H (cid:16) Q, ˜ Q (cid:17) respectively.
Therefore, it sufﬁces to get an upper bound for these Hausdorff distances.
Let us consider any vertex u
= (cid:80) i ∈ I
+ c i (cid:0) a Ti , b i (cid:1) of P .
For the vertex u ∈ P we need to choose vertex v ∈ ˜ P as close to u as possible, in order to provide an upper bound for dist (cid:16) u , ˜ P (cid:17) .
Vertex v is selected as follows.
For each i ∈ I
+ we select k such that
˜ c k (cid:0) ˜ a Tk ˜ b k (cid:1) is the center of the cluster where c i (cid:0) a Ti , b i (cid:1) belongs to.
We denote the set of such clusters by C + , where each cluster center k appears only once.
Then, vertex v is constructed as v = (cid:80) k ∈ C + ˜ c k (cid:0) ˜ a Tk
[Table]
b [Table] ∈ ˜ P .
We have that:
[Table]
i
[Equation]
I k + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:16)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:0)
[Equation]
(cid:13)
[Table]
[Equation]
[Table]
[Equation]
(cid:88)
i
[Equation]
[Table]
[Equation]
[Table]
1 −
[Equation]
| (cid:19)
[Equation]
(cid:107)
[Equation]
I k + | (cid:21)
[Equation]
[Equation]
where we denote by I k + the set of indices i ∈ I
+ that belong to the center k ∈ C
+ and ε i =
˜ c k (cid:16) ˜ a Tk , ˜ b k (cid:17) − c
i (cid:0) a Ti , b i (cid:1) is the vector of the difference of the i − th generator, with its corresponding
K-means cluster center.
The maximum value of the upper bound occurs when I + contains all indices that correspond to c i > 0 .
This value gives us an upper bound for max u ∈ P d ( u , ˜ P ) .
To compute an upper bound for max v ∈V ˜ P d ( P, v ) we assume v = (cid:80) k ∈ C + ˜ c k (cid:0) ˜ a Tk ˜ b k (cid:1) and consider the vertex (cid:80) i ∈ I
+ c i (cid:0) a Ti , b i (cid:1) ∈ P where I + is the set of indices of positive generators corresponding to the union of all clusters corresponding to the centers of C + .
Note that the occurring distance
[Equation]
was taken into account when computing the upper bound for max u ∈V P d ( u , ˜ P ) , and thus both values obtain the same upper bound.
Therefore,
[Equation]
where K + is the number of cluster centers corresponding to ˜ P and I + the indices corresponding to all positive generators of P .
Similarly,
[Equation]
where K − , I − are deﬁned in analogous way for the negative zonotope.
Combining the relations gives the desired bound.
[Equation]
C. I LLUSTRATION FOR N EURAL P ATH K- MEANS ALGORITHM
Below we illustrate the vectors on which K-means is applied for the multi-output case.
The vectors that are compressed are consist of all the edges associated to a hidden layer neuron.
The corresponding edges contain all the possible neural paths that begin from some node of the input, end in some node of the output and pass through this hidden node.
xx...
x k ... x
d ff... f i ...
f n vv... v j ... v m a
i 1 a ia
i k a i d c 1 i c 2 i c ji c m i bbb i b n
In the main text we deﬁned the null generators of zonotopes that occur by the execution of Neural
Path K-means.
Below we provide an illustration for them.
[Figure]
[Figure]
P ROOF OF P ROPOSITION
Proof.
Let us ﬁrst focus on a single output, say j − th output.
As in the proof for Zonotope K-means, we will bound H (cid:16) P j , ˜ P
j (cid:17) , H (cid:16) Q j , ˜ Q j (cid:17) for all j ∈ [ m ] .
From triangle inequality we get | v j ( x ) − ˜ v j ( x ) | ≤ | p j ( x ) − ˜ p
j ( x ) | + | q j ( x ) − ˜ q j ( x ) |
Any vertex of u ∈ P j can be written as u = (cid:80) i ∈ I j + c ji (cid:0) a Ti , b i (cid:1) where the set of indices
I j + satisfy c ji > 0 , ∀ i ∈ I j
+ and thus correspond to positive generators.
To choose a nearby vertex from ˜ P
j we perform the following.
For each i ∈ I j
+ we select the center (cid:0) ˜ a Tk ˜ b k ˜ C ( k ) T (cid:1) of the cluster to where (cid:0) a T i b i
C ( i ) T (cid:1) belongs, only if ˜ c jk > 0 .
Such a center only exists if c ji (cid:0) a T i , b i (cid:1) is not a
[Table] [Table]
[Table] [Table]
i ∈ N j + |
c ji | (cid:13)(cid:13)(cid:0) a Ti , b i (cid:1)(cid:13)(cid:13) where for all i ∈ I jk + the i − th vector of K-means is represented by the k − th center
k ∈ C j
+ .
We also assumed ˜ C
( i ) =
C : ,i + ε ( i ) ⇒ ˜
c ji = c ji + ε ji and (cid:16) ˜ a Ti , ˜ b
i (cid:17) = (cid:0) a Ti , b i (cid:1)
+ λ i .
The maximum value of the upper bound occurs when I j + contains all indices that correspond to c ji > 0 .
To compute an upper bound for max v ∈V ˜
Pj dist ( P j , v ) we write the vertex v as v =
[Equation]
˜ b
[Equation]
(cid:1) ∈ ˜
P j and choose the vertex u
= (cid:80) i ∈ I
j + c ji (cid:0) a Ti , b i (cid:1) of P j where
I j
+ is the set of all indices corresponding to generators that belong to these clusters.
As in the proof of Proposition 5, their distance was taken into account when computing the upper bound for max u ∈V Pj dist (cid:16) u , ˜ P j (cid:17) .
Hence, both obtain the same upper bound which leads to
[Equation]
i ∈
I jk + (cid:34) | ε ji | (cid:13)(cid:13)(cid:0) a Ti , b i (cid:1)(cid:13)(cid:13) + | c ji | (cid:107) λ i (cid:107) | I jk + | (cid:35) + (cid:88) i ∈ N j + |
c ji | (cid:13)(cid:13)(cid:0) a Ti , b i (cid:1)(cid:13)(cid:13) where I j + contains all indices corresponding to positive c ji .
Similarly, we deduce
H (cid:16) Q j , ˜ Q j
[Equation]
k ∈ C j − (cid:88)
i ∈
I jk − (cid:20) | ε
[Equation]
|(cid:107)
λ i (cid:107) | I jk − | + (cid:18) 1 − | I jk − | (cid:19) | c ji | (cid:13)(cid:13)(cid:0) a Ti , b i (cid:1)(cid:13)(cid:13)(cid:21) + + (cid:88) k ∈
C j − (cid:88) i ∈
I jk − (cid:34) | ε ji | (cid:13)(cid:13)(cid:0) a Ti , b i (cid:1)(cid:13)(cid:13)
| c ji | (cid:107) λ i (cid:107) | I jk − | (cid:35) + (cid:88)
i ∈ N j − |
c ji | (cid:13)(cid:13)(cid:0) a Ti , b i (cid:1)(cid:13)(cid:13) where I j − contains all i such that c ji < 0 .
In total these bounds together with Proposition 4 give
[Equation]
Here we used the notation C j = C j + ∪ C j − = { 1 , 2 , ..., K } and I jk is either equal to I jk + or I jk − depending on k ∈ C j .
Note that { i | i ∈ I jk , k ∈ C j } = { 1 , 2 , ..., n } \ N j ⊆ { 1 , 2 , ..., n } , since every generator that is not null corresponds to some cluster center with the same sign.
Also, using
N max ≥ | I jk | ≥ N min , it follows that
[Equation]
We will compute the total cost that combines all outputs by applying the inequality
[Equation]
which is a direct application of Cauchy-Schwartz inequality
[Equation]
Together with the relations (cid:107) ε ( i ) (cid:107) <
δ max , (cid:107) λ
i (cid:107) <
δ max
[Equation]
we get
[Equation]
j = n (cid:88) i = (cid:34) | ε ji | (cid:13)(cid:13)(cid:0) a Ti , b i (cid:1)(cid:13)(cid:13)
+ | c ji | (cid:107) λ
i (cid:107)
N min (cid:35) + m (cid:88)
j = (cid:88) i ∈ N j |
c ji | (cid:13)(cid:13)(cid:0) a Ti , b i (cid:1)(cid:13)(cid:13) ≤ n (cid:88) i = (cid:34) √ m (cid:13)(cid:13) ε ( i ) (cid:13)(cid:13) (cid:107) λ i (cid:107)
N min + (cid:18) 1 −
N max (cid:19) √ m (cid:107) C : ,i (cid:107) (cid:13)(cid:13)(cid:0) a Ti , b i (cid:1)(cid:13)(cid:13)(cid:35)
+ + n (cid:88)
i = (cid:34) √ m (cid:13)(cid:13)
ε ( i ) (cid:13)(cid:13) (cid:13)(cid:13)(cid:0) a Ti , b i (cid:1)(cid:13)(cid:13) + √ m (cid:107) C : ,i (cid:107) (cid:107) λ i (cid:107)
N min (cid:35) + m (cid:88)
j = (cid:88) i ∈ N j |
c ji | (cid:13)(cid:13)(cid:0) a Ti , b i (cid:1)(cid:13)(cid:13) ≤ √ mKδ 2 max + √ m (cid:18) 1 −
N max (cid:19) n (cid:88) i =
[Table]
[Equation]
Ti , b
[Table]
+
[Table]
[Equation]
[Table]
[Equation]
[Table]
(cid:1)
+
[Table]
j
[Table]
| c
[Table]
| (cid:13)(cid:13)(cid:0)
[Equation]
Ti , b i
[Table]
as desired.
D E XPERIMENTAL S PECIFICATIONS
Experimental Setup For our code’s execution we used Google’s Colab GPU machines.
Training speciﬁcations
Small CNNs such as the custom CNNs from Smyrnis et al.
; Smyrnis & Maragos (2020) and LeNet5 LeCun et al.
(1998) were trained using Adam optimizer with learning rate 10 − 3 for 20 epochs.
Larger CNNs CIFAR-VGG and modiﬁed AlexNet were trained with frozen convolutional layers with SGD at learning rate 10 − 2 , momentum 0 .
9 and weight decay 5 · 10 −for 28 epochs.
For each experiment we train from start 5 distinct models.
The reported accuracy percentages are taken as the average across these 5 models.
Standard deviation is provided both for tabular reports and on ﬁgure plots as error bars.
Pruning speciﬁcations Our algorithms are not deterministic, since K-means has random initialization.
To ensure better performance each algorithm was repeated multiple times and the pruned model with the best validation accuracy was provided as output.
In particular, we repeat each algorithm’s execution 20 on the experiment of comparison to Smyrnis & Maragos (2020), 1 for the CIFAR experiment, and 5 for the rest of the experiments.