R EDUCING THE NUMBER OF NEURONS OF D EEP R E LU N ETWORKS BASED ON THE CURRENT THEORY OF R EGULARIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We introduce a new Reduction Algorithm which makes use of the properties of
ReLU neurons to reduce signiﬁcantly the number of neurons in a trained Deep
Neural Network
This algorithm is based on the recent theory of implicit and explicit regularization in
Deep ReLU Networks from (Maennel et al, 2018) and the authors.
We discuss two experiments which illustrate the efﬁciency of the algorithm to reduce the number of neurons signiﬁcantly with provably almost no change of the learned function within the training data (and therefore almost no loss in accuracy).
I NTRODUCTION 1. M OTIVATION
In this work, we investigate a particular type of deep neural network.
Its architecture (see section 2) can be better understood, thanks to the previous work on wide shallow neural networks: Neyshabur et al.
(2014); Ongie et al.
; Savarese et al.
; Williams et al.
; Maennel et al.
Heiss et al.
and unpublished work of the authors on deep neural networks (with arbitrarily many inputs and outputs).
These results state that (cid:96) 2 weight regularization on parameter space is equivalent to L 1 -typed Pfunctionals on function space under certain conditions.
This implies that the optimal function could also be represented by ﬁnitely many neurons (Rosset et al., 2007).
With the knowledge of these properties, we were able to design a reduction algorithm which can reduce inﬁnitely wide (in practice: arbitrarily wide) layers in our architecture to much smaller layers.
This allows us to reduce the number of neurons by 90% to 99% without introducing sparsity (which allows more efﬁcient GPU-implementation (Gale et al., 2020)) and with almost no loss in accuracy.
This can be of interest for deploying neural networks on small devices or for making predictions which are computationally less costly and less energy consuming.
1. L ITERATURE / L INK TO OTHER RESEARCH
Many papers have been written on the subject of reducing neural networks.
There is the approach of weight pruning, by removing the least salient weights (LeCun et al., 1990; Hassibi & Stork, 1993; Han et al., 2015; Tanaka et al., 2020).
A different technique is pruning neurons Mariet & Sra (2015);
He et al.
(2014); Srinivas & Babu (2015), which does not introduce sparsity in the network by removing single weights, but reduces the number of neurons.
For CNNs there are ways to prune the ﬁlters (Li et al., 2016).
In transfer learning, one can prune the weights with decreasing magnitude (Sanh et al., 2020).
All these techniques require the same steps: train a large network, prune and update remaining weights or neurons, retrain.
And for too much pruning, the accuracy of the pruned models drops signiﬁcantly, also it might not always be useful to ﬁne-tune the pruned models (Liu et al., 2018).
Another approach is knowledge distillation (Hinton et al., 2015; Ba & Caruana, 2014) where one establishes a teacher/student relation between a complex and simpler network.
The lottery ticket hypothesis (Frankle & Carbin, 2018) states that ”a randomly-initialized, dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations”.
In this work, the method can be related to neuron pruning, in that we are working directly on a large already-trained network.
We are, however, trying to preserve the learned function contrary to the cited techniques which focus on the loss function and where pruning results in a different learned function.
Therefore, in our algorithm, the neurons are not only pruned but rather condensed, put together into new neurons which contain all the information learned during training.
Our method, hence, does not require retraining.
But it is beneﬁcial to further retrain the network, and reduce it again, in an iterative process.
D ESCRIPTION OF THE ARCHITECTURE
Starting from a traditional Shallow Neural Network with ReLU activation function (see ﬁg. 1) (which contains a single hidden layer) and is (cid:96) 2 regularized, we will deﬁne two variants of a One
Stack Network .
First, by adding a direct (or skip) connection between the input layer and the output layer (see ﬁg. 2), one can obtain the simpliﬁed One Stack Network.
Second, by adding a layer in the middle of this direct (skip) connection, one can get a One Stack
Network (see ﬁg. 3).
This new layer contains neurons with a linear activation function (the identity function multiplied by a constant).
It contains as many neurons as the minimum between the number of neurons in the input layer and the number of neurons in the output layer, it also has no bias.
Furthermore, we call it the afﬁne layer and the new weights before and after it the afﬁne weights .
These new weights can also be (cid:96) 2 regularized, but typically by a different hyperparameter than the non-linear weights
[Figure]
[Figure] [Figure]
The architecture that we are going to study can be described as a sequence of stacks, or a Deep
Stack Network .
We repeat the pattern described above (see ﬁgs.
4 and 5).
Since the output layer is at the end of the architecture, we call all intermediate layers related to the output layers as introduced earlier (typically containing few neurons d j ): bottlenecks .
The bottlenecks contain neurons with a linear (identity) activation function.
For every stack, all parameters are regularized except for the biases in the bottleneck.
[Figure]
Identity Identity
[Figure]
Output
[Figure]
Linear
ReLU
Output
Linear
We recommend using a higher learning rate for the afﬁne weights.
In our experiments, we used a ten-times higher learning rate for the afﬁne weights.
For every stack j , the weights and biases connecting the previous input to the (inﬁnitely-wide n j → ∞ ) ReLU layer are written as V ( j ) ∈ R n j × d j − 1 and b ( j ) ∈ R n j , the ones connecting to the bottleneck are written as W ( j ) ∈
R d j × n j and c ( j ) ∈ R d j .
Remark 2.1 .
(ordinary fully connected deep neural network) Note that our algorithm could also be applied to an ordinary fully connected deep ReLU neural network and would output a neural network with skip connections.
But if every second layer is much wider ( d j (cid:28) n j ), we have a much better theoretical understanding of what is happening and typically we can get better test accuracy with fewer neurons.
We don’t use ReLUs on the bottlenecks for easier interpretation.
The compression-rate would not suffer if we apply our algorithm on architectures without afﬁne weights.
C ONSEQUENCES FROM P- FUNCTIONAL REGULARIZATION THEORY
If we choose the numbers n j of hidden nodes sufﬁciently large for each stack while we keep ﬁnite bottleneck dimensions d j the authors have shown in an unpublished paper that the solution of the optimization problem
[Equation]
can be characterized as the solution of a much easier to interpret optimization problem
[Equation]
where we optimize over all continuous functions
f so we do not have to care about if f is representable as a neural network.
According to the unpublished paper by the authors (similarly to the work by Neyshabur et al.
Ongie et al.
; Savarese et al.
; Williams et al.
; Heiss et al.
, we can obtain the P -functional P of the deep stacked neural networks from the P -functional P j of a single stacks,
[Equation]
For a network we get
[Equation]
, where (4)
[Equation]
handles the boundary condition, S d − 1 denoted the ( d − 1) -dimensional unit sphere and we have a weighing function 3 g ( r ) = 1 √ r 2 +1 .
For the other types of stacks shown in Figures 2 and 3 the corresponding P -functionals would only need minor modiﬁcations.
If one would not regularize the afﬁne weights (skip connections) the theory would drastically change because P would assignto many highly oscillating functions if #stacks ≥ 2 , but in practice implicit regularization would weaken this effect.
For this paper, the most important aspect of eqs.
(3) and (4) is that there is (cid:107)·(cid:107) 2 (instead of (cid:107)·(cid:107) 22 ) 4 inside the integral, which reminds very much of an L 1 -norm.
Therefore the function optimizing eq.
(2) can be represented by a ﬁnite number of neurons (Rosset et al., 2007).
Similarily Maennel et al.
(2018) found that also without explicit regularization gradient decent favours solutions which could be represented by much smaller networks as well.
In the following we observe that this ﬁnite numbers n j can be very low in practice.
R EDUCTION ALGORITHM
We will now consider a one-stack network according to the description of the architecture given above (see ﬁg. 3).
Therefore, we can ﬁx j and write W :=
W ( j ) for example in this case.
Then, the reduction can be applied similarly on every stack.
4. R EMOVING AND REPLACING OUTSIDE NEURONS
The main idea of this step is to combine the information of all the non-linear (ReLU) neurons whose kink positions are outside the convex hull of the previous stack’s representation of the training data into the afﬁne neurons.
We deﬁne the outside neurons as those neurons that will always return a zero or always return strictly positive numbers for all training data points.
Therefore we ﬁnd two cases.
Either the activation function of the outside neuron acts as a zero function in which case we can simply remove it from the ReLU layer as it makes no contribution to the learned function, or the activation function of the outside neuron acts as an identity function.
In the latter case, the contribution of the neuron to the learned function is afﬁne and we can add it to the afﬁne weights and remove it from the ReLU layer.
In order to do this, we will deﬁne W afﬁne and c afﬁne :
More
− ε,ε ]( r ) (cid:13)(cid:13) 2 (cid:1) dr with the help of a convolution which inﬁnitesimally smoothens the function or
distribution f.
This results for example in (cid:82) R (cid:107) δ 0 ( r ) (cid:107) 22 dr = ∞ and (cid:82) R
[Table]
[Equation]
where M is the set of outside neurons whose activation function is positive for all training data points.
Note that we can remove the neurons whose activation function acts as a zero function without any replacement since they do not contribute to the learned function.
Note as well that w k are column vectors and v k are row vectors and their multiplication in that order gives a matrix.
Therefore, W afﬁne is the result of a sum of matrices and c afﬁne is a vector, both summarize the contributions of all outside neurons to the learned function.
We are now ﬁnally able to remove the outside neurons and add their contributions to the afﬁne weights.
The question we ask ourselves now is: how do we determine which neuron is an outside neuron?
We decided to loop through every ReLU neuron, then check their prediction for every training data point.
If the prediction was always zero or always strictly positive, we could be sure that we found an outside neuron.
Of course such a method is also computationally expensive, and would hence be very impractical if we had many neurons and many training data points.
But we observed empirically that it was not necessary to loop through every training data point (more on this in the experimentation chapter).
Remark 4.1 .
Exactly speaking, the learned function will not be the same as before.
In fact, it will be different outside of the convex hull of the training data.
But these changes further smoothen the extrapolation behavior of the learned function without affecting training accuracy.
Removing the outside neurons does not change the function on the training data and it does not change the stack’s function inside the convex hull of the previous stack’s representation of the training data.
4. R EMOVING THE WEAKEST NEURONS
The main idea of this step is to prune the neurons which do not add much to the learned function.
This can be done by calculating the quantity (cid:113)
[Equation]
each neuron k .
We sort these quantities and remove the weakest ones whose sum is less than a certain tolerance.
Determining this tolerance also serves as hyper-parameter tuning.
4. C LUSTERING THE REMAINING NEURONS
The main idea of the clustering step is to combine the information of the remaining neurons into fewer neurons.
This would not work that well without the steps before.
We ﬁrst group the neurons according to their kink-representation, using (weighted) k-means clustering.
It is up to us to determine the number of clusters, and hence the ﬁnal number of neurons after reduction, because for every cluster we will deﬁne one neuron summarizing all the information.
The kink-representation of every neuron is computed as following: a concatenation of b k v k (cid:107) v k (cid:107) 22 and v k (cid:107) v k (cid:107) 2 , and are weighted according to: (cid:107) v k (cid:107) 2 (cid:107) w k (cid:107) 2 .
First, the new kink position will be determined according to this formula:
[Equation]
We deﬁne the strength:
[Equation]
Finally we can deﬁne the new v , b , and w :
[Equation]
With g ( ξ ) = 1 √ ξ 2 +
and k the index of the neurons in a selected cluster.
The new v , b , and w can then be assigned to a new single neuron which will replace all the neurons contained in the same cluster.
E XPERIMENTS
We performed two experiments.
First, a one-dimensional easy to visualize example.
Second, we train the architecture on the MNIST dataset (handwritten digit recognition).
5. C OMPOSITE SINUS
At ﬁrst, we perform a simple experiment on artiﬁcially-generated data 6 .
We create a map f : R → R 7 : x (cid:55)→ f ( x ) :
= ( f i (sin 3 πx ))
i ∈{ 1 ,..., 7 } from a one-dimensional input to a multi-dimensional output.
All the outputs are functions of the sine of the (scaled) input.
We choose an architecture with three stacks (see ﬁg. 5), and d j = 1 neuron in each bottleneck.
We train the network at ﬁrst without regularization, and then with some regularization.
The plots are given before and after reduction in
6 and 7.
green show the function learned by the stacks themselves, red curves composition function learned network to stack.
contribution by
| respectively of all neurons.
One can observe that the intermediate (or hidden) stacks learn the inside function (the sine, see ﬁgs.
6a and 6b), and that the last stack learns the composite functions
f i on top of the sine (the square function, the cubic function, etc. see ﬁgs.
7c and 7e for example).
There is almost no difference of the plots after reduction, except of course for the yellow, red and blue points which are less numerous, and are at the right places: at the nonlinearities, where the contributions of the neurons left are essential.
The number of ReLU neurons in each stack and the MSE at every step of the algorithm is presented in Table 1.
[Table]
[Figure] [Figure]
5. MNIST
The last one-dimensional example was hopefully very intuitive and visual, yet it would now be interesting to test the architecture on a higher-dimensional dataset.
We chose to apply it to MNIST, the very popular handwritten digit recognition dataset.
Starting with a three-stack architecture (like in the previous example and illustrated in ﬁg. 5), we put d j = 16 neurons in every bottleneck.
We then proceeded to train this network:
ﬁrst without regularization and then with regularization for a longer time.
Here we applied the Reduction Algorithm twice.
We reduced the trained neural network a ﬁrst time, then trained the reduced model again which allowed us to reduce that model even further:
Removing and Replacing the Outside Neurons Instead of checking all 60000 training data points, we only checked 65 data points during the ﬁrst reduction and 30 during the second reduction to save time.
Removing the weakest neurons We used different tolerances for every stack.
For the experiment model we chose tolerances of 30, 20, and 10 for stacks 1, 2, and 3 respectively for the ﬁrst reduction, and tolerances 10, 2, and 17 for the second reduction.
The goal in this step (and the next) was to vary the hyperparameter such that we would not lose too much accuracy while removing as many neurons as possible.
Clustering Further reduction of the number of neurons as discussed before.
See table 2 for an illustration of how to typically apply the Reduction Algorithm on a given network.
Steps nnn
Train accuracy
Test accuracy
Full (large) network   0.
Replacing outside neurons   0.
Removing weak neurons   0.
Clustering   0.
After retraining   0.
Replacing outside neurons   0.
Removing weak neurons   0.
Clustering
[Table]
C ONCLUSION AND FURTHER WORK
In this paper, we have introduced a neural network architecture which allows to be reduced signiﬁcantly in the sense that we can remove or replace many of its ReLU neurons by fewer ReLU neurons.
This property alone is already of interest for a number of reasons evoked in the introductory section.
We could also analyze the learned function of each stack and observe the composition of functions.
Inspired by the lottery ticket hypothesis, we could argue that it would be difﬁcult to train a smaller neural network to generalize as well as the reduced model (when both have the same dimension).
Worth noting is that if one tries to evaluate an upper bound of the generalization gap (difference between the empirical error and true error) based on the sum of the weights, (Kawaguchi et al., 2017), one could potentially ﬁnd a better estimation by taking the sum of the weights after reduction.
Also of interest is that, after the reduction algorithm was applied, we obtain a fully-connected neural network.
Therefore this method of reduction/compression could be further improved by applying for example weight pruning on top of it.
There are many possibilities for future work, and many are going to be explored very soon.
We want to investigate every step of the Reduction Algorithm in more detail whether it is for checking fewer training data points by choosing them non-randomly in the ﬁrst step, or ﬁnding a better and more efﬁcient way to cluster the remaining neurons in the last step.
Similar ideas (of reducing the number of neurons) could potentially be applied to the bottleneck layer (at least the second and third step of the algorithm, since they do not require the presence of afﬁne weights).
Investigating the performance of the algorithm on different datasets and comparing it to other neural network architectures is certainly of interest as well as extending it to be used in
CNNs and RNNs.
Since our algorithm is highly theory driven, we might use these ideas to better theoretically explain the lottery ticket hypothesis (Frankle & Carbin, 2018).