S IGNING THE S UPERMASK : K EEP , H IDE , I NVERT
A BSTRACT
The exponential growth in numbers of parameters of neural networks over the past years has been accompanied by an increase in performance across several ﬁelds.
However, due to their sheer size, the networks not only became difﬁcult to interpret but also problematic to train and use in real-world applications, since hardware requirements increased accordingly.
Tackling both issues, we present a novel approach that either drops a neural network’s initial weights or inverts their respective sign.
Put simply, a network is trained by weight selection and inversion without changing their absolute values.
Our contribution extends previous work on masking by additionally sign-inverting the initial weights and follows the ﬁndings of the Lottery Ticket Hypothesis.
Through this extension and adaptations of initialization methods, we achieve a pruning rate of up to 99%, while still matching or exceeding the performance of various baseline and previous models.
Our approach has two main advantages.
First, and most notable, signed Supermask models drastically simplify a model’s structure, while still performing well on given tasks.
Second, by reducing the neural network to its very foundation, we gain insights into which weights matter for performance.
The code is available here.
I NTRODUCTION
In recent years neural networks have established themselves as versatile problem solvers far beyond the ﬁeld of machine learning.
However, the performance increases achieved are obtained by an even larger growth in their size.
This not only leads to growing energy costs (Strubell et al., 2020), but also complicates interpretability considerably.
There are different approaches to reduce various aspects of complexity within a neural network, which broadly can be categorized either as pruning, i.e., the technique of trimming down neural networks, or as low-precision training, i.e., representing the weights by low precision numbers.
Pruning has shown to signiﬁcantly reduce the size of a neural network (in terms of “active” weights) without affecting performance considerably, once the training phase has ended.
Compared to the original network, the pruned network is sparse, since many of the original, “useless” weights are set to zero.
In contrast, replacing weights by low precision numbers, reduces memory size, but does not reveal a sparse, likely easier to interpret network structure.
Many approaches of pruning (e.g., Janowsky (1989); LeCun et al.
; Han et al.
; Li et al.
(2016b); Molchanov et al.
(2019) among others) exist to identify “useless” weights.
This raises the question of why not train a smaller network from scratch, since the training of a large network is the computationally most expensive step and pruning is another additional step.
Frankle & Carbin (2018) present a possible solution to this question by formulating the Lottery Ticket Hypothesis (LTH), which states that any densely connected neural network contains smaller subnetworks that, when trained in isolation, perform just as well or even better than the original network.
They are able to identify such subnetworks ( winning tickets ) by iteratively pruning the network based on the magnitude of each weight.
For deeper architectures, Frankle et al.
(2019) show, that pruning after a few epochs of normal training works signiﬁcantly better over pruning at initialization.
Zhou et al.
(2019) follow the seminal idea of the LTH and are able to train neural networks by only selecting untrained weights (i.e., weights are frozen after initialization), a concept they call
Supermasks .
In other words, they ﬁnd a smaller subnetwork during training without adjusting the weights themselves.
Although this approach did not match the performance of their baselines and the pruning rate is inconsistent, it revealed a startling insight: weight values do not seem to be as important as the connection itself; a single, well-initialized value for each layer is sufﬁcient.
Ramanujan et al.
(2020) further develop Supermasks by modifying the way masks are calculated, which leads to signiﬁcant performance improvements compared to Zhou et al.
However, the number of parameters is still high.
Recently, Chijiwa et al.
modiﬁed the approach of
Ramanujan et al.
(2020) by randomizing the scores, but prune their networks only up to 60%.
In this paper, we propose a technique called signed Supermask , a natural extension of Zhou et al.
(2019) and Ramanujan et al.
We not only determine the importance of a weight by masking, but also learn the respective sign of a weight.
Signed Supermasks aim at very sparse structures and are able to uncover subnetworks with far fewer parameters in the range of 0.5% to 4% of the original network requiring little additional computational effort without sacriﬁcing performance.
This differs substantially from low precision training.
In its most extreme form, low-precision training quantizes the weight values of a neural network to three constants, including zero or two constants, excluding zero.
There, binarized neural networks (BNN) (Courbariaux et al., 2015) reduce complexity but not the size of the networks in terms of sparsity of weights.
Ternary neural networks (TNN), introduced by Li et al.
(2016a) allow in principle for sparse subnetworks, however this literature focuses almost exclusively on optimizing computational costs while maintaining predictive performance.
For TNNs to reduce computational complexity, sparsity (and interpretability) is of no importance, as they are focused on reducing the computational footprint only.
The literature presents diverse approaches, for example Shayer et al.
(2017) utilize the local reparameterization trick (Kingma et al., 2015) to learn ternarized weights stochastically, Zhu et al.
(2016) ternarize the weights but scale them layer-wise by two learned real-valued scalars and Alemdar et al.
(2016) employ a teacher-student approach.
Deng & Zhang (2020) train their networks normally with an additional regularization term in the loss function and only ternarize at the end of training by rounding.
To the best of our knowledge, this is the only work on TNNs also reporting on sparsity.
More recently,
Diffenderfer & Kailkhura (2021) combined the edge-popup algorithm by Ramanujan et al.
(2020) and the binarization of weights and achieve good results.
Thus, although TNNs and Supermasks in general appear similar at ﬁrst glance, their goals are different: while the former attempt to reduce computational complexity, the latter attempt to ﬁnd the smallest possible subnetworks within a neural network to better understand neural networks in general and work towards facilitating interpretability.
This paper takes the Supermasks perspective and our experiments show, that signing the Supermask matches or outperforms baselines and state-of-the-art approaches on Supermasks and leads to very sparse representations.
A convenient side effect of signed Supermask is the ternarization of weights, with implied reduced memory requirements (Li et al., 2016a) and further signiﬁcant speedup in inference (Hidayetoglu et al., 2020; Brasoveanu et al., 2020).
In summary, signed Supermasks provide two major advantages.
First, the network structure is simpliﬁed while maintaining or improving the performance compared to their dense counterpart.
Second, the reduction facilitates a better understanding of the inner mechanics of neural networks.
Based on that, we might be able to build smaller but equally powerful models a priori.
Additionally, once trained, signed Supermask models can be stored more efﬁciently and have the potential for faster inference.
S IGNED S UPERMASK
The idea of a signed Supermask is simple, but elegant: the network may not only switch off a given weight, but may also ﬂip its sign if deemed beneﬁcial.
We do this by multiplying each weight with a mask m ∈ {− 1 , 0 , 1 } , in contrast to m ∈ { 0 , 1 } as in previous work (Zhou et al., 2019; Ramanujan et al., 2020).
For example, if a single weight is initialized with a negative sign but the performance of the network would increase if it was positive, the signed Supermask has the additional degree of freedom to ﬂip the weight’s sign.
This step decreases the impact of random initialization, which matters speciﬁcally for a signed constant initialization , as introduced by Zhou et al.
in the context of Supermasks.
Despite including -1 in m , we have no additional information to store for the ﬁnal sparse weight matrices (i.e., the ﬁnal mask multiplied with the weight matrix), since it already includes the weight value, the sign and the zero.
In this regard, the additional ﬂexibility in the training phase comes for free.
Note that signed Supermasks are applicable to any neural network architecture.
By masking, we neither alter the information ﬂow, nor any architecture speciﬁc properties.
For the sake of brevity, however, we focus in the following on the application on feed-forward neural networks only.
The general setup is as follows:
Let W l ∈ R n l × n l − 1 denote the frozen weight matrix of layer l ∈ { 1 , . . .
, L } of a feed-forward neural network with L layers and n l denoting the number of neurons in layer l .
We initialize W l ∼ D w .
Let further M l ∈ R n l × n l − 1 , M l ∼ D
m be a learnable, real-valued matrix that is transformed to values {− 1 , 0 , 1 } by a element-wise function g , where
[Equation]
for some thresholds τ n and τ p as well as appropriate distributions D w and D m that will be speciﬁed below.
We now deﬁne the signed Supermask as ¯ M :=
g ( M ) , i.e., a Supermask as in Zhou et al.
(2019) with the additional degree of freedom of a negative sign.
Further note that, compared to
Zhou et al.
(2019), the function g ( · ) is deterministic.
In order to make full use of the signed
Supermask, we utilize the ELU ( E xponential L inear U nit) activation function (Clevert et al., 2015) with hyperparameter α .
Hereafter, we always set α = 1 .
Then, with (cid:12) symbolizing the Hadamard product, each layer l ’s output o l with z l = ( W l (cid:12)
[Equation]
T o
[Equation]
1 is computed as follows:
[Equation]
The parameters τ n,p are hyperparameters that need to be tuned: the larger they are in absolute value, the more weights are pruned at initialization and the larger values in M need to grow during the course of training to “activate” its responding weight.
The values of τ n,p might differ depending on the task and network architecture at hand.
The gradient is estimated using the straight-through estimator (Hinton, 2012; Bengio et al., 2013) as in previous work (Zhou et al., 2019; Ramanujan et al., 2020).
That is, in the backward-pass, we regard the threshold function g ( · ) as the identity function.
Thus, the gradient is going “straight through” g ( · ) .
Zhou et al.
(2019) generate their masks by sampling the entries from Bernoulli distributions with learnable Bernoulli probabilities.
Their M therefore resembles the probability, that weight w ij in some layer l is active.
Sampling from a Bernoulli distribution can also be interpreted as a stochastic threshold function.
Our ﬁxed threshold approach follows the same intuition with the difference of not using a stochastic sampling method.
That is, we deﬁne two thresholds τ n
and τ p after the initialization phase which remain ﬁxed during the course of training in contrast to Ramanujan et al.
(2020) who update τ n and τ p after each epoch such that the pruning rate stays constant.
Having ﬁxed thresholds has three main advantages which are in line with the argumentation of Zhou et al.
(2019): ﬁrst, the network can choose the optimal number of remaining weights.
Second, it can choose the optimal distribution of 1 ’s and − 1 ’s. Third, this approach is fast, as we only have to compute τ n and τ p once.
Moreover, this method, with some modiﬁcations, is frequently used in the TNN literature, which underlines its legitimacy.
As a result, an exact pruning rate cannot be determined a priori, but is learned.
However, we show in Appendix F, that the pruning rate can be indirectly inﬂuenced by adjusting other hyperparameters.
2. T HOUGHTS ON I NITIALIZATION
Signed Supermasks do not alter the value of any weight during training.
Therefore, it is especially important to initialize weights appropriately.
Common weight initialization methods such as He (He et al., 2015) or Xavier (Glorot & Bengio, 2010)
neither take masking into account nor the ELU activation function.
Weights We propose a weight initialization scheme that considers both aforementioned alterations, namely, the signed masking process and a differing activation function.
In simple terms, since each layer holds fewer weights, we can scale the weight value to counteract masking.
Considering that the goal of initialization methods is to keep the variance in the forward- and backward-pass constant, we suggest to use the following variance:
[Equation]
with fan-out n l and layer-wise equal, initial probabilities p l 0 = P ( ¯ M lij = 0) for layer l ∈ { 1 , ..., L } and ∀ i ∈ 1 , . . .
, n l , j ∈ 1 , . .
. , n l − 1 .
A detailed formal motivation of Equation 3 under speciﬁed assumptions can be found in Appendix A.
The result is still valid without masking, i.e., p l 0 = 0 .
He et al.
(2015) come to a similar conclusion for their Parametric ReLU (PReLU) activation function.
Henceforth, we refer to this initialization method as ELUS ( ELU S caled) with masking or simply
ELU in case of no masking.
Notice that ELUS is simply a scaled version of He initialization (He et al., 2015), where the scalar is only dependent on the hyperparameter α of the ELU activation function which is known before training (and set to 1 in this work).
Mask Our assumptions and observations regarding the initialization of the real-valued mask M are as follows:
We assume that D m is symmetric around zero, a standard assumption for common initialization distributions.
Given the properties of M , ¯ M and the gradient straight-through estimator, it is advantageous if the values in M and W are roughly of the same magnitude, such that the gradient, which is independent of M , is neither too small nor too large.
In other words, D m should be chosen such that it is symmetrical around zero and has a similar variance as D w .
In the case of mask initialization, we advice against a signed constant approach but recommend a normal or uniform distribution instead.
E MPIRICAL A SSESSMENT
The following paragraphs present the experiments conducted in order to assess the validity and effectiveness of ﬁxed threshold signed Supermasks.
The code for the experiments can be found here.
Experimental Setup To ensure comparability, we utilize the same experimental setup as Zhou et al.
and Frankle & Carbin (2018) i.e., models FCN, Conv2, Conv4 and Conv6 (VGG-like architectures (Simonyan & Zisserman, 2014))
with the additional Conv8 model of Ramanujan et al.
(2020) without batch normalization (Ioffe & Szegedy, 2015) and dropout (Srivastava et al., 2014).
Since we want to study the functionality of signed Supermasks in particular, any extra functionalities would disturb from the essential topic even though it might bump performance.
Details about the aforementioned model architectures can be found in Table 7 in Appendix B. Furthermore, we study signed Supermasks on ResNet20, ResNet56 and ResNet110 (He et al., 2016) with the exception of replacing ReLU with ELU (similar to Shah et al.
The batch normalization layers in the
ResNet models are frozen for signed Supermask training and are only used because its standard practice.
For all experiments the same architectures as the respective signed Supermask models but trained conventionally, act as baselines.
The fully-connected (FCN) model is trained on MNIST (LeCun et al., 2010) and all CNNs plus
ResNet20 on CIFAR-10 (Krizhevsky, 2009).
We only apply per image standardization on both of those datasets.
ResNet56 and ResNet110 are trained on CIFAR-100 (Krizhevsky, 2009), where we apply per image standardization, pad the image with 4 pixels on each side and randomly crop a 32x image out of the original or its horizontally ﬂipped counterpart.
We ran each experiment 50 times (ResNet56/ResNet110: 3 runs) with the same setup but different weights and masks drawn from the respectively same speciﬁed distribution.
This ensures that our experimental results are not affected by the randomness of the initialization step.
To obtain an intuition of the scale of memory reduction, we provide a simple metric by comparing the average network size in bytes stored in numpy arrays and scipy’s csr_matrix 1 sparse matrices after training.
For matrices with more than 2 dimensions, we reshape the matrix to two dimensions.
This does not give an exact result but provides a good heuristic.
Since we attempt to reduce complexity of a neural network, we also provide the average time needed to train a model for a single epoch to further assess practicability.
[Table]
Initialization To initialize the weights of the signed Supermask models, we use signed constants in combination with Xavier, He and ELUS.
Thus, the respective weight value has a 50% probability of being negative, where the absolute value of the constant is inﬂuenced by the respective initialization method.
We initialize the baseline models with the same method within a uniform distribution as the respective signed Supermask model.
The idea of using only one (well-chosen) signed constant is particularly compelling from the perspective of complexity reduction.
For the ELUS initialization, we scale a He initialization by √ 3 for FCN and Conv architectures and √ 1 . 5 for our ResNets, which returned best results in the preliminary testing phase.
All masks are initialized with Xavier uniform initialization.
Results of experiments with different mask initializations and distributions can be found in Appendix C.
3. F ULLY -C ONNECTED N ETWORKS
This section evaluates the performance of the fully-connected neural network (FCN) signed Supermask model.
Hyperparameters used for training the FCN are listed in Tables 8 and 10 in Appendix B.
Note that compared to Zhou et al.
(2019), the parameter choices are in line with standard practice.
Table 1 summarizes the average test accuracy, remaining weights and training time per epoch with the respective 5% and 95% quantiles for both baseline and signed Supermask models with the different weight initializations.
All models achieve similar test accuracy.
Best performing is the ELUS signed
Supermask model.
Furthermore, all signed Supermask models have a wider conﬁdence interval, indicating higher variance.
In contrast, conﬁdence intervals for the ratio of remaining weights are very small, demonstrating constant pruning rates throughout all runs.
Furthermore, ELUS’ ratio of remaining weights of 3.77% is considerably lower than for the other two initializations.
This indicates a superior initialization as the network is able to achieve a higher performance with fewer weights remaining.
For another angle of observation, Figure 5 in Appendix C visualizes the average ratio of remaining weights over the course of training for each weight initialization.
Regarding efﬁciency, ELUS signed Supermask models require about 5% more training time.
However, the obtained compression rate for the ELUS model after training is 92.01% on average, which makes up for the additional training effort.
Although the performance of the ELUS signed Supermask is trailing 0.5pp behind the FCN model of Zhou et al.
(2019), we argue that considering the pruning rate (they state the pruning rate only imprecisely, ranging from 7% to 89%) and performance, signed
Supermasks are advantageous.
To assess how the model interprets a given dataset, we examine the ﬁrst masking layer of the network, visualized in Figure 1.
The top image represents an input vector (i.e., a ﬂattened MNIST sample), with each vertical line representing one pixel.
The bottom shows the mask of the ﬁrst layer of a randomly selected trained ELUS FCN model.
The signed Supermask has learned which of the input nodes are more important than others.
Those that match a background pixel are completely masked, while the input nodes that receive actual information are allowed to let the information pass through.
Thus, the ﬁrst layer acts as a ﬁlter and feature selector.
We further analyze the behavior of the masks and draw comparisons to binary Supermasks in Appendix C.
The results so far show the potential of signed Supermasks in relatively small architectures.
Not only does the ELUS model outperform the respective baselines with only 3.8% of the original weights, it also saves 92% of memory once trained.
Furthermore, we gain insights into layer interpretability as pruning is not evenly distributed across the network.
3. C ONVOLUTIONAL N ETWORKS
Here, we present the main experimental results of convolutional neural networks (CNN) signed
Supermasks.
Overﬁtting was a major issue in the development, especially for the baseline models.
To counter that, the baseline models are only trained for 50 epochs.
In contrast, signed Supermask models were not as prone to overﬁtting.
However, once we anticipated overﬁtting tendencies of
Conv2, we reduced its learning rate compared to the other CNN models.
Apart from those alterations, the hyperparameter choices are the same throughout all CNN architectures.
All models are optimized with SGD with weight decay.
Learning rate for Conv models are exponentially reduced over the course of training.
Tables 8 and 10 in Appendix B summarize all training parameters.
We have found in initial experiments that ± 0 .
01 is a solid choice for the threshold parameters τ n and τ p .
This corresponds to an initial overall pruning rate of about 11% to 30%.Thus, larger layers initially experience a slightly higher pruning rate than smaller layers, following the intuition that weights in smaller layers impact the output of the network to a greater extent than weights in larger layers.
Table 2 presents test accuracy for our baselines, the models of Zhou et al.
and Ramanujan et al.
(2020) and signed Supermasks as well as remaining weights (i.e. 1 - pruning rate) for the latter three.
For a broader comparison to literature, see Table 14 in Appendix C. While we were able to reproduce the results of Zhou et al.
(2019), this was not the case for Ramanujan et al.
(2020) which not only delivered worse results than reported but also demanded a multiple of the training time.
The interested reader can ﬁnd additional results, experiments on different weight and initialization methods as well as investigations on extreme pruning rates in Appendices C D, E and F. First, apart from Conv2, all
CNN models signiﬁcantly outperform their respective baselines.
Conv2 reaches similar accuracy to its baseline, the difference only being 0.4pp.
The more our models grow in depth, the greater the beneﬁt of signed Supermasks over the baselines.
When considering the ratio of remaining weights, a likely reason for the slight performance glitch of Conv2 becomes apparent: it utilizes only 0.6% of the original weights.
In other words, Conv2 reaches similar performance with 99.4% of the original weights unused.
Conv4, the smallest model in terms of absolute parameter count, keeps the highest percentage of weights in relation to the original count.
As the networks grow in size and depth, we can once again note an increase in the pruning rate.
Except for Conv6, there is hardly any variance at all in this metric, indicating a robust behavior.
[Table] [Table] In terms of efﬁciency, we see in Table 3 that the signed Supermask models require 6% to 10% more training time, which can be accounted to the additional calculations necessary.
However, once the networks are trained the required memory can, on average, be compressed by 94% to 98%.
Signed
Supermasks outperform not only their respective baselines but also the methods proposed in the related work.
Compared to Zhou et al.
(2019), our Conv 2, Conv4 and Conv 6 models achieve a higher performance of at least 2pp.
Diffenderfer & Kailkhura (2021) do not report performance over a 90% pruning rate, however, their models achieve higher accuracy than ours.
We account this to their lower pruning rate.
Ramanujan et al.
(2020) do not report model performance below a pruning rate of 90% as well.
Indeed, in this case, all our Conv models outperform their equivalent models by 3 to 10 pp, while uncovering much smaller subnetworks.
[Figure]
e l .
[Figure]
oun t [
[Figure] [Figure]
R e l .
[Figure]
t [
[Figure]
needed
[Figure]
Figure 2a helps to interpret the masks by visualizing the average layer-wise mask distribution of each architecture.
It can be clearly seen that the ﬁrst layers are least affected by pruning, while all other layers reach a signiﬁcantly higher pruning rate of over 99%.
As the layer size increases, the pruning rate also increases and peaks in the largest layer of the network.
The output layers are also relatively sparse, again indicating proper initialization.
This ﬁnding conﬁrms our previous assumption that larger layers contain far fewer weights that are essential to the network.
The relative element-wise equality of the ﬁnal masks over all 50 runs for all layers and architectures is shown in Figure 2b.
This provides an intuition about the similarity of the masks and therefore subnetworks through different runs.
We observe that pairwise equality is strongly correlated with the size and sparsity of a layer: the larger, the sparser, the more alike.
To some degree this is expected as very sparse layers are alike by chance.
However, measuring similarity in matrices is a very difﬁcult task (for an overview on this literature from a graph-based perspective see, e.g. Emmert-Streib et al.
(2016)), which is why we provide visualizations in Figure 9 in Appendix C that emphasize especially the similarity in structure.
This shows, that signed Supermasks repeatedly ﬁnd strikingly similar subnetworks for different runs, meaning that it might be possible to identify certain neurons that are more important than others and hence better understand neural networks in general.
3. R ESIDUAL N ETWORKS
To further investigate the effectiveness of signed Supermasks, we turn our attention towards Residual networks which allow for deeper architectures.
The investigated ResNet consist of 9,27,54 residual blocks, each in equal parts with 16, 32 and 64 ﬁlters, respectively.
This sums up to a comparatively small parameter count of roughly 0.25M, 0.84M and 1.7M. As hypothesized above, signed Supermasks work better in deeper architectures that contain enough parameters to ﬁnd a good subnetwork.
To investigate this, we multiply the number of ﬁlters by 1.5, 2, 2.5 and 3 to create wider variants of
ResNet20.
Note that even the largest of those variants is smaller than any of the studied VGG-like architectures above, suggesting less overparameterization, especially for the more complex CIFARdataset.
Hyperparameters for baseline and signed Supermask models are depicted in Tables 9 andin Appendix B.
Compared to the other investigated architectures, we only reduce the learning rate once the train loss hits a plateau.
Furthermore, since there is no large deviation between layer sizes, we set the threshold parameters τ n and τ p such that only 25% of the weights are left for ResNet and 15% for ResNet56 and ResNet110.
Table 4 shows the test accuracy and remaining weights of both baselines and signed Supermasks trained on CIFAR-10.
Notably, signed Supermask always trail 2 to 3 pp behind their respective baseline.
However, once the signed Supermask model gets wide enough it can reach the performance of the original ResNet20 with only a third of the parameter count.
This ﬁnding is partially in line with Ramanujan et al.
(2020) and their scaling of ResNet50.
As expected, the pruning rate increases as the network grows in depth.
We suspect that weight count could be further dropped in the deep architectures by adapting the weight decay parameter, for the sake of clarity however, we decided to keep all learning parameters the same.
A possible reason for the slightly lower performance of signed
Supermasks might be the frozen batch normalization layers, which would beneﬁt learning greatly.
We investigate this aspect in Section 3.3.1.
The performance and remaining weights of ResNet56 and
[Table]
ResNet110 on CIFAR-100 are depicted in Table 5.
Signed Supermasks show, that they successfully learn from the dataset, even in deep and relatively small environments.
However, they cannot reach the same test accuracy as their respective baseline.
This can be partially explained by the relatively small original network sizes.
Furthermore, as hypothesized above, batch normalization is of great importance especially in deep architectures like ResNet110, which thereby particularly suffered.
[Table]
3.3. T HE R OLE OF B ATCH N ORMALIZATION
ResNets excel compared to conventional CNNs because of their depth.
A big role in being able to learn very deep architectures is batch normalization.
Here, we want to study the inﬂuence of batch normalization on the ResNet signed Supermask architectures.
For those experiments, we explicitly turn on the batch normalization layers in the respective architecture.
This parts a bit from the minimalist idea of Supermasks in general, however, weights are still frozen at initialization and this step might be necessary for deep architectures.
Table 6 reports the results of the baseline models (as displayed above) and the signed Supermask with learnable batch normalization layers.
Compared to the pure signed Supermask models, batch normalization helps to push performance while increasing pruning rate for all models.
However, they are unable to reach the performance of the baselines.
Nevertheless, taking the remaining weights into account, the predictive performance impresses.
More research is needed to investigate the performance gap.
Besides this, the results underline that batch normalization helps signed
Supermasks to gain better performance in very deep architectures
[Table]
S UMMARY
In this work
[Table]
we introduced the concept of signed Supermasks .
Extending the original work on
Supermasks (Zhou et al., 2019), we include the sign in the masking process and utilize a simple step-function to calculate the effective masks.
Additionally, we gave theoretical and experimental justiﬁcation for an adapted weight initialization scheme to the masking process, called ELUS .
Our experiments on MNIST and CIFAR-10 show, that signed Supermask models at best hold as little as 0.6% of the original weights, while performing on an equal or superior level compared to our baselines and current literature (Zhou et al., 2019; Ramanujan et al., 2020).
As a side effect, the trained signed Supermask models’ memory footprint can be compressed by up to 98%.
Further analyses on CIFAR-100 with ResNet56 and ResNet110 suggest that signed Supermasks are generally able to work on complex datasets and architectures as well.
E THICS S TATEMENT
The authors cannot envision any socioeconomic or ethical downsides that are speciﬁc to signed
Supermasks but not neural nets or machine learning in general.
On the contrary, we can see obvious beneﬁts, since signed Supermasks facilitate a drastic reduction of energy consumption when deploying neural nets and they can contribute to the interpretability, explainability and ultimately trustworthiness of deep learning approaches
R EPRODUCIBILITY S TATEMENT
The authors provide the complete code used to conduct all experiments in the supplementary material.
An exemplary conﬁguration and requirements ﬁle are attached as well, with which all experiments can be easily reproduced.
Importantly, all experiments were run on a single GPU, which simpliﬁes reproduction and strengthens the overall beneﬁts of signed Supermasks.
The code is written in Python, mainly using tensorﬂow and numpy.
All datasets used in this work are provided by tensorﬂow here.
A D ERIVATION OF ELUS
In the following paragraphs, we revise the initialization methods of He et al.
and Glorot &
Bengio (2010).
This is needed because ﬁrst, we introduce masking and a the ELU activation function.
Both concepts alter the assumptions of the cited work.
Here, we derive a novel adaption of the common He initialization (He et al., 2015).
In this section, we use the following notation:
• Activation function σ ( x ) • Layer l has n l neurons.
This means that a layer takes n l − 1 inputs per neuron.
In the literature, n l − 1 is sometimes called fan-in and n l is sometimes referred to as fan-out • W l ∈ R n l × n l − 1 , b l ∈ R n l ×• Weighted layer input
z l = W l o l − 1 + b l , z l ∈ R n l ×• Layer output
o l = σ ( z l ) , o l ∈ R n l ×• L denotes the network’s loss function • ∇
o l = ∂ L ∂ o l • ∇ z l = ∂ L ∂ z l = σ (cid:48) ( z l ) ∇
o l + • In order to maintain clarity, we will denote the transposed weight matrix of layer l W Tl as ˆ W l in the derivation of the backward-pass.
• ¯ W l = W l (cid:12) ¯ M l .
This leads to the suboptimal but necessary choice to denote the transpose of ¯ W l as ˜ W l (instead of ¯ W lT ) to maintain clarity.
We will further assume, that the following holds:
(1) We use the ELU activation function, introduced by Clevert et al. (2015).
(2) Variances of input features x are the same Var[ x ] .
For each layer l , the respective weight matrix W l is drawn from an independent and symmetric distribution with E [ W l ] =
and Var[ W l ] .
It follows that E
[ z l ] = 0 .
For each layer l , the respective mask matrix M l is drawn from an independent and symmetric distribution with E
[ M l ] = 0 .
To make the derivations more tangible, we assume p l 0 =
P ( ¯ M lij = 0) = 0 .
5 and p l 1 = P ( ¯ M lij = 1) =
P ( ¯ M lij = − 1) = 0 .
25 ∀ i ∈ 1 , . . .
, n l , j ∈ 1 , . .
. , n l − 1 at initialization.
In general, the variance is Var[ ¯ M l ] = 1 − p l 0 .
For each layer l , all
elements
in z l and o l share the same variance Var[ z l ] and Var[ o l ] ,
respectively.
Since we use ELU activation functions in our experiments, the question arises whether we can still use the initialization of He et al.
Clevert et al.
(2015) do so and achieve good results.
Let us therefore calculate the important values that might alter the assumptions necessary.
Like Glorot & Bengio (2010) and He et al.
(2015) differ in the choice of activation function, we can particularly investigate the parts of the derivation where the expected value of the activation function is used.
In particular, this is E [( o l − 1 ) 2 ] in the forward-pass and E [( ∇ o l ) 2 ] in the backward-pass.
We can calculate E [( o l − 1 ) 2 ] as follows:
[Equation]
From here on, we drop index l for the sake of clarity.
For the calculation of E [( αe z l − 1 − α ) 2 ] , we additionally assume that z l ∼ N (0 , 1) , although this is a simpliﬁcation and is therefore only an approximation.
With this, we obtain
[Equation]
We can observe that the last term is half of the area under the standard normal probability density function.
Note that it holds that − ax 2 + bx = − a ( x − b 2 a ) 2 + b 2 4 a .
With this, we can re-write ( a ) and ( b ) as follows (
[Equation]
We can see that ( a ) is the probability density function of N (2 , 1) and ( b ) is the probability density function of N (1 , 1) , which can be easily evaluated.
With ( a ) =
α 2 e 2 · Φ( z − 21 ) = 0 .
α 2 and ( b ) =
α 2 e 12 · Φ( z − 11 )
= 0 .
α 2 we have
[Equation]
If we take the case x > 0 into account, i.e. E [ z 2 ] = 12 Var[ z ] , we have
[Equation]
Let us now move on to calculating E [( ∇ o l ) 2 ] .
Note that the derivative of ELU is simply
if z >(which is the same as for ReLU) and αe z if z ≤ 0 .
Let us look at the squared expectation of the second case:
[Equation]
If we assume again, that z l ∼ N (0 , 1) , we immediately see that we already calculated this expected value in Equation 6.
Therefore, we have
[Equation]
For the total expected value, we then have
[Equation]
To summarize, in comparison to (He et al., 2015), we only have to scale Equations 9 and 12 with the constants h and k , respectively.
The results above are equal to Equation 15 in He et al.
(2015), where they assume a PReLU activation (PReLU as an activation function is also introduced in He et al.
However, they do not supply a derivation.
Therefore we cannot guarantee equality with conﬁdence.
Hence, we are careful with the pronoun “our” (initialization) and we will therefore simply refer to the initialization method that follows from the derivations above as “ ELU ”.
Let us now use these results and calculate Var[ W l ] .
Forward-Pass: We start with the following equation
[Equation]
Because E [ ¯ W l ] = 0 we can re-write this as
[Equation]
With Equation 9 it follows that
[Equation] [Equation]
If we want to achieve constant variance throughout the network, we can set
[Equation]
With Var[ ¯ M l ] = p l 1 · ( − 1) 2 + p l 0 ·
[Equation]
2 + p l 1 · 1 2 = 0 .
5 as stated above and α = 1 we have
[Equation] [Equation]
Note that for normal training, we would not have to account for the variance of the mask, which would result in Var[ W l ] = 1 . 5505 · 1 n l − 1 .
Backward-Pass: We know that
[Equation]
Recall that ELU derivative is
[Equation]
We know from Equation
that E [( σ (cid:48) ( z l )) 2 ] = 1
+ h and E [ ∇ o l +1 ] = 0 .
It follows, that
Var[ ∇ o l +1 ] =
E [
[Equation]
o l +1 ) 2 ] , resulting in
[Equation]
With the obtained results we can now re-write Equation 18 as
[Equation]
Herewith we are able to calculate the variance throughout all layers as
[Equation]
Like in the forward-pass, an easy way to keep the variance constant is to set
[Equation]
As above, if we use this initialization for training a normal neural network, we would use
Var[ W l ] = 1 . 49678 (cid:12) 1 n l .
We will follow the intuition of He et al.
by leaving the choice of Equation 23 or 17 to the user, as their initialization scheme has proven to outperform Glorot & Bengio (2010) in practice and takes the CNN-architecture into account.
However, since Equations 23 and 17 are relatively close to each other, we suggest to use the following variance for the sake of simplicity
[Equation]
B M ODELS AND H YPERPARAMETERS
[Table] [Table]
[Table] [Table]
[Table]
C F URTHER E XPERIMENTAL R ESULTS
[Table]
[Table]
The methods we compare our signed Supermasks to, are: Supermasks Zhou et al.
(2019), edge-popup
Ramanujan et al.
(2020), IteRand Chijiwa et al.
(2021), MPT-1/32 Diffenderfer & Kailkhura (2021),
FORCE de Jorge et al.
, SET Mocanu et al.
(2018), RigL Evci et al.
GraNet Liu et al.
(2021), TWN Li et al.
(2016a),
TTQ Zhu et al.
(2016), SCA Deng & Zhang (2020), LR-Net Shayer et al.
(2017), BinaryConnect Courbariaux et al.
(2015), BNN Courbariaux et al.
(2016) and BBG
Shen et al.
Considered models are Conv6 and Conv8, VGG-19, ResNet-20, ResNet-18 and
Wide ResNet 22-2.
VGG-Small summarizes all small VGG-like architectures, i.e. they can slightly differ.
We refer to the respective work for details.
The handling of the weights can be categorized as either a binary mask (“Masked (B)”), pruned weights where the weight values change during training, ternarized and binarized weights as well as ternary masks (“Masked (T)”)
as is the case for our signed
Supermasks.
[Table]
At ﬁrst glance the binary Supermask looks more erratic, while the signed Supermask inhibits a more ordered structure.
The impression is not deceiving: out of the 784 columns (i e. input neurons), 284 in case of the signed Supermask are completely masked, wheres the binary Supermasks only masks(we strongly assume, that the fact that one number is almost double the other is a coincidence).
That is, those columns were masked in every considered mask, which shows the robustness of Supermasks in general.
Even if we allow 100 outliers (that is, a column-wise maximum sum of 100) per column, binary Supermasks only count 260 of those columns (364 for signed Supermasks).
We can observe this fact in two parts: the signed Supermask is more sparse at the edges and the 0-columns in the center are wider and less noisy.
This leads to the conclusion that, although on average having the same pruning rate, signed Supermasks are more consistent (otherwise we would count fewer 0-columns) and more understanding of the given data, as it focuses on speciﬁc inputs and mostly disregards the image background, which (in the ﬂattened vector) is positioned at the sides.
This strongly suggests that not the pruning rate alone is of importance, but also the distribution of pruning per layer.
To summarize, although a comparison is not possible without compromise, we can see that the binary
Supermask, although exhibiting the same average pruning rate is far less consistent compared to the signed Supermask.
[Figure]
Baseline
[Figure]
Signed Supermask
[Figure] [Figure]
v
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
We also report the 5% and 95% quantiles.
Besides for the Conv6 model, all conﬁdence intervals are very narrow.
Furthermore, all architectures drop the weights heavily during the ﬁrst few epochs and plateau thereafter.
[Figure]
He
[Figure]
CN  2.
ELUS  2.
Xavier
[Figure] [Figure]
C o n v     4.
[Figure]
C
[Figure]
v
[Figure]
v
[Figure]
Taking the absolute signed Supermasks into account
[Figure]
we do not see any meaningful differences in those layers that gain high similarity in the signed Supermask case.
However, for the output layers across almost all models that were initialized with He and Xavier we see high similarity.
If contrasted with the pruning rate again, its notable that those layers are not pruned much.
This can be seen most clearly in the FCNs:
He and Xavier only prune the last layer very little, resulting in an absolute signed
Supermask that contains many 1’s.
The same holds for the CNN output layers and some CNN input layers.
Theseﬁndings suggest that the 0-elements are the main driver of mask similarity, i.e. the same weights are being pruned in every run.
Note that this metric is more takes a more heuristic-approach to analyze the problem.
Comparing matrices or tensors is a difﬁcult task.
Introducing more precise metrics would be far beyond the scope of this paper.
Interpretation of the masks, e.g. like in Figure 1 is very difﬁcult as the layers here are hidden and depend a lot on the previous and following layers.
Still, for each architecture the layers are remarkably similar which cannot be said for standard neural networks.
[Figure]
[Figure]
D I NFLUENCE OF W EIGHT D ISTRIBUTION
In this section we analyze the inﬂuence of the weight distribution on overall performance.
Therefore, we train additional models with weights drawn from a uniform distributions with the standard thresholds of He (He et al., 2015).
As a baseline, we utilize the signed Supermask models with weights as signed constants presented in Section 3.
For both variants, we arbitrarily choose He initialization as we expect similar behavior for the other initialization methods.
Hence in this Section, we call the former “He Uniform”, the latter “He SC” acting as the baseline.
Note that these experiments are not exhaustive, as we only examine the two mentioned distributions.
The purpose of this section is to give an intuition on the behavior of different weight distributions.
We ﬁrst analyze the FCN architecture, followed by a summary of the
CNN architectures.
FCN Figure 12 displays the average test accuracy and average ratio of remaining weights for both models with the respective 5%- and 95%-quantiles.
We can note that the variance of He Uniform is a lot higher than the variance of He SC.
Moreover, He Uniform performs on average slightly worse than He SC.
The ﬁnal mean result for He SC is 97.12%, whereas He Uniform achieves 96.77%.
A
Welch’s t-test for signiﬁcance yields that He SC signiﬁcantly ( p < 0 .
01 ) outperforms He Uniform.
The higher variance of He Uniform is also visible in the ratio of remaining weights.
On average,
He
SC and He Uniform achieve almost the same level of sparsity, however the 5%- and 95%-quantile are almost 2pp apart for He Uniform.
We postulate that a uniform distribution does not have additional value compared to signed constants for a dense neural network in the context of signed Supermask.
Apparently, initializing weights with small values deteriorates performance.
Recall that neither He SC nor He Uniform are scaled for the use of a signed Supermask.
We further speculate that the higher variance is a cause of random initialization such that training cannot nullify the impact of randomness and ill-initialized weights.
This leads to the conclusion that the value of a weight is important if not initialized well, as He
Uniform holds more weight values compared to He SC and performs worse.
To summarize, using signed constants instead of a uniform distribution provides a much higher level of robustness for the FCN.
Our results suggest that for signed Supermask and dense neural networks, a signed constant initialization surpasses the performance of a uniform distribution.
CNN We now investigate the inﬂuence of weight distribution on our CNN architectures.
Figure 13 visualizes the average test accuracy with the respective 5% and 95% quantiles during training for each model.
We ﬁrst note that He Uniform is not behaving consistently.
While with Conv2 and
Conv8 the variance of He Uniform is much higher compared to He SC, it is roughly equal for Conv and Conv6.
The average test accuracy of He Uniform is of the same magnitude as He SC, and we can only register a signiﬁcantly ( p < 0 .
01 ) better performance for Conv2 He SC compared to He
Uniform.
This is not sufﬁcient to conclude on one distributions signiﬁcance for the general case.
The results indicate that the additional weight values delivered by a uniform distribution do not improve the learning capability of a signed Supermask model.
This is in line with the ﬁndings of the FCN architecture.
It seems sufﬁcient to initialize the weight matrix with a well-chosen constant.
We further speculate that due to the higher parameter count of the investigated CNN architectures compared to the FCN, the optimizer is to some extent able to neglect the inﬂuence of poorly initialized weights.
Let us consider the average ratio of remaining weights over the course of training for each CNN model.
Figure 14 visualizes this metric for each network in addition to the 5% and 95% quantiles.
Unlike observed in the case of the FCN, we cannot report a high variance for He Uniform within
CNN models.
In fact, the weight distribution does not seem to have an impact on the level of sparsity, as the ratios move almost equally over the course of training for each model.
Considering both the performance and level of sparsity, we do not ﬁnd that the choice between uniform and signed constant distribution impacts the end result on average.
However, since the test accuracy of He Uniform varies a lot more compared to He SC, the ﬁndings suggest to prefer signed constants over a uniform distribution, as signed constants bring positive side-effects such as efﬁcient
[Figure] [Figure]
Epoch
[Figure]
He Uniform
He SC
Epoch
R e m a i n i ng
W e i gh
t s [ % ]
He Uniform: 4.77%
He SC: 4.93%
[Figure]
and
To summarize both FCN and CNN behavior, it can be stated that a uniform distribution is not advantageous over a signed constant approach.
We further want to re-emphasize the simplicity, yet good performance of signed constants.
They have shown a very robust behavior with regards to the
R e m a i n i ng
W e i gh
t s [ % ]
Conv
He Uniform: 1.64%
He SC: 1.69%
Conv
He Uniform: 2.9%
He SC: 3.05%
Epoch
R e m a i n i ng
W e i gh
t s [ % ]
Conv
He Uniform: 2.75%
He SC: 2.75%
Epoch
Conv
He Uniform: 1.15%
He SC: 1.19% variance in test accuracy while performing equal or better compared to uniform further simpliﬁes the of network as
E I NFLUENCE OF M ASK I NITIALIZATION
We now pursue the question to what extent mask initialization plays a role in performance.
Therefore, we use the ELUS models of Sections 3 as a baseline with Xavier uniform mask initialization (abbreviated in this section as “ELUS/Xavier”).
Furthermore, we train the same models with ELUS uniform mask initialization (abbreviated in this section as “ELUS/ELUS”).
We compare the results separately for the FCN and CNN architectures.
As in the last section, we note that these experiments are not exhaustive but give an intuition on the relationship between mask and weight initialization.
FCN Figure 15 visualizes the average test accuracy and average remaining weight ratio with the respective 5% and 95% quantiles.
We can note that there is almost no difference in variance and average test accuracy for ELUS/ELUS and ELUS/Xavier.
The former achieves an average accuracy of 97.36%, whereas the latter marginally but signiﬁcantly ( p < 0 .
01 ) exceeds ELUS/ELUS with an average accuracy of 97.48%.
As stated in Section 2.1, we presume that mask initialization is not as impactful, as long as weight and mask values are of the same magnitude.
The FCN results indicate to support this thesis.
We ﬁnd a slight deviation in training behavior in the ratio of remaining weights during training.
ELUS/ELUS reduces its weight count slightly later in training as ELUS/Xavier does.
However, the results are again very alike regarding mean and variance.
We can conclude that for the FCN, the difference in mask initialization did not inﬂuence training behavior and robustness meaningfully, yet ELUS/Xavier outperformed ELUS/ELUS slightly but
[Figure]
[Figure] [Figure]
Epoch
R e
[Figure]
i n
[Figure]
ng
[Figure]
gh
[Figure] [Figure]
ELUS/ELUS:
[Figure]
ELUS/Xavier:
[Figure] [Figure]
signiﬁcantly by 0.12pp.
Moreover, the average ratio of remaining weights differs only by 0.1pp.
Thus, we argue that both weight/mask initialization combinations might be useful, depending on the task at hand.
CNN Our focus shifts towards the CNN architectures.
Figure 16 displays the average test accuracy during training with the respective 5% and 95% quantiles for each architecture.
We can note a very robust training behavior for both mask initializations, as variance is equal across all models.
Observe that only Conv6 ELUS/ELUS performs signiﬁcantly ( p < 0 . 01 ) better than ELUS/Xavier.
We further ﬁnd that for Conv4, Conv6 and Conv8 ELUS/ELUS begins to optimize effectively a few epochs later than ELUS/Xavier.
Yet, this does not inﬂuence the ﬁnal result.
We assume this is caused by the optimizer and straight-through estimation, as the gradient might not be approximated properly in the very early learning phase.
Figure 17 visualizes the ratio of remaining weights for each CNN architecture with the respective 5% and 95% quantiles.
First, we note that each model has a distinct behavior.
Whereas Conv2 ascends its weights rapidly, it then almost linearly reduces the weights further until it plateaus.
The other architectures exhibit a similar behaviour towards the end of training, but without the linear section.
This is in line with the ﬁndings in Section 3.
Additionally, we observe that the choice of weight/mask initialization has little inﬂuence on the respective curves for each architecture.
As already seen with the FCN model, ELUS/ELUS drops its weights a few epochs later than ELUS/Xavier.
Moreover,
ELUS/Xavier utilizes 0.02pp to 0.4pp weights less than ELUS/ELUS, depending on the network architecture.
Both patterns are consistent for FCN and CNN models.
We hypothesize, that the later rise of accuracy and later drop of weight count for ELUS/ELUS is a cause of the gradient estimation.
The shift between the two initializations is consistent throughout all architectures.
It is furthermore consistent that ELUS/ELUS results in a marginally larger ratio of remaining weights.
[Figure] [Figure]
e m a i n i ng
W e i gh t s
[Figure]
F H OW FAR CAN WE GO ?
Two important questions still need to be answered.
Can we control the pruning rate?
And if so, how small can the subnetworks get before the performance is impacted negatively?
As we saw in
Table 2, Conv2 achieved the lowest pruning rate of all CNN models.
However, as described in the beginning of this section, we lowered the learning rate of Conv2 to tackle its tendencies to overﬁt.
Hence, we can investigate if all CNN models show a similar behavior when altering the learning
[Table]
parameters in the same way.
Table 15 depicts two altered parameter settings.
Why the learning rate somewhat inﬂuences pruning is for future work to investigate.
In the following, we will call the networks initialized with ELUS and trained with those hyperparameters SiNN ( S imple-m i nded
N eural N etworks) models.
As Figure 19 reports, we see that both models prune more than 99% of the original weights, SiNN 1 reaching even pruning rates above 99.5%.
Considering the predictive performance we can note, that the performance of SiNN 1 trails signiﬁcantly behind SiNN 2 for each model.
Let us compare the SiNN models to the baseline models.
Figure 18 depicts the test accuracy of the SiNN 2 and baseline models as well as remaining weights of SiNN 2 and ELUS models on the right side.
As SiNN 2 achieves higher accuracy than SiNN 1 with only few more weights, we do not further consider SiNN 1 at this point.
As the compression rate after training highly correlates with the ratio of remaining weights, we do not show this metric for the sake of brevity.
It can be observed that the SiNN 2 models achieve a similar test accuracy with a maximum of about 0.6% of the original weights.
This is a further 50% reduction of numbers of weights compared to the ELUS models.
Additionally, it is interesting to see, that the remaining absolute weight count of each CNN model is around 14.000 weights.
This suggests that this is the absolute minimum of weights that are needed to map the dataset accurately, independently of the model architecture.
The performance difference for SiNN 2 between the different architectures then stems from a deeper arrangement of weights, supporting the hypothesis that deeper architectures (with fewer weights) are beneﬁcial.
These results reveal three ﬁndings: First, we are able to indirectly control the pruning rate of signed
Supermask models by adapting the learning parameters.
Second, the SiNN 2 models perform similar to their respective baselines.
The uncovered subnetworks however, utilize only 0.3% to 0.6% of the original weights.
Third, although performance suffers compared to the ELUS models, the signed
Supermasks still perform well, even when pushed to the extreme
Ultimately
, this demonstrate not only the performance quality of the models but also how few parameters are actually needed in general to solve a given task well.
[Figure]
[Figure]
the ﬁrst layer even more.
This further supports the thesis that the ﬁrst layer in CNNs is crucial for the network’s performance, as already stated for the ELUS models.
By comparing the masks of SiNN1 and SiNNin greater detail, future work could ﬁnd e.g. speciﬁc weights that inﬂuence the performance more than other weights and subsequently draw conclusions on network architectures in general.
Epoch
[Figure] [Figure]
[Figure] [Figure]
Epoch
R e m a i n i ng
W
[Figure]
i gh t s [ % ]
ELUS
SiNN SiNN
[Figure]
CN
[Figure]
C o n v
[Figure]
C o n v
[Figure]
v
[Figure]
v
[Figure]