P LANT ’ N ’ S EEK : C AN Y OU F IND THE W INNING T ICKET ?
A BSTRACT
The lottery ticket hypothesis has sparked the rapid development of pruning algorithms that aim to reduce the computational costs associated with deep learning during training and model deployment.
Currently, such algorithms are primarily evaluated on imaging data, for which we lack ground truth information and thus the understanding of how sparse lottery tickets could be.
To ﬁll this gap, we develop a framework that allows us to plant and hide winning tickets with desirable properties in randomly initialized neural networks.
To analyze the ability of state-of-the-art pruning to identify tickets of extreme sparsity, we design and hide such tickets solving four challenging tasks. In extensive experiments, we observe similar trends as in imaging studies, indicating that our framework can provide transferable insights into realistic problems. Additionally, we can now see beyond such relative trends and highlight limitations of current pruning methods.
Based on our results, we conclude that the current limitations in ticket sparsity are likely of algorithmic rather than fundamental nature. We anticipate that comparisons to planted tickets will facilitate future developments of efﬁcient pruning algorithms.
I NTRODUCTION
Deep learning has achieved breakthroughs in multiple challenging areas pertaining to machine learning but is computationally highly demanding.
The beneﬁts of overparameterization for training with SGD (Belkin et al., 2019) seem to call for ever wider and deeper neural network (NN) architectures.
Training smaller, adequately regularized NNs from scratch to save resources could be a remedy but it seems to commonly fail, as Frankle & Carbin (2019) noted in their seminal paper.
However, they propose the lottery ticket hypothesis, which states that a small, well trainable subnetwork can be identiﬁed by pruning a large, randomly initialized NN, opening the ﬁeld to discover such subnetworks or ’winning tickets’.
Ramanujan et al.
(2020) went even further and conjectured the existence of strong lottery tickets, i.e., subnetworks of randomly initialized NNs that do not require any further training.
This hypothesis holds the promise that training NNs could potentially be replaced by efﬁcient pruning.
The existence of strong lottery tickets has been proven for networks without biases (Malach et al., 2020; Pensia et al., 2020; Orseau et al., 2020) and with biases (Fischer et al., 2021).
While these types of proofs show existence in realistic settings, the sparsity of the constructed tickets is likely not optimal, as they rely on pruning neurons to degree 1 .
The construction and proof of the generalized strong tickets raises two questions – is the suboptimal sparsity merely an artifact of existence proofs or a general limitation of pruning and hence further improvements are needed to achieve effective network compression?
And, if they exist, are current algorithms able to ﬁnd very sparse tickets?
We could answer the ﬁrst question by assessing the sparsity of tickets that have been found by network pruning, which however is limited to sparsities that current state-of-the-art pruning algorithms are able to achieve.
Hence, both raised questions require to properly evaluate the state-of-the-art algorithms for pruning networks before training. In a recent study, Frankle et al. evaluated existing pruning algorithms and concluded that there is no single winning method across considered settings and sparsity levels (Frankle et al., 2021). Furthermore, they raised the issue of missing baselines in the ﬁeld, which renders the evaluation of current and future work difﬁcult.
To ﬁll this gap and generate baselines with known ground truth, we here propose an algorithm to plant and hide arbitrary winning tickets in randomly initialized NNs and construct sparse tickets that reﬂect three common challenges in machine learning.
We use this experimental set-up to compare state-of-the-art pruning algorithms designed to search for lottery tickets.
Our results indicate that state-of-the-art methods achieve sub-optimal sparsity levels, and are not able to recover good tickets before training.
The qualitative trends are consistent with previous results on image classiﬁcation tasks (Tanaka et al., 2020; Frankle et al., 2021) indicating that our experimental set-up exposes pruning algorithms to realistic challenges.
In addition, we improve a state-of-the-art pruning algorithms towards ﬁnding strong lottery tickets of better sparsity.
Our proposed planting framework will enable the evaluation of future progress into this direction.
Contributions 1)
We prove the existence of strong lottery tickets with sparse representations. 2)
Inspired by the proof, we derive a framework that allows us to plant and hide strong tickets in neural networks and thus create benchmark data with known ground truth.
We construct sparse representations of three types of tickets that reﬂect typical machine learning problems.
We systematically evaluate state-of-the-art pruning methods that aim to discover tickets on these three problems against the ground truth tickets and highlight key challenges.
1. R ELATED WORK
Pruning approaches for neural networks can be broadly categorized into three groups, pruning before, during, or after training.
While methods that sparsify the network during (Frankle & Carbin, 2019; Srinivas & Babu, 2016; Lee et al., 2020), or after training (Savarese et al., 2020; LeCun et al., 1990; Hassibi & Stork, 1992; Dong et al., 2017; Li et al., 2017; Molchanov et al., 2017) help in reducing resources required for inference, they are, however, less helpful in reducing resources at training time but can make a difference if they prune early aggressively (You et al., 2020).
The lottery ticket hypothesis (Frankle & Carbin, 2019) has also promoted the development of neural network pruning algorithms that prune before training (Wang et al., 2020; Lee et al., 2019; Verdenius et al., 2020; Tanaka et al., 2020; Ramanujan et al., 2020).
Usually, these methods try to ﬁnd lottery tickets in a ’weak’ (but powerful) sense, that is to identify a sparse neural network architecture that is well trainable starting from its initial parameters.
These methods usually score edges in terms of network ﬂow, which can be quantiﬁed by gradients at different stages of pruning or by edge weights, and prune edges with the lowest scores until the desired sparsity is achieved.
Strong lottery tickets are sparse sub-networks that perform well with the initial parameters, hence do not need further training (Ramanujan et al., 2020).
Their existence has been proven by providing lower bounds on the width of the large, randomly initialized neural network that contains them (Malach et al., 2020; Pensia et al., 2020;
Orseau et al., 2020;
Fischer et al., 2021; Burkholz et al., 2022).
In addition, it was shown that multiple candidate tickets exist that are also robust to parameter quantization (Diffenderfer & Kailkhura, 2021).
1. N OTATION
Let f ( x ) denote a bounded function, without loss of generality f : [ − 1 , 1] n 0 → [ − 1 , 1] n L , that is parameterized as a deep neural network with architecture ¯ n = [ n 0 , n 1 , ..., n L ] , i.e., depth L and widths n l for layers l = 0 , ..., L with ReLU activation function φ ( x ) :
= max( x, 0) .
It maps an input vector x (0) to neurons x ( l ) i as x ( l ) =
φ (cid:0) W ( l ) x
( l − 1)
+ b ( l ) (cid:1) , where W ( l ) ∈ R n l − 1 × n l is the weight matrix, and b ( l ) ∈ R n l is the bias vector of Layer l .
We will establish approximation results with respect to the supremum norm (cid:107) g (cid:107) ∞ :
= sup x ∈ [ − 1 , 1] n 0 (cid:107) g (cid:107) 2 deﬁned for any function g on the domain [ − 1 , 1] n 0 .
Assume furthermore that a ticket f (cid:15) can be obtained by pruning a large mother network f , which we indicate by writing f (cid:15) ⊂ f 0 .
The sparsity level ρ of f (cid:15) is then deﬁned as the fraction of non-zero weights that remain after pruning, i.e., ρ = (cid:16)(cid:80) l (cid:13)(cid:13) (cid:13) W ( l ) (cid:15) (cid:13)(cid:13) (cid:13) 0 (cid:17) / (cid:16)(cid:80) l (cid:13)(cid:13) (cid:13) W ( l )
(cid:13)(cid:13) (cid:13) 0 (cid:17) , where (cid:107)·(cid:107) 0 denotes the l 0 -norm, which counts the number of non-zero elements in a vector or matrix.
Another important quantity that inﬂuences the existence probability of lottery tickets is the in-degree of a node i in layer l of the target f , which we deﬁne as the number of non-zero connections of a neuron to the previous layer plus 1 if the bias is non-zero, i.e., k ( l ) i :=
(cid:13)(cid:13)(cid:13) W ( l ) i, : (cid:13)(cid:13)(cid:13)
+ (cid:13)(cid:13)(cid:13) b ( l ) i (cid:13)(cid:13)(cid:13) 0 , where W ( l ) i, : is the i -th row of W ( l ) .
The maximum degree of all neurons in layer l is denoted as k l, max .
E XISTENCE OF LOTTERY TICKETS
Pruning algorithms that search for strong lottery tickets achieve sparsity levels of at best 0 .
5 when the resulting models should be able to compete with the accuracy of the entire, trained mother network (Ramanujan et al., 2020).
Proofs of the existence of strong lottery tickets give no clear indication whether this is an algorithmic shortcoming, which could be overcome, or a fundamental limitation of pruning randomly initialized networks alone.
The reason is that existing proofs (Malach et al., 2020; Pensia et al., 2020; Orseau et al., 2020; Fischer et al., 2021) guarantee high existence probabilities of subnetworks that have double the depth and 2 − 30 times the width of the target network and thus non-optimal sparsity.
Based on their 2 L construction, Malach et al.
(2020) even went so far to conclude that training by pruning might be computationally at least as hard as training shallower neural networks.
However, it is well known that speciﬁc function classes can be approximated in signiﬁcantly more parameter efﬁcient ways by deeper neural networks rather than shallower ones (Mhaskar et al., 2017; Yarotsky, 2018) and also be learned more efﬁciently (Schmidt-Hieber, 2020).
Thus, by leveraging its full depth, the randomly initialized 2 L deep neural network might contain a much sparser lottery ticket than any of the ones whose existence has been proven.
As a ﬁrst step towards making claims about the existence of very sparse representations, we therefore prove next a lower bound on the probability that a target network of general architecture is contained in a larger, randomly initialized neural network with the same depth as the target network.
As many relevant targets have known representations of lower sparsity than what is covered by this bound, we will afterwards propose a planting algorithm to design experiments that can distinguish between algorithmic and fundamental limitations of pruning for strong lottery tickets.
2. L OWER BOUND ON EXISTENCE PROBABILITY
Pruning a randomly initialized neural network usually recovers a network that has parameters close to, but not exactly like a target network.
First, we need to understand how these errors in the parameters affect the ﬁnal network output and what error ranges are acceptable.
For completeness, we restate Lem. 1 of (Fischer et al., 2021) that guarantees an (cid:15) approximation of the entire network.
Lemma 1 (Error propagation) .
Assume (cid:15) > 0 and let the target network f and its approximation f (cid:15) have the same architecture.
If every parameter θ of f and corresponding θ (cid:15) of f (cid:15) in layer l fulﬁls | θ (cid:15) − θ | ≤ (cid:15) l for (cid:15) l
[Equation]
= (cid:15) (cid:32)
[Equation]
(cid:33)
[Equation]
(cid:16)(cid:13)(cid:13)(cid:13)
[Equation]
+ (cid:15)/L (cid:17)(cid:33) −, then it follows that (cid:107) f − f (cid:15) (cid:107) ∞ ≤ (cid:15) .
Respecting the allowed errors (cid:15) l , we can next establish a lower bound on the existence probability of a speciﬁc target network assuming standard initialization schemes with necessary non-zero bias initialization (Fischer et al., 2021).
The main argument is a union bound over matching each target neuron i (with k i parameters) with neurons of the mother network in the corresponding layer.
Theorem 2 (Lower bound on existence probability) .
Assume that (cid:15) ∈ (0 , 1) and a target network f with depth L and architecture ¯ n are given.
Each parameter of the larger deep neural network f 0 with depth L and architecture ¯ n 0 is initialized independently, uniformly at random with w ( l ) ij ∼ U (cid:16)(cid:104) − σ ( l ) w , σ ( l ) w (cid:105)(cid:17) and b ( l ) i ∼ U (cid:16)(cid:104) − (cid:81) lk =1 σ ( k ) w , (cid:81) lk =1 σ ( k ) w (cid:105)(cid:17) .
Then, f 0 contains a rescaled approximation f (cid:15) of f with probability at least
[Equation]
where (cid:15) l is deﬁned as in Eq.
(1) and the scaling factor is given by λ
= (cid:81)
L l =1 1 /σ ( l ) w .
We could obtain similar results for initially normally distributed weights and biases, we would just have to substitute (cid:15) l by (cid:15) l / 2 .
A proof is provided in Appendix A.2.
Thm. 2 provides us with an intuition for what kind of lottery ticket architectures we can expect to ﬁnd.
First of all, it tells us that a large number of nodes in a layer, and more importantly nodes with [Figure] large in-degree k i , render the existence of a speciﬁc network architecture as winning lottery ticket less likely.
Each additional layer reduces the probability further.
Moreover, we observe that the last layer is a bottleneck, as it usually has the same width as in the large initial network.
To circumvent this problem we could assume that the last layer can be trained, which is common practice.
A higher width of the mother network is clearly advantageous.
Note that we could turn this theorem also into a lower bound on the width n l, 0 of the larger mother network as it is common in existence proofs.
Assuming the same width n l, 0 = n 0 and n l = n across layers, we would receive roughly n [Equation]
Even though it is polynomial in the relevant parameters, it only provides a practical existence proof for extremely sparse architectures.
We therefore have to resort to planting to answer fundamental questions about abilities of pruning algorithms.
In fact, the proof of the above theorem inspires the planting algorithm introduced next.
2. P LANTING LOTTERY TICKETS
As we have discussed, the tickets that exist with high probability rarely fulﬁll criteria of interest, such as low sparsity, favorable generalization properties, or adversarial robustness.
We therefore propose to plant winning tickets with such desirable properties within randomly initialized neural networks.
This approach offers the ﬂexibility to design experiments of different degrees of difﬁculty and generate training and test data based on a baseline ticket.
A simple approach to planting a target f in a network f 0 would be to select a random subset of neurons in each layer and set them to their target values and otherwise randomly initialize the rest.
This, however, would usually lead to a trivially detectable ticket because the target parameters are much larger than the initialized parameters of the larger mother network.
The reason is that both networks produce output that lies in a similar range (ideally the one of the training labels).
Yet, the target network has to achieve this by adding up a much smaller number of parameters.
A different perspective on the same issue is that a pruned lottery ticket needs to be scaled up to compensate for the lost parameters.
Note the scaling factor λ in Theorem 2 for that purpose.
Thus, at least, we would need to scale the target parameters appropriately during planting.
We follow a more general approach that also applies to networks f 0 whose parameters have not been randomly initialized and that captures the full variability of possible target solutions by allowing for different scaling factors per neuron.
We search in each layer of f 0 for suitable neurons that best match a target neuron in f , starting from the ﬁrst layer.
Given that we matched all neurons in layer l − 1 , we try to establish their connections to neurons in the current layer l .
A best match is decided by minimizing the l2-distance to its potential input parameters thereby adjusting for an optimal scaling factor.
For example, let neuron i in Layer l of the target f have non-zero parameters θ =
( b, w ) that point to already matched neurons in Layer l − 1 .
Each neuron j in Layer l of f 0 that we have not matched yet could be a potential match for i .
Let the corresponding parameters of j be m .
The match quality between i and j is assessed by q θ ( m )
=
(cid:107) θ − λ ( m ) m (cid:107) 2 , where λ ( m ) =
θ T m / (cid:107) m (cid:107) 22 is the optimal scaling factor.
The best matching parameters m ∗ =
arg min m q θ ( m ) are replaced by rescaled target parameters θ/λ ( m ∗ ) in f 0 .
We provide pseudocode and details in Supp.
A.3.
[Figure]
2. C ONSTRUCTION OF TARGETS FOR PLANTING
Based on the proposed planting algorithm, we generate sparse tickets for three problems that expose general pruning algorithms to common challenges in machine learning.
On purpose, these are designed to avoid high computational burdens.
Regression of a ReLU unit ( ReLU ).
The ReLU unit φ ( x ) is an essential building block of state-ofthe-art neural networks.
It is particularly interesting to study because we know the optimal solution and can guarantee that it exists with high probability.
Assuming a mother network f 0 of depth L , a ReLU can be implemented with a single neuron per layer.
Any path through the network with positive weights (cid:81)
Ll =1 φ ( w i l − 1 i l x ) deﬁnes a ReLU with scaling factor
λ = (cid:81)
Ll =
1 w i l − 1 i l for indices i l in Layer l with w i l − 1 i l > 0 .
Note that each random path fulﬁlls this criterion with probability 0 .
5 L so that even random pruning has a considerable chance to ﬁnd an optimal ticket.
A winning path exists with probability (cid:81) Ll
=1 (1 − 0 .
5 n l, 0 ) , which is almost 1 even in relatively small networks.
As a ticket with optimal sparsity exists with high probability, we would not need to plant it.
However, to give also pruning algorithms that do not speciﬁcally prune biases a chance, we set ticket biases to zero.
As we will see in experiments, despite this simpliﬁcation, pruning algorithms are severely challenged and cannot ﬁnd an optimally sparse ticket.
Classiﬁcation of rings ( Circle ).
Another basic building block of many functions, in particular, radial symmetric functions, is the radius or l2-norm of a vector.
It is also an important operation to represent products via the relation xy = 0 .
25(( x + y ) 2 − ( x − y ) 2 ) .
We derived a sparse representation that leverages the full depth of a given network, as its sparsity improves with increasing depth.
We visualize the related 4-class classiﬁcation problem with 2-dimensional inputs in Fig.
1 (left). The output is 4-dimensional, where each output unit f c ( x ) corresponds to the probability f c ( x ) that an input ( x 1 , x 2 ) ∈ [ − 1 , 1] 2 belongs to class c that is computed with softmax activation functions.
The decision boundaries are deﬁned in the last layer based on inputs of the form g ( x 1 , x 2 ) =
x 21 + x 22 .
The high symmetry of g ( x 1 , x 2 ) allows us to construct a particularly sparse representation by mirroring data points along axes as visualized in Figure 2a.
Each consecutive layer l mirrors the previous layer along the axis a ( l ) =
(cos( π/ 2 l − 1 ) , sin( π/ 2 l − 1 )) .
To enable higher precision for networks of smaller depth, the second to last layer approximates h ( x ) = x 2 for each component.
This is unnecessary for representations of high enough depth.
The details of our construction are explained in Supp.
A.4.2.
Fig.
2b shows an exemplary architecture of the planted ticket, for which we can vary the depth and width of the second to last layer.[-The lower bound on the existence probability given by Thm. 2 (without planting) for the architecture shown in Fig. 2 (b) is numerically 
-]
Identiﬁcation of a submanifold ( Helix ).
Another common problem in machine learning is to learn lower dimensional functions that are embedded in a higher dimensional space.
Fig.
1 (right) shows our minimal regression example in form of a helix.
As we have observed that many pruning algorithms have the tendency to keep a higher number of neurons closer to the input (and sometimes also the output layer), we construct a ticket that has similar properties, see Fig. 2d.
This should ease the task for pruning algorithms to ﬁnd the planted winning ticket but, as we show in our experiments, this Helix problem is surprisingly challenging.
The helix has three output coordinates f 1 ( x ) =
(5 π + 3 πx ) ∗ cos(5 π + 3 πx ) / (8 π ) , f 2 ( x ) =
(5 π + 3 πx ) ∗ sin(5 π + 3 πx ) / (8 π ) , and f 3 ( x ) =
[Figure]
(5 π +3 πx ) / (8 π ) for 1-dimensional input x ∈ [ − 1 , 1] .
We can approximate each of the components
f i ( x ) by an univariate deep neural network n i ( x ) =
(cid:80) Nj =
1 a ( i ) j φ ( x − s j )+
b ( i ) with depth L = 2 (see Fig.
2c and Supp.
A.4.3), which is achieved by the ﬁrst layers, while the remaining layers model the identity.
An interesting feature of this example is that the largest width could be assigned to almost any layer of the architecture, which makes Helix a good candidate to provoke layer collapse in different pruning algorithms.[-The lower bound on the existence probability given by Thm. 2 (without planting) for the architecture shown in Fig. 2 (d) is numerically 
-]
Strong tickets based on trained neural networks To answer our original question whether stateof-the-art pruning algorithms can ﬁnd extremely sparse strong lottery tickets, we plant a trained weak lottery ticket in a randomly initialized (VGG like) neural network.
Note that the proposed pruning algorithm can also be applied to convolutional layers in addition to fully connected ones.
E XPERIMENTS
We evaluate existing approaches that seek winning tickets on the previously introduced problems with known ground truth.
To show that our experiments reﬂect realistic conditions we also compare the general trends to results on image tasks.
For reproducibility, we provide details on data, networks, and experimental setup in Supp.
B and code online.
For comparison, we consider G RA SP, SNIP, S YNFLOW , M AGNITUDE pruning, and R ANDOM pruning as baseline (Wang et al., 2020; Lee et al., 2019; Tanaka et al., 2020; Frankle & Carbin, 2019), which are all algorithms to discover weak tickets. We do not consider partially trained tickets, such as by rewinding (Frankle et al., 2020).
Additionally, we consider EDGE - POPUP that was designed to ﬁnd strong tickets (Ramanujan et al., 2020). We use existing implementations of weak ticket pruners (Tanaka et al., 2020) and implement EDGE - POPUP ourselves, extending its score values to bias parameters.
Moreover, we test different pruning strategies. 3. P RUNING WITHOUT INTERMEDIATE TRAINING
First, we consider pruning strategies without intermediate training, where tickets are discovered by pruning a network using solemnly scores of initial network parameters.
Singleshot pruning In singleshot pruning, which is originally applied in SNIP and G RA SP, edges are scored in a single pass and then pruned to the desired sparsity.
Compared to multishot pruning, this saves signiﬁcant amounts of resources by preventing training entirely, if a strong ticket is found, or only training a small subnetwork once, in case a weak ticket is found.
For our benchmark data, we construct networks of depth 5 and width 100 and test the ability of algorithms to discover both strong and weak tickets.
The key results are visualized in Fig.
3, reporting performance of the algorithms trying to discover tickets at varying sparsity levels across 25 repetitions. Results for varying network depths and widths can be found in Supp.
B noting that they are consistent also with larger depth, but the methods fail to ﬁnd any ticket in more shallow networks of depth 3.
We ﬁnd that all approaches, including R ANDOM pruning, are able to ﬁnd weak tickets for moderate sparsity levels for Circle and ReLU , but fail to recover them on the manifold learning task Helix entirely.
Although M AGNITUDE pruning was originally not designed for this pruning strategy, it is on par with state-of-the-art singleshot methods.
For lower sparsity levels ≤ 0 .
01 , in particular baseline ticket sparsity, all methods fail to recover good subnetworks.
Dissecting the results, we observe layer collapse, meaning that entire layers are masked, thus disrupting ﬂow through the network.
Despite that S YNFLOW was proposed as a solution to this issue, we observe that it also experiences layer collapse for extreme sparsities, even when pruning for 100 rounds as suggested in the original paper, performing only slightly better than with one round of pruning (see Supp. B). In summary, with only a single pruning round, most pruning algorithms discover weak tickets at moderate sparsity, however fail to recover weak tickets of low sparsity and any strong tickets.
Robustness to noise To model real-world settings, all our datasets contain small amounts of noise as described in Supp.
B. To rule out that noise in the data is the primary source for issues with discovering tickets, we generated Circle datasets with varying levels of noise. The results indicate that on the one hand, without noise we do not see much of an improvement in terms of discovered tickets, but on the other hand observe that the algorithms are robust to even large amounts of noise, ﬁnding tickets with almost similar performance as with no noise at all (see Supp.
B). Comparison to results on image data The reported results are in line with experiments on image data as reported in the literature (Tanaka et al., 2020).
In particular, for all these methods a similar drop around 0 .
01 sparsity is observed for different VGG and Resnet architectures and image data sets.
Similarly, layer collapse has been reported for image data.
The main difference is that for our data, we know the obtainable sparsity as well as performance of tickets, setting these results into a context beyond trendline differences of methods for selected sparsity values.
While much more resource intensive, iteratively training followed by pruning and resetting to initial weights, slowly annealing to the desired sparsity, has been proven a successful approach to discover lottery tickets. Here we extend this pruning scheme to other approaches beyond magnitude pruning. Multishot pruning To investigate the effect of multishot pruning, we run each of the previous methods iteratively for 10 rounds on our benchmark datasets (see Fig. 4).
Analogous to iterative magnitude pruning, for each round r , we iteratively reduce the sparsity to ρ r/ 10 , where ρ is the desired network sparsity.
Within each round, the current subnetwork is ﬁrst shortly trained, then pruned to the current target sparsity, and then reset to initial parameters for the next round.
Compared to the singleshot results, we observe that for the classiﬁcation task, M AGNITUDE , SNIP, and
S YNFLOW are able to retrieve weak tickets of much higher sparsity.
Furthermore, these three approaches are now able to recover weak tickets of moderate sparsities also for the challenging Helix dataset.
Overall, S YNFLOW consistently performs best in discovering weak tickets, even recovering the extremely sparse baseline ticket for Circle .
We also observe that G RA SP performs poor overall, noting that it is a method designed for singleshot pruning.
Examining the results, we see that G RA SP experiences layer collapse already in early iterations with large target sparsities. None of the above approaches is able to discover a strong ticket. To discover strong tickets, Ramanujan et al.
(2020) proposed EDGE - POPUP , which falls in the same category of multishot pruning approaches.
EDGE - POPUP assigns each model parameter a score value, which is then actively trained for several rounds while freezing all original parameters, requiring a similar computational effort as multishot pruning.
Training EDGE - POPUP for 10 rounds, we observe that it discovers a strong ticket of sparsity 0 .
5 for Circle however fails to discover
[Figure]
tickets of different sparsity, which is in line with their original results Ramanujan et al.
Similar to other algorithms, we observe layer collapse.
We can extend their original algorithm using annealing as in multishot pruning by slowly decreasing the sparsity in every round, which increases performance allowing it to discover a subnetwork with reasonable accuracy at 0 .
1 sparsity.
Interestingly, EDGE - POPUP is not able to ﬁnd any good subnetwork for the regression tasks.
G RA SP with local sparsity constraints As observed before, G RA SP seems unsuitable for multishot pruning due to early layer collapse. Several works (You et al., 2020; Tanaka et al., 2020) considered local sparsity constraints, having a target sparsity for each layer, or even channel. These however impose unrealistic architecture constraints as layer sparsity is usually imbalanced (Tanaka et al., 2020), which also holds true for our Circle and Helix benchmarks. With the goal to avoid layer collapse, we still equipped G RA SP with local sparsity constraints per layer (see Supp. B).
Yet, the ﬂow through the layers stays interrupted.
A possible explanation is that G RA SP incorporates information about weight couplings via the hessian in its pruning strategy, which makes it more sensitive to removing individual connections from the mask as it happens during iterative pruning.
Comparison to results on image data Our results are coherent with the reported relative trends on image tasks both for strong and weak tickets (Ramanujan et al., 2020; Tanaka et al., 2020).
We further reproduced results for VGG16 on CIFAR10 with non-zero bias initialization (Fig. 5 left).
In particular, for strong tickets we observe the same trends for EDGE - POPUP spiking at 0.5 sparsity, performing less well at sparsity 0.1, and not recovering tickets for higher and lower sparsity.
Again consistent with previously reported results, other methods are neither suited nor designed to ﬁnd strong tickets.
For weak tickets, S YNFLOW and M AGNITUDE are on part, which is different than originally reported in Tanaka et al.
(2020), as we allow both SNIP and M AGNITUDE to learn in a multishot fashion.
This approach closely resembles the one of iterative magnitude pruning, where we do not observe layer collapse for these methods.
VGG with strong tickets Finally, to put the hypothesis to a test that EDGE - POPUP is limited to discover strong tickets of suboptimal sparsity of around 0 .
5 , we investigate its capabilities to recover [Figure] a planted baseline ticket from VGG16.
For that, we use S YNFLOW to discover a weak ticket of sparsity 0 .
01 from VGG16 with multishot pruning, trained the weak ticket, and planted it back into the network.
Running EDGE - POPUP on this network, we observe that it indeed cannot retrieve the baseline ticket of desired sparsity in this real world setting (see Fig.
5 right).
D ISCUSSION & C ONCLUSION
We investigated the optimality of existing lottery ticket pruning methods and their potential for improvement, both regarding the discovery of strong tickets – subnetworks that perform well at initialization – as well as weak tickets – subnetworks that require further training.
Recent works, in particular by Frankle et al.
(2021), evaluated pruning methods and showed that no single best method across considered settings and sparsities exists, also raising the issue of missing baselines.
To tackle this, we here proposed an algorithm that plants and hides target networks within a larger network, thus allowing to construct benchmarks for lottery tickets.
To construct parameter efﬁcient tickets, we leveraged recent results that transferred the strong lottery ticket hypothesis to neural networks with arbitrary initial biases and proved the existence of lottery tickets under realistic conditions with respect to the network width and initialization scheme.
For three challenging tasks, we hand-crafted extremely sparse network topologies, planted them in large neural networks, and evaluated the stateof-the-art pruning methods in combination with different pruning strategies.
Intriguingly, we found that none of the approaches is able to discover strong tickets, in particular not the planted ticket, in a single shot, and only at sparsity levels ≥ 0 . 1 the methods are able to ﬁnd weak tickets. One of the problems that arises is layer collapse. While layer-wise pruning could prevent this collapse, it still results in an interruption of ﬂow.
With a multishot approach, iteratively training, pruning, and resetting weights, certain methods were able to recover weak tickets for a classiﬁcation task, even at sparsity similar to the planted ticket. However, only EDGE - POPUP was able to recover strong tickets in this setting, but only at sparsity orders of magnitude larger than the planted baseline. Through annealing of the target sparsity, we were able to improve EDGE - POPUP , discovering an order of magnitude sparser strong ticket than before, yet still not as sparse as the baseline. Furthermore, we observed that all considered methods struggled more with regression than with classiﬁcation tasks, where they recovered only weak tickets of at best 10% sparsity.
These result on our benchmark data are coherent with reported as well as reproduced classiﬁcation results on image data.
This shows that our benchmarks, while artiﬁcial in nature, reﬂect realistic conditions that result in similar trends as real world image data sets would.
Moreover, we have shown that our planting framework can also be used in a real data setting to answer open questions.
For instance, by planting a trained weak ticket back into a CNN, we established that the failure of EDGE - POPUP to discover extremely sparse strong lottery tickets is likely an algorithmic rather than a fundamental limitation.
This shows how our framework enables experiments beyond relative method comparisons that are typically conducted on standard image benchmark data.
As our results indicate, several major questions pertaining to neural network pruning are still open: How can pruning approaches for weak tickets be improved to discover tickets of best possible sparsity?
How can we ﬁnd weak tickets of high sparsity that match the performance of the large network without intermediate training rounds?
And how can we discover highly sparse strong lottery tickets?
We anticipate that our contribution can be used and extended to measure progress regarding these questions against independent
sparse and well-performing baseline tickets
A T HEORY
In the following section, we present the proofs of the theorems and lemmas of the main manuscript.
A. E RROR PROPAGATION : P ROOF OF L EMMA
Statement.
Assume (cid:15) > 0 and let the target network f and its approximation f (cid:15) have the same architecture.
If every parameter θ of f and corresponding θ (cid:15) of f (cid:15) in layer l fulﬁls | θ (cid:15) − θ | ≤ (cid:15) l for (cid:15) l := (cid:15) (cid:32) L √ m l (cid:32) 1 + sup x ∈ [ − 1 , 1] n(cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
) (cid:13)(cid:13)(cid:13)(cid:33) L (cid:89) k = l + (cid:16)(cid:13)(cid:13)(cid:13) W ( l ) (cid:13)(cid:13)(cid:13) ∞ + (cid:15)/L (cid:17)(cid:33) −, then it follows that (cid:107) f − f (cid:15) (cid:107) ∞ ≤ (cid:15) .
Proof.
Our objective is to bound (cid:107) f − f (cid:15) (cid:107) ∞ ≤ (cid:15) . We frequently use the triangle inequality and that | φ ( x [Table]
( y ) | ≤ | x − y | is Lipschitz continuous with Lipschitz constant 1 to derive [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] with (cid:15) l ≤ (cid:15)/L . m l denotes the number of parameters in layer l that are smaller than (cid:15) l and (cid:107) W (cid:107) ∞ =
max i,j | w i,j | .
Note that m l ≤ n l k l, max .
The last inequality follows from the fact that all entries of the matrix (cid:16) W ( l ) − W ( l ) (cid:15) (cid:17) and of the vector ( b ( l ) − b ( l ) (cid:15) ) are bounded by (cid:15) l and maximally m l of these entries are non-zero.
Furthermore, (cid:13)(cid:13)(cid:13) W ( l ) (cid:15) (cid:13)(cid:13)(cid:13) ∞ ≤ (cid:0)(cid:13)(cid:13) W ( l ) (cid:13)(cid:13) ∞ + (cid:15) l (cid:1) follows again from the fact that each entry of (cid:16) W ( l ) − W ( l ) (cid:15) (cid:17) is bounded by (cid:15) l .
Thus, at the last layer it holds for all x ∈ [ − 1 , 1] n 0 that (cid:107) f
{+( x ) − f (cid:15) ( x ) (cid:107) 
 = (cid:13)(cid:13)(cid:13) x ( L ) − x ( L ) (cid:15) (cid:13)(cid:13)(cid:13)≤ L (cid:88) l = (cid:15) l √ m l (cid:32) 1 + sup x ∈ [ − 1 , 1] n(cid:13)(cid:13)(cid:13) x ( l − 1) (cid:13)(cid:13)(cid:13)(cid:33)+}
L (cid:89) k = l + (cid:16)(cid:13)(cid:13)(cid:13) [Equation] using the deﬁnition of (cid:15) l in the last step.
A. E XISTENCE OF SPARSE LOTTERY TICKETS : P ROOF OF T HEOREM
Next, we prove the following lower bound on the probability that a very sparse lottery ticket exists.
Statement.
Assume that (cid:15) ∈ (0 , 1) and a target network f with depth L and architecture ¯ n are given.
Each parameter of the larger deep neural network f 0 with depth L and architecture ¯ n 0 is initialized independently, uniformly at random with w ( l ) ij ∼ U (cid:16)(cid:104) − σ ( l ) w , σ ( l ) w (cid:105)(cid:17) and b ( l ) i ∼ U (cid:16)(cid:104) − (cid:81) lk =1 σ ( k ) w , (cid:81) lk =1 σ ( k ) w (cid:105)(cid:17) .
Then, f 0 contains a rescaled approximation f (cid:15) of f with probability at least
[Equation]
where (cid:15) l is deﬁned as in Eq.
(1) and the scaling factor is given by λ
= (cid:81)
Ll =1 1 /σ ( l ) w .
Proof.
As shown by Fischer et al. (2021), the scaling of the output by λ simpliﬁes the above parameter initialization to an equivalent setting, in which each parameter is distributed as w ij , b i ∼
U [ − 1 , 1] , while the overall output is scaled by the stated scaling factor λ , again assuming that all parameters are bounded by 1 − (cid:15) .
Each parameter in Layer l needs to be approximated up to error (cid:15) l according to Lemma 1.
To match the same sparsity level of f , for each neuron i in each Layer l of f , we have to ﬁnd exactly one neuron in the same layer (Layer l ) of f 0 that represents i .
We start with matching neurons at Layer 1 (given the input in Layer 0 ) and proceed iteratively by matching neurons in Layer l given the already matched neurons in Layer l − 1 .
Let us pick a random neuron in Layer l of f 0 .
How high is the probability that it is a match with a given target neuron i in layer l of f ?
The neuron i consists of k
i parameters that have to be matched.
Since the corresponding neurons in layer l − 1 of f and f 0 have already been matched according to our assumption, we only have one possible candidate θ 0 for each of the k i parameters θ i .
For uniformly distributed parameters, we have | θ i − θ [-
-]0 | ≤ (cid:15) l with probability (cid:15) l .
For normally distributed θ 0 ∼ N (0 , 1) , the probability is at least (cid:15) l / 2 (as long as | θ 0 ± (cid:15) | ≤ 1 .
This can be seen by Taylor approximation of the cdf of a standard normal Φ( z + ∆ z ) − Φ( z − ∆ z ) in z .
For the remainder of the proof, however, we assume uniformly distributed parameters.
Thus, all k i independent parameters are a match with probability (cid:15) k
i l .
Accordingly, none of the available n l,neurons in Layer l is a match with probability (cid:16) 1 − (cid:15)
k
i l (cid:17) n l, 0 .
With the help of a union bound we can deduce that the probability that at least one of the neurons i in Layer l of f has no match in f 0 is smaller or equal to (cid:80) n l i =1 (cid:16) 1 − (cid:15)
k i l (cid:17) n l, 0 .
Therefore, the converse probability that we ﬁnd a match for every single neuron in Layer l of f is at least 1 −
[Equation]
Input: target f , larger neural network f
Output
Initialize λ
= [1] 0 ; // Scaling factors for input are 1.
for l = 1 to L − 1 do for all neurons i of f in Layer l do θ :=
( b, wλ old ) // non-zero parameters of neuron i in f .
m ∗ =
arg min m q θ ( m ) //
Find best match for i in f 0 .
Replace parameters in f 0 by θ/λ ( m ∗ ) ; Deﬁne λ n = λ ( m ∗ ) .
// Remember scaling factor end
Replace λ old = λ ; end
Since we have to guarantee a match for each single layer and the matching probability of a new layer is conditional on the previous layer, we obtain a lower bound on the existence probability of a lottery ticket by multiplying the layerwise bounds. This bound is only practical for very sparse target networks f with neurons of small in-degrees k i . It still shows that the existence of very sparse lottery tickets is possible under the right conditions. This extreme sparsity, however can pose signiﬁcant challenges to pruning algorithms, as we also see in our experiments with planted solutions.
Inspired by this proof, we therefore explain next how to plant tickets (which could have variable sparsity levels). A. P LANTING ALGORITHM The idea of layerwise matching of neurons starting with the layer closest to the input can be transferred to planting. The approach applies to general initial networks f 0 whose parameters have not necessarily been randomly initialized and captures the full variability of possible target solutions by allowing for different scaling factors per neuron.
We search in each layer of f 0 for suitable neurons that best match a target neuron in f , starting from the ﬁrst hidden layer.
Given that we matched all neurons in layer l − 1 , we try to establish their connections to neurons in the current layer l .
A best match is decided by minimizing the l2-distance to its potential input parameters thereby adjusting for an optimal scaling factor.
For example, let neuron i in Layer l of the target f have non-zero parameters ( b, w ) that point to already matched neurons in Layer l − 1 .
With each of the matched neurons j (cid:48) in Layer l − 1 has been previously associated a scaling factor λ old ,j (cid:48) so that the corrected θ = ( b, wλ old ) parameters would compute the correct neuron in f 0 .
In Layer l of f 0 , each neuron j that we have not matched yet could be a potential match for i .
Let the corresponding parameters of j be m .
The match quality between i and j is assessed by
[Equation]
where λ ( m ) =
θ T m / (cid:107) m (cid:107) 22 is the optimal scaling factor.
The best matching parameters m ∗ =
arg min m q θ ( m ) are replaced by rescaled target parameters θ/λ ( m ∗ ) in f
and we remember the scaling factor λ ( m ∗ ) to consider matches of neurons in Layer l + 1 .
Note that this rescaling is necessary to ensure that the neuron is properly hidden and attains similar values as other non-planted neurons in f 0 .
In addition to the provided pseudocode (Algorithm 1), a Python implementation is submitted alongside the supplementary material.
A. C ONSTRUCTION OF TARGETS FOR PLANTING
We propose three examples for planting lottery tickets that pose different challenges for pruning algorithms.
Here, we explain the main ideas behind their construction in more detail.
A Python implementation of related regression and classiﬁcation problems is provided alongside the supplementary manuscript.
A.4. R E LU UNIT
Apart from a trivial function f ( x ) = 0 , a univariate ReLU unit f ( x ) = φ ( x ) = max( x, 0) is the most sparse lottery ticket that is possible.
Assuming a mother network f 0 of depth L , a ReLU can be implemented with a single neuron per layer.
Any path through the network with positive weights (cid:81)
Ll =1 φ ( w i l − 1 i l x ) deﬁnes a ReLU with scaling factor
λ = (cid:81)
Ll =
1 w i l − 1 i l for indices i l in Layer l with w i l − 1 i l > 0 .
Note that each random path fulﬁlls this criterion with probability 0 .
5 L so that even random pruning could have a considerable chance to ﬁnd an optimal ticket.
A winning path exists with probability (cid:81)
Ll =1 (1 − 0 .
5 n l, 0 ) , which is almost 1 even in small mother networks.
Thus, planting is not really necessary in this case.
Since not all pruning algorithms set biases to zero, however, we still set all randomly initialized biases along a winning path to zero to make the problem easier.
As we see in experiments, despite this simpliﬁcation, pruning algorithms are severely challenged in ﬁnding an optimally sparse ticket.
Even though basic, a ReLU unit seems to be a suitable benchmark that is a common building block of other tickets.
A.4. C IRCLE
For simplicity, we restrict ourselves to a 4-class classiﬁcation problem with 2-dimensional input.
The output is therefore 4-dimensional, where each output unit f c ( x ) corresponds to the probability f c ( x ) that an input ( x 1 , x 2 ) ∈ [ − 1 , 1] 2 belongs to the corresponding class c with c = 0 , 1 , 2 , 3 .
As common, this probability is computed assuming softmax activation functions in the last layer.
The decision boundaries are deﬁned in the last layer based on inputs of the form g ( x 1 , x 2 ) .
The role of the ﬁrst layers with ReLU activation functions of a Circle target f is to compute the function g ( x 1 , x 2 ) =
x 21 + x 22 , which is fundamental to many problems, in particular to the computation of radial symmetric functions.
The high symmetry of g ( x 1 , x 2 ) allows us to construct a particularly sparse representation by mirroring data points along axes as visualized in Figure 2 (a).
With the ﬁrst two layers ( l = 1 , 2 ), we map each input vector ( x 1 , x 2 ) to the ﬁrst quadrant by deﬁning x (1) =
φ ( x 1 ) + φ ( − x 1 ) and x (1) =
φ ( x 2 ) + φ ( − x 2 ) .
Thus, Layer l = 1 consists of 4 neurons, i.e., x (1) =
φ ( x 1 ) , x (1) =
φ ( − x 1 ) , x (1) =
φ ( x 2 ) , x (1) =
φ ( − x 2 ) , while Layer l = 2 consists of 2 neurons, i.e., x (2)1 = φ (cid:16) x (1)1 + x (1)2 (cid:17) , x (2) [Equation] = φ (cid:16) x (1)3 + x (1)4 (cid:17) .
Each consecutive layer l mirrors the previous layer ( x ( l − 1), x ( l − 1)) along the axis a ( l )
= (cos( π/ 2 l − 1 ) , sin( π/ 2 l − 1 )) .
It achieves this by mapping the neurons of the previous layer to three neurons, one representing the component of x ( l − 1) that is parallel to a ( l ) , and two neurons that each represent the positive or negative signal component that is perpendicular to the axis a ( l ) .
The last two neurons could be added to a single neuron in the next layer if we want to decrease the width of some layers to 2 in between.
To take more advantage of the allowed depth L , we map three neurons immediately to the next three neurons that represent the mirroring by deﬁning x ( [Equation] 1 = φ (cid:16) a ( l ) 1 x ( l − 1)+ [Equation]
− h
[Equation]
If the depth of our network is high enough, we could use the parallel component x ( l ) 1 as estimate of the radius of the input.
To enable higher precision for networks of smaller depth, however, we also apply to each remaining component a piecewise linear approximation of the univariate function h ( x ) =
x 2 and add those two components.
Note that any univariate function can be easily approximated by a neural network of depth L = 2 .
The precise approach is explained in our next example.
A.4. H ELIX
To test the ability of pruning algorithms to detect lower dimensional submanifolds, we approximate a helix with three output coordinates f 1 ( x ) = (5 π + 3 πx ) ∗ cos(5 π + 3 πx ) / (8 π ) , f 2 ( x ) = (5 π + 3 πx ) ∗ sin(5 π + 3 πx ) / (8 π ) , and f 3 ( x ) =
(5 π + 3 πx ) / (8 π ) for 1-dimensional input x ∈ [ − 1 , 1] .
As we have observed that many pruning algorithms have the tendency to keep a higher number of neurons closer to the input (and sometimes also the output layer), we construct a ticket that has similar properties.
This should ease the task for pruning algorithms to ﬁnd the planted winning ticket.
Each of the components f i ( x ) is an univariate function that we can approximate by an univariate deep neural network n i ( x ) that encodes a piece-wise linear function (see Figure 2 (c) for an explanation).
As neural networks are generally overparameterized, we have multiple options to represent n i ( x ) .
For simplicity, we write it as composition of the identity with a depth L = 2 univariate network g i ( x ) of width N in the hidden layer, which can be written as
[Equation]
where the signs p j ∈ {− 1 , 1 } can be chosen arbitrarily (and we chose alternating signs to create diversity).
The knots s = ( s j ) j ∈
[ N ] mark the boundaries of the linear regions and a = ( a ( i ) j )
j ∈
[ N ] indicate changes in slopes m ( i )
j = ( f i ( s j +1 ) − f ( s j )) /
( s j +1 − s j ) (with s N +1 :=
s N + (cid:15) ) from one linear region to the next.
a ( i ) j =
[Equation]
We thus create a helix ticket by ﬁrst mapping the input x ∈ [ − 1 , 1] to [0 , 2] .
This allows us to represent the identity in the later layers by φ ( x ) =
x , as x ≥ 0 .
We can always compensate for the bias +1 by subtracting a bias − 1 when needed.
f 3 ( x ) = (5 π + 3 πx ) / (8 π ) can therefore be represented by a path from the input to the output that only contains a single neuron per layer.
We concatenate this path with a neural network that consists of layers that approximate f 1 ( x ) and f 2 ( x ) and otherwise identity functions.
At Layer l = 2 , this network creates neurons of the form φ ( p j ( x − s j )) , where the knots s j mark an equidistant grid of [0 , 2] .
Layer l = 3 creates two neurons, one corresponding to x (2)1 =
f 1 ( x ) and one corresponding to x (2)2 = f 2 ( x ) .
These can be computed by linear combination of the previous neurons using the parameters a ( i ) j and b ( i ) .
All the remaining layers basically encode the identity.
B E XPERIMENTS
In this section, we discuss all relevant parameters and set-ups to reproduce the experimental results.
Furthermore, we provide additional results obtained for singleshot learning spanning different architectures and pruning schemes.
All source code to run pruning algorithms and to generate the data is made publicly available.
B. H YPERPARAMETERS AND DATA
For each experiment, we generate n = 10000 samples, where input data is sampled from [ − 1 , 1] , for ReLU and Helix and from and [ − 1 , 1] 3 for Circle .
The output for ReLU is computed by f ( x ) = max (0 , x ) .
For Helix , we compute the three output coordinates as f 1 ( x ) =
(5 π + 3 πx ) ∗ cos(5 π +3 πx ) / (8 π ) , f 2 ( x ) =
(5 π +3 πx ) ∗ sin(5 π +3 πx ) / (8 π ) , and f 3 ( x ) =
(5 π +3 πx ) / (8 π ) .
For
Circle , we consider circles centered at the origin with radius √ 0 . 2 , √ 0 .
5 , and √ 0 .
7 as decision boundaries for the classes.
We additionally introduce a small amount of noise to simulate real world data more closely.
For Circle we ﬂip approximately 1% of samples to the next closest class, and for the two regression problems we introduce additive noise drawn from N (0 , 0 . 01) to each output dimension.
To assess the accuracy respectively mean squared error of the tickets and trained models, we split off 10% of the data that acts as a hold out test set.
In general, all initial networks for each speciﬁc task are generated using the algorithm explained in Supp.
A. For Circle , we use 10 knots for the piecewise linear approximation, and 30 knots for the piecewise lienar approximations done in Helix .
To prune by
G RA SP,SNIP,S YNFLOW ,M AGNITUDE , and R ANDOM and train the derived tickets, we use Adam
Kingma & Ba (2015) with a learning rate of 0 . 001 .
We found that this learning rate performed well over all experiments, and leading to accurate models when there is no pruning.
It also corresponds to the default settings suggested by the authors of S YNFLOW Tanaka et al.
Training of the discovered tickets was done for 10 epochs across all experiments, where we could always observe a convergence of the respective score on the validation sets (accuracy or MSE).
We measured loss by MSE respectively cross entropy loss and used a batch size of 32 for all experiments.
We report mean and conﬁdence intervals across 10 repetitions for multishot, and across 25 repetions for the main singleshot experiments presented in Section 3.1 measured on the hold out test set.
Singleshot pruning For singleshot pruning, we considered networks of depth 3 , 5 , 10 each with layer width 100 for all three data sets on target sparsities { 0 . 01 , 0 . 1 , 0 .
5 , 1 } and the sparsity of the ground truth ticket.
Additionally, we tested for a network of depth 6 and width 1000 on Circle for the same sparsity levels.
As suggested by Tanaka et al.
(2020), we also test S YNFLOW in combination with 100 rounds of pruning for a network of depth 6 and width 100 on Circle .
For all additional singleshot experiments, we provide results in the next section.
Multishot pruning For multishot pruning, we alternated pruning and training for 10 rounds, where each training step was carried out for 5 epochs, which consistently lead to convergence of accuracy on the considered Circle data set.
Similar to singleshot pruning, we considered target sparsities { 0 . 01 , and
EDGE - POPUP pruning To prune with EDGE - POPUP , we here used the parameters suggested in the original code of Ramanujan et al.
(2020), which is SGD with momentum of 0 .
9 and weight decay 0 . 0005 , combined with cosine annealing of the learning rate.
To establish a comparison to the multishot pruning results, we train the scores for 10 epochs.
Additionally, for the experiment extending EDGE - POPUP by annealing the sparsity level, we slowly reduce the sparsity over time by ρ i/ 10 , where ρ is the desired network sparsity, and i is the current epoch.
B. A DDITIONAL RESULTS ON SINGLESHOT LEARNING
Model depth In this section, we provide all results from the singleshot learning for depths 3 , 5 ,and width 100 in Figure 6,7, and 8, respectively.
We observe that all methods have problems to deal with smaller networks, while the results for the larger networks are consistent.
Model width To investigate the effect of layer size, we run an experiment with a network of depth 6 and width 1000 on Circle – as the layer size is an important factor for the theoretical probability of the existence of tickets – and report the results in Figure 9.
We observe that although the network is much larger, there is barely any change in comparison to the previous results.
Note that the results after training of tickets of individual sparsities cannot be directly compared directly to the other singleshot results, as the tickets are much larger (due to the much larger number of parameters) and hence easier to train.
Multiple pruning rounds
We report the results of running S YNFLOW with 100 rounds of pruning on a network of depth 6 and width 100 for Circle in Figure 10.
We ﬁnd that there is again barely any change to the original singleshot results after pruning, but there is a slight increase in performance after training compared to the single-round singleshot results.
Noise experiments To test the robustness of pruning algorithms to noise in the data, we considered the Circle problem with a network of depth 6 and width 100 and varied the amount of noise in the data to be { 0 , 0 . 001 , 0 . 01 , 0 . 1 } .
We report the results before and after training in Fig. [Figure]
B. A DDITIONAL RESULTS ON MULTISHOT LEARNING
To test whether we can reach an improvement of the performance of tickets discovered by G RA SP using a multishot pruning approach and avoid layer collapse [Figure] we ran additional experiments using a local pruning rate.
In particular [Figure] we used layer-wise pruning setting the target sparsity of the parameters of each individual layer to the global sparsity level.
We report results in Fig.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]