Most Activation Functions Can Win the Lottery Without Excessive Depth
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The strong lottery ticket hypothesis has highlighted the potential for training deepneural networks by pruning, which has inspired interesting practical and theoreticalinsights into how neural networks can represent functions.
For networks with ReLUactivation functions, it has been proven that a target network with depth L can beapproximated by the subnetwork of a randomly initialized neural network that hasdouble the target’s depth 2 L and is wider by a logarithmic factor.
We show that adepth L + 1 network is sufﬁcient.
This result indicates that we can expect to ﬁndlottery tickets at realistic, commonly used depths while only requiring logarithmicoverparametrization.
Our novel construction approach applies to a large class ofactivation functions and is not limited to ReLUs.
Introduction
The Lottery Ticket Hypothesis [13] and, in particular, its strong version [36] postulate that pruningdeep neural networks might provide a promising alternative to training large, overparameterizedneural networks with Stochastic Gradient Descent (SGD).
Pruning has the potential to not onlyidentify small scale neural networks that possess a meaningful, task-speciﬁc neural network structureand generalize better due to regularization [49], but also to reduce the computational burden associatedwith deep learning [48, 10, 27].
Also from a theoretical perspective, it has been shown that for every small enough target network asufﬁciently large, randomly initialized neural network, the source network, contains a subnetwork,the lottery ticket (LT), that can approximate the target up to acceptable accuracy with high probability.[30] has been the ﬁrst to provide a probabilistic lower bound on the required network width of thelarger random network.
This bound has been improved by [35, 34] to a width that is larger thana target’s width only by a logarithmic factor.
The limitations of these works are that they all arerestricted to ReLU activation functions and assume that the larger random network has twice thedepth of the target network L s =
2 L t .
However, it is well known that deeper networks tend to have a higher expressiveness, as theycan approximate certain function classes with signiﬁcantly fewer parameters than their shallowcounterparts [31, 47].
It could therefore reduce the overall sparsity of the target and the LT to allowthe target to utilize more of the source depth L s for a sparser representation.
A lower depth requirmentcould also [11] have therefore derived lower bounds for the width of the random network of depth
L s =
L t + 1 , but these can only cover extremely sparse target networks, as the width requirement ispolynomial in the inverse approximation error 1 / ✏ .
In contrast, we derive a novel construction that only requires a logarithmic factor.
While we utilizesubset sum approximation results like [35], we allow a target network to have almost the same depth as the source network with depth L s  L t
+ 1 instead of L s = 2 L t .
Our derivations further apply toa large class of activation functions that includes but is not limited to ReLUs.
The reduced depth requirement and ﬂexibility in the activation function of our construction can resultin signiﬁcantly sparser neural network target networks and thus also LTs.
For example, f ( x ) = x 2can be approximated up to error ✏ by a shallow ReLU network with O (1 / ✏ ) parameters, while a deepenough network only needs O (log(1 / ✏ ))
parameters [31].
Also the activation function determinesthe possible target sparsity, e.g., x 2 can be represented with only 4 parameters for all ✏ > 0 if ( x ) =
(max { x, 0 } ) 2 .
Therefore, the potential sparsity of the ﬁnal lottery ticket depends criticallyon the choice of activation functions and architecture of the source network, including its depth andwidth.
We provide a theoretical foundation that provides ﬂexibility regarding these choices.
Contributions 1)
We prove the existence of strong lottery tickets as subnetworks of a largerrandomly initialized source neural network for a large class of activation functions, including ReLUs.2)
We derive a novel construction that requires the source network to have almost the same depth( L + 1 ) as the target network, thus, allowing it to leverage the potential representational beneﬁtsassociated with larger depth.
Despite the reduced depth requirement we keep the width requirementlogarithmic in the approximation error ✏ . 4)
Our proofs are constructive and deﬁne an algorithm thatapproximates a given target network by pruning a source network.
We verify in experiments that thisalgorithm is successful under realistic conditions.
Related Literature Many pruning methods have been proposed to reduce the number of neuralnetwork parameters during training [20, 33, 17, 13, 41, 22, 48, 14, 37, 26, 25, 46, 38, 6] or thereafter[39, 21, 18, 9, 24, 32, 50] and have been applied in different contexts, including graph neuralnetworks [5] and GANs [4].
Furthermore, pruning can have provable regularization and generalizationproperties [49].
The algorithms are most useful for structure learning at lower sparsity levels [42, 22]but at least Iterative Magnitude Pruning (IMP) [17, 13] can fail in identifying LTs that performsuperior to random or smaller dense networks [29].
It can therefore be beneﬁcial to start pruningfrom a sparse random architecture rather than a dense network [10, 27], which saves computationalresources.
Other options to achieve the latter are to identify and train on core sets [51] and focuson pruning before training [45, 23, 44, 43, 36].
Yet, iterative pruning methods often perform better[15, 29] but all face challenges in ﬁnding highly sparse LTs [11].
Most of the discussed pruning methods try to ﬁnd LTs in a ‘weak’ (but powerful) sense by identifyinga sparse neural network architecture that is well trainable starting from its initial parameters.
Strong
LTs are sparse subnetworks that perform well with the initial parameters and, hence, do not needfurther training [52, 36].
Their existence has been proven for fully-connected feed forward networkswith R E LU activation functions by providing lower bounds on the width of the large, randomlyinitialized source network [30, 35, 34, 12, 3].
In addition, it was shown that multiple candidate ticketsexist that are also robust to parameter quantization [8].
Our ﬁrst objective is to extend the knowntheory to other activation functions beyond R E LU S .
While this approach has been transferred toconvolutional and residual architectures [1], we focus here on fully-connected networks to developthe main construction ideas.
Constructing Lottery Tickets
Informally, our goal is to show that any deep neural network of width n t and depth L t or smallercan be approximated with probability 1   up to error ✏ by pruning a larger randomly initializedneural network of width O ( n t log[ n t L t / min {  , ✏ } ]) and depth 2 L t or a network of smaller depth
L t + 1 and width O ( n t log[ n t L t log(1 /  ) / min {  , ✏ } ]) , as long as the target network and the largerandom network rely on the same activation functions.
Note that we often omit the dependence on in our discussions because we usually care about cases when ✏ <  .
Next we explain the requiredbackground to formalize and prove our claims.
Background and Notation Let a fully-connected feed forward neural network f : D ⇢ R n 0 !
R n L be deﬁned on a compact domain D and have architecture ¯ n = [ n 0 , n 1 , ..., n L ] , i.e., depth Land width n l in layer l 2 (0 , ..., L ) , with continuous activation function  ( x ) .
On the relevantcompact domain let  ( x ) have Lipschitz constant T .
It maps an input vector x (0) to neurons x ( l ) i asx ( l ) =
h ( l )  with h ( l ) =
W ( l ) x
( l  1) + b ( l ) , where h ( l ) is the pre-activation, W ( l ) 2 R n l ⇥ n l  1 is the weight matrix, and b ( l ) 2 R n l is the bias vector of layer l .
Without loss of generality, letus assume that each parameter (weight or bias) ✓ is bounded as | ✓ |  1 .
Primarily, we distinguishthree networks.
First, we want to approximate a target network f t with architecture ¯ n t of depth L tconsisting of N t total nonzero parameters.
Second, this approximation is performed by a lotteryticket (LT) f ✏ that is obtained by pruning a larger source network f s , which we indicate by writingf ✏ ⇢ f s .
Third, this source network is a larger fully-connected feed-forward, randomly initializedneural network with architecture ¯ n s and depth L s .
While most LT existence results require exactly
L s = 2 L t , we show that any L s  L t + 1 is sufﬁcient.
[Figure]
To simplify the presentation, like most works, we assume aconvenient parameter initialization that we have to choosewith respect to the activation function if we approximate atarget layer with two source network layers.
In most cases,we make the following assumption.
Assumption 2.1 (Convenient initialization) .
We assumethat the parameters of the source network f s are independently distributed as w ( l ) ij ⇠ U ([  1 , 1]) , b (1) i ⇠
U ([  1 , 1]) and b ( l ) i = 0 for l > 1 .
Note that also other parameter distributions, e.g. normaldistributions, are covered as long as they contain a uniformdistribution [35], since this allows us to solve subset sumapproximation problems (see appendix for a formal definition).
While most works assume zero biases ( b ( l ) i = 0 ),as they focus on target networks without biases [30, 35],we initialize the biases in the ﬁrst layer as nonzero.
This issufﬁcient to approximate nonzero target biases, since wecan always construct constant neurons in each layer (seepink blocks in Fig.
1 (c)).
The convenient parameter initialization can always betransferred to a realistic one by learning or just applyingan appropriate layer-speciﬁc scaling factor  l , as also proposed in LT pruning experiments [52].
For homogeneousactivation functions like R E LU S , even a global parameter scaling is sufﬁcient [12].
For instance, let us assumethat our initial parameters in Layer l are distributed as
U ([   l ,  l ]) .
A common choice would be a He [19] or
Glorot [16] initialization with  l / 1 / p n l .
In this case,we would need to adapt our proofs by replacing { ✓ l } of a
LT by {  l ✓ l } with  l = 1 /  l to construct the same function that we have derived for convenient initializations.
For speciﬁc activation functions with  (0) 6 = 0 (e.g. SIGMOIDS ), instead of Assumption 2.1, we will assume aninitialization that has originally been derived to ensuredynamical isometry for R E LU S [2].
Interestingly, this initialization also supports LTs.
Assumption 2.2 (Looks-linear initialization) .
We assume that the weight matrices of the sourcenetwork f s are initialized as W ( l ) =
[Equation]
M ( l )  , where M ( l ) and b ( l ) are distributed as in
Assumptions 2.1.
In general, we measure the approximation error with respect to the supremum norm, which is deﬁnedas k g
k 1 :=
sup x 2 D k g k 1 for any function g and the L1-norm k x k 1 :=
P i | x i | .
In the formulation 134 of theorems, we also make use of the universal constant C that can attain different values.
Lottery Tickets as Subnetworks [30] were the ﬁrst to give probabilistic guarantees and provide alower bound on the required width of the source network that is polynomial in the width of the targetnetwork.
Their O ( n 5 t L 2 t / ✏ 2 ) requirement, or under additional sparsity assumptions O ( n 2 t L 2 t / ✏ 2 ) , has been improved to a logarithmic dependency of the form O ( n 2 t log( n t L t ) / ✏ ) for weights that followan unusual hyperbolic distribution [34] and O ( n t log( n t L t / ✏ )) for uniformly distributed weights[35].
While these assume target networks with zero biases, [12] transferred the approach by [35] tononzero biases.
All assume that the source network has exactly twice the depth of the target network( L s = 2 L t ).
Only [11] prove existence for extremely sparse tickets for L s =
L t +
but the widthrequirements are unrealistic for most target architectures.
[3] show how to leverage additional depth
L s  2 L t but still assume excessive depth in general.
Furthermore, all of these works focus on
R E LU activation functions.
Both limitations, the focus on R E LU S and the L s
= 2 L t requirementrely on the following construction idea that is also visualized in Fig.
1(a-b).
Every layer of the targetnetwork f t is represented by two layers in the lottery ticket f ✏ and equipped with R E LU activationfunctions  R ( x ) = max { x, 0 } with the possible exception of the last output layer.
We can obtaintwo layers by representing the identity as x =  R ( x )   R (  x ) and writing each target neuron x ( l ) t,ias x ( l ) t,i =  R ⇣P
[Equation]
The advantage of this 2-layer representation is that (a) we gain the ﬂexibility to select the neurons inthe middle layer among a higher number of available neurons in the source network and (b) theseneurons are univariate so that they depend on a single nonzero parameter and are thus simple toapproximate with high probability.
The question is how many nodes n 0 in the middle layer of thesource network are required to guarantee an accurate selection.
The answer distinguishes previouswork.
[35] achieve a factor  (where n s =  n t ) that is logarithmic in n s , ✏ ,  , etc. by solving aseparate subset sum approximation problem for each parameter utilizing results by [28].
[12] transfersthese results to target networks with nonzero biases.
Our ﬁrst contribution is to extend a similar construction to a wider class of activation functions thatis not restricted to R E LU S .
We need this result to approximate at least the ﬁrst layer of our targetnetwork in our second contribution, an L + 1 -construction, as shown in Fig. 1(c).
For the remaininglayers, we propose to construct the subset sum blocks for the next layer directly from the previouslayer by sharing nodes in the construction.
This has multiple advantages.
The obvious one is that wecan use the available depth of the source network to start from a potentially sparser target architectureto solve a given problem.
Also the LT itself consists usually of less neurons.
Furthermore, the subsetsum approximation of parameters becomes more efﬁcient.
Subset Sum Approximation
In the discussed construction, we generally have multiple random neurons and parameters available to approximate a target parameter z by b z up to error ✏ so that | z  b z |  ✏ .
169 Let us denote the independent random variables that we can use for this approximation as X 1 , ..., X m .
If they contain uniform distributions, Lueker [28] has shown that m  C log(1 / min { ✏ ,  } ) is sufﬁcient for the existence of a subset I ⇢ { 1 ..., m } so that the approximation of z by b z =
P k 2 I
X k is 172 successful with probability at least  .
Standard distributions of interest like uniform and normaldistributions as well as their products have this property [35].
For convenience, a precise corollary isstated in the appendix as Cor.
A.2.
Note that C depends on the distributions of the X k and is usuallylarger in the two-layers-for-one than in the one-layer-for-one construction, because the X k are givenby products X k = w ( l +1) 0 ,ik w ( l ) 0 ,kj in the former but X k = w ( l +1) 0 ,ik in the latter case.
2.1 Two Layers for One
Our ﬁrst contribution is to transfer the two-layers-for-one construction to activation functions thatcan be different from ReLUs.
We will utilize this result also in our one-layer-for-one constructionto represent the ﬁrst layer of the target network.
This is necessary as the input neurons are ﬁxed.
We need to increase their multiplicity in the ﬁrst layer of the source network to solve subset sumapproximation problems that create target Layer 1 in the LT’s Layer 2.
2.1.1 Activation Functions
The main property of ReLUs  R that is utilized in LT existence proofs is that we can represent theidentity as x =  R ( x )   R (  x ) .
This identity is exact and holds for all inputs x 2 R .
Yet, we canshow that, in combination with the right initialization of the source network, it is sufﬁcient if we canapproximate the identity on an interval that contains 0 .
This approximation is feasible with mostcontinuous activation functions that are not constant zero in a neighborhood of 0 , as we detail next.
Assumption 2.3 (Activation function (ﬁrst layer)) .
For any given ✏ 0 > 0 exists a neighborhood[  a ( ✏ 0 ) , a ( ✏ 0 )] of 0 with a ( ✏ 0 ) > 0 so that the activation function  can be approximated by b  ( x ) onthat neighborhood
such that sup x 2 [  a,a ] |  ( x )  b  ( x ) |  ✏ 0 , where b  ( x ) = m
+ x + d for x  0 andb  ( x ) =
m  x + d for x < 0 with m + , m  , d 2 R , and m + + m  6 = 0 .
We further assume that, if 193 a ( x ) is ﬁnite, g ( x ) =
x/a ( x ) is invertible on an interval ]0 , ✏ 00 ] with ✏ 00 > 0 and lim !
g ( x ) = 0 .
Most continuous functions and thus most popular activation functions fulﬁll this assumption.
Forinstance, R E LU S  R ( x ) = max( x, 0) inﬂict zero error on R (i.e., a = 1 ) with m + = 1 , m  = 0 ,and d = 0 .
Similarly, LR E LU S can be represented without error by m + = 1 , m  = ↵ , and d = 0for an ↵ > 0 .
( x ) = tanh( x ) is approximately linear so that | tanh( x )  x |  x 3 / 3 for | x | < ⇡ / 2 ,which can be seen by Taylor expansion of tanh .
This implies that the choice m + = 1 , m  = 1 , andd = 0 with a = min { (3 ✏ 0 ) 1 / 3 , ⇡ / 2 } fulﬁlls our assumption.
S IGMOIDS  ( x ) = 1 / (1 + exp
(  x ))can be analyzed in the same way with m + = m  = 0 .
25 , d = 0 .
5 , and a = min { (48 ✏ 0 ) 1 / 3 , ⇡ } ,since  ( x ) = (tanh( x/ 2) + 1) / 2 .
We should point out that not all continuous functions fulﬁll thisproperty.
Counterexamples include a shifted ReLU  ( x ) =
R ( x  1) or  ( x ) =
| x | .
However, inour L + 1 -construction, almost all activation functions can be arbitrary continuous functions.
Wewill only ask the activation functions in the ﬁrst layer to meet our assumption above.
How can werepresent the identity with such activation functions?
Lemma 2.4 (Representation of the identity) .
For any ✏ 0 > 0 , for a function  ( x ) that fulﬁlls
Assumption 2.3 with a = a ( ✏ 0 ) > 0 , and for every x 2 [  a, a ] we have
[Equation]
Note that R E LU S and LR E LU S inﬂict no approximation error so that the above statement holdsalso for ✏ 0 = 0 and a = 1 .
Some activation functions can have other advantages over R E LU S .
Forinstance, functions for which m + = m  = m and d = 0 like TANH can also be approximated by| x   ( x ) /m |  ✏ 0 /m and do not need separate approximations of the positive and the negative part.
Furthermore, SIGMOIDS and general activation functions with d 6 = 0 do not need nonzero biases inthe source network, since we can approximate a bias by random variables of the form X k = w k  (0) .
The challenge in utilizing this lemma is to incorporate the error that is inﬂicted by the approximationof the identity in addition to the pruning error into the analysis of the total approximation error.
Theallowed interval size a also inﬂuences which variables can enter our subset sum blocks and couldaffect our width requirement if we would not initialize our parameters in the right way.
2.1.2 Lottery Ticket Existence (Two-for-One)
Our ﬁrst goal is to identify a LT f ✏ that approximates a single hidden layer neural network f t ( x ) : D ⇢
R n 0 !
R with f t ( x ) =
t ⇣ P n 0 j =1 w t,j x j + b t ⌘ .
We could easily extend this result to approximateeach layer of a multilayer neural network but we will discuss a more promising alternative thatuses the following result for approximating the ﬁrst layer only.
f ✏ can be obtained by pruning afully-connected source network f s of depth L s = 2 with n s, 1 hidden neurons that are equipped withthe activation function  0 , which can be different from the outer  t .
The additional layer in f s hasthe purpose to create multiple copies of input nodes that can be used in subset sum approximations.
To achieve this despite the potential non-linear activation function  0 in the hidden layer, we haveto approximate the identity with the help of  0 .
The precise approximation can pose two additionalchallenges in comparison with the standard construction for R E LU S .
(a) If the piece-wise linearapproximation of  0 in Assumption 2.3 holds only on a ﬁnite neighborhood of 0 with a ( ✏ ) < 1 ,we can only use parameters that render the approximation valid.
Thus, the success of our approachrelies on an appropriate parameter initialization in the ﬁrst layer of the source network f s .
(b) Anonzero intercept ( d 6 = 0 ) in the approximation of  0 demands for a different initialization schemeto avoid the need for large bias approximations.
For simplicity, let us ﬁrst assume that  0 fulﬁlls
Assumption 2.3 with zero intercept ( d = 0 ).
Theorem 2.5 (LT Existence (Two-for-One)) .
Assume that ✏ 0 ,  0 2 (0 , 1) , a target network f t ( x ) :
D ⇢
R n 0 !
R n 1 with f t,i ( x ) =
t ⇣P n 0 j =1 w t,ij x j + b t,i ⌘ , and two-layer source network f swith architecture [ n 0 , n s, 1 , n 1 ] and activation functions  0 in the ﬁrst and  t in the second layer are given.
Let  0 fulﬁll Assumption 2.3 with a ( ✏ 00 ) > 0 and d = 0 ,  t have Lipschitz constant
T t , and M := max { 1 , max x 2 D ,i | x i |} .
Let the parameters of f s be conveniently initialized asw (1) ij , b (1)
i ⇠ U [   ,  ] and w (2) ij ⇠ U [  1 / ( | m + + m  |  ) , 1 / ( | m + + m  |  )] , b (2) i = 0 , where = min { 1 , a ( ✏ 00 ) /M } with ✏ 00 = g
[Equation] [Equation]
n
[Equation]
✏
[Equation]
TtM ⌘⌘⌘ .
Then, withprobability at least 1   0 , f s contains a subnetwork f ✏ 0 ⇢ f s so that each output component i is 243 approximated as max
[Equation]
[Equation]
Proof Outline.
The construction of a LT consists of three steps.
First, we prune the hidden neuronsof f s to become univariate.
Second, we identify neurons in the hidden layer for which we canapproximate  0 for small inputs according to Assumption 2.3.
Third, if n s, 1 is large enough, wecan select subsets I j and I b of the hidden neurons with small inputs so that we can use Cor.
A.2 onsubset sum approximation to approximate the parameters of the target.
The resulting subnetworkof f s is of the following form f ✏ 0 ,i ( x ) =
t
[Equation]
where I =
[ j I j and
P k 2 I j w (2)
ik w (1) kj can be used to approximate w t,ij .
f ✏ ,i qualiﬁes as LT if| f t,i ( x )  f ✏ 0 ,i ( x ) |  ✏ 0 .
A straight-forward series of bounds shows that we can achieve thisif  j =
[Equation]
⌘   ✏ 0 / ( T t ( n 0 + 1)) for each j .
This also applies to the approximation of b t,i , respectively.
We would like to approximate ⇣ w
[Equation]
⌘ ⇡ µ ± ( w (1) kj x j ) w (1) kj x j ± ✏ 00 , where µ ± ( x ) = m
+ x if x  0 and µ ± ( x ) = m
x otherwise.
Notethat µ ± ( x ) + µ ± (  x ) = m + + m  for all x 6 = 0 .
By construction, the activation functionapproximation is valid, as | w (1) kj x j | 
Ma ( ✏ 00 ) /M
= a ( ✏ 00 ) .
Hence, we can split the errorinto two subset sum approximation problems and the activation function approximation error:
[Equation]
[Equation]
where I j is split intoindices I + j for which w (1) kj >
and I  j all the ones for which w (1) kj < 0 .
The ﬁrst two terms can bebounded according to Cor.
A.2 with probability 1   00 if C log
[Equation]
⌘ randomvariables X k are available to choose from I ±
j , since the random variables
X k = | m + + m  | w (2)
ik w (1) kjare distributed as X k ⇠ U [  1 , 1] U [0 , 1] or X k ⇠ U [  1 , 1] U
[  1 , 0] and thus contain a uniformdistribution U
[  1 , 1] as shown by [35].
We can solve 2( n 0 + 1) n t, 1 independent subset sum approximation problems with probability 1   0 if each problem is solved with
=  0 / (2( n 0 + 1) n t, 1 )and if we have in total Cn 0 log
[Equation]
random variables available.
The full proof is given in the appendix.
A key result of our construction is that the activation functionapproximation error does not affect the width requirement.
As long as we choose the scaling factor small enough, we can achieve a small enough error.
Note that for R E LU S , the above theoremreduces to known results [12], since  = 1 , m + + m  = 1 , a = 1 , and d = 0 .
LR E LU Shave the same advantageous property and are now covered in addition.
The only difference is thatm + + m  = 1 + ↵ .
Another new insight is that also activation functions with ﬁnite approximationsupport a ( ✏ 00 ) <
1 support the existence of lottery tickets with the right parameter initialization with0 <
< 1 .
Interestingly, they can still achieve the same realistic width requirement.
Even though the theorem does not provide details on the universal constant, the minimum widthrequirement in this construction would be achieved by a linear activation function  0 ( x ) = x , whichdoes not require a distinction between positive and negative w (1) kj , has a ( ✏ 00 )
= 1 , does not inﬂictany approximation error, and is homogeneous, which makes it easy to transfer the above results torealistic parameter initialization schemes.
Remember that only  0 in the ﬁrst layer needs to fulﬁll Ass. 2.3.
t is an arbitrary continuousfunction.
However, it is relatively uncommon in practice to combine different activation functions in the same neural network so that  0 6 =  t .
Regardless, R E LU neural networks have been observed tolearn the identity in the ﬁrst layers (close to the input) [40].
For these reasons it could be beneﬁcial ingeneral to equip at least the ﬁrst layer with linear activation functions.
Also a ’looks-linear‘ parameter initialization can be of great beneﬁt, in particular, if the intermediaryactivation function  0 has a nonzero intercept d 6 = 0 , as shown next.
Theorem 2.6 (LT Existence (Two-for-One) with Nonzero Intercept) .
Thm.
2.5 applies also toactivation functions  0 that fulﬁll Assumption 2.3 with d 6 = 0 if the parameters are initializedaccording to Assumption 2.2 with M ( l ) 0 distributed as the weights in Thm. 2.5.
Proof Outline.
We can closely follow the steps of the previous proof.
The major difference is thatwe approximate  0 ( x ) ⇡ µ ± ( x ) x
+ d so that the activation function approximation produces anadditional error term, i.e.,
[Equation]
.
In principle, we could have modiﬁed the bias subset sumapproximation by approximating b t,i + d
P j
P k 2 I j w (2) ik instead of b t,i .
Yet,
P j
P k 2 I j w (2)
ik couldbe a large number, with which we would need to multiply our width requirement, if each w (2) ik isinitialized as in Thm.
In contrast, with the looks-linear initialization we can choose w (2) ik
=
w (2) ikso that
[Equation]
As for R E LU S [12], we could use these results to approximate every layer of a target multi-layerfeed-forward neural network for general activation functions by pruning two layers of a sourcenetwork with double the depth as the target network, as visualized in Fig.
1 (b).
As alternative, nextwe propose to prune a source network that has almost the same depth as the target network.
2.2 One Layer for One
Our second major contribution is to show how we can approximate intermediate target layers bypruning a single layer of the source network.
This is achieved by connecting subset sum approximationblocks directly as visualized in Fig.
1 (c).
The main idea is to create, instead of a single target neuron,⇢ copies to support subset sum approximations in the next layer.
In comparison with the two-layer-for-one construction, we have to solve a higher number of subsetsum approximation problems but this number is only higher by a logarithmic factor and can beintegrated in the universal constant C .
It is usually negligible.
In total, the lottery ticket might alsoconsist of a higher number of parameters, i.e., link weights, but a smaller number of neurons, whichis the more critical case to reduce computational costs [32].
The beneﬁts usually outweigh the costs,as we need less random variables to guaranty a successful subset sum approximation and, mostimportantly, can use almost the full depth of the source network to ﬁnd a sparser representation of thetarget network.
Furthermore, source networks of lower depth are easier to train and thus also moreamenable to pruning algorithms that utilize gradient based algorithms.
Interestingly, we can further relax our requirements on the activation functions, as they only inﬂuencethe error propagation through the network but are not involved anymore in creating random variablesfor the subset sum approximation problems.
Let us start with the approximation of a single layer.
Theorem 2.7.
Assume that ✏ 0 ,  0 2 (0 , 1) , a target network layer f t ( x ) : D ⇢ R n t,l ! R n t,l +1with f t,i ( x ) =
t P n t,
[Equation]
j =
[Equation]
w t,ij x j + b t,i  , and one source network layer f s with architecture[ n s,l +1 , n s,l +2 ] and the same activation function  t are given.
Let  t have Lipschitz constant 321 T , and deﬁne M := max { 1 , max x 2 D ,i | x i |} .
Let the parameters of f s be initialized according to 322 Assumption 2.1.
Then, with probability at least  0 , f s contains a subnetwork f ✏ 0 ⇢ f s so that ⇢ 323 copies of each output component i are approximated as max x
[Equation] [Equation]
Proof.
For each target input neuron j , we assume that we can create multiple copies (with indexset I j ) that we can use then as basis for subset sum approximations of target weights w ( l +1) t,ij .
Similarly, for biases we reserve neurons I b in the ﬁrst layer that are pruned so that we have  t ( c ) =
1 or another constant.
Particularly parameter efﬁcient would be  t (0) =
d .
Thus, our lotteryticket is of the form f ✏ 0 ,i 0 ( x ) =
t ⇣ P n t,l j =1 x j
P k
I
i
j ⇢ I j w ( l +2)
i 0 k +
P k
I
i
b ⇢ I b w ( l +2)
i
k  t ( c ) .
Thesubsets I i
j and I i
b are chosen so that | w ( l +1) t,ij
P k
I
i
j ⇢ I j w ( l +2)
i 0 k |  ✏ 0 / ( MT ( n t,l + 1))
and| b ( l +1) t,i
P k
I
i
b ⇢ I b w ( l +2)
i
k  t ( b ( l +1) k |  ✏ 0 /T ( n t,l + 1) , which can be achieved by subset sumapproximation based on the random variables
X k = w ( l +2)
i
k
⇠ U [  1 , 1] .
In total, we have to solvemaximally ⇢ n t,l +1 ( n t,l + 1) of these problems and can thus spend  0 / ( ⇢ n t,l +1 ( n t,l + 1)) on each ofthem.
Note that we only need a separate subset sum block for each input target neuron and not for its⇢ copies, as we can reuse each block in the construction of each output.
The advantage the construction above is that we can skip the approximation of the identity function.
The real challenge, however, lies in the derivation of the required multiplicative factor ⇢ forapproximating a multi-layer target network.
2.3 Multi-layer Lottery Ticket Existence
Our main result is to derive realistically achievable lower bounds on the width of a source networkthat has depth L s =
L t + 1 .
To prove the existence of LTs that approximate general multi-layer targetnetworks using the last theorems, we have to overcome two challenges.
First, when we considermultiple network layers, we need to understand how much error we can afford to make in each targetparameter approximation for general activation functions beyond R E LU S to stay below the globalerror bound, as error can get ampliﬁed when signal is sent through multiple layers.
Second, we needto identify the required layer-wise width scaling factor ⇢ in Thm. 2.7.
Lemma 2.8 (Error propagation) .
Let two networks f 1 and f 2 of depth L have the same architectureand activation functions with Lipschitz constant T .
Deﬁne M l := sup x 2 D  x ( l )  1 .
Then, for any✏ > 0 we have k f 1  f 2 k 1  ✏ , if every parameter ✓ 1 of f 1 and corresponding ✓ 2 of f 2 in layer l 349 fulﬁls | ✓ 1  ✓ 2 |  ✏ l for
[Equation]
For suitable target networks, this requirement essentially becomes ✏ l = C ✏ / ( n l L ) .
This estimate canbe used in two-layers-for-one as well as one-layer-for-one LT constructions.
The latter follows as ourmain result.
Theorem 2.9 (LT existence ( L + 1 construction)) .
Assume that ✏ ,  2 (0 , 1) , a target networkf t ( x ) : D ⇢ R n 0 !
R n L with architecture ¯ n t of depth L t , N t nonzero parameters, and a sourcenetwork f s with architecture ¯ n s of depth L s =
L t + 1 are given.
Let  t be the activation function off t in the layers l  2 of f s with Lipschitz constant T ,  0 be the activation function of the ﬁrst layerof f 0 fulﬁlling Assumption 2.3, and M := max { 1 , max x 2 D ,l
[Equation]
Let the parameters of f s beinitialized according to Ass.
2.1 for l > 2 and Thm.
2.5 or 2.6 for l  2 .
Then, with probability atleast 1   , f s contains a subnetwork f ✏ ⇢ f s so that each output component i is approximated as 360 max x 2 D | f t,i ( x )  f ✏ 0 ,i ( x ) |  ✏ if
[Equation]
◆ for l  1 , where ✏ l +1 is deﬁned by Lemma 2.8 and ⇢ = CN 1+  t log(1 / min { min l ✏ l ,  } ) for any > 0 .
Furthermore, we require n s, 1  Cn t, 1 log
[Equation]
The full proof is presented in the appendix.
While the layer-wise constructions are explained bythe proofs of Thms. 2.6, 2.5, and 2.7, yet with updated ✏ l , the main challenge is to determine theincreased number of subset sum problems that have to be solved to derive the scaling factor ⇢ .
Notethat ⇢ only enters the logarithm and is negligible.
By updating C , we in fact have the same asymptoticdependence n s,l +1
Cn t,l log ( n t,l L/ min { ✏ ,  } ) as in the two-layers-for-one construction.
Experiments
To demonstrate that our theoretical results make realistic claims, we prune for LTs that approximatea target network by following closely our constructive proofs.
Solving the associated subset sumproblems is NP-hard in general, as the size of the power set and thus the number of potential solutionsscales exponentially in the set size m , i.e., as 2 m .
However, as a set size of m = 15 and even smalleris sufﬁcient for our purpose, we could solve each problem optimally by exhaustively evaluating all 2 msolutions.
To reduce the size of the associated LTs, we instead identify the smallest subset consistingof up to 10 variables out of m = 20 to achieve an approximation error that does not exceed 0 .
As the inﬂuential work by [13], we use Iterative Magnitude Pruning (IMP) on LeNet networkswith architecture [784 , 300 , 100 , 10] to ﬁnd LTs that achieve a good performance on the MNISTclassiﬁcation task [7].
Using the Pytorch implementation of the Gihub repository open_lth 1 with
MIT license on a machine with Intel(R) Core(TM) i9-10850K CPU @ 3.60GHz processor and
GPU NVIDIA GeForce RTX 3080 Ti, we arrive at a target network for each of four consideredactivation functions after 12 pruning steps: R E LU, LR E LU, SIGMOID , and TANH .
Their performanceand number of nonzero parameters are reported in Table 2 alongside our results for the ( L + 1) -construction
and our 2 L construction, which achieve a similar performance.
Note that, while the
L + 1 construction relies in this case on a higher number of parameters, it uses less neurons and asmaller depth, which are the relevant criteria for fast computations and network training.
Additionalexperiments on tiny-ImageNet and the circle problem of [11] are reported in the appendix andhighlight more potential advantages of the L + 1 -construction.
We conclude that our proofs makerealistic claims and identify feasible conditions for LT existence.
Discussion
We have shown that randomly initialized fully-connected feed forward neural networks contain lotterytickets with high probability for a wide class of activation functions by deriving two types of constructions: (a) The ( 2 L t )-construction assumes that the larger random source network has at least doublethe depth of the target network and is wider only by a logarithmic factor O ( n t log( n t L t / min {  , ✏ } )in the approximation error and the LT existence probability.
(b) The ( L t + 1 )-construction allowsfor potentially sparser target network representations, as these can utilize almost the full availabledepth L s =
L t + 1 of the source network.
Remarkable about this result is that asymptotically, we canmaintain the logarithmic overparametrization.
It is important to note that in our MNIST experimentswe constructed the same target network based on source networks with different depth.
In practicehowever, the number of neurons and depth of the network are deﬁned by the source network andcould lead to a L s / 2 - or a L s  1 -construction of target networks with different representations.
An example is provided in the appendix.
The LT for the L t =
L s  1 - will often consist of fewerparameters than the LT corresponding to the L s / 2 -construction.
We have mainly discussed the case, in which the source network has exactly depth L s =
L t + 1 , butnot all target network representations become sparser with increasing depth.
What if the sparsesttarget representation has depth L t + 1 < L s ?
With the help of Lemma 2.4, we could simply addlayers that approximate the identity and thus construct a target with depth L t =
L s  1 .
In future, itcould be interesting to leverage excessive depth to distribute subset sum blocks on multiple layersinstead, as it has been proposed for R E LU S [3].
[Table]
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We discuss in detail for which
activation functions our theory holds and discuss advantages and disadvantages of the
derived LT constructions.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We do notsee any immediate negative societal impact that is speciﬁc to our contributions.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] See proof outlines inthe main paper and full proofs in the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See supplement.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See section on experiments.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We repeated each experiment with 5 independent random
seeds and report standard conﬁdence intervals as error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 3 on experiments.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We used the open_lth
package.
(b) Did you mention the license of the assets?
[Yes] .
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We submitted code in the supplement and will also make it available on Github upon
publication of the paper.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes] We only used MNIST.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]