Strong Lottery Ticket Hypothesis with ε –perturbation
Abstract
The strong Lottery Ticket Hypothesis (LTH) [18, 25] claims that there exists a subnetwork in a sufficiently large, randomly initialized neural network that approximates some target neural network without the need of training.
This work extends the theoretical guarantee of the strong LTH literature to a scenario more similar to the original LTH, by generalizing the weight change achieved in the pre-training step to some perturbation around the initialization.
In particular, we focus on the following open questions: By allowing an ε -scale perturbation on the random initial weights, can we reduce the over-parameterization requirement for the candidate network in the strong LTH?
Furthermore, does the weight change by SGD coincide with a good set of such perturbation?
We answer the first question by first extending the theoretical result on subset sum [14] to a scenario that allows perturbation on the candidates, and forms the conjecture of the perturbed strong
LTH by applying our generalized theoretical result to a neural network formulation.
To answer the second question, we show via experiments that, when a larger perturbation is allowed, the required over-parameterization of the strong LTH decreases, and the final accuracy after pruning increases.
1. Introduction
Pruning techniques for over-parameterized neural networks have drawn growing attention in recent years [1, 10–13, 19–21, 26].
Amongst them, the Lottery Ticket Hypothesis (LTH) [8, 9] claims the existence of a small (sparse) subnetwork within a large (dense) neural network such that, when trained in isolation, achieves comparable or even better performance than the original dense network.
Such subnetworks can be identified by pre-training the dense network and pruning it based on the magnitude of the learned weights [8].
Currently, to the best of our knowledge, the LTH lacks of any rigorous theoretical guarantees that justify superior performance of the subnetwork, especially under the pretraining-based pruning; yet, it has been proven to be effective in practice.
The Strong Lottery Ticket Hypothesis [18, 25] leverages a different pruning scheme: given a target dense neural network, and a randomly initialized, sufficiently over-parameterized candidate network, there exists a subnetwork in the latter that approximates the former arbitrarily well without the need of training.
While we require a significant over-parameterization in the randomly initialized network, the strong LTH enjoys extensive theoretical guarantees [15–17].
Yet, the same theory hardly applies to the LTH, as strong LTH assumes that the candidate weights pruned are fixed at initialization.
That LTH pruning is based on weights modified by pre-training motivates us to analyze the approximation behavior that emerges beyond the randomness in the candidate weights.
Further study on the pre-training process of LTH shows that the lottery ticket emerges in the early stage of training [23].
This implies that converging to a small training loss is not necessarily the intent of pre-training in the LTH procedure; in other words, achieving small loss does not necessarily explain how and why pre-training helps pruning in LTH.
Instead, one could hypothesize that the pre-training –based on loss minimization– could guide the weight perturbation to a direction that facilitate the pruning.
Based on this hypothesis, our work extends the theoretical guarantee of strong
LTH to a scenario more similar to the original LTH, by generalizing the weight change in the pretraining step to some perturbation around the initialization.
Our central question is as follows: “By allowing an ε -scale perturbation on the random initial weights, can we reduce the overparameterization requirement for the candidate network in the strong LTH?
Furthermore, does the weight change by SGD coincide with a good set of such perturbation?”
To be more specific, let f θ be a neural network with a set of parameters θ ∈ R d .
Formally, an εperturbation is a mapping P : R d → R d such that the maximum entrywise perturbation is bounded in absolute value by ε , i.e., ∥P ( θ ) − θ ∥ ∞ ≤ ε .
We point out that this definition of perturbation generalizes to two existing scenarios: when ε = 0 –i.e., we allow no weight perturbation– the question above reduces to the original strong LTH.
When ε = ∞ –i.e., we allow arbitrarily large weight perturbation– the required over-parameterization for the candidate network is at most the size of the target network.
In this case, one would use gradient based optimizers such as SGD to find such weight perturbation, but often without the need of pruning.
Yet, both cases cover only one aspect in pruning and perturbation.
In this paper, we study the inter-dependence of the two aspects above by treating ε as a variable.
In particular, we show that a larger perturbation scale ε , which corresponds to a larger amount of training, would alleviate the over-parameterization requirement, while keeping the accuracy of the pruned neural network the same.
Our contributions can be summarized as below:
• We consider a generalized version of the subset sum problem where each candidate in the summation is allowed perturbation bounded by a fixed scale ε .
We extend the analysis of the subset sum [14] to our generalized version, and show that when a larger perturbation is allowed, the required size of the candidate set can be reduced.
We empirically validate our theoretical result on the perturbed subset sum problem.
• Applying the theoretical result above to neural networks, we conjecture that, when an ε -scale perturbation is allowed, the strong LTH on randomly initialized neural network requires less overparameterization to achieve a specific approximation error.
• On neural networks, we empirically show that i ) the perturbation that alleviates the overparameterization requirement of the strong LTH can be obtained by projected SGD on the initialized weights; and, ii ) under fixed overparameterization, neural networks with a larger freedom over the level of perturbation achieves a higher accuracy after pruning.
This result establish the connection between the amount of pre-training and the accuracy of the pruned network.
2. Related Works
Lottery Ticket Hypothesis.
Several works attempt to explain the LTH theoretically.
[7] empirically study the behavior of gradient flow in the pruned network.
[24] assumes that the optimal mask is given, and proves that the pruned network achieves faster convergence and better generalization when trained from initialization.
[22] provides a theoretical guarantee of the loss in a pruning-after-training fashion.
However, these works differ from ours in the following: i ) they usually consider neuron pruning on small neural network archictures (e.g., [22] focuses on a twolayer neural network with smooth activations), while our work considers weight pruning of a deep
ReLU neural network; ii ) they consider minimizing the loss on a specific dataset, and require an over-parameterization that scales quadratically with the number of samples, while we approximate a target network with a fixed architecture in terms of the function norm, and require an overparameterization that scales with the width of the target network.
Strong Lottery Ticket Hypothesis.
The strong LTH originates from the empirical observation that, by fixing the weights at initialization and learning the mask the weights, one can identify subnetworks that achieve comparable accuracy to the dense one with learned weights [25].
[18] improved this idea by proposing the edge-popup algorithm to efficiently learn the mask.
[15] first proved such hypothesis under the assumption that the dense network’s size scales polynomially with the target network’s width and depth.
Leveraging the advantage of weight decomposition and theoretical results on the subset sum problem [14], [16] and [17] improved the over-parameterization requirement to a logarithm factor times the size of the target network.
Later work explores different variations of the strong LTH: [6] and [2] show that the strong LTH holds in the case of convolutional neural networks.
[3] extends the strong LTH proof to a universial family of functions.
Finally, [4] further reduces the over-parameterization requirement by employing the iterative randomization.
3. Notations and Setup
Notation.
For a vector a , we use ∥ a ∥ 2 to denote its ℓ 2 (Euclidean) norm, and ∥ a ∥ ∞ to denote its ℓ ∞ norm.
For a matrix A , we use ∥ A ∥ max = max ij | A ij | to denote its max norm.
Moreover, Unif ( I ) denotes the uniform distribution on the interval I , and Geom ( · ) and Bin ( · , · ) denotes the geometric and binomial distributions, respectively.
We use σ ( a ) = max { 0 , a } to denote the ReLU activation.
We use min {·} refers to the entrywise minimum and abs ( · ) refers to the entrywise absolute value.
Setup.
Similar to [17], our focus is to approximate an L -layer, ReLU activated target multi-layer perceptron (MLP) f ( x ) by pruning a 2 L -layer, ReLU activated candidate MLP g θ ( x ) .
For a input vector x ∈ R d 0 , we assume f ( x ) and g θ ( x ) are represented by:
[Equation]
We consider the pruning of g θ ( x ) with masks for the weights M = { M ℓ } 2 Lℓ =1 , denoted as g M , θ ( x ) :
[Equation]
For a set of weights θ , we consider its ε -perturbation
P ( θ )
= {P (cid:0) U ℓ (cid:1) } 2 Lℓ =1 , with P applied to each
U ℓ such that ∥P ( U ℓ ) − U ℓ ∥ max = max ij |P
( U ℓ ) ij − U ℓij | ≤ ε .
Our focus in this paper is on the approximation error defined as:
[Equation]
Subset Sum with ε -Perturbation
For each layer in the target network, [17] constructed a two-layer subnetwork with block structure, such that each block approximates a single entry in the weight matrix of the target network.
In particular, they obtain a logarithmic-scale over-parameterization by formulating the approximation as a subset sum problem [5, 14] Given a candidate set of values { x i } ni =1 of size n and a target value z , the solution to the subset sum problem finds the best approximation of z using the sum of a subset of { x i } ni =1 .
From an optimization perspective, the optimal approximation error
η ⋆ is the solution to the following problem
η ⋆ = min δ ∈{ 0 , 1 } n | (cid:80) ni
=
δ i x
i − z | where δ i ∈ { 0 , 1 } is the indicator variable on whether x i is selected in the sum to approximate z .
From the perspective of strong LTH, we can treat z as the weight entry in the target network that we wish to approximate, and { x i } ni =1 as the weights in the candidate network we will prune.
Here, δ i = 1 means that the i -th weight is kept, while δ i =
means that the i -th weight is pruned.
[14] shows that, with high probability over the randomness of x i ∼ Unif ([ − 1 , 1]) , a candidate set with size of the order n = Ω (cid:0) log η − 1 (cid:1) is enough to guarantee that η ∗ ≤ η for all z ∈ [ − 1 / 2 , 1 / 2 ] .
As an extension to the strong LTH, our setup incorporates an ε -perturbation on the weights of the random neural network.
This calls for the attention of extending the subset sum problem to a version with ε -perturbation.
In particular, we consider the following joint minimization problem:
[Equation]
We denote the values that lead to the optimal approximation error as δ ⋆ and y ⋆ , respectively.
In words, the above problem aims to select values from the set { x i } ni =1 such that, after potential entrywise perturbation by some tunable y i ∈ [ − ε, ε ] , the summation of the selected and perturbed (cid:80) ni =
δ i ( x i + y i ) will approximate z .
A central technical difficulty in our work is to extend the result of [14] to incorporate εperturbation.
Intuitively, as the perturbation scale ε becomes larger, each candidate is susceptible to a larger change in order to better approximate the objective z .
Intuitively, this implies that we should require a smaller size for the candidate size.
We formally show this in the theorem below.
Theorem 1 Given a candidate set { x i } ni =1 with x i ∼ Unif ([ − 1 , 1]) for all i ∈ [ n ] .
Consider the ε -perturbed subset sum problem in Equation 2.
Let the number of candidates be n = K 1 + K 2 with
[Equation]
Then with probability at least 1 − exp (cid:16) − K [Equation] 1+ [Equation]
− [Equation] −
[Equation] −
[Equation]
[Equation]
η } K 1 ) we have that all z ∈ [ − 1 / 2 , 1 / 2 ] has a 2 η -approximation.
Sketch of proof: The proof of Theorem (1) is provided in Appendix (1).
Compared with the proof of [14], we included the ε -perturbation when constructing the recurrence of the size of the target range that can be approximated.
We sketch the proof below:
We start by defining an indicator function f k,η ( z ) corresponding to the event that z has an ηapproximation by the first k candidates.
However, this recursively
defined sequence is hard to control as it involves the f k,η + ε ( z ) . We further study the behavior of the set where f k,η + ε ( z ) = 1 , and, by introducing the notion of ε -extensionm, we construct another sequence of indicator functions { ˆ f k } nk =1 that lower bound f k,η yet shows the advantage of large ε .
As in [14], we define p k to be the fraction of z on the interval [ − 1 / 2 , 1 / 2 ] such that ˆ f k = 1 .
Differently, we show that the expectation of p k +1 − p k is lower bounded by 1 / 2 (1 − p k )( p k + ε ) .
This demonstrates the expected growth p k +1 enjoys from p k .
Noticeably, this growth is larger when ε is larger.
Note that this property implies a lower bound on the expectation of p n .
We first show the lower bound on K 1 such that p K 1 ≥ 1 / 4 with a high probability. We do this by partitioning the interval [0 , 1 / 4 ] into sub-intervals that represents geometric grown.
We then show that the sum of the number of steps of growth that escapes these intervals can be represented as a binomial random variable, which can be bounded by applying Hoeffding’s inequality.
Next, starting from p k ≥ 1 / 4 , we lower-bound the summation of Z k +
=
p k
+1 − p k p k
(1 − p k ) using Azuma’s inequality.
To relate the summation of Z k +1 to the growth of p k , we define a function ψ ( p ) such that ψ ( p k +1 ) − ψ
( p k ) ≥ Z k +1 .
In this way, we arrive at a lower bound on ψ ( p K 1 + K 2 ) − ψ
Enforcing a lower bound on p K 1 + K 2 gives a lower bound on K 2 .
Remark.
Notice that the lower bound on the candidate set n depends on K 1 and K 2 , where Kscales inversely with log(5 + ε ) and K 2 scales inversely with 1 + 2 ε .
This implies that n decreases monotonically as ε increases.
We utilize this result to analyze the approximation error in Equation 1.
5. Strong Lottery Ticket Hypothesis with ε Perturbation
Next, we extend this idea to the approximation of a deep neural network. The idea is to approximate each layer in the target network using a two-layer ReLU MLP . Each entry in the weight matrix of the target network is approximated by a subnetwork of the MLP as in Lemma (8).
Therefore, a concatenation of these MLPs gives an approximation of the target network. Theorem 2 Consider approximating f with g as defined above.
Assume that assumption (2) holds. Also, assume that for 1 ≤ ℓ ≤ L ,
[Equation]
Then with probability at least 1 − d
[Equation]
2 L
[Equation]
max { ε
[Equation]
η } K[-2 − 
-]
[Equation]
we have L ( f, g ) < η , where L ( f, g ) is defined in equation (1).
6. Experiments 6.1. Approximating Neural Nets with SubsetSum and ε Perturbation
We study the effect of weight perturbation on the required overparametrization by approximating a two-layer, 500 hidden node target.
Each weight was approximated using a subset sum of n randomly initialized candidates where each candidate was allowed to perturbed by at most ε .
In particular, for some given ε , η , we say n satisfies the over-parametrization requirement if η ∗ in Equation 2 satisfies η ∗ ≤ η .
We randomly generate 10 sets of x i ’s, and e record the minimum n such that 8 of such sets make n satisfying the over-parametrization requirement.
We vary η from 1 e − 2 to 1 e − 4 and choose ε such that ε/η varies between 0 and 10 .
We are interested how such n changes as η and ε/η change.
From Figure 1( a )subfigure, we can observe that for fixed η , n decreases as ε/η increases.
[Figure] More specifically, as ε increases, more changes in ε are required to make a decrease in the minimum over-parameterization requirement n , which coincides with Theorem 1.
6.2. Perturbation Using SGD
We hypothesize that weight perturbation using SGD coincide with the desirable perturbation δ ∗ in equation (2 ).
We propose a two-stage algorithm to validate this hypothesis.
With a given perturbation scale ε , we first train an over-parameterized neural network using projected SGD to convergence.
Note that by applying the projected SGD we guarantee that each value of the neural network weight stays in an ε -neighborhood of initialization.
Then, we run edge-popup for a range of pruning (sparsity) levels.
We then consider the best accuracy amongst all pruning levels to be the optimal approximation scale.
Details referred to Algorithm 1.
We train a four-layer MLP with width 500 on MNIST.
We use Algorithm 1 to train the network: we use a learning rate of 0 . 03 for projected SGD epochs; for pruning we use edge-popup with a learning rate of 0 . 1 .
All weights in the network are initialized from Unif ([ − 1 / 2 , 1 / 2 ]) , and ε ranges from 0 to 0 .
The results are shown in Table 1 and Figure 1( c )subfigure,1( b )subfigure.
Note that as the perturbation scale ϵ increase, the optimal approximation error decreases.
Also, as ϵ increase, the pruning level that achieves the best approximation error decreases.
[Figure]
2 · 10 −3 · 10 −4 · 10 −5 · 10 −10 −2 · 10 − 0.
Appendix A. Using Projected Gradient Descent to Find ε -Perturbation
Data: Perturbation scale ε , neural network loss L , initial weight W 0 , learning rate { α t } T − 1 t = Result: Optimal loss ℓ ∗ , mask M ∗ and sparsity level s ∆ W ← 0 ; foreach t ∈ { 0 , . . , T − 1 } do ∆ W ← sign (∆ W − α t ∇L ( W t )) · min { abs (∆ W − α t ∇L ( W t )) , ε } ; W t +1 ← W 0 + ∆ W ; end ℓ ∗ ← ∞ , M ∗ ← None; foreach pruning level s ∈ { 0 . 1 , 0 . 2 , . . .
, 0 . 9 } do ℓ, M ← Edge-Popup ( L , W T , s ) ; if ℓ ≤ ℓ ∗ then ℓ ∗ ← ℓ, M ∗ ← M end end return ℓ ∗ , M ∗ , s
Appendix B. Proof of Theorem
The subset sum problem considers finding s ∈ { 0 , 1 } n that minimizes ℓ ( z, s ) =
| z − (cid:80) ni
=
s i x i | for a given z and given x i ’s. Previous work finds that, with n = Ω(log 1 / η ) , it holds with high probability that there exists s ∈ { 0 , 1 } n such that ℓ ( z, s ) ≤ η .
Alternatively, this problem can be started as finding the smallest n such that η ∗ ≤ η with
[Equation]
In our case, we would like to give the freedom of each x i to be perturbed for a small degree ε .
In particular, we extend the definition of ℓ to [Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
− n (cid:88) i =
s [Equation] [Equation] (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) and seeks condition of n such that η ∗ ≤ η with
[Equation]
If this condition is met for a fixed z , we say that such z has an η approximation.
Assumption 1 Let the candidate values x i ∼ Unif ([ − 1 , 1]) for all i ∈ [ n ] , and the target value z ∈ [ − 1 / 2 , 1 / 2 ] . Let 0 ≤ ε ≤ η ≤ 1 be given. Notice that if ε > 1 , then by Hoeffding’s inequality, [Equation]
When | (cid:80) ni =1 x i | ≤ nε 2 holds, we have that | (cid:80) ni
=1 ( x i + y i ) | can be anything in [ − nε / 2 , nε / 2 ] by varing y i . Therefore, as long as n = 1 ε 2 log s − 1 , it holds with probability at least 1 − s that η ∗ =for all z ∈ [ − 1 / 2 , 1 / 2 ] . Thus, our focus is on the case of ε ≤ 1 . Under this assumption, we attempts to prove the following theorem
Theorem 1 Let the number of candidates satisfy n = K 1 + K 2 with [Equation] Then with probability at least 1 − exp (cid:16) − K 2 (1+
[Equation]
2 8(3 − [Equation] ) 2 (cid:17) − exp (cid:0) − K 1 18 (cid:1) − exp ( − max { ε, η } K 1 ) we have that all z ∈ [ − 1 / 2 , 1 / 2 ] has a 2 η -approximation.
We define the indicator function for the existence of ˆ η -approximation within the first k candidate.
[Equation]
This indicator function has the following recurrence
[Equation]
Define the following random variable (depending on { x k } ki =1 )
[Equation]
This random variable denotes the portion of z ∈ [ − 1 / 2 , 1 / 2 ] that can be approximated within η error. Definition 1 For a candidate set { x i } n i =1 , and some k ∈ { 0 } ∪ [ n ] , define its ( k, η ) -feasible set as
[Equation]
By definition, F k,η is the union of finitely many mutually disjoint closed intervals on [ − 1 / 2 , 1 / 2 ] .
Let µ denote the Lebesgue measure on R . Consider the following definition of ε -extension of a set Definition 2 Let I ⊂ [ − 1 / 2 , 1 / 2 ] be a closed interval. A set S is called an ε -extension of I, denoted
S ∈ Ξ ε ( I ) if
S ⊆
[ − 1 / 2 , 1 / 2 ] \ I 2.
for all s ∈ S , we have that min
a ∈ I | s − a | ≤ ε µ ( S ) = min { ε, 1 − µ ( I ) } By definition, for each I ⊂ [ − 1 / 2 , 1 / 2 ] , there is at least one ε -extension of I , since we can choose
S =
[ − 1 / 2 , 1 / 2 ] ∪ (cid:40) [ − 1 / 2 , inf I ) ∪ (sup I, sup I − inf
I + ε − 1 / 2 ] if inf I
≤ ε −[inf I − ε, inf I ) otherwise
Let F = ∪ mj
{+=
+}
I j be a finite union of closed intervals.
A set S is called an ε -extension of F , denoted by S ∈ Ξ ε ( F ) if S ⊆ ∪ mj =1 ∪ ξ j ∈ Ξ ε ( I j ) ξ j \ F 2.
µ ( S ) =
min { ε, 1 − µ ( F ) }
By lemma 2, there is at least one ε -extension of F .
{+Lemma 2 There is at least one ε -extension for each F of the form F = ∪ mj =
+}
{+I j . Proof Suppose there is no ε -extension of some F = ∪ mj =
 I j .+}
Consider two cases:
Case 1 : µ ( F ) ≥ 1 − ε .
Since F has no ε -extension, there must be a subset A of [ − 1 / 2 , 1 / 2 ] with nonzero Lebesgue measure such that every element in A is at least ε away from F . This is, however, a contradiction, since by µ ( F ) ≥ 1 − ε , every point in [ − 1 / 2 , 1 / 2 ] must be within ε distance of F . Case 2 : µ ( F ) ≤ 1 − ε . Let S = ∪ mj =1 ∪ ξ j ∈ Ξ ε ( I j ) ξ j .
Since F has no ε -extension, we must have µ ( S ) < 1 .
This implies that there exist a ∈ [ − 1 / 2 , 1 / 2 ] such that a / ∈ S .
Thus inf a ′ ∈F | a − a ′ | ≥ f .
Let such a ′ be given, then if a > a ′ , ( a ′ , a ′ + ε ] is an ε -extension, and if a < a ′ , [ a ′ − ε, a ′ ) is an ε -extension.
This is a contradiction.
Let { S k } nk =0 be given such that S k ∈ Ξ ε ( F k,η ) . Moreover, let g k ( z ) = I { z ∈ S k } .
We define another recurrence of indicator function
[Equation] {+Lemma 
+}
{+The sequence (cid:110) ˆ f k (cid:111) n k =
 satisfies ˆ f k ( z ) ≤ f k,η ( z ) for all z ∈ [ − 1 / 2 , 1 / 2 ] . Proof We show this by induction.+}
For k = 0 , we have ˆ f 0 ( z ) = f 0 ,η ( z ) by definition. Assume ˆ f k ( z ) ≤ f k,η ( z ) , we would like to show ˆ f k +1 ( z ) ≤ f k +1 ,η ( z ) . To do this, we first notice that, by definition of g k
[Equation]
Moreover, if g k ( z ) = 1 , we must have f k,η ( z ) = 0 .
Therefore (1 − f k,η ( z )) g k ( z ) = g k ( z ) . This implies that
[Equation]
Using this, we have
[Equation]
This completes the proof
[Equation]
Based on the definition of (cid:110) ˆ f k (cid:111) n k = [Equation] we define { ˜ p k } nk =0 as
[Equation]
p k
[Equation] [Equation]
k
[Equation]
z
[Equation]
dz
{+Then by definition we have ˜ p k ≤ p k , with ˜ p 
+}
= p 0 .
Moreover [Equation] we have ˜ p k +1 =
(cid:90) 1 /− 1 /
[Equation]
k
[Equation]
Furthermore
[Equation]
by definition of ˜ p
k
+ [Equation]
we have ˜ p k +1 ≤ 1 .
For ˜ p
k
[Equation]
we can compute its expectation with respect to { x i } ki =1 as
[Equation]
We define K 1 as below
[Equation]
if
[Equation] [Equation]
ε
[Equation]
otherwise To upper bound K 1 , we consider 14 ≤ 1 − ε . {+Lemma 
+}
For all 0 ≤ k ≤ K , it holds that
[Equation]
Proof Given that ˜ p k ≤ 14 , we can show that
[Equation]
Moreover, recall that we have that ˜ p k +1 ≤ 2˜ p k + ε .
Thus, we can apply the reverse Markov’s inequality
[Equation]
Lemma 5 With probability at least 1 − exp (cid:0) − 118 K 1 (cid:1) we have that K ≤ O [Equation] [Equation]
(cid:1) (cid:33)
Proof By using ˜ p k ≤ 14 , it thus follows from lemma (4) that
[Equation]
Denote β = (cid:0) 54 + ε 2 (cid:1) . As in previous work, we define the following partition of the interval (0 , 1 / 4 ) .
[Equation]
where i ∗ is the smallest integer such that β i ∗ η ≥ 14 , that is
[Equation]
For i > 1 , let ˆ k i be given such that ˜ p
ˆ k
i ≥ β i − 1 η , let ˆ Y i be the smallest number of steps such that ˜ p ˆ k
i +ˆ Y i > β i η .
Then we have that
ˆ Y i ≤ Y i ∼ Geom ( 1 / 6 ) , since, according to lemma (4), we have
[Equation]
Therefore, for all K ∗ , we have that
[Equation]
where B K ∗ ∼ Bin ( K ∗ , 4 / 7 ) .
Given that E [ B K ∗ ] = 16 K ∗ , we can apply the Hoeffding’s inequality for binomial distribution
[Equation]
choose K ∗ = 12 i ∗ and t = 16 K ∗ gives that
[Equation]
B.2. Growth from 1 / 4 to 1 − max { ε, η }
Recall the recurrence
[Equation]
We define
[Equation]
Then we have E [ Z k +1 ] ≥ 1 / 2 .
Let Y k
= − k / 2 + (cid:80) K
+ k +1 i
=
K
+1 Z i , then Y k is a submartingale.
We bound Z k +1 as follows
Lemma
[Equation]
Proof We notice that ˜ p k ≤ ˜ p k +1 ≤ min { 2˜ p k + ε, 1 } .
Consider two cases of p k :
Case 1 : ˜ p k ≤ 1 − ε 2 .
In this case, we have 1 − ˜ p k ≥ 1+ ε
[Equation]
Case 2 : ˜ p k ≥ 1 − ε 2 .
In this case, we use ˜ p k +1 ≤ 1 . Moreover, we have ˜ p k
+ ε ≥ 1+ ε 2 :
[Equation]
Thus,
[Equation]
Let n = K 1 + K 2 + 1 .
Therefore, we can apply Azuma’s inequality to get that
[Equation]
Let t = K 2 4 gives that
[Equation]
We use the following function to track the growth of p k , but starting from 1 / 4 .
[Equation]
Lemma
For all p k ≥ 14 , we have that
[Equation]
Proof We first notice that
[Equation]
It suffice to show that
[Equation]
The first- and second-order derivative of ψ are
[Equation]
Therefore, ψ ′ attains its minimum at p ∗ = min (cid:8) 1 , 1 − ε 2 (cid:9) , and ψ ′ decreases monotonically on ( 1 / 4 , p ∗ ] and increases monotonically on [ p ∗ , 1] . Notice that the function 1 (˜ p k + ε )(1 − ˜ p k ) also decreases monotonically on ( 1 / 4 , p ∗ ] and increases monotonically on [ p ∗ , 1] .
We consider two cases of p k :
Case 1 : ˜ p k ∈ ( 1 / 4 , p ∗ ] .
In this range the function 1 (˜ p k + ε )(1 − ˜ p k ) decreases monotonically.
Thus it achieves its maximum at ˜ p k = 1 / 4 with a value of {+16 
+}
+12 ε . However, since 0 ≤ p ≤ 1 , we have
[Equation]
Thus min
[Equation]
Case 2 : ˜ p k ∈ ( p ∗ , 1] .
Notice that ψ increases monotonically on ( p ∗ , 1] .
Thus
[Equation]
Therefore, we have
[Equation]
Plugging in the value of ψ (˜ p n ) and ψ (˜ p K 1 ) , and notice that ψ increases monotonically with p
[Equation]
Since (cid:80) ni
= K 
 {++
+}
Z i ≥ K 2 4 with probability at least 1 − exp (cid:16) − K 2 (1+ ε ) 2 8(3 − ε ) 2 (cid:17) ,
as long as K 2 ≥ 2log η − 1 1+ ε + 4 =
O ( log η − 1 1+ ε + 1) , we have that with high probability (cid:80) ni
= K
+1 Z i ≥ log η − 1 1+ ε [Equation] 4 , which implies that
[Equation]
which implies that ˜ p n ≥ 1 − max { η, ε } .
This shows that, with high probability, for K 1 , K 2 defined above, n = K 1 + K 2 + 1 candidates guarantess that each point z ∈ [ − 1 / 2 , 1 / 2 ] either has an η approximation or is max { η, ε } away from an η approximation.
In the case of η ≥ ε , we have that each z has a 2 η approximation.
Otherwise, if ε > η , we need an additional set of candidates to grown from 1 − ε to 1 − η .
B.3. Growth from 1 − ε to 1 − η under ε > η
Consider another set of candidates { ˆ x i } K 3 i =1 . By Hoeffding’s inequality, we have that [Equation] This implies that with probability at least 1 − exp (cid:16) − (( K 3 − 1) ε + η ) 2 2 K
[Equation]
, for each ˆ y ∈ [ η − ε, ε − η ] , there exists y ∈ [ − ε, ε ] K 3 such that ˆ y = (cid:80) K ′ i
=
1 ( x i
+ y i ) .
This implies that with probability at least 1 − exp (cid:16) −
[Equation] [Equation] [Equation]
(cid:17) , for all z ∈ [ − 1 / 2 , 1 / 2 ] we have that z has an η approximation. For convenience we can choose K 3 = K 1 + 1 , which results in a success probability at least 1 − exp ( − max { ε, η } K 1 ) Thus, as long as n = 2 K 1 + K 2 + 1 with
[Equation]
we have that with probability at least 1 − exp (cid:16) −
[Equation]
− [Equation] max { ε, η } K 1 ) , each point in [ − 1 / 2 , 1 / 2 ] either has η approximation or lies with η distance to a point with η approximation. Therefore, each point in [ − 1 / 2 , 1 / 2 ] has an 2 η approximation.
Appendix C. Proof of Theorem
Lemma 8 Let g : R → R be a randomly initialized network of the form g ( x ) = v ⊤ σ ( u x ) , where v , u ∈ R 2 n , n = K 1 + K 2 , [Equation] where u i = 1 for i ≤ n , u i = − 1 for i ≥ n + 1 , and v ′
i s are drawn from Unif [ − 1 , 1] . Then, with probability at least 1 − δ , there exist s ∈ { 0 , 1 } 2 n , y ∈ [ − ε, + ε ] 2 n such that [Equation] for all w ∈ [ − 12 , 12 ] with
[Equation]
Proof Note that wx = σ ( wx ) − σ ( − wx ) and without loss of generality we assume w ≥ 0 . The case of w < 0 can be handled by changing x to − x . Furthermore, we decompose u , v , y , s by u = (cid:18) uu [Equation]
(cid:19) , v = (cid:18) vv(cid:19) , s = (cid:18)
ss
[Equation]
(cid:19) ,
y = (cid:18) yy(cid:19) , where u 1 = 1 n , u 2 = − 1 n , v 1 , v 2 ∈ R n , s 1 , s 2 ∈ { 0 , 1 } n , and y 1 , y 2 ∈ [ − ε, ε ] n .
Then we have
[Equation]
We use the first half of the RHS to approximate σ ( wx ) and use the second half of the RHS to approximate − σ ( − wx ) .
Approximating σ ( wx ) .
Note that since w ≥ 0 , then for x ≤ 0 , ( v 1 + y 1 ) ⊤ σ (( u 1 ⊙ s 1 ) x ) = σ ( wx ) = 0 .
Consider x > 0 .
By definition of u 1 , we have
[Equation]
Now consider ( (cid:80) ni
=1 v 1 ,i ) , Theorem 1 states that with probability at least 1 − δ 4 ,
[Equation]
Since
[Equation]
with probability at least 1 − δ 4 , we have
[Equation]
Since | x | ≤ 1 , with probability at least 1 − δ 4 , we have ∀ w ∈
[Equation]
, ∃
[Equation]
Recall that ( v 1 + y 1 )
⊤ σ (( u 1 ⊙ s 1 ) x ) = σ ( wx ) =
for x ≤ 0 .
Also, for x > 0 , σ ( wx ) = wx and ( v 1 + y 1 ) ⊤ σ (( u 1 ⊙ s 1 ) x ) = ( v 1 + y 1 ) ⊤ ( s 1 ⊙ u 1 ) x . Therefore, with probability at least 1 − δ 4 , we have ∀
[Equation]
Approximating − σ ( − wx ) . For x ≥ 0 , ( v 2 + y 2 ) ⊤ σ (( u 2 ⊙ s 2 ) x ) = − σ ( − wx ) = 0 . Now, consider x < 0 . By definition of u 2 , it holds that
[Equation]
Therefore, similarly we have that with probability at least 1 − δ 4 , ∀ w ∈
[Equation]
Hence by a union bound
[Equation]
with probability at least 1 − δ 2 ,
[Equation]
<η.
Note that for the case w ≤ 0 , the result has the same probability and the approximation error, so by a union bound, the lemma hold with probability at least 1 − δ .
Lemma 9 Let g : R d 1 → R d 2 be a randomly initialized network of the form g ( x ) =
V σ ( Ux ) , where V ∈ R d 2 × 2 n , U ∈ R 2 n × d 1 , n = K 1 + K 2 ,
[Equation]
where weights in V are drawn i.i.d.
from Unif [ − 1 , 1] , U = (cid:18) U
+
U − (cid:19) , with U + being a matrix of all 1 s and U − being a matrix of all − 1 s .
Let ˆ g ( x ) =
( S ⊙ ( V + Y ))
σ (( B ⊙ U ) x ) be the pruned network for masks S ∈ { 0 , 1 } d 2 × 2 n , B ∈ { 0 , 1 } 2 n × d 1 and perturbation matrix Y ∈ [ − ε, ε ] 2 n × d 1 .
Let the target network be f W ( x ) =
Wx , then with probability at least 1 − d 1 d 2 (cid:16) exp
[Equation]
− K
[Equation]
1+
[Equation]
−
[Equation]
[Equation]
− exp
[Equation]
−
[Equation]
1 18 (cid:1) − exp ( − max { ε, η } K 1 ) (cid:17) , there exist S , B , Y such that sup
[Equation]
for all W such that ∥ W ∥ ∞ ≤ 12 .
Proof Since U can be written as (cid:18)
[Equation]
(cid:19) , with U + being a matrix of all 1 s and U − being a matrix of all − 1 s, we choose ˆ B such that ˆ B ⊙ U is of the form
ˆ B ⊙ U =    u +  . . .
u + . . .
u + du − . . .
u −. . .
u − d   where u + j = 1 and u − j = − 1 .
Moreover, we decompose S ⊙ ( V + Y ) as
[Equation]
[Equation]
d
[Equation]
v
[Equation]
 
[Equation]
y
[Equation]
[Equation]
+ ⊤
[Equation]
y
[Equation]
−⊤ d
[Equation]
d
[Equation]
  where each s ± i,j , v ± i,j , y ± i,j ∈ R n/d 1 .
Then we have
[Equation]
Letting v ij = (cid:18) v + ij v − ij (cid:19) , s ij = (cid:18) s + ij s − ij (cid:19) , y ij = (cid:18)
y
+ ij y − ij (cid:19)
and y ij = (cid:18) y
+ ij y − ij (cid:19) , we then have (cid:104) ( S ⊙ ( V + Y ))
σ (( ˆ B ⊙ U ) x ) (cid:105) i =
d 1 (cid:88)
j
=
(( v i,j + y i,j ) ⊙ s i,j ) ⊤ σ ( u j x j )
Now define the event
F i,j,η :=   sup w : | w |≤inf s i ∈{ 0 , 1 } 2 n/d 1 , y i,j ∈ [ − ε,ε ] 2 n/dsup x : | x |≤(cid:12)(cid:12) (cid:12) wx − (( v i,j + y i,j ) ⊙
s i,j ) ⊤ σ ( u j x ) (cid:12)(cid:12) (cid:12)
<
η   .
Define F η := (cid:84) d 2 i =1 (cid:84) d 1 j =1 F i,j,η , then
P (cid:16) F η d 1 d(cid:17) ≥ 1 − d 1 d(cid:18) exp (cid:18) − K 2 (1 + ε )8(3 − ε )(cid:19) − exp (cid:18) − K (cid:19) − exp ( − max { ε, η } K 1 ) (cid:19) .
On event F η d 1 d 2 , we have sup
[Equation]
i
[Equation]
d
[Equation]
=
[Equation]
d
[Equation]
j =
[Equation]
j
[Equation]
j
[Equation]
i
[Equation]
j
[Equation]
j
[Equation]
i
[Equation]
d
[Equation]
(cid:88)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
=
[Equation]
With the help of this lemma, we are ready to prove theorem 2.
Recall that our goal is to approximate an L -layer, ReLU activated target multi-layer perceptron (MLP) f ( x ) by pruning a 2 L -layer,
ReLU activated candidate MLP g ( x ) .
For some input vector x ∈ R d 0 , we assume f ( x ) =
f L ( x ) has a fixed set of parameters { W ℓ } Lℓ =1 , represented by:
[Equation]
where W ℓ ∈ R d ℓ × d ℓ − 1 .
Similarly, let g ( x ) =
g 2 L ( x ) with parameters { U ℓ } 2 Lℓ =1 , represented by:
[Equation]
where U ℓ ∈ R ˆ d ℓ × ˆ d ℓ − 1 .
In particular, g is a neural network with twice the depth of f .
We consider the pruning and ε -perturbation of g ( x ) with a set of masks for the weights S
= { S ℓ } 2 Lℓ =1 and perturbation matrices Y =
{ Y i } Li =1 , denoted as g S , Y ( x ) =
g 2 L S , Y ( x ) :
[Equation]
Let F Y denote the feasible set of the perturbation Y .
Also recall our assumptions for the setup
Assumption 2 We assume the following condition for f, g and F Y :
For all ℓ ∈ { 0 }∪ [ L ] , the weight matrix W ℓ of the target neural network f satisfies ∥ W ℓ
∥
≤and (cid:13)(cid:13) W ℓ (cid:13)(cid:13) ∞ ≤ 12 . 2.
The initialization of g satisfies U 2 ℓij ∼ Unif [ − 1 , 1] , and U 2 ℓ − 1 ij = 1 if i ≤ ˆ d 2( ℓ − 1) / 2 and U 2 ℓ − 1 ij =
[Equation]
1 if i > ˆ d 2( ℓ − 1) / 2 for all ℓ ∈
[ L ] and j ∈ [ ˆ d 2 ℓ − 3 ] .
The feasible set of Y is defined as
[Equation]
We focus on the approximation error defined as:
[Equation]
We state the theorem here for convenience
[Equation]
Theorem 10 Consider approximating f with g as defined above.
Assume that assumption (2) holds.
Also, assume that for 1 ≤ ℓ ≤ L ,
[Equation]
2 ) × d ℓ − 1 .
Then with probability at least 1 − 2 d 1 d 2 L (cid:16) exp (cid:16) − K 2 (1+
[Equation]
[Equation]
(cid:17) − exp (cid:0) −
[Equation]
18 (cid:1) − exp ( − max { ε
[Equation]
K
[Equation] where g S , Y is a pruning & ε -perturbation of g . Proof By Lemma 9, for ℓ -th layer, with probability 1 − d 1 d 2 δ with [Equation]
we have
[Equation]
[Equation]
where F W ℓ = { W ∈ R d ℓ × d ℓ − 1 : : ∥ W ∥ ≤ 1 , ∥ W ∥ ∞ ≤ 12 } .
Since ReLU is 1 -Lipschitz, with same probability, we have
[Equation]
[Equation]
Then with probability at least 1 − 2 d 1 d 2 Lδ , (5) and (6) hold simultaneously for every layer 1 ≤ ℓ ≤
L .
Equation (6) implies for 1 ≤ ℓ ≤ L − 1 ,
[Equation]
[Table]
Since (cid:13)(cid:13) W ℓ
[Equation]
≤ 1 for all ℓ ∈ [ L ] , we have that
[Equation]
This implies that, for all x such that ∥ x ∥ ≤ 1 ,
[Equation]
Thus, we have that for all x such that ∥ x ∥ ≤
[Equation]
Solving the recurrence thus gives
[Equation]