P ROVING
THE
L OTTERY
T ICKET
H YPOTHESIS
FOR
C ONVOLUTIONAL N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
The lottery ticket hypothesis states that a randomly-initialized neural network contains a small subnetwork such that, when trained in isolation, can compete with the performance of the original network.
Recent theoretical works proved an even stronger version: every sufﬁciently overparameterized (dense) neural network contains a subnetwork that, even without training, achieves comparable accuracy to the trained large network.
These works left as an open problem to extend the result to convolutional neural networks (CNNs).
In this work we provide such generalization by showing that, with high probability, it is possible to approximate any CNN by pruning a random CNN whose size is larger by a logarithmic factor.
I NTRODUCTION
Many impressive successes in machine learning are reached through neural network architectures with a huge number of trainable parameters.
Consequently, substantial research in the ﬁeld aims at reducing the size of such networks while maintaining good accuracy, e.g. for deployment in resource constrained devices (Yang et al., 2017).
A major empirical fact of such endeavour is the contrast between the initial model overparametrization, which appears necessary for effective training, and the extent to which the size of the resulting model can be reduced through compression techniques.
Among the latter, pruning methods appear as a mature and efﬁcient way of achieving signiﬁcant compression, often without incurring any accuracy loss (Blalock et al., 2020).
Recently, the aforementioned contrast between the initial and ﬁnal number of parameters has been addressed by the lottery ticket hypothesis (Frankle & Carbin, 2019), or LTH for short, which states that any randomly initialized network contains lottery tickets , that is, sparse subnetworks that can be trained just once and reach the performance of the fully-trained original network.
This hypothesis was ﬁrst veriﬁed experimentally, leveraging pruning methods to identify the lottery tickets (Frankle & Carbin, 2019; Lee et al., 2019).
(Ramanujan et al., 2020) then proposed a stronger version of the hypothesis, named strong lottery ticket hypothesis (SLTH) by (Pensia et al., 2020): it stipulates that a network with random weights contains, with high probability, sub-networks that can approximate any given sufﬁciently-smaller neural network.
In other words, a sufﬁciently large and randomly initialized network that can be successfully trained for a task, could instead be suitably pruned to obtain a network that, even without training, achieves good accuracy.
Experimental support for this stronger version were reported by Ramanujan et al.
; Zhou et al.
; Wang et al.
(2020), making use of pruning methods to ﬁnd lottery tickets in a range of architectures, including convolutional neural networks (CNNs).
A ﬁrst rigorous proof of the SLTH has been given by Malach et al.
(2020) for the case of dense networks (i.e. consisting of fully connected layers).
Pensia et al.
(2020) and Orseau et al.
(2020) successively improved this result by showing that logarithmic over-parametrization is sufﬁcient.
Their results are also restricted to dense networks and they leave as an open problem to extend it to
CNNs.
Our contributions.
We extend and complete the proof of the SLTH (and thus, also, of the LTH), for classical network architectures which can combine convolutional and fully connected layers.
More precisely, we prove that any CNN with given weights can be approximated by pruning a CNN with random weights (random CNN for short), with the latter being larger than the former by a logarithmic factor.
We also provide basic experiments showing that starting from a random CNN which is roughly 30 times larger than LeNet5, it is possible to compute in few hours a pruning mask that allows to approximate the trained convolutional layers of LeNet5 with relative error 10 − 3 , even when ignoring some hypothesis of our theoretical result.
Our theoretical analysis follows the approach of Malach et al.
and make use of two layers to approximate one.
We borrow from
Pensia et al.
the use of random subset sum (RSS) (Lueker, 1998) to approximate a given weight via the sum of a subset of a sample of random weights, and carefully design instances of
RSS via a combination of two convolutional layers.
By controlling the error accumulated by each layer with Young’s convolution inequality, we establish the following result.
Informal version of Theorem 1.
Given ε, δ > 0 , any CNN with k parameters and (cid:96) layers, and kernels with (cid:96) 1 norm at most 1, can be approximated within ε by pruning a random CNN with O ( k log( k(cid:96) min { ε,δ } )) parameters and 2 (cid:96) layers with probability at least 1 − δ .
This result generalizes those by Pensia et al.
(2020), Orseau et al.
(2020) and Malach et al.
(2020), as the dense-network case is retrieved as a special case of our theorem, when the kernel size matches the input size.
Roadmap.
After discussing related work in the next section, we state our theoretical results alongside a high-level idea of the proofs.
Successively, we report our experimental results.
Finally, in
Section 4, we provide detailed proofs of our statements.
1. R ELATED W ORK
Pruning methods are classical neural network compression strategies that date back to the 80’s (LeCun et al., 1989b; Mozer & Smolensky, 1988).
We recommend the recent survey Blalock et al.
(2020) for an overview of the current state of research on these techniques.
As for the lottery ticket hypothesis, Lange (2020) provides a survey which summarizes the progress on the topic until the results by Malach et al.
In the following we brieﬂy mention works which are not discussed in Lange (2020).
Cosentino et al.
(2019) shows that lottery tickets can be adversarially trained, yielding sparse and robust neural networks.
Soelen & Sheppard (2019) shows that lottery tickets are transferable, in the sense of showing remarkable accuracy for tasks other than the original one for which they have been found.
Sabatelli et al.
(2021) further shows that minimal retraining on a new task allows lottery tickets to often achieve better generalization than models trained ad-hoc for the task.
(Yu et al., 2020) empirically supports that the LTH holds also in the context of reinforcement learning and natural language processing.
(Diffenderfer & Kailkhura, 2021) shows that lottery tickets are also robust against extreme quantization of the weights.
Aladago & Torresani (2021) provides a method to train networks where each initial weight is restricted to few possible random values.
An extreme case of the latter is to share only a single (random) value among all weights, and focus the training solely on ﬁnding the best architecture (Gaier & Ha, 2019).
Our work also relates to recent papers investigating properties of random CNNs, such as Ulyanov et al.
(2020) which observes that random CNNs already seem to capture some natural image statistics required for tasks such as de-noising and inpainting.
T HEORETICAL R ESULTS
We start by introducing some of our notation.
The rest of it follows Goodfellow et al.
(2016) with minor modiﬁcations, so we defer a full description to Section 4.1.
Given n ∈ N , we denote the set { 1 , . . . , n } by [ n ] .
The symbol ∗ represents the convolution operation, (cid:12) represents the element-wise (Hadamard) product, and σ represents ReLU activation function.
Finally, the notation (cid:107)·(cid:107) 1 refers to the sum of the absolute values of each entry in a tensor while (cid:107)·(cid:107) max denotes the maximum norm: the maximum among
We restrict our setting to convolutional neural networks f : [0 , 1] D × D × c 0 → R D × D × c
[Equation]
of the form
[Equation]
where K i ∈ R d i × d i × c
i − 1 × c i , and the convolutions have no bias and are suitably padded with zeros.
The restrictions on tensor sizes and the exclusion of bias terms 1 aim only to ease presentation.
Our initial goal is to approximate a convolution with a single kernel, as depicted in Figure 1, using convolutions with (pruned) random kernels.
We achieve this by the means of the structure presented in Figure 2, using two convolutions with random tensors.
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
Lemma 1 asserts that, with high probability, we can prune this structure to approximate the output of a convolution with any given kernel as long as the amount of kernels is large enough.
[Figure]
[Equation]
The elements of the set U = (cid:8) U (1) , . .
, U ( n ) (cid:9) and V are random tensors.
Notice that the intermediate tensor U ∗ X has size D × D × n and yet ﬁnal output is a D × D matrix.
Lemma 1 (Single kernel) .
Let D, d, c, n ∈ N , and ε, C ∈ R > 0 , where n ≥ C log d 2 cε , U ∈ R
d
×
d × c × n , V ∈ R 1 × 1 × n × 1 , and S ∈ { 0 , 1 } d × d × c × n .
Let us further assume that the entries of U and V are i.i.d.
random variables following a Unif([ − 1 , 1]) distribution, and deﬁne g ( X )
= V ∗ σ ( U ∗ X ) and g S ( X ) =
V ∗ σ (( U (cid:12) S ) ∗ X ) thus yielding a random CNN g : [0 , 1] D × D × c → R D × D × 1 and its pruned version g S w.r.t.
the mask S . Then, we can choose constant C independently from other parameters so that, with probability at least 1 − ε , for all K ∈ [ − 1 , 1] d × d × c × 1 with (cid:107) K (cid:107) 1 ≤ 1 , there exists a pruning mask S such that
[Equation]
Proof idea.
We leverage the absence of negative entries in the input and an initial pruning of U to bypass the ReLU non-linearity.
This allows us to virtually replace the operations in g by a single convolution with a random kernel obtained by combining U and V .
Each entry of this resulting kernel is the sum of n random variables, where we can choose to include/exclude each term in the sum by choosing to keep/prune the relevant weights.
We ﬁnish the proof by applying Theorem 2 to conclude that n variables sufﬁce to approximate all entries, simultaneously, with enough precision to ensure the thesis.
We now extend Lemma 1 to an entire layer.
As before, a detailed proof is provided in Section 4.3.
Lemma 2 (Convolutional Layer) .
Let D, d, c 0 , c 1 , n ∈ N , and ε, C ∈ R > 0 , where n ≥
C c 1 log d 2 c 0 c 1 ε , U ∈ R
d × d × c 0 × n , V ∈ R 1 × 1 × n × c 1 , S ∈ { 0 , 1 } d × d × c 0 × n and T ∈ { 0 , 1 } 1 × 1 × [ n ] × [ c 1 ] .
Let us further assume that the entries of U and V are i.i.d.
random variables following a Unif([ − 1 , 1]) distribution, and deﬁne
[Equation]
thus yielding a random CNN g : [0 , 1] D × D × c 0 → R D × D × c 1 and its pruned version g T , S ( X ) w.r.t.
the masks S and T . Then, we can choose constant C independently from other parameters so that, with probability at least 1 − ε , for all K ∈ [ − 1 , 1] d × d × c 0 × c 1 with (cid:107) K (cid:107) 1 ≤ 1 , there exist masks S and T such that sup
[Equation]
Proof Idea.
The lemma follows by applying Lemma 1 to each kernel independently, so that all of them are approximated by a factor at most εc 1 ; the latter approximation allows to apply the union bound so that the desired approximation holds simultaneously for all c 1 output kernels with probability at least 1 − ε .
We are now ready to extend the previous approximation result from a single layer to the entire network, thus proving our main result.
A detailed proof is given in Section B.
Theorem 1 (Convolutional Network) .
Let D, d, c 0 , c 1 , (cid:96) ∈ N , and ε, C ∈ R > 0 .
For each i ∈ [ (cid:96) ] , let L 2 i − 1 and L 2 i be tensors with size d i × d i × c
i − 1 ×
( Cc i log c
i − 1 c i d 2 i (cid:96) min { ε,δ } ) and
1 × 1 ×
( Cc i log c
i − 1 d
i c i (cid:96) min { ε,δ } ) × c i , respectively.
Furthermore, for each i ∈ [2 (cid:96) ] , let S i ∈ { 0 , 1 } size( L i ) .
Finally, let us assume that the entries of L 1 , ..., L 2 (cid:96) are i.i.d.
random variables following a Unif([ − 1 , 1]) distribution, and deﬁne
[Equation]
]] thus yielding a random 2 (cid:96) -layer CNN g : [0 , 1] D × D × c 0 → R D
× D × c 1 and its pruned version g S [Equation] X ) w.r.t. masks S 1 , ..., S 2 (cid:96) .
Let F be the class of functions from [0 , 1] D × D × c 0 to R D × D × c (cid:96) such that, for each f ∈ F f
[Equation]
, where, for each i ∈ [ (cid:96) ] , K i ∈ [ − 1 , 1] d i × d i × c
i − 1 × c i and (cid:107) K i (cid:107) 1 ≤ 1 .
Then, we can choose constant C independently from other parameters so that, with probability at least 1 − δ , the following holds for every f ∈ F :
[Equation]
Proof Idea.
The proof leverages Lemma 2 in an analogous way to how the latter relied on Lemma 1; namely, we apply Lemma 2 by requiring an approximation factor that guarantees, with sufﬁcient probability, that a suitable approximation is reached across all layers simultaneously.
The latter requirement is responsible for the (cid:96) factor which appears in the logarithms of the dimensions of each random tensor L i .
2. D ISCUSSION ON T HEOREM
Size analysis.
For each layer, we emulate a 4-D kernel K with size d i × d i × c
i − 1 × c i with two 4-D kernels U and V with size d i × d i × c
i − 1 × n and 1 × 1 × n × c i respectively with n =
Cc i log c
i − 1 c i d 2 i (cid:96) min { ε,δ } .
Under the technical assumption c i
= O
( d 2 i c i − 1 ) for i ∈
[ (cid:96) ] , the size of V is within a constant factor of that of U , and the size of the whole random network we prune has size O ( k log k(cid:96) min { ε,δ } ) where k is the size of the network we want to approximate.
Note that this technical assumption is met for all classical convolutional networks used for image processing with a reasonably small constant in the big O notation.
We come back on this assumption below.
Limitations.
The properties of convolutional layers require stronger hypotheses in Theoremwhen compared with the results for dense layers Malach et al.
(2020) or Pensia et al.
First, we require non-negative inputs for all layers, however, since the output of the ReLU function is never negative, this restriction is only relevant for the input of the ﬁrst layer.
The mentioned works avoid this restriction by exploiting the identity a = σ ( a ) − σ ( − a ) to deal separately with the positive and negative entries.
The fact that each entry of the output of a convolution is affected by potentially multiple input entries prevents us from employing a similar strategy.
Nonetheless, we remark that, while this is a relevant theoretical indication of the challenges imposed by the operation of convolution, in practice the inclusion of biases sufﬁces to easily convert any CNN with a domain including negative values into an equivalent CNN that takes only non-negative inputs.
Finally, the possibly multidimensional entries of convolutions also motivates the restriction on the norm of the target weight tensors in terms of the 1-norm.
Generalizations.
For the sake of simplicity, we state and prove Theorem 1 in a restricted setting.
It is worth remarking a series of generalizations that can be obtained at the mere cost of making the proofs more technically involved.
We defer a more complete treatment of such extensions to a journal version of this work.
First, the proof could consider also other parameters, such as stride, padding, average pooling and other operations that can be seen as convolutions.
Moreover, we could consider more general convolutions, not necessarily 2-D, operating on tensors of any sufﬁciently large dimension.
In particular, it is not necessary to assume that V has size 1 × 1 × n × c i in the above analysis.
Using a 5-D tensor with size d × d × c i − 1 × ( n/c i ) × c 1 for U , an appropriate convolution
U ∗ X would result in a D × D × ( n/c i ) ×
c i tensor, and we could use a 1 × 1 × ( n/c i ) ×
c i tensor for V without the need for mask T by performing in parallel c i appropriate convolutions.
Note that the size of V is then smaller than the size of U .
The technical assumption used in the above size analysis is thus not necessary to guarantee that logarithmic over-parametrization is sufﬁcient.
Finally, observe that that our results generalize to any probability distribution for the weights that contains a b -scaled Unif([ − a, a ]) for some constant a > 0 (in the sense of Deﬁnition 2 in Section
C), where the parameters a and b only impact the constants in the theorem.
E XPERIMENTS
As networks with higher parameter counts tend to be more robust to noise, we stick to the small CNN architecture used by Pensia et al.
(2020), namely LeNet5 (LeCun et al., 1989a).
We conduct our experiments by ﬁrst training a LeNet5 with ReLU activations to 98.99% test accuracy on MNIST dataset (Lecun et al., 1998) and, to avoid well-known limitations of the MNIST dataset (in particular its large number of zero entries), we also trained it on the Fashion-MNIST dataset (Xiao et al., 2017) to 89.12% test accuracy.
We adopted Kaiming Uniform (He et al., 2015) for weight initialization, a batch size of 64 and trained for 50 epochs using ADAM optimizer (Kingma & Ba, 2015) with learning rate of 0.001, exponential decay of 0.9 and momentum estimate of 0.999, the default values in Flux.jl (Innes et al., 2018) machine learning library.
Once the network is trained we change its weights for a random subset sum approximation of them.
More precisely, for each weight w we sample x from Unif([ − 1 , 1] n ) and use Gurobi optimization software (Gurobi Optimization, LLC, 2021) to solve the mixed-integer program
[Equation]
where n is the sample size.
Solving this subset sum problem with n = 30 for the 2572 parameters in the convolutional layers of LetNet takes around 1 hour on 32 cores of a Intel® Xeon® Gold
CPU @ 2.60GHz.
[Figure]
Figure 3 shows the accuracy of the approximation for different sample sizes.
We start to obtain good approximations (error smaller than 10 − 2 ) from sample sizes around 15-20.
Also, when comparing to the weights obtained for MNIST and for Fashion-MNIST, we have better approximations for the smaller sample sizes for MNIST.
We believe this is due to the fact that the training on FashionMNIST resulted in ﬁlters with larger weights (up to a factor 2, roughly), since a larger sample size is necessary to approximate a larger interval of values (see the statement of Lueker (1998,
Corollary 3.3) in Appendix C).
The high precision in the approximation of most weights leads to negligible change in the accuracy of the network.
For this reason, we focus on studying the error at the output of the convolutional section of LeNet5, right before the ﬂattening.
Also, at this point the activation tensor has dimension 7 × 7 × 16 as opposed to the vector of size 10 at the end of LeNet5.
[Figure]
Figure 4 shows the maximum relative error for all approximated outputs compared to original ones.
The relative error of the output for an input image is computed as the maximum activation error divided by the maximum original activation (both maxima are taken over all 7 × 7 × 16 activations).
Once again, MNIST leads to better precision for the smaller sample sizes.
This can be explained by the fact that weights are better approximated in that range with MNIST as seen in Figure 3.
In both cases, we get a relative error close to 10 − 3 with sample size 20, and even better with larger sample sizes.
Within the settings of Theorem 1, this corresponds to expanding the convolutional portion of the network by a factor of, roughly, 30 if we take into account kernel sizes and number of channels.
Note that this high precision is achieved even though the trained weights do not satisfy the norm restrictions of Theorem 1.
Indeed, as we do not use any explicit regularization, the 1-norms of the kernels obtained are quite high (from 50 to 15000 roughly for both datasets).
T ECHNICAL A NALYSIS 4. N OTATION AND C ONVENTIONS
We employ the notation from Goodfellow et al.
(2016) with minor adjustments.
For selfcontainment, we detail below not only our adaptations but all the relevant notation inherited from the textbook.
We identify the type of the objects with the font used for their symbols.
This applies to scalars (e.g. x ), to vectors (e.g. x ) and its entries (e.g. x i ), to tensors (e.g. X ) and its entries (e.g. X i,j,k ), and to sets (e.g. X ).
We denote slices of tensors by indexing it with colons.
For example, the expression X : , : ,i represents a 2-D slice of a 3-D tensor.
Finally, we refer to the axis of a 4-D tensors as rows , columns , channels , and kernels (a.k.a. ﬁlters), in this order.
This work considers explicitly only 2-dimensional convolutions with multiple channels, multiple kernels and enough zero-padding to preserve the output size.
However, as we discuss in Section 2.1, our results can be generalized in many ways.
Deﬁnition 1 (Convolution) .
Given tensors K ∈ R d × d × c and X ∈ R D
× D × c , the 2-dimensional discrete convolution between K and X is the D × D matrix with entries given by
[Equation]
for i, j ∈
[ D ] × [ D ] .
where X is suitably zero-padded.
Similarly, if K ∈ R d × d × c 0 × c 1 is a 4-dimensional tensor, we deﬁne
K ∗ X as the D × D × c 1 tensor with entries
[Equation]
Notice that the output of a convolution with a 4-D kernel can also be can be expressed in terms of the 3-D case using tensor slices.
For instance, the last equality in the previous deﬁnition can be rephrased as
[Equation]
4. S INGLE K ERNEL A PPROXIMATION (P ROOF OF L EMMA 1)
Our ﬁrst goal is to bypass the non-linearity so we can combine the two convolutions in g ( X ) =
V ∗ σ ( U ∗ X ) into a single one.
Given that the activation function under consideration is the ReLU, it sufﬁces to ensure that its input has no negative entry.
Hence, we prune all negative entries of U , obtaining the tensor U + = max { 0 , U } , where the maximum is applied entry-wise.
Since, by hypothesis, the entries of the input X are non-negative, it follows that the entries of the tensor U + ∗ X are also non-negative.
Therefore,
[Equation]
We now look at the ﬁrst convolution on the right side of Equation 1.
By Deﬁnition (1), we have
[Equation]
The equation above shows that performing V ∗ ( U + ∗ X ) is equivalent to performing a single convolution between X and
a tensor L ∈ R d × d × c × 1 whose coordinates are given by
[Equation]
This reveals a RSS conﬁguration where we can choose to include/exclude each value V 1 , 1 ,t, 1 · U + i,j,k,t in the sum by choosing to keep/prune U
+ i,j,k,t .
Since Equation 1 continues to hold after further pruning U + , we ﬁnish our proof by doing exactly that: we leverage Theorem 2 to ensure that, with high probability, we can solve this RSS problem for each entry of L to approximate the respective entry of K .
To see that we can apply Theorem 2 in this setting, for ε (cid:48) > 0 , i, j ∈ [ d ] , and k ∈ [ c ] , deﬁne the event E i,j,k,ε (cid:48) to be the event (cid:110)
[Equation]
We now use the RSS result Lueker (1998, Corollary 3.3) (Theorem 2 in Section C) to show that there exists constants a, b such that
[Equation]
It is not hard to show that, since ( V 1 , 1 ,t, 1 ) 1 ≤ t ≤ n and ( U + i,j,k,t ) 1 ≤ t ≤ n are i.i.d.
Unif([ − 1 , 1]) random variables, then the value of the density of ( V 1 , 1 ,t, 1 · U + i,j,k,t ) 1 ≤ t ≤ n is at least log on [ − 1 / 2 , 1 / 2] , and it thus contains a log22 -scaled Unif([ − 1 / 2 , 1 / 2])
(see Lemma 3 in Section C for details).
In particular, setting X = V 1 , 1 ,t, 1 · U + i,j,k,t , we have that µ − =
E ( 1 X ≤ 0 X ) ≤ − log28 < − 1 / 16 and µ + = E ( 1 X> 0 X ) ≥ log28 > 1 / 16 .
We can thus apply Theorem 2 with ξ = 1 / 32 : there exist constants a, b > 0 such that the expected value of the [ − n/ 32 , n/ 32] -subset-sum gap for ( V 1 , 1 ,t, 1 · U
+ i,j,k,t ) 1 ≤ t ≤ n is at most ae − bn , that is
[Equation]
Assuming n ≥ 32 , Markov’s inequality yields P ( E i,j,k,ε (cid:48) ) ≤ ae − bn ε (cid:48) .
Setting ε (cid:48)
= εd 2 c and C
=b + log a b , and supposing w.l.o.g.
ε (cid:48) <
e − 1 , the condition n ≥ C log 1 ε (cid:48) implies bn ≥ 2 log ε (cid:48)
+log a and ae − bn ε (cid:48) <
ε (cid:48) , and we get
[Equation]
Now deﬁne the simultaneous event E ε (cid:48)
=
(cid:84) i,j,k E i,j,k,ε (cid:48) .
By a union bound over the inequality above for i, j ∈ [ d ] , k ∈ [ c ] , we have
[Equation]
Finally, conditioning on E εd 2 c , it holds that
[Equation]
[Equation]
where (a) follows from Equation 1, (b) from the distributivity of the convolution operation, (c) from
Proposition 1, (d) from the fact that X ∈ [0 , 1] D × D × 1 , (e) from the inequality (cid:107) x (cid:107) 1 ≤ m (cid:107) x (cid:107) max for x ∈ R m , and (f) from the deﬁnition of E εcd 2 .
4. C ONVOLUTIONAL L AYER A PPROXIMATION (P ROOF OF L EMMA 2)
Our goal is to approximate K ∗ X with ( V (cid:12) T ) ∗ σ [( U (cid:12) S ) ∗ X ] where T and S are binary masks.
For (cid:96) ∈ [ c 1 ] let K ( (cid:96) ) be K ’s (cid:96) -th kernel.
That is,
[Equation]
Notice that K ∗ X is the concatenation along the third dimension of each K ( (cid:96) )
∗ X : for (cid:96) ∈
[ c 1 ] , we have ( K ∗ X ) : , : ,(cid:96) =
K ( (cid:96) ) ∗ X .
We ﬁx T a priori to be the block diagonal matrix B with entries given by B 1 , 1 ,t,(cid:96)
=
1 ( (cid:96) − 1) n (cid:48) <
t ≤ (cid:96)n (cid:48) for t, (cid:96) ∈ [ n ] × [ c 1 ] , where n (cid:48)
=
n/c 1 .
In the rest of the proof, we show how to choose S , based on
U and V , in order to approximate the kernels K ( (cid:96) ) .
We perform the approximation of each K ( (cid:96) ) using different sections of the tensors.
To this end, for (cid:96) ∈ [ c 1 ] , let
[Equation]
and
[Equation]
As we did in the proof of Lemma 1, we perform an initial pruning on U by restricting S to the space of masks that prune all of its negative entries.
This allows us to ignore the ReLU activation and conclude that
[Equation]
For (cid:96) ∈ [ c 1 ] and ε (cid:48) > 0 , deﬁne the event E (cid:96),ε (cid:48) to be the event (cid:110) sup
K
[Equation]
inf S (cid:96)
[Equation]
[Equation] [Equation]
(cid:48) sup
X ∈ [0 , 1]
[Equation]
c(cid:13) (cid:13) (cid:13) K
( (cid:96) )
∗ X − V (cid:96) ∗ σ [( U (cid:96) (cid:12) S (cid:96) ) ∗ X ] (cid:13) (cid:13) (cid:13) max < ε (cid:48) (cid:111) .
Now deﬁne the event E ε (cid:48) =
(cid:84) (cid:96) E (cid:96),ε (cid:48) and consider the event E ε/c 1 .
Since n (cid:48) = n/c 1 = C log d 2 c 0 ε/c 1 , for each (cid:96) ∈ [ c 1 ] , thanks to Lemma 1 we have P ( E (cid:96),ε/c 1 ) ≥ 1 − ε/c 1 , which implies P ( E ε/c 1 ) ≥− ε.
Finally, conditioning on E ε/c 1 and using the fact that the output channels of a convolutional layer are calculated independently, we conclude sup
[Equation]
inf
[Equation]
sup
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
(cid:12) S
[Equation]
(cid:13)(cid:13)(cid:13) max ≤ sup
[Equation]
inf
[Equation]
sup
X ∈ [0 , 1] D × D × c(cid:13)
(cid:13)(cid:13) K ∗ X − ( V (cid:12) B ) ∗ σ [( U (cid:12) S ) ∗ X ] (cid:13)(cid:13)(cid:13) max = max (cid:96) ∈ [ c 1 ] sup
K ( (cid:96) ) ∈ [ − 1 , 1] d × d × cinf S (cid:96) ∈{ 0 , 1 } d × d × c 0 × n (cid:48) sup
X ∈ [0 , 1] D
× D × c(cid:13)(cid:13)(cid:13) K
( (cid:96) )
∗ X − V (cid:96) ∗ σ [( U (cid:96) (cid:12) S (cid:96) ) ∗ X ] (cid:13)(cid:13)(cid:13) max < ε.
R EPRODUCIBILITY S TATEMENT
We made our best effort to describe in detail all the relevant parameters for Section 3.
We also took care to initialize the random number generator not only for Julia but also for Gurobi, so that the experiments can be reproduced using the source code available at https://shared03.
opsone-cloud.ch/index.php/s/mWPK8AMRrrdJr8E .
The latter also include a BSON serialized version of our trained models and solutions to RSS instances, that can readily be imported with Julia.
A B OUND ON THE N ORM OF A C ONVOLUTION
The following proposition can be seen as a special case of Young’s Convolution Inequality for functions over Z n where the norms in the inequality are the (cid:96) 1 and (cid:96) ∞ norms.
Proposition 1 (Tensor Convolution Inequality) .
Given K ∈ R d × d × c 0 × c 1 and X ∈ R D
× D × c 0 , we have
[Equation]
Proof.
We have
[Equation] [Equation]
i (cid:48)
[Equation]
c ] (cid:12)(cid:12)
[Equation]
(cid:48)
[Equation]
(cid:48)
[Equation]
i
[Equation]
i (cid:48)
[Equation]
(cid:48)
[Equation]
(cid:12)(cid:12)
[Equation]
i
[Equation]
(cid:88) i (cid:48)
[Equation]
j (cid:48) ,k (cid:12)(cid:12)
[Equation]
i (cid:48) ,j (cid:48) ,k
[Equation]
(cid:96) (cid:12)(cid:12) (cid:107)
[Equation]
(cid:107) max
[Equation]
(cid:18) (cid:88)
[Equation]
(cid:48)
[Equation]
(cid:48)
[Equation]
|
[Equation]
i (cid:48) ,j (cid:48) ,k
[Equation]
(cid:96) | (cid:19) (cid:107) X (cid:107) max
[Equation]
(cid:107)
[Equation]
(cid:107) max .
B CNN A PPROXIMATION (P ROOF OF T HEOREM 1)
Since Lemma 2 provide bounds in terms of the output of a layer, the study of the propagation of this error through the network is mostly independent of the layer type.
Hence, the next proof follows the structure of Pensia et al.
(2020, Theorem 1), where our Lemma 2 assumes the role of their Lemma 3, and where we leverage our Proposition 1 in order to address the problem of bounding the maximum norm of a convolution.
Proof (of Theorem 1).
For the sake of brevity, in the proof we denote the max -norm simply by (cid:107)·(cid:107) .
Let X i be the input of the i -th layer of the network f .
Thus,
X 1 =
X , 2.
X i +
=
σ ( K i ∗ X i ) for 1 ≤ i ≤ (cid:96) − 1 and 3.
f ( X ) =
K (cid:96) ∗ X (cid:96) .
By applying Lemma 2 to each layer, we choose masks S 2 i and S 2 i − 1 so that
[Equation]
with probability at least 1 − ε 2 (cid:96) .
Since the ReLU function is 1-Lipschitz with respect to the max norm, the above event implies the following for all i ∈ [ (cid:96) − 1] :
[Equation]
By a union bound, with probability 1 − ε , Equation 3 and Equation 4 hold for all layers simultaneously.
In the rest of the proof, we implicitly condition on the latter event.
For any ﬁxed function f , let g be the pruned network constructed layer-wise, by pruning with binary masks which satisfy Equation 3 and Equation 4.
Let these pruned tensors be ˜ L i , and let ˜ X
i be the input to the (2 i − 1) -th layer of g .
We note that ˜ X i satisﬁes the the recurrence relation
˜ X
1 = X , 2.
˜
X i +1 =
σ (˜ L 2 i ∗ σ (˜ L 2 i − 1 ∗ X i )) for 1 ≤ i ≤ (cid:96) − 1 .
Because (cid:107) X (cid:107) ≤ 1 , Equation 4 implies that (cid:107) ˜
X
i (cid:107) ≤
[Equation]
1 + ε
[Equation]
(cid:17) i − 1 .
To see this, note that
Equationimplies, for 1 ≤ i ≤ (cid:96) − 1 ,
[Equation]
thus
[Equation]
By the reverse triangle inequality, the last inequality implies
[Equation]
Applying this inequality recursively, we get that (cid:107) ˜
X
i
[Equation]
≤
[Equation]
1+
[Equation]
i − 1 for 1 ≤ i ≤ (cid:96) − 1 .
This allows us to bound the error between X i and ˜ X i .
For 1 ≤ i ≤ (cid:96) − 1 , we have
[Equation] [Equation]
where for the last inequality we use Equation 3.
Unrolling Equation
we get
[Equation]
Finally
[Equation]
this last inequality leads
[Equation]
with probability at least 1 − ε , to
[Equation]
[Table]
[Equation]
where the last inequality holds because ε < 1 .
Replacing ε in this proof with min { ε, δ } concludes the proof of the theorem.
C R ANDOM S UBSET -S UM T HEOREM
For the sake of completeness, in this section we recall a result by Lueker (1998) together with the necessary deﬁnitions.
Deﬁnition 2.
Given two positive constants a and b , we say that a distribution with density f contains a b -scaled Unif([ − a, a ]) distribution if for each x ∈ [ − a, a ] it holds f ( x ) ≥ b .
We simply say that a distribution F contains a uniform distribution if there exist positive constants a and b such that F contains a b -scaled Unif([ − a, a ]) distribution.
The following is a weaker version of Corollary 1 in the Appendix of Pensia et al.
Lemma 3.
Let X
and X 2 be two independent random variables following a Unif([ − 1 , 1]) distribution.
Then X 1 · X 2 contains a log22 -scaled Unif([ − 12 , 12 ]) distribution.
We say that z is 2 η -subsetsum-approximated with S
= { X 1 , ..., X n } if there exists a subset I z ⊆
[ n ] s.t. | (cid:80) i ∈ I
z X i − z
| ≤ 2 η .
Deﬁnition 3.
The [ a, b ] -subset-sum gap of S =
{ X 1 , ..., X n } is the smallest value of η such that each z ∈
[ a, b ] , can be 2 η -subsetsum-approximated with S .
Theorem 2 (Corollary 3.3 in Lueker (1998)) .
Let S
=
{ X 1 , ..., X n } be n i.i.d.
bounded random variables and ξ > 0 any constant.
Suppose that the distribution of X 1 contains a uniform distribution.
Let µ − =
E [ 1 X ≤ 0 X ] , µ + = E
[ 1 X ≥ 0 X ] , µ abs =
E [ | X | ] = µ + − µ − .
The expected value of the [( µ − + ξ ) n, ( µ + − ξ ) n ] -subset-sum gap of S is exponentially small w.r.t.
n .