Paper: Generalization Bounds for Magnitude-Based Pruning via Sparse Matrix Sketching (ICLR-2024)

Reviewer comment (Theory):
Theorem 5.2 requires an assumption $\gamma$ to be no smaller than a number. It seems that this number would be very large since it has exponential dependency on $L$, and $\Gamma_l$ is also large according to the definition. If $\gamma$ is large, then $\hat{R}_\gamma$ would be very large.

Author response:
A. When $\gamma$ is large, $\hat{R}_{\gamma}$ is smaller as the acceptable margin is smaller. This is identical to the case of Arora et al. This is not an unusual statement unique to this paper.

Reviewer comment (Theory):
In Theorem 5.2, the dependency of the generalization bound on $d$ seems to be $d^{3/4}$. If I understand correctly, if we do not use the matrix sketching, the dependency would be $d$. Therefore, the improvement is $d^{1/4}$, which seems not to be quite significant. As stated in the conclusion, the analysis still leads to a vacuous generalization bound.

Author response:
A. This is not correct. Here, $d$ is a coefficient of the strength of the pruning. The dependence on $d$ is roughly $\frac{1}{d^{.25}}$ so as $d$ increases and prunes more, the generalization bound gets tighter and tighter. In every other generalization bound, there is no mention of this $d$ and they do not get tighter as you prune more. $d$ is not the dimension of the matrix. For heavily pruned matrices, our bounds are significantly tighter.

Reviewer comment (Theory):
Section 3.1: it seems that the definition of $\hat{R}_\gamma$ is not correct, i.e., $\geq \gamma$ should be $\leq\gamma$?

Author response:
A. Yes, you are correct. This has been fixed in the updated version.

Reviewer comment (Theory):
Theorem 3.1 only shows the existence of a $A\in\mathcal{A}$. How to find such a $A$?

Author response:
A. Lemma 4.1 proves that using Algorithm 1 to prune a model finds such an $A$ that achieves a difference in output with small margin.

Reviewer comment (Writing):
Lemma 4.1: $\Gamma_l$ is used without giving its meaning.

Author response:
A. We apologize for this. This was put into the appendix for space, but we will move it back for the Camera Ready Version.

Reviewer comment (Theory):
The assumptions, especially those concerning Gaussian weight distribution, lack adequate empirical validation.

Author response:
Thank you for the responses! We now better understand your questions. We did not show this graphic since this is a well-known empirical phenomenon. For example, both Figure 7 by Han et al. 2015 and Figure 1 by Qian and Klabjan 2021 run experiments on the weight distributions of Feed-Forward Networks. After training, they *do* see that the distributions are indeed Gaussian approximately. They plot this graphical verification. We have reproduced their experiments and *added them to the newly uploaded version in Figure 1*. Please look at our figures. I believe these are what you are looking for. I appreciate all the help and quick response! Again, if this alleviates your concerns, we would greatly appreciate an increase in scores!

Xin Qian and Diego Klabjan. A probabilistic approach to neural network pruning. CoRR,
abs/2105.10065, 2021. URL https://arxiv.org/abs/2105.10065.

Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for
efficient neural network. Advances in neural information processing systems, 28, 2015.

Reviewer comment (Experiments):
Considering the assumed Gaussian distribution of weights, it is worth exploring whether better discretization methods exist.

Author response:
A. We believe this reviewer has misinterpreted the point of discretization. We only discretize so that there are a limited number of parameterizations so that we can use the compression framework. Different discretization techniques are unlikely to improve the bounds significantly.

Reviewer comment (Evaluation):
The empirical verification of bounds in Fig~3 displays bound values that are excessively large, impairing their practical utility. It would be advisable to offer theoretical comparisons rather than relying solely on empirical evidence.

Author response:
A. As stated in this paper, all of the baselines do not capture the sparsity and do not incorporate sparsity into their generalization bounds. This means that our bounds are significantly tighter when the models are heavily pruned.

Reviewer comment (Writing):
This paper could significantly benefit from improved writing clarity. The current version may be confusing for readers.

Author response:
A. We thank this reviewer for their comments on clarity. We have taken the time to improve these elements of our paper.

Reviewer comment (Theory):
The proof follows a structure of bounding the layerwise error and then accounting for discretization of the parameters in order to conform with the setting of Theorem 3.1 based on the proof of Arora et al. [1]. However, the need for discretization has not been motivated clearly in the paper and can benefit from additional explanation regarding the same, which will make the proof easier to follow.

Author response:
We apologize that this has not been made clear. We discretize so that the number of different parameterizations is finite. We have made this clearer in the uploaded new version.

Reviewer comment (Theory):
The magnitude pruning algorithm assumed by the authors uses a Bernoulli based construction of the sparse mask, however, in practice magnitude pruning is done based on sorting the parameters in each layer (or the entire network). Does the $(j_r, j_c)$ structure in each layer hold then? For pruning methods like Iterative Magnitude Pruning, larger layers are known to have dead neurons (corresponding to all zeros in a row). This problem is especially amplified in a CNN where IMP sets most channels to zero while having very few dense channels. The current proof seems to be unable to handle these situations and how will the assumption change for CNNs.

Author response:
This is not true. There are different forms of Magnitude Based pruning. There are many forms of Magnitude Based pruning. For example, both random and threshold based Magnitude Based Pruning are discussed in Xian and Klabjan. This is not an abnormal form of pruning. However, our analysis can be slightly altered to prove for threshold based pruning as well. We only need that the pruning does not induce large error, which is known, and that there does not exist rows with many nonzeros, which we can also demonstrate. I am unsure what the second part of this weakness means. We only require that there are no rows with many nonzeros, i.e. distributed. Our analysis will hold if there are rows with no nonzeros.

Reviewer comment (Theory):
How will the generalization bounds change for different pruning methods, for eg: random pruning or other gradient based pruning criterions. Would random be strictly worse or similar? Insights on different pruning methods will help understand the relevance of the proposed bounds for different pruning criteria.

Author response:
As long as one can prove that the pruning does not induce large error and that the sparsity induced is distributed, our analysis will hold. In fact, our method is not largely dependent on the type of pruning used, only that it is evenly distributed and does not induce large errors. This makes our results much more generally applicable.

Reviewer comment (Writing):
The abstract is confusing as it seems to focus on ML in general and never mentions the context — e.g. neural networks.

Author response:
A. Magnitude-based Pruning is a very specific term as it applies to neural networks. We hope this clears up the confusion.

Reviewer comment (Writing):
Overall, the paper has lots of typos and grammatical errors and would need significant proofing before final publication. The introduction in particular is very vague and difficult to read, and it is unclear what the paper is actually doing. For example, the term "sparse matrix sketching" is used without definition or citation when introduced. It is unclear whether this refers to random projection with sparse random matrices or something else, and how it is being used (e.g., as an alternative to magnitude-based pruning, as an analysis tool for magnitude-based pruning, or something else). While some of these questions become clearer later, the introduction needs to be significantly reworked to be more concrete.

Author response:
A. Matrix Sketching is a well-known technique in matrix theory. I have added the relevant citations in the main text. We use Matrix Sketching to represent the space of sparse matrices in the set of smaller dense matrices to create tight generalization bounds from “Specifically, we show that our set of pruned matrices can be efficiently represented in the space of dense matrices of much smaller dimensions via the Sparse Matrix Sketching.”  We have fixed this in the uploaded new version.

Reviewer comment (Writing):
In Section 3.1, it is unclear what the difference is between R(M) and R₀(M). Both notations seem to denote the case when γ = 0 (i.e., no margin).

Author response:
A. We state in the paper that “The population risk $R(M)$ is obtained as a special case of $R_{\gamma}(M)$ by setting $\gamma = 0.” The fixed string is mentioned as we are repeating the lemma exactly from the work of Arora, which has the flexibility to use such fixed strings, but we do not use them. We have fixed this in the newly uploaded version. Moreover, yes, this is a small typo. We have dropped the string notation, as mentioned in the newly uploaded version.

Reviewer comment (Theory):
The role of the ‘fixed string’ s in Definition 3.1 is unclear. It is not used in either the definition of G_{A,s} nor in the definition of compressibility, and it seems that dropping it would not change either the definition or Theorem 3.1. Relatedly, G_{A,s} is defined differently in Definition 3.1 and Theorem 3.1, which is presumed to be a typo.

Author response:
A. We state in the paper that “The population risk $R(M)$ is obtained as a special case of $R_{\gamma}(M)$ by setting $\gamma = 0.” The fixed string is mentioned as we are repeating the lemma exactly from the work of Arora, which has the flexibility to use such fixed strings, but we do not use them. We have fixed this in the newly uploaded version. Moreover, yes, this is a small typo. We have dropped the string notation, as mentioned in the newly uploaded version.

Reviewer comment (Theory):
In Assumption 3.1, it is unclear whether the weights are independent of each other, or merely marginally Gaussian but potentially correlated. Later in the paper independence is needed, but this was not clarified in the original assumption statement.

Author response:
A. We agree with this comment. We have specified the need for independence in the newly uploaded version.

Reviewer comment (Theory):
In the explanation of the proof in Section 4.1, it is unclear why the assumption of Gaussianity is needed, and why randomized pruning is necessary. The reviewer suggests that as long as the original distribution is mean 0 and symmetric, pruning all values below some threshold t would yield Δ^l with i.i.d. mean 0 entries with bounded moments, which would then allow the application of random matrix theory results.

Author response:
1. You have made a great observation. In fact, *we truly only need the weights' independence and identicality and a zero mean distribution. Any distribution satisfying this satisfies our analysis. For example, the uniform distribution from Qian and Klabjan would have also been simple to analyze and would work with our current analysis. We take the Gaussian assumption *only* for the reason that, empirically, weights tend to look Gaussian, not uniform. We can very simply extend our analysis to work on different distributions. If this reviewer believes it is better, we can remove the Gaussian assumption with an IID and zero mean and bounded variance distribution for the Camera Ready Version. It is a very simple change and does not affect our analysis at all.  

2.  Our analysis *can* support deterministic pruning. You would want to assume the weights are i.i.d. random and zero mean. If we adopt the pruning analysis for deterministic pruning from Qian and Klabjan for deterministic pruning, using our generalization bound with deterministic pruning is easy. The only difference would be the proof that pruning does not induce large errors, but that is also very simple to prove. Extending to different forms of pruning is simple, hence why we extend our analysis for Lottery Tickets. If this reviewer wants, we can extend our analysis to deterministic pruning in the appendix for the camera ready. It is very simple to do.

Reviewer comment (Theory):
The reviewer cannot make intuitive sense of Lemma 4.1. The bound seems questionable as it does not depend on any norm bound for the original input points. For example, with one linear layer and arbitrary input scaling by a large constant C, both x^L and \hat{x}^L would scale by C, and so would the error—implying the error bound must depend on input scaling.

Author response:
A. You are correct; there is a small typo in the proof. The norm of the input should scale with this error. We have corrected this in the newly uploaded version. This is a minor change. The latter part about the significant error compared to $\|x^L\|_2$ is invalid. As the error induced by pruning decreases, epsilon decreases, and the error decreases. The main point of this proof is that the error scales linearly with epsilon. This margin is not tight in practice, as seen in Figure 1a. This looseness is normal in margin bounds and suffices for our generalization bounds. This looseness and style of analysis are similar to the results of Arora et al. 2018. While one can trivially reduce this dependence using noise stability, such as Arora et al. 2018, this only slightly improves this bound.

Reviewer comment (Presentation):
In Figure 1, it is unclear whether "Lemma B.5" is meant to refer to Lemma 4.1.

Author response:
A. Yes, there are many ways to prove that pruning does not induce too much error. We agree with this reviewer that there are many ways to do this. Again, analyzing the spectral norm of the difference between the original and pruned matrix is a straightforward way to analyze this and follows some of the analysis of the Probalistic pruning change. We do not believe alternative methods to proving similar bounds make this standard way any less strong. Random Matrix Theory is useful for demonstrating that the spectral norm of the difference between a pruned matrix and the original matrix is relatively small. There are many ways to show this. However, trivially applying Markov’s Inequality results in a very poor bound. Using it results in an upper bound of $\|\hat{\mathbf{A}} - \mathbf{A}\|_2 \leq c\|\mathbf{A}\|_2$ where $c$ is some large constant. This bound is weaker than ours.

Reviewer comment (Theory):
The reviewer asks where random matrix theory is actually applied in Lemma 4.1. They note that for a matrix with random mean 0 entries, the spectral norm can be bounded by the Frobenius norm, which by Markov’s inequality yields a bound of order n*E[Var] with good probability, while random matrix theory gives bounds of order √n. They question how Lemma 4.1 would change if this trivial Frobenius norm bound were used instead.

Author response:
3. Lemma 4.1 would change to have a dependence on $d_1^ld_2^l \frac{d^\frac{3}{2}}{(d+2)^{\frac{3}{2}}}$ using your analysis. This bound is much looser than ours since ours is dependent on dimension roughly $\sqrt{d_1^l} + \sqrt{d_2^l}$. This is much better. We hope this tightness justifies our style of analysis.

Reviewer comment (Novelty):
In Section 5, it is unclear whether the Sparse Matrix Sketching idea is novel to this paper or taken from the existing literature, as no citations are given.

Author response:
A. In Theorem 5.1, we cite Dasarthy et al. Matrix Sketching is an idea from existing papers, but its use for generalization bounds is indeed novel.

Reviewer comment (Theory):
Considering a p × p matrix with j*p nonzeros, representing it in sparse matrix format requires j*p values and j*p indices (~ log p bits each), giving roughly j*p*log p parameters. The reviewer does not see how sparse matrix sketching improves over this simple argument, and suggests that the proposed approach loses a log factor since (\sqrt{jp} log p)^2 = jp log² p parameters.

Author response:
A. This is an equivalent argument to the combinatorial argument in the appendix. We agree with this reviewer. We thank you for this catch. Indeed, the argument in the appendix is just as tight as matrix sketching. However, the argument in the appendix is still strong and improves on existing bounds. We hope this is enough in your opinion for acceptance.

Reviewer comment (Writing):
In Lemma 5.1, the variable λ_l is undefined, making the theorem impossible to interpret; it could potentially be as large as max(d₁,d₂). The definition of χ is also unclear.

Author response:
A. As stated, Lemma 5.1 holds for any $\lambda$ in $\mathbb{R}$. Moreover, $\chi$ is exactly as stated in Lemma 5.1.

Reviewer comment (Theory):
In Theorem 5.2, it states that d must be chosen to satisfy some expression, but that expression does not depend on d, leading to confusion about the statement's meaning.

Author response:
A. $\epsilon_l$ depends on d from the pruning error. This comes from Lemma 4.2.