Paper: How Sparse Can We Prune A Deep Network: A Geometric Viewpoint (ICLR-2024)

Reviewer comment (Evaluation):
For Table 3, I am confused by the margin of improvement. Could the author give a detailed illustration of your GOP. From my point of view, you use the magnitude pruning and the magnitude is determined by l1 norm. In addition you prune the network globally. However, for pruning using magnitude, using l1 or l2 norm will not alter the rank of these weights which means the remaining weights after pruning is the same. Then why your GOP can improve significantly against LTH.

Author response:
**Question 1: For Table 3, I am confused by the margin of improvement. Could the author give a detailed illustration of your GOP. From my point of view, you use the magnitude pruning and the magnitude is determined by l1 norm. In addition you prune the network globally. However, for pruning using magnitude, using l1 or l2 norm will not alter the rank of these weights which means the remaining weights after pruning is the same. Then why your GOP can improve significantly against LTH. In addition, could the authors further validate the theory on large scale dataset such ImageNet or Place365?**

A brief description of the GOP algorithm is provided in Section 2 of our paper. Basically, GOP runs as follows: we modify the original loss function with an $l_1$ regularization term and then perform network training, whose final outcome of weight will be pruned based on the magnitude.

As for the mentioned $l_1$ norm against $l_2$ norm, indeed they make no difference when conducting the magnitude-based pruning. The key enabler of our algorithm as compared with the LTH lies in that the pruning ratio in GOP is determined by formulas, in a global and one-shot way, while the pruning ratio in LTH is iterative and predetermined in nature, easily causing performance loss due to that important parameters might be removed inappropriately during the pruning iterations.

As for the large-scale dataset, due to the limited computational resources in our lab, experiments based on the ImageNet dataset is of difficulty for now. Therefore, we chose to use tiny version of Imagenet for validation, which shows that our theory aligns well with experimental outcomes. Relevant results can be found in Section 5. If this paper was finally accepted and experiments were finished then, we will provide the  validation results on Imagenet  in the final version.

Reviewer comment (Experiments):
Could the authors further validate the theory on large scale dataset such ImageNet or Place365?

Author response:
**Question 1: For Table 3, I am confused by the margin of improvement. Could the author give a detailed illustration of your GOP. From my point of view, you use the magnitude pruning and the magnitude is determined by l1 norm. In addition you prune the network globally. However, for pruning using magnitude, using l1 or l2 norm will not alter the rank of these weights which means the remaining weights after pruning is the same. Then why your GOP can improve significantly against LTH. In addition, could the authors further validate the theory on large scale dataset such ImageNet or Place365?**

A brief description of the GOP algorithm is provided in Section 2 of our paper. Basically, GOP runs as follows: we modify the original loss function with an $l_1$ regularization term and then perform network training, whose final outcome of weight will be pruned based on the magnitude.

As for the mentioned $l_1$ norm against $l_2$ norm, indeed they make no difference when conducting the magnitude-based pruning. The key enabler of our algorithm as compared with the LTH lies in that the pruning ratio in GOP is determined by formulas, in a global and one-shot way, while the pruning ratio in LTH is iterative and predetermined in nature, easily causing performance loss due to that important parameters might be removed inappropriately during the pruning iterations.

As for the large-scale dataset, due to the limited computational resources in our lab, experiments based on the ImageNet dataset is of difficulty for now. Therefore, we chose to use tiny version of Imagenet for validation, which shows that our theory aligns well with experimental outcomes. Relevant results can be found in Section 5. If this paper was finally accepted and experiments were finished then, we will provide the  validation results on Imagenet  in the final version.

Reviewer comment (Theory):
For LTH, the key point is to keep the same initialization when doing training. Could the theory provide an explanation for this phenomenon?

Author response:
**Question 2: For LTH, the key point is to keep the same initialization when doing training. Could the theory provide an explanation for this phenomenon?**

Since our theory is based on global and one-shot pruning, though  it can achive better  performance than the multi-shot pruing, say LTH, currently it cannot provide explanations for phenomena in the latter. We do think this is a much worthy-to-explore question.

Reviewer comment (Writing):
For the part of Approximate Kinematics Formula from [1], a concrete introduction could be given in the paper for better understanding, for this part is the key component of this paper.

Author response:
**Question 4: For the part of Approximate Kinematics Formula from [1], a concrete introduction could be given in the paper for better understanding, for this part is the key component of this paper.**

Thanks for the suggestion. We have modified the main idea description in a more intuitive manner in the revised version. Specifically, we have provided an intuitive description of the key tool in our work, i.e., the Approximate Kinematics Formula. Please refer to the following two excerpts:

In Section 1:

> Despite the above progress, it however still remains elusive about the *fundamental limit* of network pruning while maintaining the performance. To tackle this problem, we'll take a first principle approach by imposing the sparsity constraint directly on the loss function, thus converting the original pruning limit problem to a set intersection problem, i.e., deciding whether the *$k$-sparse set* intersects with the *loss sublevel set* (i.e., the set of weights whose corresponding loss is no larger than the original loss plus  tolerance $\epsilon$)  and obtaining the smallest value of $k$.

>
> Intuitively speaking, the larger the loss sublevel set (higher complexity), the smaller the sparse set required for intersection, i.e., the network can be more sparse. To rigorously characterize the complexity of a set, by exploiting the high dimensional nature of DNNs, we are able to harness the notion of *statistical dimension* and *Gaussian width* in high dimensional convex geometry. Through this geometric perspective, it's possible for us to take advantage of the universal *concentration effect* [1] in the high-dimensional world, so as to get sharp results about the above set intersection problem. In specific, we will exploit the powerful Approximate Kinematics Formula in high-dimensional geometry, which roughly says that for two convex cones, if the sum of their statistical dimension exceeds the ambient dimension, then these two cones would intersect with probability 1, otherwise they would intersect with probability 0. We notice that a sharp phase transition emerge here, thus enabling a precise and succinct characterization of the fundamental limit of network pruning.

and in Section 2:

> To characterize the sufficient and necessary condition of the set (or cone) intersection, the Approximate kinematics Formula is a powerful and sharp result, which basically says that for   two convex cones (or generally, sets), if the sum of their statistical dimension exceeds the ambient dimension, then these two cones would intersect with probability 1, otherwise they would intersect with probability 0.

 [1] Basically, the concentration effect says that a function of a *large amount* of independent (or weakly dependent) variables tends to concentrate to its expectation value. Notable examples include the Johnson-Lindenstrauss lemma and related results in Compressive Sensing.

Reviewer comment (Theory):
Since the theoretical analysis did not include $\lambda$ in Eq.3, it brings some additional problems. For example, modern deep neural network training could be very expensive, assume we trained a model, and we calculate the theoretical pruning rate. But this pruning rate does not meet our expectation, then the only thing we can do is to adjust the $\lambda$ and retrain the model. We never know the theoretical pruning rate before the model is fully trained. As a result, the theoretical pruning rate does not give a better guidance when we want to train a model given an expected pruning rate.

Author response:
Yes, our current theoretical result can only be obtained after the training. However, we do think it is of much value. Beyond its theoretical value of the fundamental limit, we are also able to identify the most important factors that impact the pruning performance, i.e., the network flatness (namely, the trace of the Hessian matrix) and the magnitude of weight. If we take into account these factors into the training objective, we think it will offer help for sparsifying the network during training (rather than sparsifying after training).

Furthermore, we do think determining the pruning ratio before the training is a very intersting and important avenue for research.

Reviewer comment (Novelty):
The pruning method itself is not novel, it is simply $L_1$ regularized training + magnitude pruning.

Author response:
It's true that our pruning algorithm is not novel. Actually, the main purpose of our work is to characterize the fundamental limit of network pruning, thus obtaining insights on the key factors that most impact the pruning.  Our proposed pruning algorithm is a byproduct, which serves as validating our theoretical results.

Reviewer comment (Presentation):
Section 5.3 appears to be somewhat disconnected from the primary focus of the work. Given the widespread use of regularization-based pruning algorithms in the literature (e.g., [1,2]), it might be worth considering how this section better aligns with the core contributions of the paper.

Author response:
Regularization-based pruning algorithms are indeed not new. For example, in [1] the $l_2$ regularization is employed,  while in [2] the $l_0$-norm is used, however the regularization is not directly imposed on the weight, rather it is on the auxiliary variables.

To the best of our knowledge, our work is the *first* to employ the $l_1$ regularization directly on the weight and *one-shot pruning* is performed, in the spirit of first principles modeling and problem-solving.  Therefore, we think the performance of our proposed $l_1$-regularized one-shot pruning algorithm, which turns out to outperform all typical pruning algorithms, is a valuable complement, to our theoretical result. 

[1] https://openreview.net/pdf?id=o966_Is_nPA

[2] https://proceedings.neurips.cc/paper_files/paper/2019/file/4efc9e02abdab6b6166251918570a307-Paper.pdf

Reviewer comment (Theory):
In the era of Large Language Models (LLM), how do the theoretical results align with LLM pruning results [3]?

Author response:
This is an excellent question. In our opinion, our theoretical result aligns well with LLM pruning results in [3], due to the following reasons: 1) Our theory imposes no constraints on network models or loss functions, thus enabling its high universality. 2) The pruning methods in [3] also utilizes one-shot magnitude pruning, similar as ours, therefore its limit (referred as Essential sparsity in [3]) should align with our proposed fundamental limit.  

Due to that  currently we lack the required computing resources for running LLMs, we're unable to conduct experiments for validation purpose. 

[3] https://arxiv.org/abs/2306.03805