Paper: What Makes a Good Prune? Maximal Unstructured Pruning for Maximal Cosine Similarity (ICLR-2024)

Reviewer comment (Experiments):
My major concern is that pruning results on CIFAR-10 often are too easy and need more investigation at Tiny ImageNet and ImageNet scale to verify if the empirical insights translate. I would be very happy to increase my score and advocate for acceptance with the presence of ImageNet results on one or two networks (see Blalock et al., 2020 for best practices).

Author response:
Thank you for your constructive review. 

To address your main concern, we have run our pruning experiments on three networks trained on ImageNet. The results are discussed in the general comment above and detailed in the new version of the paper (specifically in sections 5.4, 5.3 and 5.2 - edits are in blue).

Reviewer comment (Experiments):
On similar lines, the networks used for CIFAR-10 are often way too overparameterized and that would be handled by experiments on ImageNet.

Author response:
Thank you for your constructive review. 

To address your main concern, we have run our pruning experiments on three networks trained on ImageNet. The results are discussed in the general comment above and detailed in the new version of the paper (specifically in sections 5.4, 5.3 and 5.2 - edits are in blue).

Reviewer comment (Experiments):
Also, the approach does 1-epoch fine-tuning. Is it just for the pruned network? Then what’s the similarity after a 1-epoch fine-tuning? Or if it does not care about cosine similarity after fine-tuning, why does it fine-tune only for 1 epoch, but not multiple times as the SOTA pruning approaches do?

Author response:
What happens to the cosine similarity during fine-tuning is an orthogonal question. We know that networks become more functionally similar to the parent during fine-tuning (5.2), and it is therefore possible that the cosine similarity increases as well. Note that as a response to reviewer U3zy, we have repeated our experiments with 10 epochs of fine-tuning (see public response to this reviewer for more details).

Reviewer comment (Experiments):
This work lacks necessary comparisons with SOTA pruning approaches, such as Weight rewinding [1], Learning rate rewinding [2], and Gradual magnitude pruning [3][4][5]. Please consider comparing it with them. Comparing with Random pruning does not provide extremely interesting information.

Author response:
The motivation behind this work was to (a) understand how and why global L1-pruning works, and (b) give some theoretical underpinnings to answer the question "By how much do I need to prune?". We are not trying to advocate that a particular pruning method is competitive with other methods for certain models. Comparison to SOTA is thus not entirely relevant. Rather, we are trying to understand the mechanisms of pruning. This is exemplified by e.g. our analysis of the rule of kurtosis (and kurtosis of kurtoses) in how much and how a network can be pruned.

Reviewer comment (Theory):
It looks to me you're computing cosine similarity of a vectorized weight vector and its pruned version, the former containing ALL weights in the network and its size would be humongous, how do you deal with that? Also, that weight vector contains weights of different DNN layers which're segregated by nonlinear activations in the network, why grouping them into one huge vector would work at all? More insights or analytical explanations are needed here.

Author response:
Turning now to the following question.

"It looks to me you're computing cosine similarity of a vectorized weight vector and its pruned version, the former containing ALL weights in the network and its size would be humongous, how do you deal with that? Also, that weight vector contains weights of different DNN layers which're segregated by nonlinear activations in the network, why grouping them into one huge vector would work at all? More insights or analytical explanations are needed here."

The size, although significant, is fine as calculating the cosine similarity between two vectors can be effectively computed on the CPU and take advantage of the RAM available and thus is not limited to GPU VRAM. Similarly, computing the optimal level of pruning (see 5.4) can be done efficiently.

As to grouping all of the layers into one vector, given a network topology (including choices of activation functions) and an enumeration scheme for weight matrices (e.g. stacking columns or stacking rows) and biases, there is a one-to-one correspondence between the standard representation of a network and its vectorisation. Since we're doing unstructured pruning, the network topology is fixed, which means that we can vectorise, prune the vector, and reconstruct a network without any issues.

The general comment above and edits to sec 5.2-5.3-5.4 discuss what happens if a pruned segment of a vector corresponds an entire layer.

Reviewer comment (Experiments):
Only LeNet, ResNet18 and VGG11 are experimented. I would be interested in seeing edge networks like MobileNet to see how "brittle" they are, and whether these edge nets are already tight for further pruning.

Author response:
Thank you for your review.

Your point about other types of networks is well taken. We have followed your advice and considered MobileNet, which turned out to be very informative. This model (and two others) were trained on ImageNet in order to add another training set to our experiments.  The results are discussed in the general comment above and detailed in the new version of the paper (specifically in sections 5.4, 5.3 and 5.2 - edits are in blue).

Reviewer comment (Experiments):
Can the findings be extended to other types of neural networks, such as recurrent neural networks or Transformers?

Author response:
Thank you for your review.

Your point about other types of networks is well taken. We have followed your advice and considered MobileNet, which turned out to be very informative. This model (and two others) were trained on ImageNet in order to add another training set to our experiments.  The results are discussed in the general comment above and detailed in the new version of the paper (specifically in sections 5.4, 5.3 and 5.2 - edits are in blue).

Reviewer comment (Experiments):
Experiments only utilize 3 architectures, 2 pruning strategies, and 1 dataset (CIFAR10). I would expect an empirical paper at ICLR to consider at least one additional dataset (ImageNet) and some additional unstructured pruning strategies (e.g., lottery ticket rewinding) would increase impact of findings.

Author response:
We have addressed your constructive criticism and Question 2 by repeating our pruning experiments on one more dataset (ImageNet) trained on several networks. The results are discussed in the general comment above and detailed in the new version of the paper (specifically in sections 5.4, 5.3 and 5.2 - edits are in blue).

Reviewer comment (Experiments):
At the top of p. 7 you state “It is still unclear whether, given more fine-tuning steps, these models can return to the low-loss region from their current position.” Did you consider exploring this more? I think it would be interesting and worthwhile to empirically explore this direction by increasing the number of fine-tuning steps to see if the pruned models with lower cosine similarity can regain accuracy lost due to pruning.

Author response:
To answer your Question 1 we have repeated our experiments with ten epochs of fine-tuning instead of one. The conclusions (discussed in sec 5.2 and additional graphs in the appendix) are two-fold.

First, additional fine-tuning magnifies the effect of the prune. After a relatively small prune, 10 epochs of fine-tuning result in a network which is more similar to the original than 1 epoch of fine-tuning. Conversely, after a big prune, 10 epochs of fine-tuning will lead to a bigger functional change w.r.t. to the parent than 1 epoch of fine-tuning.

Second, in terms of accuracy, 10 epochs of fine tuning yield some small improvements but the network remains in the same loss region as 1 epoch of fine-tuning. This holds across all levels of pruning.

Reviewer comment (Experiments):
While I find the premise and findings to be interesting, I think the evaluation is limited in that it is only performed on a single dataset. I think the addition of empirical results on at least a larger scale dataset, like ImageNet, and additional unstructured pruning strategies would better support the generalizability of the takeaways.

Author response:
We have addressed your constructive criticism and Question 2 by repeating our pruning experiments on one more dataset (ImageNet) trained on several networks. The results are discussed in the general comment above and detailed in the new version of the paper (specifically in sections 5.4, 5.3 and 5.2 - edits are in blue).