Paper: Pruning neural networks using FishLeg estimation (ICLR-2024)

Reviewer comment (Experiments):
I believe that paper contribution is not good enough for ICLR standards. Since the authors adapted the FishLeg implementation for pruning, I would have expected a broader evaluation process. The AutoEncoder benchmark on MNIST (which is also used in the FishLeg paper) and ImageNet on ResNet-50 are not that relevant for pruning.

Author response:
We appreciate your thoughtful review. In our main rebuttal, we addressed the common concerns of all reviews, including yours. As discussed above, this work focuses on improving second-order importance scores and, therefore, pruning using FishLeg surgeon without entering into any specialization of pruning tactics (recipes and distillation strategies) or network architectures. Therefore, our experiment uses the same benchmarks as previous second-order methods [3]. Applying FLS to LLMs pruning is undoubtedly interesting but outside this project's scope. We want to highlight that FLS can be combined with any state-of-the-art pruning methods as long as they are compatible with second-order pruning, further enhancing their capabilities.

Reviewer comment (Experiments):
The M-FAC and oBERT baselines are not state of the art for ImageNet/ResNet-50 benchmark. For example, in the Figure 8 from the CAP approach [1] also show around 70% accuracy for M-FAC for the same benchmark (ImageNet/ResNet-50 @ 75% sparsity), but the CAP approach is much better than M-FAC, reaching about 75% accuracy for 75% sparsity.

Author response:
We appreciate your thoughtful review. In our main rebuttal, we addressed the common concerns of all reviews, including yours. As discussed above, this work focuses on improving second-order importance scores and, therefore, pruning using FishLeg surgeon without entering into any specialization of pruning tactics (recipes and distillation strategies) or network architectures. Therefore, our experiment uses the same benchmarks as previous second-order methods [3]. Applying FLS to LLMs pruning is undoubtedly interesting but outside this project's scope. We want to highlight that FLS can be combined with any state-of-the-art pruning methods as long as they are compatible with second-order pruning, further enhancing their capabilities.

Reviewer comment (Experiments):
The paper does not have any experiments on LLMs pruning. Since this paper addresses the one-shot pruning too, some good results can be obtained using SparseGPT [2] technique, which showed good results on one-shot pruning on large models.

Author response:
We appreciate your thoughtful review. In our main rebuttal, we addressed the common concerns of all reviews, including yours. As discussed above, this work focuses on improving second-order importance scores and, therefore, pruning using FishLeg surgeon without entering into any specialization of pruning tactics (recipes and distillation strategies) or network architectures. Therefore, our experiment uses the same benchmarks as previous second-order methods [3]. Applying FLS to LLMs pruning is undoubtedly interesting but outside this project's scope. We want to highlight that FLS can be combined with any state-of-the-art pruning methods as long as they are compatible with second-order pruning, further enhancing their capabilities.

Reviewer comment (Theory):
The parameterization of the inverse of Fisher information matrix (Eq. 9) is not unique even under the practicality constraint. There might exist a practical tradeoff between the capacity and form of the parameterization and the quality of the resulting importance metric for pruning.

Author response:
With respect to your concerns about the non-uniqueness of the $Q(\lambda)$ parameterization of the inverse Fisher, we claim that this flexibility is in fact an advantage of our method, as it allows the user to trade off accuracy for scalability. Nevertheless, in this work, we propose concrete approximations for the dense and convolutional layers that allow us to achieve better pruning performance than other second-order computational methods while having lower computational costs, as described in the general answer above.

Reviewer comment (Experiments):
As the authors demonstrated, the procedure of meta-optimization of $\lambda$ has hyperparameters that are tricky to tune. This leads to practical complexity.

Author response:
This reviewer also has expressed concern about the possible practical complexity of finetuning the hyperparameters of meta-optimization of $\lambda$. We are the first to acknowledge that hyperparameter tuning was an issue that was left unaddressed by the original FishLeg paper, and we did in fact spend considerable time thinking of ways to mitigate this:

- Section 3.2 shows that a good diagonal initialization of $Q$ should have a magnitude equal to the largest eigenvalues of $F_\gamma^{-1}$, which is roughly $1/\gamma$ (Figure 1).

- Section 3.3 acknowledges that learning the full $F^{-1}$ is a complex problem when $F$ is ill-conditioned, as the auxiliary loss inherits this ill-conditioning. Nevertheless, our theoretical analysis of this problem (Appendix F) has led to the discovery of a good preconditioner which only costs a single additional $Q(\lambda)v$ product per iteration (Algorithm 1).
This preconditioner greatly accelerates the asymptotic convergence of the auxiliary loss (Figure 5A), leading to better estimates of the inverse FIM.

Reviewer comment (Experiments):
Lack of demonstration with large models in comparison against competing techniques.

Author response:
We appreciate your thoughtful review. In the main rebuttal above, we have addressed the concerns common to all reviews, including your questions about the lack of demonstration with large models and the superiority of FLS compared to other second-order methods.