Paper: Soft iEP: On the Exploration Inefficacy of Gradient Based Strong Lottery Exploration (ICLR-2024)

Reviewer comment (Experiments):
Need comparison to some "drop and then grow" pruning methods in LTH like Rigging the Lottery and its following works. In the related studies, the authors explained that the difference between SLTH and LTH is that SLTH assumes there are strong subnets in dense models without training. However, the methods and experiments involve model training. Therefore, it is necessary to compare dynamic sparse training methods in LTH.

Author response:
Thank you for providing interesting literature. Firstly, we admit that a performance gap exists between weight-based LT and SLT, and the paper should discuss this point more. Therefore, we have included a detailed discussion on the drawbacks of the SLT-based method compared to the weight-based LT in section 5. Besides we also included a comparison with weight-based LT in Appendix C.2. 

However, we would like to clarify again that SLTH still differs from LTH in the sense the weights are never updated in SLTH, which makes the SLTH unique in both intriguing properties of random neural networks and practical benefits. For practical benefit, please see the general response. Therefore, we believe that the performance gap between weight-based LT and SLT does not hinder the value of the paper. In addition, to strengthen the paper, we added an SLTH-based baseline (see general response 2-3).

Reviewer comment (Experiments):
Soft pruning was directly added to iEP and the best-performing learning rate rewinding setting during the experiment. Need ablation studies: (1) decompose soft pruning and iEP to examine which contributes most; (2) combine soft pruning with different variants of iterative pruning to demonstrate its effectiveness.

Author response:
Thank you for the comments. However, since the difference between Hard iEP w/ LR and Soft iEP w/ LR is whether it employs hard or soft pruning, I’m afraid I think the effect of soft pruning and rewinding strategy is already decomposed in experiments. Besides, since soft pruning w/ SRw degenerated to the standard EP, we can think of the difference between EP and Soft iEP w/ LR as the difference in rewinding strategy.

Reviewer comment (Experiments):
Similar to cons 1, related studies compared in the experiment only include IteRand about iterative pruning. And IteRand is not implemented with the Wide ResNet-50 setting.

Author response:
We directly obtained the value from the original manuscript, which does not provide results of Wide ResNet-50. We have clarified that in the revised manuscript. Due to the lack of enough computational budget (A100 specifically), we can not conduct the experiments on ImageNet during the rebuttal.

Reviewer comment (Presentation):
Figure 6(b) is difficult to comprehend, as the x-axis and the red and blue lines all refer to the weight remaining rate.

Author response:
Sorry for the confusion. I have updated the explanation. To clarify, blue lines do not refer to the weight remaining ratio. It represents the transition of score ranking of each edge. Color represents whether the edges are finally selected or not. See the end of the P8 in the revised manuscript.

Reviewer comment (Experiments):
Also, the dense baseline performance is weak. Based on previous literature like https://arxiv.org/pdf/2210.04092.pdf, the dense ResNet-18 on CIFAR10 and CIFAR100 can reach ~95% and ~77% respectively, which are ~5% and ~17% accuracies better then the ones in Figure 5.

Author response:
Thank you for carefully reading the paper and providing detailed comment, but I’m afraid the reviewer might have some sort of misunderstanding. The baseline with a lower prune ratio performs on par with the provided results, and Fig 5. of CIFAR10 does not provide a dense baseline.

Reviewer comment (Experiments):
One of the reasons for having increased popularity of the EP algorithm is its computational efficiency compared to iterative techniques. Specifically, without iterative pruning, we can easily get the desired subnetwork (winning ticket) in one-shot training. The proposed technique misses the key advantage of the EP algorithm as the proposed Soft iEP requires iterative pruning. Therefore, the proposed technique may be computationally expensive and may limit its applicability in crucial domains such as large language models (LLM) where the computational cost is very expensive.

Author response:
As you pointed out and we have originally discussed in the limitation section, the computational costs to obtain STL are one of the major drawbacks of the proposed method. On the other hand, we believe that the practical benefit of the SLT-based method is deployment-friendly since it only needs to store the binary matrix and random seed to generate weights. See the general response (1) and the revised introduction for a more detailed discussion. 

In addition, we updated the manuscript to clarify the concerns raised by the reviewer for the fairer comparison between the SLT-based method and the dense baseline in Fig 5. See the last paragraph of P7.

Reviewer comment (Experiments):
The proposed techniques miss the multiple baselines that do not require iterative pruning but still perform exploration [1, 2, 3]. It is important to discuss how their proposed technique compares with those techniques in terms of methodology as well as experimental results.

Author response:
Thank you for providing multiple references. We have carefully checked the papers, but we believe they are developed in the context of weight-base LT (multicoated lottery tickets [a]). Instead, we implemented a recently proposed extension of EP, which shares the concept of the provided literature in the sense it uses multiple (cheap) tickets to improve performance. We included the comparison with [a] in section 5 and Appendix and also verified that the proposed method can be combined with [a] to improve the performance further. See general response 2-3 as well. 

[a] "Multicoated Supermasks Enhance Hidden Networks", ICML2022

Reviewer comment (Experiments):
It would be interesting to see how the dying edge phenomenon scales with respect to the size of the network. It may be the case that for the bigger architecture model, the impact of the dying edge is less. To assess the robustness of the proposed technique for different architectures, the authors may be required to consider the higher capacity models (such as ResNet101, ViT) especially for the Cifar10 and Cifar100 datasets.

Author response:
Following the reviewer’s comment, we have included additional models, including higher capacity ResNet18 (with 2x width), ResNext, and ConvMixer. We are sorry that we have not included ViT, as ViT typically requires large-scale datasets to train. In addition, as we have discussed in the original manuscript, the performance improvement tends to be larger when we use higher capacity models. See second paragraph of section 5.

Reviewer comment (Theory):
It seems that higher exploration efficacy implies better performance, which from my point of view has not really been given any rationales.

Author response:
Thank you for the question. I think Figure 2 in the manuscript explains the motivation of the higher exploration efficacy to provide performance improvement. Assume that we have target networks that are hidden in random networks. If the exploration efficacy is low, which means there exist many dying edges as with the standard EP, whether the algorithm can be reached at the solution strongly depends on the initial structure. In other words, low exploration efficiency means that they actually do not (or can not) use entire parameter spaces effectively, thus trapped to the suboptimal solution around the initial solution. Additional analysis of the structure difference among subnetworks obtained by Soft iEP, Hard iEP, and EP also supports this perspective (see Appendix C.3 and general response 3).

Reviewer comment (Reproducibility):
It would be great to give a detailed description in formulas (Appendix should be fine) to summarize the algorithms.

Author response:
We have updated the proposed method's entire description and included the pseudocode in Appendix B.

Reviewer comment (Theory):
Please give rationale and motivation of the higher exploration efficacy, the better performance?

Author response:
Thank you for the question. I think Figure 2 in the manuscript explains the motivation of the higher exploration efficacy to provide performance improvement. Assume that we have target networks that are hidden in random networks. If the exploration efficacy is low, which means there exist many dying edges as with the standard EP, whether the algorithm can be reached at the solution strongly depends on the initial structure. In other words, low exploration efficiency means that they actually do not (or can not) use entire parameter spaces effectively, thus trapped to the suboptimal solution around the initial solution. Additional analysis of the structure difference among subnetworks obtained by Soft iEP, Hard iEP, and EP also supports this perspective (see Appendix C.3 and general response 3).

Reviewer comment (Experiments):
For training such soft iEP, how does this converge compared to the hard one?

Author response:
Thank you for the question. We have included the convergence speed of hard and soft iEP. See general response 2-7 for more explanation.

Reviewer comment (Experiments):
Please discuss the dying ratio and pruning performance for different randomized initializations of the network.

Author response:
We have added experiments on the dying ratio and burning performance for different randomized initializations of the network. For the dying ratio, we included the results of different weight and score initialization and how it affects the dying ratio and performance in section 3 (see general response 2-5). Besides we also tested the proposed iterative pruning with different weight initialization in Appendix C.2.4.

Reviewer comment (Evaluation):
How many only visited once edges are finally retained?

Author response:
Following the reviewer's comments, we extend the analysis and compare the cumulative visit count of edges between EP and iEP. In short, EP first provides almost 10x dying edges, and Soft iEP tends to activate edges a few times. See Appendix C.3.3 for a more detailed discussion.

Reviewer comment (Evaluation):
To what degree are the final pruned edges similar between soft and hard iEP edges? (Maybe compare the edge graph similarity?)

Author response:
Following prior works investigating the mask similarity, we have investigated mask similarity among EP, Hard iEP, and Soft iEP. See appendix C.3.2 and general response 3.

Reviewer comment (Novelty):
The concept of soft pruning, also known as soft filter pruning, is not novel, having been explored in such previous works [1,2], which should be cited in this paper as it is connected to the idea of finding SLTs.  

[1] Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks, IJCAI 2018  
[2] Operation-Aware Soft Channel Pruning using Differentiable Masks, ICML 2020

Author response:
Thank you for providing the missing references. We have cited them in the revised paper and discussed the connection in section 4. Besides, we have developed a new variant of soft pruning inspired by [1]. See the general response 2-6.

Reviewer comment (Experiments):
It would be great to see more experiments with more advanced models. For example with an advanced CNN such as ConvNeXt and the recently proposed vision transformers, the proposed method would be stronger.

Author response:
We have included other backbone networks, including (ResNet18x2, RexNext, and ConvMixer). Due to resource limitations (lack of A100 GPU), we have only conducted experiments on the CIFAR10 and CIFAR100, but we believe the results are enough to show their generality. See general response 2-2.

Reviewer comment (Writing):
The proposed method of soft iEP presumably has many variations for reviving edges, but the details in the manuscript are unclear to me.

Author response:
Sorry for the confusion. Since soft iEP is only valid for the LR and FT setup (if we use SR, it degenerates to the standard EP), we used the LR option throughout the paper. We have clarified on this point in the sanction 4.

Reviewer comment (Evaluation):
Can the authors give any (architectural) differences between the two final networks pruned by iEP and soft IEP having the same pruned weights ratio?

Author response:
Thank you for the insightful comments. Inspired by the reviewer’s comment, we have conducted additional analysis on how different the obtained subnetworks are. Please see the general response 3.

Reviewer comment (Experiments):
This reviewer suggests that the paper could be strengthened by comparing the proposed method with other approaches in the domain of strong lottery tickets if such methods exist and are available.

Author response:
Following the reviewer’s comment, we included a new baseline. Please see the general response 2-3.

Reviewer comment (Experiments):
This reviewer is curious about the applicability of the proposed method to pretrained models, given that hard pruning (hard-EP) has been shown to underperform with randomly initialized weights as shown in [1].  

[1] Lottery Jackpots Exist in Pre-trained Models, TPAMI 2023

Author response:
Thank you for the interesting question. We have tested the applicability of soft iEP to the pre-trained weights. See general response 2-4.

Reviewer comment (Experiments):
It’s better if this paper could discuss more benefits of SLTH. What is the major benefits of only learning mask, rather than weights? What is the current performance gap between learning mask and learning weights. For example, based on Figure 5, we can see that a ResNet-50 with 8 millions number of parameters only achieve a performance around 70%, while at this 66% (1 - 8 / 23) sparsity level, many previous LTH with unstructured pruning have show that they can still almost match the performance of the dense net (~76%). Thus, there is still a huge performance gap for SLTH-based methods. I would suggest the authors highlight the practical benefit of SLTH, and also show the performance gap with weight-based LTH.

Author response:
Following the reviewer's comments, we have updated the introduction to discuss the practical benefit of the SLTH. In short, it is more robust to the quantization and, moreover, it can easily recover weight matrix only via random seed. Thus, only need to store binary matrix during the deployment, providing practical merits on edge device scenario. We have clarified this point mainly in the revised introduction. 

On the other hand, the SLT-based method still usually underperforms the method involving weight updates like dynamic sparse training, the standard lottery tickets, and its variants. We have clarified this point as well in the experiment parts (section 5). Besides, we have conducted additional experiments to clarify this point in Appendix C.2.1; see the general response 2-1.

Reviewer comment (Novelty):
The novelty of applying iterative pruning to the SLTH is kind of limited. Although this paper claims that iterative pruning has not been evaluated in SLTH, the proposed idea is not exciting considering that iterative pruning has been massively explored in the LTH community.

Author response:
I agree that iterative pruning itself is widely used in the LTH community; however, as we have discussed in the introduction, the application to SLTH is limited. In this sense, our paper can be posed as the first empirical analysis of the effectiveness of iterative pruning on SLTH, using a variety of datasets (CIFAR10, CIFAR100, and ImageNet), architecture (CNN, ResNet, ResNext, and Conv Mixer), retraining procedure (hard pruning, soft pruning, and various rewinding procedure). In addition, our main novelty is also posed in the analysis of the pitfall of the SLTH-based method (the dying edge problem).

Reviewer comment (Evaluation):
The investigation of the relationship between dying ratio and performance may be unclear. This paper explores different hyperparameters to optimize EP, including batch size, learning rate, etc. However, it’s unclear whether these hyperparameters themselves will hinders the performance. For example, it’s common that different optimizer will result in different performance when you learn the weights. Thus, the causal relationship between them (“high dying ratio hinders the performance”) cannot be verified. Besides, it’s unclear why change these hyperparameters can result in different dying ratio. It’s better for this paper to illustrate their relationship.

Author response:
I'm sorry for the confusion. I agree that this part is misleading and overclaimed; thus, I revised and moved it to the appendix. We agree that it is difficult to investigate the causal relationship and thus do not intend to claim that. 

Thank you for the comments. We have included the discussion on this point in appendix A.2. While it might be a bit obvious, learning rate and batch size affect the dying ratio since the larger learning rate (or equivalently smaller batch size) possibly swaps more edges. Similarly, the weight decay reduces the magnitude of the score, making it easier to swap edges with smaller gradients.

Reviewer comment (Theory):
It would be advantageous to include mathematical formulations for the proposed soft edge. The current draft employs plain language to convey the concept, ensuring ease of comprehension but lacking the necessary technical rigor.

Author response:
Thank you for the comment. As we mentioned in the general comment, we have tried to improve the technical rigor of the proposed method.