Paper: Sparse Weight Averaging with Multiple Particles for Iterative Magnitude Pruning (ICLR-2024)

Reviewer comment (Experiments):
While the authors address the computational efficiency of SWAMP+ in section 4.3, stating that it can utilize a single particle for the first few iterations, this claim seems to work because the networks are already very sparse. My concern lies in the computational cost of SWAMP+ at lower sparsity levels.

Author response:
__[W1] *(Re. SWAMP+ in low sparsity regime):*__ Thank you for pointing out the effectiveness of SWAMP+ in low sparsity regimes. We agree with the reviewer that the effectiveness of SWAMP+ fades under low sparsity regimes. That said, IMP is a SOTA algorithm for extremely high sparse networks (sparsity over 90%). We thus suggest two alternatives instead of SWAMP+ for low sparsity regimes. Firstly, dynamic sparse methods (e.g., RigL) can also benefit from SWAMP as shown in Table 9 which exhibits significantly lower training costs than IMP. Secondly, leveraging the robustness of SWAMP (refer to Table 7), one can employ higher pruning ratio which can still boost the training speed even in low sparsity regimes. We believe the general response could further resolve your concerns regarding the training cost of the algorithm. Additionally, as per your great suggestion, we hereby provide the results of IMP+ on ImageNet in Table R.10. Please note that the results in Table R.10 differ from Table 3, as the experiment was conducted earlier with a different set of hyperparameters.

Reviewer comment (Experiments):
Did the authors test SWAMP+ on ImageNet?

Author response:
__[W1] *(Re. SWAMP+ in low sparsity regime):*__ Thank you for pointing out the effectiveness of SWAMP+ in low sparsity regimes. We agree with the reviewer that the effectiveness of SWAMP+ fades under low sparsity regimes. That said, IMP is a SOTA algorithm for extremely high sparse networks (sparsity over 90%). We thus suggest two alternatives instead of SWAMP+ for low sparsity regimes. Firstly, dynamic sparse methods (e.g., RigL) can also benefit from SWAMP as shown in Table 9 which exhibits significantly lower training costs than IMP. Secondly, leveraging the robustness of SWAMP (refer to Table 7), one can employ higher pruning ratio which can still boost the training speed even in low sparsity regimes. We believe the general response could further resolve your concerns regarding the training cost of the algorithm. Additionally, as per your great suggestion, we hereby provide the results of IMP+ on ImageNet in Table R.10. Please note that the results in Table R.10 differ from Table 3, as the experiment was conducted earlier with a different set of hyperparameters.

Reviewer comment (Evaluation):
The results in Figure 4 show that interpolated weights yield even lower errors compared to IMP weights. Could the authors provide a detailed explanation or hypothesis as to why this is the case?

Author response:
__[Q1] *(Re. superior performance of interpolated networks in Fig. 4):*__ We appreciate the reviewer for highlighting empirical analysis in Figure 4. Following [Paul et al., 2023], Figure 4 illustrates that there exists a linear path on the loss landscape connecting two successive IMP/SWAMP solutions. In case of both IMP/SWAMP, as the reviewer pointed out, the interpolated weights exhibit lower loss compared to obtained solutions which is also corroborated by several prior works [Garipov et al., 2018; Wortsman et al., 2021; Nam et al., 2022; Paul et al., 2023]. This finding suggests that one can obtain a better solution interpolating successive solutions which is the main motivation for Lottery Pools [Yin et al., 2023]. Further, while we only highlight the existence of linear connectors in the paper, we here provide additional experimental results in Tables R.11 and R.12 suggesting that SWAMP can also benefit from such a phenomenon, i.e., combining SWAMP and Lottery Pools leads to further performance enhancement.

Reviewer comment (Experiments):
Appendix B states, "The learning rate for this phase (SWA phase) is set to a constant value of 0.05." Does this imply that the minimum learning rate is set at 0.05 for SWAMP, and for other baselines such as IMP?

Author response:
__[Q2] *(Clarification: learning rate setting):*__ Thank you for the attentive review. In our IMP experiments, we explored two learning rate schedules: cosine annealing (initiating from 0.1 and decaying to 0.001) and step learning rate (adopted from SWA; dropping to a constant value of 0.05). The cosine annealing schedule turned out to be the best for IMP, while SWAMP exhibited better performance with step lr schedule. In short, we employed the cosine annealing schedule for IMP and the step lr schedule for SWAMP.

Reviewer comment (Experiments):
Do IMP and SWAMP use the same epoch T_0 to rewind weights?

Author response:
__[Q3] *(Clarification: rewinding epochs):*__ Yes, indeed. We set the same matching epoch for both IMP and SWAMP.

Reviewer comment (Novelty):
In terms of methodology, the proposed technique provides an empirically guided straightforward extension over the IMP technique. The proposed SWAMP therefore has a trivial contribution and therefore lacks novelty.

Author response:
__[W1] *(Re. lack of novelty):*__ Our primary contribution extends beyond the development of an algorithm; it involves empirically demonstrating that sparse networks trained with varying SGD noise can benefit from weight-averaging, especially as they become sparser – a finding that is not immediately obvious and has been acknowledged by Reviewer-tk5N. Moreover, as Reviewer-MGna highlighted, our work is inspired by the theoretical framework of [Paul et. al, 2023] which enabled us to empirically validate whether SWAMP retains the key property of IMP; we kindly refer the reviewer to Section 3.3.

Reviewer comment (Evaluation):
In terms of experimentation, the performance gain over other techniques seems to be marginal and reduces the significance of their proposed methodology.

Author response:
__[W2] *(Marginal improvement of the algorithm):*__ Please refer to General Response B.

Reviewer comment (Evaluation):
In Figure 3, for relatively lower sparsity (e.g., sparsity of 20%), the proposed Weighted Average (WA) technique seems to underperform the individual particle performance. Does this mean the proposed technique harms the performance on the lower sparsity? The authors may need to provide more extensive justification to explain this phenomenon.

Author response:
__[W3, Q1] *(Analysis under lower sparsity regime):*__ Thank you for highlighting the failure of sparse weight averaging under low sparsity regimes. We can see that weight averaging fails at the earlier stages of IMP due to the highly non-convex nature of the landscape. However, as sparsity increases, particles tend to locate in the same wide basin which enables weight-averaging. Such a finding is in line with [Frankle et al., 2020; Paul et al., 2023] demonstrating the ease of finding a low-loss curve with a smaller network compared to a larger one, i.e., a sparse network tends to be more stable. Additionally, it further supports that our algorithm benefits more with sparser networks.

\
__References__  
- Frankle et al., “Linear mode connectivity and the lottery ticket hypothesis.” ICML 2020.   
- Paul et al., “Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?” ICLR 2023.

Reviewer comment (Experiments):
The authors have not empirically justified their choice of using Stochastic Weight Averaging (SWA) as opposed to SGD in the manuscript. It would be important to understand the impact of SWA on the proposed approach by demonstrating how IMP performs when it uses SWA as opposed to SGD.

Author response:
__[W1] *(Ablation study w.r.t. SWA & multiple particles):*__ Thank you for pointing out the issue. We indeed provided an ablation study regarding SWA in Table 5. In detail, the second and the third row each correspond to SWAMP without SWA and IMP with SWA, respectively. We can readily find that each component, SWA and multi-particles, contributes similarly to the performance gain, and the optimal performance is attained when used in combination.

Reviewer comment (Experiments):
The authors have not empirically justified their choice of using Stochastic Weight Averaging (SWA) as opposed to SGD in the manuscript. It would be important to understand the impact of SWA on the proposed approach by demonstrating how SWAMP performs when it uses SGD as opposed to SWA.

Author response:
__[W1] *(Ablation study w.r.t. SWA & multiple particles):*__ Thank you for pointing out the issue. We indeed provided an ablation study regarding SWA in Table 5. In detail, the second and the third row each correspond to SWAMP without SWA and IMP with SWA, respectively. We can readily find that each component, SWA and multi-particles, contributes similarly to the performance gain, and the optimal performance is attained when used in combination.

Reviewer comment (Experiments):
I would like to understand why the authors choose to average the weights in SWAMP. As demonstrated in Figure 1, there might be individual IMP runs that outperform SWAMP. Why not take the best of multiple pruned weights?

Author response:
__[Q1] *(Re. best seed performance):*__   As per your insightful suggestion, we provide the performance of an IMP ensemble, wherein each member is chosen as the best-performing model from multiple random seeds. Tables R.8 and R.9 below verify that exploring more random seeds does not result in additional improvements.

\
__Table R.8.__ Further comparison between (a) IMP, (b) IMP selected from best seed, and (c) SWAMP  algorithm on CIFAR-10.
|     | Sparsity 50% | Sparsity 75% | Sparsity 90% | Sparsity 95% |
| :-  | :-:          | :-:          | :-:          | :-:          |
| (a) | 93.97 ± 0.16 | 94.02 ± 0.23 | 93.90 ± 0.15 | 93.58 ± 0.09 |
| (b) | 93.91 ± 0.23 | 94.14 ± 0.13 | 93.81 ± 0.14 | 93.72 ± 0.27 |
| (c) | 94.74 ± 0.04 | 94.88 ± 0.09 | 94.73 ± 0.10 | 94.23 ± 0.11 |

__Table R.9.__ Further comparison between (a) IMP, (b) IMP selected from best seed, and (c) SWAMP  algorithm on CIFAR-100.
|     | Sparsity 50% | Sparsity 75% | Sparsity 90% | Sparsity 95% |
| :-  | :-:          | :-:          | :-:          | :-:          |
| (a) | 75.40 ± 0.23 | 75.72 ± 0.41 | 75.24 ± 0.25 | 74.60 ± 0.37 |
| (b) | 75.79 ± 0.53 | 75.05 ± 0.39 | 74.92 ± 0.58 | 74.35 ± 0.43 |
| (c) | 77.29 ± 0.53 | 77.35 ± 0.39 | 77.14 ± 0.33 | 76.48 ± 0.73 |

Reviewer comment (Novelty):
The method comes down to applying the model soup paper to sparse training/IMP. I believe there is sufficient novelty in applying a method only shown on dense training and not necessarily repeatable in the sparse training context, never mind the extensive analysis shown by the authors in this work. Saying that, it's also not the most novel research direction out there compared to many papers.

Author response:
__[W1] *(Extension to different hyperparameters):*__ We thank the reviewer for recognizing the novelty of our work, where our main objective is to empirically validate sparse weight averaging technique and whether it preserves the characteristics of IMP rooted in the theoretical framework of [Paul et al., 2023]. As the reviewer pointed out the work of model soups [Wortsman et al., 2022], we hereby provide extended results of SWAMP with multiple particles obtained from various hyperparameters settings (weight decaying rate, learning rate) beyond random seeds in Table R.5. Our findings indicate that weight averaging with particles perturbed by means other than random seeds do not markedly improve performance. We acknowledge that further optimization of hyperparameters, given enough time, could potentially boost SWAMP's performance making it an interesting direction for future research. That said, one should also consider whether the improvements of SWAMP variants justify the additional computational cost they demand.

Reviewer comment (Experiments):
As presented in the main body of the paper, SWAMP is *much* more expensive than most of the compared sparse training methods in e.g. Table 2 at *training time*. This is because IMP with weight rewinding is extraordinarily expensive in practice. However, the authors do demonstrate that the SWAMP methodology applies to other much more efficient sparse training methods in the appendix, notably RiGL, a state-of-the-art sparse training method, and one that is reasonably efficient. I believe the authors should focus their method as being widely applicable to sparse training methods in the main body of the paper, rather than focusing on IMP however - this is especially important given the motivation that SWAMP is better than training an ensemble (which is in fact likely cheaper than SWAMP when using more practical sparse training methods than IMP!).

Author response:
__[W2, Q2] *(Heavy training cost & extension to other pruning methods):*__ As per your constructive feedback, we will revise the final manuscripts to include the SWAMP-extended dynamic pruning results in the main body of the paper, not the appendix. As shown in Table 9, SWAMP is indeed applicable to other sparse training methods. During the rebuttal phase, as time permits, we aim to investigate whether dynamic pruning methods, aside from RigL, can also benefit from SWAMP. Moreover, we choose to focus on IMP in order to verify whether SWAMP preserves the theoretical characteristics outlined in [Paul et al., 2023] as discussed in Section 3.3. Also, we believe that General Response A could further resolve your concerns regarding the training cost of the algorithm.

Reviewer comment (Experiments):
While the paper is motivated by comparing the generalization of a SWAMP to an ensemble of two IMP solutions, what is the comparison in generalization when using other sparse training methods, e.g. RiGL, given that these other methods often generalize better than IMP?

Author response:
__[Q1] *(Re. ensemble of dynamic pruning methods):*__  As per your detailed feedback, we hereby provide additional results on the ensemble of RigL and DST solutions. Tables R.6 and R.7 provide clear evidence that SWAMP exhibits comparable generalization to test sets when compared to ensembles of other baseline methods.

\
__Table R.6.__ Additional accuracy results on the ensemble of two particles of (a) VGG13 with RigL, (b) VGG13 with DST on CIFAR-10. And (c) is the performance of a single SWAMP solution. Reported values are averaged over three random seeds.
|     | Sparsity 50% | Sparsity 75% | Sparsity 90% | Sparsity 95% |
| :-  | :-:          | :-:          | :-:          | :-:          |
| (a) | 94.34 ± 0.13 | 94.12 ± 0.15 | 94.15 ± 0.22 | 93.33 ± 0.09 |
| (b) | 94.54 ± 0.09 | 94.48 ± 0.19 | 94.29 ± 0.14 | 94.11 ± 0.02 |
| (c) | 94.14 ± 0.08 | 94.39 ± 0.15 | 94.40 ± 0.16 | 94.34 ± 0.11 |

__Table R.7.__ Additional accuracy results on the ensemble of two particles of (a) VGG16 with RigL, (b) VGG16 with DST on CIFAR-100. And (c) is the performance of a single SWAMP solution. Reported values are averaged over three random seeds.
|     | Sparsity 50% | Sparsity 75% | Sparsity 90% | Sparsity 95% |
| :-  | :-:          | :-:          | :-:          | :-:          |
| (a) | 74.43 ± 0.21 | 73.66 ± 0.20 | 73.15 ± 0.15 | 71.91 ± 0.09 |
| (b) | 75.06 ± 0.14 | 74.87 ± 0.21 | 74.46 ± 0.16 | 73.67 ± 0.16 |
| (c) | 73.27 ± 0.26 | 73.54 ± 0.36 | 73.40 ± 0.33 | 73.53 ± 0.32 |