Paper: Masks, Signs, And Learning Rate Rewinding (ICLR-2024)

Reviewer comment (Experiments):
Although the main ideas explored in the paper and the theoretical insights are strong, the empirical evaluation is limited to ResNet architectures. The empirical results are mainly reported for CIFAR10 and CIFAR100. Extending the findings to more architectures and other domains would be helpful to understand the significance of the findings. Additionally, to reviewer's understanding, several figures do not fully support the claims made in the text (see questions below).

Author response:
We further confirm our findings on the larger and more complex datasets: Tiny ImageNet (see Figure 2d) and ImageNet (see Figure 15 in the appendix).

- The main insight we draw from Figure 4 is that LRR is able to improve parameter optimization for different sparse masks, like the mask found by IMP (IMP mask). This finding actually holds for multiple datasets and learning settings. LRR with the IMP mask achieves a significantly higher performance than IMP on CIFAR10. If we modify LRR with BN parameter rewinding (red line) this holds also true on CIFAR100. Furthermore, note that we can draw similar conclusions for Tiny ImageNet (Fig. 2d), ImageNet (Fig. 15), different pruning criteria (Fig. 16), and VGG networks (Fig. 17).

- Fig 7(c, d) additionally indicates that LRR depends significantly on the learned sign information, as sign perturbations can greatly hurt its performance. Yet, it is robust to magnitude rewinding (IMP reset amplitude in Fig 7). LRR signs and masks seem to contain sufficient information to achieve the high LRR performance. This conclusion is supported by the results on all studied datasets.

Reviewer comment (Experiments):
Magnitude pruning is explored as the only strategy to train sparse networks. Can the findings regarding the superiority of LRR be generalized and extended to other pruning strategies?

Author response:
So far, LRR has only been proposed as a variant of IMP, which is the most common pruning baseline for its simplicity and high accuracy. It was therefore restricted to magnitude as a pruning criterion. Nevertheless, we have transferred the same principle to random pruning (see Figure 2. b,c and Figure 13 in the appendix), for which our insights hold as well. To extend the analysis, we have further added experiments for iterative Synflow and iterative SNIP in Fig. 16. The superiority of LRR can be generalized to all these different criteria. Also, our additional experiments with a VGG16 architecture on CIFAR10 in Figure 17 in the appendix are in line with these results.

Reviewer comment (Experiments):
An exploration of the impact of the learning rate would also help to better understand the practical value of the proposed analysis.

Author response:
In Figure 8 and 9 in the appendix, we provide a comparison on the effect of different learning rate schedules and highlight the benefits of using warmup epochs at the beginning of every training cycle to improve performance of both IMP and LRR.

Reviewer comment (Experiments):
The conclusions for CIFAR10 and CIFAR100 often diverge (Fig. 4: LRR with IMP mask vs IMP, Fig. 7), however the explanation for this divergence is not well understood. Although the authors list potential reasons, there is no experiment to support or reject the hypothesis. Evaluation on more datasets and more architectures should help to provide a stronger evidence for the claims and the relevance of the obtained theoretical findings.

Author response:
We further confirm our findings on the larger and more complex datasets: Tiny ImageNet (see Figure 2d) and ImageNet (see Figure 15 in the appendix).

- The main insight we draw from Figure 4 is that LRR is able to improve parameter optimization for different sparse masks, like the mask found by IMP (IMP mask). This finding actually holds for multiple datasets and learning settings. LRR with the IMP mask achieves a significantly higher performance than IMP on CIFAR10. If we modify LRR with BN parameter rewinding (red line) this holds also true on CIFAR100. Furthermore, note that we can draw similar conclusions for Tiny ImageNet (Fig. 2d), ImageNet (Fig. 15), different pruning criteria (Fig. 16), and VGG networks (Fig. 17).

- Fig 7(c, d) additionally indicates that LRR depends significantly on the learned sign information, as sign perturbations can greatly hurt its performance. Yet, it is robust to magnitude rewinding (IMP reset amplitude in Fig 7). LRR signs and masks seem to contain sufficient information to achieve the high LRR performance. This conclusion is supported by the results on all studied datasets.

Reviewer comment (Presentation):
What is the meaning of the light blue line in Fig. 11?

Author response:
The blue line in Fig 11 (Fig 12 in the updated draft) refers to experiments, where we fix several aspects that have been learned by LRR. Concretely, we train an architecture with LRR mask and initial parameters that have the same signs as the learned LRR parameters. The only part that is still learned in the iterative scheme is the parameter magnitude, which starts from the initial IMP magnitude. The fact that this blue line achieves a similar performance as LRR suggests that the high LRR performance can primarily be attributed to the learned signs and the mask.

Reviewer comment (Evaluation):
How does Fig. 12 support the claim made in the figure caption "LRR enables considerably more sign flips than IMP and thus improving mask identifcation and parameter optimization"?

Author response:
Figure 12 (13 in the updated draft) plots the difference between the total number of signs flipped from the initialization. It shows that LRR flips many more signs in comparison with IMP. We have removed the conclusion in the caption, as not only this figure but the collection of different experiments provide evidence for the stated hypothesis.

Reviewer comment (Experiments):
While the authors argue LRR finds a better mask than WR in Figure 3, I wonder if a longer training epochs within each IMP cycle would help WR to find a superior mask. In other words, are both WR and LRR fully converged? If that’s the case, does the mask configuration stay constant after convergence? Further, if the optimal mask can be attained only at the end of the training epochs, it could pose challenges in efforts to reduce the computational cost associated with IMP (both WR & LRR).

Author response:
- Both IMP and LRR are trained until convergence in each prune-train iteration (All details on the procedure are provided in Appendix 3 (Table 1)). The mask is identified by pruning the smallest parameters at the end of each training level once the model has converged. Hence it stays constant for each training iteration between pruning levels. The mask can also be identified after fewer training epochs optimizing a trade-off between model accuracy and computational costs (see e.g. [1,2]). However, we chose to identify the mask after convergence, because our work is focused on insights into the mechanisms that make iterative pruning algorithms succeed. 
- These insights might inspire computationally less expensive variants of LRR and IMP in future. For instance, we find the fact encouraging that LRR identifies good sign configurations relatively early in training.

Reviewer comment (Evaluation):
Concerning the flexible LRR mask analysis (see Figure 4), there appear to be some questionable findings. For instance, the "LRR with IMP mask (blue)" does not appear to show significant improvement over WR (orange) except for the case of Cifar-10 with a moderate level of sparsity. Moreover, it is unclear regarding the implication of "LRR w/ BN rewind (yellow)" in Figure 4 in the context of "flexible LRR training."

Author response:
- The main insight we draw from Figure 4 is that LRR is able to improve parameter optimization for different sparse masks, like the mask found by IMP (IMP mask). This finding actually holds for multiple datasets and learning settings. As correctly noted, LRR with the IMP mask achieves a significantly higher performance than IMP on CIFAR10. If we modify LRR with BN parameter rewinding and use it to train a network with the IMP mask (red line) this holds also true on CIFAR100. Note that the yellow line is not designed to showcase the flexibility of LRR in optimizing different masks, the red line is. In summary, the comparison of the red and the orange lines support our hypothesis.

- Furthermore, note that we can draw similar conclusions for ImageNet (Fig. 15), different pruning criteria (Fig. 16), and VGG networks (Fig. 17).

Reviewer comment (Experiments):
In Figures 3 and 6, there are only two sets of experimental results available for analysis, Cifar-10 and Cifar-100. The authors argue that Cifar-100 results may not fully meet expectations due to its higher complexity. However, it remains an open question whether the same trend would hold for different network architectures, such as VGG networks.

Author response:
We have added experiments for VGG16 in Fig. 17 in the appendix. The same general trends transfer also to this setting.

Reviewer comment (Experiments):
Is there any further results on ImageNet or any large-scale datasets?

Author response:
We have added experiments for ImageNet in Fig. 15 in the appendix. The results are consistent with our previous findings on CIFAR10, CIFAR100, and Tiny Imagenet.

Reviewer comment (Reproducibility):
In Figure 3, is the presented LRR results with or without BN rewinding?

Author response:
Figure 3 is without BN rewinding in LRR. However, note that IMP always rewinds BN parameters.

Reviewer comment (Experiments):
Considering that hyperparameter tuning is generally a problem-specific task, do you believe this sensitivity might hinder the practical application of LRR? Could you provide some advice or guidelines for hyperparameter selection or tuning when using LRR and discuss how the learning rate schedule influences the performance of this algorithm?

Author response:
Both IMP and LRR appear to be similarly sensitive to hyperparameters. In general, generalization on CIFAR10 is less sensitive to different learning rate schedules. However, for the larger CIFAR100 dataset, generalization is more susceptible to a poorly tuned learning rate schedule for iterative pruning. Both IMP and LRR have been used with a step or cosine learning rate schedule (Renda et al. [1]). We find that adding a few warmup epochs (usually 10) at each pruning-training iteration (with linearly increasing LR during warmup) improves the overall performance marginally on CIFAR10 and significantly on CIFAR100, for both LRR and IMP (see Figures 8 and 9 in the appendix). Hence, we propose to use warmup epochs for all experiments.

Reviewer comment (Theory):
Your paper also indicates that LRR can benefit from the overparametrization of neural networks. Can you elaborate on how this overparametrization impacts the functionality of LRR? Is it possible to have too much overparametrization, which could negatively impact the performance of LRR?

Author response:
LRR benefits from overparameterization. 
  - Our theoretical insights on the single neuron model highlight that LRR can leverage overparameterization in early prune-train iterations to enable sign switches of initial parameters and find stable sign configurations which ensure that LRR succeeds more often than IMP. While an increasing input dimension does not harm LRR, at some point its positive impact saturates and further overparameterization is not required anymore (see Figure 18 in the appendix). 
   - Also experimentally, ResNet architectures tend to benefit from an increased width [2]. Yet, this benefit saturates at a certain point (so that the performance does not improve significantly by further increasing the width). This benefit also translates to pruning the corresponding architectures. 
 - We are not aware of a case, in which width overparameterization harms performance. However, we could imagine that it could cause issues with trainability and machine precision in extreme cases, but this would be a different topic of investigation.