Paper: Sparse Cocktail: Every Sparse Pattern Every Sparse Ratio All At Once (ICLR-2024)

Reviewer comment (Novelty):
The whole pipeline looks like a huge ensemble of existing techniques, such as the "Dense Pivot Co-training" stage from USNet and BigNAS, the "Sparse Network Interpolation" stage from AutoSlim and LotteryPool … However, the author did not make meaningful discussions in each stage, on their differences from prior arts. I would like to hear the authors clarify.

Author response:
We would like to argue that one of our major novelty is to expand the scope of sparse co-training to cover diverse sparsity patterns and multiple sparsity ratios at once. This research goal stands out as novel because previous works have not addressed such a wide range of sparse patterns and ratios. The harmony among different sparsity patterns is made possible by UMG and Dense Pivot Co-training. By using UMG as a universal pruning criterion and producing closely aligned sparse masks, we relieve the gradient conflicts of different sparsity patterns during training. Then by Dense Pivot Co-training that inserts a dense mini-batch step at every alternative sparse mini-batch step, we further enforce the optimizing directions of subnetworks of different sparsity patterns to be aligned with the same dense network. Meanwhile, the dense network at each IMP iteration has the same initialization as in the Lottery Ticket Hypothesis, thus the optimization directions from different IMP iteration are aligned because of the same dense network initialization. Thus, the optimization directions from different sparsity ratios and patterns are all regularized to be aligned together. We are also the first one to apply LTH for sparse co-training to amortize the sparse co-training pressure (we only co-trains subnetworks of a single sparsity ratio from different sparsity patterns at the same time in each IMP iteration, while finally it produces a lot more subnetworks in total without the need to co-train them together.), while related work such as Lottery Pool only considers aiming to produce a single stronger subnetwork.


We also do not simply reuse existing methods but develop novel adaptations for the sparse co-training circumstance. Specifically, (1) we adapt the refilling method in Chen et al[1] as UMG by incorporating N:M sparsity and letting both unstructured and N:M sparsity decide which channels to refill; (2) we adapt the AST[2](instead of USNet or BigNAS) as Dense Pivot Co-training by not just alternating mini-batches among sparse networks of the same sparsity ratio and pattern but inserting a dense mini-batch step and combine it with IMP to achieve optimization alignments across different sparsity ratios and patterns. (3) we adapt the network interpolation method in Lottery Pool[2] as our network interpolation for sparse co-training and the core difference is that they only aim to produce a single sparse network with an interpolated sparsity ratio, while we need to use the interpolation to produce multiple subnetworks with a shared parameter set.

Reviewer comment (Evaluation):
I would like to see some more relevant metrics such as training time, memory savings, or inference speed ups if any. Without those, it is hard or meaningless to fetch any real benefit of training with sparsity.

Author response:
Thanks for the question. we want to emphasize that the motivation of our sparse co-training is parameter efficiency instead of training or inference speed up. Since for each kind of sparsity pattern, the Sparse Cocktail produces multiple subnetworks with exactly the same sparsity ratios w.r.t. other corresponding methods, we thus have almost the same memory savings, and inference speed up and omit the comparison. We also empirically find that Sparse Cocktail has only around 1/8 extra total wall-clock training time primarily due to 1 extra distillation step every 2 mini-batches.

Reviewer comment (Novelty):
Is Dense Pivot Co-training just weight rewinding (which is a pretty standard trick), or are they different (in which way)?

Author response:
The dense pivot co-training is an orthogonal technique w.r.t. the weight-rewinding. The weight-rewinding means rewinding the weights to initialization weights(or very early epochs) at the start of each IMP iteration, which is operated on the epoch level. For example, if the IMP has 10 iterations and each iteration trains the network for 100 epochs, then the weight rewinding will be performed for 9 times at the start of each iteration, from the 2nd to the last iteration. 
In contrast, the Dense Pivot Co-training inserts a dense network training step before each mini-batch of the sparse co-training, where parameters are shared and continuously updated without rewinding and it is operated on a mini-batch level. For example, at the i-th iteration of Sparse Cocktail, the mini-batch orders during training are arranged like this: dense —> unstructured subnetwork —> dense  —> channel-wise subnetwork  —> dense  —> N:M subnetwork  —> dense  —> unstructured subnetwork ...
To summarize, weight rewinding is an epoch-level operation and dense-pivot co-training is a mini-batch-level operation, and they are thus orthogonal.

Reviewer comment (Evaluation):
No discussion of training time cost. The proposed joint/switchable training appears to take much longer time than any single sparse training method. Please report the details and provide a fair discussion on training cost.

Author response:
Thanks for your question. We have discussed the training cost in Appendix D and will make this point more clear in the revised version. Overall, we keep the batch size and total training epochs of all the methods, including Sparse Cocktail, AST, AC/DC, and MutualNet, as 1500 epochs to have a fair comparison. Specifically, (1) for Sparse Cocktail, we use 10 iterations for IMP and each iteration contains 150 epochs. (2) for AST and MutualNet, we directly perform the co-training for 1500 epochs. (3) for AC/DC we co-train each of the 10 dense-sparse network pairs for 150 epochs. In this way, all the methods have the same number of training iterations and batch size (regardless of which subnetwork will be trained at each mini-batch), and thus the same training cost.  We also empirically find that Sparse Cocktail has only around 1/8 extra total wall-clock training time primarily due to 1 extra distillation step every 2 mini-batches.

Reviewer comment (Reproducibility):
Hyperparameter setting was missed in Appendix C (empty - though mentioned multiple times in the main paper)!! This paper has so many moving widgets and it seems challenging to get all the hyper-parameters and settings right in practice.

Author response:
We apologize for this problem, we have presented the hyper-parameters in Table 5 and Table 6 in the Appendix, but this is misplaced away from the Appendix C section title due to the typesetting problem. We will fix this issue and add textual references in Appendix C in the revised version.

Reviewer comment (Novelty):
I am concerned about the novel contributions of this paper, and the results presented in this paper are the combination of existing works with little novelty of its own.

Author response:
A1: We would like to argue that one of our major novelty is to expand the scope of sparse co-training to cover diverse sparsity patterns and multiple sparsity ratios at once. This research goal stands out as novel because previous works have not addressed such a wide range of sparse patterns and ratios. The harmony among different sparsity patterns is made possible by UMG and Dense Pivot Co-training. By using UMG as a universal pruning criterion and producing closely aligned sparse masks, we relieve the gradient conflicts of different sparsity patterns during training. Then by Dense Pivot Co-training that inserts a dense mini-batch step at every alternative sparse mini-batch step, we further enforce the optimizing directions of subnetworks of different sparsity patterns to be aligned with the same dense network. Meanwhile, the dense network at each IMP iteration has the same initialization as in the Lottery Ticket Hypothesis, thus the optimization directions from different IMP iteration are aligned because of the same dense network initialization. Thus, the optimization directions from different sparsity ratios and patterns are all regularized to be aligned together. We are also the first one to apply LTH for sparse co-training to amortize the sparse co-training pressure (we only co-trains subnetworks of a single sparsity ratio from different sparsity patterns at the same time in each IMP iteration, while finally it produces a lot more subnetworks in total without the need to co-train them together), while related work such as Lottery Pool only considers aiming to produce a single stronger subnetwork.

We also do not simply reuse existing methods but develop novel adaptations for the sparse co-training circumstance. Specifically, (1) we adapt the refilling method in Chen et al[1] as UMG by incorporating N:M sparsity and letting both unstructured and N:M sparsity decide which channels to refill; (2) we adapt the AST[2] as Dense Pivot Co-training by not just alternating mini-batches among sparse networks of the same sparsity ratio and pattern but inserting a dense mini-batch step and combine it with IMP to achieve optimization alignments across different sparsity ratios and patterns. (3) we adapt the network interpolation method in Lottery Pool[3] as we state in A6 below.

Reviewer comment (Experiments):
The results are shown on different sparse subnetworks obtained from multiple sparse masks. However, it is likely that the performance of these sparse subnetworks is stable merely because of the relatively low sparsity reported in the paper. In order to see the effectiveness of the method, I would like to see the performance of the subnetworks with higher sparsity (> 90%) especially for unstructured sparsity patterns.

Author response:
A2: We would like to argue the compared sparse co-training methods all have performance degradation when the sparsity gets very high. In the two compared unstructured sparse co-training methods, AC/DC [4] has ~3.5% performance degradation (compared to vanilla dense network) at 95% sparsity and ~8.5% degradation at 98% sparsity; AST [2] has less performance degradation at high sparsity primarily likely because it doesn't involve co-training with different sparsity ratios and only focus on single sparsity ratio but different masks.

Reviewer comment (Experiments):
The algorithm is not entirely clear from the Figure and methodology section. For example, how many sparsities is each sparse pattern trained for, what are the performances of each sparsity pattern and how does a subnetwork’s performance improve after merging (if it does).

Author response:
A3: Thanks for your question. The number of sparsity ratios can be a variable hyper-parameter while in our setting, we use 10 sparsity ratios for both unstructured and channel-wise sparsity, and 3 sparsity ratios for N:M sparsity (1:2, 2:4, 4:8) following prior sparsity work as we discussed in the start of Section 4. The performance of each sparsity pattern are shown in Table 1 and Table 2, generally speaking, unstructured and N:M subnetworks have comparable performance to the dense network and channel-wise subnetworks have degraded performance due to the structural sparsity nature. Regarding the performance influence of network interpolation, we list the average performance before and after Interpolation of Sparse Cocktail on image classification datasets. The results show that there will be slight performance degradation after interpolation, but we consider this a necessary sacrifice since we aim to produce a single network with shared parameters of multiple subnetworks.


On CIFAR10 dataset:
|  Sparsity Pattern  |   Dense |   Unstructured|  Channel-wise | N:M |
|-------------:|----------:|-----------:|-----------:|-----------:|
|Before Interpolation |  92.56 |   92.45 |   90.97  |    92.32  |
|After Interpolation |    92.48 |   92.09 |   90.02  |    91.83  |


On ImageNet dataset:
|   Sparsity Pattern |   Dense |   Unstructured|  Channel-wise | N:M |
|-------------:|----------:|-----------:|-----------:|-----------:|
|Before Interpolation | 76.45  |   74.64 |   74.02 |    76.13  |
|After Interpolation |   76.32  |   73.23 |   72.22  |    75.19  |

Reviewer comment (Evaluation):
The authors don’t comment on the loss landscape of each of the subnetworks obtained during training. From previous work by Paul et al. [1] I would expect each of the obtained subnetworks to lie in the same loss basin. In order to assess the effectiveness of the dynamic distillation step I would expect to look at the Hessian or the linear mode connectivity between the subnetworks obtained.

Author response:
A4: Thanks for your comment. We empirically find that different subnetworks of the same sparsity pattern obtained by IMP with weight rewinding are located in the same loss basin, i.e. there are no significant error barriers for interpolations similar to Figure 2 in [3], while subnetworks from different sparsity patterns have at most ~3.2% error barriers on average due to the divergence in sparse masks. We show the latter phenomenon in Figure 7 in appendix, by plotting the performance change using different interpolation factors as in Algorithm 1. However, by finding proper interpolations the error barrier problem can be mitigated or avoided.

Reviewer comment (Experiments):
The performance of the proposed method on ImageNet is poorer than AC/DC (in Table 1) which is a well established method.

Author response:
A5: We would like to argue that AC/DC has slightly better performance mainly because the original AC/DC algorithm only co-trains 2 networks (one dense and one sparse) at the same time. In our experiments, we compare Sparse Cocktail, which co-trains 24 subnetworks with different sparsity patterns and ratios at once, with AC/DC which co-trains 2 networks for 10 separate times to produce 10 unstructured subnetworks without parameter sharing. Our Sparse Cocktail achieves much higher parameter efficiency than AC/DC (24 vs 2 avg subnetwork number as in Table 3) with slight sacrifices in the performance of individual subnetworks (which we believe is more a merit than a downside).

Reviewer comment (Novelty):
How does Network Interpolation help, and at what stage of training is it used. It seems to be similar to the implementation of Lottery Pools [1].

Author response:
A6: The network interpolation is used to create a single network with shared parameters among different subnetworks generated by Sparse Cocktail since we have different output parameter values of subnetworks at different iterations. It will only be performed once after the whole IMP process is finished. Our network interpolation is developed based on the interpolation method of Lottery Pool, the core difference is that they only aim to produce a single sparse network with an interpolated sparsity ratio, while we need to use the interpolation to produce multiple subnetworks with a shared parameter set. In terms of technical details, the Lottery Pool evaluates a single network at every greedy step, while our algorithm performs interpolation using the dense networks obtained at the end of IMP iteration (please refer to Appendix B for algorithm details), and then evaluates the performance of every subnetworks obtained so far at i-th iteration by applying their sparse masks.

Reviewer comment (Experiments):
It’s not made clear how the N:M network and unstructured networks obtained from IMP are kept similar to each other such that their weights can be interpolated.

Author response:
A7: Thanks for the question. The weight interpolation among different sparsity patterns is made possible by UMG and Dense Pivot Co-training. By using UMG as a universal pruning criterion, we relieve the gradient conflicts of different sparsity patterns during training. Then by Dense Pivot Co-training, we further enforce the optimizing directions of subnetworks of different sparsity patterns to be aligned with the dense network. Meanwhile, the dense network at each IMP iteration has the same initialization as in Lottery Ticket Hypothesis, thus the optimization directions from different IMP iterations are aligned because of the same dense network initialization. Thus, overall weight interpolation from different sparsity ratios and patterns is possible.

Reviewer comment (Experiments):
It is not clear to me why the authors choose to generate a total of 24 subnetworks by restricting the unstructured and structured sparse networks to 10 each. Is this a hyperparameter and why not choose additional networks at higher sparsity ratios?

Author response:
A8: Please refer to the response in A3. This is an adjustable hyperparameter and we elaborate on how we choose to generate a total of 24 subnetworks at the start of Section 4. As we discussed in A3, co-training at higher sparsity ratios along with a dense network will cause performance degradation problems not only for Sparse Cocktail but also for the compared methods.