From: ICLR.cc/2023/Conference/Paper5619/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper5619/Authors
title: Thanks to Reviewer AyCo 
comment: Thank you again for your comments. These comments and suggestions greatly help to improve the quality of our paper. Your recognition of our paper in writing, motivation, and experiments encourages us a lot. We hope our response and revised paper have addressed your concern. If you have any other questions or suggestions, please kindly let us know.

From: ICLR.cc/2023/Conference/Paper5619/Authors
title: Thanks to Reviewer mNhR 
comment: Thank you again for your responses and for recognizing our paper's strengths in paper writing and the idea of adaptive BN statistics. We are glad that our efforts have addressed your interests. In our final version, we will wrap up our additional results in the main paper. We hope our response and revised paper have addressed your concern. If you have any other questions or suggestions, please kindly let us know.

From: ICLR.cc/2023/Conference/Paper5619/Authors
title: Thanks to Reviewer X987 
comment: Thank you again for your time and valuable comments, and in particular for recognizing our paper's strengths in motivation and experiments. Your suggestion on paper writing help us a lot and we will polish our writing again in our final version. Please kindly let us know if you have any additional questions or suggestions.

From: ICLR.cc/2023/Conference/Paper5619/Authors
title: Thank you for your response!
comment: Thank you for your response! We address the points you raised.

Q 1.1 Although the subnets are not specialized for each client, I am curious if the specialized subnet can outperform the predefined subnets.

A 1.1  Since predefined subnets are hard to adapt to the data distribution of clients, the predefined subnets often result in a lower accuracy compared to the specialized subnet, which has been shown in existing works [1, 2]. We also demonstrate this in our experiments. Predefined methods (FL-PQSU, SNIP, and SynFlow) often result in lower accuracy compared to specialized methods (PruneFL, LotteryFL, and FedTiny).

[1] Jiang, Yuang, et al. "Model pruning enables efficient federated learning on edge devices." IEEE Transactions on Neural Networks and Learning Systems (2022).

[2] Frankle, Jonathan, et al. "Pruning neural networks at initialization: Why are we missing the mark?." arXiv preprint arXiv:2009.08576 (2020).

From: ICLR.cc/2023/Conference/Paper5619/Authors
title: Thank you for your comments 
comment: Thank you again for reviewing our work and for the valuable feedback, especially in terms of paper writing. 

Should you need any further clarification or have any questions, please do not hesitate to contact us. We are happy to address them before the rebuttal ends.

From: ICLR.cc/2023/Conference/Paper5619/Authors
title: Thank you for your comments 
comment: We gratefully thank you again for your precious time making constructive remarks.

Please feel free to contact us if you have any questions or need any further clarification. We are happy to address them before the rebuttal ends.

From: ICLR.cc/2023/Conference/Paper5619/Authors
title: Thank you for your review
comment: We are very grateful for your valuable comments, and in particular for recognizing the strengths of our paper in terms of motivation and paper writing.

If you have any questions or need further clarification, please feel free to let us know. We are very happy to address these issues before the rebuttal is over.

From: ICLR.cc/2023/Conference/Paper5619/Authors
title: Part 2 of Response to Annette Cheng 
comment: **Q4.** FedDST[2] leveraged RiGL without the BN adaptation at standard compression with negligible accuracy drop. And many models don't use BN, for example, VGG.

**A4.** We argue that adaptive BN selection is a key step to addressing the bias in federated learning. Without adaptive BN selection, a highly biased pruned structure caused by non-IID requires more resources and time to adjust to the optimal structure, especially in the low-density regime. FedTiny largely increases the accuracy compared with the baseline approaches in the low-density regime. In our experiments, FedDST suffers a serious accuracy drop compared to the proposed FedTiny.

VGG[4] does not use BN because its original paper is published in 2014, but BN[5] is published in 2015. In fact, multiple experiments show that VGG will have a better result if it adds BN layers [1], as shown in the following Table. In fact, BN is standard in modern neural networks, e.g., ResNet, DenseNet, and EfficientNet.

| Model    | Acc@1  | Acc@5  |
|----------|--------|--------|
| VGG11_BN | 70.37  | 89.81  |
| VGG11    | 69.02  | 88.628 |
| VGG13_BN | 71.586 | 90.374 |
| VGG13    | 69.928 | 89.246 |
| VGG16_BN | 73.36  | 91.516 |
| VGG16    | 71.592 | 90.382 |
| VGG19_BN | 74.218 | 91.842 |
| VGG19    | 72.376 | 90.876 |

Accuracies are reported on ImageNet-1K[1]

**Q5.** It might be significantly hard to find any practically deployable use case of such a method that requires much higher storage cost and communication costs compared to the target compression ratio.

**A5.** As mentioned in A3, the communication overhead is small compared with the overall communication cost. For example, in the VGG11, the communication overhead is around 20% of a full-size model. For memory cost, although the gradients should be fully computed, the memory just needs to store a_{t,l} gradients during the computing. In the experiment, using VGG11 with 0.1% density, FedTiny only requires 0.9% memory cost of the best existing pruning methods, while achieving 33% higher accuracy. We have clarified this in Section 4.2. FedTiny enjoys low communication and computation costs compared with FedDST and other existing approaches.

Many edge devices, in the real world, are still equipped with very limited memory space, especially for clips in smart security cameras. For example, the memory size of security cameras SoC (e.g., Fullhan) is only about 64MB [3]. However, it requires more than 1.5GB of memory to train a full-size VGG11 model for computer vision tasks. Therefore, the pruning model in the extremely low-density region (e.g., 0.001) still has significant applications in practice.


[1] https://pytorch.org/vision/stable/models.html

[2]Raihan, Md Aamir, and Tor Aamodt. "Sparse weight activation training." Advances in Neural Information Processing Systems 33 (2020): 15625-15638.

[3] https://www.fullhan.com/en/index.php?c=article&a=type&tid=9

[4] Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014).

[5] Ioffe, Sergey, and Christian Szegedy. "Batch normalization: Accelerating deep network training by reducing internal covariate shift." International conference on machine learning. PMLR, 2015.

From: ICLR.cc/2023/Conference/Paper5619/Authors
title: Part 1 of Response to Annette Cheng
comment: We would like to thank you for providing the feedback. Below please find our response. Since your comment was posted on the last day of the discussion stage, we cannot update the paper now. We will include the new results in the final version of the paper.

Q1. The authors might have missed significant development in advanced pruning techniques in FL that incorporate sparse learning, for example ZeroFL (ICLR 2022), FedDST (AAAI 2022). 

A1. Compared to ZeroFL, our proposed FedTiny requires much smaller memory footprints in clients. ZeroFL[1] partitions weights into active weights and non-active weights in the inference and sparsified weights and activations for backward propagation. However, this approach still needs a large memory space because the non-active weights and the gradients generated through the training process are still stored in a dense fashion. Therefore, ZeroFL is not applicable to memory-constrained devices.

Compared to FedDST, our proposed FedTiny requires less computation cost and converges faster. There is a key difference between FedDST and the proposed FedTiny. FedDST deploys the mask adjustment on the clients and the servers generate a new global model via sparse aggregation and magnitude pruning. Compared to the proposed FedTiny, FedDST needs much more computation costs because it needs extra training epochs to recover the growing weights before uploading, which may lead to straggling issues in federated learning. Moreover, without BN adaption, the bias in the initialized model will also slow down the convergence. 

We added a new experiment on the CIFAR-10 dataset to compare the performance of FedDST and the proposed FedTiny. In the experiment, we use the same setting in Figure 1 and use the open-source implementation of FedDST(https://github.com/bibikar/feddst)

The experiment results show the proposed FedTiny outperforms FedDST.  FedDST is not even the best baseline. We will discuss ZeroFL and FedDST in the final version.

| Density | SynFlow | FedDST | LotteryFL | FedTiny |
|:-------:|:-------:|:------:|:---------:|:-------:|
|    1    |  0.9048 | 0.9048 |   0.9048  |  0.9048 |
|   0.1   |  0.8906 | 0.8923 |   0.8992  |  0.9122 |
|   0.05  |  0.8801 | 0.8334 |   0.8869  |  0.8979 |
|   0.01  |  0.8034 | 0.7495 |   0.8262  |  0.8523 |
|  0.005  |  0.7206 | 0.6245 |   0.736   |  0.7972 |
|  0.003  |  0.6279 |  0.515 |   0.6453  |  0.7572 |
|  0.001  |  0.2862 | 0.1868 |   0.2955  |  0.6311 |

Q2. Existing FL work FedDST[2] used the same sparse learning, as adapted in this paper

A2. The sparse learning in our proposed FedTiny addresses three key issues in FedDST. First, FedDST applies two-stage pruning and clients need extra training epochs to recover the growing weights before uploading, which brings more computation cost in a round. For example, FedDST needs to train 10 epochs to adjust the model structure, while our work only trains for 5 epochs because clients just need to upload gradients in FedTiny.  Second, sparse aggregation on weight in FedDST slows down the convergence. FedDST lets clients prune their model structure and apply sparse aggregation to aggregate them and produce a global model. However, in the global model, some weights are only from one or few clients, which makes the serious accuracy drop and slows down the convergence. The proposed FedTiny lets clients have one common model structure to avoid this issue. Third, FedDST gets its initialized model by random pruning, the bias in the initialized model will slow down the convergence. The proposed FedTiny uses adaptive BN selection to produce the adaptive initialized model, which can speed up the model training. 

Our experimental results also show FedTiny outperforms FedDST under various densities. 

Q3 As the server needs to send C models to the clients, also during progressive pruning clients need to have the coarse model (higher density as the fine model), the communication overhead may be significant.

A3. As discussed in the response to Reviewer AyCo, the communication overhead of FedTiny is marginal. First, we investigate the communication cost of adaptive BN selection. Given the optimal pool size ($C^*=\frac{0.1}{d_{target}}$), the communication overhead introduced by adaptive BN selection is around 20% of a full-size model. As federated learning needs to transfer the model for a large number of rounds. A one-time 20% model-size communication is very small compared with the entire communication cost. Therefore, we argue that the overhead introduced by adaptive BN selection is marginal. We clarified the cost of the proposed FedTiny in Section 4.4. We include a quantified cost analysis in Appendix C.

Second, during progressive pruning, clients do not need to have the coarse models, they just need to maintain one sparse model and upload/download it. The additional communication overhead in progressive pruning is sparse gradients, which can be negligible.

From: ICLR.cc/2023/Conference/Paper5619/Reviewer_r2Vc
title: Thank you for your responses
comment: Thank you for the responses, which have addressed most of my concerns. Thus, I increase my scores. Below are some follow-up questions.

**Q1.1**: I appreciate the clarification and realize that HeteroFL, FjOrd, Split-Mix are designed for heterogeneous devices and the proposed method is for low-end devices. But both Split-Mix should be applicable for the setting, where each client only trains a small subnet. Although the subnets are not specialized for each client, I am curious if the specialized subnet can outperform the predefined subnets.

From: ~Annette_Cheng1
title: To reviewers and authors, regarding the motivations and contributions of the work.
comment: Dear authors and reviewers,
I really enjoyed reading the paper and understanding the scope of the work, particularly from the reviewer’s comments. In this context I would like to highlight the following points that I think may further help the evaluation of the current manuscript.

**Regarding the motivation**

Till mid 2021, the issue of training a compute heavy full model in yielding pruned FL model was valid, as pointed out by the authors in the manuscript (in the caption of Fig. 1, in the introduction, in the last sentence of Related Work). However, the authors might have missed significant development in advanced pruning techniques in FL that incorporate sparse learning, example ZeroFL [1] (ICLR 2022), FedDST [2] (AAAI 2022). Both these works ensure only a fraction of the model gets trained at the clients, thus weakening the primary motivation of the work.

**Existing FL work [2] used the same sparse learning, as adapted in this paper**

*RigL's adaptation in FL.* I would like to draw the attention of the reviewers that with similar intention as that of the current manuscript, FedDST[2] adapted the same sparse learning method, RigL, to efficiently prune. Thus to our understanding, efficient adaptation of RigL in FL without the issue of straggler is not new. 

*Efficient gradient update also proposed in RigL[4].* Moreover, as the authors highlighted that their method is more efficient compared to RigL in terms of computing the gradients. I would like to highlight that RigL also has a similar efficient online gradient update policy mentioned (refer to paragraph above Algo. 1 in RigL paper[4]).  Thus, the process of keeping a fraction of gradients for online update, is very similar to updating a fraction of gradients in FL settings (the top-$a_{t,l}$ as mentioned in the rebuttal by the authors).

I now highlight few potential limitations of the current manuscript

**Significant increase in communication cost**

As the server needs to send C models to the clients, also during progressive pruning clients need to have the coarse model (higher density as the fine model), the communication overhead may be significant. Not to mention, the storage cost also gets increased significantly at the client compared to what is needed to have the target pruning budget.

**Utility of the BN adaptation not clear**

From fig. 3, it looks like the BN adaptation is effective only for high sparsity when the accuracy drop is significant (even with adaptive BN), particularly for both CIFAR-10 and CIFAR-100. Based on my understanding of the rebuttal, the problem of significantly low accuracy at high compression is still there. 
Also FedDST[2] leveraged RiGL without the BN adaptation at standard compression with negligible accuracy drop.

Moreover, if BN adaptation plays an important role in the algorithm, many models (that is a use case of the current motivation) don't use BN,  for example VGG (without BN, as proposed in the original manuscript). Also, normalization free large networks [3] has been proven to be significantly effective compared to the ones with BN. This current algorithm is not applicable for the recent models or for the models without BNs.

**Practical use case limitation**

High compression is where the authors demonstrated the benefits. However, at such compression the drop is significant, around 20% pp for CIFAR-100, which is not negligible (as otherwise claimed by the authors) and also not acceptable in safety-critical applications. And medical safety-critical application is one of the wide range use cases of FL. There are other examples as well where accuracy is of high priority. This makes such application space even weaker, as it might be difficult to withstand such a huge drop in performance. At low compression, the method seems to be significantly less effective compared to the shown baselines. Thus it might be significantly hard to find any practically deployable use case of such a method that requires much higher storage cost and communication cost compared to the target compression ratio.

[1] ZeroFL: Efficient On-DeviceTrining for Federated Learning with Local Sparsity, ICLR 2022.
[2] Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better, AAAI 2022.
[3] High performance large-scale image recognition without normalization, ICML 2021.
[4] Rigging the Lottery: Making All Tickets Winners, ICML 2020.

From: ICLR.cc/2023/Conference/Paper5619/Authors
title: Response to Reviewer AyCo
comment: Thank you for your valuable review. We try to summarize your questions and answer them below.

**Q1.** This method may suffer from high communication costs, especially for the adaptive BN selection step.

**A1.** We investigate the communication cost of adaptive BN selection. Given the optimal pool size ($C^*=\frac{0.1}{d_{target}}$), the communication overhead introduced by adaptive BN selection is around 20% of a full-size model. As federated learning needs to transfer the model for a large number of rounds. A one-time 20% model-size communication is very small compared with the entire communication cost. Therefore, we argue that the overhead introduced by adaptive BN selection is marginal. We clarified the cost of the proposed FedTiny in Section 4.4. We include a quantified cost analysis in Appendix C.

**Q2.** The experiments are on small/outdated networks with small datasets.

**A2.** In this work, we focus on specialized tiny models. Hence, we evaluate the proposed FedTiny on a new model, EfficientNet[1], a state-of-the-art neural network model for small devices. We conduct the experiments on CINIC-10 and set the density as 1%. As shown in the table below, FedTiny largely outperforms the existing methods (SynFlow, PruneFL, LotteryFL), which is similar to the result in our paper. The experimental results suggest our proposed FedTiny can be generalized to different models and outperforms the existing methods.

We evaluate a large dataset, CINIC-10, in our work. CINIC-10 includes a total of 270K images, 4.5 times that of CIFAR-10 which was compiled as a bridge between CIFAR-10 and ImageNet. As shown in Figure 2 in the main text, FedTiny also performs better than the existing methods in the CINIC-10 dataset.

| Full-size | SynFlow | PruneFL | LotteryFL | FedTiny |
|:---------:|:-------:|:-------:|:---------:|:-------:|
|   0.725   |  0.6546 |  0.6809 |   0.6661  |  0.7096 |

**Q3.** How to determine the per-layer density is not clear.

**A3.** Given a target density $d_{target}$, We derive the per-layer density $d^l$ by adding target density $d_{target}$ with random noise $e^l$, i.e., $d^l =d_{target}+e^l$ . A candidate is added to the candidate pool only if its density for the entire model $d$ satisfies $d \le d_{target}$. We have clarified the details of per-layer density in Appendix H.

**Q4.** Is taking the weighted average of variances a correct way to combine variances from multiple datasets? 

**A4.** In FedAvg, the variance of batch normalization is aggregated by weighted averaging. Hence, in this work, we use the weighted average of variance following FedAvg.

**Q5.** I don't understand how the top-a_t gradients of pruned parameters are computed. Because of the existence of the fixed binary mask, the gradients of pruned weights are zero.

**A5.** To calculate the gradients of pruned parameters, we follow the common procedure in RigL[3]. When one parameter $x$ is pruned (is masked), its value is set as $x \cdot m$. When we want to calculate gradients for pruned parameters, we actually calculate gradients of the masked value $\frac{\partial f(x \cdot m)}{\partial x \cdot m}$.  Although  $x \cdot m$  is set to 0 due to pruning, the gradients of pruned weights $\frac{\partial f(x \cdot m)}{\partial x \cdot m}$ will not be 0.

**Q6.** Doing layer-wise updates may slow down convergence. The impact of this method on convergence speed and running time is not reported. 

**A6.** We investigate the impact of different types of updates, including layer-wise updates, block-wise updates, and model-wise updates. If the updating granularity is too small (e.g., layer-wise update), the model structure will converge slowly, and the optimal structure cannot be achieved with limited training resources. But high updating granularity leads to more intensive computation in one round. We find that pruning a block per round is an optimal choice for the progressive pruning module. We have included the results in appendix D.

**Q7.** According to the experimental results, it seems that this method usually outperforms related works in the extra low-density region. I am not sure whether there are lots of these cases in practice.

**A7.** Many edge devices, in the real world, are still equipped with very limited memory space.  For example, the memory size of security cameras SoC (e.g., Fullhan) is only about  64MB [2]. However, it requires more than 1.5GB of memory to train a full-size VGG11 model for computer vision tasks. Therefore, the pruning model in the extremely low-density region (e.g., 0.001) still has significant applications in practice.


[1] Tan, Mingxing, and Quoc Le. "Efficientnet: Rethinking model scaling for convolutional neural networks." International conference on machine learning. PMLR, 2019.

[2] https://www.fullhan.com/en/index.php?c=article&a=type&tid=9

[3] Evci, Utku, et al. "Rigging the lottery: Making all tickets winners." International Conference on Machine Learning. PMLR, 2020.

From: ICLR.cc/2023/Conference/Paper5619/Authors
title: Response to Reviewer mNhR 
comment: Thank you for the insightful comment. Below, we address the points you raised.

Q1. There is not enough analytical or empirical evidence in the paper to support that the proposed method is more memory efficient compared to other pruned FL variants.

A1. In progressive pruning, clients create a buffer that stores the top-$a_{t,l}$ gradients only, other gradients will be discarded immediately once they are computed. Other pruning methods can not discard importance scores like ours, since their all importance score will be used in further processing. More importantly, FedTiny achieves much higher accuracy compared with the other FL pruning methods at the same level of model density. For example, using VGG11 with 0.1% density, FedTiny achieves 33% higher accuracy than the best of the existing pruning methods, while requiring 0.9% memory cost and 2.4% computation cost. We have clarified this in Section 4.2.

Q2. The BN stats adjustment step adds additional overhead which is not present in prior works. Can the authors please quantify this overhead and provide an end-to-end cost analysis difference with prior work?

A2. We investigate the communication and computation cost of adaptive BN selection. First, given the optimal pool size ($C^*=\frac{0.1}{d_{target}}$), the communication overhead introduced by adaptive BN selection is around 20% of a full-size model. As federated learning needs to transfer the model for a large number of rounds. A one-time 20% model-size communication is very small compared with the entire communication cost. Second, the extra FLOPs in adaptive BN selection are less than one round of sparse training. Since federated learning usually involves more than one hundred rounds of training, the extra computational overhead is neglectable. Therefore, we argue that the overhead introduced by adaptive BN selection is marginal. We clarified the overhead of the proposed FedTiny in Section 4.4. We also included a quantified cost analysis in Appendix C.

Q3. A Big O analysis of the computation cost differences between the two techniques for one round of training 

A3. In progressive pruning, we prune a number of parameters and grow the same number of parameters. The major memory cost lies in the growing process. We assume $n$ parameters have been pruned in the model and we need to grow $m$ parameters from them, where $m \ll n$. The existing works need to calculate the importance scores for all $n$ pruned parameters. Since the existing work can not discard the importance scores during the calculation, it needs $O(n)$ memory space. In our approach, we set a buffer that can store the $m$ gradient. Once a gradient is calculated, it will be verified whether its magnitude is larger than the smallest gradient magnitude in the buffer. If it does, it will be pushed into the buffer, otherwise, it will be discarded. Therefore, the memory complexity of the proposed method is $O(m)$. For example, in the experiments, existing methods need to store 10M importance scores in ResNet18 with 1% density, but FedTiny just needs to store 10K importance scores.  We have clarified this in Section 3.4

Q4. Quantify the memory saving, clarify where it is coming from, and how it differs from the memory consumption in existing works.

A4. As we have mentioned in A3, although the gradients should be fully computed, the memory just needs to store $a_{t,l}$ gradients during the computing. Clients will create a buffer that only stores $a_{t,l}$  gradients. once a new gradient is calculated, it will be verified whether its magnitude is larger than the smallest gradient magnitude in the buffer. If it does, it will be pushed into the buffer. Otherwise, it will be discarded immediately. In the experiment, using VGG11 with 0.1% density, FedTiny only requires 0.9% memory cost of the best existing pruning methods, while achieving 33% higher accuracy. We have clarified this in Section 4.2.

From: ICLR.cc/2023/Conference/Paper5619/Authors
title: Response to Reviewer X987 
comment: Thank you for your valuable suggestions. We carefully and thoroughly revise the paper to address the issues. The revision is marked in blue in the revised version. 

We highlighted the main points in the experimental sections and make our major findings more clear to read.

We remove redundant or overused sentences to make the paper more precise.

Your suggestions are most welcome!

From: ICLR.cc/2023/Conference/Paper5619/Authors
title: Part 2 of Response to Reviewer r2Vc
comment: **Q 2.1** The proposed method outperforms baselines only with a very low density like 1%. Maybe a small network can be more effective.

**A 2.1** Although LotteryFL can partially outperform FedTiny under high density, it requires more resources to process dense models, as shown in Table 1. We investigate the performance of the small network. We train a small network with 3 convolutional layers. First, we evaluate the small network with a similar number of parameters to ResNet18 with 1% density on different datasets. Second, we evaluate the small network with a similar number of parameters to ResNet18 with different densities on CIFAR-10.

The experimental results show that the small network is competitive compared to other baselines. However, our proposed FedTiny achieves much better performance compared with the small network, which demonstrates the advantage of FedTiny. We have included these experimental results in Appendix F.

|  Dataset  | SynFlow   | LotteryFL   | Small Network   |  FedTiny |
|:---------:|:---------:|:-----------:|:---------------:|:--------:|
|  CIFAR-10 |  0.8034   |   0.8262    |     0.8019      | 0.8523   |
|  CINIC-10 |   0.6057  |    0.6379   |      0.5578     | 0.6712   |
|    SVHN   |   0.8683  |    0.8927   |     0.8395      | 0.8826   |
| CIFAR-100 |   0.4413  |    0.4373   |      0.4277     |  0.4865  |

Table A. The accuracy for ResNet18 with 1% density and a small network on different datasets.

| Density | SynFlow | LotteryFL | Small Network   | FedTiny |
|:-------:|:-------:|:---------:|:---------------:|:-------:|
|   0.01  |  0.8034 |   0.8262  |      0.8019     |  0.8523 |
|  0.005  |  0.7206 |   0.736   |      0.7201     |  0.7972 |
|  0.003  |  0.6279 |   0.6453  |      0.6921     |  0.7572 |
|  0.001  |  0.2862 |   0.2955  |      0.6158     |  0.6311 |

Table B. The accuracy for ResNet18 and small networks with different densities on the CIFAR-10

**Q 2.2** The Fl setting is not convincing by using 10 clients.

**A 2.2** We conduct federated learning with 100 clients to verify the scalability of our proposed FedTiny. We present the accuracy of FedTiny on the CIFAR-10 dataset with ResNet18 in 1% density. The experimental results show that FedTiny outperforms other baselines in 100-client settings. Compared to SynFlow’s 59.25%, PruneFL’s 62.08%, and LotteryFL’s 56.80%,  FedTiny gets 71.12% accuracy and still achieves the best performance compared with the existing pruning methods. We have included the experimental results in Appendix G. 

**Q3.** how the proposed progressive pruning differs from RigL and what is the limitation of RigL directly applied in FL.

**A3.** We argue that there is a critical limitation of directly applying RigL in FL. RigL needs to compute gradients for all parameters, which is computationally expensive and may lead to straggling issues in federated learning. To reduce intensive computation over the gradients, the proposed progressive pruning only computes gradients for partial model parameters (e.g., gradients in one layer) for each communication round. The experimental results show that our proposed FedTiny can still achieve high accuracy while reducing computational costs. We have clarified the difference between RigL and our work in Section 2

From: ICLR.cc/2023/Conference/Paper5619/Authors
title: Part 1 of Response to Reviewer r2Vc
comment: We appreciate your valuable suggestions. We address the points you raised.

**Q 1.1** Pruning is not the only technique to achieve this goal in federated learning and why do people have to adopt pruning in FL? The authors should also discuss related work including HeteroFL, FjOrd, Split-Mix which are also designed for training heterogeneous networks for resource-constrained edge devices. 

**A 1.1** We agree with the reviewer that pruning is not the only technique in federated learning. Pruning can be further integrated with other orthogonal methods such as quantization to reduce the precision of model parameters and to improve learning efficiency in federated learning.  In the paper, we focus on using neural network pruning to identify a specialized tiny model structure for memory- and computing-constrained devices.
We argue that the goal of the related works (e.g., HeteroFL, FjOrd, Split-Mix) is different from ours. These works aim to handle the resource heterogeneity in federated learning to enable collaboration between high-end and low-end devices. Different from these works, we assume all the devices are low-end with limited memory and computing resources. We aim to find a specialized structure for all the low-end devices. In addition, high-end devices with sufficient computing resources used in the related works are not available in our settings. We have revised Section 1 and Section 3.1 to make our setting clear.

**Q 1.2** Does the distributional bias cause the pruning to be biased? I did not see solid evidence to support the conjecture.

**A 1.2** Neural network pruning requires training data to determine the proper model structure. However, due to privacy concerns in federated learning, the server cannot access the clients' training data. Due to the heterogeneous data distributions over clients, the pruning becomes biased. Therefore, our strategy is to use adaptive BN selection to select one pruned model with less bias. 
 
We add a new experiment to demonstrate the impact of data heterogeneity on pruning in Appendix E.  We train the models on CIFAR-10 with ResNet under different non-iid degrees. We set different non-iid degrees by using different $\alpha$ in the Dirichlet distribution. Lower $\alpha$ indicates a higher Non-IID degree. 

Our experiments show that 1) the performance of the existing pruning methods (e.g., SynFlow, LotteryFL) in Federated Learning will be significantly degraded given a higher non-iid degree; 2) Our proposed FedTiny mitigates the bias in pruning and achieves the best performance compared with the existing pruning methods.
We have clarified the details of heterogeneous data in Section 1 and included this experiment in Appendix E.

| $\alpha$ | SynFlow | LotteryFL | FedTiny |
|:--------:|:-------:|:---------:|:-------:|
|     1    |  0.8597 |   0.855   |  0.8642 |
|    0.7   |  0.8191 |   0.8303  |  0.8541 |
|    0.5   |  0.8034 |   0.8262  |  0.8523 |
|    0.3   |  0.772  |   0.8023  |  0.8317 |

From: ICLR.cc/2023/Conference/Paper5619/Reviewer_r2Vc
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper considers the pruning in FL. The major challenges are biased global pruning and the inefficiency of pruning large models on local data. To address these challenges, the authors proposed a novel method FedTiny with two strategies: one is BN selection and progressive pruning. Experiments on 4 datasets and two models demonstrate that FedTIny can improve accuracy with a lower level of memory and computational costs.
strength_and_weaknesses: 
Strength:
1. Efficiency of model inference is critical for edge devices with limited resources.
2. The proposed method is evaluated in multiple dimensions, including accuracy, FLOPs, and memory footprints, which are critical for edge devices.
3. The adaptive BN selection is novel for addressing the bias for data-heterogenous edge devices, when local evaluation of pruned models could be biased by local data distributions.
4. Progressive pruning could greatly reduce the memory footprint and computation costs for pruning a large model. This method could make it possible to carry out federated learning on memory-constrained devices.

Weakness:
1. The motivation for the method is unclear, which is in two aspects.
- 1.1 Though training small networks suitable for edge devices is important, pruning is not the only technique to achieve this goal in federated learning and why people have to adopt pruning in FL. The authors should also discuss related work including HeteroFL [A], FjOrd [B], Split-Mix [C] which are also designed for training heterogeneous networks for resource-constrained edge devices. I suppose that pruning has a finer-grained strategy for providing better small networks personalized for edge devices. However, I did not see experiments to show how much improvement could be gained through fine-grained pruning compared to the coarse-grained model personalization, like HeteroFL, FjOrd, and Split-Mix. Therefore, I don't think that the motivation to focus on pruning in FL is reasonable.
- 1.2 The proposed method is motivated by the conjecture that data heterogeneity could result in pruning bias, but I did not see how this happens. The authors propose adaptive BN selection to address the bias. Does the distributional bias cause the pruning to be biased? I did not see solid evidence to support the conjecture.
2. The empirical results are weak. 
- 2.1 Only when models are very sparse like 1% parameters are maintained, the proposed method outperforms baselines. However, in this case, the accuracy significantly drops compared to the full-size models, for example, by 20%. I doubt if the small network could be effective in practice.
- 2.2 The Fl setting is not convincing by using 10 clients, which is a typical cross-silo setting. In the cross-silo setting, it is not reasonable to say the device is resource-limited when each node could collect a large number of samples. Therefore, the motivation for pruning models sacrificing accuracy is substantially undermined.
3. As prorgessive pruning is inspired by RigL, the authors should elaborate on how the proposed progressive pruning differs from RigL and what is the limitation of RigL directly applied in FL.

[A] Diao, E., Ding, J., & Tarokh, V. (2020). HeteroFL: Computation and communication efficient federated learning for heterogeneous clients. ICLR.

[B] Horvath, S., Laskaridis, S., Almeida, M., Leontiadis, I., Venieris, S., & Lane, N. (2021). Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. NeurIPS.

[C] Hong, J., Wang, H., Wang, Z., & Zhou, J. (2022). Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization. ICLR.


clarity,_quality,_novelty_and_reproducibility: Clarity: The motivation of the methodology is not clear. Why pruning is considered but not other heterogeneous architecture designs in FL? The meaning of coarse pruning is not explained in the introduction and method sections. Therefore, it is hard to understand why coarse pruning could result in bias.

Quality: The evaluation is flawed. The FL setting with 10 clients (cross-silo setting in the FL domain) is not a proper scenario for resource-limited edge devices. More clients, like 100 to 1000 clients are more proper.

Novelty: Some parts of the method is novel. The adaptive BN selection is novel. The progressive RigL seems a straightforward application of RigL.

Reproducibility: The authors provided the pseudo codes and details of hyper-parameters, which should merit reproducibility. But considering the complexity of the algorithm, I encourage the authors to publish codes in the future.
summary_of_the_review: The paper considers a critical problem in federated learning, where client devices may have limited resources for executing or training large models. However, the motivation for the method is unclear and the evaluation is flawed for the assumption of resource-limited devices. Missing literature in the related work (like HeteroFL) and comparison weaken the motivation of the method, too. The novelty is a concern as the difference between the proposed progress pruning and the RigL (Evic, et al., 2020) is not clarified. The empirical evaluation is conducted in an improper setting, with a very small set of clients (typically cross-silo FL), when these clients typically are not short at computation resources, conflicting with the motivation of the paper.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5619/Reviewer_X987
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The authors propose a distributed pruning technique, FedTiny, to generate tiny models for Federated Learning clients which are heterogeneous and resource-constrained. In contrast to the infeasibility of server-based pruning model which relies on importance scores from clients, FedTiny pushes pruning to the edge devices

FedTiny is based on

(i) initial model derived by adaptive batch norm for deployment 

(ii) lightweight progressive pruning of the initial model
strength_and_weaknesses: **Strengths**
- The paper is well-motivated with related studies and extensive experiments. 
- The authors use popular techniques in a unique FL setting to address the challenges associated with resource-constrained devices with varying access to private data.


**Weaknesses**
- It would be helpful for clarity if each paragraph highlights the main points discussed or observed, especially in experimental section.
- At several places, the text is repeated in some form or other. While it is understandable to re-iterate the main points, but here it seems to be overused and redundant.
clarity,_quality,_novelty_and_reproducibility: The paper could be improved in terms of clarity and highlighting main points/outcomes. The paper is application of popular schemes in pruning literature applied in a unique FL setup.
summary_of_the_review: This work targets pruning global model for FL clients with goal on reducing communication overhead with sparse and cheap local computations.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5619/Reviewer_mNhR
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes a pruning technique for federated learning to reduce the training overhead. The server first queries the clients' batch normalization statistics, which are then used to course-prune the initial model and generate several candidates. The candidates are then sent to the users to find the one most suitable for the majority of the users. The FL training is then conducted on this coarse-grained model. Every few iterations the pruning mask is adjusted to make the model more sparse amid training, aligned with the user data.
strength_and_weaknesses: Strengths:
- The paper is well-written and easy to follow.
- The idea of using BN statistics to adjust the original pruned model on the server is interesting.
Weaknesses: It seems like the main contribution in this work is the BN statistics adjustment step for the initial coarse pruning on the server side. The authors claim the proposed method is more memory efficient compared to other pruned FL variants, however, there is not enough analytical or empirical evidence in the paper to support this claim. Please see below for more details.
- The BN stats adjustment step adds additional overhead which is not present in prior works. Can the authors please quantify this overhead and provide and end-to-end cost analysis difference with prior work?
- The authors mention the high cost of computing importance scores as a shortcoming for prior work at several points in the paper. However, it is not clear what overheads the score computation adds to the original training (forward + backward pass on the batches) and what exactly the bottlenecks are for importance score computation, and more importantly, how the proposed method in the paper avoids them. Perhaps a Big O analysis of the computation cost differences between the two techniques for one round of training can help clarify.
- The authors mention that the proposed method provides "memory saving". In Equation 3, all theta_{t+E} values must be fully computed and sent to the server to compute the I drop (line 9 of algorithm 2). Therefore, there seems to be no "memory saving in the communication" other than the saving from the sparsity of the model (which is also available in other prior work that do pruned FL). Additionally, the gradients should be fully computed before the top a_{t,l} values can be identified and removed. Therefore, there also seems to be no "memory saving during training", other than the sparsity of the model (which is again equal to the sparsity of the pruning masks m_t and not specific to this particular proposed work). So can the authors please quantify the memory saving, clarify where it is coming from, and how it differs from the memory consumption in existing works?

clarity,_quality,_novelty_and_reproducibility: While the proposal for adaptive BN adjustment seems novel, the remaining process performed during the training is not. Specifically, the proposed method simply calculates the gradients, removes the bottom a_t values based on the magnitude, and sends the remaining unpruned values to the server to be accumulated. This method of magnitude-based pruning has been studied in many prior works (see for example [1]) and is not novel on its own. There are also some concerns about the claims made regarding the memory savings of the proposed training compared to prior pruned FL works, as detailed in the weaknesses above.

[1] Lin, Yujun, et al. "Deep gradient compression: Reducing the communication bandwidth for distributed training." ICLR 2018.
summary_of_the_review: Please see the weaknesses listed above.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5619/Reviewer_AyCo
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper aims to prune a network under the application of federated learning. It starts with selecting the best coarsely-pruned architecture from a pool. For comparing architectures in the pool, only their means and the variances in batch normalization layers are updated. This selected architecture is then adjusted by growing and pruning the same amount of parameters from a chosen block per federated round.
strength_and_weaknesses: Strength:
- This paper is easy to read and understand.
- This paper is well-motivated and addresses an important issue of federated learning.
- The proposed method makes sense to me.
- The experimental results demonstrate better performance when compared to related works on the low-density region.

===============================================================

Weakness:
- This method may suffer from high communication costs, especially for the adaptive BN selection step. It requires sending 2C models to clients, where C is the number of models in the pool. For example, under the same experimental setting, it requires sending 100x more models to clients. Also, given the fact that other methods do not have this step, this step leads to extra computation costs.
- The experiments are on small/outdated networks (ResNet-18, VGG11) with small datasets (CIFAR-10/100, CINIC-10, SVHN). It is not clear whether the proposed method can also achieve good performance on modern networks with much larger datasets in practice. For example, as the network gets larger, the number of possible candidates grows exponentially. A small C may not give good results while a large C may result in unacceptable communication costs.
- How to determine the per-layer density is not clear. The paper only states "determined by the pruning schedule".
- Is taking the weighted average of variances a correct way to combine variances from multiple datasets? Why does it make sense?
- I don't understand how the top-a_t gradients of pruned parameters are computed. Because of the existence of the fixed binary mask, the gradients of pruned weights are zero.
- This method seems to be more complicated than other methods. Moreover, doing layer-wise updates may slow down convergence. The impact of this method on convergence speed and running time is not reported.
- According to the experimental results, it seems that this method usually outperforms related works in the extra low-density region. I am not sure whether there are lots of these cases in practice.
clarity,_quality,_novelty_and_reproducibility: Clarity and Quality:
Well written and easy to understand.

===============================================================

Novelty:
Marginally novel.

===============================================================

Reproducibility:
Yes.
summary_of_the_review: This is an interesting paper and may be useful for on-device federated learning under strict resource limitations. However, its higher accuracy seems to be an outcome of much higher communication costs and slower convergence rates. Moreover, whether its good performance can still hold in practice is susceptible.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold