Paper: FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity (ICLR-2024)

Reviewer comment (Theory):
Absence of Theoretical Insights: This work seems lacks a strong theoretical foundation or interpretation.

Author response:
We are currently working on the theoretical analysis and will provide a response before the conclusion of this author-reviewer period.

Reviewer comment (Experiments):
More Ablations: The current model does not explore all potential ablations, for instance, different data aggregation techniques.

Author response:
We acknowledge this point and have included the results and analysis of various FL aggregation strategies in Section 4.3.3 of our revised paper.

Reviewer comment (Evaluation):
The authors claim that this method is "privacy-preserving" and designed to maximise privacy overall. At the same time, there is no evaluation at all with respect to privacy (either analytical or through some empirical attacks). Furthermore, the privacy aspect is not discussed, practically assuming that fewer layers sent -> more privacy. While there might be some correlation there, I would expect these claims to be backed up with some thorough evaluation/analysis.

Author response:
Our "privacy-friendly" method enhances privacy by not sharing the complete network architecture from the client to the server. This strategy effectively hides critical architecture details, reinforcing our privacy stance, which we support analytically. The training process is designed to extract features using only a subset of layers, rather than relying on the entire network that memorizes local data. By transmitting only a limited selection of layers in each iteration, our approach becomes more privacy-friendly.

It is important to note that the concept of gradient pruning as a means of preserving privacy was initially popularized by the foundational work of Zhu et al. in "Deep leakage from gradients" (NeurIPS, 2019). Subsequently, studies such as "Privacy-preserving learning via deep net pruning" by Huang et al. (arXiv:2003.01876, 2020) have further explored the effectiveness of DNN pruning in maintaining privacy. Our framework presents a broad scope for meaningful exploration and lays the groundwork for future research. However, the quantitative assessment of privacy in FL remains a complex and open issue. Exploring the privacy dimensions of our framework is an exciting and valuable direction for forthcoming studies.

Reviewer comment (Evaluation):
Similarly, the authors claim that resources are saved (energy, memory, CPU) and larger models can be trained. But there is no evaluation with respect to any such savings. There are no numerical results regarding any energy savings, the memory consumption savings, or understanding how wide the heterogeneity can be with respect to device capabilities. Finally, it would be great to have a thorough study on the convergence speed.

Author response:
We specify that clients in our study have a fixed number of layers, not equal parameters from fully-trained layers, mirroring a heterogeneous environment with diverse memory and communication capacities. For example, in CIFAR10, the first convolutional layer contains 4,864 parameters, contrasting sharply with the 1,638,400 parameters in the first fully-connected layer. As demonstrated in Appendix B.4 (Figure 6), clients training FC2+FFC layers, in comparison to those training Conv1+FFC layers, face communication costs that are over 10,815 times higher. Additionally, they encounter 24.91\% and 57.93\% increases in the sizes of deployed models at global pruning ratios of 0.7 and 0.5, respectively. Section 4.3.3 delves into the assignment of varying layer numbers to clients, analyzing different weighting strategies and detailing our experimental approach.

Our primary experiments with CNNs (2 Conv and 4 FC layers) on CIFAR-10/100 and Fashion-MNIST demonstrate network size reduction. In OPU2, where the last FC layer and two randomly chosen layers (2Conv+2FC) are trained without global or local pruning, we find a 40\% reduction in parameter communication compared to the full FedAvg model. OPU3, with 20\% less parameter communication, matches FedAvg's performance. Dual pruning significantly decreases both the deployed (global pruning) and locally trained (local pruning) model sizes. Figure 4 highlights our method's efficiency for larger models in collaborative training, with a detailed quantitative analysis in Figure 6. The revised paper's Section 4.2 also presents an in-depth analysis of ResNet18. 

We are currently working on the theoretical analysis and will provide a response before the conclusion of this author-reviewer period.

Reviewer comment (Experiments):
Given that there are a number of sub-model FL training methods proposed (the authors cite a few), it would be great if the evaluation could be expanded to compare with the state of the art.

Author response:
Our framework is not only versatile but also well-suited for integrating existing dual-pruning methods. As an illustration, Table 2 in our paper showcases the effective incorporation of the Ordered Dropout method from FjORD (NeurIPS21) into our system. We have enriched the revised version of our paper with an extended related work section, offering a comprehensive comparison with a wide array of relevant studies, as detailed in Appendix A.

Furthermore, our analysis extends beyond mere comparisons with the standard FedAvg algorithm. We have undertaken a thorough comparative study with the cutting-edge algorithm FedCR. The details of this comparison, highlighting the efficacy and advancements of our framework in relation to state-of-the-art methods, are meticulously presented in Figure 2 of Section 4.2. This set of comparisons underscores the robustness and innovation of our framework in the context of FL.

Reviewer comment (Writing):
The relation between privacy-friendly property and network pruning is implicit. It is better to provide a detailed illustration of the reason why existing pruning-based FL methods have privacy concerns. This can help readers to better understand the motivation of the privacy-preserving part.

Author response:
Firstly, it is important to note that the concept of gradient pruning as a means of preserving privacy was initially popularized by the foundational work of Zhu et al. in "Deep leakage from gradients" (NeurIPS, 2019). Subsequently, studies such as "Privacy-preserving learning via deep net pruning" by Huang et al. (arXiv:2003.01876, 2020) have further explored the effectiveness of DNN pruning in maintaining privacy.

In our setting, we ensure that our training process focuses on extracting partial features without relying on all layers to memorize local training data. This is achieved by transmitting only a limited subset of layers from the client to the server in each iteration. Under this approach, sending fewer layers—effectively implementing more pruning from clients to the server—enhances the privacy-friendliness of our framework.

Reviewer comment (Experiments):
The authors have emphasized the possibility of utilizing the proposed method in LLMs-based scenarios but there is a lack of related experiments supporting that point. Only shallow neural networks and ResNet are considered in the experimental part.

Author response:
We appreciate the interest in applying our framework to scenarios involving LLMs, and we plan to address this in future work. The main objective of this paper is to validate the effectiveness of our novel framework and to ascertain whether existing best practices in FL that are orthogonal to our approach can be successfully integrated.

Reviewer comment (Presentation):
The presentation of the paper should be further enhanced and some parts should be reorganized. For example, the training goal and variable notation should be given in the Preliminary or Methodology section rather than the Introduction section.

Author response:
In the Introduction, we introduced two optimization objectives: eq (1) illustrates the standard finite-sum, while eq (2) represents our optimization goal in a high-level sense. Our intention is for the comparison of these two objectives to offer intuitive insights to readers. Starting with the standard FL objective and continuing through the related work section, we aim to furnish essential background information for readers less familiar with FL. Meanwhile, our formal and detailed methodologies are comprehensively outlined in Section 3 - Approach. We've provided a revised version and we remain receptive to suggestions for improving the organization of this content based on further discussions.

Reviewer comment (Experiments):
Lack of empirical comparison with existing FL methods that also address model heterogeneity problem.

Author response:
Firstly, it's important to clarify that our main objective is not necessarily to surpass existing methods addressing model heterogeneity. Instead, our focus is on assessing the viability of our novel framework, which integrates personalized and privacy-friendly network pruning in FL. We are encouraged by extensive experiments that validate the effectiveness of our approach; while there may be some compromise in final testing accuracy, our method notably reduces communication costs and supports flexible dual-pruning, enhancing privacy.

Secondly, the concept of dual-pruning, encompassing both global and local pruning, inherently introduces model heterogeneity. Consequently, we view any pruning method as a potential solution for addressing model heterogeneity. In this respect, our framework is orthogonal to existing approaches, allowing for the safe integration of different pruning methods. The details are provided in Sec 4.3. 

Lastly, we are also keen on exploring performance enhancements beyond the standard FedAvg. Our ablation studies include the state-of-the-art FL method FedCR (ICML23), with detailed experiments and analyses provided in Figure 2 in the revised version. These studies reveal that our framework remains efficient, achieving promising performance and significantly reducing the number of required parameters.

Reviewer comment (Novelty):
To solve the model heterogeneity problem, some existing work needs to be discussed, e.g. Knowledge Distillation-based FL [1], Prototype-based FL [2], and NAS-based FL.

Author response:
We are grateful to the reviewer for suggesting relevant literature. We have incorporated a comparison of these works in Appendix A, titled "Extended Related Work."

Reviewer comment (Reproducibility):
How to conduct federated training of LLMs based on the proposed method?

Author response:
We have not yet performed experimental verifications on federated LLMs, a field that remains in its early stages and likely warrants dedicated projects. Currently, we must clarify that the application of our method to federated training of LLMs is theoretical. A crucial initial step would involve developing a reliable FedAvg-like method specifically for LLMs. Given that our method, FedP3 as detailed in Algorithm 1, is not limited by the architecture of the global model, it offers the potential for integration into federated LLM frameworks. This integration would aim at enabling personalized and privacy-aware pruning, albeit with necessary adaptive adjustments.

Reviewer comment (Novelty):
In the algorithm, predefined pruning mechanisms are assigned for each client. Is this a common operation in FL under model heterogeneity issues? Are there any previous works conducting similar operations?

Author response:
In addressing the question, our focus is twofold: layer-wise assignment for each client and dual-pruning. In our unique scenario, where clients train only a subset of layers, we lack direct references from existing literature. We chose a simple approach where each client trains a fixed set of layers, although our model could, in theory, permit variable layer selection. However, this adjustment may not substantially benefit our contributions. As for dual-pruning, we follow the conventional design of both global and local pruning, maintaining a consistent pruning mechanism without mixing various strategies, in line with standard practices outlined in Section 2.1.

Reviewer comment (Experiments):
There are multiple pruning and averaging strategies that can be adopted in the proposed FedP3. For a specific case, how to select the most appropriate strategy?

Author response:
In Figure 5, our analysis demonstrates that weighted averaging consistently outperforms simple averaging, albeit marginally. Table 2 further illustrates that Order Dropout local pruning consistently falls short in Top-1 accuracy when compared to Uniform strategies. This is likely because Uniform pruning encompasses all weights, thereby providing a more comprehensive global perspective. The versatility and adaptability of our framework with different pruning and averaging techniques open up a wide array of opportunities for more targeted experimentation across various datasets and contexts. A key, consistent observation with our FedP3 method is its ability to achieve comparable performance while significantly reducing communication costs and friendly to privacy.

Reviewer comment (Experiments):
In practice, what if we set several types of models to tackle various heterogeneous devices? Can you set a baseline algorithm to test it?

Author response:
We appreciate the reviewer's interest in training different types of local models. To the best of our knowledge, two possible approaches are knowledge distillation and building model prototypes. However, knowledge distillation operates on loss or other agency, often requiring additional shared data or constraints. Building model prototypes doesn't align with our objective, which is to train a *single* large global model.

When different models with varying semantic meanings of weights are aggregated, as in combining ResNet18 and ResNet50, it presents numerous unresolved challenges. For example, the distinct semantics of each layer in these models render standard mean aggregation impractical. Developing suitable aggregation strategies for such diverse models is complex and could lead to many interesting research projects. While potential new strategies might be adapted to our general framework with some modifications, there is currently a lack of literature on aggregating outputs from local clients with different model types.

In the context of dual-pruning, our FedP3 framework can accommodate various local models. We also acknowledge the importance of mimicking heterogeneous device environments, including varying memory and bandwidth constraints among devices, an approach we have adopted in our research.

Reviewer comment (Experiments):
There is a discrepancy between the experiment setting and the motivation problem setting in the introduction. The experiment setting does not reflect the heterogeneity of memory and communication constraints between clients, even though this is a major motivation mentioned in the introduction. Specifically, in all experiments, different clients train subnetworks roughly the same size.

Author response:
**A1** Thanks for the insightful questions.

**a**: We specify that clients in our study have a fixed number of layers, not equal parameters from fully-trained layers, mirroring a heterogeneous environment with diverse memory and communication capacities. For example, in CIFAR10, the first convolutional layer contains 4,864 parameters, contrasting sharply with the 1,638,400 parameters in the first fully-connected layer. As demonstrated in Appendix B.4 (Figure 6), clients training FC2+FFC layers, in comparison to those training Conv1+FFC layers, face communication costs that are over 10,815 times higher. Additionally, they encounter 24.91\% and 57.93\% increases in the sizes of deployed models at global pruning ratios of 0.7 and 0.5, respectively.

**b**: Our primary experiments with CNNs (2 Conv and 4 FC layers) on CIFAR-10/100 and Fashion-MNIST demonstrate network size reduction. In OPU2, where the last FC layer and two randomly chosen layers (2Conv+2FC) are trained without global or local pruning, we have a 40\% reduction in parameter communication compared to the full FedAvg model.
OPU3, with 20\% less parameter communication, matches FedAvg's performance. Dual pruning significantly decreases both the deployed (global pruning) and locally trained (local pruning) model sizes. Figure 4 highlights our method's efficiency for larger models in collaborative training, with a detailed quantitative analysis in Figure 6. The revised paper's Section 4.2 presents an in-depth analysis of ResNet18.

Reviewer comment (Experiments):
The subnetwork is not significantly smaller than the global model. For the ResNet-18 experiments, the subnetwork of each client appears to be at least around half the scale of the global model. This differs from the interesting scenario mentioned in the introduction, where each subnetwork is significantly smaller than the global model.

Author response:
**A1** Thanks for the insightful questions.

**a**: We specify that clients in our study have a fixed number of layers, not equal parameters from fully-trained layers, mirroring a heterogeneous environment with diverse memory and communication capacities. For example, in CIFAR10, the first convolutional layer contains 4,864 parameters, contrasting sharply with the 1,638,400 parameters in the first fully-connected layer. As demonstrated in Appendix B.4 (Figure 6), clients training FC2+FFC layers, in comparison to those training Conv1+FFC layers, face communication costs that are over 10,815 times higher. Additionally, they encounter 24.91\% and 57.93\% increases in the sizes of deployed models at global pruning ratios of 0.7 and 0.5, respectively.

**b**: Our primary experiments with CNNs (2 Conv and 4 FC layers) on CIFAR-10/100 and Fashion-MNIST demonstrate network size reduction. In OPU2, where the last FC layer and two randomly chosen layers (2Conv+2FC) are trained without global or local pruning, we have a 40\% reduction in parameter communication compared to the full FedAvg model.
OPU3, with 20\% less parameter communication, matches FedAvg's performance. Dual pruning significantly decreases both the deployed (global pruning) and locally trained (local pruning) model sizes. Figure 4 highlights our method's efficiency for larger models in collaborative training, with a detailed quantitative analysis in Figure 6. The revised paper's Section 4.2 presents an in-depth analysis of ResNet18.

Reviewer comment (Writing):
Algorithm descriptions sometimes need more clarity; see question 3 for more details.

Author response:
**A3**: Thanks for your valuable questions. Our response is as follows:

**a**: We would like to clarify that this assertion is not correct. Our focus is on retaining mainly the *pruned* global model layers and training a smaller portion of the *unpruned* fully-trained layers. We substantiate this with quantitative analysis in the right panel of Figure 4 and throughout Figure 6.

**b**: Correct. Our primary aim is to showcase the feasibility of training large models with reduced communication costs in a privacy-friendly manner. Consequently, our method does not utilize pruned weights for each client.

**c**: Equation (2) outlines our process in a general form. We have not specified the function $h$ here, as it can refer to various aggregation strategies. More details are provided in Algorithm 3 and Section 4.3.3.

Reviewer comment (Experiments):
Could the authors comment on whether the algorithm would perform well if the subnetworks trained by individual clients are significantly smaller than the global model? Table 2 shows that the algorithm performs poorly when each client only trains one layer.

Author response:
**A4**: We acknowledge some potential misunderstandings. average communication cost reduction of 20\% for OPU3, 40\% for OPU2, and 60\% for LowerB. We provide a detailed analysis in our revised paper. We treat LowerB as the lower bound benchmark for providing analytical insights. OPU3, notably, delivers performance on par with FedAvg, yet requires only 80\% of the parameter communication. Specifically, LowerB exhibits an average 8.53\% drop in performance but benefits from 60\% lower communication costs. The size of the locally deployed and trained model parameters is quantitatively analyzed in the right part of Figure 4 and throughout Figure 6. These figures demonstrate that our method achieves significantly smaller deployed model sizes, effectively balancing accuracy trade-offs, particularly when the global model is large.

Reviewer comment (Experiments):
Could the authors comment on an alternative approach of global model pruning + standard FL on the pruned model?

Author response:
**A2**: We actually denote this baseline as to the "Full" method depicted in Table 1 and the "Fixed" method in Table 2. We have updated their captions to clarify this. By default, we set the global pruning ratio to 0.9, which entails transferring approximately 10\% fewer parameters from the server to the client. Both "Full" and "Fixed" represent the implementation of standard FedAvg during local training and subsequent model aggregation. Further details are available in Sections 4.3.1 and 4.3.2 of our revised paper.

Reviewer comment (Presentation):
Several parts of the algorithm could be clarified more. Figure 1 shows that each client still needs to store a large proportion of the unpruned global model. Is that correct?

Author response:
**A3**: Thanks for your valuable questions. Our response is as follows:

**a**: We would like to clarify that this assertion is not correct. Our focus is on retaining mainly the *pruned* global model layers and training a smaller portion of the *unpruned* fully-trained layers. We substantiate this with quantitative analysis in the right panel of Figure 4 and throughout Figure 6.

**b**: Correct. Our primary aim is to showcase the feasibility of training large models with reduced communication costs in a privacy-friendly manner. Consequently, our method does not utilize pruned weights for each client.

**c**: Equation (2) outlines our process in a general form. We have not specified the function $h$ here, as it can refer to various aggregation strategies. More details are provided in Algorithm 3 and Section 4.3.3.

Reviewer comment (Writing):
Several parts of the algorithm could be clarified more. In Algorithm 1, the pruned weights \(P_i(W_t^l)\) are not used anywhere later.

Author response:
**A3**: Thanks for your valuable questions. Our response is as follows:

**a**: We would like to clarify that this assertion is not correct. Our focus is on retaining mainly the *pruned* global model layers and training a smaller portion of the *unpruned* fully-trained layers. We substantiate this with quantitative analysis in the right panel of Figure 4 and throughout Figure 6.

**b**: Correct. Our primary aim is to showcase the feasibility of training large models with reduced communication costs in a privacy-friendly manner. Consequently, our method does not utilize pruned weights for each client.

**c**: Equation (2) outlines our process in a general form. We have not specified the function $h$ here, as it can refer to various aggregation strategies. More details are provided in Algorithm 3 and Section 4.3.3.

Reviewer comment (Writing):
Several parts of the algorithm could be clarified more. In eq (2), the objective function \(h\) does not appear to be defined for the problem considered in this paper.

Author response:
**A3**: Thanks for your valuable questions. Our response is as follows:

**a**: We would like to clarify that this assertion is not correct. Our focus is on retaining mainly the *pruned* global model layers and training a smaller portion of the *unpruned* fully-trained layers. We substantiate this with quantitative analysis in the right panel of Figure 4 and throughout Figure 6.

**b**: Correct. Our primary aim is to showcase the feasibility of training large models with reduced communication costs in a privacy-friendly manner. Consequently, our method does not utilize pruned weights for each client.

**c**: Equation (2) outlines our process in a general form. We have not specified the function $h$ here, as it can refer to various aggregation strategies. More details are provided in Algorithm 3 and Section 4.3.3.

Reviewer comment (Evaluation):
Few-layer communication can also significantly reduce communication costs and save bandwidth, a detailed numerical analysis would help.

Author response:
In the paragraph "Layer Overlapping Analysis" of Section 4.2 of our revised paper, we assess the impact of few-layer communication on reducing costs. Our findings indicate significant savings: 20\% for OPU3, 40\% for OPU2, and 60\% for LowerB on average. Particularly, OPU3 achieves comparable results to FedAvg with only 80\% of the parameters communicated. This method is also applicable to ResNet18 and EMNIST-L datasets. As detailed in Appendix B.4 (Figure 6), clients training FC2+FFC layers, in comparison to those training Conv1+FFC layers, face communication costs that are over 10,815 times higher. Additionally, they encounter 24.91\% and 57.93\% increases in the sizes of deployed models at global pruning ratios of 0.7 and 0.5, respectively.

Reviewer comment (Experiments):
The results on various FL aggregation strategies shall be considered for completeness.

Author response:
Acknowledging this, we've added a detailed examination of various FL aggregation strategies in Section 4.3.3 of our revised paper. We show a slight improvement using weighted averaging based on the relative number of layers on each client.

Reviewer comment (Experiments):
Hyperparameter tuning part is not so crystal. More explanation is needed on how to choose and tune the hyperparameter (maybe via grid-search?) to deliver the best possible results for each model.

Author response:
We optimized key hyper-parameters, including the local training learning rate and the personalized factor of FedCR, through grid search, while maintaining other parameters at default settings for fair comparison. Comprehensive details of the training procedure can be found in Appendix B.3.

Reviewer comment (Novelty):
Broader Literature Review is expected. While this work focuses primarily on the discussion and analysis of the most relevant and typical works, this approach might overlook other pertinent past research that holds tangential relevance to their study.

Author response:
In the Related Work section of our paper, we initially compared our study to the most pertinent existing research within our knowledge. Responding to feedback, we have expanded this section to encompass and contrast additional studies, particularly those highlighted by reviewers, along with other relevant papers. Further details are provided in Appendix A.

Reviewer comment (Evaluation):
How much the few-layer communication can save communication costs and bandwidth?

Author response:
In the paragraph "Layer Overlapping Analysis" of Section 4.2 of our revised paper, we assess the impact of few-layer communication on reducing costs. Our findings indicate significant savings: 20\% for OPU3, 40\% for OPU2, and 60\% for LowerB on average. Particularly, OPU3 achieves comparable results to FedAvg with only 80\% of the parameters communicated. This method is also applicable to ResNet18 and EMNIST-L datasets. As detailed in Appendix B.4 (Figure 6), clients training FC2+FFC layers, in comparison to those training Conv1+FFC layers, face communication costs that are over 10,815 times higher. Additionally, they encounter 24.91\% and 57.93\% increases in the sizes of deployed models at global pruning ratios of 0.7 and 0.5, respectively.

Reviewer comment (Experiments):
How the other FL aggregation strategies work in combination with FedP3?

Author response:
Acknowledging this, we've added a detailed examination of various FL aggregation strategies in Section 4.3.3 of our revised paper. We show a slight improvement using weighted averaging based on the relative number of layers on each client.

Reviewer comment (Experiments):
How to choose and tune the hyperparameter?

Author response:
We optimized key hyper-parameters, including the local training learning rate and the personalized factor of FedCR, through grid search, while maintaining other parameters at default settings for fair comparison. Comprehensive details of the training procedure can be found in Appendix B.3.

Reviewer comment (Experiments):
The outcomes of the experiment need to be made more convincing.

Author response:
**A1**: In response, we have implemented several modifications to enhance the clarity and illustrative quality of our experiments. Details are attached in the general response to all reviewers. Please also find our revised paper with more evidence. 

We trust that these enhancements will make our experiments and analyses more transparent and insightful. We are readily available to address any specific queries regarding the experimental components.

Reviewer comment (Experiments):
Limited in-depth comparison with state-of-the-art solutions.

Author response:
**A2**: We wish to clarify that the primary objective of our paper is not to surpass existing FL algorithms on popular datasets. Instead, our focus is on introducing a general pruning framework that not only allows for personalization but also operates within a privacy-friendly setting. This claim is substantiated by comprehensive experiments conducted throughout the paper.

Given the general nature of our framework, we are also interested in investigating its applicability to cutting-edge personalized FL methodologies. To this end, we have conducted a comparative analysis with the state-of-the-art FedCR (ICML23). The details of our experimental setup, results, and analysis are thoroughly documented in the right part of Figure 2. Our findings are encouraging, indicating that even when applied to FedCR, our framework maintains comparable performance while significantly reducing the volume of parameters communicated.

Reviewer comment (Theory):
Privacy analysis and convergence analysis need to be included.

Author response:
**A3**: Our "privacy-friendly" method enhances privacy by not sharing the complete network architecture from the client to the server. This strategy effectively hides critical architecture details, reinforcing our privacy stance, which we support analytically. The training process is designed to extract features using only a subset of layers, rather than relying on the entire network that memorizes local data. By transmitting only a limited selection of layers in each iteration, our approach becomes more privacy-friendly.
 
It is important to note that the concept of gradient pruning as a means of preserving privacy was initially popularized by the foundational work of Zhu et al. in "Deep leakage from gradients" (NeurIPS, 2019). Subsequently, studies such as "Privacy-preserving learning via deep net pruning" by Huang et al. (arXiv:2003.01876, 2020) have further explored the effectiveness of DNN pruning in maintaining privacy. 

Our framework presents a broad scope for meaningful exploration and lays the groundwork for future research. However, the quantitative assessment of privacy in FL remains a complex and open issue. Exploring the privacy dimensions of our framework is an exciting and valuable direction for forthcoming studies.

Reviewer comment (Experiments):
Empirical verification of the claimed contributions is necessary.

Author response:
**A4**: We would like to emphasize all our claimed contributions are clearly stated with enough evidence. Here is the response for all our contributions:

- **Contribution 1: general framework.** "A versatile design that caters to model heterogeneity, enabling tailored personalization based on clients’ local capacities (computational, memory and communication constraints)."
    - Our general framework, FedP3, detailed in Algorithm 1, uses $L_i$, $P_i$, and $Q_i$ as personalization factors for client $i$, aligning with each client's computational capacity and bandwidth. This framework is central to this paper and the feasibility has been verified through all our experiments.
    
- **Contribution 2: dual-pruning approach.** "A dual-pruning approach that supports both global pruning (from server to client) and local pruning by individual clients."
    - We provide empirical evidence of this methodology throughout Sections 4.3.1 and 4.3.2.
    
- **Contribution 3: Privacy-friendly design.** " A commitment to privacy, ensuring that complete client information is never disclosed to the server as only a few layers are communicated from the clients to the server after local training."
    - Our approach ensures privacy-friendliness by not sending the full network architecture from client to server. The quantitative efficiency of this method is demonstrated throughout our experimental section.
  
- **Contribution 4: handling data and model heterogeneity.** "Comprehensive handling of both data and model heterogeneity. We allow data distributed among clients to be class-wise or Dirichlet non-iid. Meanwhile, we allow the server-client models and client-wise models to be different.
      - Our experiments, detailed in Section 4, focus on non-iid data, including class-wise or Dirichlet cases. After dual-pruning, the local models naturally exhibit heterogeneity, a key focus in our paper. 

We also revised the contribution part in the introduction for better clarity.

Reviewer comment (Theory):
Privacy analysis and convergence analysis need to be included. Indeed, separating model parameters from the network parameter architecture is intuitively beneficial to FL privacy protection, but whether existing gradient reconstruction attacks or other privacy attacks will challenge this personalization technology has not been fully explored. Therefore, it would be better if the authors could analyze the privacy performance empirically or theoretically.

Author response:
**A3**: Our "privacy-friendly" method enhances privacy by not sharing the complete network architecture from the client to the server. This strategy effectively hides critical architecture details, reinforcing our privacy stance, which we support analytically. The training process is designed to extract features using only a subset of layers, rather than relying on the entire network that memorizes local data. By transmitting only a limited selection of layers in each iteration, our approach becomes more privacy-friendly.
 
It is important to note that the concept of gradient pruning as a means of preserving privacy was initially popularized by the foundational work of Zhu et al. in "Deep leakage from gradients" (NeurIPS, 2019). Subsequently, studies such as "Privacy-preserving learning via deep net pruning" by Huang et al. (arXiv:2003.01876, 2020) have further explored the effectiveness of DNN pruning in maintaining privacy. 

Our framework presents a broad scope for meaningful exploration and lays the groundwork for future research. However, the quantitative assessment of privacy in FL remains a complex and open issue. Exploring the privacy dimensions of our framework is an exciting and valuable direction for forthcoming studies.

Reviewer comment (Theory):
The convergence analysis of the proposed method has not given any explanation. For model heterogeneous scenarios, reviewers expect to see rigorous analysis and discussion of the convergence and stability of this method.

Author response:
**A4**: We are currently working on the theoretical analysis and will provide a response before the conclusion of this author-reviewer period.

Reviewer comment (Experiments):
This paper claims that the proposed personalized pruning technique can well alleviate the system heterogeneity and model heterogeneity problems, but the reviewer has not seen any discussion and numerical results on the system heterogeneity scenario.

Author response:
**A5**: There seems to be a misunderstanding regarding systems heterogeneity, which, as defined in reference [a] below, pertains to the varying storage, computational, and communication capabilities of each device. Our paper directly addresses this issue throughout. The number of personalized layers $L_i$, alongside the global pruning mechanism $P_i$ and the local pruning mechanism $Q_i$ presented in Algorithm 1, are specifically designed to cater to system heterogeneity. Intuitively, different clients may need to train a vastly different number of parameters. 

For example, in CIFAR10, the first convolutional layer contains 4,864 parameters, contrasting sharply with the 1,638,400 parameters in the first fully-connected layer. As demonstrated in Appendix B.4 (Figure 6), clients training FC2+FFC layers, in comparison to those training Conv1+FFC layers, face communication costs that are over 10,815 times higher. Additionally, they encounter 24.91\% and 57.93\% increases in the sizes of deployed models at global pruning ratios of 0.7 and 0.5, respectively. Section 4.3.3 delves into the assignment of varying layer numbers to clients, analyzing different weighting strategies and detailing our experimental approach.

Our primary experiments with CNNs (2 Conv and 4 FC layers) on CIFAR-10/100 and Fashion-MNIST demonstrate network size reduction. In OPU2, where the last FC layer and two randomly chosen layers (2Conv+2FC) are trained without global or local pruning, we find a 40\% reduction in parameter communication compared to the full FedAvg model. OPU3, with 20\% less parameter communication, matches FedAvg's performance. Dual pruning significantly decreases both the deployed (global pruning) and locally trained (local pruning) model sizes. Figure 4 highlights our method's efficiency for larger models in collaborative training, with a detailed quantitative analysis in Figure 6. The revised paper's Section 4.2 also presents an in-depth analysis of ResNet18. 

[a] Li, Tian, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. "Federated learning: Challenges, methods, and future directions." IEEE signal processing magazine 37, no. 3 (2020): 50-60.

Reviewer comment (Experiments):
Limited in-depth comparison with state-of-the-art solutions. In fact, personalized pruning technique is not the first time to be applied to FL to solve efficiency and heterogeneity problems. The following literature needs to be included in the baseline solutions and explain the differences and connections between this paper and them.  

[1] Zhou X, Jia Q, Xie R. NestFL: efficient federated learning through progressive model pruning in heterogeneous edge computing[C]//Proceedings of the 28th Annual International Conference on Mobile Computing And Networking. 2022: 817-819.  

[2] Li A, Sun J, Li P, et al. Hermes: an efficient federated learning framework for heterogeneous mobile clients[C]//Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. 2021: 420-437.  

[3] Pase F, Isik B, Gunduz D, et al. Efficient federated random subnetwork training[C]//Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022). 2022.

Author response:
**A4**: Firstly, it's important to clarify that our main objective is not necessarily to surpass existing methods addressing model heterogeneity. Instead, our focus is on assessing the viability of our novel framework, which integrates personalized and privacy-friendly network pruning in FL. We are encouraged by extensive experiments that validate the effectiveness of our approach; while there may be some compromise in final testing accuracy, our method notably reduces communication costs and supports flexible dual-pruning, enhancing privacy.

Secondly, the concept of dual-pruning, encompassing both global and local pruning, inherently introduces model heterogeneity. Consequently, we view any pruning method as a potential solution for addressing model heterogeneity. In this respect, our framework is orthogonal to existing approaches, allowing for the safe integration of different pruning methods. The details are provided in Sec 4.3. 

Lastly, we are also keen on exploring performance enhancements beyond the standard FedAvg. Our ablation studies include the state-of-the-art FL method FedCR (ICML23), with detailed experiments and analyses provided in Figure 2 in the revised version. These studies reveal that our framework remains efficient, achieving commendable performance while significantly reducing the number of required parameters.