From: ICLR.cc/2023/Conference/Paper1005/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: Summary: 

This paper claims that, due to device heterogeneity, the main bottleneck of training efficiency is the workflow in the server. The paper proposes a parallel federated learning (PFL) strategy (which is asynchronous) to decouple the global aggregation in server with the communication between server and client (uploading & broadcasting), so that the server and clients can do global aggregation and local optimization in parallel. A key difference here is, instead of receiving the up-to-date global model, a client now receives a stale model and the model updates are computed based on the stale model. 

Strengths:

1. This paper tackles an important problem, the training inefficiency of federated learning due to device heterogeneity. 
2. The proposed idea is simple to understand and is a straightforward re-ordering of the conventional FL process with swapping of aggregation and broadcast steps on the server side.

Weaknesses:

1. There exist works in the literature that apply asynchronous federated learning. These are not even mentioned in the list of publications. E.g., see the paper "Asynchronous Federated Optimization" for one of the first works on this topic; see paper "Asynchronous Federated Learning on Heterogeneous Devices: A Survey" for a **survey** on asynchronous methods on this topic; see paper "Federated Learning with Buffered Asynchronous Aggregation" and "Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropout" for more recent approaches. The experimental validation does not include at least a subset of these methods (some of the above are available after this paper was submitted, but others are quite old); even if there are no asynchronous methods that the authors are aware of, one option is to turn some existing methods into asynchronous (like asynchronous FedAvg/FedProx/...). This makes the experimental validation incomplete.

2. In addition to the above comment, it is not clear how the current paper differs from existing work (intellectually). This is missed at the current version.

3. There is no theoretical justification about the algorithmic proposal and how it performs on paper.

4. Maybe the AC is mistaken, but the experiments are not complete: There is a lack of discussion on **testing** accuracy; tables mostly focus on training accuracy/efficiency, which is part of the full story. This backs up some concerns that other reviewers had. 

Recommendation:

First of all, the authors should be stayed assured that there were discussions regarding the paper, and the area chair read carefully i) the paper, ii) the reviews, and iii) the discussions after the rebuttal. We acknowledge that the paper addresses concerns raised, but the reviewers have remaining concerns.

Better comparison to existing work (not only empirical but also better placement of this work in comparison to existing async. FL methods) is needed; theoretical support is encouraged. Further, more experimental results that show all the aspects of training and testing phase of learning should be included
justification_for_why_not_higher_score: See above
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper1005/Authors
title: About the novelty of our work
comment: Dear Reviewer Ech9,

Thanks very much for your kind feedback! 

We would like to clarify the contribution of our work. In this paper, we specially study the training efficiency issue of FL over heterogenous devices. We claim that the latency bottlenecks include the network straggler introduced by device heterogeneity as well as the time cost for complicated aggregation. 

 -- For the former, instead of remedying the network latency issue from the edge side, we propose a novel approach by changing the classical aggregation-then-broadcasting workflow of the server.  **By swapping the computational order of aggregation and broadcasting**, we achieve parallel federated learning framework that unlocks the edge nodes during global computation and the central server during local computation. 

-- For the latter,  the training latency induced by complicated aggregation has not been explored yet by the literature. By carefully examining the workflow of classical FedAvg scheme, it can be seen that the server would be blocked to wait for receiving all local updates in order to perform the global aggregation while the edge nodes would be blocked to wait for receiving the broadcasted global model and then can begin the next local optimization. We found that the cause of training inefficiency actually lies in the dependency of server on all local updates uploading and the dependency of clients on global aggregation. It is exciting that we can decouple these dependencies **by swapping the computational order of aggregation and broadcasting**!

-- Without redesigning the whole FL framework, we propose a very simple operation---exchange the execution order of global aggregation and broadcasting, and **achieve a unified FL framework to address these two bottlenecks simultaneously**.

As stated above, the novelty of our work is remarkable. We believe our work could offer new inspiration to the community.


Best,

The authors

From: ICLR.cc/2023/Conference/Paper1005/Reviewer_bvyd
title: Thank you for Clarifications
comment: I appreciate the response provided to my main concerns. I have updated my score to reflect the above.

From: ICLR.cc/2023/Conference/Paper1005/Reviewer_Ech9
title: Good revision, novelty still seems a tad incremental
comment: The responses fixed most of my questions about presentation and flow of the paper. I think the paper is well-executed now. But I still feel the novelty is a tad limited and incremental. I slightly adjusted my score given the revision and response work.

From: ICLR.cc/2023/Conference/Paper1005/Authors
title: Any concerns about our response?
comment: Dear Reviewer,

Thanks again for your valuable time in reviewing our paper. The response to your comments has been posted for three weeks. If you have any concerns about our response, we would be happy to answer them.


Best regards,

The authors

From: ICLR.cc/2023/Conference/Paper1005/Authors
title: Any concerns about our response?
comment: Dear Reviewer,

Thanks again for your valuable time in reviewing our paper. The response to your comments has been posted for three weeks. If you have any concerns about our response, we would be happy to answer them.


Best regards,

The authors

From: ICLR.cc/2023/Conference/Paper1005/Authors
title: Any concerns about our response?
comment: Dear Reviewer,

Thanks again for your valuable time in reviewing our paper. The response to your comments has been posted for three weeks. If you have any concerns about our response, we would be happy to answer them.


Best regards,

The authors

From: ICLR.cc/2023/Conference/Paper1005/Authors
title: Response To Reviewer Ech9
comment: We thank Reviewer Ech9 for the thoughtful comments and suggestions. We address each of your questions below.

**Q1:** *"This paper only introduced the asynchronous PFL in section 2.2. ("Due to page limitation. we only present asynchronous PFL in this section.") However, section 2.3 mainly proves the convergence of synchronous PFL, which is inconsistent with the prior section. Thus, when reading section 2.3, which discusses the synchronous PFL, the terms explained in section 2.2 are no longer applicable. And those terms for synchronous PFL are not explained in section 2.3 a fresh. Therefore, it is hard to understand what the terms in section 2.3 mean. (Even the explanations for synchronous PFL in the appendix are not detailed enough.)"*

**R:** Thanks very much for this valuable comment, which is pretty helpful in improving the readability of our manuscript. As suggested, in Section 2.1 of the revised manuscript, we present the details of both synchronous and asynchronous PFL, as well as their difference; Section 2.2 includes the convergence analysis of both synchronous and asynchronous cases. Please refer to Section 2 for more details.

**Q2:** *"The global time clock in the system is based on the communication to the servers. (in section 2.2) However, for each clock time (t-1), only one client will receive the global model $(w^{(g)}{t-1})$ from the server and settle its local model $(w^{(i)}{t})$. For the next time that the client receives the model from the server, there will be a large interval between the time clock of the previous model and the current model. For the local models inside the interval, their notations and meanings still remain undefined. Nevertheless, those terms for the local model are used in the proof process in the appendix."*

**R:** Thank you for the correction. In the proof of asynchronous PFL, we turn to use $\omega_t$, since each global clock corresponds to only one local model. And in synchronous PFL, we still use $\omega_t^{(i)}$. Here we would like to explain that our proof relies mainly on the virtual sequence, which is fully defined, while for the local model $\omega_t^{(i)}$ we just use its locally updated formula, which is always valid and therefore does not affect the results of our proof.

**Q3:** *"In section 3.2, the time cost for each clock time for PFL is referred to as $max(T_l, T_g)$ + $T_c/2$. It is not clear how the $T_c/2$ is obtained."*

**R:** Thank you for the insightful correction. When the central server broadcasts the global model, the central aggregation is also being performed. Here, we assume that the time for uploading the local model is the same as the time for downloading so that the communication time $T_c$ is reduced by half. However, when the aggregation time of the central server is greater than $T_l+T_c$, then the central server still executes the last aggregation task when the node uploads the model, and the communication time can be considered $0$ at this point. Therefore, we corrected the time cost of each clock for the synchronous / asynchronous PFL to $T=\max(T_l+T_c+T_w, T_g)$ and $T=\max(T_l+T_c, T_g)$, respectively. We have added a timing diagram when the aggregation time is $T_g = T_l+T_c+T_w$ to help with understanding. Please refer to Fig. 5 in the Appendix.

From: ICLR.cc/2023/Conference/Paper1005/Authors
title: Response To Reviewer bvyd (1/3)
comment: We thank reviewer bvyd for the informative feedback, the comments and questions are very relevant. On the other hand, we also realize there is some misunderstanding of our method. Please allow us to clarify them in more detail.

**Q1:**  *"The biggest concern is that there is no evidence (empirical or theoretic) to support the claim that the bottleneck is the aggregation and broadcast step on the server side when compared to the edge. The entire paper's premise lies on the above claim and this needs to be validated."*

**R:** Thanks very much for your comment. Sorry, this comment makes us confused. We do not claim that *"the bottleneck is the aggregation and broadcast step on the server side when compared to the edge."* Instead, in this paper, we study two bottlenecks induced by network stragglers on the edge side and complicated aggregation operation on the server side. The training efficiency issue induced by these two bottlenecks is actually in the server workflow. By simply exchanging the execution order of global aggregation and broadcasting, global aggregation is decoupled from the communication and can be performed in parallel with local training. 

Maybe we did not represent these two bottlenecks clearly. We would like to clarify them by case study: 

- **For the bottleneck induced by network stragglers:** Suppose in a FL system, there are four edge devices, which are with diverse computation and communication abilities, leading to various local training time costs: edge node \#1 (e.g., an institution with HPC cluster) costs 30 mins, edge node \#2 (e.g., a user with 8 tesla v100 GPUs) costs 1 hour, edge node \#3 (e.g., a user with one 3090Ti GPU) costs 6 hours,  edge node \#4 (e.g., a user with mobile device) costs 60 hours. In the current Sync FL, according to the rule that aggregation is performed after **all** local updates are received, the slowest-responding client, i.e.,  edge node \#4, becomes the bottleneck of the Sync FL system. The server has to wait for 60 hours until all local training tasks are completed, then can perform global aggregation.

- **For the bottleneck induced by complicated aggregation:** We consider the scenario of FL over edge devices with heterogeneous neural network based models. In this case, directly averaging the local models is not feasible anymore, since local models have diverse structures. A popular approach is to perform knowledge distillation on the server side, leading to a complicated aggregation operation. The local edge models have to wait for the completion of knowledge distillation-based aggregation and then can begin the next local training. Accordingly, the complicated aggregation operation is another bottleneck of the FL system.

**Q2:** *"Moreover, the authors need to elaborate or justify (empirically or theoretically) their assertion/examination stated as "By carefully examining the above process, it can be found that the cause of training inefficiency actually lies in the dependency of server on all local updates uploading and the dependency of clients on global aggregation.""*

**R:** Thanks for your comments. We will point out separately the dependencies among each of the stages in the FL training process as follows.

- **The dependency of the global aggregation in server on all local updates uploading:** The central server must wait until it has received all the local models for the current round, and then it can perform aggregation. This leads to the "straggler" bottleneck. 

- **The dependency of clients' local optimization on global aggregation:** The edge devices must wait until the central server completes global aggregation, and then it can get the global model to perform its local optimization. As a result, the edge devices are idle while the central server completes aggregation, and the central server is idle while the edge devices perform local optimization. This leads to the "complicated aggregation" bottleneck. 

We provide extensive experiment results to empirically show that our framework can tolerate both network stragglers and complicated aggregation operation, which brings up to $5.56\times$ speed-up compared to traditional FL.

From: ICLR.cc/2023/Conference/Paper1005/Authors
title: Response To Reviewer bvyd (2/3)
comment: **Q3:** *"There is a lack of discussion on accuracy. Even though Table 1 simply reports various accuracies, however, there is no discussion on analytical or intuition behind why working with a stale global model does not significantly affect the accuracy. Why PFL achieves lower and better accuracies for different datasets and what impact of iid vs non-iid on the proposed scheme?"*

**R:** Thanks very much for your comment. In fact, in Section 2.2 of our manuscript, we have already offered a theoretical analysis to show that using the stale global model achieves a similar convergence as using the up-to-date global model. Furthermore, in the revised manuscript, we added experimental results in Fig. 4 to verify the influence of the stale global model on the final accuracy. Please refer to Fig. 4 in the Appendix and its discussion for more details.

**Revision in "Study on the Influence of Stale Global Model":**  *"In our scheme, the key to achieving parallel federated learning is to send the stale global model to the edge devices. It would be interesting to investigate the influence of the stale global model on the final performance. In this section, we conduct convergence analysis through an empirical experiment with respect to both accuracy and communication rounds on the CIFAR-10 dataset with 20 edge devices.
The convergence curve is provided in Fig. 4. It can be found that, in the early training stage, the proposed PFL suffers from lower accuracy and stronger oscillation compared with FedAvg; while with the communication round increases, PFL finally achieves similar performance to FedAvg. This is consistent with the intuition that when the algorithm converges, there is no significant difference between using the stale global model and using the up-to-date global model."*

In addition, a series of recent studies [1-5] offer similar conclusions that asynchronous FL can achieve the same convergence rate and accuracy as synchronous FL when the asynchrony between the nodes involved in aggregation is less than a certain threshold.  In contrast to asynchronous FL, we actively broadcast the stale model, so that the server and clients can do global aggregation and local optimization in parallel. And we offered a theoretical analysis in Section 2.2 to show that using the stale global model achieves a similar convergence as using the up-to-date global model. 

[1] Avdiukhin, Dmitrii, and Shiva Kasiviswanathan. "Federated learning under arbitrary communication patterns." International Conference on Machine Learning. PMLR, 2021.

[2] Huba, Dzmitry, et al. "Papaya: Practical, private, and scalable federated learning." Proceedings of Machine Learning and Systems 4 (2022): 814-832.

[3] Mishchenko, Konstantin, et al. "Asynchronous sgd beats minibatch sgd under arbitrary delays." arXiv preprint arXiv:2206.07638 (2022).

[4] Koloskova, Anastasia, Sebastian U. Stich, and Martin Jaggi. "Sharper convergence guarantees for asynchronous sgd for distributed and federated learning." arXiv preprint arXiv:2206.08307 (2022).

[5] Nguyen, John, et al. "Federated learning with buffered asynchronous aggregation." *International Conference on Artificial Intelligence and Statistics*. PMLR, 2022.

**Q4:** *"The paper lacks clarity in writing."*

**R:** Thanks very much for your kind suggestion. 

- *"There are some grammatical errors/typos":* We have carefully proofread the manuscript to correct typos. 

- *"Section 2.1 is repetitive and thus redundant.":* As suggested,  Section 2.1 has been removed from the revised manuscript, part of which was merged into Section 1. 

**Q5:** *"Moreover, the authors claim to be the first to identify this bottleneck without any support."*

**R:** To the best of our knowledge, there is no existing work considering the bottleneck induced by the complicated aggregation operation. It would be appreciated if the reviewers could tell us about other works that considered this issue.

From: ICLR.cc/2023/Conference/Paper1005/Authors
title: Response To Reviewer bvyd (3/3)
comment: **Q6:** *"The computational (hardware and network) infrastructure is unclear with N=100 mobile devices. Are these real physical mobile devices or whether these are simulated? More details are definitely required."*

**R:** Thanks for your comment. Due to limitations on hardware resources, following the common approach adopted in FL [1-4], we performed the experiments with four GeForce RTX 3090-type GPUs on one machine.

[1] Gu, Xinran, et al. "Fast federated learning in the presence of arbitrary device unavailability." Advances in Neural Information Processing Systems 34 (2021): 12052-12064.

[2] Ruan, Yichen, et al. "Towards flexible device participation in federated learning." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

[3] Avdiukhin, Dmitrii, and Shiva Kasiviswanathan. "Federated learning under arbitrary communication patterns." International Conference on Machine Learning. PMLR, 2021.

[4] Nguyen, John, et al. "Federated learning with buffered asynchronous aggregation." International Conference on Artificial Intelligence and Statistics. PMLR, 2022.

In our experiments, the 100 mobile devices are implemented through simulation. It will not affect the soundness of our results. This is because the main difference between reality and simulation is the communication time required for the upload and download of the model, which is not our main concern. In our experiments, the bandwidth is set to 4 MB/s. So, the model upload/download time is 3 s, which is a small percentage of the overall FL training process. Theoretical proofs and simulation experiments are sufficient to demonstrate the feasibility of our solution.

**Q7:** *"Why the ratios chosen are $T_w/T_l$ and $T_g/T_l$ ? What performance metric do they represent in layman terms? It should be discussed in the opening paragraph of Experiments"*

**R:** Thanks for your comment. The core of our approach lies in the parallelism between the aggregation stage of the central server and the local optimization stage of the edge devices. Therefore, the relative ratio of the time spent in each stage can directly determine the speedup of our approach. For example, the time cost of each round in synchronous PFL can be calculated by $T = \max(T_l+T_c+T_w, T_g)$ and the time cost of each round in standard federated learning can be calculated by $T=T_l+T_c+T_w+T_g$. When the time-cost ratio for each stage is $1:1:1:1$, then the speedup of our framework is $1.33$. And if $T_g = T_l+T_w+T_c$, the speedup of our framework is $2$. We have added a timing diagram when the aggregation time is $T_g = T_l+T_c+T_w$ to help with understanding. Please refer to Fig. 5 in the Appendix.

From: ICLR.cc/2023/Conference/Paper1005/Authors
title: Response To Reviewer PWvv
comment: We thank the reviewer for the positive reviews and insightful suggestions. Our responses to the reviewer's comments are as follows:

**Q1:**  *"There are some typos (e.g., speedup radio $\rightarrow$ speedup ratio in Figure 3 and page 9). The caption of Table 1 says 'The best result is boldfaced', but there is no 'boldfaced' number in the table."*

**R:** Thanks very much for your kind reminding. We have carefully proofread the manuscript to correct typos.

**Q2:** *"Small errors in the asynchronous PFL algorithm: the model being sent back to device i is not necessary to be w\_(t+n), say if the global aggregation is being executed when the update from device $i$ arrives along with updates from some other devices."*

**R:** Thank you for your kind reminding. Yes, this case does exist. What we actually do is to send the stale global model in the buffer to the edge node $i$, which is the most recent result of the aggregation. We clarified this point in the manuscript.

**Revision in Section 2.1.2:** *"Right after receiving the uploaded signal $\delta^{(i)}$ from edge device $i$, the central server sends back to it the stale global model $\omega^{(g)}$ in the buffer,  which is the most recent aggregation result. It would be $\omega_{t+n}^{(g)}$ if the central server is idle, or $\omega_{t+n-r}^{(g)}$ if the central server is performing aggregation, where $r$ denotes the number of local devices that previously arrived at the sever but did not participate in aggregation. The former is a special case of the latter with $r=0$. For simplicity, we denote the stale global model as $\omega_{t+n-r}^{(g)}$. If the local model uploaded in the previous clock by that device has not completed aggregation, then the device must wait until it has."*

**Q3:** "*As this paper mainly describes asynchronous PFL in Sec 2.2, it might be better to include a convergence analysis of asynchronous training instead of synchronous training in Sec 2.3."*

**R:** Thanks very much for your kind suggestion. In the revised manuscript, as also following the suggestion of reviewer Ech9, in Section 2.1 we presented both synchronous and asynchronous PFL, in Section 2.2, we provided the convergence analysis including both synchronous and asynchronous cases. Please refer to Section 2 for more details.

**Q4:** *"The speedup is measured in run time for per global clock between the device and the central server, but the total number of global clocks (training round) needed for convergence is not reported. It would be interesting to see whether PFL affects the convergence speed."*

**R:** Thank you very much for the insightful suggestion. In the revised manuscript, as suggested, we added experimental results with respect to accuracy and communication rounds. Please refer to Fig. 4 in the Appendix and its discussion for more details. 

**Revision in "Study on the Influence of Stale Global Model":**  *"In our scheme, the key to achieving parallel federated learning is to send the stale global model to the edge devices. It would be interesting to investigate the influence of the stale global model on the final performance. In this section, we conduct convergence analysis through empirical experiments with respect to both accuracy and communication rounds on the CIFAR-10 dataset with 20 edge devices.
The convergence curve is provided in Fig. 4. It can be found that, in the early training stage, the proposed PFL suffers from lower accuracy and stronger oscillations compared with FedAvg; while with the communication round increases, PFL finally achieves similar performance to FedAvg. This is consistent with the intuition that, when the algorithm converges, there is no significant difference between using the stale global model or using the up-to-date global model."*

From: ICLR.cc/2023/Conference/Paper1005/Reviewer_PWvv
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper claims that, due to device heterogeneity, the main bottleneck of training efficiency is the workflow in the server. The workflow of FedAvg is uploading (client's local updates -> server), global aggregation (server), broadcasting (server's global model -> client), local optimization (client). Two main dependencies play the role here: the dependency of the global aggregation in server on all local updates uploading, and the dependency of clients' local optimization on global aggregation. This paper proposes a parallel federated learning (PFL) strategy to decouple the global aggregation in server with the communication between server and client (uploading & broadcasting), so that the server and clients can do global aggregation and local optimization in parallel. A key difference here is, instead of receiving the up-to-date global model, a client now receives a stale model and the model updates are computed based on the stale model. The paper provides convergence analysis and empirically shows that PFL is faster than FedAvg and two optimized methods (DGA and MIFA), brings 1.77X to 7.32X speedup.
strength_and_weaknesses: Strength
1. This paper tackles an important problem, the training inefficiency of federated learning due to device heterogeneity. The proposed solution, PFL, is simple but efficient.
2. The paper designs both synchronous and asynchronous versions of PFL, provides convergence analysis. Figures 1&2 are helpful in illustrating the workflows and interactions between the server and clients.
3. The paper has a thorough evaluation on the performance of FPL and demonstrates significant speedups with experiments.

Weakness
1. There are some typos (e.g., speedup radio -> speedup ratio in Figure 3 and page 9). The caption of Table 1 says 'The best result is boldfaced', but there is no 'boldfaced' number in the table.
2. Small errors in the asynchronous PFL algorithm: the model being sent back to device i is not necessary to be w_(t+n), say if the global aggregation is being executed when the update from device i arrives along with updates from some other devices.
3. As this paper mainly describes asynchronous PFL in Sec 2.2, it might be better to include a convergence analysis of asynchronous training instead of synchronous training in Sec 2.3.
4. The speedup is measured in run time for per global clock between the device and the central server, but the total number of global clocks (training round) needed for convergence is not reported. It would be interesting to see whether FPL affects the convergence speed.
clarity,_quality,_novelty_and_reproducibility: Reproducibility: this paper doesn't provide detailed training parameters in the experiment part. 
summary_of_the_review: In summary, this paper tackles an important problem of federated learning and proposes a simple solution to address the training inefficiency issues brought by device heterogeneity effectively. The work effectively accelerates federated learning without scarifies the model accuracy. The presentation of the paper can be improved, and it would be helpful to include more experiment results regarding the total training time or a comparison of the speed of convergence.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1005/Reviewer_bvyd
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors propose a Parallel Federated Learning (PFL) scheme to deal with the challenges associated with device heterogeneity. 
- Unlike the conventional approaches which tackle the above challenges by focusing on edge-based solutions, authors assert that this work is "first-of-its-kind" in focusing on the server as the bottleneck, specifically, the aggregation-then-broadcasting on the central server is the root cause of training inefficiency.
- PFL has both versions: synchronous and asynchronous
- PFL simply works on swapping the order for aggregation and broadcast at the server where the stale global model is broadcasted on receiving the local model update from any client. Then, the server aggregates the local updates to create the next global update while the edge device simultaneously (in parallel) uses the stale model to compute the next local update.
- PFL can achieve the similar convergence rate as synchronous FL
- PFL can tolerate both stragglers and complicated aggregation tasks, which brings 1.77× to 7.32× speedup.


strength_and_weaknesses: **Strengths:**
- The paper discusses an important problem in the field of FL associated with device heterogeneity.
- Fig. 1, Fig. 2, and Algo 1 are much appreciated in helping to understand the proposed workflow and put it in the context of conventional approaches.
- The proposed idea is simple to understand and is a straightforward re-ordering of the conventional FL process with swapping of aggregation and broadcast steps on the server side.


**Weaknesses:**
- The biggest concern is that there is no evidence (empirical or theoretic) to support the claim that the bottleneck is the aggregation and broadcast step on the server side when compared to the edge. The entire paper's premise lies on the above claim and this needs to be validated.
- The paper lacks clarity in writing (see the next section for details)
- There is a lack of discussion on accuracy. Even though Table 1 simply reports various accuracies, however, there is no discussion on analytical or intuition behind why working with a stale global model does not significantly affect the accuracy. Why PFL achieves lower and better accuracies for different datasets and what impact of iid vs non-iid on the proposed scheme?
clarity,_quality,_novelty_and_reproducibility: 
1. There are some grammatical errors/typos (e.g., cause root, hinder to,  making it have,  In summery, does can achieve, which to conduct, does not participant, The pipeline of PFL is shown as Fig. 4. (Fig. 2 not referenced), the best result is claimed to be bold-faced in Table 1 but is not visible, etc) which would require the work to be thoroughly proof-read.
2.  Section 2.1 is repetitive and thus redundant. It might be better to tag the last paragraph of Section 1 as Motivation. 
3.  Moreover, the authors need to elaborate or justify (empirically or theoretically) their assertion/examination stated as "By carefully examining the above process, it can be found that the cause of training inefficiency actually lies in the dependency of server on all local updates uploading and the dependency of clients on global aggregation"  
4. The computational (hardware and network) infrastructure is unclear with N=100 mobile devices. Are these real physical mobile devices or whether these are simulated? More details are definitely required.
5. Why the ratios chosen are $\frac{T_w}{T_l}$ and $\frac{T_g}{T_l}$ ? What performance metric do they represent in layman terms? It should be discussed in the opening paragraph of Experiments
summary_of_the_review: My recommendation is rejection since the paper lacks evidence to support the claim that the bottleneck is the aggregation and broadcast step on the server side when compared to the edge. Moreover, the authors claim to be the first to identify this bottleneck without any support.

---

EDIT (after authors' responses): The authors have addressed my main concerns. I appreciate their responses and clarifications. I am increasing the score in favor of acceptance.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1005/Reviewer_Ech9
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper presented parallel federated learning (PFL), a new federated learning framework that is mostly based on conventional frameworks but with some innovative adjustments to enhance framework efficiency. PFL inherits most of the workflow from previous approaches like FedAvg. However, it swaps the order of global aggregation and global broadcasting and proves that such an adjustment can lead to similar convergence. The order swap allows the local optimization and the global aggregation to occur simultaneously, guaranteeing better parallelism. This paper also prepared some experiments to verify the feasibility of the proposed framework, and the experimental result indicates a better performance of PFL compared to the baselines.

strength_and_weaknesses: Strength:

1. This paper gives solid proof for changing the order between the aggregation and the broadcasting process and shows that such change will not influence its convergence. Compared to previous approaches from which it inherits, its workflow is changed, leading to the queries of readers on how such change will influence the result performance. The solid proof eliminated the queries here. 

2. This paper introduced a sufficient number of baselines and showed some of the experimental results for them in comparison to the proposed framework. Based on the comparisons to those baselines, the feasibility of the proposed work can be guaranteed better.



Weakness:

1. This paper only introduced the asynchronous PFL in section 2.2. ("Due to page limitation. we only present asynchronous PFL in this section.") However, section 2.3 mainly proves the convergence of synchronous PFL, which is inconsistent with the prior section. Thus, when reading section 2.3, which discusses the synchronous PFL, the terms explained in section 2.2 are no longer applicable. And those terms for synchronous PFL are not explained in section 2.3 afresh. Therefore, it is hard to understand what the terms in section 2.3 mean. (Even the explanations for synchronous PFL in the appendix are not detailed enough.)

2. The global time clock in the system is based on the communication to the servers. (in section 2.2) However, for each clock time (t-1), only one client will receive the global model (w^{(g)}_{t-1}) from the server and settle its local model (w^{(i)}_{t}). For the next time that the client receives the model from the server, there will be a large interval between the time clock of the previous model and the current model. For the local models inside the interval, their notations and meanings still remain undefined. Nevertheless, those terms for the local model are used in the proof process in the appendix.

3. In section 3.2, the time cost for each clock time for PFL is referred to as max(T_l, T_g) + T_c/2. It is not clear how the T_c/2 is obtained.


clarity,_quality,_novelty_and_reproducibility: This paper is mainly an extensively modified version of the conventional approaches to federated learning. Its novelty on the entire framework design is limited, and the idea is relatively straightforward. However, it presents most of its inferences clearly with solid support and proves its feasibility and effectiveness with experiments. 

However, some drawbacks exist in illustrating how the synchronous and asynchronous PFL work. This paper emphasizes that it covers those two versions. But the illustration and comparison for the two versions are incomplete, leaving readers hard to know detailed improvements for the two versions. For example, this paper states that they have both the synchronous and the asynchronous version. But in each section, they seem only to include one version: either the synchronous one or the asynchronous one. For instance, section 2.2 mostly focuses on the asynchronous version, and section 2.3 focuses on the synchronous version. Moreover, the comparison between the two versions and the illustration of their differences are not included as well. Those two versions should have distinct advantages and implementation details, but their distinction is not presented in the paper sufficiently.

summary_of_the_review: This paper presented an approach to parallelize global aggregation and broadcasting in conventional federated learning. The proposed method is proven theoretically correct generally. But there may exist some unclear or incomplete parts in the proof. The explanation for the synchronous or asynchronous PFL is also not appropriately arranged to some extent. However, the experimental result for the proposed framework compared to the baselines indeed shows its feasibility and improvement that cannot be neglected, although more experiments can make this work more convincing.

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold