From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper proposes a new approach, TimelyFL, to handling stragglers in federated learning (FL). The approach combines several ideas: adapting the duration of each round based on the devices participating and their capabilities, and adapting the amount of training to be done on each device based on their capabilities. Although some of these ideas (e.g., partial model training) have been explored independently, this is the first time they have explored jointly. The ideas are evaluated through simulations and experiments with a testbed of Android phones.

**Strengths:** The ideas introduced in the paper appear to be promising, especially from some of the experimental results. We also appreciate the effort that went into running the Android testbed experiments.

**Weaknesses:** The general sense was that the paper was borderline at best, for the following reasons:
1. Limited novelty in terms of how the partial model training was done;
2. Without theory, the bar is higher on the experiments, and there are several ways in which the experimental results could be strengthened.

In particular, the ways in which the paper could be strengthened:
* Use target thresholds closer to the performance achieved by SyncFL (for example, in Table 1). More generally, what happens when one targets accuracy at or above that achieved by the SyncFL baseline?
* Improve TimelyFL, if needed, so that after training longer it achieves performance comparable to SyncFL (e.g., may need to wait for more clients, train on larger portions of the model, or otherwise increase the time-per-round as training progresses)
* Include more analysis and discussion of the communication costs and overhead of TimelyFL
* Elaborate on the major design challenges involved in the testbed experiments

There was an extensive discussion about this paper among the reviewers and AC, and the ultimate conclusion was that the weaknesses currently outweigh the strengths, leading to the recommendation to reject the paper. The authors are encouraged to submit again in the future after addressing some of the weaknesses. In particular, using target thresholds closer to the performance achieved by SyncFL would have gone a very long way to better convincing the reviewers of the merits of this work.
justification_for_why_not_higher_score: There could be an argument for a higher score based on the fact that the paper includes both simulations and experiments with an Android testbed, which is above and beyond the typical FL submission to ICLR. I expect that aspect of the contribution might be more appreciated at a systems conference.

The algorithmic contribution is modest, and the main ingredient missing is better ablations to illustrate when and why the proposed approach is preferable over previous approaches, and what are the limitations of the proposed approach.
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: We had a virtual meeting to discuss this paper. All reviewers attended and contributed significantly to the discussion.

There was a healthy back-and-forth discussion about the relative strengths and weaknesses of the proposed approach summarized above. Most of that discussion directly fed into the meta-review.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Author's response 
comment: Re 1/6, 2/6: We politely disagree with the reviewer’s statement. Our work does have in-depth insights: the key insights of our work are on disclosing why state-of-the-arts (FedBuff) fall short, which is non-trivial and indeed improves the understanding of practical FL systems. Moreover, we took a further step to illustrate such key insights in Figure 2 to help the readers understand better. We used key insights as the primary motivation and proposed a novel heterogeneity-aware asynchronous FL framework based on adaptive partial training to address the key limitations of state-of-the-arts, pushing the state-of-the-arts further.  

Re 3/6: Regarding the training parameters mentioned by the reviewer, such as the selection between epoch and iteration for local training, we consider them as different variants of TimelyFL. We will take this suggestion seriously and would like to conduct extra experiments to compare the performance between TimelyFL and the variant suggested by the reviewer.

Re 4/6: The techniques mentioned by the reviewer such as random dropout and weighting schemes have been proposed by existing works. In our work, we did not combine those existing techniques into TimelyFL. However, we will take this suggestion seriously and summarize those existing techniques mentioned in the review in the related work section.

Re 5/6: Agreed. Will make this format change accordingly.

Re 6/6: We believe this is not an ethical issue but an interpretation difference. We consider that paragraph a reproducibility statement because it outlines how we reproduce the results on real devices. However, we will take this suggestion seriously to resolve this editorial issue.

From: ICLR.cc/2023/Conference/Paper1260/Reviewer_L6W9
title: Thanks for the response
comment: Thanks for the response.

Re 1/6, 2/6: The original response by the authors was to my point about lack of non-trivial findings from the experiments. Now the authors seem to agree that these two points are not new findings, so my original comment in the review that says `although a decent amount of experiments has been done, there is not much discussion on any unique or surprising results that was seen, or any in-depth insights that can improve the understanding of practical federated learning systems` is valid now.

Re 3/6: It is fine to consider it as future work. The authors should understand the difference between a SGD iteration and an epoch. An epoch includes multiple SGD iterations. If different clients have different amount of data, then the number of iterations in an epoch would be different, which would cause additional complexity for scheduling. This problem would not exist if you fix the number of iterations, instead of epoch, between two communication rounds, which is why I wonder why the authors choose to use epoch instead of iterations.

Re 4/6: $T_k$ is just an example. There may be other parameters that can also be adapted to guarantee a good final result. For example, if you train a model for a very long time, you may want to consider gradually increasing the model size at all the clients and waiting for a longer time in each aggregation round. In the end, all the clients will have the full model. Although it takes longer to train, it will be possible to get the same performance as with SyncFL. Meanwhile, random dropout itself should not cause converging to a worse final result, there are techniques such as weighting the client updates according to the reciprocal of its participation rate or simply skipping the updates of clients that participate more often so that the overall contribution among clients is equal. 

Re 5/6: It usually makes sense to include more insightful discussions in the main paper, while dataset details can go into the appendix. 

Re 6/6: The reproducibility statement should be after all the sections in the main paper including the conclusion section. This is a purely editorial issue though, which can be easily fixed in the final version. I just don't know how this will be handled at this stage (will leave it to the AC and up). However, I really cannot see how Section 4.4 can be the reproducibility statement. This section talks about the results in Figure 8 and it is the only place talking about Figure 8, so it is definitely **not** a reproducibility statement. I'm more disappointed by the attitude the authors take here to argue something that is clearly not a reproducibility statement to be the reproducibility statement. Originally, this was a minor issue (at least to me), but this clearly inappropriate argument could raise ethical concerns...

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Post Rebuttal Response (6/6) 
comment: ```
Please also note the ICLR 2023 submission instructions (https://iclr.cc/Conferences/2023/CallForPapers) which states that There will be a strict upper limit of 9 pages for the main text of the submission, with unlimited additional pages for citations. This page limit applies to both the initial and final camera ready version. The paper currently has the conclusion section on the 10th page, which violates the page limit.
```

In our original version, the submission is within 9 pages. We edited some writing based on the reviewers’ suggestions during the rebuttal period. To avoid confusion, we do not delete the previous writing, which makes the paper longer than the initial version.

However, even though the conclusion section is on the 10th page of the current version, it still obeys the page limit. The last section before the conclusion section (section 4.4 real-world deployment) is our **reproducibility statement**, which does not count toward the page limit. As stated in the ICLR 2023 Author Guide (https://iclr.cc/Conferences/2023/AuthorGuide), ```this optional reproducibility statement will not count toward the page limit, but should not be more than 1 page.``` As a result, we do not violate the page limitation. 

In our final camera-ready version, we would finalize our writing based on your valuable opinions and ensure it fits the conference's page limitation. 

We thank again the reviewer for the time and effort in giving feedback on our paper. If you feel our responses have clarified all your concerns, we hope you can positively re-evaluate our paper in light of our responses and let us know! We are happy to take any follow-up questions and look forward to discussing them with you for the remaining period!

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Post Rebuttal Response (5/6)
comment: ```
(5/6) It is also not quite clear to me why the authors have made some relatively less important additions (e.g. details of datasets) to the main paper, while more important discussions such as those mentioned in the response have not been added.
```

We understand the reviewer’s concern about our edition for the details of datasets. We provide the details of the Google Command dataset and its pre-processes method as suggested by the other reviewers, as we value all the opinions from the reviewers. **We believe that all the reviews aim to make our work better. By this chance, we want to thank again to all the reviewers for the time and effort put into this work.** 

For other concerns and questions raised by the reviewers, we all have responded in the discussion forum. If there are remaining concerns, please let us know and we would be happy to engage in discussing them for the remainder of the discussion period.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Post Rebuttal Response (4/6) 
comment: ```
(4/6) I do not understand the last response (5/5). Since you are developing an adaptive scheduling method, the scheduling should guarantee convergence to the optimal result, at least when the system is run for a long enough time. This means that parameters such as Tk may need to be different in initial training and when approaching convergence. It seems the gap is caused by not fully adapting the scheduling of devices to the latest system and model status, which is a significant limitation.
```

There might have been some misunderstandings for the $T_k$ in our work, which is adaptively changing during the training. The aggregation interval $T_k$ in each round equals the $k$th smallest value among the estimated unit total time for all clients, as stated in line 8 in Algorithm 1. As the estimated unit total time is dynamic during the training, the $T_k$ is changing round by round based on the real-time situation for the whole system. 

As we stated earlier, the reasons for the gap between TimelyFL and syncFL are based on the followings. Firstly, as shown in Figure 1b, some clients could not catch up with the aggregation interval in each round due to sudden communication or computation disturbance. For example, some devices may suddenly go out of battery or service and disconnect from the system, or the required resources have been occupied by other high-priority tasks compared to model training during the FL training. Secondly, the weak device only reports the local update for the partial model instead of the update from full model training. We believe these are the fundamental reasons for the gap.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Post Rebuttal Response (3/6)
comment: ```
(3/6) Regarding the two options (1) Fully train the model with one or more local epochs. (2) Training fewer layers of the model and for only one epoch mentioned in the response, I wonder why the authors only chose these two options instead of allowing training fewer layers with multiple iterations/epochs. Moreover, an epoch usually includes multiple mini-batch iterations. It is not clear why the authors chose epoch as the smallest unit, while it may be possible that training for multiple iterations that do not constitute a full epoch can actually give a better performance.
```

The motivation for these two options begins from the following. The low-resource devices could not finish even one epoch under the full model training within the required time. Based on this, we would calculate the limit layer numbers they could complete within the time threshold, which is training fewer layers of the model for only one epoch. In the system, we wish the server could get the update for the full model in each communication round, which means that some devices should conduct the full model training in each round. As a result, the high-resource devices would fully train the model with one or more local epochs.

We believe that the reviewer’s suggestion, ​​training fewer layers with multiple iterations/epochs, is another bright direction for layer-wise efficient training in federated learning, which needs a comprehensive discussion on the impact of each layer’s update to the server aggregation, and is out of the scope for this work. We would consider it as a bright direction for our future work.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Post Rebuttal Response (2/6)
comment: ```
(2/6) Regarding the design of the training system should adaptively schedule the training parameters based on the real-time capability of each device, this again is not new by itself and scheduling algorithms for federated/distributed learning have been proposed in both ML and distributed computing communities, including FjORD and HeteroFL. I understand that your method is somewhat different from these, but the need for adaptive scheduling itself is not a new conclusion from this work.
```

**Again, we need to clarify that in our paper, we never claim “the need for adaptive scheduling” is new. Instead, what we claimed is new in our work is a new staleness control technique based on adaptive partial training, which is different from FjORD and HeteroFL.** Specifically, our proposed method is different from FjORD and HeteroFL in the following aspect: Fjord and HeteroFL need to change the model structures for efficient training on the low-capability devices, while TimelyFL does not.

From the empirical point of view, compared to the Fjord and HeteroFL, the TimelyFL brings more flexibility in adaptive scheduling. Fjord and HeteroFL need to change the model structures for efficient training on the low-capability devices. From the coding perspective, all the required changes to the model structures should be prepared beforehand, which scarifies the flexibility during the distributed training. On the other hand, under the TimelyFL, the devices do not need to change the model structure for partial training, which brings more adaptivity from the empirical perspective.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Post Rebuttal Response (1/6)
comment: We thank the reviewer for the feedback, and address the reviewer's comments in our response below.
```
(1/6) Regarding a critical observation that stale local updates would significantly harm the model accuracy for asynchronous federated learning, this is not new and it is well-known that having bounded staleness (i.e. having a "maximum age of the stale values") can improve the performance.
```

We need to clarify that in our paper, we never claim the observation is new. As the reviewer stated, staleness is a well-known problem that hurts model accuracy. SOTA works on asynchronous FL such as FedBuff [1], PAPAYA [2] and FedAysnc [3] have proposed staleness control techniques to address the problem. Unfortunately, as we show in the Introduction Section, existing staleness control techniques still fall short, and this is the key motivation of our work. The key novelty of our work is a new staleness control technique based on adaptive partial training that outperforms existing works.

[1] Nguyen, John et al. “Federated Learning with Buffered Asynchronous Aggregation.” International Conference on Artificial Intelligence and Statistics (2021).

[2] Huba, Dzmitry et al. “Papaya: Practical, Private, and Scalable Federated Learning.” ArXiv abs/2111.04877 (2021): n. pag.

[3] Xie, Cong et al. “Asynchronous Federated Optimization.” ArXiv abs/1903.03934 (2019): n. pag.

From: ICLR.cc/2023/Conference/Paper1260/Reviewer_L6W9
title: Thank you for the response
comment: Thank you for the response. I still maintain my opinion that the contribution in this paper is overly straightforward and lacks rigorous consideration (even from an empirical perspective). The following are some further comments: 
- From the authors' response, I couldn't identify what is particularly non-trivial or challenging. 
- Regarding `a critical observation that stale local updates would significantly harm the model accuracy for asynchronous federated learning`, this is not new and it is well-known that having bounded staleness (i.e. having a "maximum age of the stale values") can improve the performance. See for example: More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server, NeurIPS 2013.
- Regarding `the design of the training system should adaptively schedule the training parameters based on the real-time capability of each device`, this again is not new by itself and scheduling algorithms for federated/distributed learning have been proposed in both ML and distributed computing communities, including FjORD and HeteroFL. I understand that your method is somewhat different from these, but the need for adaptive scheduling itself is not a new conclusion from this work.  
- Regarding the two options `(1) Fully train the model with one or more local epochs. (2) Training fewer layers of the model and for only one epoch` mentioned in the response, I wonder why the authors only chose these two options instead of allowing training fewer layers with multiple iterations/epochs. Moreover, an epoch usually includes multiple mini-batch iterations. It is not clear why the authors chose epoch as the smallest unit, while it may be possible that training for multiple iterations that do not constitute a full epoch can actually give a better performance. 
- I do not understand the last response (5/5). Since you are developing an adaptive scheduling method, the scheduling should guarantee convergence to the optimal result, at least when the system is run for a long enough time. This means that parameters such as $T_k$ may need to be different in initial training and when approaching convergence. It seems the gap is caused by not fully adapting the scheduling of devices to the latest system and model status, which is a significant limitation. 

It is also not quite clear to me why the authors have made some relatively less important additions (e.g. details of datasets) to the main paper, while more important discussions such as those mentioned in the response have not been added. 

Please also note the ICLR 2023 submission instructions (https://iclr.cc/Conferences/2023/CallForPapers) which states that `There will be a strict upper limit of 9 pages for the main text of the submission, with unlimited additional pages for citations. This page limit applies to both the initial and final camera ready version.` The paper currently has the conclusion section on the 10th page, which violates the page limit.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Thank you for your response!
comment: Dear Reviewer 21P2:

Thank you for your response and positive evaluation of our work! As the revision deadline has passed, we will revise the writing of line 3 in Algorithm 1 based on your suggestion in the camera-ready version. We will keep updating the writings based on your further suggestions if it is needed. We thank the reviewer for the time and effort in giving feedback on our paper.

Best regards, Authors

From: ICLR.cc/2023/Conference/Paper1260/Reviewer_21P2
title: Post-rebuttal feedback
comment: I'd like to thank the Authors for the effort put during the Rebuttal. Responses 1,2,4 and 5 adequately clarified the points I raised in my review. I particularly appreciate the time spent to obtain the SpeechCommands results with the lightweight model, this makes the evaluation stronger in my opinion.

I still think that line 3 in Algo1 isn't too clear: instead of 

_Sample $n$ clients for $S$ uniformly at random and send $W^r_s$ to clients in $S$ ;_

it would be clearer to write it as (which closely matches what's later shown in the paragraph in blue font):

_Sample $n$ clients uniformly at random to define $S$, and send $W^r_s$ to clients in $S$ ;_

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer touo (6/6)
comment: ```
(6/6) The current version of this work shows most of the results about the comparison with FedBuff to present the technical issue. It would raise the question of whether the phenomenon also exists in other asynchronous FL methods.
```

As the current SOTA work, FedBuff has already compared its performance with some other popular asynchronous FL method [1] and heterogeneous FL method [2]. These two baselines used to be the best in the asynchronous FL and still are popular in the FL community.

[1]. Xie, C., Koyejo, O., & Gupta, I. (2019). Asynchronous Federated Optimization. ArXiv, abs/1903.03934.

[2]. Sahu, A., Li, T., Sanjabi, M., Zaheer, M., Talwalkar, A.S., & Smith, V. (2020). Federated Optimization in Heterogeneous Networks. arXiv: Learning.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer touo (5/6)
comment: ```
(5/6) If there is enough time for training, what will happen after 14 hr in Figure 1 (c)? It seems that the FedBuff is still gaining higher test performance along with the training, considering the current green curve.
```

Figure 1c is a zoomed-in portion of Figure 4.a. During the last half hour of the training, the accuracy is stable for the FedBuff baseline, and the best performance is around 73.30% within the 15.6 hours of training. The performance gap between the final accuracy of syncFL and FedBuff is around 5.38%, while the gap between the final accuracy of syncFL and TimelyFL is 3.23%.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer touo (4/6)
comment: ```
(4/6) "asynchronous communication" is also an important and practical problem in distributed systems [1], it seems there are also some proposals or strategies in traditional distributed systems. Considering FL, the new learning paradigm with distributed models, what is the unique point of this problem? Could the author also give more discussion? (not only in the FL area, as the problem is general)
```

The most unique point that FL brings is the non-iidness of both local data distribution and device heterogeneity. The traditional distributed systems would consider more about how to finish a centralized training task by parallel computation, whereas the non-iid data distribution has rarely been considered. However, under the FL scope, the server does not have much information about the local data from each client or the computation and communication availability of each participant. These are the main challenges that FL brings to the distributed system.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer touo (3/6)
comment: ```
(3/6) Could the author discuss more the negative effects of waiting for all clients participating in each round or give some reference about this problem?
```

The most direct negative effect of waiting for all clients participating in each round is the significant delays due to stragglers. As we stated in the introduction, under a large-scale cross-device FL setup, not all clients are available at the training time. In addition, client devices may have limited communication bandwidth and compute power; some clients may even be temporarily disconnected during the training process, making synchronous federated learning challenging under realistic settings. It is commonly accepted by the FL community [1,2,3] that waiting for all clients participating in each round will not succeed in the cross-device setups, where the number of participants could be millions, and reckon that the synchronous federated learning limits the scalability because only a small fraction of the client population may be available at any given time for training.

[1]. Bonawitz, K.A., Kairouz, P., McMahan, H.B., & Ramage, D. (2021). Federated Learning and Privacy. Queue, 19, 87 - 114.

[2]. Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M.G., Malek, M., & Huba, D. (2022). Federated Learning with Buffered Asynchronous Aggregation. AISTATS.

[3]. Wang, J., Charles, Z.B., Xu, Z., Joshi, G., McMahan, H.B., Arcas, B.A., Al-Shedivat, M., Andrew, G., Avestimehr, S., Daly, K., Data, D., Diggavi, S.N., Eichner, H., Gadhikar, A., Garrett, Z., Girgis, A.M., Hanzely, F., Hard, A., He, C., Horváth, S., Huo, Z., Ingerman, A., Jaggi, M., Javidi, T., Kairouz, P., Kale, S., Karimireddy, S.P., Konecný, J., Koyejo, S., Li, T., Liu, L., Mohri, M., Qi, H., Reddi, S.J., Richtárik, P., Singhal, K., Smith, V., Soltanolkotabi, M., Song, W., Suresh, A.T., Stich, S.U., Talwalkar, A.S., Wang, H., Woodworth, B.E., Wu, S., Yu, F.X., Yuan, H., Zaheer, M., Zhang, M., Zhang, T., Zheng, C., Zhu, C., & Zhu, W. (2021). A Field Guide to Federated Optimization. ArXiv, abs/2107.06917.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer touo (2/6)
comment: ```
(2/6) This paper lacks some theoretical analysis of the focused "learning" problem as well as the convergence. It makes readers hard to understand the underlying mechanism of TimelyFL in federated learning. The authors may consider reorganizing the method section and highlighting the previous issues and corresponding improvements in TimelyFL.
```

Compared to the SOTA asynchronous FL work [1] we cite in this paper, TimelyFL introduces partial model training and adaptive workload scheduling that brings more complexity from the system level. Under the TimelyFL setting, the condition is complex as the training parameters, such as partial training rate and local epoch number, are changed in every communication round. We do plan to add a proof framework for the method. But you know that it is very difficult to analyze convergence for a training framework like ours, which considers both system and learning. Similar to other SOTA works for federated learning with partial model training [2,3], we mainly emphasize our empirical analysis and real-world deployment. We prefer to try theoretical analysis as our future independent work. Our empirical evidence in Section 4 and Appendix sufficiently supports our proposed method. We have marked it as a promising future work for this paper.

We highlight the previous issues and corresponding improvements in TimelyFL in the Introduction part. On the second page of this paper, we present a critical observation that stale local updates would significantly harm the model accuracy in FedBuff, the current SOTA work. As illustrated in Figure 2, we give the audience a direct view of the drawback for FedBuff and how TimelyFL tackles the staleness issues by adaptively adjusting the workload for each client. We hope this answers your question and are happy to discuss further if needed.

[1] Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M.G., Malek, M., & Huba, D. (2022). Federated Learning with Buffered Asynchronous Aggregation. AISTATS.

[2] Diao, E., Ding, J., & Tarokh, V. (2021). HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients. ArXiv, abs/2010.01264.

[3] Horváth, S., Laskaridis, S., Almeida, M., Leondiadis, I., Venieris, S.I., & Lane, N.D. (2021). FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout. ArXiv, abs/2102.13451.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer touo (1/6)
comment: Thank you for your detailed and constructive comments! Our responses are summarized below.

```
(1/6) The motivation for the proposed method is a little bit unclear, which seems to be based on the intuitive explanation in the second paragraph of page two. It may need more in-depth analysis or justification
```

We understand the reviewer's perspective on the motivation of our work. Our motivation starts from the drawbacks of the current SOTA work in asynchronous federated learning. In the introduction, we detailedly elaborate on the reason why the current SOTA, FedBuff, leads to an accuracy drop compared to the final accuracy of SyncFL. Our paper makes a critical observation that stale local updates would significantly harm the model accuracy for asynchronous federated learning, as we stated in the introduction (page 1 & 2). To solve this issue, we implement two techniques in the TimelyFL system: adaptive model training and adaptive aggregation interval. The adaptive aggregation interval improves inclusiveness, and adaptive model training brings heterogeneity awareness. The combination of two techniques brings advantages over the previous SOTA work. Based on the reviewers' suggestion, in the revised version, we have updated some writings and highlighted them in blue. We will keep updating the writings based on your further suggestions if it is needed.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer 21P2 (8/8)
comment: ```
(8/8) But something still hasn't been discussed regarding speechcommands: how is the input (a 1second-long audio signal) pre-processed before passing it to the CNN? I presume the Authors do the usual and compute the MFCC features shaped as a matrix? Could the Authors ellaborate
```
For the VGG11-based experiment on Google Speech Commands, we use the Mel-frequency cepstral coefficients (MFCC) method. Specifically, to pre-process the raw audio data, a sequence of overlapping Hamming windows is applied to the raw speech signal with a time shift of 10 ms and window size of 25ms. The MFCC is used for training the keyword spotting model. Take a 1-second long audio signal as an example, the input to the CNN-based model would be a 98 * 40 (T * d) matrix.

For the lightweight model experiment, to pre-process the raw audio data, a sequence of overlapping Hamming windows is applied to the raw speech signal with a time shift of 10 ms. We calculate the discrete Fourier transform (DFT) with a frame length of 1,024 and compute the Mel-spectrogram with a dimension of 128. The Mel-spectrogram is used for training the keyword spotting model. We exactly follow one related work [1] for this setup.

We have addressed the description of the pre-processed method for audio data in the revised version of the paper and highlighted it in blue. Thank you again for your valuable suggestions.

[1]. Zhang, T., Feng, T., Alam, S., Lee, S., Zhang, M., Narayanan, S.S., & Avestimehr, S. (2022). FedAudio: A Federated Learning Benchmark for Audio Tasks. ArXiv, abs/2210.15707.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer 21P2 (7/8)
comment: ```
(7/8) I must say that stating that "TimelyFL improves inclusiveness" is pushing the buzzwords game a bit too much. Why not use "fairness" or just present it in the same way as later in 4.3: "TimelyFL brings heterogeneity-awarenes". Both quotes refer to the same aspect that TimelyFL is offering a solution (a good one!) for.
```

Thank you for pointing out this issue. The term “improving inclusiveness” in our paper indicates the increment of the participation rate, as shown the first time in the 5th paragraph of the Introduction. We have no meaning in introducing the buzzwords to the audience. Based on your suggestion, we have added a footnote to Section 4.3 to explain the term “TimelyFL improves inclusiveness” and highlighted it in blue.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer 21P2 (6/8)
comment: ```
(6/8) Experiments in Table 1 are conducted on a server with a single GPU (as indicated in A.1.2). How would TimelyFL benefit/suffer from running on a server with more GPUs? I understand this would only applicable to simulation scenarios (but still would be a good contribution to any FL framework that wants to present this as a useful feature to have for asyncFL). Could the Authors comment upon this?
```

We believe that TimelyFL would benefit from running on a server with more GPUs. Currently, the simulation experiments are conducted on a single GPU, which means that the single GPU simulates both local client training and server aggregating. Suppose we have the resources to run with multiple GPUs. In that case, we could run the experiments in a distributed manner, which would accelerate the simulation training and enlarge the scalability of the simulation experiment.

If we could run it with multiple GPUs, we also could implement the TimelyFL on a real-world cross-silo setup. One application scenario is that serval users would operate different computational tasks on the same GPU server, making the availability of each GPU becomes a skewed distribution. For example, some GPUs may have a utility rate of around 80%, but the rest may only have around 30%. With the implementation of TimelyFL, the user could fully use the GPUs with different resource availability by adapting partial model training. We would implement our framework on a real server with multiple GPUs and mark it as a promising future work direction.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer 21P2 (5/8)
comment: ```
(5/8) It would had been more interesting to see the performance of TimelyFL when the tasks at hand is very lightweight (e.g. w/ SpeechCommands) AND when the model is adequately chosen for such lightweight task. VGG11 is not appropriate. How many parameters does it have? Just for context, keywordspotting is regularly done with very small models (often with less than 1M parameters -- https://arxiv.org/abs/2010.11267). Why not using a small model?
```

The VGG11 we used in the paper has a parameter size of 32,449,188. We select large models such as VGG11 and ALBERT to see the effectiveness of the adaptive partial training on devices with heterogeneous resource availability.

We do agree that lightweight models could be chosen for the keyword-spotting task. Based on your suggestion, we implement a lightweight model on the Speech Commands dataset in order to see the effectiveness of TimelyFL on a small model. Following one recent work [1], we choose the model that consists of two convolution layers followed by one Gated Recurrent Units (GRU) layer. An average pooling layer is connected to the GRU output, which is then fed through two dense layers to generate the predictions. The parameter size of this model is equal to 79,044. We followed [1] to sample 106 clients per round. The results are listed in the table below.

| Dataset       | Agg. Function | Accuracy | TimelyFL         | FedBuff                  | SyncFL                    |
|---------------|---------------|----------|------------------|--------------------------|---------------------------|
| Google Speech | FedAvg        | 70%      | 2.23 +/- 2.1% hr | 3.55 +/- 1.9% hr (1.59x) | 18.37 +/- 0.6% hr (8.24x) |
|               |               | 80%      | 4.16 +/- 1.3% hr | 6.13 +/- 1.4% hr (1.47x) | 32.36 +/- 0.4% hr (7.80x) |
|               | FedOpt        | 70%      | 0.48 +/- 1.7% hr | 1.66 +/- 1.0% hr (3.46x) | 4.61 +/- 2.1% hr (9.60x)  |
|               |               | 80%      | 1.13 +/- 1.2% hr | 3.25 +/- 0.8% hr (2.88x) | 7.47 +/- 1.1% hr (6.61x)  |

As shown in the table, TimelyFL achieves a higher convergence speed compared with the other two strategies before reaching the test accuracy, which confirms the simulation results elaborated in Section 4.2. We have updated these results in the revised version of the paper and highlighted them in blue. 

[1]. Zhang, T., Feng, T., Alam, S., Lee, S., Zhang, M., Narayanan, S.S., & Avestimehr, S. (2022). FedAudio: A Federated Learning Benchmark for Audio Tasks. ArXiv, abs/2210.15707.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer 21P2 (4/8)
comment: ```
(4/8) The description provided for Google Speech Commands is not correct. SpeechCommands (V2) has 105829 .wav files from 2618 speakers (2112 for training), 256 for validation and 250 for testing.
```

Thank you so much for pointing out this error. We have fixed this description carefully and highlighted it in blue. We will do more carefully and detailedly in the dataset description in our future works. Thanks again for the comments.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer 21P2 (3/8)
comment: ```
(3/8) Algorithm 1 isn't clear. First the client samples n clients out of S, but then all clients do local time update, and parallel training? (both stages do Clients c belong to S). Is this correct?
```
As shown in the third line of Algorithm 1, the TimelyFL server would randomly sample n clients from all clients to construct S (Sample n clients for S uniformly at random). As a result, S stands for the collection of the sampled clients per round, not for all of the clients. Only the sampled clients would perform the local time update and parallel training.

To further clarify the concerns, we added detailed descriptions for the S in the paper, which is highlighted in blue.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer 21P2 (2/8)
comment: ```
(2/8) It is not clear from the the results in Table 1, which scenario is this? the one with Android phones or the one with CPU/GPU training nodes? I'm assuming this is the latter, since Figure 8 then states in the caption "real-world deployment".
```

The results in Table 1 are from the CPU/GPU-based training. We have addressed it in our revised version of the paper.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer 21P2 (1/8)
comment: Thank you for your positive feedback and constructive comments. Our review responses are summarized below. We also have addressed your suggestions in the revised version of the paper. The revision part is highlighted by the blue color on the paper.

```
(1/8) Is Figure 1.c a zoomed-in portion of Figure 4.a?
```

Yes, it is. The blue dot line in Figure 1.c indicates the final accuracy for the synchronous FedOpt baseline.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer L6W9 (5/5)
comment: 
```
(5/5) In Figure 1c, there is a gap between the final accuracy achieved by synchronous FL and TimelyFL. Why is this the case? With proper scheduling, it should be possible for asynchronous methods to achieve the same final accuracy as synchronous methods.
```

Some previous works have pointed out that the effects of the stale update on distributed machine learning can directly harm the overall convergence behavior for the final model, aligned with the asynchronous distributed optimization theory suggested by [1, 2, 3], which functions as the main reason for the performance gap between the async and synchronous FL. By implementing TimelyFL, we aim to shrink this gap through adaptive partial training. Theoretically speaking, the syncFL would harvest the strongest performance among all baselines, as each client could fully train and upload the local model, which is nearly impossible to make it deployable due to real challenges, such as unstable communication and limited computational resources for some participants. The difference between syncFL and TimelyFL is mainly due to two reasons. Firstly, as shown in Figure 1b, some clients still could not catch up with the aggregation interval in each round due to communication or computation disturbance. Secondly, the weak device only reports the local update for the partial model instead of the update from full model training.

[1]. How slow can you go? In ICML, 2018.

[2]. Toward understanding the impact of staleness in distributed machine learning. In ICLR, 2018

[3]. At stability’s edge: How to adjust hyperparameters to preserve minima selection in asynchronous training of neural networks? In ICLR, 2019

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer L6W9 (4/5)
comment: ```
(4/5) The adaptive (partial) model training in TimelyFL gives a tradeoff between: (1) training more layers and for less iterations, (2) training less layers and for more iterations. Both can consume the same amount of time. It would be interesting to study what is the best way to balance this tradeoff, but the paper does not seem to include such a study. Moreover, it is worth discussing how to choose the best number of layers to train.
```

There might be some misunderstanding regarding Algorithm 3 in our paper. The workload scheduling function would give each client one of the below options: (1) Fully train the model with one or more local epochs. (2) Training fewer layers of the model and for only one epoch. Both cases would consume nearly the same amount of time to eliminate the local update's staleness. As a result, there is no tradeoff between training more layers with fewer iterations and training fewer layers with more iterations.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer L6W9 (3/5)
comment: ```
(3/5) The proposed TimelyFL algorithm has some key parameters, such as Tk. The choice of Tk (or more specifically, k) can have a critical impact on the performance. However, the paper does not seem to discuss this.
```

As stated in Appendix A.1.4, the aggregation goal and aggregation participation target are searched from 30% to 50% of training concurrency per round for FedBuff and TimelyFL, respectively.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer L6W9 (2/5)
comment: ```
(2/5) Although a decent amount of experiments has been done, there is not much discussion on any unique or surprising results that was seen, or any in-depth insights that can improve the understanding of practical federated learning systems. This makes this paper insufficient also from a practical/empirical point of view.
```

We believe our algorithmic and system contribution bring solid insight for future work in this direction, as we summarized below:

1. Our paper brings a critical observation that stale local updates would significantly harm the model accuracy for asynchronous federated learning, which serves as the main drawback for the current SOTA work.
2. To fully optimize the training efficiency under cross-device FL setups, the design of the training system should adaptively schedule the training parameters based on the real-time capability of each device, due to the unstable computation and communication availability of the mobile device.
3. Our empirical results demonstrate that for the weak devices, the local update on partial model training without staleness would bring better performance contribution for the global model compared to the update on the full model with staleness.

Based on our current progress, we believe that the layer-wise adaptive training in federated learning would be a promising direction. Currently, our layer-wise selection mechanism emphasizes the hardware-heterogeneity aspect. We would continue working on a more detailed layer-wise analysis as promising future work in the efficient training domain.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer L6W9 (1/5)
comment: We thank the reviewer for the valuable feedback, and our response is below. We sincerely hope we can clarify your concerns based on our feedback!

```
(1/5): The contribution of this paper is very straightforward and hence the novelty is low. It is a combination of a few simple and known techniques. Therefore, this work is mostly an engineering effort and there does not seem to be much research needed to achieve this.
```

We understand the reviewer's perspective on the novelty of our work. Our work starts from the current SOTA work [1] in the asynchronous FL system design, as we cite and compare against in the paper. To our best knowledge, we are the first work to point out the drawback of the current SOTA works in an asynchronous federated learning framework. In the introduction, we detailedly elaborate on why the current SOTA, FedBuff, leads to an accuracy drop compared to the final accuracy of SyncFL. Specifically, our paper brings a critical observation that stale local updates would significantly harm the model accuracy for asynchronous federated learning. We want to emphasize that our core contribution is how to implement the partial model training technique to tackle the drawbacks of the SOTA asynchronous federated learning framework. Our evaluation results show that our result significantly improves the performance compared to the SOTA asynchronous system design. 

In addition, the partial training technique we present is different compared to other SOTA partial training techniques in FL, such as HeteroFL [2] and FjORD [3]. For the asynchronous FL specifically, our method would bring more adaptivity and flexibility to the training system, as it does not require changing the model size or architecture every round for each client. Based on the above two points, we believe our algorithmic and system contribution can bring insight for future work in this direction.


[1] Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M.G., Malek, M., & Huba, D. (2022). Federated Learning with Buffered Asynchronous Aggregation. AISTATS.

[2] Diao, E., Ding, J., & Tarokh, V. (2021). HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients. ArXiv, abs/2010.01264.

[3] Horváth, S., Laskaridis, S., Almeida, M., Leondiadis, I., Venieris, S.I., & Lane, N.D. (2021). FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout. ArXiv, abs/2102.13451.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer rkLW (2/2)
comment: ```
(2/2): There is no theoretical analysis on the convergence of the proposed method
```
Compared to the SOTA asynchronous FL work [1] we cite in this paper, TimelyFL introduces partial model training and adaptive workload scheduling that brings more complexity from the system level. Under the TimelyFL setting, the condition is complex as the training parameters, such as partial training rate and local epoch number, are changed in every communication round. We do plan to add a proof framework for the method. But you know that it is very difficult to analyze convergence for a training framework like ours, which considers both system and learning. Similar to other SOTA works for federated learning with partial model training [2,3], we mainly emphasize our empirical analysis and real-world deployment. We prefer to try theoretical analysis as our future independent work. Our empirical results in Section 4 and Appendix supports our proposed method. We have marked it as a non-trivial future work for this paper.

[1] Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M.G., Malek, M., & Huba, D. (2022). Federated Learning with Buffered Asynchronous Aggregation. AISTATS.

[2] Diao, E., Ding, J., & Tarokh, V. (2021). HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients. ArXiv, abs/2010.01264.

[3] Horváth, S., Laskaridis, S., Almeida, M., Leondiadis, I., Venieris, S.I., & Lane, N.D. (2021). FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout. ArXiv, abs/2102.13451.

From: ICLR.cc/2023/Conference/Paper1260/Authors
title: Response to Reviewer rkLW (1/2)
comment: We thank the reviewer for the thoughtful review. Here are our responses.

```
(1/2) I am a little concerned about the novelty of the proposed method. Two of the main components of the methods, different local epochs and partial model training based on device capability, might be proposed by previous papers.
```

We understand the reviewer's perspective on the novelty of our work, especially on partial training. We want to emphasize that our core contribution is how to implement the partial model training technique to tackle the drawbacks of the SOTA asynchronous federated learning framework. In addition, our proposed partial training technique in TimelyFL is different from the previous related works, such as HeteroFL [1] and FjORD [2].

As the reviewer points out, HeteroFL [1] and FjORD [2] showed promising FL performance on heterogeneous systems. They commonly assign only a part of every network to weak clients to reduce the workload. One notable point is that under the above system frameworks, the weak client would have a different model size than the global model. While this width reduction method alleviates the weak clients’ workload burden, they suffer from a substantial accuracy drop when there are many weak clients, as shown in the recent work [3]. 

Unlike the above methods, the layer-wise partial training method that we implemented in the TimelyFL does not alter the model size or architecture on the client side. Instead, our core idea is to assign an output-side consecutive subset of network layers to the weak clients and only let them perform the backward pass within the assigned layers. It is a simple and efficient way for weak clients to contribute to the global model training when they focus on the outside layers only. It differs from the previously proposed efficient training method under the FL framework. For the asynchronous FL specifically, our method would bring more adaptivity and flexibility to the training system, as it does not require changing the model size or architecture every round for each client. 

We also want to emphasize that our core contribution is how to implement the partial model training technique to tackle the drawbacks of the SOTA asynchronous federated learning framework. As we propose an asynchronous aggregation framework, our direct comparison is to other asynchronous system designs rather than partial model techniques. In addition, our paper brings a critical observation that stale local updates would significantly harm the model accuracy for asynchronous federated learning. Our evaluation results show that our result significantly improves the performance compared to the SOTA asynchronous system design. In summary, we believe that we are very different from the previous works in the above two aspects.

[1] Diao, E., Ding, J., & Tarokh, V. (2021). HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients. ArXiv, abs/2010.01264.

[2] Horváth, S., Laskaridis, S., Almeida, M., Leondiadis, I., Venieris, S.I., & Lane, N.D. (2021). FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout. ArXiv, abs/2102.13451.

[3] Niu, Y., Prakash, S., Kundu, S., Lee, S., & Avestimehr, S. (2022). Federated Learning of Large Models at the Edge via Principal Sub-Model Training. ArXiv, abs/2208.13141.

From: ICLR.cc/2023/Conference/Paper1260/Reviewer_rkLW
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: Recent asynchronous FL methods have been proposed to overcome straggler effect occurred in FL with heterogeneous clients. However, such asynchronous methods suffer from accuracy drop and slower convergence rate compared with synchronous methods. In this paper, authors hypothesize that the main reason is that “ fast-speed devices contribute to many more rounds of aggregation while others join more intermittently or not at all, and with stale model updates.” To overcome this, authors propose TimelyFL which assigns smaller workload to slower clients (by reducing local epochs and freezing subsets of network layers) such that all clients finish local training at about the same time and participate in global model update, which is closer to synchronous method. Authors claim TimelyFL allows more client participation and no staleness update, which results in significantly higher final accuracy and faster convergence.
strength_and_weaknesses:  Strength:

1: The motivation is clear and important. As authors said, closing the gap between asynchronous and synchronous FL methods is very important for FL community. 

2: The proposed method is simple but seems to be very effective in all experiment settings. TimelyFL shows significant increase in final accuracy and convergence rate with heterogeneous data distribution and heterogeneous device.

3: Authors perform experiments on real mobile devices, which is very valuable and rare in most of the federated learning papers. This should be commended on its own.

Weakness:

1:  I am a little concerned about the novelty of the proposed method. Two of the main components of the methods, different local epochs and partial model training based on device capability, might be proposed by previous papers. As stated by authors in the related work section, partial model training is already proposed to solve device heterogeneity (additional references such as FjORD and HeteroFL). Although authors claim they add dynamic partial ratio, I am not convinced this is novel enough. 

2: There is theoretical analysis on the convergence of the proposed method. Using different local epochs and partial model training will significantly change the training dynamics of the model. Thus, authors should include a more theoretical analysis, as in other asynchronous FL papers.

clarity,_quality,_novelty_and_reproducibility: Clarity, quality:

Overll very good.

Novelty:

Might have some concern in novelty as stated above.

Reproducibility:

Seems good but no code is provided.
summary_of_the_review: I recommend weak accept as this is in general well-motivated paper with a simple but effective method, though some might argue about its lack of novelty and theoratical analysis.

correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1260/Reviewer_L6W9
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper presents a method (called TimelyFL) for efficient federated learning. The main ideas include allowing asynchronous updates until a certain amount of time has elapsed and adapting the number of layers to update via backpropagation. The performance has been evaluated on various datasets in different domains and a platform involving mobile phones.
strength_and_weaknesses: Strength:
- It is important to improve the efficiency of federated learning.
- There is an extensive set of experiments to evaluate the performance.

Weakness:
- The contribution of this paper is very straightforward and hence the novelty is low. It is a combination of a few simple and known techniques. Therefore, this work is mostly an engineering effort and there does not seem to be much research needed to achieve this. 
- Although a decent amount of experiments has been done, there is not much discussion on any unique or surprising results that was seen, or any in-depth insights that can improve the understanding of practical federated learning systems. This makes this paper insufficient also from a practical/empirical point of view. 
- The proposed TimelyFL algorithm has some key parameters, such as $T_k$. The choice of $T_k$ (or more specifically, $k$) can have a critical impact on the performance. However, the paper does not seem to discuss this. 
- The adaptive (partial) model training in TimelyFL gives a tradeoff between: (1) training more layers and for less iterations, (2) training less layers and for more iterations. Both can consume the same amount of time. It would be interesting to study what is the best way to balance this tradeoff, but the paper does not seem to include such a study. Moreover, it is worth discussing how to choose the best number of layers to train. 
- In Figure 1c, there is a gap between the final accuracy achieved by synchronous FL and TimelyFL. Why is this the case? With proper scheduling, it should be possible for asynchronous methods to achieve the same final accuracy as synchronous methods. 
clarity,_quality,_novelty_and_reproducibility: Clarity: The contribution itself is clear, but many key discussion points are missing as mentioned above. 

Quality: The quality of this paper is low and it is below the acceptance bar of high-quality conferences such as ICLR.

Novelty: The novelty of this work is limited.

Reproducibility: The algorithm itself is easy to implement. It may be difficult to reproduce the result on mobile phones though. 
summary_of_the_review: The contribution is straightforward and the paper lacks technical depth overall. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1260/Reviewer_21P2
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: TimelyFL addresses an important problem in previous asynchronous FL methods: training is unintentionally biased towards more capable devices or those with less data (i.e. those that train the local-training stage faster). This results in slower convergence rates and lower global accuracy since some of the data in slower clients might not be considered often enough. To address this issue, the Authors present a simple yet effective method to normalise the workload done at each client: more capable clients do more work (i.e. train the whole model) whereas more constrained devices train only a fraction of the layers. In this way, the number of participating clients that contribute to each aggregation step varies over time relative to the capabilities of the sampled set of clients in each round.
strength_and_weaknesses: ### Strengths
*    TimelyFL seems simple to apply and brings clear advantages.


### Weaknesses

*    Is Figure 1.c a zoomed-in portion of Figure 4.a?
*    It is not clear from the the results in Table 1, which scenario is this? the one with Android phones or the one with CPU/GPU training nodes? I'm assuming this is the latter, since Figure 8 then states in the caption "real-world deployment".
*    Algorithm 1 isn't clear. FIrst the client samples `n` clients out of `S`, but then all clients do local time update, and parallel training? (both stages do Clients $c\in\mathcal{S}$). Is this correct?
*    The description provided for Google Speech Commands is not correct.  SpeechCommands (V2) has 105829 .wav files from 2618 speakers (2112 for training), 256 for validation and 250 for testing.
*    It would had been more interesting to see the performance of TimelyFL when the tasks at hand is very lightweight (e.g. w/ SpeechCommands) AND when the model is adequately chosen for such lightweight task. VGG11 is not appropriate.  How many parameters does it have? Just for context, keywordspotting is regularly done with very small models (often with less than 1M parameters -- https://arxiv.org/abs/2010.11267). Why not using a small model?

Experiments in Table 1 are conducted on a server with a single GPU (as indicated in A.1.2). How would TimelyFL benefit/suffer from running on a server with more GPUs? I understand this would only applicable to simulation scenarios (but still would be a good contribution to any FL framework that wants to present this as a useful feature to have for asyncFL). Could the Authors comment upon this?

I must say that stating that "TimelyFL improves inclusiveness" is pushing the buzzwords game a bit too much. Why not use "fairness" or just present it in the same way as later in 4.3: "TimelyFL brings heterogeneity-awarenes". Both quotes refer to the same aspect that TimelyFL is offering a solution (a good one!) for.
clarity,_quality,_novelty_and_reproducibility: The paper is correctly written with only some minor grammar mistakes here and there. I would had appreaciate if captions in Figures and Tables would be more self-contained. It seems that most of the hyperparameters are disclosed in the appendix. But something still hasn't been discussed regarding speechcommands: how is the input (a 1second-long audio signal) pre-processed before passing it to the CNN? I presume the Authors do the usual and compute the MFCC features shaped as a matrix? Could the Authors ellaborate
summary_of_the_review: TimelyFL does not bring new ideas to the FL domain or even to the asynchronous FL paradigm. However, this doesn't mean is not a valuable work. The Author used a relatively simple methodology to adjust the workload on each client based on their capabilities and enable more devices in a given round to push their update to the server before the end of the synchronisation window. I would had like to see TimelyFL applied to other settings where the workload of the clients is much smaller, in particular for SpeechCommands. I encourage the Authors to include that additional study during the rebuttal. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1260/Reviewer_touo
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper focuses on the asynchronous federated learning (FL) problem. Specifically, the authors found that the recent methods (e.g., FedBuff) show lower training accuracy and slower convergence rate due to the imbalance in the contribution to aggregation among different models. To address this issue, they proposed TimelyFL, a heterogeneity-aware asynchronous FL method, which improves the asynchronous FL methods by adjusting the local training workload. Both effectiveness and feasibility are well demonstrated by conducting extensive experiments in various settings.
strength_and_weaknesses: Strength:
1. The problem focused on by this paper is very practical and important in federated learning, especially the scalability and feasibility of pushing federated learning toward a more practical scenario.
2. The proposed TimelyFL shows empirical effectiveness in achieving similar test accuracy faster than previous asynchronous FL methods. 
3. Extensive experiments and analysis, as well as the android-based FL testbed, are provided to justify the proposed method, which shows the efficiency and effectiveness of TimelyFL.

Weaknesses:
1. The writing and presentation of the current version could be further improved, and some clarification in this paper is not well supported. It may lead to some parts being hard to follow (at least for me). The detailed question and comments are listed below.
2. The motivation for the proposed method is a little bit unclear, which seems to be based on the intuitive explanation in the second paragraph of page two. It may need more in-depth analysis or justification.
3. This paper lacks some theoretical analysis of the focused "learning" problem as well as the convergence. It makes readers hard to understand the underlying mechanism of TimelyFL in federated learning. The authors may consider reorganizing the method section and highlighting the previous issues and corresponding improvements in TimelyFL.

Some further questions and comments:
1. Could the author discuss more the negative effects of waiting for all clients participating in each round or give some reference about this problem?
2. "asynchronous communication" is also an important and practical problem in distributed systems [1], it seems there are also some proposals or strategies in traditional distributed systems. Considering FL, the new learning paradigm with distributed models, what is the unique point of this problem? Could the author also give more discussion? (not only in the FL area, as the problem is general)
3. If there is enough time for training, what will happen after 14 hr in Figure 1 (c)? It seems that the FedBuff is still gaining higher test performance along with the training, considering the current green curve.
4. The current version of this work shows most of the results about the comparison with FedBuff to present the technical issue. It would raise the question of whether the phenomenon also exists in other asynchronous FL methods.

[1] F. Cristian and C. Fetzer, "The timed asynchronous distributed system model," in IEEE Transactions on Parallel and Distributed Systems, vol. 10, no. 6, pp. 642-657, June 1999, doi: 10.1109/71.774912.
clarity,_quality,_novelty_and_reproducibility: Clarity:
1. Some clarifications in the current version of this work are not well supported, which could be further improved, like the general existence of the drawback of previous asynchronous FL (since it is important as the motivation of TimelyFL, if I understand correctly) and the further analysis about the convergence rate, either theoretical or empirical. 

Quality:
1. The presentation quality of the current version could be further improved to highlight and describe more about the proposed method, especially the working mechanism for the learning problem (which directly affects the drop in test accuracy).

Novelty:
1. The novelty of TimelyFL is somehow new compared with the previous asynchronous FL method. While the partial model training and different learning schedule design in FL are common in the area, this work can be further improved by adding more analysis to highlight the uniqueness of the addressed issue and corresponding mechanism.

Reproducibility:
1. This paper provides detailed information about experimental settings and algorithm descriptions. It would be better if the authors could open-source the code.


summary_of_the_review: Overall, this paper proposed a new asynchronous federated learning (FL) method, i.e., TimelyFL, to achieve a compatible test performance and a faster convergence rate, introducing the local training adjustment and partial model training in the FL framework. Extensive experiments and the adaptation to a real-world scenario, are provided to justify the efficiency of TimelyFL. However, the current version could be further improved by strengthening its clarification and adding more discussion or analysis to perfect its presentation.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold