Paper: FedCompass: Efficient Cross-Silo Federated Learning on Heterogeneous Client Devices Using a Computing Power-Aware Scheduler (ICLR-2024)

Reviewer comment (Novelty):
The motivation is a bit unconvincing. The straggler issue is considered important primarily in cross-device FL since the organizations are expected to have sufficient computational/communication resources in cross-silo FL [arXiv’22].

Author response:
**Unconvincing motivations**

The straggler issues in cross-device FL are mainly caused by the unreliability of the clients due to intermittent compute availability and connectivity. On the other hand, in cross-silo FL, though each client is assumed to have reliable and dedicated resources, the straggler issues come from the disparity/difference of the computing power among clients. In practical settings, there usually exists a substantial gap in computing capability between cross-silo FL clients.

Firstly, the increasing adoption of specialized ASICs like those from Cerebras, SambaNova, and Graphcore, particularly by large institutions engaged in FL projects such as national laboratories, is noteworthy. At the other end of the spectrum, institutions like hospitals, also interested in utilizing federated learning, may only have access to standard consumer GPUs or solely CPU-based computing resources. This diversity in the types of accelerators and computing machines used by cross-silo FL clients inevitably leads to substantial disparities in computational capabilities.

Secondly, disparities in computing power are also evident even among clients using the same type of accelerator, such as GPUs. For instance, HPC clusters built with different generations of GPUs and associated support hardware, like networking components, can exhibit significant differences in training capabilities. This is further elaborated in the following NVIDIA post (https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/).

Considering these disparities among cross-silo clients' computing resources, it becomes crucial to address resource utilization efficiently. The aim is to prevent resource wastage, which can occur by having faster clients idly wait for slower ones and underutilizing the available resources. A typical scenario in HPC clusters illustrates this: when a user requests a node with 8 GPUs through a job scheduler like Slurm, they are billed for a fixed number of compute hours, regardless of whether all 8 GPUs are fully utilized throughout the job duration.

Reviewer comment (Experiments):
Most experiments were conducted on datasets with a small number of classes. To confirm the scalability of the proposed method, could the authors conduct experiments on datasets with more classes like CIFAR-100?

Author response:
**Using CIFAR-100 as datasets**

In our experiments, we selected the MNIST and CIFAR-10 datasets to evaluate the performance of FedCompass. These two datasets are widely recognized and frequently utilized in FL research, as evidenced by Ma et al. (2022), who noted their prevalent use in the field. According to a survey presented in their study (Figure 9), MNIST is used in 32% of FL experiments, while CIFAR-10 is used in 28%. CIFAR-100, in contrast, is seldom chosen for such experiments. Beyond these traditional datasets, we also assessed FedCompass using two datasets from FLamby (Terrail et al., 2022), a recently introduced benchmark specifically designed for cross-silo FL. These datasets are significantly larger, at 444MB and 9GB, respectively. We believe this diverse range of datasets, both in terms of size and complexity, adequately demonstrates the versatility and efficacy of FedCompass in various scenarios.

Reviewer comment (Novelty):
The idea of adjusting local epochs for each client is straightforward and the technical novelty of the proposed method seems somewhat limited.

Author response:
**Limited technical novelty**

At first glance, the concept of assigning varying local steps to clients based on clients' computing speeds might seem straightforward. However, in practice, this approach encounters several challenges and edge cases. For example, these include synchronizing clients when there is no prior knowledge of their computing speeds, addressing minor and significant fluctuations in client speeds, and handling scenarios where there is a substantial disparity in client computing power (e.g., exponential client computing speed distributions).

FedCompass not only addresses these practical challenges effectively but also provides theoretical proof of convergence and demonstrates robust performance through an extensive series of experiments. Therefore, we believe that this paper possesses sufficient novelty, underscoring its contribution to advancing FL methodologies.

Ma, X., Zhu, J., Lin, Z., Chen, S., & Qin, Y. (2022). A state-of-the-art survey on solving non-IID data in Federated Learning. Future Generation Computer Systems, 135, 244-258.

Terrail, J. O. D., Ayed, S. S., Cyffers, E., Grimberg, F., He, C., Loeb, R., ... & Andreux, M. (2022). Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. arXiv preprint arXiv:2210.04620.

Reviewer comment (Writing):
There are some typos in Appendix that should be corrected (e.g., in equation (6) of the proof for Lemma 2, the gradient symbol is missing).

Author response:
**Typos**

Thanks for the reminder! We have fixed it in the revised paper.

Reviewer comment (Theory):
The description of the algorithm in section 3.2 is unclear and further discussion is required. Specifically, the assignment of groups and the group aggregation are major components of the proposed method and they are not presented or discussed sufficiently in the main body of the paper (I strongly believe that including these components or at least an extensive discussion about them in the main body of the paper would significantly improve the presentation of this work). It is unclear how the implementation of these components alleviates the bias introduced by faster nodes performing more updates than slower nodes (a common issue met in previously proposed asynchronous methods). It is also unclear how the overall method reaches an equilibrium (as discussed in section 3.1) and what properties it has. I would appreciate it if the authors could elaborate on the above.

Author response:
**Unclear description in Section 3.2**

For Section 3.2, we now rewrote the paragraph for describing the server-side algorithm of FedCompass by focusing on the higher-level idea instead of the low-level details (last two paragraphs at the end of page 5). We believe that now it is a more clear presentation of the algorithm.

**How FedCompass alleviates the bias towards faster clients**

Compared with other asynchronous FL algorithms such as FedAsync (Xie et al., 2019) and FedBuff (Nguyen et al., 2022), FedCompass alleviates the bias towards faster clients by using group aggregation to reduce global aggregation frequency and avoids the frequent application of staleness factors. We would like to further elaborate this via a simple example:

Suppose that client A is 4 times faster than client B. In FedAsync, if A and B perform the same number of local steps in each local training round, then client A will perform 4 times more local training rounds than client B. When client B’s model arrives, the global model has already been updated four times using A’s model, so B’s local model will become very stale/outdated, and another penalty staleness factor (e.g., say 0.25) will be applied to B’s model during the aggregation. In such a case, B not only performs fewer updates, but its model also experiences an additional penalty factor to further cause the global model to drift away from it. For FedBuff, although it has a buffer to reduce the global update frequency, it is very possible that the buffer contains several local models from the same fast clients and slower clients may also experience some staleness factor. However, for FedCompass, we are grouping the arrival of clients, so in this example, A will perform 4 times more local steps than B and they will arrive almost simultaneously at the server for a group aggregation. In this case, no staleness factor (i.e., that 0.25) is applied to B, so the drift towards A is mitigated compared to other methods.

Of course, from the above description, it is clear that FedCompass only mitigates the drift caused by the stateless factor but does not eliminate it, as client A still performs 4 times more steps than B. The drift can be further mitigated by combining FedCompass with other strategies such as normalized averaging (Nova) proposed in FedNova (Wang et al., 2020). Nova basically normalizes the client’s local gradient according to the number of local steps to mitigate objective bias towards faster clients. Now we also include the experiment results for this combination (FedCompss+N) in the last row of Tables 1 and 2 of the revised paper (which is also the response to another review’s comments). However, it is noteworthy that there is still a tradeoff here: though Nova can mitigate the objective bias, it also diminishes the updates from the faster nodes and may sometimes even lead to slower convergence of the global model.

**Questions about equilibrium**

Equilibrium means that the number of existing groups and the group assignments do not change. Take the scenario in Figure 1 as an example. Initially, there are two groups {1,2,3} and {4,5}. At 12:00, when clients {1,2,3} finish local training together, the Compass scheduler assigns them to the group of clients 4 and 5 and now all clients are within the same group. Then, we can see that all the clients will continue the training following the same grouping (assuming no drastic changes in the computing speed of clients during training), and the number of local steps performed by each client is proportional to the computing speed of each client.

Xie, C., Koyejo, S., & Gupta, I. (2019). Asynchronous federated optimization. arXiv preprint arXiv:1903.03934.

Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M., Malek, M., & Huba, D. (2022, May). Federated learning with buffered asynchronous aggregation. In International Conference on Artificial Intelligence and Statistics (pp. 3581-3607). PMLR.

Wang, J., Liu, Q., Liang, H., Joshi, G., & Poor, H. V. (2020). Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33, 7611-7623.

Reviewer comment (Theory):
In the description in section 3.2 it seems that when a client arrives later than $G[g].T_{max}$ it is immediately assigned a new training task whereas the clients that arrive earlier than $G[g].T_{max}$ are assigned training tasks when the next aggregation takes place. What is the justification for this differentiation?

Author response:
**Different behaviors for clients arriving before and after $T_{\max}$**

All clients within the same group are expected to arrive at almost the same time for a group aggregation. Therefore, if a client arrives earlier than the latest arrival time $T_{\max}$, then the server will wait for other pending clients (or $T_{\max}$, whichever is earlier) for a group aggregation. Then the clients can obtain new training tasks with the **most updated global model**. However, if a client arrives later than $T_{\max}$, it means that all other clients within the same group have already finished the group aggregation after waiting until $T_{\max}$. Therefore, there is no point in waiting and the server will immediately assign new training tasks to the client. We think the confusion is mainly caused by the unclarity in the original draft and we hope the revised version conveys the idea more clearly.

Reviewer comment (Theory):
Some of the assumptions required for the theoretical results are very restrictive. Specifically, the bounded gradient assumption rules out simple functions such as the quadratic. Further, the bounded heterogeneity and staleness assumption is rather strong and the derived results in Theorem 1 and Corollary 1 have a heavy dependence on quantity $\mu$. As a result, the impact of those results is diminished.

Author response:
**Restrictive assumptions for convergence analysis**

The first three assumptions are commonly used in convergence analysis of non-convex functions in federated learning and distributed learning literature (Stich, 2019; Li et al., 2019, Yu et al., 2019, Nguyen et al., 2022). In terms of the bounded heterogeneity and staleness assumption, it is valid and reasonable in cross-silo FL, as each client is considered to be reliable. We would also like to clarify that the main focus/contribution of our convergence analysis section is deriving the convergence behavior of our newly proposed semi-asynchronous algorithm FedCompass using the commonly-used assumptions and obtaining further insights (such as the impacts of different hyperparameters/setting on convergence), instead of trying to get rid of any commonly-used assumption such as bounded staleness. Additionally, this paper actually first studies convergence behavior of a semi-asynchronous FL algorithm with a scheduler, and we really believe this is still a good contribution to the FL community.

Reviewer comment (Novelty):
The related work section could be extended with works on device heterogeneity (Reisizadeh et al., 2022; Horvath et al., 2022) and on asynchronous FL (So et al., 2021). 

Reisizadeh, A., Tziotis, I., Hassani, H., Mokhtari, A., and Pedarsani, R. (2022). Straggler-resilient federated learning: Leveraging the interplay between statistical accuracy and system heterogeneity. 

Horváth, S., Sanjabi, M., Xiao, L., Richtárik, P., and Rabbat, M. (2022). Fedshuffle: Recipes for better use of local work in federated learning.

So, J., Ali, R. E., Güler, B., and Avestimehr, A. S. (2021). Secure aggregation for buffered asynchronous federated learning.

Author response:
**Citations to other related works**

Thanks a lot for providing those additional papers. We find them relevant and useful and have cited them in suitable places in the revised paper.

Reviewer comment (Novelty):
The adaptation of the number of local updates based on clients' computation speed is not new. An important and well-known baseline is FedNova (Wang et al., 2020), which has been cited in the paper but not compared against. Compared with the proposed FedCompass method, FedNova is much easier to implement and supports both cross-silo FL and cross-device FL with partial client participation. The advantage of FedCompass over FedNova is not clear.

Author response:
**Comparison between FedCompass and FedNova**

The main focus of FedNova (Wang et al., 2020) is how to eliminate objective inconsistencies when the number of local steps for clients is different (via adjusting weighting factors). In practice, FedNova is typically implemented in the following two **synchronous** manners:
(1) Each client performs the same number of local epochs, thus different number of local steps due to different data sizes. In this case, the algorithm suffers a lot from client computing speed variations like the FedAvg algorithm. In the worst case, the slowest client may have the largest dataset, and the whole training process can be extremely slow.
(2) Setting a universal completion time in each round and letting each client perform local updates until the completion time is reached. This approach eliminates the long waiting time and also has a simple implementation process, but it requires the universal completion time to be manually selected for different training tasks, model sizes, and average client computing capabilities.

On the other hand, FedCompass is designed to be an **asynchronous** federated learning algorithm, and focuses on how to employ a scheduler to have asynchronous clients arrive almost simultaneously. This is achieved by assigning different numbers of steps dynamically using real-time performance metrics. This design choice, while marginally complicating the implementation, significantly streamlines the experimental setup, as users only need to provide a range of desired local update steps in each round (which is more intuitive to users) and do not need to empirically estimate a suitable universal completion time. There is a tradeoff between ease of implementation and usage convenience.

Furthermore, we would like to clarify that FedNova and FedCompass are not analogous but actually “orthogonal” to each other. As FedCompass involves different client local steps in each round, it can be combined with FedNova’s normalized averaging strategy as a manner to further mitigate the drifting to faster clients. To further emphasize the FedNova paper and explain the orthogonal relationship between FedCompass and FedNova, we now provide a new set of experiment results for their combination in the last rows of Tables 1 and 2 (FedCompass+N) of the revised paper (as well as all result tables and figures in the Appendix).

Wang, J., Liu, Q., Liang, H., Joshi, G., & Poor, H. V. (2020). Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33, 7611-7623.

Reviewer comment (Experiments):
The proposed FedCompass approach is heuristic and a lot of its description is based on simplified examples. The paper argues that FedBuffer includes the buffer size $K$ as a hyper parameter that needs to be tuned. However, FedCompass also includes hyper parameters $Q_\mathrm{min}$, $Q_\mathrm{max}$, and parameters in sub-procedures, such as $\lambda$ in Algorithm 2, that need to be tuned heuristically. It is worth noting that the experiments in this paper use FedBuffer with small values of $K$ up to 5 (as listed in Appendix E) while the original FedBuffer paper (Nguyen et al., 2022) uses $K=10$ in the majority of experiments. It is not clear how the authors selected $K$ in the experiments presented in this paper, but I would expect that FedBuffer with a properly tuned $K$ would give a similar performance as FedCompass with properly tuned hyper parameters.

Author response:
**Hyperparameter problems for FedBuff and FedCompass**

First, thanks for the reminder to adopt FedBuff instead of FedBuffer for consistency with the original paper (Nguyen et al., 2022). We have changed all FedBuffer to FedBuff in the revised paper now.

Second, we would like to answer why we do not simply use $K=10$ as the original FedBuff paper does, especially given that their paper mentions that “Our extensive empirical evaluation finds that $K=10$ is a good setting across benchmarks and does not require tuning”. This is because all the experiments in FedBuff are in cross-device FL settings, and have **at least thousands of clients in all experiment settings** (CelebA dataset: 9343 clients, Sent140: 660120 clients, and CIFAR-10: 5000 clients). However, that statement does not hold for cross-silo cases, as we have less than or equal to 10 clients in most experiments. When the number of clients becomes small, the users have to decide the buffer size themselves instead of directly using $K=10$.

Then, we would like to describe how we decide the value of $K$ for different numbers of clients. We use the *class partitioned* MNIST dataset with {5, 10, 20} clients and normal distribution as the client heterogeneity setting. For 5 clients, we search $K$ among values {2,3}, for 10 clients, we search $K$ among {2, 3, 5}, and for 20 clients, we search $K$ among {3, 5, 10}. We select $K$ with the highest average accuracy of three independent runs. According to the table below, we finally select $K=3/3/5$ for 5/10/20 clients. For Fed-IXI, we choose $K=2$ as there are only three clients. For Fed-ISIC2019 with 6 clients, we choose $K=3$ as we use this buffer size for both 5 and 10 clients. Additionally, we would like to mention that from the training process visualization figures (Figure 2 and more in Appendix F), we can clearly see that there is a non-marginal gap between the FedCompass curve and the FedBuff curve in many cases, especially for the relatively complicated datasets, CIFAR10 and Fed-ISIC2019.

| Number of Clients | K=2     | K=3     | K=5     | K=10    |
|-------------------|---------|---------|---------|---------|
| 5 clients         | 94.13   | 94.77   |    -     |    -     |
| 10 clients        | 93.88   | 94.60   | 94.54   |    -    |
| 20 clients        |    -     | 97.65   | 98.21   | 98.16   |

Finally, regarding the hyperparameter tuning, we would like to elaborate on the reason why we state that FedCompass requires easier tuning. For FedCompass, it has three algorithm-specific hyperparameters: $Q_{\min}$, $Q_{\max}$, and $\lambda$, or equivalently, $Q_{\max}$, $Q_{\min}/Q_{\max}$, and $\lambda$. For FedBuff, it requires $Q$ and $K$. $Q_{\max}$ in FedCompass and $Q$ in FedBuff are both used to define the amount of computational work we want each client to do in each round, so they cancel each other out. The ratio $Q_{\min}/Q_{\max}$ is irrelevant to the number of clients, and we empirically show in the ablation study that a large range of values can lead to faster convergence than FedBuff (we now also added new ablation study results on the CIFAR-10 datasets to address another reviewer’s request, and the observation remains on the more difficult CIFAR-10 dataset). Similarly, for $\lambda$, its impact on the convergence is also minor and a number slightly above 1 is fine. On the other hand, for $K$ in FedBuff, though their paper mentions $K=10$ is a good choice when the client numbers are large, there does not exist any rule of thumb for selecting $K$ when the number of clients is small in cross-silo settings, so we had to do some small preliminary experiments to choose $K$ for different numbers of clients. For FedCompass, we simply choose $Q_{\min}/Q_{\max}=0.2$ and $\lambda=1.2$ for all experiments (which are not even optimal in some cases), and FedCompass still outperforms FedBuff all the time with a non-marginal gap.

Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M., Malek, M., & Huba, D. (2022, May). Federated learning with buffered asynchronous aggregation. In International Conference on Artificial Intelligence and Statistics (pp. 3581-3607). PMLR.

Reviewer comment (Writing):
Side note: The algorithm is called FedBuff in the original paper (Nguyen et al., 2022), not sure why the authors call it FedBuffer in this paper.

Author response:
**Hyperparameter problems for FedBuff and FedCompass**

First, thanks for the reminder to adopt FedBuff instead of FedBuffer for consistency with the original paper (Nguyen et al., 2022). We have changed all FedBuffer to FedBuff in the revised paper now.

Second, we would like to answer why we do not simply use $K=10$ as the original FedBuff paper does, especially given that their paper mentions that “Our extensive empirical evaluation finds that $K=10$ is a good setting across benchmarks and does not require tuning”. This is because all the experiments in FedBuff are in cross-device FL settings, and have **at least thousands of clients in all experiment settings** (CelebA dataset: 9343 clients, Sent140: 660120 clients, and CIFAR-10: 5000 clients). However, that statement does not hold for cross-silo cases, as we have less than or equal to 10 clients in most experiments. When the number of clients becomes small, the users have to decide the buffer size themselves instead of directly using $K=10$.

Then, we would like to describe how we decide the value of $K$ for different numbers of clients. We use the *class partitioned* MNIST dataset with {5, 10, 20} clients and normal distribution as the client heterogeneity setting. For 5 clients, we search $K$ among values {2,3}, for 10 clients, we search $K$ among {2, 3, 5}, and for 20 clients, we search $K$ among {3, 5, 10}. We select $K$ with the highest average accuracy of three independent runs. According to the table below, we finally select $K=3/3/5$ for 5/10/20 clients. For Fed-IXI, we choose $K=2$ as there are only three clients. For Fed-ISIC2019 with 6 clients, we choose $K=3$ as we use this buffer size for both 5 and 10 clients. Additionally, we would like to mention that from the training process visualization figures (Figure 2 and more in Appendix F), we can clearly see that there is a non-marginal gap between the FedCompass curve and the FedBuff curve in many cases, especially for the relatively complicated datasets, CIFAR10 and Fed-ISIC2019.

| Number of Clients | K=2     | K=3     | K=5     | K=10    |
|-------------------|---------|---------|---------|---------|
| 5 clients         | 94.13   | 94.77   |    -     |    -     |
| 10 clients        | 93.88   | 94.60   | 94.54   |    -    |
| 20 clients        |    -     | 97.65   | 98.21   | 98.16   |

Finally, regarding the hyperparameter tuning, we would like to elaborate on the reason why we state that FedCompass requires easier tuning. For FedCompass, it has three algorithm-specific hyperparameters: $Q_{\min}$, $Q_{\max}$, and $\lambda$, or equivalently, $Q_{\max}$, $Q_{\min}/Q_{\max}$, and $\lambda$. For FedBuff, it requires $Q$ and $K$. $Q_{\max}$ in FedCompass and $Q$ in FedBuff are both used to define the amount of computational work we want each client to do in each round, so they cancel each other out. The ratio $Q_{\min}/Q_{\max}$ is irrelevant to the number of clients, and we empirically show in the ablation study that a large range of values can lead to faster convergence than FedBuff (we now also added new ablation study results on the CIFAR-10 datasets to address another reviewer’s request, and the observation remains on the more difficult CIFAR-10 dataset). Similarly, for $\lambda$, its impact on the convergence is also minor and a number slightly above 1 is fine. On the other hand, for $K$ in FedBuff, though their paper mentions $K=10$ is a good choice when the client numbers are large, there does not exist any rule of thumb for selecting $K$ when the number of clients is small in cross-silo settings, so we had to do some small preliminary experiments to choose $K$ for different numbers of clients. For FedCompass, we simply choose $Q_{\min}/Q_{\max}=0.2$ and $\lambda=1.2$ for all experiments (which are not even optimal in some cases), and FedCompass still outperforms FedBuff all the time with a non-marginal gap.

Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M., Malek, M., & Huba, D. (2022, May). Federated learning with buffered asynchronous aggregation. In International Conference on Artificial Intelligence and Statistics (pp. 3581-3607). PMLR.

Reviewer comment (Theory):
In practice, the clients' computation speeds can vary over time due to the varying amount of concurrent tasks running in the system. The paper has a mentioning of this but only studies the behavior of the algorithm in some oversimplified cases of computation speed variation. It is unclear whether the scheduling procedure in Section 3.1 is robust to all types of speed variation, or does there exist a worst case scenario where the scheduler completely loses track in its estimation of clients' computation speeds. This needs a separate theoretical analysis IMHO, which is different from the convergence bound.

Author response:
**Evaluating the algorithm under drastic speed variations**

In our ablation study on FedCompass’s resilience to client speed changes in Appendix G.2, every client has a 10% chance of randomly sampling a new speed from the established speed distribution, irrespective of its previous speed. Consequently, when client speeds are modeled using an **exponential distribution**, there's a substantial likelihood of encountering drastic speed changes—some clients might experience speed alterations by a factor of 5x or even 10x. Take our experiments with the MNIST dataset as an example. By calculating the area under the curve of the exponential distribution's probability density function, we find that each client has a 15.35% chance of operating at speeds less than 0.025 seconds per update. Conversely, there's an 18.89% probability for speeds to exceed 0.25 seconds per update, which translates to more than a 10x difference in speed. These numbers intuitively illustrate the potential for drastic speed variations in our evaluation settings, emphasizing the robustness and adaptability of FedCompass in handling such extreme changes.

**Theoretical foundation of the FedCompass scheduler**

First, our scheduler aims to minimize the blocking time in a group of clients by varying the number of local steps, facilitating their near-simultaneous arrival. The contribution of this design lies in three key aspects:

(1) The scheduler introduces a novel approach for running FL experiments in a semi-asynchronous manner, which combines the benefits of synchronous algorithms (less client drift) with the quick convergence typical of asynchronous algorithms.

(2) The separation of the scheduling and the server aggregation makes FedCompass have the compatibility with a wide range of FL optimization techniques found in the literature, including server-side optimizations (Hsu et al., 2019; Reddi et al., 2020) and normalized averaging (Wang et al., 2020).

(3) FedCompass has shown superior performance across various datasets and client heterogeneity settings empirically and also comes with proof for convergence under certain assumptions.

Regarding the scheduler's optimality, proving such a claim is exceedingly challenging, if not impossible. The scheduler designs inherently involve trade-offs, and defining a universally "optimal" scheduler is complex. In FedCompass, this trade-off manifests in balancing synchrony and asynchrony. Also, we would like to mention that the trade-off happens in the scheduler design across various computer science fields. For instance, the SLURM job scheduler (Yoo et al., 2003), widely used for resource management in HPC clusters and supercomputers, is based on a complex finite state machine without a formal "optimality" proof. Similarly, in the realm of the operating system CPU schedulers, numerous algorithms exist (https://www.geeksforgeeks.org/comparison-of-different-cpu-scheduling-algorithms-in-os/), each with its trade-offs and none universally acknowledged or formally proved as the optimal.

In conclusion, while this paper does not present proof of the scheduler's optimality, considering the aforementioned advantages, we believe that FedCompass is still a great contribution to the FL community.

Hsu, T. M. H., Qi, H., & Brown, M. (2019). Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335.

Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., ... & McMahan, H. B. (2020). Adaptive federated optimization. arXiv preprint arXiv:2003.00295.

Wang, J., Liu, Q., Liang, H., Joshi, G., & Poor, H. V. (2020). Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33, 7611-7623.

Yoo, A. B., Jette, M. A., & Grondona, M. (2003, June). Slurm: Simple linux utility for resource management. In Workshop on job scheduling strategies for parallel processing (pp. 44-60). Berlin, Heidelberg: Springer Berlin Heidelberg.

Reviewer comment (Novelty):
The convergence result appears fairly straightforward, since results with bounded staleness and different numbers of local updates both exist in the literature. It is not quite clear what is the challenge and novelty in the convergence analysis.

Author response:
**Convergence analysis [updated]**

(1) The main focus/contribution of our convergence analysis section is deriving the convergence behavior of our newly proposed semi-asynchronous algorithm FedCompass using the commonly-used assumptions and obtaining further insights (such as the impacts of different hyperparameters/settings on the convergence), instead of trying to get rid of any commonly-used assumptions such as bounded staleness.

(2) Though we are not the first ones to study the convergence behavior of the general asynchronous FL algorithms or FL algorithms with different client local steps, this paper actually first studies convergence behavior of a semi-asynchronous FL algorithm with a scheduler, and we believe this is still a good contribution to the FL community.