Paper: FedCDA: Federated Learning with Cross-rounds Divergence-aware Aggregation (ICLR-2024)

Reviewer comment (Experiments):
Considering that the proposed method is about a new aggregation method, 20 clients and a participation ratio of 20% are too unrealistic.

Author response:
Thanks for the comments which help improve the comprehensiveness of our evaluations. As we know, there are many papers of which the experiments are conducted with 20 clients [1,2,3].  To avoid the sensitivity of our method over the number of clients, we conduct extensive experiments over **100** clients with a **participation ratio of 10%** and compare our method with FedAvg. The dataset is cifar-100, the learning rate is 0.1, the weight decay is 1e-3, the local epoch is 10, and the optimizer is SGD without momentum. The results are presented below:

| Method | Dir(0.1) | Dir(0.3) | Dir(0.5) |
|--------|---------|---------|---------|
| FedAvg | 38.89%(+-0.85%) | 40.38%(+-0.55%) | 42.23%(+-0.30%) |
| FedProx | 39.86%(+-0.45%) | 39.48%(+-0.37%) | 40.18%(+-0.46%) |
| FedExP | 38.04%(+-3.37%) | 44.10%(+-1.69%) | 41.79%(+-1.62%) |
| FedSAM | 16.35%(+-0.25%) | 20.53%(+-0.25%) | 25.70%(+-0.28%) |
| FedDF | 41.04%(+-0.57%) | 47.06%(+-0.74%) | 47.63%(+-0.53%) |
| FedGEN | 39.91%(+-1.72%) | 41.65%(+-1.35%) | 43.39%(+-1.08%) |
| FedMA | 39.12%(+-0.52%) | 40.42%(+-0.61%) | 42.89(+-0.21%) |
| GAMF | 39.89%(+-0.67%) | 40.98%(+-0.32%) | 43.25(+-0.34%) |
| FedLAW | 40.88%(+-0.66%) | 41.77%(+-0.78%) | 41.89(+-0.33%) |
| FedCDA | **47.38%** (+-0.23%) | **49.96%** (+-0.21%) | **51.04%** (+-0.19%) |



Our method can still achieve higher accuracy than FedAvg. 

[1] Zexi Li, Tao Lin, Xinyi Shang, Chao Wu: Revisiting Weighted Aggregation in Federated Learning with Neural Networks. ICML 2023	

[2] Zhuangdi Zhu, Junyuan Hong, Jiayu Zhou: Data-Free Knowledge Distillation for Heterogeneous Federated Learning. ICML 2021.

[3] Tao Lin, Lingjing Kong, Sebastian U. Stich, Martin Jaggi: Ensemble Distillation for Robust Model Fusion in Federated Learning. NeurIPS 2020

Reviewer comment (Experiments):
In addition to the local epoch, other hyperparameters used in the experiment are not suitable for comparison with baselines.

Author response:
Thanks for raising the concern. 

*Our algorithm is not sensitive to the number of local epochs, but benefits from it.* Considering that the communication cost is generally the bottleneck of federated learning, many works seek to reduce the number of rounds by benefiting from the increasing of local epochs [1,2,3]. Figure 2(c) presents that our algorithm has this advantage. To reach the same accuracy, our algorithm requires fewer communication rounds with larger local epochs. As a comparison, FedAvg does not necessarily achieve improvements as the number of local epochs increases. Besides, as shown in Figure 2(c) and (d), our algorithm always outperforms FedAvg over various local epochs. In fact, many existing works also adopt the 20 epochs as the basic configuration [4,5]. 

To demonstrate the effectiveness of our proposed method over a broader range of hyper-parameters. We conduct more experiments by using a different learning rate and smaller local epochs. The results are shown in the following three tables. 


Table 1. The learning rate is 0.1, weight decay is 1e-3, local epoch is 5, and the optimizer is SGD without momentum.

| Method| Accuracy|
|--|--|
|FedAvg|48.20% (±0.42%)|
|FedProx|54.46% (±1.01%)|
|FedExP|57.00% (±1.02%)|
|FedSAM|57.94% (±0.36%)|
|FedDF|58.83% (±0.81%)|
|FedGEN|55.29% (±1.37%)|
|FedMA| 49.45% (±0.61%)|
|GAMF| 50.87% (±0.27%)|
|FedLAW|51.78% (±0.97%)|
|FedCDA|**65.50%** (±0.25%)|


Table 2. The learning rate is 0.1, weight decay is 1e-3, local epoch is 10, and the optimizer is SGD without momentum.

| Method | Accuracy |
| :--: | :--: |
| FedAvg | 46.92% (+- 0.72%) |
| FedCDA| **66.25%** (+- 0.25%) |

Table 3. The learning rate is 0.1, weight decay is 1e-3, local epoch is 20, and SGD without momentum. 

|Method|Accuracy|
| :--: | :--: |
|FedAvg|45.01% (+- 0.81%) |
|FedCDA|**66.62%** (+- 0.25%) |

We can see that our method still achieves great improvement under hyperparameters of the learning rate being 0.1 and the number of local epochs being 5, 10, and 20. In fact, a larger local learning rate helps the convergence of local training, where this configuration is more suitable for our method. 

We are very pleased that you have tested our code of FedAvg. **We sincerely hope that you can further directly test our proposed algorithm under various configurations.** We believe that the results you obtain are more convincing than our claim. 

[1] Yan Sun, et al. Fedspeed: Larger local interval, less communication round, and higher generalization accuracy. ICLR 2023

[2] Sebastian U. Stich: Local SGD Converges Fast and Communicates Little. ICLR 2019

[3] 	Konstantin Mishchenko, et al. ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally! ICML 2022

[4] Xin-Chun Li, et al. Federated Learning with Position-Aware Neurons. CVPR 2022

[5] Hongyi Wang, et al. Federated Learning with Matched Averaging. ICLR 2020

Reviewer comment (Experiments):
The learning rate was set to 1e-3, which helps the local model to converge quickly with a large local epoch. However, it is unclear whether such a small learning rate and large local epoch are suitable for other baselines.

Author response:
Thanks for raising the concern. 

*Our algorithm is not sensitive to the number of local epochs, but benefits from it.* Considering that the communication cost is generally the bottleneck of federated learning, many works seek to reduce the number of rounds by benefiting from the increasing of local epochs [1,2,3]. Figure 2(c) presents that our algorithm has this advantage. To reach the same accuracy, our algorithm requires fewer communication rounds with larger local epochs. As a comparison, FedAvg does not necessarily achieve improvements as the number of local epochs increases. Besides, as shown in Figure 2(c) and (d), our algorithm always outperforms FedAvg over various local epochs. In fact, many existing works also adopt the 20 epochs as the basic configuration [4,5]. 

To demonstrate the effectiveness of our proposed method over a broader range of hyper-parameters. We conduct more experiments by using a different learning rate and smaller local epochs. The results are shown in the following three tables. 


Table 1. The learning rate is 0.1, weight decay is 1e-3, local epoch is 5, and the optimizer is SGD without momentum.

| Method| Accuracy|
|--|--|
|FedAvg|48.20% (±0.42%)|
|FedProx|54.46% (±1.01%)|
|FedExP|57.00% (±1.02%)|
|FedSAM|57.94% (±0.36%)|
|FedDF|58.83% (±0.81%)|
|FedGEN|55.29% (±1.37%)|
|FedMA| 49.45% (±0.61%)|
|GAMF| 50.87% (±0.27%)|
|FedLAW|51.78% (±0.97%)|
|FedCDA|**65.50%** (±0.25%)|


Table 2. The learning rate is 0.1, weight decay is 1e-3, local epoch is 10, and the optimizer is SGD without momentum.

| Method | Accuracy |
| :--: | :--: |
| FedAvg | 46.92% (+- 0.72%) |
| FedCDA| **66.25%** (+- 0.25%) |

Table 3. The learning rate is 0.1, weight decay is 1e-3, local epoch is 20, and SGD without momentum. 

|Method|Accuracy|
| :--: | :--: |
|FedAvg|45.01% (+- 0.81%) |
|FedCDA|**66.62%** (+- 0.25%) |

We can see that our method still achieves great improvement under hyperparameters of the learning rate being 0.1 and the number of local epochs being 5, 10, and 20. In fact, a larger local learning rate helps the convergence of local training, where this configuration is more suitable for our method. 

We are very pleased that you have tested our code of FedAvg. **We sincerely hope that you can further directly test our proposed algorithm under various configurations.** We believe that the results you obtain are more convincing than our claim. 

[1] Yan Sun, et al. Fedspeed: Larger local interval, less communication round, and higher generalization accuracy. ICLR 2023

[2] Sebastian U. Stich: Local SGD Converges Fast and Communicates Little. ICLR 2019

[3] 	Konstantin Mishchenko, et al. ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally! ICML 2022

[4] Xin-Chun Li, et al. Federated Learning with Position-Aware Neurons. CVPR 2022

[5] Hongyi Wang, et al. Federated Learning with Matched Averaging. ICLR 2020

Reviewer comment (Experiments):
Based on the code included in the supplementary material, the reviewer reproduced the FedAvg experiment of CIFAR100 - Dirichlet (0.5) in Table 1. Using the same hyperparameters presented in the paper, FedAvg showed an accuracy of about 38%, which is almost similar to 38.86% recorded in Table 1. However, when the learning rate was 0.1, weight decay was 1e-3, local epoch was 5, and the same experiment was performed with SGD without momentum, the accuracy was about 49%, which is almost similar to the proposed method (49.31%). This does not mean that the proposed method shows similar performance to FedAvg, but the reviewer thinks that the hyperparameters should have been chosen more carefully. The reviewer also requests, if possible, that the experiment be run again with the hyperparameters mentioned above.

Author response:
Thanks for raising the concern. 

*Our algorithm is not sensitive to the number of local epochs, but benefits from it.* Considering that the communication cost is generally the bottleneck of federated learning, many works seek to reduce the number of rounds by benefiting from the increasing of local epochs [1,2,3]. Figure 2(c) presents that our algorithm has this advantage. To reach the same accuracy, our algorithm requires fewer communication rounds with larger local epochs. As a comparison, FedAvg does not necessarily achieve improvements as the number of local epochs increases. Besides, as shown in Figure 2(c) and (d), our algorithm always outperforms FedAvg over various local epochs. In fact, many existing works also adopt the 20 epochs as the basic configuration [4,5]. 

To demonstrate the effectiveness of our proposed method over a broader range of hyper-parameters. We conduct more experiments by using a different learning rate and smaller local epochs. The results are shown in the following three tables. 


Table 1. The learning rate is 0.1, weight decay is 1e-3, local epoch is 5, and the optimizer is SGD without momentum.

| Method| Accuracy|
|--|--|
|FedAvg|48.20% (±0.42%)|
|FedProx|54.46% (±1.01%)|
|FedExP|57.00% (±1.02%)|
|FedSAM|57.94% (±0.36%)|
|FedDF|58.83% (±0.81%)|
|FedGEN|55.29% (±1.37%)|
|FedMA| 49.45% (±0.61%)|
|GAMF| 50.87% (±0.27%)|
|FedLAW|51.78% (±0.97%)|
|FedCDA|**65.50%** (±0.25%)|


Table 2. The learning rate is 0.1, weight decay is 1e-3, local epoch is 10, and the optimizer is SGD without momentum.

| Method | Accuracy |
| :--: | :--: |
| FedAvg | 46.92% (+- 0.72%) |
| FedCDA| **66.25%** (+- 0.25%) |

Table 3. The learning rate is 0.1, weight decay is 1e-3, local epoch is 20, and SGD without momentum. 

|Method|Accuracy|
| :--: | :--: |
|FedAvg|45.01% (+- 0.81%) |
|FedCDA|**66.62%** (+- 0.25%) |

We can see that our method still achieves great improvement under hyperparameters of the learning rate being 0.1 and the number of local epochs being 5, 10, and 20. In fact, a larger local learning rate helps the convergence of local training, where this configuration is more suitable for our method. 

We are very pleased that you have tested our code of FedAvg. **We sincerely hope that you can further directly test our proposed algorithm under various configurations.** We believe that the results you obtain are more convincing than our claim. 

[1] Yan Sun, et al. Fedspeed: Larger local interval, less communication round, and higher generalization accuracy. ICLR 2023

[2] Sebastian U. Stich: Local SGD Converges Fast and Communicates Little. ICLR 2019

[3] 	Konstantin Mishchenko, et al. ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally! ICML 2022

[4] Xin-Chun Li, et al. Federated Learning with Position-Aware Neurons. CVPR 2022

[5] Hongyi Wang, et al. Federated Learning with Matched Averaging. ICLR 2020

Reviewer comment (Writing):
The overall writing quality of the paper is not good. There are many incorrect parts in the notation related to formulas and figures in the paper, and many sentences require revision.

Author response:
Thanks for pointing out the limitations of our presentation. We have revised the presentation throughout the paper. For example, we refine the motivation example of Figure 1. We are glad that you would like to provide some suggestions for further improvement.

Reviewer comment (Theory):
First of all, the motivation about Figure 1 is difficult to understand, where the authors attempt to illustrate the usefulness of cross-round local models. But according to the reviewer's understanding, the model aggregation strategy of Fig. 1(b) will cause the global model to overfit to the local node, resulting in extremely unstable global convergence. Further clarification is needed on why it is better to update model via Fig. 1(b).

Author response:
Thanks for your comment that indicates that our illustration may cause misunderstandings. We have refined Figure 1 and its corresponding description, which can be found in the revised manuscript. 

In the cross-round method of our example, the global model $w_{t+1}$ is the aggregation of the $t$-th round local model $w_{t}^1$ of client 1 and the $t+1$-th round local model $w_{t+1}^2$ of client 2. It does not overfit to any specific client. In the single-round method, the global model $w_{t+1}$ is the aggregation of the local model $w_{t+1}^1$ of client 1 and $w_{t+1}^2$ of the client 2 in the same $t+1$-th round. 

$w_{t+1}$ obtained through cross-round aggregation is close to the local optimum 1 of client 1 and the local optimum 2 of client 2, while the single-round aggregation is distant from any local optima. Therefore, $w_{t+1}$ obtained through cross-round aggregation can fit the data of all clients.

Reviewer comment (Theory):
Why the optimization objective function presented in Eq. (2) is derived based on the effectiveness of cross-round local models remains unclear. The connection between the optimization objective and the effectiveness of cross-round local models requires further elucidation.

Author response:
Thank you for pointing out the limited explanation in this part.  We have made revisions to present the derivation process of Eq. (2) and connections to cross-round aggregation. It is shown in the blue texts of Section 4 of our revised manuscript.

Eq. (2) (corresponding to Eq. (3) in the revised manuscript) is derived by separately applying the L-smooth assumption over the local objective function $F_n(w^n)$ of each client n. Then, after a series transformation, the objective becomes Eq. (5) in the revised manuscript. As explained by the blue text, Eq. (5) is to essentially select local models that exhibit minimal divergence across multiple rounds.

Reviewer comment (Reproducibility):
Another issue lies in the optimization objective presented in Eq. (7) on the server side, where model derivatives appear to be computed for inactivate nodes. Question then arises as to whether these non-active models are stored on the server side. In federated scenarios characterized by a large number of nodes and typically large models, this could lead to substantial storage overhead.

Author response:
Thank you for raising this concern. The server needs to store K local models of all clients. Although our algorithm requires more storage of the server than FedAvg, the cost is totally manageable in practical settings:

*First*, the number $K$ of cached models is small. Our evaluation shows that merely setting $K=3$ can achieve significant improvement. 

*Second*, the models of non-active clients can be stored in the disk. Disk is cheap and can satisfy the storage requirement of models designed for clients. Taking the commonly used ResNet-18 (44MB) and MobileNet-V3-Large (21MB) in edge devices as examples, a disk of $1h$ TB can approximately store $20,000$ local models and $40,000$ local models, respectively. Considering that a mobile phone can have 1TB storage in the current era, we believe that this storage cost is well within the budget of an aggregation node which is usually provided by a cloud.

In fact, our algorithm can be applied to most FL scenarios that existing methods considered:

*Cloud-edge scenario*. Cloud serves as the aggregation node and edge devices serves as clients. Generally, constrained resources of the edge devices are the bottleneck of this scenario while the capability of the cloud is strong. Our algorithm requires no extra resources of edge devices as compared to basic FedAvg and thus can be applied to this scenario. 

*Cross-silo scenario*. Cloud serves as both the aggregation server and clients. The participated clients are usually institutes of which its number is usually small. Hence, the memory cost of our algorithm is acceptable in such a scenario.

Reviewer comment (Evaluation):
The proposed method incurs a substantial storage cost, significantly constraining the practicality of the algorithm.

Author response:
Thank you for raising this concern. The server needs to store K local models of all clients. Although our algorithm requires more storage of the server than FedAvg, the cost is totally manageable in practical settings:

*First*, the number $K$ of cached models is small. Our evaluation shows that merely setting $K=3$ can achieve significant improvement. 

*Second*, the models of non-active clients can be stored in the disk. Disk is cheap and can satisfy the storage requirement of models designed for clients. Taking the commonly used ResNet-18 (44MB) and MobileNet-V3-Large (21MB) in edge devices as examples, a disk of $1h$ TB can approximately store $20,000$ local models and $40,000$ local models, respectively. Considering that a mobile phone can have 1TB storage in the current era, we believe that this storage cost is well within the budget of an aggregation node which is usually provided by a cloud.

In fact, our algorithm can be applied to most FL scenarios that existing methods considered:

*Cloud-edge scenario*. Cloud serves as the aggregation node and edge devices serves as clients. Generally, constrained resources of the edge devices are the bottleneck of this scenario while the capability of the cloud is strong. Our algorithm requires no extra resources of edge devices as compared to basic FedAvg and thus can be applied to this scenario. 

*Cross-silo scenario*. Cloud serves as both the aggregation server and clients. The participated clients are usually institutes of which its number is usually small. Hence, the memory cost of our algorithm is acceptable in such a scenario.

Reviewer comment (Experiments):
In FedCDA, each local client needs to store a set of recent K local models, which requires a much higher demand of memory space compared to other FL methods.

Author response:
Thank you for your careful reviewing. These K local models are stored in the server instead of the client. Our algorithm does not require that the client has a high storage capability. Although our algorithm consumes more memory of the server than FedAvg, the cost is totally manageable in practical setting:

*First*, the number $K$ of cached models is small. Our evaluation shows that merely setting $K=3$ can achieve significant improvement. 

*Second*, the models of not-sampled clients can be stored in the disk. Disk is cheap and can satisfy the storage requirement of models designed for clients. For example, a disk of $1$ TB can approximately store $20,000$ ResNet-18 models which are generally used in edge devices and costs $44$ MB. Given that a mobile-phone is even equipped with $1$ TB storage, we believe that the cost can be ignored by the aggregation node which is usually served as by a cloud.

In fact, our algorithm can also be applied to most FL scenarios that existing methods considered:

*Cloud-edge scenario*. Cloud serves as the aggregation server and edge devices serves as clients. Generally, constrained resources of the edge devices are the bottleneck of this scenario while the capability of the cloud is strong. Our algorithm requires no extra resources of edge devices as compared to basic FedAvg and thus can be applied to this scenario. 

*Cross-silo scenario*. Cloud serves as both the aggregation server and clients. The participated clients are usually institutes of which its number is usually small. Hence, the memory cost of our algorithm is acceptable in such a scenario.

Reviewer comment (Evaluation):
Although FedCDA employed the approximation strategy, communicating the model set is still too heavy and the considerable extra communication may be a severe problem when there exists a large number of engaging clients.

Author response:
Thanks for raising the concern. Our algorithm has the same communication cost as the basic FedAvg, i.e., broadcasting a global model to all sampled clients and receiving a single local model from each sampled client. These past K local models are stored in the server.

Reviewer comment (Evaluation):
FedCDA maintains a consistent global model that will not interfere with those outlier local models as the authors selected the closer local model to do aggregation. The potential problem is that the aggregated model may not adapt to those outliers and be quite sub-optimal.

Author response:
Thank you for raising such an insightful comment. We agree that our algorithm cannot exclude all outliers. But, as compared to existing aggregation methods, our algorithm can greatly mitigate this issue via an aggregation manner of minimizing model divergence. Besides, our objective is to train a global model which has high accuracy over the global dataset. Although the global model cannot well adapt to these outliers, it still can have high global accuracy.

Reviewer comment (Theory):
The variables in the formula are very complex, and a lot of space was spent on deriving the objective function. At the same time, it is still difficult to understand why selectively aggregating them can make the global model approach local models more closely and affect the targeting of local data knowledge?

Author response:
Thanks for your comment that indicates that the presentation of our motivation is not clear enough. We have refined Figure 1 and its corresponding description, which can be found in the revised manuscript. 

As shown in the Figure 1.(b), in the cross-round method, the global model $w_{t+1}$ is the aggregation of the $t$-th round local model $w_{t}^1$ of client 1 and the $t+1$-th round local model $w_{t+1}^2$ of client 2. In the single-round method, the global model $w_{t+1}$ is the aggregation of the local model $w_{t+1}^1$ of client 1 and $w_{t+1}^2$ of the client 2 in the same $t+1$-th round. 

$w_{t+1}$ obtained through cross-round aggregation is close to the local optimum 1 of client 1 and the local optimum 2 of client 2, while the single-round aggregation is distant from any local optima. Therefore, by referring to the concepts of the field of continual learning where the knowledge of old model will be maintained when the new model is close to the old model, we claim that the global model $w_{t+1}$ obtained through cross-round aggregation can maintain the local data knowledge of local models.

Reviewer comment (Presentation):
It seem that the caption of Fig(2) is wrong.

Author response:
Thank you very much for the careful reviewing. We have reordered the captions to match their corresponding sub-figures.

Reviewer comment (Evaluation):
There seems to be a lack of discussion on memory and communication costs.

Author response:
Thanks for raising the concern. The memory costs have been discussed by the Proposition 1, 2, 3 and the last-second paragraph before section 4.1 of our original paper. As compared to FedAvg which only caches $1$ model of each client sampled in current round, our algorithm requires the server to cache $K$ models for all clients respectively. Although the memory cost is several times than FedAvg, our algorithm is also totally manageable in a practical setting. 

*First*, the number of cached models $K$ is small. Our evaluation shows that merely setting $K=3$ can achieve significant improvement. 

*Second*, the models of not-sampled clients can be stored in the disk. Disk is very cheap now and can satisfy the storage requirement of models designed for clients. For example, a disk of $1$ TB can approximately store $20,000$ ResNet-18 models which are generally used in edge devices and costs $44$ MB. The same disk can store more models when the model is MobileNet designed specifically for devices. Given that a mobile phone is even equipped with $1$ TB storage, we believe that the cost is within the budget of the aggregation node which is usually served by a cloud.

It is worthwhile to note that our algorithm has the same communication cost as the basic FedAvg, i.e., broadcasting a global model to all sampled clients and receiving a single local model from each sampled client. Our algorithm does not modify the sampling strategy and the interaction strategy of FedAvg.

Reviewer comment (Writing):
How to understand that there are varied received global models and they result in local models? (Quote from Introduction: "each client often converges to different models due to varied received global models and multiple local optima of deep neural networks”?)

Author response:
A1: We view the local training stage of each round in a client as a complete training process corresponding to the centralized setting. The initialized model of each local training stage is the global model. Since the parameters of the global model are different across rounds, the initialized models are different in different local training stages. Considering the model may converge to local optima close to its initialized value when there are multiple local optima of the DNN, the converged local model naturally varies across rounds.

Reviewer comment (Experiments):
Do the local models achieve different local minimal only due to their own private local data?

Author response:
A2: We consider that there are three main factors that affect the convergence of local models of different clients.

First, heterogeneous data. Dataset formalizes the loss function. It is necessary to train different parameters to fit different data distributions.

Second, different parameters of the global model across rounds. Different clients are sampled in different rounds and thus they perform the local training process with different global models.

Third, stochastic training process. The local training process usually adopts SGD or its variants, which also affects the convergence of local minima.

Reviewer comment (Evaluation):
Why are there varied received global models?

Author response:
A3: The “varied global models” means that the parameters of the global model of different rounds are different. We have revised the corresponding texts in the abstract and introduction in the manuscript, which are highlighted by the blue color. 

For example, to avoid ambiguity, the corresponding text in the abstract is revised to be “The principle behind FedCDA is that due to the different global model parameters received in different rounds and the non-convexity of deep neural networks, the local models from each client may converge to different local optima across rounds”.

Reviewer comment (Theory):
Does Equation (2) have used the approximation (∇wn Fn(wn))’ ≈ 0 ?

Author response:
A5: No.  Equation (2) is derived based on the L-smooth assumption. The detailed derivation process can be found in the revised manuscript, which is denoted by the blue color.

Reviewer comment (Theory):
Why selectively aggregating them can make the global model approach local models more closely and affect the targeting of local data knowledge?

Author response:
A7: Considering the aggregation of the two-client FL, given the local model $w^1_t$ of the client 1 and two local models ${w^2_{t-1}, w^2_{t}}$ of the client 2, there are two aggregation manners, i.e., $w_t = 1/2(w^1_t+ w^2_{t-1})$ and $w_t = 1/2(w^1_t+ w^2_{t})$. Choosing which combination manner relies on their distance. Specifically, if $w^1_t$ is closer to $w^2_{t-1}$ than $w^2_{t}$, then the first aggregation manner is adopted. Otherwise, the second manner is adopted. In this way, the obtained global model $w_t$ is close to at least one local model of each client. When merely adopting existing aggregation methods, i.e., the second aggregation manner, the obtained global model $w_t$ may be far from any local models of all clients. More details can also be found in the response R1 for the weakness W1.

Reviewer comment (Evaluation):
The authors aim to make the global model approach any of the different local models in multiple rounds, how to define the “approach”, and how approach between the multi-local epoch global model and last-epoch local models?

Author response:
A9: The “approach” is measured as the variance among local models, i.e., the average of L2-norm distance of the global model to all local models. This description can be found in the eq.(5) and (6) of the revised manuscript. 

In the question of “how approach between the multi-local epoch global model and last-epoch local models”, we guess you may be asking about “how approach between the global model obtained by cross-rounds aggregation and the local models of the latest round”. Unfortunately, the model distance can hardly be analyzed due to the unknown distribution of local optima. Yet, our paper has shown that the difference between their losses is bounded, as shown in Lemma 1 of Appendix B.4.

Reviewer comment (Novelty):
What differences between this work “Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initialization. Dacheng Tao et.al .Nips2023.”

Author response:
A10: FL consists of two main steps: step 1 of local update and step 2 of global aggregation. This related work focuses on step 1 by applying an offset to the local model, while our work focuses on step 2 by applying a cross-round aggregation of all local models. Our method is orthogonal to the related work and can be jointly applied to FL together with this work. We have discussed the difference in the first paragraph of the related work section in our revised manuscript.