Paper: Fake It Till Make It: Federated Learning with Consensus-Oriented Generation (ICLR-2024)

Reviewer comment (Experiments):
In the experiment part, what are the sample numbers of the synthetic data in your setup?

Author response:
**W2:** What are the sample numbers of the synthetic data in your setup?

**Response:** The sample number of the synthetic data in our setup is 256. Please also note that this number is significantly smaller than the sample number of real data. For example, when the client number is 10 for CIFAR-10, each client has 5,000 samples on average.

Reviewer comment (Experiments):
I am concerned that none of the selected baselines is a synthetic data-based method. I see the paper cites FedGen in the related work section. Why does the author not compare with the recent synthetic data-based methods such as FedGen[1] and DynaFed[2]?

Author response:
**W4:** None of the selected baselines is a synthetic data-based method? Why does the author not compare with methods such as FedGen?

**Response:** Thanks for the advice. Actually, **FedReg (ICLR 2022) [3] in Table 1 is a synthetic data-based method**, where we can see that our method achieves significantly better performance.

Following your advice, we now include comparison between with FedGen (ICML 2021) in Table 1 in the revision. Here, we show the comparison between FedCOG and FedGen below for convenience.

From the table, we see that **FedCOG outperforms FedGen** across different datasets and heterogeneity types. This could result from the fact that FedCOG modifies the dataset itself to adjust the training of the whole model, which more fundamentally addresses the issue of data heterogeneity, while FedGen generates features to adjust the last layer of model only. Beside performance improvement, another key advantage of FedCOG is that FedCOG is compatible with Secure Aggregation while FedGen is not, because in FedGen the server needs to access each individual local model.

[**Table R1.** Comparison between FedCOG and FedGen.]
|   Method   | FMNIST-1  | FMNIST-2  | CIFAR10-1 | CIFAR10-2 | CIFAR100-1 | CIFAR100-2 |    Avg    |
|:--:|:--:|:--:|:--:|:---:|:-:|:--:|
|   FedGen   |   72.64   |   61.34   |   62.45   |   49.12   |   37.99    |   26.93    |   51.75   |
| **FedCOG** | **77.34** | **73.68** | **64.83** | **54.00** | **42.88**  | **34.80**  | **57.92** |

[3] Xu et al., Acceleration of Federated Learning with Alleviated Forgetting in Local Training, ICLR 2022.

Reviewer comment (Reproducibility):
How would the FedCOG be harmonized with FedProx? I am curious about how the FedCOG does the local proximal term in the KD-based model training?

Author response:
**Q1:** 1. How would the FedCOG be harmonized with FedProx? Curious about how FedCOG does the local proximal term in the KD-based training?

**Response:** FedCOG can be seamlessly harmonized with FedProx. Take a conventional classification task as an example. 

FedCOG has two loss terms, which are both applied at the logit level (one for cross-entropy loss and one for KD-based loss), while FedProx introduces a model-level loss, which calculates element-wise l2 distance between local and global models. Thus, these **three loss terms** can be applied together to compute the loss and backward to update local model.

Reviewer comment (Evaluation):
In Table 4, why FedProx took longer local training time compared to the FedCOG?

Author response:
**Q2:** In Table 4, why FedProx took longer local time compared to FedCOG?

**Response:** 
The reason is that during local training, FedProx needs to launch two models, while FedCOG only needs to launch one model.

In FedCOG, the soft labels of generated data are obtained by a one-time inference of the global model, which is fast. After this, during training, **only one model** (the local model) needs to be launched.

In FedProx, **one local model and one global model** need to be launched during local training because proximal term in FedProx is computed by computing element-wise distance between local and global model. This cost will be larger, especially when the model size is larger.

Please note that for FedCOG, the generation time, time for inferring soft labels and training time are all included in Table 4, making it a fair comparison.

---

**Q2:** I have a question regarding the "the soft labels of generated data are obtained by a one-time inference of the global model". Do you mean the data generation in FedCOG is faster than the proximal term in FedProx?

**Response:** Sorry for the confusion. We mean at each round, the soft labels of generated data are obtained by a one-time inference of the global model, indicating that the global model is no longer needed during local model training. Thus, FedCOG only needs to launch one local model during training while FedProx needs to launch two models during training. 

Regarding the second question, the answer is yes. In our experiments, the generation process is fast (0.81 seconds), which makes our method faster than FedProx in a fair comparison.

Reviewer comment (Experiments):
C2 It's possible the network is not converged. To eliminate such a possibility, please provide a graph depicting the trend of convergence with the number of rounds on the server side.

Author response:
Thanks for your advice, we include the convergence figures in Figure 6 in the revision. We show that the network has converged, and our method achieves the best performance.

Reviewer comment (Evaluation):
C3 What's more, the proposed method only achieved a little improvement in accuracy, it's not sure whether it's caused by insufficient experiments, please repeat and provide mean and standard error for all results.

Author response:
Thanks for the advice. Please note that:

(1) In Table 1, we have reported mean and standard error, which shows that our method can significantly and consistently outperform other methods.

(2) Sorry for the confusion. For other results, the reported values are mean values of 3-5 trials, and we did not include standard error due to space limits, such that we could show more comparisons.

Here we list several examples.

[**Table R2.** Mean and std.]
| Method |     FMNIST-1     |     FMNIST-2     |    CIFAR10-1     |    CIFAR10-2     |
|:------:|:----------------:|:----------------:|:----------------:|:----------------:|
| FedAvg | 73.07 $\pm$ 0.08 | 64.11 $\pm$ 0.78 | 64.36 $\pm$ 0.11 | 50.55 $\pm$ 0.45 |
| FedCOG | **77.34** $\pm$ 0.07 | **73.68** $\pm$ 0.38 | **64.83** $\pm$ 0.12 | **54.00** $\pm$ 0.84 |

Reviewer comment (Experiments):
C4 We kindly request further experimentations involving the generation of datasets of varying sizes, with corresponding meticulous documentation of the associated overhead. Furthermore, if feasible, we encourage experimentation on datasets comprising high-resolution images uniformly, to enhance the comprehensiveness of the analysis.

Author response:
Thanks for the helpful advice! We now include a table to demonstrate the relationship between generation time and number of generated samples in Table 11 in the revision. For convenience, we also put the table here. We can see that the generation time is little (we generate 256 samples throughout the paper), indicating the efficiency of our method. As a reference, the conventional local training takes 14.81 seconds.

[**Table R3.** Generation time (seconds) v.s. number of generated samples.]
| Number of generated samples  | 16   | 32   | 64   | 128  | 256  | 512  | 1024 |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| Generation time (s) | 0.72 | 0.78 | 0.79 | 0.80 | 0.81 | 0.83 | 1.27 | 

Please note the images in the FLAIR dataset (Table 2) have a high resolution of 256\*256.

Reviewer comment (Theory):
C5 Please add proofs for the convergence analysis. If possible, please add a formal security analysis to your method.

Author response:
Here, we provide convergence analysis and some interpretations. 

**1. Convergence:**

Please refer to Section A.6 in the Appendix in the revision. We show that our method has a **theoretical convergence guarantee besides achieving pleasant performance**.

**2. Interpretations:**

From standard FL theoretical literature, it is well-known that lower dissimilarity contributes to faster convergence both empirically and theoretically [2,3]. At the same time, our method contributes to **lower model difference**, which corresponds to lower dissimilarity among clients in the standard FL theoretical literature (e.g., bounded dissimilarity in [1,2]); please refer to evidence in Figure 2 (a). 

These two key observations can provide implications and evidence for the convergence of our method.

Please also note that the focus of our paper is to improve the performance of federated learning from a practical view and our contribution lies on proposing a new attempt on tackling data heterogeneity but not theory. Thus, we did not include a convergence analysis previously. 

[1] Karimireddy, Sai Praneeth, et al. "Scaffold: Stochastic controlled averaging for federated learning." International conference on machine learning. PMLR, 2020.

[2] Wang, Jianyu, et al. "Tackling the objective inconsistency problem in heterogeneous federated optimization." Advances in neural information processing systems 33 (2020): 7611-7623.

[3] Li, Tian, et al. "Federated optimization in heterogeneous networks." Proceedings of Machine learning and systems 2 (2020): 429-450.

Reviewer comment (Evaluation):
This paper introduced data distribution from other clients, will this cause privacy leakage, making it easier for the attacker to learn data information from the clients?

Author response:
Sorry for the potential confusion. We would like to emphasize that **our method does not introduce data distribution from other clients, which does not cause privacy leakage**. The shared information is exactly the same as FedAvg: the local model. No information about distribution is required for sharing.

Reviewer comment (Theory):
Will generating new data for each client be identical to amplifying the global weight update direction collected in the last epoch?

Author response:
We believe that generating new data for each client is not equivalent to amplifying the global weight update direction. The reasons are twofold. On one hand, this can be supported by the results of FedAvgM, which introduces momentum-based global update for each round. On the other hand, we keep the number of local model updates the same for all methods, making sure that all methods are with the same update scale.

Reviewer comment (Theory):
This paper would further benefit from theoretical derivations to interpret why generated data on the client side helps in improving global model performance.

Author response:
Here, we provide some interpretations and a theoretical convergence analysis of our method.

**1. Interpretations:**

From standard FL theoretical literature, it is well-known that lower dissimilarity (between clients) contributes to faster convergence both empirically and theoretically [2,3]. At the same time, our method contributes to lower model difference, which corresponds to lower dissimilarity among clients in the standard FL theoretical literature (e.g., bounded dissimilarity in [1,2]); please refer to evidence in Figure 2 (a). These two key observations can provide implications and evidence for the convergence of our method.

**2. Convergence:**

Please refer to Section A.6 in the Appendix in the revision. We show that our method has a **theoretical convergence guarantee** besides achieving pleasant performance.

Please also note that the focus of our paper is to improve the performance of federated learning from a practical view and our contribution lies on proposing a new attempt on tackling data heterogeneity but not theory. Thus, we did not include a convergence analysis previously. 

[1] Karimireddy, Sai Praneeth, et al. "Scaffold: Stochastic controlled averaging for federated learning." International conference on machine learning. PMLR, 2020.

[2] Wang, Jianyu, et al. "Tackling the objective inconsistency problem in heterogeneous federated optimization." Advances in neural information processing systems 33 (2020): 7611-7623.

[3] Li, Tian, et al. "Federated optimization in heterogeneous networks." Proceedings of Machine learning and systems 2 (2020): 429-450.

Reviewer comment (Experiments):
Concerns on Experiments: All methods in Table 1 achieve notably lower accuracies than SOTA FL methods. I am concerned that the model arch, communication round, or optimizer setting is not well set up for appropriate comparison.

Author response:
We would like to emphasize that we follow exactly the same setups as [1,2] including model architecture (CNN), optimizer (SGD), and learning rate (0.01). We set the number of rounds as 70 because we find that after 70 rounds the performance remains nearly the same as the 70th round's.

The reason why the performance seems low on average is that many methods are not robust towards diverse settings (e.g., datasets and heterogeneity types). For example, SCAFFOLD achieves 4% higher than FedAvg on NIID-1 of Fashion-MNIST but 6% lower than FedAvg on NIID-2 of Fashion-MNIST. 

[1] Li, Qinbin, Bingsheng He, and Dawn Song. "Model-contrastive federated learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[2] Luo, Mi, et al. "No fear of heterogeneity: Classifier calibration for federated learning with non-iid data." Advances in Neural Information Processing Systems 34 (2021): 5972-5984.

Reviewer comment (Experiments):
Since tackling data heterogeneity is the key of this paper, I suggest authors conduct more experiments on data with Dirichlet distribution by varying the hyper-parameter $\beta$. More results with changing heterogeneities would further validate the effects of the proposed methods.

Author response:
Thanks for the advice. Actually, we have included such experiments in Figure 3 (a). Here, we show the results again for convenience. We see that our proposed FedCOG consistently performs the best at different beta.

[**Table R1.** Effects of hyper-parameter beta.]
| Beta       | 0.1       | 0.5       | 1.0       | 5.0       |
| ---------- | --------- | --------- | --------- | --------- |
| FedAvg     | 63.50     | 64.51     | 66.55     | 67.70     |
| FedProx    | 63.00     | 63.94     | 67.29     | 67.01     |
| MOON       | 63.28     | 64.09     | 64.07     | 67.78     |
| **FedCOG** | **64.46** | **65.87** | **67.99** | **69.40** |

Reviewer comment (Novelty):
The motivation for why we need to use data generation, instead of recent popular methods based on model correction, is somewhat not clear. When the training dataset is very large, the proposed method therefore needs to generate a large amount of data in order to achieve alignment, which is costly, then it seems like model correction is a better choice in such a scenario.

Author response:
**W1-1:** The motivation for why we need to use data generation, instead of recent popular methods based on model correction, is somewhat not clear.

**Response:** Sorry for the confusion. Our motivations are twofold. 

1. **Heterogeneous data is the fundamental cause of performance degradation while model-correction-based methods do not directly operate on data.** Previous model correction methods focus on remedying the negative effects of data heterogeneity by model correction techniques while leaving the dataset as heterogeneous as before. While data is the root of performance degradation, this motivates us to deal with the heterogenous datasets themselves to more fundamentally address the issue of data heterogeneity. We believe that this will be an interesting future direction, especially because there are more and more advanced data generation techniques.
2. **Merely relying on correction-based methods cannot fully address data heterogeneity,** which motivates us to explore a new direction that addresses this from an orthoganal perspective. We expect to see that in the future the issue of data heterogeneity can potentially be organically addressed by integration of techniques from diverse perspectives.

**W1-2:** The proposed method needs to generate more data when training dataset is large?

**Response:** Sorry for the potential misunderstanding we have caused. Actually, the generated number does not scale up with the number of training samples. Throughout the paper, **the generated number keeps the same (256)** no matter how many samples each client has (e.g., 10k, 5k). Note that the required generation time is only **0.81 seconds** (on CIFAR-10) since the only learnable parameters are the inputs while the models are fixed, and **only 100 steps** of optimization is sufficient.

Here is the rationale. In FL, the number of local updates is strongly related to convergence, thus it is set to a fixed moderate number no matter how many data samples the client has. This means that for each round, the number of samples used for updating the model is the same no matter how many data samples the client has (number of local updates multiplied by batch size). Thus, the generated number for each round can also be set as the same number no matter how many data samples the client has.

Besides, our method is **actually efficient** because our method only needs to launch **one model** during local model training (the soft labels from global model is obtained for only **one-time inference** and only the local model needs to be launched). On the contrary, for example, during local model training, FedProx needs to launch **two models** (one local model and one global model); MOON needs to launch **three models** (two local models and one global model); SCAFFOLD needs to launch **three models** (one local model, one local control variate and one global control variate).

Reviewer comment (Novelty):
As the paper mentioned, the proposed method FedCOG has two advantages, i.e., plug-and-play and compatibility with standard FL protocols, however, the unique advantages of this data generation method, compared to previous model correction methods, are still ambiguous. Could you please elaborate more regarding them?

Author response:
**W2:** The unique advantages of this data generation method, compared to previous model correction methods.

**Response:** The unique advantage of our method is that our method not only more fundamentally alleviates the heterogeneity level of clients' data, but also enhance alignment among local models at the same time. 

FedCOG consists of two key parts: data generation and knowledge distillation, which organically alleviate the issue of data heterogeneity. (1) First, the generated consensus data can effectively decrease the difference among clients' datasets, therefore reducing the data heterogeneity level to more fundamentally alleviate this issue. (2) Second, based on the consensus data, we force each local model's outputs to align with global model's outputs, which functions to implicitly align local models like correction-based methods. Besides, unlike correction-based methods that need to launch several models, our method only needs to launch one local model during training as the soft labels of consensus data can be obtained through one-time inference, which is efficient.

Please note that besides the technical contribution, our contribution also lies on exploring a **novel perspective of dealing with data heterogeneity**, which can potentially inspire more future works to effectively address data heterogeneity.