Paper: Federated Learning Empowered by Generative Content (ICLR-2024)

Reviewer comment (Theory):
Other than considerations regarding generating data locally for each client (i.e. the four aspects at generating the data above), there is no significant theoretical contribution. There is no theorem, no proposal. Not a single equation is found in the paper.

Author response:
**W1/W3:** There is no theorem, no proposal. Not a single equation is found in the paper. I think this part of the paper is good from a practical point of view. However, it appear to be dominantly engineering contributions, which does not seem suitable for ICLR, a conference about learning representations.

**Response:** 

First, federated learning itself is a practical setting and we propose a practical solution, which naturally fits the overall objective.

Second, this judgment varies from person to person. We kindly request the reviewer to refer to the following ICLR papers, where they might seem to be 'unrelated' and 'engineering' for the reviewer.

[1] Chen, Hong-You, et al. "On the importance and applicability of pre-training for federated learning." The Eleventh International Conference on Learning Representations. 2023.

[2] Nguyen, John, et al. "Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning." The Eleventh International Conference on Learning Representations. 2023.

[3] Oh, Jaehoon, SangMook Kim, and Se-Young Yun. "FedBABU: Toward Enhanced Representation for Federated Image Classification." International Conference on Learning Representations. 2022.

Reviewer comment (Novelty):
The paper solely focuses on data generation for local training. However, there is nothing in the communication among the clients that carries any information about data generation from one client to another, other than the number of samples to be generated for each client. In other words, the use of federated learning and data generation appear to be unrelated. I wonder if the whole work could have been better presented in a non-federated setting.

Author response:
**W2:** The paper solely focuses on data generation for local training. However, there is nothing in the communication among the clients that carries any information about data generation from one client to another, other than the number of samples to be generated for each client. In other words, the use of federated learning and data generation appear to be unrelated. I wonder if the whole work could have been better presented in a non-federated setting.

**Response:** Focusing on data heterogeneity, we propose a new method that can effectively alleviate its negative effects. The generative data can significantlly mitigate the level of data heterogeneity and the issue of overfitting, promoting the performance of FL. Thus, it is not 'unrelated'.

Data heterogeneity is a representative and common issue in federated setting while in a non-federated setting there is no defination of data heterogeneity. Thus, the idea of using generative content to address data heterogeneity is quite **unique and suited for FL** but not non-FL setting. From the following table, we can see that local training (without FL) with generative content performs significantly worse.

[**Table R1.** Performance comparison between local training with generative content and our FedGC.]
| Method | CIFAR-High | CIFAR-Low | EuroSAT-High | EuroSAT-Low |
|:--------:|:----------:|:---------:|:------------:|:-----------:|
| Local+GC |   46.89    |   50.47   |    24.87     |    35.48    |
|  FedGC   | **74.50**  | **79.93** |  **74.83**   |  **84.46**  | 


To make the design 'more like' FL, there could be many methods following FedGC framework. For example, to mitigate the potential negative effects of dissimilarity between generative and real data, we can use soft-label supervision to train local model on generative data uwhile hard-label supervision for real data, where the soft labels come from the FL global model. We leave such methods to future works.

Reviewer comment (Novelty):
In terms of data generation for each client itself, the proposed approaches for each of the 4 aspects above are simple and straightforward. I think this part of the paper is good from a practical point of view. However, it appears to be dominantly engineering contributions, which does not seem suitable for ICLR, a conference about learning representations.

Author response:
**W1/W3:** There is no theorem, no proposal. Not a single equation is found in the paper. I think this part of the paper is good from a practical point of view. However, it appear to be dominantly engineering contributions, which does not seem suitable for ICLR, a conference about learning representations.

**Response:** 

First, federated learning itself is a practical setting and we propose a practical solution, which naturally fits the overall objective.

Second, this judgment varies from person to person. We kindly request the reviewer to refer to the following ICLR papers, where they might seem to be 'unrelated' and 'engineering' for the reviewer.

[1] Chen, Hong-You, et al. "On the importance and applicability of pre-training for federated learning." The Eleventh International Conference on Learning Representations. 2023.

[2] Nguyen, John, et al. "Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning." The Eleventh International Conference on Learning Representations. 2023.

[3] Oh, Jaehoon, SangMook Kim, and Se-Young Yun. "FedBABU: Toward Enhanced Representation for Federated Image Classification." International Conference on Learning Representations. 2022.

Reviewer comment (Experiments):
The dataset for experimenting was made up by merging from a few existing known datasets which were not designed for federated learning. Heterogeneity in the dataset was synthesized using Dirichlet distribution. This is probably one major weakness of the paper. It is questionable whether the dataset reflects real world. Results in this dataset are therefore not very convincing. I reckon the authors to use real-world datasets, or if that task is not feasible, at least use the same datasets that other federated learning approaches have used, rather than creating one of your own.

Author response:
**W4:** The dataset for experimenting was made up by merging from a few existing known datasets which were not designed for federated learning?

**Response:** 

Please note that most of the used datasets are **quite common in federated learning literature and we follow exactly the same setting. Here are some examples.**

- Sentiment140, which is from FL LEAF benchmark [1] (1052 citations).
- CIFAR10, which is commonly used in most of FL papers [2,3,4].
- HAM10000 [5,6,7], PACS[8,9,10].

[1] Caldas, Sebastian, et al. "Leaf: A benchmark for federated settings." arXiv preprint arXiv:1812.01097 (2018).

[2] Chen, Hong-You, et al. "On the importance and applicability of pre-training for federated learning." The Eleventh International Conference on Learning Representations. 2023.

[3] Nguyen, John, et al. "Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning." The Eleventh International Conference on Learning Representations. 2023.

[4] Oh, Jaehoon, SangMook Kim, and Se-Young Yun. "FedBABU: Toward Enhanced Representation for Federated Image Classification." International Conference on Learning Representations. 2022.

[5] Thapa, Chandra, et al. "Splitfed: When federated learning meets split learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 8. 2022.

[6] Yang, Fu-En, Chien-Yi Wang, and Yu-Chiang Frank Wang. "Efficient model personalization in federated learning via client-specific prompt generation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[7] Pennisi, Matteo, et al. "Experience Replay as an Effective Strategy for Optimizing Decentralized Federated Learning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[8] Zhang, Ruipeng, et al. "Federated domain generalization with generalization adjustment." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[9] Nguyen, A. Tuan, Philip Torr, and Ser Nam Lim. "Fedsr: A simple and effective domain generalization method for federated learning." Advances in Neural Information Processing Systems 35 (2022): 38831-38843.

[10] Chen, Haokun, et al. "Fraug: Tackling federated learning with non-iid features via representation augmentation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Reviewer comment (Evaluation):
On the one hand, the authors show that increasing the amount of generative data always increases the learning performance (Table 2 and Figure 2). On the other hand, the authors also show that FL on only generative contents (no private data) performs poorly in Table 5. This seems counterintuitive. Could the authors explain why?

Author response:
That is a good point. Acturally, **they are not counterintuitive. Please note that in Table 2, the performance of using 50000 samples is worse than performance of using 20000 samples.**

Here is the rationale. (1) First, the results showing that only generative contents performs poorly indicates that the generative data cannot fully represent real data. (2) Second, when generative data is combined with real heterogeneous data, the generative data serves to mitigate data heterogeneity and overfitting issue. Therefore, the performance is increased after adding generative data.

However, when the number of generative samples is too large, the issue of dissimilarity between generative and real data tends to bring negative effects.


[**Table R1.** Performance under different number of generative samples.]
| Baseline |   0   | 1000  | 10000 |   20000   | 50000 |
|:--------:|:-----:|:-----:|:-----:|:---------:|:-----:|
|  FedAvg  | 61.25 | 66.98 | 74.50 | **76.93** | 76.39 |
| FedProx  | 64.02 | 68.55 | 74.36 | **76.81** | 76.73 |
| SCAFFOLD | 63.98 | 71.33 | 73.96 | **74.88** | 73.98 |

Reviewer comment (Experiments):
Additionally, could authors discuss the possible dataset contamination? That is, whether the benchmark dataset is already used in training dataset for the generative model.

Author response:
Sure, we also take this factor seriously and that is why we have shown the following results in the submission.


1. The experiments on **medical dataset HAM10000 in Table 4**. Since stable diffusion can not generate medical with good quality, we choose this experiment to alleviate the concern of contamination. Please note that the generated images are completely different from the real iamges and we decided not to posting them to avoid causing discomfort. Here, we put the results below for convinience. **Despite the dissimilarity, our method still brings significant gain.**

[**Table R1.** Results on medical dataset.]
| Baseline | Without FedGC | With FedGC |
| :------: | :-----------: | :--------: |
| FedAvg   | 48.57         | **56.67**      |
| FedProx  | 49.52         | **56.19**      |
| SCAFFOLD | 54.76         | **58.57**      | 

2. The experiments of training on **generated data only in Table 6.** Here, we put the results below for convinience. From the table, we see that **merely using generative data to train a model fail to perform well**, indicating that there is a huge gap between generative data and real data. Generated data can only exhibit its effectiveness when used in conjunction with real data in our proposed FedGC.

[**Table R2.** Comparisons with training on genrative data only. Training on generative data only achieves low performance.]
| Baseline | Real Data | Generative Data | FedGC (ours) |
|:--------:|:---------:|:---------------:|:------------:|
|  FedAvg  |   60.77   |      41.85      |  **73.99**   |
| FedProx  |   63.62   |      40.93      |  **73.69**   |
| SCAFFOLD |   65.00   |      43.45      |  **75.79**   |

3. We compare real data and generative data in Figure 8 via visualization, where we see that there is a large amount of generative data with incorrect concept and can not fully represent real data.

Reviewer comment (Evaluation):
The author does not consider the generation cost in the paper. The Stable Diffusion model needs at least 4.2GB space to deploy locally, and the memory consumption of generation is huge for the IoT or cross-device FL setup. For the black-box foundational models such as ChatGPT, the prompts directly leak the data privacy to the server of ChatGPT. As a result, both methods do not fit the FL setups.

Author response:
Foremost, we want to reiterate the emphasis of our paper: (1) the **idea** of leveraging advanced generative models to assist training on private data, and (2) the **general framework** with diverse solutions for each sub-step.

Regarding the concern of generation cost, it can be easily alleviated by generating data using **online interfaces (rather than generating locally)**, which are commonly available right now (e.g., huggingface) [1]. And we can generate generate data without real-data guidance for such cases since our proposed FedGC provides diverse solutions for each generation step. In this case, **both the issues of generation burden and privacy issue are alleviated.**

Besides, we acturally have recognized such potential concern so that we **do not use real-data-guidance for all the experiments except Table 4**. However, since our focus is to propose a general and multifunctional framework, we still decided to include this because such strategy can bring additional benefit if the computational resource is available.

[1] https://huggingface.co/runwayml/stable-diffusion-v1-5

Reviewer comment (Evaluation):
I am concerned about the in-domain generation problem. As the Stable Diffusion is trained with the LAION-5B dataset, we cannot guarantee that the Stable Diffusion does not meet with the test data, such as the CIFAR dataset, during the training. As a result, we could not distinguish whether the performance boost-up is coming from the FedGC or the in-domain generation of the foundational models.

Author response:
Acturally, we have put efforts to alleviate such concerns from three aspects.

1. The experiments on **medical dataset HAM10000 in Table 4**. Since stable diffusion can not generate medical with good quality, we choose this experiment to alleviate the concern of in-domain generation. Please note that the generated images are completely different from the real iamges and we decided not to posting them to avoid causing discomfort. Here, we put the results below for convinience. **Despite the dissimilarity, our method still brings significant gain.**

[**Table R1.** Results on medical dataset.]
| Baseline | Without FedGC | With FedGC |
| :------: | :------: | :------: |
| FedAvg   | 48.57         | **56.67**      |
| FedProx  | 49.52         | **56.19**      |
| SCAFFOLD | 54.76         | **58.57**      | 

2. The experiments of training on **generated data only in Table 6.** Here, we put the results below for convinience. From the table, we see that **merely using generative data to train a model fail to perform well**, indicating that there is a huge gap between generative data and real data. Generated data can only exhibit its effectiveness when used in conjunction with real data in our proposed FedGC.

[**Table R2.** Comparisons with training on genrative data only. Training on generative data only achieves low performance.]
| Baseline | Real Data | Generative Data | FedGC (ours) |
|:---:|:---:|:---:|:---:|
|  FedAvg  |   60.77   |      41.85      |  **73.99**   |
| FedProx  |   63.62   |      40.93      |  **73.69**   |
| SCAFFOLD |   65.00   |      43.45      |  **75.79**   |

3. We compare real data and generative data in Figure 8 via visualization, where we see that there is a large amount of generative data with incorrect concept and can not fully represent real data.

Reviewer comment (Experiments):
During local data generation, does the local client only generate the data for the label it holds or generate the data for the whole label space among all participants?

Author response:
Local client generates data for the whole label space.