Paper: GPT-FL: Generative Pre-trained Model-Assisted Federated Learning (ICLR-2024)

Reviewer comment (Novelty):
The technical contribution is incremental; the framework mainly applies existing generative models.

Author response:
We believe our contributions are not incremental from three aspects. 
1) Generated data-based FL is an emerging and very promising FL topic. At a high level, our work stands as one of the early and very few generated data-based FL works. More importantly, our work identifies the key limitations of state-of-the-art generated data-based FL approaches, and explains why such limitations are the fundamental reasons for their less competitive performance. 

2) The design of our proposed framework, though seemingly straightforward, is not trivial at all. Our design decouples the synthetic data generation from the federated training process and leverages the knowledge from the generative pre-trained models, which directly addresses the limitations of state-of-the-art generated data-based FL approaches. Such design opens up new directions that no prior works explored, and pushes the state-of-the-arts forward. 

3) Performance wise, our proposed approach not only outperforms generated data-based FL approaches, but also standard FL and public data-based FL approaches significantly, pushing the state-of-the-art FL performance forward.

In the 4th paragraph of the introduction section, we summarize five key merits of our proposed approach over existing FL approaches. We also summarize the key empirical performance advantages in the last paragraph of the introduction section.

Reviewer comment (Novelty):
The methodology is agnostic to the FL framework. The generative-model-based pretraining can be done without FL. The specific relation with FL needs to be justified.

Author response:
We want to clarify that our framework is indeed a FL framework. The purpose of the centralized downstream model training (step 3 in Figure 1) is to provide a better initialized model for the following standard FL process (step 4 in Figure 1). This is the key difference from conventional FL where the model is usually randomly initialized.

Reviewer comment (Experiments):
The motivation for why clients choose to finetune models in a federated way rather than locally is not justified. Better add the performance of performing local finetuning based on the GPT-pretrained server model.

Author response:
Based on your suggestion, we conducted an ablation study comparing local fine-tuning against FL fine-tuning with FedAvg. We did the experiments with Google Command and the Cifar100 dataset. For each experiment in local fine-tuning, we randomly select 10 clients to fine-tune locally in isolation and report the average accuracy. We partitioned the datasets the same as in the paper. More details about these experiments are shown in the revised version of the Appendix. The results are as follows:

|                | Locally Fine-tune | FL Fine-tune with FedAvg |
|----------------|-------------------|--------------------------|
| CIFAR-100      | 35.53%            | 47.80%                   | 
| Google Command | 23.00%            | 81.90%                   |

These findings clearly demonstrate the value of federated learning in fine-tuning, especially given the limitations of local data in terms of volume and diversity.

Reviewer comment (Experiments):
The comparison may not be proper. Some FL methods are agnostic to the pretraining. A better way would be to combine the GPT-FL and these methods to show the relative improvements.

Author response:
We must emphasize that the results of GPT-FL in Table 2 are based on random initialization, the same as other methods in Table 2. As a result, it is a fair and proper comparison between GPT-FL and other baselines in Table 2. 

In fact, we indeed did experiments regarding GPT-FL’s harmonization with the existing pre-train model in the last part of Section 4.2. Specifically, besides training from scratch, GPT-FL
could utilize the existing pre-train model to start training the synthetic data to generate a downstream model and finetune it again with private data in FL. As shown in Table 5, GPT-FL outperforms FL methods starting from a pre-trained model by a large margin.

Reviewer comment (Experiments):
Fig.2 may not be fair. These methods start from scratch, and it is intuitive that these methods would require more communication to converge.

Author response:
We must emphasize that the results of GPT-FL in Table 2 are based on random initialization, the same as other methods in Table 2. As a result, it is a fair and proper comparison between GPT-FL and other baselines in Table 2. 

In fact, we indeed did experiments regarding GPT-FL’s harmonization with the existing pre-train model in the last part of Section 4.2. Specifically, besides training from scratch, GPT-FL
could utilize the existing pre-train model to start training the synthetic data to generate a downstream model and finetune it again with private data in FL. As shown in Table 5, GPT-FL outperforms FL methods starting from a pre-trained model by a large margin.

Reviewer comment (Novelty):
The fourth finding that ‘downstream model generated by synthetic data controls gradient diversity during FL training’ is not new. This has been discussed by Nguyen et al. [1]

Author response:
In Section 4.2, we indeed cite and discuss Nguyen et al.'s work in our paper to provide context for our own findings. Our contribution extends beyond the scope of Nguyen et al.’s research. While Nguyen et al. focused on the impact of starting federated learning from a pre-trained model, our work specifically examines the effects of using a downstream model generated by synthetic data. Our findings reveal that such models can similarly control gradient diversity during FL training, which aligns with Nguyen’s finding.

Reviewer comment (Theory):
Why does the MOON method involve the use of public data? The main idea of MOON is contrastive learning; the claim may not be correct and needs to be further justified.

Author response:
We thank the reviewer for their insightful comment regarding our reference to the MOON method and the use of public data. Upon re-evaluation, we agree with your observation. In our initial manuscript, we incorrectly stated that MOON requires public data, following the assertions made in the Fed-ET paper [1]. We realize now that this interpretation was not accurate. We have revised the relevant sections in our paper.

[1] Cho, Y.J., Manoel, A., Joshi, G., Sim, R., & Dimitriadis, D. Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning. IJCAI 2022.

Reviewer comment (Evaluation):
Why does FedOPT underperform 3x synthetic on image data but outperform it on audio data?

Author response:
This is because the quality of synthetic audio data is worse than that of synthetic image data. The experiment results with synthetic audio align with the previous study [1] that utilizes pure synthetic speech data to train the automatic speech recognition system leading to substantial performance degradation. We have thoroughly discussed it in the paper, and more details are stated in the third paragraph of Section 4.2.

[1] Li, J., Gadde, R.T., Ginsburg, B., & Lavrukhin, V. (2018). Training Neural Speech Recognition Systems with Synthetic Speech Augmentation. ArXiv, abs/1811.00707.

Reviewer comment (Experiments):
Why change the model to ResNets in Table 4? Here needs to be further clarified.

Author response:
In our experiments in Table 2, we opted for VGG19 and ConvNet to maintain consistency with the experimental setups of prior works [1,2] in the field. This choice was intended to facilitate direct comparison and benchmarking. 

However, we recognize that VGG19, being a relatively large model, may not be ideally suited for cross-device FL scenarios due to its computational and memory demands. In response to this consideration, we chose to employ ResNet models for the experiments presented in Table 4. ResNet models are not only lighter but also more commonly used in cross-device FL research. This makes them a more practical choice for such settings. The shift to ResNets also aligns with our objective to demonstrate the versatility of our GPT-FL framework in supporting various model architectures effectively.

[1] Cho, Y.J., Manoel, A., Joshi, G., Sim, R., & Dimitriadis, D. Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning. IJCAI 2022.

[2] Pi, R., Zhang, W., Xie, Y., Gao, J., Wang, X., Kim, S., & Chen, Q. (2022). DYNAFED: Tackling Client Data Heterogeneity with Global Dynamics. CVPR 2023.

Reviewer comment (Experiments):
Using synthetic data seems promising. It would be interesting to further explore the effects of training larger models such as transformer-based methods.

Author response:
Thank you for your insightful suggestions regarding the exploration of larger models. Inspired by your recommendation, we initiated an experiment using the Vision Transformer (ViT) model to further validate our approach. However, due to the extensive computational resources and time required for training such large models, we were unable to complete this experiment within the rebuttal period.

We plan to continue this line of investigation and will include the results in our paper as soon as they are available. This endeavor not only aligns with your suggestion but also opens up new avenues for future research. We recognize the potential of exploring various model architectures in our study and intend to delve deeper into this area. We believe that this will not only enrich our current work but also inspire further research within the community.

Reviewer comment (Experiments):
Does this method still work for more finer tasks? E.g., detection or segmentation.

Author response:
Based on your suggestion, we have also added two new experiments regarding more domain-specific tasks with Food 101[1] and the COVID-19 X-rays[2] dataset in the revised Appendix. 

|                 | 3x Synthetic | FedAvg | GPT-FL w/ FedAvg | 
|-----------------|--------------|--------|------------------|
| COVID-19 X-rays | 37.71%       | 78.36% | 94.65%           | 
| Food 101        | 35.14%       | 43.25% | 70.57%           | 

The results demonstrate that even though the synthetic data has significant differences compared to actual data, GPT-FL still provides performance benefits compared to exclusive reliance on FL.

[1]. Bossard, Lukas et al. “Food-101 - Mining Discriminative Components with Random Forests.” ECCV 2014.

[2]. Chowdhury, Muhammad Enamul Hoque et al. “Can AI Help in Screening Viral and COVID-19 Pneumonia?” IEEE Access 8 (2020): 132665-132676.

Reviewer comment (Experiments):
GPT-FL seems to do very well on standard vision benchmarks, but it is unclear whether the generative model was previously trained on these benchmarks. There is a possibility that the StableDiffusion model has ingested enough test data, thus providing an advantage to the downstream model.

Author response:
**Response**: We evaluated the quality of the generated data by following the well-recognized practices outlined in the latest work on measuring the quality of synthetic data from generative models [1]. Specifically, we evaluated the quality of the generated data by measuring the test accuracy of the model trained on the generated data but tested on real data. The accuracies are presented in Table 3, Table 4, and Table 5. This generated data quality evaluation method has also been adopted by the community in recent works [2,3]. The higher accuracy represents the closer the synthetic data is to the original data.

We have taken into account the impact of out-of-domain data generation, as discussed at the beginning of Section 4.2. We evaluated the synthetic audio data generated by SpeechT5 [4] and AudioLDM [5] models. The downstream task dataset, Google Command, does not overlap (e.g. speakers or utterances) with the LibriSpeech dataset [6], which is used to train the SpeechT5 model. The experience results in Table 3 show that centralized training with synthetic audio performs worse than FL.

However, it is worth noting that GPT-FL does not solely rely on the quality of synthetic data. As highlighted in the benefits of GPT-FL and demonstrated in Table 4, the FL fine-tuning step consistently leads to significant performance gains, surpassing models trained solely with FL or FL combined with synthetic training by a large margin. These results emphasize that regardless of whether the target data falls within or outside the domain of the pre-trained generative model, GPT-FL can efficiently utilize the client data in FL, consistently achieving substantial performance improvements and faster convergence, surpassing the results obtained by models trained solely with real or synthetic data respectively.

[1] He, Ruifei, et al. "Is synthetic data from generative models ready for image recognition?." ICLR 2023.

[2] Shipard, Jordan, et al. "Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusion." 
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Azizi, Shekoofeh, et al. "Synthetic data from diffusion models improves imagenet classification.

[4] Ao, Junyi et al. “SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing.” ACL 2021.

[5] Liu, Haohe et al. “AudioLDM: Text-to-Audio Generation with Latent Diffusion Models.” ICML 2023.

[6] Panayotov, Vassil et al. “Librispeech: An ASR corpus based on public domain audio books.” ICASSP 2015

Reviewer comment (Evaluation):
It is suggested to measure the Fréchet Inception Distance to the training/test images (or use similar generative model evaluation metrics) to gain a better understanding of performance. For audio data, this may be challenging, but the hypothesis is that the closer the synthetic data are to the training/test distribution, the better GPT-FL will perform—potentially explaining Table 3 results.

Author response:
**Response**: We evaluated the quality of the generated data by following the well-recognized practices outlined in the latest work on measuring the quality of synthetic data from generative models [1]. Specifically, we evaluated the quality of the generated data by measuring the test accuracy of the model trained on the generated data but tested on real data. The accuracies are presented in Table 3, Table 4, and Table 5. This generated data quality evaluation method has also been adopted by the community in recent works [2,3]. The higher accuracy represents the closer the synthetic data is to the original data.

We have taken into account the impact of out-of-domain data generation, as discussed at the beginning of Section 4.2. We evaluated the synthetic audio data generated by SpeechT5 [4] and AudioLDM [5] models. The downstream task dataset, Google Command, does not overlap (e.g. speakers or utterances) with the LibriSpeech dataset [6], which is used to train the SpeechT5 model. The experience results in Table 3 show that centralized training with synthetic audio performs worse than FL.

However, it is worth noting that GPT-FL does not solely rely on the quality of synthetic data. As highlighted in the benefits of GPT-FL and demonstrated in Table 4, the FL fine-tuning step consistently leads to significant performance gains, surpassing models trained solely with FL or FL combined with synthetic training by a large margin. These results emphasize that regardless of whether the target data falls within or outside the domain of the pre-trained generative model, GPT-FL can efficiently utilize the client data in FL, consistently achieving substantial performance improvements and faster convergence, surpassing the results obtained by models trained solely with real or synthetic data respectively.

[1] He, Ruifei, et al. "Is synthetic data from generative models ready for image recognition?." ICLR 2023.

[2] Shipard, Jordan, et al. "Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusion." 
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Azizi, Shekoofeh, et al. "Synthetic data from diffusion models improves imagenet classification.

[4] Ao, Junyi et al. “SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing.” ACL 2021.

[5] Liu, Haohe et al. “AudioLDM: Text-to-Audio Generation with Latent Diffusion Models.” ICML 2023.

[6] Panayotov, Vassil et al. “Librispeech: An ASR corpus based on public domain audio books.” ICASSP 2015

Reviewer comment (Experiments):
An additional ablation study could involve conducting FL training on tasks that are more domain-specific (e.g., medical images) and therefore less likely to overlap with the training data of the respective generative models.

Author response:
**Response**: Based on your suggestion, we have also added two new experiments regarding more domain-specific tasks with Food 101[1] and the COVID-19 X-rays[2] dataset in the revised Appendix. 

|                 | 3x Synthetic | FedAvg | GPT-FL w/ FedAvg | 
|-----------------|--------------|--------|------------------|
| COVID-19 X-rays | 37.71%       | 78.36% | 94.65%           |  
| Food 101        | 35.14%       | 43.25% | 70.57%           |  


The results demonstrate that even though the synthetic data has significant differences compared to actual data, GPT-FL still provides performance benefits compared to exclusive reliance on FL or synthetic data.

[1]. Bossard, Lukas et al. “Food-101 - Mining Discriminative Components with Random Forests.” European Conference on Computer Vision (2014).

[2]. Chowdhury, Muhammad Enamul Hoque et al. “Can AI Help in Screening Viral and COVID-19 Pneumonia?” IEEE Access 8 (2020): 132665-132676.

Reviewer comment (Experiments):
A related work, "PS-FedGAN: An Efficient Federated Learning Framework Based on Partially Shared Generative Adversarial Networks For Data Privacy" (Wijesinghe et al., 2023), does not assume having a generative model and instead learns one. It is worth questioning whether this approach would work better in cases where the FL task data are out-of-distribution relative to the pretrained generative model.

Author response:
**Response**: PS-FedGAN proposed to train a GAN-based generator at the server to cope with non-IID data distribution in FL training. As noted in our Introduction, one primary challenge with training generative models concurrently with federated learning is the instability of synthetic data quality during the early stages of convergence. This instability can adversely affect the FL training process. Our concern, as supported by evidence in Table 2, is that such unstable synthetic data, especially in the context of high-resolution imagery, would in turn jeopardize the federated training process. In PS-FedGAN, the absence of consideration for high-resolution images raises questions about its capability in high-resolution image generation. Our observations of related generator-based FL methods, specifically FedGen and DynaFed, indicate difficulties in converging on high-resolution datasets like Flower102. Considering these factors, we infer that our GPT-FL framework would likely offer superior performance compared to PS-FedGAN when dealing with OOD tasks. This inference is also based on the results of the audio datasets and the new-added ablation studies with the Food101 and Covid X-rays dataset. 

We initially wanted to do an experiment with PS-FedGAN for a direct comparison, but the paper does not provide the open-source code, which makes it difficult to reproduce within the rebuttal period. We have included PS-FedGAN in the related work section and marked it as a baseline for future studies.

Reviewer comment (Evaluation):
My main concern is that GPT-FL may significantly increase the burden of the central server. Notice that in the standard FL, the server only needs to aggregate parameters received from clients. However, GPT-FL needs to run pre-trained models on the central server, which requires additional computational cost.

Author response:
Central servers in FL are cloud servers which have significantly more computational power compared to clients. The additional computational cost incurred by running pre-trained models is minimal compared to how many computational resources a central server has. In standard FL, the server only needs to aggregate parameters. Although this incurs little computational cost, the communication cost of collecting parameters from clients through hundreds of iterations is the bottleneck. In contrast, GPT-FL significantly cuts the communication cost. This is the significant benefit of running the pre-trained models on the central server. 

Utilizing this server capability is crucial for mitigating client drift, a limitation of traditional FL methods as evidenced by our results in Table 2 and supported by prior research (referenced in Table 2). The additional computational cost is a justified trade-off for the enhanced performance and drift mitigation provided by GPT-FL. Our approach is aligned with current trends in leveraging server power for more effective FL, such as the baselines we listed in Table 1 and Table 2.

Reviewer comment (Experiments):
I think the experiments may compare the time and memory overhead used by GPT-FL and standard FL methods.

Author response:
It is worth noting that GPT-FL does not alter the standard FL framework, which means that each client in GPT-FL would not have an additional computational or communicational burden, even any additional hyper-parameters beyond the standard FL framework.

As demonstrated in Figures 2, 3, and 4, GPT-FL requires significantly fewer communication rounds compared to standard FL methods for achieving the same target accuracy. This reduction in communication rounds translates to considerable computational savings for the clients. The primary reason behind this efficiency is the offloading of a substantial portion of computation to the server side, specifically through the pre-training of the downstream model. By leveraging the computational power of the server in this manner, GPT-FL not only maintains client-side efficiency but also enhances the overall performance and speed of the federated learning process.