Paper: Federated Generative Learning with Foundation Models (ICLR-2024)

Reviewer comment (Experiments):
The evaluation of the Federated Generative Learning (FGL) framework is limited to simpler domains like ImageNet and doesn't extend to other areas, casting doubt on whether prompts can encapsulate complexity.

Author response:
**Q1: The evaluation of the Federated Generative Learning (FGL) framework is limited to simpler domains like ImageNet and doesn't extend to other areas, casting doubt on whether prompts can encapsulate complexity.** 

**A1**: To further validate the effectiveness of our method, we conducted experiments on several fine-grained image classification datasets, namely CUB-200, Stanford Cars, and also the satellite image dataset EuroSAT. CUB-200 is a challenging dataset consisting of 200 bird species, while Stanford Cars contains 16,185 images belonging to 196 classes of cars. As for EuroSAT, the official dataset did not provide predefined training and testing splits, so we performed a split in an 8:2 ratio. The size of fine-grained recognition datasets is typically smaller compared to general image classification datasets. In previous work, a common practice is to utilize a pretrained model that has been trained on the ImageNet dataset. In this study, we present two approaches: training the model from scratch and loading a pretrained ResNet34 model.
As shown in the table, our method achieves excellent performance even in these challenging domains. This can be attributed to the fact that regardless of the magnitude of domain differences, pretraining a well-performing model on our synthetic data is beneficial for the downstream federated tasks. We have added these results in the appendix.

| |               |         |       |             |              ||  |             |       |  |
|:---------------:|:-------------:|:-------:|:------------:|:-----------:|:------------:|:---------------:|:--------------:|:-----------:|:-----:|:-----------:|
|   Prompt type   | Training type | Dataset | FedAvg,$\beta=0.01$ |FedAvg, $\beta=0.5$ | FedAvg (IID) |    Ours (one-shot)              |  Ours (5-round) ,$\beta=0.01$  |Ours (5-round),$\beta=0.5$ |  Ours (5-round),IID  |        Centralized     |
|     instance    |    scratch    | CUB-200 |     35.04    |    36.61    |     36.62    |      44.17      |      64.53     |    69.19    | 71.01 |    48.31    |
|     instance    |   pretrain    | CUB-200 |     78.98    |    79.08    |     78.48    |      54.02      |      75.13     |    78.96    | 80.72 |    81.77    |
|      class      |    scratch    | CUB-200 |     35.04    |    36.61    |     36.62    |      45.34      |      67.66     |     71.9    | 73.33 |    48.32    |
|      class      |   pretrain    | CUB-200 |     78.98    |    79.08    |     78.48    |      52.73      |      74.68     |     78.7    | 80.32 |    81.77    |
|      class      |    scratch    |   Cars  |     55.18    |    42.43    |     44.48    |      54.48      |      83.31     |    87.22    | 88.07 |    64.72    |
|      class      |   pretrain    |   Cars  |     87.71    |    88.91    |     88.96    |      60.55      |      87.31     |    90.05    | 90.73 |    91.21    |
|      class      |    scratch    | EuroSAT |     43.94    |    74.48    |     84.87    |      38.37      |      37.59     |    82.94    | 91.01 |    94.31    |

Reviewer comment (Evaluation):
While FGL aids in data generation for non-IID data, achieving congruence with a global distribution is yet to be addressed.

Author response:
**Q2: While FGL aids in data generation for non-IID data, achieving congruence with a global distribution is yet to be addressed.**
 
**A2**: Thank you for your valuable feedback. We acknowledge that FGL is effective in generating data for non-IID scenarios, aligning it with a global distribution (e.g., IID settings) also works in our experiments (see Table 1 for IID results).

Reviewer comment (Evaluation):
Security risks of prompts require more analysis. Could prompts be reverse-engineered to obtain private data?

Author response:
**Q3: Security risks of prompts require more analysis. Could prompts be reverse-engineered to obtain private data?**

**A3**:  During the communication phase, traditional Federated Learning (FL) methods typically transmit model parameters or gradients. However, these parameters can be vulnerable to adversarial attacks and model inversion attacks if intercepted by an adversary. To enhance security, some FL methods utilize prompts for communication. However, the potential for attackers to reconstruct private data using prompts has received limited research attention, both in black-box and white-box scenarios.
Recent work[1,2] has identified risks associated with the reconstruction of pretrained data in Diffusion models. Nevertheless, there is currently no available method that can solely reconstruct previously unseen private data in Diffusion models based solely on prompts. This presents an interesting and promising research direction for future investigations.
Consequently, considering the lack of research in this area, our method can be regarded as relatively safe and privacy-preserving.

[1]Shen, Xinyue, et al. "Prompt Stealing Attacks Against Text-to-Image Generation Models." arXiv preprint arXiv:2302.09923 (2023).

[2]Carlini, Nicolas, et al. "Extracting training data from diffusion models." 32nd USENIX Security Symposium (USENIX Security 23). 2023.

Reviewer comment (Experiments):
The framework hasn't been benchmarked against other federated learning methods that employ generative models.

Author response:
**Q4: The framework hasn't been benchmarked against other federated learning methods that employ generative models.**

**A4**: Unfortunately, we were unable to find any existing methods in the literature that directly address our specific setting, making it difficult to perform a fair comparison. In light of this, and following the suggestions of other reviewers, we conducted experiments using various types of generative models to demonstrate the applicability of our proposed method. To investigate the impact of various generative models on the results, we followed the setting in [1]. Our experiments primarily focus on three prevalent conditional diffusion models: DiT[2], GLIDE[3], and Stable Diffusion. We use these off-the-shelf models to generate synthetic images. Specifically, for GLIDE and Stable Diffusion, the prompt was configured as "a photo of {label name}, real-world images, high resolution." For DiT, the input comprised the label ID corresponding to the ImageNet1k dataset. The images synthesized by DiT and GLIDE are of dimensions 256x256, whereas those produced by Stable Diffusion are of dimensions 512x512. As shown in the following table, even when we vary the foundation models used in our method, FGL consistently outperforms FedAvg by a significant margin. This observation serves as evidence for the generality of our approach. We have added these results in the appendix.

|           Method          | one-shot | 5-round, beta=0.01 | 5-round, beta=0.5 |    IID   |
|:-------------------------:|:--------:|:------------------:|:-----------------:|:--------:|
| Ours w/  Stable Diffusion | **85.2** |      **82.8**      |       **94.1**      | **95.6** |
|       Ours w/ Glide       |    79.0    |        76.2        |        89.4       |   89.4   |
|        Ours w/ Dit        |   76.2   |        74.6        |        90.2       |   92.8   |
|     FedAvg (120-round)    |     -    |        51.6        |        75.1       |   79.2   |

[1] Li, Zheng, et al. "Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?." arXiv preprint arXiv:2305.12954 (2023).

[2] Nichol, Alex, et al. "Glide: Towards photorealistic image generation and editing with text-guided diffusion models." arXiv preprint arXiv:2112.10741 (2021).

[3]Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Reviewer comment (Experiments):
The training framework is predominantly tailored for image datasets, limiting its applicability.

Author response:
**A1**: Our approach is based on existing generative models that are widely used in various domains, such as Computer Vision with Stable Diffusion and Natural Language Processing with GPTs. This means that our framework can easily be applied to other domains, including NLP. However, due to time constraints, we were unable to conduct additional experiments on NLP tasks. We believe that further research in this area would be valuable and should be pursued in the future.

Reviewer comment (Theory):
The method heavily depends on the congruence between the captioning and generative models, making it challenging to ensure the proxy dataset's distribution aligns with the private data.

Author response:
**A2**: To further validate the effectiveness of our method, we conducted experiments on several fine-grained image classification datasets, namely CUB-200, Stanford Cars, and also the satellite image dataset EuroSAT. CUB-200 is a challenging dataset consisting of 200 bird species, while Stanford Cars contains 16,185 images belonging to 196 classes of cars. As for EuroSAT, the official dataset did not provide predefined training and testing splits, so we performed a split in an 8:2 ratio. The size of fine-grained recognition datasets is typically smaller compared to general image classification datasets. In previous work, a common practice is to utilize a pretrained model that has been trained on the ImageNet dataset. In this study, we present two approaches: training the model from scratch and loading a pretrained ResNet34 model.
As shown in the table, our method achieves excellent performance even in these challenging domains. This can be attributed to the fact that regardless of the magnitude of domain differences, pretraining a well-performing model on our synthetic data is beneficial for the downstream federated tasks. 

| |               |         |       |             |              ||  |             |       |  |
|:---------------:|:-------------:|:-------:|:------------:|:-----------:|:------------:|:---------------:|:--------------:|:-----------:|:-----:|:-----------:|
|   Prompt type   | Training type | Dataset | FedAvg,$\beta=0.01$ |FedAvg, $\beta=0.5$ | FedAvg (IID) |    Ours (one-shot)              |  Ours (5-round) ,$\beta=0.01$  |Ours (5-round),$\beta=0.5$ |  Ours (5-round),IID  |        Centralized     |
|     instance    |    scratch    | CUB-200 |     35.04    |    36.61    |     36.62    |      44.17      |      64.53     |    69.19    | 71.01 |    48.31    |
|     instance    |   pretrain    | CUB-200 |     78.98    |    79.08    |     78.48    |      54.02      |      75.13     |    78.96    | 80.72 |    81.77    |
|      class      |    scratch    | CUB-200 |     35.04    |    36.61    |     36.62    |      45.34      |      67.66     |     71.9    | 73.33 |    48.32    |
|      class      |   pretrain    | CUB-200 |     78.98    |    79.08    |     78.48    |      52.73      |      74.68     |     78.7    | 80.32 |    81.77    |
|      class      |    scratch    |   Cars  |     55.18    |    42.43    |     44.48    |      54.48      |      83.31     |    87.22    | 88.07 |    64.72    |
|      class      |   pretrain    |   Cars  |     87.71    |    88.91    |     88.96    |      60.55      |      87.31     |    90.05    | 90.73 |    91.21    |
|      class      |    scratch    | EuroSAT |     43.94    |    74.48    |     84.87    |      38.37      |      37.59     |    82.94    | 91.01 |    94.31    |

Reviewer comment (Experiments):
The experimental setup, with only five clients, may not adequately represent real-world scenarios; expanding the evaluation to include 50 or 100 clients could provide more insightful results.

Author response:
**A3**:Thanks for your suggestion. We extended our analysis to include the results obtained from the ImageNette dataset with 50 and 100 clients. As depicted in the table, our method continues to exhibit superior performance compared to FedAvg across all scenarios. Additionally, the improvements achieved by our method remain significant. See more details in the Appendix.

| # Client | FedAvg, $\beta$=0.5 | FedAvg, IID | Ours (one-shot) | Ours (5-round), $\beta$=0.5 | Ours (5-round), IID | Centralized |
|:--------:|:----------------:|:-----------:|-----------------|:------------------------:|:-------------------:|-------------|
|     5    |        75.0        |     79.2    |       85.2      |            94.0            |         95.6        |     92.2    |
|    50    |        72.1        |      77.0     |       85.2      |           93.8           |         91.2        |     92.2    |
|    100   |        70.1        |     67.2    |       85.2      |           92.8           |         93.2        |     92.2    |

Reviewer comment (Experiments):
The comparison to a single baseline, FedAvg, falls short; including comparisons to advanced Federated Learning frameworks could better highlight the proposed method's effectiveness.

Author response:
**A4**: We have compared the two popular FL methods, Moon[1] and Fedopt[2]. We conducted experiments on the ImageNette and ImageNet100 datasets, considering a scenario with 50 clients under non-IID settings (beta=0.5). To the best of our knowledge, there is currently no federated learning method that surpasses centralized training. However, our proposed method even outperforms centralized trained models in many scenarios (see Table 1 in main text). Therefore, as shown in this table, our method still outperforms other federated learning approaches. 

| Method | FedAvg | FedOpt | Moon | Ours (one-shot) | Ours (5-round) | 
|:----------------------:|:------:|:------:|:-----:|:---------------:|:--------------:|
 | ImageNette (beta=0.5) | 72.01 | 73.21 | 74.27 | 85.21 | 93.80 | 
| ImageNet100 (beta=0.5) | 40.13 | 41.25 | 41.43 | 48.31 | 72.67 |

[1]Li, Qinbin, Bingsheng He, and Dawn Song. "Model-contrastive federated learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[2]Reddi, Sashank J., et al. "Adaptive Federated Optimization." International Conference on Learning Representations. 2020.

Reviewer comment (Evaluation):
Table 2 shows the proposed method outperforming centralized learning significantly; a thorough explanation of this phenomenon is warranted.

Author response:
**A5**: This is because our method synthesizes a balanced dataset using all collected prompts during the first round of communication. We then pretrain a "well-initialized" model on this dataset. Once we have this well-initialized model, several rounds of communication can quickly bring the model to a good performance. In the first table, we present the results of directly loading a pretrained model on ImageNet. It can be observed that directly loading a pretrained model on ImageNet reduces the gap between our method and FedAvg. This is because the pretrained model provides a good starting point. However, training a pretrained model on ImageNet requires a significant computational cost on a dataset of 1.3M samples. In contrast, our method only requires training on a small amount of synthesized data to provide a well-initialized model, hence achieving better performance than models trained in a centralized manner.

Reviewer comment (Experiments):
I wonder if the approach can be applied to other types of datasets, besides the image datasets.

Author response:
**A1**: Our approach is based on existing generative models that are widely used in various domains, such as Computer Vision with Stable Diffusion and Natural Language Processing with GPTs. This means that our framework can easily be applied to other domains, including NLP. However, due to time constraints, we were unable to conduct additional experiments on NLP tasks. We believe that further research in this area would be valuable and should be pursued in the future.

Reviewer comment (Experiments):
What the experimental results would be when the number of clients becomes bigger, e.g., 100.

Author response:
**A3**:Thanks for your suggestion. We extended our analysis to include the results obtained from the ImageNette dataset with 50 and 100 clients. As depicted in the table, our method continues to exhibit superior performance compared to FedAvg across all scenarios. Additionally, the improvements achieved by our method remain significant. See more details in the Appendix.

| # Client | FedAvg, $\beta$=0.5 | FedAvg, IID | Ours (one-shot) | Ours (5-round), $\beta$=0.5 | Ours (5-round), IID | Centralized |
|:--------:|:----------------:|:-----------:|-----------------|:------------------------:|:-------------------:|-------------|
|     5    |        75.0        |     79.2    |       85.2      |            94.0            |         95.6        |     92.2    |
|    50    |        72.1        |      77.0     |       85.2      |           93.8           |         91.2        |     92.2    |
|    100   |        70.1        |     67.2    |       85.2      |           92.8           |         93.2        |     92.2    |

Reviewer comment (Experiments):
Settings where FL is helpful—such as medical images across hospitals, user-generated text across mobile phones—are often where the data distributions aren’t covered by the pre-training data of foundational models. The datasets used by the experiments are all natural image datasets (ImageNette, ImageFruit, etc.), which can be well-represented in the pre-training dataset of foundation generative models. Results on non-natural image datasets would be appreciated.

Author response:
**A1:** Although the pretraining dataset and the private dataset may exhibit some domain similarities (e.g., both may contain common real-world scenes), the tasks of lmageSquawk (fine-grained bird classification) and QuickDraw (non-realistic domain) in DomainNet that we demonstrate in our experiments are inherently challenging. Training a model solely on synthetic data generated by foundation models to achieve high accuracy on the ImageNet or DomainNet test sets is a non-trivial task. 
To further validate the effectiveness of our method, we conducted experiments on several fine-grained image classification datasets, including CUB, Cars, and the satellite image dataset EuroSAT. As the official EuroSAT dataset did not provide predefined training and testing splits, we performed a split in an 8:2 ratio. The size of fine-grained recognition datasets is typically smaller compared to general image classification datasets. In previous work, a common practice is to utilize a pretrained model that has been trained on the ImageNet dataset. In this study, we present two approaches: training the model from scratch and loading a pretrained ResNet34 model. As shown in the table, our method achieves excellent performance even in these challenging domains. Additionally, in the cross-silo federated learning scenario, when clients have strong computational capabilities, one can simply finetune the foundation models on these domains, achieving better performance than normal federated learning methods. We have added these results in the appendix. 


| |               |         |       |             |              ||  |             |       |  |
|:---------------:|:-------------:|:-------:|:------------:|:-----------:|:------------:|:---------------:|:--------------:|:-----------:|:-----:|:-----------:|
|   Prompt type   | Training type | Dataset | FedAvg,$\beta=0.01$ |FedAvg, $\beta=0.5$ | FedAvg (IID) |    Ours (one-shot)              |  Ours (5-round) ,$\beta=0.01$  |Ours (5-round),$\beta=0.5$ |  Ours (5-round),IID  |        Centralized     |
|     instance    |    scratch    | CUB-200 |     35.04    |    36.61    |     36.62    |      44.17      |      64.53     |    69.19    | 71.01 |    48.31    |
|     instance    |   pretrain    | CUB-200 |     78.98    |    79.08    |     78.48    |      54.02      |      75.13     |    78.96    | 80.72 |    81.77    |
|      class      |    scratch    | CUB-200 |     35.04    |    36.61    |     36.62    |      45.34      |      67.66     |     71.9    | 73.33 |    48.32    |
|      class      |   pretrain    | CUB-200 |     78.98    |    79.08    |     78.48    |      52.73      |      74.68     |     78.7    | 80.32 |    81.77    |
|      class      |    scratch    |   Cars  |     55.18    |    42.43    |     44.48    |      54.48      |      83.31     |    87.22    | 88.07 |    64.72    |
|      class      |   pretrain    |   Cars  |     87.71    |    88.91    |     88.96    |      60.55      |      87.31     |    90.05    | 90.73 |    91.21    |
|      class      |    scratch    | EuroSAT |     43.94    |    74.48    |     84.87    |      38.37      |      37.59     |    82.94    | 91.01 |    94.31    |

Reviewer comment (Novelty):
In particular, if we consider horizontal FL settings (as with the paper), the server may even know about the possible classes / labels (e.g., federating binary classifiers) without communicating to the clients, in which case the “class-level prompts” may not be needed at all since the server can just generate images by itself.

Author response:
**A2**: Yes, if the server-side has knowledge of the specific labels for the classification task, it can generate them directly. However, class-level is just a simple case. We propose the instance-level approach to address more complex domains, where client-side customized prompt generation is more advantageous in improving the performance of the overall model.

Reviewer comment (Theory):
What exactly is client privacy in this case? Can the client data be still considered “private” if you could already generate them with public foundation models? Does the privacy of the data lie in the pixels, or simply the description of the pixels?

Author response:
**A4**: client privacy: In this paper, similar to differential privacy, we primarily focus on individual privacy, as it is more challenging for attackers. For instance, Attack A perfectly targets a known subset of 0.1% of users in a client, but succeeds with a random 50% chance on the rest. Attack B succeeds with a 50.05% probability on any given user in a client. On average, these two attacks have the same attack success rate. However, the second attack is practically useless, while the first attack is much more powerful in the real-world. This is precisely what LiRA[1] emphasizes, as it evaluates the privacy attack by computing their true-positive rate at very low (e.g., ≤ 0.1%) false-positive rates (as illustrated in our experimental results in Figure 8), demonstrating that our method can better defend against privacy attacks. 

[1] Carlini, Nicholas, et al. "Membership inference attacks from first principles." 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022.

Reviewer comment (Evaluation):
In many cases, the descriptions of the images can already be leaking privacy. If we apply the proposed method to cross-device federated learning on user’s photo data, the server could already learn a lot about the user data distribution and preferences. For example, following Sec 5.4 and Figure 6, knowing that a user has lots of golf photos (without knowing the pixels of the photos) already allows the FL service provider (e.g., Google) to sell targeted ads.

Author response:
**A5**:This could be the difference between individual privacy and group privacy. The majority of the current papers on data protection focuses on the individual ‘user,’ or ‘data subject,’ who’s right to privacy will grow exponentially with the enforcement of the GDPR (General Data Protection Regulation). However, group privacy is not mentioned in the GDPR , which is not well-defined. Also, if the server knows some of the user data distribution means potential privacy risks, our proposed method does not introduce additional risk in this regard. This is because in traditional gradient/parameter-based methods, using model inversion, the server can still infer this information[1,2]. But for individual privacy, this information won't increase the leakage of membership in private data. 

[1] Geiping, Jonas, et al. "Inverting gradients-how easy is it to break privacy in federated learning?." Advances in Neural Information Processing Systems 33 (2020): 16937-16947.

[2] Hatamizadeh, Ali, et al. "Do gradient inversion attacks make federated learning unsafe?." IEEE Transactions on Medical Imaging (2023).

Reviewer comment (Theory):
[Intro section] Why exactly does the proposed method provide robustness to data heterogeneity? Heterogeneity can still surface in the (instance-level) client prompts and subsequently the generated images.

Author response:
**A6**: For one-shot Federated Learning (FL), regardless of the extreme data distributions among different clients, the server can always collect prompts corresponding to all the data, thus obtaining a balanced synthetic dataset on the server. Therefore, compared to FedAvg, our method is not sensitive to data heterogeneity in the first round of communication. In the subsequent model updates, the clients are still affected by non-IID data. However, due to the well-trained initial model obtained in the first round of communication, only a few rounds of communication are needed for local updates, making it more robust to data heterogeneity. As shown in Table 1 in the main text, our method exhibits significantly smaller gaps compared to FedAvg under different non-IID scenarios.

Reviewer comment (Presentation):
Minor comment: consider using different citation commands `\citet`, `\cite`, etc. in LaTeX to make the formatting of the in-text references consistent.

Author response:
**A7:** Thanks for pointing this out. we will check it in the updated version.

Reviewer comment (Experiments):
The proposed method may be highly dependent on the performance of both diffusion models and visual-captioning models. An ablation study of varying the foundation models is needed.

Author response:
**A1**: Thanks for this insightful point. To investigate the impact of various generative models on the results, we followed the setting in [1]. Our experiments primarily focus on three prevalent conditional diffusion models: DiT[2], GLIDE[3], and Stable Diffusion. We use these off-the-shelf models to generate synthetic images. Specifically, for GLIDE and Stable Diffusion, the prompt was configured as "a photo of {label name}, real-world images, high resolution." For DiT, the input comprised the label ID corresponding to the ImageNet1k dataset. The images synthesized by DiT and GLIDE are of dimensions 256x256, whereas those produced by Stable Diffusion are of dimensions 512x512. As shown in the following table, even when we vary the foundation models used in our method, FGL consistently outperforms FedAvg by a significant margin. This observation serves as evidence for the generality of our approach. We have added these results in the Appendix A.4.3.

|           Method          | one-shot | 5-round, $\beta$=0.01 | 5-round, $\beta$=0.5 |    IID   |
|:-------------------------:|:--------:|:------------------:|:-----------------:|:--------:|
| Ours w/  Stable Diffusion | **85.2** |      **82.8**      |       **94.1**      | **95.6** |
|       Ours w/ Glide       |    79.0    |        76.2        |        89.4       |   89.4   |
|        Ours w/ Dit        |   76.2   |        74.6        |        90.2       |   92.8   |
|     FedAvg (120-round)    |     -    |        51.6        |        75.1       |   79.2   |


[1] Li, Zheng, et al. "Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?." arXiv preprint arXiv:2305.12954 (2023).

[2] Nichol, Alex, et al. "Glide: Towards photorealistic image generation and editing with text-guided diffusion models." arXiv preprint arXiv:2112.10741 (2021).

[3]Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Reviewer comment (Experiments):
The local training dataset should be unseen for pretraining foundation models and should be more difficult than ImageNet, which is a standard image classification dataset. As mentioned in the Introduction section, the local training data are more likely to be privacy sensitive, so they are more likely to be unseen or not contained for pre-training foundation models such as BLIPv2 and Stable Diffusion. Evaluation on ImageNet or DomainNet implicitly uses the assumption that local data have a similar or subset domain to the pretraining dataset of foundation models, which are publicly accessible or have no privacy issue.

Author response:
**A3**: Thanks for pointing this out. Here, we would like to address this from three aspects: 
- Although the pretraining dataset and private dataset may have some domain similarities (e.g., both may contain common real-world scenes), the tasks of lmageSquawk (fine-grained bird classification) and QuickDraw (non-realistic domain) in DomainNet that we show in our experiments are challenging. It is a non-trivial task to train a model only using synthetic data generated by foundation models to achieve good accuracy on the ImageNet or DomainNet test sets. 

- Furthermore, even when there are some domain similarities between the pretraining dataset and private dataset, does it mean that there is no need to discuss the privacy risk? Definitely not! Let's consider a scenario where a public dataset contains various images of cats, while a private dataset contains personal images of cats belonging to individual users. Although both datasets involve images of cats, the private dataset may contain users' personal information, such as their family photos or addresses. Therefore, even if the two datasets are similar in some aspects, the private data still carries privacy risks and needs to be properly protected. Taking Membership Inference Attack (MIA) as an example, consider an adversary that wants to probe a ML model to test membership of an individual's data in the model's training data. In this scenario, an adversary is more likely to have access to some representative images of the target individual, but not necessarily the ones used for training the model. As shown in Figure 8, we implemented the state-of-the-art LiRA algorithm in MIA. The experimental results demonstrate that our approach ensures the protection of sensitive information of the members in the clients' data (since the model training process has never been exposed to any private data). In contrast, traditional federated learning methods directly train on private data, posing a high risk of exposing the sensitive information of the members in the clients' data (i.e., for certain private data samples, attackers have a high confidence in identifying the client from which the sample originates). To the best of our knowledge, prior to our proposed approach, no one has put forth a training paradigm that effectively defends against LiRA while concurrently modeling utility (i.e., achieving high test accuracy). 

- Finally, even for particularly challenging domains such as remote sensing images or fine-grained classification datasets, our method can easily adapt to these scenarios. We conducted experiments on several fine-grained image classification datasets, namely CUB-200, Stanford Cars, and also the satellite image dataset EuroSAT. CUB-200 is a challenging dataset consisting of 200 bird species, while Stanford Cars contains 16,185 images belonging to 196 classes of cars. See more details in the `Appendix A.4.2`.

| |               |         |       |             |              ||  |             |       |  |
|:---------------:|:-------------:|:-------:|:------------:|:-----------:|:------------:|:---------------:|:--------------:|:-----------:|:-----:|:-----------:|
|   Prompt type   | Training type | Dataset | FedAvg,$\beta=0.01$ |FedAvg, $\beta=0.5$ | FedAvg (IID) |    Ours (one-shot)              |  Ours (5-round) ,$\beta=0.01$  |Ours (5-round),$\beta=0.5$ |  Ours (5-round),IID  |        Centralized     |
|     instance    |    scratch    | CUB-200 |     35.04    |    36.61    |     36.62    |      44.17      |      64.53     |    69.19    | 71.01 |    48.31    |
|     instance    |   pretrain    | CUB-200 |     78.98    |    79.08    |     78.48    |      54.02      |      75.13     |    78.96    | 80.72 |    81.77    |
|      class      |    scratch    | CUB-200 |     35.04    |    36.61    |     36.62    |      45.34      |      67.66     |     71.9    | 73.33 |    48.32    |
|      class      |   pretrain    | CUB-200 |     78.98    |    79.08    |     78.48    |      52.73      |      74.68     |     78.7    | 80.32 |    81.77    |
|      class      |    scratch    |   Cars  |     55.18    |    42.43    |     44.48    |      54.48      |      83.31     |    87.22    | 88.07 |    64.72    |
|      class      |   pretrain    |   Cars  |     87.71    |    88.91    |     88.96    |      60.55      |      87.31     |    90.05    | 90.73 |    91.21    |
|      class      |    scratch    | EuroSAT |     43.94    |    74.48    |     84.87    |      38.37      |      37.59     |    82.94    | 91.01 |    94.31    |

Reviewer comment (Experiments):
Clients in federated learning are often assumed to have limited capacity in memory or computation. Generating prompts using a large visual captioning model in each client is impractical.

Author response:
**A2**: 
- First, compared to FedAvg, our method introduces only one additional operation on the client side, i.e., prompt generation, which involves only forward propagation and does not impose significant computational costs. All computational operations are executed on the server side during the initial communication. The server trains a model with an excellent initial state, and subsequently, the client performs regular model updates, which means no additional cost compared with FedAvg. 
- Secondly, our method is particularly well-suited for cross-silo FL, where the clients represent organizations or companies. In this context, the number of clients is typically small, but they possess substantial computational resources. Furthermore, this scenario emphasizes the importance of protecting clients' local data from potential leaks, which constitutes a significant contribution of our approach towards preserving privacy.

Reviewer comment (Experiments):
The quality of synthetic data could be highly different according to domain discrepancy between the local training data and the pretraining data for the foundation model. Instead of using standard image classification datasets, does the proposed method work for federated learning on fine-grained classification such as CUB-200, Cars, and medical image datasets?

Author response:
**A4**: Please refer to the table in A3.