Paper: Differentially Private Synthetic Data via Foundation Model APIs 1: Images (ICLR-2024)

Reviewer comment (Theory):
Though the result looks nice, I am so unconvinced it is true. A generative model basically learns the training distribution, so that it can generate something similar to the training set. However, if the training distribution is different from what you want to generate, how is it possible? Sec 5.1.1 looks reasonable, because the distribution of CIFAR10 has a large overlap with (or maybe is even covered by) ImageNet. However, CAMELYON17 is totally different from ImageNet, why is a diffusion pretrained on ImageNet able to generate CAMELYON17 images? The authors did not give any formal or intuitive explanations on this point.

Author response:
**A2:** Even though the diffusion model is trained on natural images, the support of the generated distribution could be very large due to the formulation and the large latent space (same size as images). More concretely, let’s look at two examples: score-based models (https://arxiv.org/abs/1907.05600, closely related to diffusion models) and the diffusion model we used (Nichol & Dhariwal, 2021). For score-based models, its modeling target is a distribution perturbed by Gaussian noise, and therefore, the generated distribution spans the entire sample space. For diffusion models (Ho et al., 2020), the latent space has the same size as the image, and the last denoising step is modeled as a distribution derived from a Gaussian distribution that spans the entire pixel space (see Section 3.3 of Ho et al. (2020)). Therefore, the generated distribution of diffusion models also spans the entire sample space.

In other words, for any (trained) score-based models or diffusion models, it is theoretically possible to generate images similar to the private dataset (or in fact, generate any images). The problem is that the probability of generating such images is small if there is a large distribution shift from the pre-training data to the private data. PE is effective in guiding the diffusion model to generate samples from the region that is low-density in the original pre-trained distribution but high-density in the private distribution.

To make it more concrete, we have provided how the generated images evolve from natural images to Camelyon17 dataset in Figure 19 in the original submission (Figure 23 in the revision) and the selected/filtered samples by PE in Figure 26 in the revision. At every iteration, PE selects the set of images that are most similar to the Camelyon17 dataset (Figure 26 in the revision). Those images might still appear different from Camelyon17 in early iterations. However, as long as we get images that are more similar to Camelyon17 through VARIATION_API at every iteration, we will make progress, and finally, we can get images similar to Camelyon17 (Figure 17 in the original submission or Figure 21 in the revision).

We added brief explanations in Section 5.1.2, and an expanded version in Appendix J in the revision. We also want to note that (1) we have provided *theoretical* justification of why PE works in Section D in the original submission, and (2) we have provided the source code for reproducing the experiments in the supplementary material.

Reviewer comment (Presentation):
Based on the second concern, it would be more clear and intuitive to highlight samples that are used to update the histogram in each iteration. For example, in Figure 19, iteration=0, what generative samples (I believe not all of them) are closer to the private set and thereby used to update the histogram?

Author response:
**A3:** Thanks for the great idea! Following your suggestions, we add samples with **the highest and the lowest count** for **both CIFAR10 and Camelyon17** in Figure 19 and Figure 26 in the revision. The key observations are summarized below.
* On the CIFAR10 dataset, we can see that the DP NN Histogram picks samples similar to the private data as desired. This is more obvious in the first two PE iterations, where the DP NN Histogram assigns high votes on the samples with the correct classes and puts low votes on the samples with incorrect classes. 
* On the Camelyon17 dataset, we can see that the DP NN Histogram picks samples with the right patterns as the private data and drops samples with more different patterns as desired. As the Camelyon17 dataset is further away from the pre-training dataset ImageNet than CIFAR10, we can see that it converges slower than the case in CIFAR10.

Reviewer comment (Theory):
By reading Alg. 2, it looks to me that it is possible that the histogram will be more and more concentrated on some samples when the overlap between the generation set and the private set is small, thus implying that the variation of the generation can be low.

Author response:
**A4:** The reality is on the contrary: at the beginning, the histogram is concentrated; as more iterations go on, the histogram becomes more uniform. The reason is as follows. In the beginning, only a few generated samples are close to the private data. Those generated samples get most of the votes from private samples, which results in a concentrated histogram. PE will then pick those highly-voted generated samples and do more variations over them. As a result, private samples that voted for the same generated sample may now find different closest generated samples and distribute the votes, which results in a “more uniform” histogram. 

To support the above claim, we show the standard deviation of the counts in the DP NN Histogram across different PE iterations on both CIFAR10 and Camelyon17 in Figure 20 and Figure 29 in the revision. The figure shows that the histogram becomes “more uniform” instead of more concentrated across iterations.

Reviewer comment (Reproducibility):
I find it hard and a bit confusing to track the pretrained model information in the paper. The term "pretrained model" and "API" seem to be interchangeably used, but you have two APIs. For example, in App. I, you mentioned only one pretrained model, is that your random_API? In API implementation, you mention both APIs, and the variation_API is SDEdit. Is it also pretrained on ImageNet?

Author response:
**A5:** Sorry for the confusion! We want to clarify that:

* In each experiment, there is only *one* pre-trained model (a diffusion model), and both APIs (RANDOM_API and VARIATION_API) are implemented using the same model. 
* RANDOM_API is implemented by drawing the samples from the diffusion models using the standard diffusion model sampling process (Appendix I).
* SDEdit (Meng et al., 2021) is a method, *not* a model. VARIATION_API is implemented by applying SDEdit on the same diffusion model. SDEdit adds noise to input images and lets the diffusion model denoise them to get their variation (Appendix I).

We add clarification in Section 5.1. Hope it is clearer now!

Reviewer comment (Experiments):
It makes no sense to compare the performance at different $(\epsilon, \delta)$-DP, such as the comparison in Sec 5.1.2. Why not targeting the same DP guarantee?

Author response:
**A6:** Thank you for the question! Following your suggestions, we rerun PE with a re-computed noise multiplier so that the DP parameter epsilon is exactly 10, the same as the baseline method. The accuracy is 80.33%, slightly higher than 79.56% at epsilon=7.58. We update the results in the revision. 

Note that these results do not change the takeaway messages in Section 5.1.2: (1) Even with a large distribution shift, PE can still achieve a non-trivial utility of DP synthetic data (much higher than random guess accuracy of 50%). (2) The traditional training-based methods (91.1% accuracy) are still more promising in this case if the privacy-utility trade-off is the only goal. However, given the benefit of API-only assumption and the non-trivial results that PE already got, it is worth further exploiting the potential of PE in future work.

Reviewer comment (Evaluation):
Fig. 19(a) is the output of random_API (iteration=0), but some images look weirdly pink-ish, e.g. col 1 row 7, col 7 row 6, col 8 row 10, while the rest looks gray-ish. This is weird if the random_API is well pretrained. Do the authors have any interpretations?

Author response:
**A8:** This is because we used the DDIM sampler (Song et al., 2020) with only 10 sampling steps to draw samples from the diffusion model. It is known in the community that the DDIM sampler with a small number of sampling steps can result in samples with bad quality and/or weird colors. See Figure 6 in https://arxiv.org/pdf/2302.04867v1.pdf, Figures 1,3,5 in https://arxiv.org/pdf/2206.00927.pdf, and Figures 1,6 of https://arxiv.org/pdf/2211.01095.pdf for such examples in other datasets and diffusion models. Better samplers (such as the papers listed above) and more sampling steps can improve the results. In our experiments, we used the DDIM sampler because it is the default one implemented in the Improved Diffusion codebase (the model we used as the API). We only used 10 sampling steps because it is faster to generate samples. We found that such a less optimal configuration is sufficient for PE to work reasonably well so we did not try better configurations.

Reviewer comment (Theory):
**Privacy guarantee**: 1) The query at each iteration depends on the output of the last iteration; such adaptive queries lead to correlated outputs. This may potentially amplify the privacy loss.

Author response:
**A1.1:** We want to clarify that our privacy analysis already takes the adaptive queries into account. The related discussions are in Step 3 and Step 4 of the privacy analysis in Section 4.3. 
It is well known that differential privacy satisfies the adaptive composition theorem (see [1]), i.e., if $M_1$ and $M_2$ are two DP algorithms, then first running $M_1$ on database $D$ to get $M_1(D)$ and then running $M_2$ on $D$ as well as the output of $M_1$, i.e., $M_2(D,M_1(D))$ is still differentially private. (Here we need that $M_2(D,z)$ is DP for all values of z, which is true in our case.) This is how we can care of any correlations that arise.

In particular, in Step 3, we mention that PE can be regarded as $T$ (adaptive) compositions of the Gaussian mechanism. Then, in Step 4, we use the property that such composition can be regarded as one Gaussian mechanism with noise multiplier $\sigma/\sqrt{T}$. It is a standard result from Dong et al. (2022) (see Corollary 3.3 therein).

Please let us know if it is unclear, and we are happy to clarify further!

[1] Dwork, Cynthia, and Aaron Roth. "The algorithmic foundations of differential privacy." Foundations and Trends® in Theoretical Computer Science 9.3–4 (2014): 211-407.

Reviewer comment (Theory):
**Privacy guarantee**: 2) If the underlying model of the API memorizes information from the queries (and can even update the model based on user interactions, such as what OpenAI does for GPT-4), there could be long-term privacy implications, especially that at later iterations, the data fed back to the API are highly similar to the private data.

Author response:
**A1.2:** We want to clarify that releasing all the (intermediate) generated sets $S_1,\dots,S_T$ also satisfies the same DP guarantees as the final generated dataset. Note that the API only sees $S_1,\dots,S_T$ during the interaction. Therefore, even if the API memorizes all the previous queries and feeds them back at later PE iterations, it does not violate the DP guarantee. We discussed this point in Section 4.3; please let us know if you have further questions.

Reviewer comment (Experiments):
**Conditional generation**: The authors treat the conditional generation as unconditional generation for each class. This is a valid approach, but think about the scenario where there are many classes and only a few images per class (e.g., CelebA dataset with 10k classes and ~20 images per class), it may be better for the algorithm to not limit itself to one class of data. Can the authors comment on how they would adapt the algorithm for such type of data, and the utility? (One might expect that the threshold needs to be low to accommodate for the few number of images per class, but this means more noise).

Author response:
**A3:** Thanks for the great question! 

In the paper, we test PE in the setting where there are only 100 images (the cat dataset), and PE works reasonably well. Although we did not test it on an even smaller dataset, we believe that the number of private samples ($N_{priv}$) itself is not the bottleneck of the performance of PE; rather, the ratio between the number of generated samples ($N_{syn}$) and the number of private samples, i.e., $N_{syn}/N_{priv}$, is more important for the performance of PE. Note that $N_{priv}$ determines the total of the votes from the private samples in the DP NN histogram (Line 2-4 in Algorithm 2), and Gaussian noise is added to all positions in the DP NN histogram of size $N_{syn}$ (Line 5 in Algorithm 2). Therefore, if $N_{syn}/N_{priv}$ is too large, the true votes in the DP NN histogram will be destroyed by the Gaussian noise. In all our experiments, we set $N_{syn}/N_{priv}=1$ and it works well empirically. Note that it does not mean that PE can only generate as many samples as the private dataset; as we illustrated in Sections 4.2 and 5.1.3, PE can generate unlimited (useful) samples by calling VARIATION_API on these $N_{syn}$ samples. 

We also want to clarify that the threshold $H$ does not affect the privacy guarantees; it only affects the utility. We conducted the ablation study across a wide range of values (0~15) (see Figure 31 in the original submission or Figure 40 in the revision). Empirically, we found that setting threshold $\in[1,2]\sigma$ ($\sigma$ is the noise multiplier) works well across all datasets of different sizes we evaluated. 

Hope this answers your question :)

Reviewer comment (Evaluation):
**Monetary cost of the algorithm**: The authors mentioned the foundation models "including GPT4, Bard, and DALLE2 (that) only provide API access without releasing model weights or code". For those models: 1) often the API calls are not inexpensive; 2) they may support fine-tuning access. Can the authors discuss the costs of the two routes (DPSDA vs. DP finetuning)?

Author response:
**A4:** We discuss the two sub-questions separately below.

* Indeed, the APIs of those *black box models* could be expensive. In this paper, we did not take the number of API calls into account when optimizing PE. One future work is to optimize the number of API calls along with privacy-utility tradeoffs. We discussed this point in Section 6.

  We also want to mention that, besides *black box models*, PE can also use *local models* such as Improved Diffusion and Stable Diffusion (as in our experiments). In this case, both PE and DP fine-tuning are feasible. We showed in Appendix O of the original submission that **the computational cost of PE can be smaller than DP fine-tuning**. This demonstrates the practical value of PE.
* Indeed, some of those models provide fine-tuning APIs. However, so far they do not support DP fine-tuning and do not provide gradients. Also, uploading sensitive data to these APIs controlled by other companies can lead to privacy violations. We explained it in footnote 2.

Reviewer comment (Evaluation):
**Diversity-utility trade-off**: I'm curious about whether calling more VARIATION_API will hurt the utility. Apparently the diversity increases because there's no longer constraints in producing nearest neighbor images. But without this constraint, and without the knowledge of how the foundation model API produces a "similar" image, how can we be sure about retaining similar level of data utility? The results in Fig. 5 are encouraging results, but I'd appreciate more evaluation to directly corroborate the argument, e.g., using the initial 50k images to train a classifier, and compare with another 50k after calling the API for a few times.

Author response:
**A5:** Sorry for the confusion, but we want to clarify how “generating more samples” in Figure 5 is done. The reviewer is right that if we recursively call VARIATION_API multiple times, i.e., VARIATION_API(VARIATION_API(...($S_{syn}$))), then the generated samples may not have good utility because the generated samples drift away from the private samples with each call of VARIATION_API. **However, it is not what “generating more samples” is implemented (see Section 4.2).** Instead, there is only one call of VARIATION_API for each sample. For example, to generate 1M samples from the $S_{syn}$ with 50000 samples, we call VARIATION_API 20 times, each with $S_{syn}$ as input: [VARIATION_API($S_{syn}$), …, VARIATION_API($S_{syn}$)]. If one wants to tune the diversity of these augmented samples, the variation degree parameter $v$ (Section 3.3) can be adjusted. 

We clarify the above points in Section 4.2. Thank you for pointing it out!

Reviewer comment (Evaluation):
**Evaluation**: Fig. 6 presented generated Camelyon17 images with (9.92, 3x10^{-6})-DP. I wonder why the authors didn't report the accuracy on this generated dataset while instead reported the numbers for the \epsilon=7.58 dataset? Clearly it's more fair to compare the \epsilon=9.92 results with the \epsilon=10 results from the prior work. I appreciate it if the authors can add the result.

Author response:
**A7.1:** Thank you for the question! Following your suggestions, we add two experiments: (1) We compute the classification accuracy of the epsilon=7.58 dataset. (2) We rerun PE with a re-computed noise multiplier so that the DP parameter epsilon is exactly 10, the same as the baseline method. 

The results are as follows. (1) The epsilon=9.92 dataset has an accuracy of 78.53%, slightly lower than the accuracy under epsilon=7.58 (79.56%). This accords with the finding in Appendix M that the data quality might degrade at the later stage of PE when the hyper-parameters are not optimal. Nevertheless, these two accuracies are very close. (2) The epsilon=10 dataset has an accuracy of 80.33%, slightly higher than the above two cases. We update the results in the revision. 

Note that these results do not change the takeaway messages in Section 5.1.2: (1) Even with a large distribution shift, PE can still achieve a non-trivial utility of DP synthetic data (much higher than random guess accuracy of 50%). (2) The traditional training-based methods (91.1% accuracy) are still more promising under large distribution shifts if the privacy-utility trade-off is the only goal. However, given the benefit of API-only assumption and the non-trivial results that PE already got, it is worth further exploiting the potential of PE in future work.

Reviewer comment (Experiments):
**Evaluation**: More details on ensemble in the main paper is appreciated. Are the 5 classifiers trained on disjoint datasets each of 0.2M samples? Is this the same setup as Ghalebikesabi et al.? (because 0.2M samples for a CIFAR classifier seem a lot.)

Author response:
**A7.2:** We follow the same setup as Ghalebikesabi et al. (2023). For the right-most point in Figure 5 (1M samples), we train 5 classifiers with the same hyper-parameters on all 1M samples; the only difference between these 5 classifiers is the random seed. The ensemble of the classifier is implemented by ensembling the logits (Appendix I).

Reviewer comment (Experiments):
**Evaluation**: I appreciate the effort of the authors putting up their own private dataset for evaluation. However, I have reservations about the experimental setting in Sec 5.2. In its current form, Sec 5.2 only serves the visual demonstration purpose. But one dataset with datapoints contributed all by one single identity makes little sense when talking about DP. A dataset with e.g. 5 cats and 20 images each would be better. It would be possible to further evaluate the utility of the synthesized data then.

Author response:
**A7.3:** This is a great question! We also gave thoughtful consideration to this point when we did the experiment. Even though the training set of Stable Diffusion is public, it is hard to check if a public image or its variants (e.g., cropped, scaled) have been used to produce images in it. Therefore, we decided not to use public datasets, and we resorted to constructing our own private dataset so that we are 100% sure that the private data and the pre-training data have no overlap. We discussed the above considerations in footnote 6 in the original submission. 

To construct our private dataset, we did not come up with a better way than using the photos of the *only* two cats we have. That being said, we agree with the reviewer that having multiple samples from the same identity is not meaningful from the practical perspective of DP. Instead of regarding this dataset as a real-world use case, we would rather treat it as a “toy dataset”, the same as other widely used benchmarks in DP literature including CIFAR10 and ImageNet, where there are also duplicated and/or similar samples from the same identity. The experiment of PE on this cat dataset is not for showing a real use case but is rather a proof-of-concept for showing “the feasibility of applying PE on large foundation models with Stable Diffusion” (Section 5.2).

We added more clarifications to the above question in Section K. Thank you again for bringing it up!

Reviewer comment (Writing):
**Minor**: Grammar issue: Sec 4.1, "the (privatized) number of private numbers whose"

Author response:
**A8:** Thank you for catching these issues! We fixed them in the revision.

Reviewer comment (Presentation):
**Minor**: The position and the order of the footnotes are messed up.

Author response:
**A8:** Thank you for catching these issues! We fixed them in the revision.

Reviewer comment (Writing):
**Minor**: It is not appropriate to use "Cat Cookie" and "Cat Doudou" in a research paper. Use Cat A and Cat B instead.

Author response:
**A8:** Thank you for catching these issues! We fixed them in the revision.

Reviewer comment (Experiments):
Incomplete downstream classification accuracy v.s. privacy. Figure 5 compares the downstream classification accuracy of the proposed method versus another DP synthetic data generation approach. Can you compare the proposed method with SOTA DP training models on raw data to see which one is better for the downstream classification tasks?

Author response:
**A2:** The SOTA DP classifier pre-trained on ImageNet (without DP) and fine-tuned on CIFAR10 (with DP) [1] achieves 94.8% and 95.4% accuracies with epsilon=1 and 2 respectively. It is not surprising that DP classifiers outperform PE (and other DP synthetic data approaches as well) on classification tasks, as DP classifiers are targeted at and optimized for a single task whereas DP synthetic data is general-purpose. See similar discussions in Section 5.2 of the SOTA DP Diffusion work (Ghalebikesabi et al., 2023).

We added the numbers and discussions in Section 5.1.1 of the revision.  

[1] De, Soham, et al. "Unlocking high-accuracy differentially private image classification through scale." arXiv preprint arXiv:2204.13650 (2022).

Reviewer comment (Theory):
Overclaim. Section 4 starts by saying "Data type. While our framework above and our algorithms in § 4 are general for any data type, we focus on images in our experiments." However, it is not clear how this can be done for example for text because of the complexity of textual information and also for audio because of the lack of speech-based APIs. I would either remove this statement or try to discuss how this extension can be done.

Author response:
**A3:** Thank you for the question. We discussed briefly in the original submission how PE can be implemented for text, and we added initial results showing that PE indeed works for texts below. 

* **How PE can be implemented for text.** We discussed in Section 3.3: “In our algorithm design and experiments, we use 2 APIs, both of which are either directly provided in the APIs of popular models (e.g., DALLE 2, Stable Diffusion **or can be easily implemented by adapting current APIs (e.g., using appropriate text prompts in GPT APIs)**”. Indeed, the natural language interface of large language models (LLMs) gives us even more flexibility in implementing RANDOM_API and VARIATION_API (for texts) than the diffusion models (for images). Let’s take email datasets as an example. For RANDOM_API, we can use the prompt “Please generate a random email:” to query the LLM. For VARIATION_API, we can use the prompt “Please rephrase the following email: <the original email>” to query the LLM. Such an approach works for any LLMs that provide chat capability (e.g., GPTs, LLaMAs). The above prompts are just a simple example; given LLMs’ powerful instruction-following capability, more prompt engineering can potentially give better results.

* **Results of PE on texts.** We evaluate PE on text in our follow-up work. We compare PE with the recent ACL 2023 paper on DP synthetic text generation [1]. We follow their experimental setting:

  * We take GPT2, GPT2-Medium, and GPT2-Large as the pre-trained model (to implement RANDOM_API and VARIATION_API in PE, or to DP-fine-tune in [1]).
  * We consider the Yelp review dataset and the private data.
  * We consider rating classification and category classification as the two downstream tasks for evaluating the utility.
  * We set the DP parameter epsilon=4.

The following table shows the results. 



|                |             | Classification accuracy on rating | Classification accuracy on category |
|----------------|-------------|-----------------------------------|-------------------------------------|
| DP fine-tuning [1] | GPT2        | 66.56%                            | 74.78%                              |
| PE             | GPT2        | 65.62%                            | 74.75%                              |
| DP fine-tuning [1] | GPT2-Medium | 67.56%                            | 74.86%                              |
| PE             | GPT2-Medium | 66.85%                            | 75.01%                              |
| DP fine-tuning [1] | GPT2-Large  | 69.36%                            | 75.68%                              |
| PE             | GPT2-Large  | 67.49%                            | 74.64%                              |

As a reference, directly training a DP classifier on the original data achieves 70.14% accuracy on rating and 76.44% accuracy on category. We can see that PE achieves similar (and sometimes better) accuracies compared to the state-of-the-art DP fine-tuning approach, no matter whether the pre-trained model is GPT2, GPT-Medium, or GPT2-Large. 

Nevertheless, given that this submission only focuses on images, we removed the above claim in the revision following the reviewer’s suggestion.

[1] Yue, Xiang, et al. "Synthetic text generation with differential privacy: A simple and practical recipe." arXiv preprint arXiv:2210.14348 (2022).

Reviewer comment (Experiments):
As this paper says in Section 4 "Foundation models have a broad and general model of our world from their extensive training data." so how do you make sure the images that you consider as private (for example CIFAR-10 in your case) are not used in the training of foundation models of used APIs? I think this somehow affects the practicality of the proposed method in practice. After reading the experiment section, it seems this paper does not use real-world APIs and instead, this paper tries to simulate this in a controlled environment by considering pre-trained models.

Author response:
**A4:** It is a great point! Your understanding is correct. We added the following discussions to Section 7 of the revision.

As PE does **not** provide DP guarantees for the pre-training data of the foundation models, one should be careful that data considered private is not used in pre-training. Depending on whether the APIs are from *black box models*, which can only be accessed through APIs  (e.g., DALLE3), or *local models*, whose weights and architectures are accessible by the users (e.g., Stable Diffusion), it has different implications.
* *Using APIs from black box models.* Since most black box models do not reveal their training dataset, it is safer to only consider private data that was never been shared or posted online. This includes scenarios where hospitals generate synthetic medical images using their proprietary patient records.
* *Using APIs from local models.* For local models, we have full control over the model weights and architectures. We can pre-train the models on data that surely has no overlap with the private data. In all experiments of the paper, we use local models including Improved Diffusion and Stable Diffusion. We directly take the pre-trained models from prior work, and we make sure that the private data and the pre-training data have no overlap.

Reviewer comment (Experiments):
The claim of the proposed method being easier to deploy than existing methods is not supported. I would at least empirically validate this by measuring its costs in comparison to existing ones.

Author response:
**A5:** In Appendix O of the original submission and the revision, we provide experimental results showing that the **computational cost** of PE can be smaller than training-based methods under the same setting (e.g., using the same pre-trained model, generating the same number of generated samples). This means that if practitioners want to run the APIs locally (i.e., downloading the foundation models and running the APIs locally without using public API providers), PE provides computational benefits.

Reviewer comment (Evaluation):
It would be good to also talk about the limitations of DP synthetic data in general especially when data is distributed across users and cannot be centralised in one place.

Author response:
**A6:** Thanks for the great point. We agree that decentralized settings are more challenging to solve. In Section 6 of the original submission, we list it as one future work: “Solving DPSDA in the Local/Shuffle DP model and in federated learning settings.” We believe it is interesting to study how to extend PE to federated learning settings,
In fact, PE is quite amenable to adapting to decentralized settings. Suppose each client/user has one private sample for simplicity. If we share the current iteration of generated samples with each user (these are differentially private anyway), then the user can vote for the nearest sample. These votes can be aggregated by a central server either by employing Differentially Private Frequency Estimation under Local Differential Privacy (such as in [1]) or by using frequency estimation under the shuffle-DP model which further amplifies the privacy guarantees as shown in [2].


[1] Private Frequency Estimation via Projective Geometry. Feldman et al. ICML 2022. 

[2] Hiding Among the Clones: A Simple and Nearly Optimal Analysis of Privacy Amplification by Shuffling. Feldman et al. FOCS 2021.

Reviewer comment (Novelty):
The core DP component of the approach (i.e., adding DP noise to the histogram) is largely an existing idea/mechanism, which slightly diminishes the overall technical contribution of this submission.

Author response:
**Q1: The core DP component of the approach (i.e., adding DP noise to the histogram) is largely an existing idea/mechanism, which slightly diminishes the overall technical contribution of this submission.**

**A1:** We agree with the reviewer that “adding DP noise to the histogram” is an existing idea. But we also want to mention that it is only one step among the other components we proposed, and the way we construct the histogram (using private samples to vote for the nearest generated samples) is different from the usual application of this idea (e.g., DP density estimation where the histogram is constructed by binning the private samples). Overall, our contributions are not on “adding DP noise to the histogram”, but on the following aspects:
* **New problem:** We highlight the importance of studying DP Synthetic Data via foundation model APIs.
* **New framework:** We propose Private Evolution, a new framework for generating DP synthetic data, that only requires APIs of the foundation models and does not need any model training.
* **Promising results:** Private Evolution can even outperform the SOTA DP-fine-tuning-based approaches on the privacy-utility trade-off in the CIFAR10 experiments.

Reviewer comment (Experiments):
While the proposed approach demonstrates encouraging results on Camelyon17 dataset where the distribution shifts between the public and private distribution is relatively large, its performance may start to fall short compared to the more straightforward pre-training and fine-tuning paradigm (as the curves look not saturating and only one privacy level is presented). A more in-depth exploration of the limits (in terms of tolerance to distribution shifts) of the pure API method (potentially by considering other medical datasets) and a discussion on potential refinements or extensions of the proposed approach would further strengthen the submission.

Author response:
**Q2: While the proposed approach demonstrates encouraging results on Camelyon17 dataset where the distribution shifts between the public and private distribution is relatively large, its performance may start to fall short compared to the more straightforward pre-training and fine-tuning paradigm (as the curves look not saturating and only one privacy level is presented). A more in-depth exploration of the limits (in terms of tolerance to distribution shifts) of the pure API method (potentially by considering other medical datasets) and a discussion on potential refinements or extensions of the proposed approach would further strengthen the submission.**

**A2:** Following your suggestion, we conducted a new experiment showing how PE performs under different levels of distribution shifts. To do that, we take the Camelyon17 dataset and modify the *saturation* of the images to create a sequence of datasets, each with a different saturation change. This way, we create a sequence of datasets with different levels of distribution shifts from ImageNet. We decided to use this methodology because it is: (1) more controllable, as we can use saturation change to control the level of the distribution shifts, (2) more comparable, as these datasets are derived from the same base dataset, and (3) comprehensive, as we can show the performance under *multiple* levels of distribution shifts.

The results are shown in Figure 30 in the revision. We can see that among all levels of distribution shifts we test here, the FID scores decrease with more PE iterations. This means that PE is effective in pushing the generated distribution towards private data. At the time we post the rebuttal, the FID scores are still decreasing and have not converged yet; we will keep updating this figure as the experiment continues.

However, as we discussed in Section 5.1.2 and Section 6, larger distribution shifts do make the convergence slower and the results tend to be worse compared to the DP fine-tuning approaches. One possible improvement is to consider better variation degree ($v$ in Section 3.3) choices. Currently, we use a single variation degree in each step. As the theoretical analysis in Section D suggested, using multiple variation degrees (so that the variations can cover the entire space no matter whether the distribution shift is large or small) benefits convergence. It would be interesting to explore this direction in future work.

Reviewer comment (Evaluation):
I appreciate the qualitative results, such as the generated samples and their NN among real data. However, I would expect additional quantitative evaluation, perhaps in the form of histograms or average results, for the NN distances (both from real to closest generated and from generated to closest real). This may help provide critical insights into the fidelity of the generated samples, the approximated data likelihood, and the empirical privacy leakage.

Author response:
**Q4: I appreciate the qualitative results, such as the generated samples and their NN among real data. However, I would expect additional quantitative evaluation, perhaps in the form of histograms or average results, for the NN distances (both from real to closest generated and from generated to closest real). This may help provide critical insights into the fidelity of the generated samples, the approximated data likelihood, and the empirical privacy leakage.**

**A4:** Thanks for the great idea! Following your suggestions, we plotted the distributions of the distances between (1) generated samples and their nearest real samples and (2) real samples and their nearest generated samples, on both CIFAR10 (Figures 17, 18) and Camelyon17 (Figures 27, Figure 28), for generated samples from different PE iterations. We plot the distribution in CDFs instead of histograms, as CDFs are easier to read when plotting multiple distributions in one figure. 

Two key observations are: (1) During the early PE iterations, the inception distances tend to decrease. This means that PE is effective in pushing the generated distribution to be closer to the private distribution. (2) However, as PE continues, the inception distances stop decreasing. It is expected, as DP upper bounds the probability of reconstructing any sample in the private dataset. Please refer to the revision for more discussions.