Paper: Harnessing large-language models to generate private synthetic text (ICLR-2024)

Reviewer comment (Novelty):
The main weakness of the paper is that although it follows a similar direction to the prior work (Bommasani et al., 2019; Yue et al., 2022; Putta et al., 2023; Mattern et al., 2022), the authors committed an injustice in their presentation and comparison with the prior work. These prior works present impressive results that show the effectiveness of generating high-fidelity synthetic text datasets with DP. The authors shockingly present as if the reverse is true that the prior work failed to obtain good fidelity synthetic data, which is unacceptable.

Author response:
We wrote a detailed explanation of our comparison to (Yue at al) and other prior work into a shared reply to all reviewers. We also updated section 2 of our paper to be more clear and only focus on the most important points.

We acknowledge that all of prior work presented important building blocks necessary for DP-synthetic data. However, we wanted to emphasize that none of the prior work takes into account dataset contamination between LLM pre-training dataset and dataset used for synthetic data generation. As we show in appendix D this problem is real and both training and test examples from downstream task datasets are potentially present in GPT2 pre-training data.

Reviewer comment (Experiments):
For the downstream tasks, the authors choose the binary classification problem (4-way for AGNews?), which does not let the utility of the approach to be seen for more challenging scenarios.

Author response:
We would like to point out that we considered both binary and multi-class classification problem. Specifically downstream task on AGNews dataset is 4-class classification. This is generally in line with all prior work which consider either binary classification downstream problems or few class (4..5 classes) classification problem.

Reviewer comment (Novelty):
The difference from prior work (Putta et al., 2023) and (Mattern et al., 2022) appears to be that the authors do not augment the training objective, and the difference from prior work (Yue et al., 2022) is merely applying parameter efficient fine-tuning instead of full fine-tuning. Looking at the results of Table 1, there is only a 2-3% difference in performance between real and non-private (private) synthetic data, which is similar to prior work results—despite the much larger 8B model used here compared to the GPT2 series in prior works, and the fact that prior works also considered multiclass classification. The reviewer does not see the claimed contradiction with prior work and questions statements such as "Firstly, our results in Table 1 indicate that obtaining good fidelity non-private synthetic data is possible, contrary to the results reported in (Yue et al., 2022) and (Putta et al. (2023)." Putta et al. (2023) reports 91.3 for non-private synthetic data compared to 93.7 for real data, which the authors omit mentioning. Similar misrepresentation is suspected for results from Mattern et al. (2022) and Yue et al. (2022). The reviewer therefore questions how the authors can state "Previous approaches either show significant performance loss, or have, as we show, critical design flaws" given that their approach and results are similar.

Author response:
We wrote a detailed explanation of our comparison to (Yue at al) and other prior work into a shared reply to all reviewers. We also updated section 2 of our paper to be more clear and only focus on the most important points.

We acknowledge that all of prior work presented important building blocks necessary for DP-synthetic data. However, we wanted to emphasize that none of the prior work takes into account dataset contamination between LLM pre-training dataset and dataset used for synthetic data generation. As we show in appendix D this problem is real and both training and test examples from downstream task datasets are potentially present in GPT2 pre-training data.

Reviewer comment (Experiments):
The reviewer asks how the hyperparameter search was performed for the comparison between full fine-tuning and parameter efficient fine-tuning. Were the same comprehensive hyperparameter search protocols applied to both approaches? The reviewer doubts the claim of significant superiority of parameter efficient fine-tuning, given that prior work (Li et al., 2021) demonstrates that both methods are competitive when proper hyperparameter search is performed.

Author response:
While conducting hyperparameter sweep we followed the best practices reported in various prior work (Li et al., 2021, and https://arxiv.org/abs/2303.00654) for all settings (i.e. full-finetuning, prompt-tuning and Lora). Specifically, we conduct a sweep over both learning rate and clipping norm. Due to resource constraints, we typically tune the clipping norm only for a single value of epsilon and re-used it for others. Additionally, for some of the full finetuning experiments we have to reduce granularity of learning rate sweep, i.e. (1e-5, 1e-4, 1e-3, …) for full finetuning compared to (1e-5, 3e-5, 1e-4, 3e-4, …) for parameter efficient tuning.
Nevertheless, in an attempt to get better results with full finetuning, we trained LLM for 40 epochs, while for Lora and prompt tuning we found it is sufficient to train LLM for only 10 epochs. Even with longer training, full finetuning could not match parameter efficient tuning.

Reviewer comment (Novelty):
The paper requires significant revision regarding how prior work is treated. Although the reviewer acknowledges the paper’s contributions—such as attention to the pre-training phase, using a larger 8B model, and parameter efficient training along with promising results—proper justice must be done in their presentation and comparison with prior work.

Author response:
We wrote a detailed explanation of our comparison to (Yue at al) and other prior work into a shared reply to all reviewers. We also updated section 2 of our paper to be more clear and only focus on the most important points.

We acknowledge that all of prior work presented important building blocks necessary for DP-synthetic data. However, we wanted to emphasize that none of the prior work takes into account dataset contamination between LLM pre-training dataset and dataset used for synthetic data generation. As we show in appendix D this problem is real and both training and test examples from downstream task datasets are potentially present in GPT2 pre-training data.

Reviewer comment (Experiments):
Minor comment – "For each dataset we formulated a binary classification problem (sentiment classification) as the downstream prediction task." The reviewer supposes for AGNews this must not be the case because the dataset is for news topic classification, which has 4 labels.

Author response:
We would like to point out that we considered both binary and multi-class classification problem. Specifically downstream task on AGNews dataset is 4-class classification. This is generally in line with all prior work which consider either binary classification downstream problems or few class (4..5 classes) classification problem.

Reviewer comment (Novelty):
One weakness coming to my mind is the lack of novelty as there is not really anything new proposed in the paper. All the experiments are results of combining existing techniques. At the same time, I do think these are really valuable experimental results and the lack of theoretical novelties is not necessarily a problem.

Author response:
We would like to thank you for your work and detailed comments. Below we provide answers to your questions and concerns.

**Novelty**

We put a detailed description of the novelty of our work into a shared reply to all reviewers.

Reviewer comment (Experiments):
I would have liked to see more about the computational costs of the experiments. I see there are some compute cost numbers in Appendix M, but would have been interesting too more detailed analysis. I find it interesting that fine-tuning the 20-40k parameter prompt tensor can lead to such impressive results, better than fine-tuning the full model and almost the same as fine-tuning the 20M LoRa parameters. Would have also been interesting to see about the memory requirements.

Author response:
We would like to thank you for your work and detailed comments. Below we provide answers to your questions and concerns.

**Computational cost**

Computational cost of LoRa finetuning is roughly 2x compared to prompt tuning in our experiments. Thus LoRa is still much faster than full finetuning. We added these details to appendix M.

Reviewer comment (Reproducibility):
You write in Appendix M: "Diferentially private finetuning of 8B model required between 20 hours (for shortest run on IMDB) and up to 80 hours for some of the longer runs. On the other hand prompt tuning required 1.5 hour for short run and up to 20 hours for longest runs." Does this mean that LoRa fine-tuning had the same cost as the full fine-tuning of the model? I cannot see it said explicitly anywhere, what is the cost of DP-fine tuning the 20M parameter LoRa part of the model. I find it interesting that DP fine-tuning the 20-40k parameter prompt tensor gives so good results. What makes the cost then relatively high anyways, the forward pass through the 8B parameter pre-trained model? Is there any rule of thumb, how much does each part cost?

Author response:
We would like to thank you for your work and detailed comments. Below we provide answers to your questions and concerns.

**Computational cost**

Computational cost of LoRa finetuning is roughly 2x compared to prompt tuning in our experiments. Thus LoRa is still much faster than full finetuning. We added these details to appendix M.

Reviewer comment (Evaluation):
What kind of differences in the memory requirements are there between the different alternatives?

Author response:
We would like to thank you for your work and detailed comments. Below we provide answers to your questions and concerns.

**Memory requirements**

We have not done a detailed analysis of accelerator memory requirements. However we observed that for full finetuning and LoRa we can fit a similar batch size into accelerator memory (32 examples), while we can fit about 256 examples for prompt tuning. This suggests that finetuning and LoRa should have similar memory requirements and prompt tuning have lower memory requirements.

Note that in all setups our effective batch size is 1024 or more examples per step, thus we have to use gradient accumulation technique to be able to train with such batch size. We split the entire batch into smaller chunks of either 32 or 256 examples, sequentially compute gradients for each chunk and then aggregate to compute final gradients for the entire batch.
Even taking into account the difference in chunk size, we still observe that LoRa is only about 2x slower compared to prompt tuning when using effective batch size of 1024 and the same number of training epochs.

Reviewer comment (Novelty):
The discussion and comparison with Yue et al. (2022) might be confusing to the readers. It would be helpful if the authors could clarify the difference between 'conditioning on some features' and 'augmenting the fine-tuning process.' mentioned in the Section 2 related work. In Yue et al. (2022), labels are used in the prompt for conditional generation, with labels considered as non-private. Therefore, there seems to be no “augmentation” during the fine-tuning process. This approach seems to be the same as the proposed method in section 4.1 of this paper, where the authors also use label names in the prefix as condition generation.

Author response:
**Comparison to Yue at al**

We wrote a detailed explanation of our comparison to (Yue at al) and other prior work into a shared reply to all reviewers. We also updated section 2 of our paper to be more clear and only focus on the most important points.

We acknowledge that all of prior work presented important building blocks necessary for DP-synthetic data. However, we wanted to emphasize that none of the prior work takes into account dataset contamination between LLM pre-training dataset and dataset used for synthetic data generation. As we show in appendix D this problem is real and both training and test examples from downstream task datasets are potentially present in GPT2 pre-training data.

Reviewer comment (Experiments):
While the authors acknowledge the presence of IMDB in Pile and perform deduplication accordingly, it might be more beneficial to directly use a dataset from an unseen domain, such as a medical dataset.

Author response:
**Dataset choice**

We agree that the ideal scenario would be to separately gather a dataset for downstream task from an unseen domain. However we would argue that this is difficult to achieve in practice if we consider datasets which are non-proprietary and easily available to researchers for experimentation.
Most of the text datasets available to researchers are essentially scraping text from the internet. ThePile contains parts of common crawl, C4 is crawled from the internet, WebText and OpenWebText are constructed by scraping various web-sites. So all of these pre-training datasets would likely contain all or some portions of public data we treat as "private" for our experiments.

Reviewer comment (Experiments):
“We used the suffix arrays to find common sequences of 50 or more tokens which appear in The Pile” — A justification for the choice of the hyperparameter value of 50 in the use of suffix arrays would be appreciated.

Author response:
**De-duplication - why we used 50 tokens**

We have chosen 50 tokens following the suggestion of https://arxiv.org/abs/2107.06499. The general idea is that sequences which are too short likely correspond to some common words or phrases. Note that this approach removes any common sequence 50 tokens or longer. Also note that for our training process, we use instances of up to 512 token long - so we remove the pretraining data that even PARTIALLY matches our "private" instances (that is why we also mention that it is a stronger deduplication than simply removing exact matches/exact instances).

Reviewer comment (Experiments):
Could the authors elucidate why this de-duplication is “stronger” than simply removing the datasets from the Pile?

Author response:
**De-duplication - why not simply remove the dataset from The Pile**

We wanted to clarify that there is no simple way of removing IMDB, Yelp, AGNews from The Pile (or any other large text dataset). Running de-duplication code is essentially our way of removing these three datasets from The Pile.

Let’s consider IMDB as an example. IMDB dataset is obtained by scraping and parsing movie reviews from imdb.com web-site. The Pile or OpenWebText do not directly include examples of IMDB dataset. Instead they include content of web-pages from imdb.com which contain the same movie review as IMDB dataset. So what typically happens is that a content of the web-page with multiple movie reviews is included into The Pile or OpenWebText. At the same time some of the reviews from this web-page are also included into the IMDB dataset in a slightly more sanitized manner. Note that The Pile does not provide annotations of URLs of dataset examples, thus we can not search for all “imdb.com” domains and remove them from The Pile.
In this case we need to search for common substrings between The Pile and IMDB to find and remove corresponding reviews. This is exactly what our deduplication algorithm does - search for all common substrings.

Reviewer comment (Experiments):
All downstream tasks in the study are binary sentiment classification tasks, which may appear monotonous and simplistic. The utility of synthetic data for more complex downstream tasks, such as multi-class classification or classification tasks beyond sentiment, as considered in previous studies, remains unclear.

Author response:
**Downstream Tasks**

We wanted to clarify that the downstream task for AGNews is a multi-class classification task. All of the prior work on synthetic text also operates on binary or few class classification tasks only. At the same time it is a valid question whether downstream models for much harder tasks (e.g. summarization) will demonstrate similar performance on DP synthetic data vs DP on real data directly.

Reviewer comment (Evaluation):
The interpretation of results under different metrics in Table 3 could be more clearly explained, particularly for readers who may not be familiar with RBO, Spearman, and Kendall metrics. For example, how good or bad is the value 0.56 under RBO 25% metric for BERT trained on real data with $\epsilon=\infty$?

Author response:
**Presentation and Interpretation of Results**

W.r.t to metrics in Table 3 and the interpretation of the results, as with many metrics in ML/stats, these metrics don't have absolute "good" threshold. E.g. for tuning some of the models, if synthetic data exhibits correlation of 70% it might be good enough, for others it might be insufficient to obtain the best possible model. Also the row "real data" with eps inf can serve as a good comparison point.  
However these metrics can be used in relative terms for comparison of different types of synthetic data. For example in Table 3 we can clearly see (as expected)  that tuning on real data is the best (column Mean 25% accuracy) however it does not exhibit perfect correlation with downstream performance (spearman of 0.96 or RBO of 0.56, stil  achieving the best test accuracy on top 25% trials of 93.55)  We will expand on this discussion in the text, space permitting.

Reviewer comment (Writing):
In section 6, there is a missing space after the comma: “monitoring,and sharing,” –> “monitoring, and sharing,”

Author response:
**Typos**

Thanks for indicating the typos. We will update the text accordingly.

Reviewer comment (Experiments):
The data de-duplication only checks the suffix of the samples. If I understand it correctly, it does not detect duplications that happen in the middle of the samples?

Author response:
Our deduplication method finds all common substrings between two datasets, including substrings in the middle of text.
We use the same approach as  https://arxiv.org/abs/2107.06499 and adapted their code https://github.com/google-research/deduplicate-text-datasets 
The deduplication algorithm is based on a data structure, which is called “suffix array”, see https://en.wikipedia.org/wiki/Suffix_array . This data structure helps to facilitate efficient search for duplicates in a large dataset, but the approach finds all duplicates not just suffixes.

Reviewer comment (Experiments):
The paper shows that the classifiers trained on DP synthetic data can have even better downstream classification accuracy than the classifier trained on real data with DP. As explained in Section 5.1, it is because "the private synthetic data beneﬁts from massive amount of public data that was used for pretraining of the LLM". While it is a nice result to have, it would also be informative to show results when both classifiers are exposed to the same amount of public information (e.g., by pre-training the BERT model on the same public data used for pre-training the LLM, and then fine-tuning the BERT model with either synthetic data or private data (with DP) as the classifier). This way, we can isolate the effect of public information and understand the true gap between training downstream classifiers with DP synthetic data and training downstream classifiers on real data with DP directly.

Author response:
When both the classifier that generates the DP synthetic data and the final downstream classifier is of the same capacity and uses the same amount of public information, we believe that DP on real will be an upper bound on performance that any DP synthetic data with the same eps can achieve. For this use case, DP synthetic data might be not the best approach (if performance maximization is the ultimate goal).

However, among other benefits of synthetic data we mentioned in the paper (e.g. data sharing, eval and hyperparameter tuning), consider a slightly different use case when one is to train several downstream classifiers (using the same private data) and reveal all of them to the end user. In this case, the budget for DP on real should be split between all classifiers, while DP synthetic data will be created once and can be reused for all models. In this case training multiple models on DP synthetic data may perform better than training several models with DP on real data directly.

Reviewer comment (Presentation):
There are some missing numbers in Table 1. Why is that?

Author response:
The two remaining experiments on AGNews were not complete by the time of ICLR submission due to high cost of full finetuning and limited computational resources.
At that time we decided to omit them since we didn’t expect them to be particularly good and LoRa results would be better.

Currently we re-ran all three DP-finetuning experiments on AGNews using a smaller computational budget (train for 10 epochs instead of original 40 epochs) and update the numbers in the paper. Note that the accuracy of eps=1 full finetuning on AGNews also changed, since this experiment was re-run with a smaller number of training epochs.

Reviewer comment (Experiments):
It would also be useful to show sample length distribution as in Yue et al.

Author response:
We added it to appendix L, which talks about evaluation of synthetic data quality. We also added comparison of bi-gram distributions.