Private Synthetic Text Generation with Diffusion Models (2025)

Abstract: How capable are diffusion models of generating synthetics texts? Recent research shows their strengths, with performance reaching that of auto-regressive LLMs. But are they also good in generating synthetic data if the training was under differential privacy? Here the evidence is missing, yet the promises from private image generation look strong. In this paper we address this open question by extensive experiments. At the same time, we critically assess (and reimplement) previous works on synthetic private text generation with LLMs and reveal some unmet assumptions that might have led to violating the differential privacy guarantees. Our results partly contradict previous non-private findings and show that fully open-source LLMs outperform diffusion models in the privacy regime. Our complete source codes, datasets, and experimental setup is publicly available to foster future research.

## introduction how can we share sensitive textual data and protect privacy of individuals in there at the same time? a go-to method to circumvent this issue is synthetic data generation, used mainly for tabular data @xcite , or images @xcite in the medical domain. however, synthetic data generation alone is not sufficient to protect privacy of the underlying data. for example, @xcite show that outliers in synthetic data suffer from membership inference attacks. achieving formal privacy guarantees for the underlying data is possible by combining a generative model with differential privacy @xcite ) where we trade off privacy for reduced utility of the synthetic data. @xcite and @xcite demonstrated feasibility of synthetic data generation with dp in nlp. however, both @xcite and @xcite violate several assumptions about the underlying data which casts doubts on the validity of their findings. recent advances also show the success of diffusion models in private synthetic data generation in images @xcite . despite recent achievements in conditional text generation using diffusion models @xcite @xcite , the capabilities of private synthetic text generation with diffusion models remain unexplored. we thus ask two research questions. first, what performance on downstream tasks can we achieve using diffusion models for synthetic text generation with varying strengths of differential privacy? second, which factors might have artificially inflated performance in previous works and can we mitigate invalid assumptions in empirical experiments? we address these questions as follows. our first hypothesis is that diffusion models for text generation might not suffer from noise added in dp, as they inherently work with a de-noising objective. we address this question empirically by conducting extensive (and expensive) experiments with three state-of-the-art diffusion models. we address the second research question by two arguments. our first argument is that previous experiments mostly focused on 'old' public datasets, such as the imdb movie reviews @xcite . as these datasets may very likely have been seen during pretraining of the utilized llms (gpt-2), the reported effectiveness of the privacy-preserving synthetic texts may be overestimated. we factor out the potential data leaking by introducing five new 'fresh' unseen datasets into the experiments. our second argument is the violation of dp by ignoring group privacy in the datasets. we provide evidence in §4. we strive for transparency, reproducibility, and accountability-three key ingredients of privacyrelated research. therefore we experiment with fully open-source and transparent models, such as bloom @xcite . all our source codes and datasets are also publicly available for further scrutiny. our main contributions are (1) evidence of severe underestimation of dp guarantees in previous works, (2) empirical evidence showing that, unlike in image domain, diffusion models for synthetic text generation suffer severely under dp training, and (3) complete re-implementations of unpublished previous works enabling transparency and potentially increasing trust in privacy-oriented research. as the theory of dp is considerably extensive, we refer to @xcite work as an easily accessible introduction for the interested reader. furthermore, @xcite present a tutorial about how dp can be harnessed for nlp applications, while @xcite provide a comprehensive review about the current state of the art in dp research. ## related work to enable the privacy-preserving sharing of labeled text data, @xcite propose a method called prompt-based dp fine-tuning, which they utilize to train a gpt2-large model @xcite . the process can be described as a 'reverse' classification task where instead of learning to predict the label for a text, the generative model is trained on generating texts suitable for a given label. the privacy of each author in the dataset is protected by applying dp-sgd to the fine-tuning procedure. additionally, a mismatch loss is applied during training by maximizing the negative loglikelihood of purposely mislabeled texts. @xcite experiment on two publicly available datasets to validate their approach. the reported results claim that text classifiers trained on the privatized, synthetic data and evaluated on original data experience no significant performance loss compared to a classifier directly trained on the original texts. yue et al. (2023) use the same methodology in their approach, apart from the mismatch loss. in contrast to @xcite , they include a private customer feedback dataset in their experiments, where the synthetic texts also proved to be useful for downstream task classification performance. furthermore, they empirically evaluate the privacy-preservation of the dp generation models by injecting 'canaries' into the training data, as suggested by @xcite . after those models generate a synthetic dataset, it is then possible to track if any canaries have been replicated, which was not the case for models trained with dp guarantees. although not in the text domain, diffusion models have also been explored for dp synthetic data generation. @xcite train diffusion models with dp-sgd on several lowresolution image datasets, such as the mnist dataset of handwritten digits @xcite and generate synthetic images for the downstream classification task. classifiers trained on the synthetic data reportedly reach performances close to the state of the art. notably, in most experiments, before the diffusion models were trained with dp-sgd on smaller datasets, they were pretrained without dp on the large-scale imagenet32 @xcite dataset. so far, pretraining large models on public data and fine-tuning those with dp-sgd on smaller, private datasets seems to be an efficient method to produce privacy-preserving and useful synthetic data. however, neither diffusion-lm, nor diffuseq, nor most other current diffusion model include pretraining in their methodology. one exception is genie, introduced by @xcite , which is the only available diffusion model for text-to-text generation that is pretrained. genie is pretrained on a large corpora of texts by using an objective similar to span-based masked language modeling, however, instead of predicting the correct text span, gaussian noise is continuously added to the selected token span, which genie learns to denoise. when fine-tuned, the model outperforms the base versions of t5 @xcite and bart @xcite in several natural language generation (nlg) tasks. ## critical analysis of existing works in the following, we explain in detail how dp synthetic text generation has been accomplished in prior work. we also critically assess their underlying assumptions and evaluate their respective validity in regards to privacy protection. prompting llms trained with dp-sgd is correct. both yue et al. @xcite and @xcite describe a scenario, where a data holder wishes to benefit from public research on their inhouse sensitive text resources (such as medical reports or customer data), but cannot release them due to privacy concerns. the first assumption is that the sensitive documents are labeled (categorized) and the task to be solved is classification. the authors address this problem as follows. first, they create prompts for each original text based on the category to which it belongs. second, a pretrained language model is fine-tuned on the prompt:text pairs (e.g., <write a positive review: original review @xmath0, learning to create texts resembling the data from the instructed category. since synthetic data generation alone is not enough to protect privacy (recall the discussion in section 1), both authors train their models with dp-sgd. afterwards, synthetic texts are sampled from the resulting models using the prompts formulated earlier. it is worth mentioning that the dp guarantee remains the same regardless of the amount of synthetic generated texts. the utility of the resulting samples is then evaluated on a downstream classification task, namely by fine-tuning bert on the generated texts and testing it on the original sensitive test data. multiple texts from the same person violates differential privacy. the main requirement in dp is that in the underlying dataset, there is a one-toone correspondence of a person and its data point. from the ml perspective, this means that each ex-ample for training or fine-tuning belongs exactly to a unique person. this allows us to spell out the necessary notion of neighboring datasets and the very guarantee of dp, such that the difference of a private analysis will be 'roughly the same' (governed by ε and δ) on two datasets of size n and n -1, respectively. this assumption is violated when a textual dataset contains multiple examples from the same author. due to unique writing style, vocabulary or other implicit features, those texts may correlate with each other even though they do not share the same explicit information. a workaround for this issue is group privacy (dwork and roth, 2014, theorem 2.2), which translates to the following: when assuming that each author provides at most k contributions to a dataset and texts of different authors do not correlate with each other, the privacy bounds of any (ε, δ)-dp mechanism increase to (kε, k exp((k -1)ε), δ)-dp. however, to the best of our knowledge, no works utilizing dp-sgd have ever used group privacy, and it is actually unclear whether the dp-sgd is compatible with it. 2 we offer additional perspectives on this matter in the "limitations and ethics statement" section below. our analysis of the datasets used in previous works reveals that there is a clear violation of this dp assumption. in the book reviews from the amazon multi domain data @xcite experiments, users 'shalom freedman', 'prairie pal' and 52 more contributed at least more than one review. similarly, the yelp open dataset foot_2 provides concrete proof in its documentation that some users have written more than one review. nonetheless, the dataset was part of the experiments carried out by @xcite . given these violated assumptions, we cannot really tell whether or not previous works truly guarantee the reported privacy strength. we suspect that under fair conditions the protection would be much lower. no evidence that the sensitive data were not part of lm pretraining. another assumption in @xcite and @xcite work is that the 'sensitive' data (in this case imdb, yelp, etc.) have not been 'seen' during pretraining of the generative language model. we identify two potential issues here. first, any data point accessed during llm pretraining can be potentially leaked by adversarial prompting @xcite . if the sensitive data were used both in (1) llm pretraining and in (2) private fine-tuning, privacy had been breached in (1) already and no claims about protection in (2) are valid. second, pre-training and fine-tuning on the same data will most likely boost the effectiveness of the synthesized texts, as opposed to synthesizing outof-domain 'fresh' data. this has been demonstrated in previous work by @xcite who found that such leaking led to unrealistically good results in other works. since yue et al. (2023) and mattern et al. (2022) use gpt2 @xcite in their method, and the pretraining data of that model is not disclosed, we doubt that the performance reported on the yelp and imdb dataset is fully transferable. hence, in our experiments we only include datasets that are guaranteed not to be part of the pretraining data. ## experiments we investigate the capabilities of diffusion models to generate synthetic texts under dp with the following experimental setup. ## discussion in contrast to our experiments, the samples provided in @xcite ) demonstrate that diffusion models are capable of generating semantically and syntactically correct texts. this may be mainly caused by the task differences between our works. @xcite albeit the experimental results of combining dp with text diffusion models are not promising, we still encourage the exploration of this method in future work. we believe that major improvements to this model class are possible, as demonstrated by @xcite , which drastically improved the training and inference speed of diffuseq. for privacy-preserving nlp research involving dp, we recommend to take section 4 into consideration for their experiments. future research should not fine-tune llms with dp-sgd without justifying how the main requirements of dp are met in their work. ## conclusion we explored the capability of three text-to-text diffusion models to generate private synthetic texts. we also revealed that reported dp guarantees are severely underestimated by previous works for synthetic text generation, due to omitting group privacy rules and potential leakage of the employed datasets into the pretraining data of the involved llms. while our experiments demonstrate that diffusion models do not exceed their baselines for private synthetic text generation, we are the first to train a text diffusion model with dp constraints. hopefully, this enables further exploration of privacy methods incorporating these type of models.