Private prediction for large-scale synthetic text generation (2024)

Abstract: We present an approach for generating differentially private synthetic text using large language models (LLMs), via private prediction. In the private prediction framework, we only require the output synthetic data to satisfy differential privacy guarantees. This is in contrast to approaches that train a generative model on potentially sensitive user-supplied source data and seek to ensure the model itself is safe to release.We prompt a pretrained LLM with source data, but ensure that next-token predictions are made with differential privacy guarantees. Previous work in this paradigm reported generating a small number of examples ( < 10) at reasonable privacy levels, an amount of data that is useful only for downstream in-context learning or prompting. In contrast, we make changes that allow us to generate thousands of high-quality synthetic data points, greatly expanding the set of potential applications. Our improvements come from an improved privacy analysis and a better private selection mechanism, which makes use of the equivalence between the softmax layer for sampling tokens in LLMs and the exponential mechanism. Furthermore, we introduce a novel use of public predictions via the sparse vector technique, in which we do not pay privacy costs for tokens that are predictable without sensitive data; we find this to be particularly effective for structured data.

## introduction differentially private mechanisms process a source dataset potentially containing sensitive user information and perform a useful computation -as simple as calculating a mean, or as complex as training an ml model -whose output can be safely shared while protecting the privacy of users who contributed to the dataset. perhaps the most general-purpose differentially private mechanism is one that produces a synthetic version of its input dataset, as the output of such a mechanism would be suitable for all the same purposes as the original dataset. for example, a private synthetic dataset can be used to train an ml model, but can also be used for auxiliary tasks such as feature engineering, hyperparameter tuning, and quality monitoring. there has been recent interest in using largelanguage models (llms) to generate differentially private versions of text datasets. existing approaches can be classified into several categories. private fine-tuning methods privately adjust the parameters of an llm on the input dataset, using an algorithm such as differentially private stochastic gradient descent (dp-sgd), and then prompt the llm to generate similar text. fine-tuning methods have been used to produce high-quality synthetic data, but the training procedure can be prohibitive, available only to those with the time, compute, and access necessary to train state-of-the-art llms containing billions of parameters. private prediction methods do not modify the llm parameters at all. instead, they directly prompt the llm with text from the source dataset, asking for similar text in response, and then perturb the llm's token distribution (i.e., its last layer) to ensure that each sampled token, and thus the entire generated response, is private. since no training is required, private prediction methods can quickly generate synthetic data, typically producing some data within minutes instead of hours, which allows for rapid prototyping and iteration. however, unlike private fine-tuning, the guarantees of private prediction methods degrade with the volume of data that is generated. consequently, existing private prediction methods have mostly been used in applications that require only small amounts of synthetic data @xcite , sharply limiting their practicality. is z public similar to z 1 , z 2 , ..., z |s| ? a sample token from public logits. yes b dp sample token from sensitive logits. @xmath0, z 2 , ..., z |s| ) go to and repeat. figure 1: algorithm 1, visualized. an llm receives a batch of prompts, each instructing to generate text similar to a piece of sensitive text. synthetic text is generated token by token, by running inference on the batch in parallel. in each step, the logit vectors produced downstream of sensitive text are aggregated and sampled from with differential privacy. every token sampled in such way incurs a privacy cost, motivating us to include an auxillary public prompt and sample from its logits when similar to the sensitive logits. ## related work private fine-tuning is widely used for synthetic text generation. @xcite created private synthetic versions of text datasets by using dp-sgd @xcite to fine-tune an llm on the sensitive data. @xcite showed that parameter efficient approaches to fine-tuning, such as lora @xcite can improve the quality of the synthetic data. @xcite took a two-stage approach: first they fine-tuned an llm on a public dataset that closely resembled the sensitive data (which was itself generated by an llm using carefully designed prompts); then they completed the fine-tuning process by running dp-sgd on the sensitive dataset. concurrent to the present work, tran and xiong (2024) describe a private fine-tuning approach for generating synthetic tabular data that is formatting compliant. private prediction @xcite ) is an alternate approach to private machine learning that only guarantees the privacy of the predictions output by an ml model, and not the model itself. the predominant way this is realized is via subsample-and-aggregate @xcite : first sensitive data is split into disjoint partitions; then non-private predictions are made from each partition and privately aggregated. pate @xcite employs this approach to get answers to a limited set of image classification queries, which are then used to train a student model that can be queried indefinitely. private prediction has been applied to synthetic text generation by viewing each token sampled by an llm as a 'prediction', and perturbing the llm's token distributions to ensure their privacy. @xcite added noise to several independent token distributions and averaged them, while @xcite selected the most popular token among the token distributions using the limited-domain mechanism @xcite . these methods can avoid the time, compute, and access required to fine-tune an llm with billions of parameters. however, a privacy loss is suffered for each token produced in this manner. as a result, previous work has only been able to generate a very small number of synthetic examples at reasonable privacy levels (fewer than 10). other work has applied private prediction techniques to llms @xcite , including in combination with fine-tuning @xcite , but not for the purpose of synthetic text generation. finally, another distinct set of approaches are private filtering methods. private filtering methods operate directly on whole llm responses and a large corpus of public data that does not require protection. @xcite and @xcite used the sensitive responses to privately select similar responses from the public dataset. similarly, @xcite aggregate response embeddings and select the public response that is closest in embedding space. 1 one limitation of filtering methods is that the menu of possible responses is constructed without signal from the new source dataset. ## privacy analysis in this section we state formally how algorithm 1 preserves the privacy of the sensitive prompts it uses to generate synthetic examples. let d be the set of all possible prompt datasets. a mechanism is a randomized function with domain d. note that algorithm 1 is a mechanism. we say that a pair of prompt datasets d, @xmath1. in the differential privacy literature this is commonly referred to as the add/remove neighbor relation. definition 1 @xcite ). a mechanism m satisfies (ε, δ)-differential privacy if pr[m (d) ∈ o] ≤ e ε pr[m (d ) ∈ o] + δ for any neighboring datasets d, @xmath2. theorem 1 below provides a differential privacy guarantee for algorithm 1. the proof of theorem 1 requires a technical assumption about how the prompts are partitioned into batches in the first step of the algorithm. assumption 1. in algorithm 1, the assignment of a prompt to a batch depends only on the prompt itself, and not on the other prompts. the most straightforward way to satisfy assumption 1 is to apply a hash function to each prompt and then use the hash value to determine its assigned batch. for example, if h is the hash value, n is the number of prompts and s is the expected batch size, then we can assign the prompt to the (h mod n s )th batch. if we want to batch together prompts that share a certain attribute (like a label), we can apply another hash function to that attribute and concatenate the hash values. using hash functions for batch assignment can lead to batches whose sizes differ from the expected batch size s, but this does not impact the validity of theorem 1. theorem 1 (privacy of algorithm 1). suppose assumption 1 holds. let ρ = r 1 2 c sτ foot_1 + 2 (sσ) 2 . for all ε ≥ 0, algorithm 1 satisfies (ε, δ)differential privacy, where the proof is in §c and makes use of sharp privacy analyses of: (1) zcdp to approximate dp conversion @xcite ; and (2) zcdp bounds for the exponential mechanism @xcite . in-context learning ( §5.1). we generate examples to use as in-context exemplars for prompting an llm. we report downstream accuracy on real test examples, when prompted with synthetic data, on 3 classification tasks (agnews @xcite , dbpedia @xcite , trec @xcite ) and 2 extraction tasks (mit-g, mit-d @xcite ). fine-tuning ( §5.2). we generate synthetic examples to use for fine-tuning a bert classifier. we report downstream accuracy on real test examples for 3 classification tasks (imdb @xcite , yelp @xcite , agnews @xcite ). ## discussion we believe that our significantly improved performance relative to @xcite is primarily attributable to two algorithmic innovations. first, for each generated token, tang et al. (2024) preserve the privacy of the entire distributions from which the token is sampled (by taking argmax), even though only the token itself is included in the synthetic data. by contrast, our method uses a discrete choosing mechanism, the exponential mechanism. as a result, we do not need to maintain a dp version of the entire token distribution to release a single token. this decision leads to significantly lower noise requirements, as a straightforward calculation reveals. empirically, we obtained good synthetic data quality with @xmath3, τ = 2, @xmath4. in order to switch to the gaussian mechanism using its standard (ε, δ)-dp guarantee, and achieve comparable privacy guarantees we would would require σ ≈ 0.53 to achieve a comparable privacy guarantee. (see §d). better analyses of the gaussian mechanism exist, but do not offer much help. using the improved analysis in @xcite to attain the same ε would require σ ≈ 0.34. conducting the analysis so that both mechanisms have equivalent privacy loss under zcdp yields σ = 0.2. these are all very large noise magnitudes relative to probabilities in [0, 1]. 5 secondly, @xcite generated each token using a different random sample of the sensitive prompts, which is computationally very expensive, as it prevents the use of kv cache-accelerated decoding, since the cache is invalidated upon every resample. while resampling less often would be more practical, @xcite noted that in this case the privacy amplification benefits of subsampling would not be adequately realized, and characterized this limitation as the "main weakness" of their approach. instead, our method generates each synthetic example using a fixed disjoint subset of the sensitive prompts, allowing us to leverage parallel composition in our analysis, and thus avoid this privacy versus computation tradeoff. ## conclusion as proprietary models become increasingly powerful, we anticipate more practitioners will be able to generate inferences from state-of-the-art models, while fewer practitioners will be able to train networks that perform like state-of-the-art models. this makes it increasingly important to develop private prediction methods that compete with private fine-tuning. we demonstrate that private prediction can be used to generate large amounts of synthetic text with reasonable differential privacy guarantees. we produce 2-3 orders of magnitude more private synthetic data than what was demonstrated in prior work in this paradigm. access to more synthetic data lets us fine-tune downstream models, as well as yields performance improvements via many-shot in-context learning. furthermore, we introduce a novel use of public models in which we are able to sample predictable tokens at no privacy cost, which is particularly effective for structured data.