Differentially Private Next-Token Prediction of Large Language Models (2024)

Abstract: Ensuring the privacy of Large Language Models (LLMs) is becoming increasingly important. The most widely adopted technique to accomplish this is DP-SGD, which trains a model to guarantee Differential Privacy (DP). However, DP-SGD overestimates an adversary’s capabilities in having white box access to the model and, as a result, causes longer training times and larger memory usage than SGD. On the other hand, commercial LLM deployments are predominantly cloud-based; hence, adversarial access to LLMs is black-box. Motivated by these observations, we present Private Mixing of Ensemble Distributions (PMixED): a private prediction protocol for next-token prediction that utilizes the inherent stochasticity of next-token sampling and a public model to achieve Differential Privacy. We formalize this by introducing RD-mollifers which project each of the model’s output distribution from an ensemble of fine-tuned LLMs onto a set around a public LLM’s output distribution, then average the projected distributions and sample from it. Unlike DP-SGD which needs to consider the model architecture during training, PMixED is model agnostic, which makes PMixED a very appealing solution for current deployments. Our results show that PMixED achieves a stronger privacy guarantee than sample-level privacy and outperforms DP-SGD for privacy \epsilon = 8 on large-scale datasets. Thus, PMixED offers a practical alternative to DP training methods for achieving strong generative utility without compromising privacy.

## introduction large language models (llms) are being deployed to improve societal productivity, from troubleshooting complex systems to autocompletion tools and interactive chatbots. their commercial success is largely attributed to their ability to generate humanlike text. however, when given query access to an llm, it has been shown that llms are susceptible to training data extraction attacks @xcite due to memorization of training samples @xcite . these security vulnerabilities have recently catalyzed government intervention, most notably the eu's ai act @xcite and the us executive order on safe, secure, and trustworthy ai (us, 2023). thus, it is becoming a requirement that entities deploying llms must do so in a privacy-preserving way. the gold standard for achieving strong privacy is differential privacy (dp), a mathematical framework that reduces how much an llm memorizes individual data samples @xcite . the most widely known technique injects strategic noise into the training algorithm, called dp-sgd @xcite . it has recently been shown that applying dp-sgd during fine-tuning on pre-trained llms provides acceptable results @xcite . unfortunately, scaling these results to larger datasets and models becomes challenging since (1) the magnitude of the noise added by dp-sgd scales with the total number of parameters of the model, i.e., √ d @xcite , in the worst case; and (2) ml accelerated hardware is designed to exploit batch operations but is underutilized by dp-sgd since it requires per-sample gradient calculations, which cause large runtime and memory consumption @xcite . a key observation that motivates this work is that many commercial deployments of llms are only accessible via an api, e.g. chat-gpt. hence, an adversary has only black-box access to the model, yet dp-sgd assumes the adversary has white-box access to the model since it applies differential privacy to the model parameters. such a pessimistic assumption on adversarial capabilities can lead to overestimating the privacy loss bounds @xcite . we believe that improving the privacy of llms under the black box assumption is the key to enabling wider adoption of privacy-preserving llms. however, prior work has demonstrated that private prediction algorithms generally perform worse than private training algorithms (van der @xcite . to address the aforementioned limitations, we propose pmixed, depicted in figure 1 . rather than providing dp during training, pmixed instead provides dp for a private corpus during inference by utilizing two crucial features: (1) randomness comes for free when predicting the next-token by sampling from the output probability distribution of a language model; (2) we can bound the privacy leakage of a prediction using a public model 1 to mix the predictions of privacy-sensitive llms. we formalize these two observations by introducing (α, β)-rd mollifiers which generalize ϵ-mollifers introduced in @xcite and could be of interest independent of this work. pmixed can be broken down into two phases: training and private prediction. during phase-i: training, pmixed follows the sample-andaggregate paradigm @xcite by partitioning a private corpus into n pairwise disjoint subsets, then fine-tuning a pre-trained llm on each subset to produce an ensemble. using an ensemble is crucial for guaranteeing privacy, but also has been shown to boost perplexity @xcite . during phase-ii: private prediction, pmixed selects a subset of the ensemble using 1 while it is assumed that a public llm does not compromise the privacy of a private corpus, in practice, such models can inadvertently leak information. this assumption is implicitly relied upon in works that employ differentially private fine-tuning of public models. poisson subsampling. each selected model produces its output probability distribution, then follows the rd-mollification process by projecting its output distribution onto a renyi divergence ball centered at a public model's output distribution. lastly, pmixed averages these optimal projections and then samples from it. because pmixed does not employ differentially private training algorithms, it does not incur substantial training overhead like dp-sgd. furthermore, we reduce computational and storage costs by employing parameter-efficient fine-tuning methods. in summary, our key contributions are the following: (1) we introduce and formalize (α, β)-rd mollifiers, a generalization of ϵ-mollifers, to avoid additive dp noise. ## related works most of the previous differential privacy work in llms has focused on private training. @xcite first explored private training of language models using a small recurrent neural network to achieve user-level differential privacy in the federated learning setting. recent breakthroughs in differentially private llms involve self-supervised pre-training on public data, followed by privately fine-tuning on a private corpus @xcite . an orthogonal approach, but conceptually similar to ours, is pate @xcite . pate also uses an ensemble of models trained on pairwise disjoint subsets of a private dataset to generate dp labels. however, pate and pmixed rely on substantially different private aggregation schemes. pate uses the gaussian/laplacian mechanism to perturb its output vote count histogram while our method utilizes the inherent stochastic nature of sampling from a probability distribution and a public model in llms. furthermore, pate does not satisfy the dp prediction definition @xcite since its data-dependent privacy accounting causes its privacy loss to be a function of the private data. hence, testing that the datadependent loss does not exceed the privacy budget will leak additional privacy @xcite . there is a much smaller body of work that has focused on private prediction for generative language models, mainly due to prior work empirically showing that private prediction methods perform worse than private training (van der maaten and hannun, 2020). private prediction can be broadly categorized into two methods: prediction sensitivity, which adds noise to the output distribution of an llm; and sample-and-aggregate, which involves the same process as pate for generating noisy labels. for prediction sensitivity, @xcite used the uniform distribution as their perturbation mechanism and showed that it satisfies ϵ-dp. however, the privacy loss needs to be large, ϵ ≈ 60, in order to be practical. one work closely related to pmixed, which motivated our work, also falls under the sampleand-aggregate method as ours @xcite . however, their ensemble could exceed the privacy budget before completing all t queries due to their data-dependent privacy accounting. hence, they had to define a new privacy notion to account for this. our work does not have this limitation. ## pmixed: a protocol for private next token prediction we now introduce and describe pmixed, a private prediction protocol for next token prediction. first, we will introduce a concept called renyi divergence (rd) mollifiers, which formalize the projection of a private distribution onto a set around a public distribution, and help us prove the privacy guarantees of pmixed. next, we will discuss the fine-tuning process of the ensemble, which follows the sample-and-aggregate paradigm. lastly, we will describe the prediction protocol and show its privacy guarantees. ## privacy analysis we begin our privacy analysis of pmixed by first considering the case of no poisson subsampling. then we invoke the privacy amplification theorem c.4 to derive our final privacy guarantees. lastly, we will discuss the implications of our privacy guarantees. let p denote our private prediction protocol, pmixed. note that d and d ′ are neighboring datasets if d ′ adds or removes a subset d i from d, which is equivalent to adding or removing the model p i from the ensemble. ## experiments 6.1 experimental setup we experimentally evaluated the privacy-utility tradeoff of pmixed by using lora @xcite to fine-tune pre-trained gpt-2 models (radford et al., 2019) from huggingface @xcite on the wikitext-103 @xcite and one billion word @xcite each other, in that they give a good comprehensive evaluation of differentially private next token prediction. wikitext-103 tests the ability of long-term dependency modeling, while one billion word mainly tests the ability to model only short-term dependency @xcite . the public model in our experiments is a pretrained gpt-2 small model. we compare pmixed to three baselines: the public model, a non-private fine-tuned model, and a private fine-tuned model produced by dp-sgd. although dp-sgd has a weaker privacy level, we compared pmixed to per-sample dp-sgd as our baseline because it illustrates how well pmixed performs against the most widely-used dp solution. the non-private and private fine-tuned baselines also used lora. lora based fine-tuning can outperform full private fine-tuning since the lora updates a much smaller set of parameters during fine-tuning @xcite . the lora parameters used for each model p i contain 0.11% of the total number of parameters of the pre-trained model, allowing us to fit the entire ensemble into one gpu during prediction. appendix a contains more details about fine-tuning. for private prediction, unless stated otherwise, the parameters used are set to the default values shown in ## conclusion and future directions pmixed is inspired by the observation that most llm deployments are cloud-based where an adversary only has black-box access to the model, rather than access to the entire model parameters. under this setting, pmixed presents a novel private prediction protocol that provides dp during prediction, rather than during training. thus pmixed is model agnostic, which has significant practical implications since it avoids the complexity of privately training models with millions or even billions of parameters. our approach relies on an ensemble of fine-tuned llms and the novel idea of (α, β)-rd mollifiers, which generalizes ϵ-mollifers, to project each of the model's output distribution onto a set around a public llm's output distribution. pmixed is compatible with various privacy granularities, such as group-level dp, and is at least stronger than sample-level dp. furthermore, pmixed does not require per-sample gradients and can operate on batch-level data, significantly reducing the training overhead compared to dp-sgd. we experimentally showed that pmixed outperforms dp-sgd in terms of utility on large-scale datasets using llms. thus, pmixed substantially progresses private prediction methods in llms and offers a practical alternative to dp training methods. although our experiments only used a pretrained gpt-2 small as the public model, in general, we can plug and play any public model for inference, highlighting the versatility of our approach. for example, we could use a gpt-2 model that is already fine-tuned on a public corpus from a different domain, or we could use a larger pre-trained model like gpt-2 xl. as a result, we can further boost the performance of our approach without additional training costs. we leave this exploration as a future work.