Just Fine-tune Twice: Selective Differential Privacy for Large Language Models (2022)

Abstract: Protecting large language models from privacy leakage is becoming increasingly crucial with their wide adoption in real-world products. Yet applying *differential privacy* (DP), a canonical notion with provable privacy guarantees for machine learning models, to those models remains challenging due to the trade-off between model utility and privacy loss. Utilizing the fact that sensitive information in language data tends to be sparse, Shi et al. (2021) formalized a DP notion extension called *Selective Differential Privacy* (SDP) to protect only the sensitive tokens defined by a policy function. However, their algorithm only works for RNN-based models. In this paper, we develop a novel framework, *Just Fine-tune Twice* (JFT), that achieves SDP for state-of-the-art large transformer-based models. Our method is easy to implement: it first fine-tunes the model with *redacted* in-domain data, and then fine-tunes it again with the *original* in-domain data using a private training mechanism. Furthermore, we study the scenario of imperfect implementation of policy functions that misses sensitive tokens and develop systematic methods to handle it. Experiments show that our method achieves strong utility compared to previous baselines. We also analyze the SDP privacy guarantee empirically with the canary insertion attack.

## introduction with the rapid advancement in natural language processing (nlp), it has become increasingly important to protect nlp models from leaking privacy information. previous work has attempted to tackle this challenge by applying differential privacy (dp, @xcite on these models @xcite . however, existing dp learning algorithms suffer from limited user control and low utility, as they protect the entirety of each training example (e.g., one complete sentence) regardless of users' privacy preference, and therefore tend to be overly pessimistic when only partial information in a training example is sensitive. this problem is particularly pertinent in nlp, as nlp training data are often mixed with sparse domain-dependent private information, and not all tokens need to be protected. for example, for the sentence "my ssn is 123-45-6789", only the last few tokens of the actual ssn need to be protected. in fact, the definition of dp does not prevent us at all from protecting only the sensitive part of data. specifically, dp ensures that the output of a data analysis algorithm stays roughly the same for neighboring datasets, while providing the flexibility to adjust the definition of neighboring relation to specific application contexts. @xcite recently proposed an instantiation of dp, called selective-dp (sdp), which defines neighboring datasets to differ only in the sensitive part of a training example and as a result, sdp selectively hides the difference in the sensitive part only. sdp is particularly suitable for nlp and many other unstructured, high-dimensional data, wherein sensitive information only accounts for a small part. but their privacy mechanism to achieve sdp suffers from three problems: 1) it requires substantial knowledge about the model to separate the private and public variables, and it is unclear how their algorithm tailored to recurrent neural networks could be extended to modern transformer-based nlp models; 2) it has only been evaluated with explicit private entities but not with contextual sensitive information; 3) it doesn't provide protection for undetected sensitive tokens; these constraints limit the applicability of sdp in real-world scenarios. large language models (llms) @xcite have achieved tremendous success in nlp. they are pretrained on a massive amount of public textual data, and thus excel at capturing general language structures. a common practice in nlp is to fine-tune these llms on downstream tasks. such a fine-tuning process also works well in the private training context. previously, @xcite showed that privately fine-tuning an additional small set of parameters on top of off-theshelf llms with private data achieves comparable performance to non-private baselines. inspired by their findings, in this paper, we propose a two-phase fine-tuning privacy mechanism, just fine-tune twice (jft), to achieve sdp for llms. instead of directly using off-the-shelf models to fine-tune once, we have two fine-tuning steps: 1) we first redact the in-domain data of the downstream tasks, and fine-tune the model with these in-domain redacted data (redacted-fine-tune), and 2) then privately finetune the model on the original private data (privatefine-tune). this additional redacted-fine-tune step allows the model to directly learn information from the in-domain data and thus leads to a better model initialization for the second private-fine-tune step. moreover, in the redacted-fine-tune step, we show that even with limited public data (where manual screening is possible), jft achieves better utility than fine-tune-once baselines. additionally, we can apply lightly-noised optimizers and privacy amplification to protect undetected sensitive tokens. our contributions are as follows. first, we propose an effective and generalizable privacy mechanism to achieve sdp for large language models for various nlp tasks. second, we design secret detectors of different privacy levels (explicit and contextual sensitive data) and study their implications on the models. third, our method can utilize even a small amount of public data to achieve better utility, and mitigate the missed sensitive token problem with lightly-noised optimizer and privacy amplification. finally, we show that, opposite to the common belief that privacy is at odds with utility, private learning doesn't have to conflict with the utility because private information in the data could be irrelevant to the learning task. ## preliminary a differentially private algorithm hides the difference between two neighboring datasets. ## jft: just fine-tune twice now we describe jft, a two-phase privacy mechanism to achieve sdp for large language models (figure 1 ). in the first redacted-fine-tune phase, we redact the private data d with a secret detector to obtain the redacted version d ′ , and learn a redacted model from d ′ in a privacy-preserving way. in the second private-fine-tune phase, we further fine-tune the redacted model (from phase one) on the private data d with a private optimizer to achieve sdp guarantees. ## secret detectors of different levels typical private information includes personalidentifiable information (pii) such as name and birthday. but as pointed out in @xcite , one key challenge in nlp is that private information is often contextual. for example, they presented a dialogue between alice and bob about alice's divorce ( to build generalizable secret detectors, we utilize off-the-shelf ner, dependency parser, and pos tagger in spacy @xcite to label each token, and redact different sets of tokens to achieve the different privacy levels below (entity level and contextual level). to qualitatively show their protection levels, we apply them to redact two sentences from the divorce dialogue in @xcite . the results are in the two secret detectors above rely on named entities, so they are more explicit than the two de-tectors below, which consider the overall sentence structure and thus are more contextual. low contextual protects all the 18 entities plus proper nouns, pronouns, and sentence subjects and objects. this detector drastically increases the privacy level: we cannot get any useful information from the left example in human language is diverse, and private information can take various forms. so instead of designing sophisticated algorithms, we intentionally rely on common nlp tools to build easy-to-use domain-agnostic secret detectors with high recalls to protect privacy as much as possible. as shown in ## experiments we conduct experiments on two nlp tasks: 1) natural language understanding (nlu, on glue) and 2) language generation (on wikitext-2 and abcd). datasets. 1) glue @xcite is a widelyused multi-task benchmark dataset for nlu. it contains sensitive information such as name and date. 2) wikitext-2 @xcite contains wikipedia articles with private information such as name and date. 3) abcd @xcite is a human-human customer service dialogue dataset under real-world scenarios with user private information such as name and order ids. models. we use roberta @xcite for the nlu classification task and gpt2 @xcite for the language generation task. due to computational constraints, we use roberta-base and gpt2-small for the experiments. we use an effi-cient implementation of dpsgd in @xcite . based on previous studies @xcite , larger dp models usually achieve better results and thus we expect that larger sdp models will achieve even better performances. baselines. 1) no-dp: the model is fine-tuned using regular adam optimizer (kingma and ba, 2014) without extra noise and hence it does not have any privacy guarantees (i.e., ϵ @xmath1. ## results we show three major findings: 1) the impacts of secret detectors are task-dependent on the resulting jft models, but even for conservative contextual detectors (30%+ tokens are redacted), jft still achieves better results than naive dpsgd (section 6.1); 2) despite the small scale, using the manual screened in-domain data still improves the jft model utility (section 6.2); 3) lightly noised optimizer with privacy amplification protects missed sensitive tokens from attacks (section 6.3). there is always a privacy-utility trade-off, so larger epsilons lead to better utilities but worse privacy. and when comparing models, we need to look at the model utility under a similar privacy budget. an epsilon of 1 to 3 is commonly used in various privacy literature @xcite @xcite . in our experiments, we pre-calculated the privacy parameter so that an ϵ of around 3 is spent when training ends. ## related work recent work studied private language models on various model architectures such as rnns @xcite and large language models @xcite @xcite . @xcite proposed ghostclipping to reduce the computational cost in persample gradients of dpsgd, and achieved strong private llms. @xcite added a small set of private parameters to off-the-shelf llms and privately tune them on private data and obtained performant private models. most previous works achieve canonical dp. @xcite proposed selective-dp for applications with sparse sensitive information like nlp, and a privacy mechanism for rnn models. our work proposes an effective mechanism for llms to achieve sdp and study the impact of secret detectors at different levels. our work is also closely related to utilizing public data for private learning @xcite @xcite . one working direction assumes access to large unlabeled public data to train dp models. for example, pate @xcite used unlabeled public data and knowledge distillation to build dp models. @xcite used public medical data to pre-train domain-specific private vocabulary and models. another direction leveraged small public data to guide the private updates in lower-dimension subspaces @xcite . our work is distinct from previous studies: instead of querying public data from outside, we utilize the public portion of in-domain data and achieve sdp with better model utility. ## conclusions in this paper, we propose jft, which can achieve selective-dp for large language models. we also design generalizable secret detectors to provide protection at different levels and study their impacts on the resulting sdp models, and address the problem of missed sensitive tokens via selective manual screening and private training with reduced noise, which is justified by privacy amplification. the results show that the proposed jft produces sdp models with strong performance while remaining robust to the canary insertion attack.