Selective Differential Privacy for Language Modeling (2022)

Abstract: With the increasing applications of language models, it has become crucial to protect these models from leaking private information. Previous work has attempted to tackle this challenge by training RNN-based language models with differential privacy guarantees. However, applying classical differential privacy to language models leads to poor model performance as the underlying privacy notion is over-pessimistic and provides undifferentiated protection for all tokens in the data. Given that the private information in natural language is sparse (for example, the bulk of an email might not carry personally identifiable information), we propose a new privacy notion, selective differential privacy, to provide rigorous privacy guarantees on the sensitive portion of the data to improve model utility. To realize such a new notion, we develop a corresponding privacy mechanism, Selective-DPSGD, for RNN-based language models. Besides language modeling, we also apply the method to a more concrete application – dialog systems. Experiments on both language modeling and dialog system building show that the proposed privacy-preserving mechanism achieves better utilities while remaining safe under various privacy attacks compared to the baselines. The data and code are released at https://github.com/wyshi/lm_privacy to facilitate future research.

## introduction language models have been widely used in various kinds of applications, such as google smart compose, amazon alexa, and so on. however, these models are often trained on highly sensitive data such as emails and chat logs, while having the tendency to memorize the training data unintentionally @xcite . therefore, how to protect user privacy while preserving the model utility has become an increasingly important topic. several methods have been developed to protect the data, such as data anonymization, k-anonymity, figure 1 : "data anonymization" and training with "no dp" cannot provide knobs to adjust the trade-off between privacy and utility. selective-dp improves the privacy-utility trade-off of traditional dp, to get closer to the ideal model with both high privacy and high utility. and differential privacy (dp). among them, dp has become a dominant privacy notion as it provides formal and provable guarantees for people to understand the privacy-utility trade-off (figure 1 ). it works by carefully randomizing the algorithm so that the model does not rely too much on any single data point. however, traditional dp notion protects each data point as a whole regardless of the property of individual attributes inside the data @xcite . training large models with this overly pessimistic privacy notion could lead to poor model performance or even a non-convergent training process @xcite . our work is inspired by an important observation that in many scenarios including language modeling, private information is sparse, and not all attributes need to be protected. for example, for the sentence "my ssn is 123-45-6789", only the last token with the actual ssn number needs to be protected. but if we protect the entire sentence, we may fail to learn the underlying language pattern well. to solve this problem, we propose a new dp notion, namely selective differential privacy (s-dp), to provide focused protection for sensitive attributes in a training record and improve model utility. we follow the traditional dp setup, where each training record is contributed by a different user. the key difference is that we consider only partial dimensions of a training record as sensitive, which better abstracts the language data privacy problems. whether a given dimension in a record is sensitive is specified by a user-defined policy function that encodes application-specific privacy regulations, which gives users the freedom to protect any type of sensitive information according to the use cases. we also develop a corresponding privacy mechanism, selective-dpsgd, for rnn-based language models under the new notion. moreover, to process the variable-length sequences, we propose a batching method to group private and public tokens together and alternatively update them for a more efficient implementation of selective-dpsgd. one important concept to note is that, as shown in figure 1 , there is always a trade-off between privacy and utility: data anonymization achieves high privacy guarantee at the cost of low utility on private tokens; models trained without dp have high utility but low privacy. however, for both data anonymization and "no dp", there is no way to tune the trade-off, while dp-related methods provide knobs to adjust the privacy-utility trade-off. this paper proposes s-dp to improve the trade-off of canonical dp to get closer to the ideal model with both high-utility and high-privacy. we evaluate s-dp on two tasks, 1) a language generation task and 2) a more concrete application of dialog systems. besides reporting the model utility and theoretical privacy guarantees, we also empirically demonstrate their robustness under popular privacy attacks on language data @xcite . the experiments suggest that training with selective-dpsgd improves the model utility while remaining safe to the attacks. our contributions are as follows. first, we propose a new selective differential privacy notion that ensures targeted privacy protection for sensitive attributes, and a corresponding mechanism to realize the new s-dp notion for rnn-based models. second, we propose a dialog dataset for future privacy research. next, we show both theoretically and practically that our models are safe to attacks with improved utilities on both the language generation task and the dialog system application. moreoever, we discuss the case of imperfect policy func-tion, and compare s-dp with data anonymization to show that s-dp achieves better utilities when the policy function is imperfect. we also show preliminary results on contextual policy functions and transformer models. in the era of information explosion and large-scale pretraining, protecting data privacy becomes more and more important. with s-dp, we march one step closer towards more privacy-preserving and high-performing language models and hope to inspire more research in this direction in the nlp community. moreover, despite our focus on language-related applications in this paper, the proposed s-dp notion could be useful for a much broader range of applications where only partial data require privacy protection. ## related work language modeling is a key research problem in nlp. however, although language models are often trained on sensitive data such as emails, most related studies focus on improving model without considering privacy, leaving the models vulnerable to attacks. for example, @xcite showed that it is possible to infer a secret in the training data by attacking published language models. therefore, it is of great importance to introduce privacy protection mechanisms to the nlp community and train the models in a much safer way. differential privacy (dp) @xcite ) has been applied to various domains @xcite . @xcite introduced dpsgd to train dp-protected deep-learning models. pate @xcite leveraged knowledge distillation to train differentially private models. but dp-protected algorithms suffer from low utility, so the dp notion often needs to be adjusted according to the applications: @xcite proposed a personalized dp notion to provide different levels of protection for different users; @xcite developed one-sided dp to protect sensitive users only. we also propose a new selective-dp notion to protect only the sensitive attributes in a record to improve utility. recently, dp has also been applied to nlp tasks @xcite @xcite . for example, mcmahan et al. (2018) proposed dp-fedavg and dp-fedsgd to train rnn language models with user-level privacy guarantees. adelani et al. (2020) developed a probabilistic text deidentification algorithm with formal privacy guar-antees. different from existing work that directly applied dpsgd and provided undifferentiated protection for all training examples, we propose a new privacy notion and a corresponding mechanism to protect the sensitive portion of the data in centralized learning. such a new notion can be easily adapted to federated learning settings as well. there is also a line of work that attempts to improve differentially private deep learning performance via modifying training process @xcite @xcite or model architecture @xcite . our work is complementary to this line of work as we propose a new privacy notion. particularly, our work can be combined with the aforementioned methods to further improve model utility. ## backgrounds we will introduce language modeling and differential privacy as preliminaries in this section. language modeling. consider a text sequence that consists of multiple tokens, i.e., @xmath0, x 2 , . . . , x n ), where x i is the i-th token. the goal of language modeling is to learn the probability of the sequence p(x), which can be factorized with the chain rule as in equation (1). given a corpus @xmath1, . . . , x |d| }, we train a neural network (e.g., rnn) parameterized by θ to learn p(x) by minimizing the negative log-likelihood over d with the loss function in equation (2). differential privacy (dp) @xcite . a differentially private algorithm ensures that its output cannot help much to distinguish whether an individual record is contained in the input dataset. in other words, dp hides the presence of individual records. the formal definition is as follows. definition 1. (differential privacy). given a domain d, any two datasets d, d ′ ⊆ d that differ in exactly one record are called neighboring datasets. a randomized algorithm m : @xmath2, δ)differential private if for all neighboring datasets d and d ′ and all t ⊆ r, ## selective differential privacy canonical dp notion treats all records as sensitive. prior work has studied variants of dp notions, such as personalized dp @xcite and onesided dp @xcite , to exploit different privacy levels between records. however, existing privacy notions do not allow different attributes in a given record to have different privacy levels, which could otherwise potentially enable additional utility gains, especially for nlp tasks where private attributes are sparse. hence, we propose a new privacy notion-selective differential privacy-to distinguish between private and non-private attributes inside one data point with a policy function and protect sensitive part of one data point. ## experiments we conduct our experiments on two datasets: 1) a traditional text corpus for language modeling, and 2) a dialog dataset for a more concrete application of dialog systems. below are the dataset details. wikitext-2. to minimize real-world harm, we choose the already-public wikitext-2 @xcite . it contains articles from wikipedia with potentially sensitive information, and is a classical dataset for language modeling. for simplicity, we treat all the digits as privacy information. so the policy function f is a digit detector: if the token is a digit, f will output 0, otherwise, 1. customersim. with the emergence of virtual assistants, more and more private information is being exchanged during daily interactions. so we algorithm 1 selective-dpsgd 1: input: dataset d with n examples, policy function f , privacy bit matrix @xmath3, max sequence length k, loss function l(θ). parameters: learning rate η, noise multiplier σ, gradient norm bound c, group size l. 2: for @xmath4,2,... do 3: take a random batch b of max sequence length k, with sampling probability l/n 4: using p, split b into a sequence of non-private and private tuples {(bnp,i, bp,i)} 5: l, @xmath5,i, h) 9: l, @xmath6,i, h) 12: calculate sample gradient 13: note that although we use digits and names as running examples for sensitive information, s-dp can protect any sensitive attributes specified by the policy function. building better policy functions is orthogonal to s-dp, and thus beyond the scope of this paper. any improvements on policy functions are compatible with s-dp to achieve better results. model training details. we use one-layer lstms with an embedding size of 200 and a hidden size of 200, and a bpe tokenizer @xcite to avoid information leakage from the tokenizer: with bpe, a secret "1234" will be split into multiple tokens, e.g., "12" and "34", while traditional tokenizer will release "1234" in the dictionary. all private states (hidden, cell states) in the lstms are protected. baselines. we have two baselines, one without dp ("no-dp"), and the other trained with dpsgd ("dpsgd"). we refer to our models trained with s-dpsgd as "s-dpsgd". "no-dp" is simply an lstm optimized with a regular sgd and a starting learning rate (lr) of 20. the learning rate was annealed and decreased as training proceeded. "dpsgd" is optimized with dpsgd and a starting learning rate of 0.05. all the models are trained five times to reduce randomness, and the parameters are tuned on the validation set. we compare with dpsgd because it's the backbone of most of existing dp learning algorithms. existing modifications of dpsgd are mainly focused on optimization algorithms and objectives, thus are compatible with our work that tailors the privacy notions to realistic privacy needs in the nlp context. ## preliminary results on contextual policy function and transformer in this section, we present the preliminary results on different policy functions and large transformer models @xcite on wikitext-2. ## conclusions to conclude, we develop a new privacy notion, selective differential privacy (s-dp), to improve model utility while providing rigorous privacy guarantees for the sensitive portion of the data. we also develop a privacy mechanism, selective-dpsgd, to achieve the new s-dp notion for rnns. we experiment with wikitext-2 and a synthetic customer service dialog dataset. results on both tasks show that models trained with s-dp achieve better utilities than traditional dp, and are more robust under various attacks than models without dp protection. with s-dp, we march one step closer towards safer and better language models and hope to inspire more related research. moreover, s-dp could be applied to domains beyond nlp where only partial data require protection, such as image recognition. please see section a.1 for ethical consideration.