Adaptive Differential Privacy for Language Model Training (2022)

Abstract: Although differential privacy (DP) can protect language models from leaking privacy, its indiscriminative protection on all data points reduces its practical utility. Previous works improve DP training by discriminating privacy and non-privacy data. But these works rely on datasets with prior privacy information, which is not available in real-world scenarios. In this paper, we propose an Adaptive Differential Privacy (ADP) framework for language modeling without resorting to prior privacy information. We estimate the probability that a linguistic item contains privacy based on a language model. We further propose a new Adam algorithm that adjusts the degree of differential privacy noise injected to the language model according to the estimated privacy probabilities. Experiments demonstrate that our ADP improves differentially private language modeling to achieve good protection from canary attackers.

## introduction language modeling is a foundation problem in natural language processing @xcite . recent large language models @xcite are usually trained at scale. unfortunately, large language models have a tendency to remember training data in the absence of appropriate privacy protection mechanisms @xcite . since data, which are usually collected from public sources, e.g., tweets, blogs, may contain sensitive information (personal address, ssn numbers, and so on) learning a safe large language model has become increasingly important. in recent years, differential privacy @xcite has become a key privacy preservation method, which attempts to ran-domize the training algorithm so that the model does not rely too much on any single training instances. @xcite propose differential private stochastic gradient descent (dp-sgd) to protect deep learning models by adding random noise to gradients. however, traditional differential privacy ignores individual attributes of data @xcite . this overly pessimistic privacy protection results in poor performance or even mis-convergence of training for differentially private language models @xcite . therefore, approaches are proposed to mitigate this problem by treating private and non-private data separately during the dp training process, such as selective differential privacy @xcite and sensory-based privacy-χ @xcite . these methods require training data to provide privacy information as a hard label. unfortunately, it is usually difficult and expensive to manually annotate privacy labels to data. other studies @xcite learn to detect privacy information in unstructured texts. however, the prerequisite is knowing keywords or reference texts of privacy information @xcite . therefore, learning differentially private language models on data without prior privacy information is an open problem yet to be investigated. in this paper, we propose an adaptive differential privacy (adp) framework without resorting to prior privacy information. the basic assumption behind adp is that linguistic items containing private information do not occur frequently in realworld texts. hence, the probability that a linguistic item contains privacy information (hereinafter privacy probability) is inversely proportional to the frequency of the linguistic item occurring in the dataset. with this assumption, we can estimate the privacy probability of a linguistic item based on a language model. after estimating these probabilities, we relax the constraint of differential privacy, and propose an adaptive differential privacy method, which adjusts the guassian noise of differential privacy based on privacy probabilities. to enable this adaptive differential privacy strategy, we further present adaptive-dp-adam algorithm to train differentially private language models. to evaluate our approach, we train transformerbased language models, and compare the performance of adaptive differential privacy against traditional differential privacy methods. additionally, we verify the protection effectiveness of adp models with canary attackers @xcite . the results suggest that our adaptive differential privacy method can achieve good performance and protection from canary attackers. the main contributions of this paper are threefold. ## related work large language models @xcite have been attracting growing attention. powerful large language models can achieve substantial improvements on a wide range of downstream nlp tasks. unfortunately, large language models have a tendency to memorize training data @xcite . @xcite have successfully induced gpt-2 @xcite to output sensitive information in its training data. differential privacy @xcite is widely used to protect private information of data. @xcite propose the dp-sgd algorithm to train deep learning models, and apply moment accounting to calculate cumulative privacy loss during training. although dp-sgd can limit the risk of leaking information from training data, random noise on gradients usually degrades corresponding models @xcite , and even cause training to not converge when a large model is trained. to improve dp-sgd, one way is to change training settings @xcite , e.g., increasing the batch size or decreasing clipping norm. however, these methods are usually at a higher cost. other attempts to improve the utilization of dataset information by relaxing the constraints of differential privacy. for example, @xcite propose personalized differentiated privacy to provide different levels of privacy protection for different users. @xcite develop one-sided differential privacy that only protects sensitive users. @xcite introduce selective differential privacy to add noise only into private data. these methods all need to know which items in the dataset contain private information, which is prohibitively expensive for large-scale datasets. there are some previous works @xcite detecting sensitive information in unstructured texts, but relying on labeled keywords or reference texts. ## preliminary we will introduce differential privacy @xcite , and the dp-sgd algorithm @xcite as preliminaries in this section. ## adaptive differential privacy in this section, we will elaborate the proposed adaptive differential privacy. first, we introduce a method to evaluate the privacy probability of a linguistic item. second, we propose an adaptive noise method, which adjusts the noise magnitude according to the privacy probability of an item in dp-sgd process. finally, an adam gradient optimization algorithm based on adaptive noise is proposed. ## conclusion we have presented a new method to estimate the privacy probability of a linguistic item when the privacy information of the dataset is not known. with estimated privacy probabilities, we propose adaptive differential privacy (adp), to improve the model utility. we also present a privacy optimization algorithm, adaptive-dp-adam, to train differentially private models. our experiments show that models trained with adp achieve better utilities than traditional dp and are capable of protecting sensitive information from being leaked.