Differentially Private Language Models Benefit from Public Pre-training (2020)

Abstract: Language modeling is a keystone task in natural language processing. When training a language model on sensitive information, differential privacy (DP) allows us to quantify the degree to which our private data is protected. However, training algorithms which enforce differential privacy often lead to degradation in model quality. We study the feasibility of learning a language model which is simultaneously high-quality and privacy preserving by tuning a public base model on a private corpus. We find that DP fine-tuning boosts the performance of language models in the private domain, making the training of such models possible.

## introduction language modeling, the task of assigning a probability to sequences of words, is a key problem in natural language processing. modern language models are data-driven, relying on a large corpus of text. many such models are trained on corpora from a specific domain, such as wikipedia or news articles @xcite . these models often suffer from generalization issues when used to model language from a different domain. this motivates the use of model fine-tuning, in which the weights of a pre-trained language model are tuned by gradient descent on a second dataset of interest @xcite @xcite . in some cases, we would like to fine-tune our model with respect to a dataset containing private information. as such, there is an obligation to preserve the privacy of individuals who contribute text to the private training corpus. for example, training a medical chat-bot may require learning a language model from transcribed patient-doctor conversations; it would be critical that this model not expose sensitive information about the patients whose conversations are used as training data. in recent years, differential privacy (dp) has been a key quantitative measure of privacy which allows one to use aggregate statistical information about a dataset while preserving the privacy of its individual datapoints. in the case of language modeling, we are interested in preserving the privacy of individuals who contribute text to a private corpus. as each individual who contributes to this dataset could potentially contribute several sentences, our notion of privacy is group differential privacy @xcite , in which all sentences from a single individual are grouped. in practice, group dp is equivalent to dp with re-scaled parameters.a potential limitation of this approach is that the number of contributed sentences may not be uniform over users, leading to sub-optimal bounds on the privacy guarantee. there has been some success in directly training differentially private language models, but these often require access to large datasets in order to achieve a reasonable level of quality @xcite . other work has trained a differentially private base model which was then fine-tuned through active learning on a non-private dataset @xcite . we instead train a non-private base model on a large, public dataset, which we proceed to finetune on a private out-of-distribution dataset through differentially private stochastic gradient descent (dpsgd) @xcite . by doing so, we successfully train a high-quality model which is differentially private with respect to our tuning dataset. our experimental results show that dp fine-tuning not only boosts the performance of dp language modeling, but makes it possible. ## related work training a feedforward neural network with dp is achievable through the popular dp-sgd algorithm @xcite . however, this method may lead to significant decreases in the accuracy (or other metrics) of the resulting model. recent work considers the use of metric privacy for language modeling @xcite , which is a relaxation of differential privacy where noise is instead added to the vector embedding of a word. we leave the exploration of metric privacy for the private fine-tuning task as a direction for future work. many high-quality language models rely on some form of recurrent neural architecture, such as rnns or lstms @xcite . @xcite , the authors develop a method for training such models while achieving differential privacy. however, this approach requires a large private dataset, and the mechanisms to achieve privacy lead to a significant decrease in model quality. in @xcite , the authors attempt to train a language model which is simultaneously differentially private and of high quality. the first solution proposed in @xcite is to finetune the language model with publicly available data, but as this public data is likely distributed differently than the private data, the resulting model is likely mistuned. the second proposed approach is to augment the training data by actively selecting non-private data instances. this effectively reduces the privacy cost incurred during each training step, but still requires training with potentially out-ofdistribution data. in contrast, our work begins with a pre-trained model which only has access to publicly available data. this base model is then fine-tuned through dpsgd on our private domain of interest, resulting in a model that is both differentially private and tuned with respect to our protected dataset. by tuning a pre-trained public model, we achieve higher quality models without incurring any additional costs to our privacy budget. ## approach let d be a publicly available corpus, and p be a protected corpus whose contents we would like to protect the privacy of. denote by x the fixed, shared vocabulary of these corpora. at a high level, our approach is to first train a language model m d : x @xmath0, 1]. in practice, we choose a feedforward architecture for m d due to limited computing resources. we fine-tune this model with respect to p by using the dpsgd algorithm @xcite on batches of sentences from p. ## analysis finetuning improves dp perplexity we summarize the perplexities of our final small and large models in ## conclusions training neural models with differential privacy often significantly degrades model performance. however, differential privacy could prove crucial when doing language modeling on private datasets. our work shows that dp fine-tuning not only boosts the performance of dp language modeling, but makes it possible. we also compared our experiments across two different model sizes and found that increasing the model size while decreasing the number of training epochs does not significantly impact the results in the differentially private transfer learning scenario. future research could experiment with stronger model architectures (e.g., lstm's, transformers) instead of regular feedforward neural networks, as well as train models longer in order to increase performance.