Differentially Private Language Models for Secure Data Sharing (2022)

Abstract: To protect the privacy of individuals whose data is being shared, it is of high importance to develop methods allowing researchers and companies to release textual data while providing formal privacy guarantees to its originators. In the field of NLP, substantial efforts have been directed at building mechanisms following the framework of local differential privacy, thereby anonymizing individual text samples before releasing them. In practice, these approaches are often dissatisfying in terms of the quality of their output language due to the strong noise required for local differential privacy. In this paper, we approach the problem at hand using global differential privacy, particularly by training a generative language model in a differentially private manner and consequently sampling data from it. Using natural language prompts and a new prompt-mismatch loss, we are able to create highly accurate and fluent textual datasets taking on specific desired attributes such as sentiment or topic and resembling statistical properties of the training data. We perform thorough experiments indicating that our synthetic datasets do not leak information from our original data and are of high language quality and highly suitable for training models for further analysis on real-world data. Notably, we also demonstrate that training classifiers on private synthetic data outperforms directly training classifiers with DP-SGD.

## introduction rapid advancements in the field of deep learning and natural language processing (nlp) have enabled companies, public institutions and researchers to extract information and gain knowledge from large-scale data generated by individuals. in many cases, it is desirable to share such data figure 1 : main idea of our paper: to share potentially sensitive datasets with third parties, we train a language model (lm) on the sensitive data in a differentially private manner and consequently prompt the lm to generate synthetic samples with privacy guarantees. with third parties, for example when analyses are performed by external consultants or in order to provide high quality benchmarks for the research community. this, however, entails a variety of risks related to privacy that cannot merely be solved by pseudonymization: a variety of deanonymization attacks enable the re-identification of individuals from tabular data such as movie ratings @xcite , geolocation data @xcite and notably also text @xcite @xcite . it is therefore highly desirable to develop anonymization mechanisms enabling secure data sharing, ideally with mathematical privacy guarantees as granted by differential privacy (dp) @xcite . existing approaches anonymize every text sample individually by obtaining differentially private vector representations @xcite or using sequence-to-sequence approaches that rewrite a given sample to eliminate user-revealing information @xcite @xcite , thereby following local differential privacy. as pointed out by @xcite , local dp requires a very high degree of noise which often leads to incoherent language and only little semantic overlap. the strict requirements of local dp are, however, not necessary if we assume that an entity aiming to share data already has access to the full collection of user-written texts and only wants to release an anonymized version of it. in this paper, inspired by recent advances demonstrating the feasibility of training large language models (llms) in a differentially private manner @xcite , we propose a globally differentially private data release mechanism relying on the generation of a "twin" dataset of the original, sensitive user data from large language models. as depicted in figure 1 , we train gpt-2 @xcite to generate texts of our original dataset based on prompts inferred from the sample's individual attributes such as sentiment or topic. for fine-tuning, we use a differentially private optimization algorithm in order to protect the content of our training data. subsequently, we sample from the trained model to generate a large number of synthetic, anonymous texts, resulting in a verifiably private "twin" dataset. we carefully evaluate our proposed method using popular nlp datasets such as imdb movie reviews or amazon product reviews. here, we find that even after learning with strong privacy guarantees such as ϵ = 3 or ϵ = 8 from only a very limited amount of training samples such as 25 or 50, our generated data is of high quality and the classifiers trained on it achieve accuracies only ∼3% lower than those trained on the full original dataset containing thousands of samples. notably, we also find that transformer based classification models trained on private data outperform models trained on real data with differentially private optimization. finally, we show that the differentially private fine-tuning procedure effectively minimizes the risk of data leakage from language models that was previously discovered by @xcite . ## approach we consider the following scenario to motivate our approach: an entity wants to implement nlp pipelines to gain insights from internal data, e.g., emails from customers. to seek advice and get support for modeling the data and building pipelines, the entity aims to share an excerpt of the internal data with a third party such as a consultant or a group of researchers. in order to do this without compromising the privacy of its customers, the aim is to synthesize a verifiably private "toy" dataset that reflects the properties of the original data without leaking private information. on such a toy dataset, a third party could research how to best solve the task at hand and train a model to perform inference on the actual internal data, without being able to access sensitive information about cus-tomers. formally, we aim to achieve the following goal: we consider a dataset consisting of a training set d train and test set d test . given d train or a subset of it, we want to train a generative model to synthesize a dataset ‹ d train that does not leak information from the original d train . furthermore, the synthesized dataset should share statistical properties with the original one so that a classification model trained on ‹ d train performs as well as if it was trained on d train when making predictions about d test . to achieve this, we use the pretrained autoregressive transformer model @xcite gpt-2 @xcite and use natural language prompts to enable the conditional generation of text based on desired textual attributes such as its sentiment, domain or genre provided in the prompt. furthermore, we introduce a new training objective that penalizes the generation of samples fitting another label to reduce the risk of faulty labeled samples in our synthetic dataset. finally, we fine-tune our model using a differentially private optimizer to provide privacy guarantees for our training data and to prevent information leakage from our model when subsequently sampling our synthetic dataset. ## evaluation we conduct extensive evaluation measuring the utility and privacy of our generated data as well as the quality of its language. in this section, we describe the datasets we use as well as our evaluation metrics, implementation details and results. ## conclusion in this paper, we explored the generation of synthetic datasets from differentially private language models as a solution for publicly sharing textual data while protecting the privacy of users whose data is being shared. our experiments show that synthetic data from differentially private language models is of high quality and is very well suited as training data for further tasks while significantly reducing the risk of leaking the original data. our approach can be applied in a variety of use cases working with sensitive data. an interesting challenge for future work is the anonymization of multimodal datasets consisting of tabular, visual and text data.