Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe (2023)

Abstract: Privacy concerns have attracted increasing attention in data-driven products due to the tendency of machine learning models to memorize sensitive training data. Generating synthetic versions of such data with a formal privacy guarantee, such as differential privacy (DP), provides a promising path to mitigating these privacy concerns, but previous approaches in this direction have typically failed to produce synthetic data of high quality. In this work, we show that a simple and practical recipe in the text domain is effective: simply fine-tuning a pretrained generative language model with DP enables the model to generate useful synthetic text with strong privacy protection. Through extensive empirical analyses on both benchmark and private customer data, we demonstrate that our method produces synthetic text that is competitive in terms of utility with its non-private counterpart, meanwhile providing strong protection against potential privacy leakages.

## introduction the issue of privacy has gained increasing attention in natural language processing (nlp). privacy attacks against common nlp pipelines have demonstrated that models trained without formal privacy guarantees can reveal membership information and enable training data reconstruction @xcite . privacy concerns manifested through tightening legislation (e.g., gdpr (art. 29 wp, 2014)) and growing discussions on policy and ethics call for improved approaches for privacy-preserving machine learning. among different approaches for learning with private data, learning with differential privacy (dp) @xcite has become the gold standard as its formal guarantee enables reasoning about the privacy loss in a principled manner and makes the approach resilient to strong privacy attacks @xcite . recent developments have substantially improved the computational efficiency and privacy-utility trade-off of dp machine learning @xcite @xcite @xcite @xcite , demonstrating gains for learning models that perform specific downstream tasks. in contrast to the above works, we study synthetic text generation by building generative text models with dp training algorithms (figure 1 ). the goal of this approach is to learn a generative model that faithfully captures distributional properties of the training data (and the underlying distribution), as opposed to learning task-oriented models with specific functions. compared to directly learning models for target tasks, this paradigm has several advantages: (1) dp-trained generative models can be used to draw synthetic data for learning an expanding set of task models without incurring any additional privacy loss (due to the post-processing property of dp); (2) dataset debugging is made easy as synthetic text generated from dp-trained models can be shared more freely, and inspecting its samples poses less of a privacy concern compared to examining the original private data @xcite ; (3) synthetic data generated from dp-trained models can be retained for a longer time under certain existing policies (e.g., right to be forgotten) thanks to the fact that dp implies some degree of approximate machine unlearnwe propose to generate synthetic text with a formal privacy guarantee: we fine-tune a generative language model with dp and then leverage it for synthetic text generation using control codes. privacy loss of the overall procedure can be controlled by the data generation stage as, by the robustness to post-processing property of dp, the downstream task stage does not incur any additional privacy loss. in this work, we initiate a systematic empirical study of the problem and show that dp language model (lm) fine-tuning can be an effective solution to synthetic text generation with privacy. in particular, we show that simply fine-tuning progressively larger autoregressively pre-trained language models on (private) data leads to models that generate increasingly useful synthetic text. for instance, we fine-tune a gpt-2 large model @xcite on a review dataset with dp at ϵ = 4 and then use it to generate synthetic text to build downstream classifiers. the classification models achieve comparable performance (only 2-4% in accuracy away) to the classifiers trained on the original dataset. furthermore, we demonstrate that generating a small amount of synthetic data with dp is sufficient to create classification models that are on par with those trained directly on the entire original dataset with dp. one of the advantages of the synthetic data approach is that the privacy loss is fixed, and an unlimited number of downstream models can be built without incurring additional leakage. in contrast, training additional downstream models on the original data with dp accumulates privacy loss. distributional similarity evaluation additionally confirms that the synthetic text distribution resembles the original data distribution. we also uncover a novel phenomenon in dp-trained lms that is of independent interest. specifically, we observe a length truncation effect in text generation with dptrained models, resulting in completions that are generally shorter than their non-dp counterparts and instances in the original dataset. we further extensively study learning dynamics with dp by injecting specially-crafted canaries @xcite in the training data. this allows for (i) stress-testing the extent to which dp fine-tuning limits the leakage of private information and (ii) understanding the conditions under which a subject of interest would appear in synthetic generations. finally, we conclude our studies on an industriallevel private customer feedback dataset to show the feasibility of our approach in real-world scenarios. ## method in this section, we formally state the problem and present our method (see figure 1 for an illustration) that produces a synthetic version of private text data with differential privacy. ## analyses on a public review dataset in this section, we extensively analyze our method with experiments on a public benchmark dataset: yelp open dataset, 2 which has been widely adopted for language modeling and text classification tasks. we then apply our method to an internal private customer feedback dataset in section 5. ## results on private customer feedback to demonstrate the effectiveness of our method in safeguarding utility and privacy in practical scenarios, we evaluate its performance using a microsoft private feedback dataset obtained from customers. background. industrial applications often receive a significant volume of customer feedback regarding their products. customer feedback is valuable as it provides insights into product performance, user satisfaction, and areas for improvement. while customer feedback may not typically contain personally identifiable information, it may still include sensitive details that could potentially disclose the customer's identity. for example, customers might mention specific job titles, company names, or locations in their feedback. when combined with other publicly available information, these details could potentially be used to identify the customer and compromise their privacy. protecting the privacy of this information is crucial to comply with privacy regulations such as the gdpr (art. 29 wp, 2014), build trust with customers, and mitigate the risk of unauthorized access or misuse. dataset. in our scenario, 1m customer feedback is collected on a set of microsoft products. for downstream tasks, we are interested in three attributes of the feedback, which we call a(ttribute)1, a2 and a3. attributes can be a number of product characteristics including, but not limited to, user satisfaction scores, date and time range, product name, product type, location, etc. using the attributes (a1, a2, a3) together with a particular combination of their respective values, such as (v a1 , v a2 , v a3 ), the conditional text generation prompt becomes: "a1: we use the gpt2-large model with the settings described in section 4.1 in our scenario. downstream task performance. similar to section 4.2, to measure the quality of synthetic data, we evaluate the performance of classification models trained on them. we train three classification models, to predict three attributes a1, a2, and a3 with 5, 45, and 5 classes respectively. we present the results in ## conclusion in this paper, we present a simple and practical recipe for generating synthetic text data with privacy guarantees. our method is built upon pretrained language models and differential privacy, where the former enables us to generate highquality synthetic text data and the latter provides formal privacy guarantees that no single example in the training dataset can influence the trained model by a substantial amount probabilistically. we conduct comprehensive experiments evaluating both utility and privacy risks of the synthetic data. the results demonstrate that our method can generate high-quality text while mitigating privacy risks.