Locally Differentially Private Document Generation Using Zero Shot Prompting (2023)

Abstract: Numerous studies have highlighted the privacy risks associated with large language models. Our research offers a unique perspective by demonstrating that pretrained large language models can effectively contribute to privacy preservation. We propose a locally differentially private mechanism called DP-Prompt, which leverages the power of pretrained large language models and zero-shot prompting to counter author de-anonymization attacks while minimizing the impact on downstream utility. When DP-Prompt is used with a powerful language model like ChatGPT (gpt-3.5), we observe a notable reduction in the success rate of de-anonymization attacks, showing that it surpasses existing approaches by a considerable margin despite its simpler design. For instance, in the case of the IMDB dataset, DP-Prompt (with ChatGPT) perfectly recovers the clean sentiment F1 score while achieving a 46% reduction in author identification F1 score against static attackers and a 26% reduction against adaptive attackers. We conduct extensive experiments across six open-source large language models, ranging up to 7 billion parameters, to analyze various effects of the privacy-utility tradeoff.

## introduction the vast amount of online text data has the potential to reveal numerous user attributes, making individuals easily identifiable @xcite @xcite . while private information can be directly disclosed through specific phrases in the text, it can also be implicitly inferred. for instance, linguistic patterns embedded within the text can inadvertently facilitate authorship attribution @xcite , leading to unintended privacy leakage. an illustrative real-world scenario is the aol search data leak in august 2006 @xcite , 2006). the incident unfolded when aol mistakenly released detailed search logs of their users, wrongly assuming that the data had been adequately anonymized through the use of random user ids. unfortunately, the released logs contained sufficient personally identifiable information, leading to the identification of numerous individuals @xcite . this breach of privacy triggered widespread public outcry and led to the initiation of class action lawsuits. this case is just one among many that highlights the limitations of ad-hoc privacy approaches that may give the impression of providing privacy but ultimately fall short. differential privacy (dp) provides a rigorous treatment for the notion of data privacy by providing plausible deniability by precisely quantifying the deviation in the model's output distribution under modification of a small number of data points @xcite . the provable guarantees offered by dp, coupled with its compelling properties such as immunity to arbitrary post-processing and graceful composability, have established it as the de facto standard for privacy. dp has witnessed widespread adoption and numerous deployments in both private @xcite @xcite and public organizations @xcite . to address the issue of deanonymization attacks, various approaches have been proposed within the dp framework. these approaches encompass word-level strategies @xcite @xcite where noise is added at the word level, as well as sentence-level techniques @xcite where noise is added at the sentence level. however, recent research by @xcite has identified limitations in word-level approaches, particularly their disregard for contextual information. to overcome these limitations, mattern introduced a mechanism that fine-tunes the gpt-2 model @xcite specifically for paraphrasing tasks, resulting in the generation of sanitized versions of documents. while promising, the approach is limited by their reliance on annotated paraphrasing data, extensive computing resources for larger models, and the quality of annotations. we propose dp-prompt, a novel and straightforward solution to address deanonymization attacks. our method leverages pretrained large language models by directly prompting them to generate paraphrases. these paraphrases are then released as sanitized documents in a zero-shot manner (see figure 1 ). our motivation for this approach stems from two important factors. firstly, recent research @xcite has shown that paraphrasing is a robust defense mechanism against deanonymization attacks. secondly, growing evidence suggests that pretrained large language models can effectively tackle complex tasks without the need for task-specific and expensive fine-tuning @xcite @xcite @xcite , through zero-shot prompting. by harnessing the capabilities of pretrained large language models, dp-prompt offers a straightforward and powerful solution to mitigate the risk of deanonymization. it provides a promising alternative that can be widely applicable, particularly in the context of on-device large language models where text completion tasks require significantly fewer resources. we summarize the contributions as follows: the resulting paraphrased document is then released as a sanitized document, as illustrated in figure 1 . ## preliminaries a mechanism m : @xmath0, for all inputs d, d ′ ∈ d that differ in one element, and for all @xcite . metric differential privacy (metric-dp) @xcite is a relaxation of pure-dp that applies to data represented in a general metric space. for a given distance met- local differential privacy (ldp) @xcite @xcite is a privacy framework where data is locally perturbed before transmission, considering the presence of an untrusted data collector or server. the formal definition of ldp is as follows: definition 1 (pureldp). a randomized mechanism m : @xmath1, d ′ ∈ d and for all there is a growing consensus that, despite the assurance of formal guarantees, it is imperative to subject differentially private mechanisms to robust privacy attacks that simulate strong and malicious adversaries @xcite . such evaluation allows to effectively assess the empirical privacy provided by the mechanism in real-world scenario. to this end we define four attack models depending its adaptivity and mode of access. definition 2 (attack models). consider a collection of private documents (d 1 , . . . , d n ) from distribution d with associated author identities (a 1 , . . . , a n ) and embeddings for text-to-text sanitization using mechanism m text , the sanitized documents are represented as (p 1 , . . . , p n ) ∼ p mtext . for text-to-embedding sanitization via mechanism m embedding , the sanitized embeddings are denoted as (n 1 , . . . , n n ) ∼ n m embedding • static attacker with embedding access: has access to clean documents (d 1 , . . . , d n ) but lacks access to sanitized versions (p 1 , . . . , p n ). it is important to note that the adaptive attacker is a more formidable adversary since it adapts to the characteristics of the mechanism m, whereas the static attacker only has access to clean documents/clean embeddings without any added noise. the mode of access-either raw text or abstracted embeddings-offers further nuances, determining the exact nature of the data an attacker can exploit. ## dp prompt language models use a decoder network to generate sequential text output. given a context or prompt represented by a sequence of tokens @xmath2, . . . , c m ), the language model generates text by sampling tokens from a conditional distribution in this distribution, the logits @xmath3, where definition 3 (exponential mechanism). given an utility function u : d × @xmath4. the exponential mechanism m exp : @xmath5,v) 2∆u , where ∆@xmath6, the utility of token v @xmath7. hence, one can make text generation differentially private using the exponential mechanism. extensive research has shown that paraphrasing documents helps conceal author identity @xcite @xcite . considering recent advancements where tasks are formulated as prompts and language models are tasked to complete them @xcite @xcite , we directly prompt the language model to generate paraphrases. therefore, given a private document d and a specific prompt template instructing the language model to generate a paraphrase, such as t := "paraphrase of the document:" we combine d and t to create a context c. by utilizing this context, we execute the text generation procedure in a differentially private manner to produce a paraphrase. we refer to this procedure as dp-prompt. algorithm 1 outlines the specific steps of our pro-algorithm 1: dp-prompt input: language model (lm), private document (d), prompt template (t), clipping vector @xmath8, temperature @xmath9, paraphrase tokens n. output: sanitized doc (p) 0.0 0.2 0.4 0.6 0.8 author identification f1 score (privacy) 0.3 0.4 0.5 0.6 0.7 0.8 sentiment f1 score (utility) 46% reduction clean random madlib mahalanobis tem truncated-laplace deep-candidate paraphraser(gpt2-xl) dp-prompt(chatgpt) (a) imdb (static) 0.2 0.4 0.6 0.8 author identification f1 score (privacy) 0.50 0.55 0.60 0.65 0.70 0.75 0.80 sentiment f1 score (utility) 25% reduction clean random madlib mahalanobis tem truncated-laplace deep-candidate paraphraser(gpt2-xl) dp-prompt(chatgpt) (b) imdb (adaptive) 0.0 0.2 0.4 0.6 0.8 author identification f1 score (privacy) 0.50 0.55 0.60 0.65 0.70 0.75 0.80 sentiment f1 score (utility) 53% reduction clean random madlib mahalanobis tem truncated-laplace deep-candidate paraphraser(gpt2-xl) dp-prompt(chatgpt) (c) yelp (static) 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 author identification f1 score (privacy) 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 sentiment f1 score (utility) 29% reduction clean random madlib mahalanobis tem truncated-laplace deep-candidate paraphraser(gpt2-xl) dp-prompt(chatgpt) (d) yelp (adaptive) 0.0 0.2 0.4 0.6 0.8 1.0 author identification f1 score (privacy) 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 sentiment f1 score (utility) 54% reduction clean random madlib mahalanobis tem paraphraser(gpt2-xl) dp-prompt(chatgpt) (e) imdb (static) 0.2 0.4 0.6 0.8 1.0 author identification f1 score (privacy) 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 sentiment f1 score (utility) 10% reduction clean random madlib mahalanobis tem paraphraser(gpt2-xl) dp-prompt(chatgpt) (f) imdb (adaptive) 0.0 0.2 0.4 0.6 0.8 1.0 author identification f1 score (privacy) 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 sentiment f1 score (utility) 73% reduction clean random madlib mahalanobis tem paraphraser(gpt2-xl) dp-prompt(chatgpt) (g) yelp (static) 0.0 0.2 0.4 0.6 0.8 1.0 author identification f1 score (privacy) 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 sentiment f1 score (utility) 24% reduction clean random madlib mahalanobis tem paraphraser(gpt2-xl) dp-prompt(chatgpt) sessment. as a result, we plot the author identification f1 score, which is calculated by conducting de-anonymization attacks on the sanitized documents. this score indicates the potential for privacy breaches. on the other hand, the y-axis represents the sentiment f1 score, which measures the utility of the sanitized documents. ## issue of data memorization it is important to acknowledge the possibility that chatgpt and other open-source language models (llms) may have been exposed to online review datasets such as imdb and yelp. this exposure is not unique to our approach. similar questions can be raised for word-level approaches @xcite @xcite and sentence-level approaches @xcite since glove embeddings @xcite and sentence transformers (bert) @xcite are pretrained on the common crawl dataset. hence assumption in theorem 1 may not hold. we argue that this exposure does not pose a major concern for dp-prompt due to the following reasons. no availability of paraphrases of imdb, yelp: dp-prompt essentially evaluates the performance of llms in the task of zero-shot paraphrasing at higher temperatures. although the language models may have been fine-tuned using open annotated paraphrase datasets @xcite , it's important to note that there is no specific annotated paraphrasing data available for the imdb and yelp datasets. therefore, the language models used in dp-prompt are not explicitly fine-tuned for paraphrasing tasks related to these datasets. consequently, the results obtained in section 4 are likely to generalize to a large extent. robustness of evaluation setup: our evaluation methodology goes beyond relying solely on the sentiment f1 score (utility) or even comparing sentiment f1 score (utility) with ϵ (theoretical privacy) as presented in @xcite . instead, we adopt a comprehensive approach that considers the merit of our proposed approach based on the sentiment f1 score in conjunction with empirical privacy demonstration against de-anonymization attacks. simply copying reviews verbatim would yield a high author identification f1 score, failing to provide empirical privacy. additionally, we consider not only static attack models but also stronger ones like adaptive attackers. superficially altering reviews may deceive static attackers but would likely fail against adaptive attackers. ## related work the previous work on releasing private documents can be categorized into three approaches based on the level at which noise is added (see ## conclusion this paper introduces dp-prompt, a locally differentially mechanism called dp-prompt that generates sanitized versions of private documents by prompting large language models to generate paraphrases. notably, our method offers a simpler approach compared to existing methods. through extensive experiments, we show that our approach achieves significantly improved utility compared to current methods for any required level of privacy. as the demand for on-device large language models (llms) continues to grow, our method emerges as a reliable safeguard for users' privacy and provides robust defense against de-anonymization attacks.