Thinking Outside of the Differential Privacy Box: A Case Study in Text Privatization with Language Model Prompting (2024)

Abstract: The field of privacy-preserving Natural Language Processing has risen in popularity, particularly at a time when concerns about privacy grow with the proliferation of large language models. One solution consistently appearing in recent literature has been the integration of Differential Privacy (DP) into NLP techniques. In this paper, we take these approaches into critical view, discussing the restrictions that DP integration imposes, as well as bring to light the challenges that such restrictions entail. To accomplish this, we focus on **DP-Prompt**, a recent method for text privatization leveraging language models to rewrite texts. In particular, we explore this rewriting task in multiple scenarios, both with DP and without DP. To drive the discussion on the merits of DP in NLP, we conduct empirical utility and privacy experiments. Our results demonstrate the need for more discussion on the usability of DP in NLP and its benefits over non-DP approaches.

## introduction the topic of privacy in natural language processing has recently gained traction, which has only been fueled by the prominent rise of large language models. in an effort to address concerns revolving around the protection of user data, the study of privacy-preserving nlp has presented a plethora of innovative solutions, all investigating in some form the optimization of the privacy-utility trade-off for the safe processing of textual data. a well-studied solution comes with the integration of differential privacy (dp) @xcite into nlp techniques. essentially, the use of dp entails the addition of calibrated noise to some stage in a pipeline, e.g., directly to the data or to model weights. this is performed with the ultimate goal of protecting the individual whose data is being used, aligned with the objective of differential privacy set out in its inception nearly 20 years ago. the incentive of proving differential privacy is the mathematical guarantee of privacy protection that it offers, so long as its basic principles are adhered to. particularly, important dp notions must be strictly defined, such as who the individual is, how data points are adjacent, and how data can be bounded. as such, the fusion of differential privacy and nlp introduces several challenges @xcite @xcite . when generalized forms of dp are used or well-defined notions of dp concepts are lacking, the promise of dp becomes more of a shallow guarantee. in this work, we critically view the pursuit of dp in nlp, focusing on the particular method of dp-prompt @xcite . this method leverages generative language models to rewrite (paraphrase) texts with the help of a dp token selection method based on the exponential mechanism @xcite . we run experiments on three rewriting settings: (1) dp, (2) quasi-dp, and (3) non-dp; the purpose of this trichotomy is to explore the benefits and shortcomings of dp in text rewriting. we define our research question as: what is the benefit of integrating differential privacy into private text rewriting methods leveraging lms, and what effect can be observed by relaxing this guarantee? our empirical findings show the advantages that incorporating dp into text rewriting mechanisms brings, notably higher semantic similarity and resemblance to the original texts, along with strong empirical privacy results. this, however, comes with the downside of generally lower quality text in terms of readability, particularly at stricter privacy budgets. these findings open the door to discussions regarding the practical distinction between dp and non-dp text privatization, where we present open questions and paths for future work. the contributions of our work are as follows: ## related work natural language can leak personal information @xcite and it is possible to extract training data from machine learning models @xcite @xcite . in the global dp setting, user texts are collected at a central location and a model is trained using privacy-preserving optimization techniques @xcite such as dp-sgd @xcite . the primary drawback of this model is that user data must be collected at a central location, giving a data curator access to the entire data @xcite . to mitigate this, text can be obfuscated or rewritten locally in a dp manner before collecting it at a central location @xcite @xcite . the earliest set of approaches of dp in nlp began at the word level @xcite @xcite @xcite ), yet these methods do not consider contextual and grammatical information during privatization @xcite . other works operate directly at the sentence level by either applying dp to embeddings @xcite or latent representations @xcite @xcite . dp text rewriting methods using generative lms @xcite @xcite or encoder-only models @xcite have also been proposed. ## method here, we describe the base text privatization method that we utilize, as well as the variations which form the basis of our experiments. ## experimental setup and results as stated by @xcite , a practical text privatization mechanism should: (1) protect against deanonymization attacks, (2) preserve utility, and (3) keep the original semantics intact. as such, we design our experiments by leveraging multiple dimensions of a single dataset. the results of all described experiments can be found in ## discussion in analyzing the results, we first discuss the merits of dp text privatization. at stricter privacy budgets (lower ε), only the original dp-prompt is able to present significant gains, as showcased with ε = 25. at these lower values, one can also observe the benefits of enforcing dp via logit clipping, which results in higher cs and bleu retention while outputting generally more readable text (much lower ppl). this trend with ppl holds for all scenarios of dp vs. quasi-dp, making a clear case for proper bounding in dp applications. in studying dp vs. quasi-dp further, we notice that the distinction between the two, particularly at higher ε values, becomes somewhat opaque. in fact, quasi-dp outperforms dp in terms of empirical privacy in many of the higher privacy budget scenarios. this would imply that a dp mechanism leveraging temperature sampling only becomes effective and sensible with stricter privacy budgets. an important point of comparison also comes with the study results of our non-dp method. a strength of this method is highlighted by its ability at lower k values (analogous to less strict privacy budgets) to maintain high levels of semantic similarity (cs), while still achieving competitive empirical privacy scores. for example, in the case of @xmath0, this method is able to outperform all ε ≥ 100 for both dp and quasi-dp. the bleu scores for non-dp would also imply that this method is better able to rewrite texts in a semantically similar, yet lexically different manner, as opposed to dp methods at high ε values (see appendix d). these results make a case for non-dp privatization in certain cases, and in parallel, provide a critical view of using dp at high ε values which lead to ineffective empirical privacy. a final point that is crucial to discuss is grounded in the observed relative gains. looking to the cumulative scores ( γ) of ## conclusion central to this work is the debate on the merits of differential privacy in nlp. to lead this discussion, we conduct a case study with the dp-prompt mechanism, juxtaposed with two "relaxed" variants. our results show that while the theoretical guarantee of individual privacy may be important in some application settings, in others, it may become too restrictive to apply effectively. conversely, the merits of dp may be observed in stricter privacy scenarios, where the need for tight guarantees does bring favorable privacy-utility trade-offs. we call for further research in two directions: (1) rigorous studies on the theoretical and practical implications of dp vs non-dp privatization, and relatedly, (2) the continued design of privatization mechanisms outside the realm of differential privacy that aim to balance strong privacy protections with practical utility preservation. we hope that researchers may be able to harmonize the "best of both worlds", keeping in sight the need for practically usable privacy protection of text data.