A Customized Text Sanitization Mechanism with Differential Privacy (2023)

Abstract: As privacy issues are receiving increasing attention within the Natural Language Processing (NLP) community, numerous methods have been proposed to sanitize texts subject to differential privacy. However, the state-of-the-art text sanitization mechanisms based on a relaxed notion of metric local differential privacy (MLDP) do not apply to non-metric semantic similarity measures and cannot achieve good privacy-utility trade-offs. To address these limitations, we propose a novel Customized Text sanitization (CusText) mechanism based on the original \epsilon -differential privacy (DP) definition, which is compatible with any similarity measure.Moreover, CusText assigns each input token a customized output set to provide more advanced privacy protection at the token level.Extensive experiments on several benchmark datasets show that CusText achieves a better trade-off between privacy and utility than existing mechanisms.The code is available at https://github.com/sai4july/CusText .

## introduction in many natural language processing (nlp) applications, input texts often contain sensitive information that can infer the identity of specific persons @xcite , leading to potential privacy leakage that impedes privacy-conscious users from releasing data to service providers @xcite @xcite . moreover, legal restrictions such as ccpa 1 and gdpr 2 may further limit the sharing of sensitive textual data. this makes nlp service providers difficult to collect training data unless the privacy concerns of data owners, including individuals and institutions, are well discoursed. ## related work there have been numerous studies on the vulnerability of deep learning models @xcite , including language models @xcite ) (lms), against privacy attacks. in particular, such attacks can recover sensitive user attributes or raw texts from trained models. therefore, incorporating privacy mechanisms with rigorous guarantees is vital to protect lms from privacy attacks. a few attempts at applying anonymization techniques for i.i.d. data @xcite fail to provide strong privacy protection for textual data @xcite . then, many efforts @xcite @xcite @xcite have been made to preserve the utility of lms on textual data with provable differential privacy (dp) guarantees. following the application of dp in deep learning @xcite , they mainly focus on adding calibrated noise to gradients or text representations during the training phase for both utility and privacy. however, they need a trustworthy server to collect original texts from data owners for model training and thus cannot be applied to the scenario without trusted servers. to address privacy issues from the root, different (customized) local differential privacy @xcite (ldp) mechanisms have been proposed to allow data owners to sanitize their data locally before releasing them to the server. due to the high dimensionality and complicated features of textual data, compared with statistical analytics on i.i.d. data with ldp @xcite , it is much more challenging to achieve good utilityprivacy trade-offs for lms with ldp. to improve the model utility, existing methods @xcite @xcite rely on a relaxed notion of metric local differential privacy @xcite (mldp, also known as d χ -privacy) for text sanitization. however, they either achieve reasonable accuracy only at a very low privacy protection level (e.g., with a privacy parameter ϵ > 10) or become unusable (around 50% accuracy rate for the benchmark binary classification tasks) with appropriate privacy guarantees (e.g., ϵ = 2). thus, there remains much room for improvement in terms of utility-privacy trade-off for differentially private text sanitization, which is the goal of this work. ## preliminaries before introducing our custext mechanism, we briefly review the key concepts, including ϵ-dp and exponential mechanism (em). definition 1 (ϵ-differential privacy @xcite ). for a given privacy parameter ϵ ≥ 0, all pairs of adjacent inputs x, x ′ ∈ x , and every possible output @xmath0, a randomized mechanism m is ϵ-differentially private (dp) if it holds that by definition, a smaller value of ϵ corresponds to a higher level of privacy protection. conceptually, the notion of ϵ-dp means that an unlimited adversary cannot distinguish the two probabilistic ensembles with sufficiently small ϵ because the probabilities of adjacent tokens producing the same output token y are similar. in the context of nlp, we consider any pair of input tokens that share the same output set y to be adjacent to each other. in the rest of this paper, we follow the above definition of adjacent inputs for ϵ-dp. next, we define the exponential mechanism (em) commonly used for differentially private item selection from a discrete domain, which naturally fits nlp applications due to the discrete nature of textual data. definition 2 (exponential mechanism @xcite ). for a given scoring function u : x × @xmath1, an exponential mechanism (em) m(x , u, y) satisfies ϵ-differential privacy if it samples an output token @xmath2,y) 2∆u , where u(x, y) denotes the score of output token y for input token x. in addition, we use ∆u := max @xmath3,x ′ ∈x |u(x, y)u(x ′ , y)| to denote the sensitivity of u for em. from definition 2, we can see that smaller sensitivity makes it harder for adversaries to distinguish the original token from its adjacent tokens. in practice, for simplicity, we can normalize the scoring function u to scale its sensitivity ∆u to a specific real number (e.g., 1). as such, the sampling probability of each output token y for input token x is only related to u(x, y), as ϵ and ∆u are known beforehand, and a larger u(x, y) indicates a higher sampling probability. in an nlp task, we suppose that each document @xmath4. we formulate our text sanitization task as follows: given an input document d containing sensitive information, a set x of all possible input tokens, a set y of all possible output tokens, and a differentially private mechanism m (e.g., em in this work), it performs the mechanism m on each input token t @xmath5. all the tokens after replacement form the sanitized document, i.e., d ′ = ⟨r ′ i ⟩ m @xmath6. following the prior work on text sanitization @xcite @xcite , we consider a semi-honest threat model under the ldp setting where data owners (e.g., individuals or institutions) only submit their sanitized documents to the service provider. malicious service providers may try to infer sensitive information from their received data. we assume that adversaries only have access to sanitized texts, and all algorithms and mechanisms are publicly known. moreover, adversaries have unlimited computation resources. ## the custext mechanism an overview of our customized text (custext) sanitization mechanism is presented in figure 2 . in general, it replaces each token in the original text document with a new token to achieve the privacy guarantee. it consists of two components: (1) a mapping function f map : @xmath7; (2) a sampling function foot_0 f sample : x ′ → y ′ based on the exponential mechanism to sample a new token from ## concluding remarks in this work, we study the problem of differentially private text sanitization. we propose a novel cus-text mechanism consisting of a mapping scheme to assign each input token a customized output set and sampling function generation methods based on the mapping scheme and exponential mechanism to reduce privacy costs while improving the utilities of sanitized texts. extensive experiments demonstrate that custext achieves better privacyutility trade-offs than state-of-the-art text sanitization mechanisms. in the future, we will explore how to improve our mechanism by adaptively allocating privacy costs across tokens and find a better way to decide whether a token is sensitive than based on a pre-defined stopword list.