Differentially Private Representation for NLP : Formal Guarantee and An Empirical Study on Privacy and Fairness (2020)

Abstract: It has been demonstrated that hidden representation learned by deep model can encode private information of the input, hence can be exploited to recover such information with reasonable accuracy. To address this issue, we propose a novel approach called Differentially Private Neural Representation (DPNR) to preserve privacy of the extracted representation from text. DPNR utilises Differential Privacy (DP) to provide formal privacy guarantee. Further, we show that masking words via dropout can further enhance privacy. To maintain utility of the learned representation, we integrate DP-noisy representation into a robust training process to derive a robust target model, which also helps for model fairness over various demographic variables. Experimental results on benchmark datasets under various parameter settings demonstrate that DPNR largely reduces privacy leakage without significantly sacrificing the main task performance.

## introduction many language applications have involved deep learning techniques to learn text representation through neural models @xcite @xcite , performing composition over the learned representation for downstream tasks @xcite . however, the input text often provides sufficient clues to portray the author, such as gender, age, and other important attributes. for example, sentiment analysis tasks often have privacy implications for authors whose text is used to train models. many user attributes have been shown to be easily detectable from online review data, as used extensively in sentiment analysis results @xcite . private information can take the form of key phrases explicitly contained in the text. however, it can also be implicit. for example, demographic information about the author of a text can be predicted with above chance accuracy from linguistic cues in the text itself @xcite . on the other hand, even the learned representation, rather than the text itself, may still contain sensitive information and incur significant privacy leakage. one might argue that sensitive information like gender, age, location and password should not be leaked out and should have been removed from representation. however, on the intermediate representation level, which is trained from the input text to contain useful features for the prediction task, it can meanwhile encode personal information which might be exploited for adversarial usages, especially a modern deep learning model has vastly more capacity than they need to perform well on their tasks. and, it has been justified that an attacker can recover private variables with higher-than-chance accuracy, only using hidden representation @xcite . therefore, the fact that representations appear to be abstract real-numbered vectors should not be misconstrued as being safe. the naive solution of removing protected attributes is insufficient: other features may be highly correlated with, and thus predictive of, the protected attributes @xcite . to tackle with these privacy issues, @xcite proposed to train deep models with adversarial learning, which explicitly obscures individuals' private information, while improves the robustness and privacy of neural representation in part-of-speech tagging and sentiment analysis tasks. in a parallel study, @xcite proposed defence methods based on modifications of the training objective of the main model. however, both works provide only empirical improvements in privacy, without any formal guarantees. prior works have approached formal differential privacy guarantee by training dif-ferentially private deep models @xcite @xcite . however, these works generally only considered the training data privacy rather than the test data privacy. while cryptographic methods can be used for privacy protection, it could be resource-hungry or overly complex for the user. to alleviate the above limitations, we take inspirations from differential privacy @xcite to provide formal privacy guarantee of the extracted representation from user-authored text. meanwhile, we propose a robust training algorithm to derive a robust target model to maintain utility, which also offers fairness as a by-product. to the best of our knowledge, our work is the only work to date that can provide formal differential privacy guarantee of the extracted representation, while ensuring fairness. our main contributions include: a novel approach called differentially private neural representation (dpnr) is proposed to perturb the extracted representation.also, we prove that masking words via dropout can further enhance privacy. ## preliminary: differential privacy differential privacy @xcite provides a mathematically rigorous definition of privacy and has become a de facto standard for privacy analysis. within dp framework, there are two general settings: central dp (cdp) and local dp (ldp). in cdp, a trusted data curator answers queries or releases differentially private models by using ran-domisation mechanisms @xcite @xcite . for scenarios where data are sourced from end users, and end users do not trust any third parties, dp should be enforced in a "local" manner to enable end users to perturb their data before publication, which is termed as ldp @xcite . compared with cdp, ldp offers a stronger level of protection. in our system, we aim to protect the test-phase privacy of the extracted neural representations from end users, we therefore adopt ldp. ldp has shown the advantage that the data is randomised before individuals disclose their personal information, so the server and the middle eavesdropper can never see or receive the raw data. in terms of ldp mechanisms, randomised response @xcite and its variants have been widely used for aggregating statistics, such as frequency estimation, heavy hitter estimation, etc @xcite . definition 2.1. let a : @xmath0. the algorithm a is ( , δ)-local differentially private if for all data entries x, @xmath1, we have pr{a(x) = o} ≤ exp( ) pr{a(x ) = o} + δ if δ = 0, a is said to be -local differentially private. a formal definition of ldp is provided in definition 2.1, the privacy parameter captures the privacy loss consumed by the output of the algorithm: = 0 ensures perfect privacy in which the output is independent of its input, @xmath2. for every pair of adjacent inputs x and x , differential privacy requires that the distribution of a(x) and a(x ) are "close" to each other where closeness are measured by the privacy parameters and δ. typically, the inputs x and x are adjacent inputs when all the attributes of one record are modified. in real scenario, the adjacent input is an application specific notion. for example, a sentence is divided into several items for every 5 words, and two sentences are considered to be adjacent if they differ by at most 5 consecutive words @xcite . in this work, we consider a word-level dp, i.e., two inputs are considered to be adjacent if they differ by at most 1 word. for brevity, we use ( , δ)-dp to represent ( , δ)-ldp for the rest of the paper. we remark that all the randomisation mechanisms used for cdp, including laplace mechanism and gaussian mechanism @xcite , can be individually used by each party to inject noise into local data to ensure ldp before releasing @xcite @xcite . in particular, we adopt laplace mechanism which ensures -dp with δ = 0 throughout the paper. in a nutshell, data universe can be expressed as @xmath3, a, y), which will be convenient to partition as (x , y) × a @xcite . given one person's record x, we can write it as a pair @xmath4, x s ) where x @xmath5, y) represents the insensitive attributes and x @xmath6. our main goal is to promise differential privacy only with respect to the sensitive attributes. write x s ∼ x s to denote that x s and x s differ in exactly one coordinate (i.e. one word/token in nlp domain). an algorithm is ( , δ)differentially private in the sensitive attributes if for all x @xmath7, y) and for all x s ∼ x @xmath8, we have: post-processing. dp enjoys a well-known postprocessing property @xcite : any computation applied to the output of an ( , δ)-dp algorithm remains ( , δ)-dp. this nice property allows the attacker to implement any sophisticated post-processing function on the privatised representation from the user, without compromising dp or making it less differentially private. ## experiments in this section, we conduct comprehensive studies over different tasks and datasets to examine the efficacy of the proposed algorithm from three facets: 1) main task performance, 2) privacy and 3) target model fairness. ## discussion privacy and fairness are two emerging but important areas in nlp community. prior efforts predominantly focus on either privacy or fairness @xcite @xcite @xcite , but there is no systematic study on how privacy and fairness are related. this work fills this gap, and discovers the impact of differential privacy on model fairness. we empirically show that privacy and fairness can be simultaneously achieved through differential privacy. we hope that this work highlights the need for more research in the development of effective countermeasures to defend against privacy leakage via model representation and mitigate model bias in a general sense, and not only specific to a particular attack. more generally, we hope that our work spurs future interest into developing a better understanding of why differential privacy works. meanwhile, differential privacy may incur a reduction in the model's accuracy. it is worthwhile to explore how to get a better trade-off between privacy, fairness and accuracy. ## conclusion and future work in this paper, we take the first effort to build differential privacy into the extracted neural representa-tion of text during inference phase. in particular, we prove that masking the words in a sentence via dropout can further enhance privacy. to maintain utility, we propose a novel robust training algorithm that incorporates a noisy layer into the training process to produce the noisy training representation. experimental results on benchmark datasets across various tasks, and parameter settings demonstrate that our approach ensures representation privacy without significantly degrading accuracy. meanwhile, our dp method helps reduce the effects of model discrimination in most cases, achieving better fairness than the non-private baseline. our work makes a first step towards understanding the connection between privacy and fairness in nlp -which were previously thought of as distinct classes. moving forward, we believe that our results justify a larger study on various nlp applications and models, which will be our immediate future work.