How reparametrization trick broke differentially-private text representation learning (2022)

Abstract: As privacy gains traction in the NLP community, researchers have started adopting various approaches to privacy-preserving methods. One of the favorite privacy frameworks, differential privacy (DP), is perhaps the most compelling thanks to its fundamental theoretical guarantees. Despite the apparent simplicity of the general concept of differential privacy, it seems non-trivial to get it right when applying it to NLP. In this short paper, we formally analyze several recent NLP papers proposing text representation learning using DPText (Beigi et al., 2019a,b; Alnasser et al., 2021; Beigi et al., 2021) and reveal their false claims of being differentially private. Furthermore, we also show a simple yet general empirical sanity check to determine whether a given implementation of a DP mechanism almost certainly violates the privacy loss guarantees. Our main goal is to raise awareness and help the community understand potential pitfalls of applying differential privacy to text representation learning.

## introduction differential privacy (dp), a formal mathematical treatment of privacy protection, is making its way to nlp @xcite . unlike other approaches to protect privacy of individuals' text documents, such as redacting named entities @xcite or learning text representation with a gan attacker @xcite , dp has the advantage of quantifying and guaranteeing how much privacy can be lost in the worst case. however, as @xcite showed, adapting dp mechanisms to nlp properly is a non-trivial task. representation learning with protecting privacy in an end-to-end fashion has been recently proposed in dptext @xcite . dptext consists of an autoencoder for text representation, a differential-privacy-based noise adder, and private attribute discriminators, among others. the latent text representation is claimed to be differentially private and thus can be shared with data consumers for a given down-stream task. unlike using a predetermined privacy budget ε, dptext takes ε as a learnable parameter and utilizes the reparametrization trick (kingma and welling, 2014) for random sampling. however, the downstream task results look too good to be true for such low ε values. we thus asked whether dptext is really differentially private. this paper makes two important contributions to the community. first, we formally analyze the heart of dptext and prove that the employed reparametrization trick based on inverse continuous density function in dptext is wrong and the model violates the dp guarantees. this shows that extreme care should be taken when implementing dp algorithms in end-to-end differentiable deep neural networks. second, we propose an empirical sanity check which simulates the actual privacy loss on a carefully crafted dataset and a reconstruction attack. this supports our theoretical analysis of non-privacy of dptext and also confirms previous findings of breaking privacy of another system adept. 1 2 differential privacy primer suppose we have a dataset (database) where each element belongs to an individual, for example alice, bob, charlie, up to m. each person's entry, denoted with a generic variable x, could be an arbitrary object, but for simplicity consider it a real valued vector @xmath0. an important premise is that this vector contains some sensitive information we aim to protect, for example an income (@xmath1, a binary value whether or not the person has a certain disease (@xmath2.0, 1.0}), or a dense representation from sentencebert containing the person's latest medical record (@xmath3. this dataset is held by someone we trust to protect the information, the trusted curator. 2 this dataset is a set from which we can create 2 m subsets, for instance x 1 = {alice}, x 2 = {alice, bob}, etc. all these subsets form a universe x , that is x 1 , x 2 , • • • ∈ x , and each of them is also called (a bit ambiguously) a dataset. definition 2.1. any two datasets x, @xmath4, if they differ in one person. for example, @xmath5, @xmath6, bob}, @xmath7, while @xmath8, @xmath9, bob, charlie} are not. definition 2.2. numeric query is any function f applied to a dataset x and outputting a realvalued vector, formally f : @xmath10. for example, numeric queries might return an average income (@xmath11, number of persons in the database (@xmath12, or a textual summary of medical records of all persons in the database represented as a dense vector (@xmath13. the query is simply something we want to learn from the dataset. a query might be also an identity function that just 'copies' the input, e.g., f (@xmath14, 0)}) → (1, 0) for a real-valued dataset @xmath15, 0)}. an attacker who knows everything about bob, charlie, and others would be able to reveal alice's private information by querying the dataset and combining it with what they know already. differentially private algorithm (or mechanism) m(x; f ) thus randomly modifies the query output in order to minimize and quantify such attacks. smith and ullman (2021) formulate the principle of differential privacy as follows: "no matter what they know ahead of time, an attacker seeing the output of a differentially private algorithm would draw (almost) the same conclusions about alice whether or not her data were used." let a dp-mechanism m(x; f ) have an arbitrary range r (a generalization of our case of numeric queries, for which we would have @xmath16. differential privacy is then defined as for all neighboring datasets x, x and all @xmath17, where pr(x) and pr(x ) is our prior knowledge of x and x . in words, our posterior knowledge of x or x after observing z can only grow by factor exp(ε) @xcite , where ε is a privacy budget @xcite . 3 ## analysis of dptext in the heart of the model, dptext relies on the standard laplace mechanism which takes a realvalued vector and perturbs each element by a random draw from the laplace distribution. formally, let z be a real-valued d-dimensional vector. then the laplace mechanism outputs a vector z such that for each index @xmath18, . . . , d the laplace mechanism satisfies differential privacy @xcite . ## empirical sanity check algorithm it is impossible to empirically verify that a given dp-mechanism implementation is actually dp @xcite . however, it is possible to detect a dp-violating mechanism with a fair degree of certainty. we propose a general sanity check applicable to any real-valued dp mechanism, such as the laplace mechanism, dptext, or any other. 5 we start by constructing two neighboring datasets x (alice) and x (bob) such that @xmath19, . . . , 0 n ) consists of n zeros and @xmath20, . . . , 1 n ) consists of n ones. the dimensionality @xmath21, 2, . . . } is a hyperparameter of the experiment. we employ a synthetic data release mechanism (also called local dp). the mechanism takes x or x and outputs its privatized version of the same dimensionality n, so that the zeros or ones are 'noisified' real numbers. the query sensitivity ∆ is n. 6 thanks to the post-processing lemma, any postprocessing of dp output remains dp. we can thus turn the output real vector back to all zeros or all ones, simply by rounding to closest 0 or 1 and applying majority voting. this process is in fact our reconstruction attack: given a privatized vector, we try to guess what the original values were, either all zeros or all ones. what our attacker is doing, and what dp protects, is that if alice gives us her privatized data, we cannot tell whether her private values were all zeros or all ones (up to a given factor); the same for bob. by definition (1) and having no prior knowledge over x and x apart from the fact that the values are correlated, our attacker cannot exceed the guaranteed privacy loss exp(ε): we can estimate the conditional probability pr(x|m(x; f ) = z) using maximum likelihood estimation (mle) simply as our attacker's precision: how many times the attacker reconstructed true x values given the observed privatized vector. we can do the same for estimating the conditional probability of x . in particular, we repeatedly run each dp mechanism over x and x 10 million times each, which gives very precise mle estimates even for small ε. note that it does not guarantee dp, the reconstruction attack might be just weak. area above the green line: the algorithm almost certainly violates dp as our attack caused bigger privacy loss than allowed by ε. extreme baselines show two extreme scenarios, as random output is absolutely private (but provides zero utility) and copy input provides maximal utility but no privacy by revealing the data in full. ## results and discussion for the sake of completeness, we implemented two extreme baselines: one that simply copies input (no privacy) and other one completely random regardless of the input (maximum privacy); these are shown in figure 2 left. the vanilla laplace mechanism behaves as expected; all empirical losses for all dimensions (1 up to 128) are bounded by ε. we re-implemented the laplace mechanism from adept @xcite that, due to wrong sensitivity, has been shown theoretically as dp-violating @xcite . we empirically confirm that adept suffered from the curse of dimensionality as the privacy loss explodes for larger dimensions. the last panel confirms our previous theoretical dptext results, which (regardless of dimensionality) has infinite privacy loss. note that we constructed the dataset carefully as two neighboring multidimensional correlated data that are as distant from each other as possible in the (0, 1) n space. however, dp must guarantee privacy for any datapoints, even the worst case scenario, as shown by the correct laplace mechanism. ## conclusion we formally proved that dptext @xcite @xcite is not differentially private due to wrong sampling in its reparametrization trick. we also proposed