Differentially Private Instance Encoding against Privacy Attacks (2022)

Abstract: TextHide was recently proposed to protect the training data via instance encoding in natural language domain. Due to the lack of theoretic privacy guarantee, such instance encoding scheme has been shown to be vulnerable against privacy attacks, e.g., reconstruction attack. To address such limitation, we revise the instance encoding scheme with differential privacy and thus provide a provable guarantee against privacy attacks. The experimental results also show that the proposed scheme can defend against privacy attacks while ensuring learning utility (as a trade-off).

## introduction machine learning models have been widely deployed in a wide range of applications/domains, such as speech recognition @xcite , computer vision @xcite and natural language processing @xcite @xcite . meanwhile, the privacy issues have also aroused more and more attention as machine learning-based systems usually aggressively collect large amounts of data for better performance, which could contain user's personal information and thus jeopardize user's privacy. for instance, the hospital admission information and diagnosis report can be processed by language models to predict the readmission rate of a patient @xcite . another example is that the prediction of keyboard input would require personal users' daily input texts for better accuracy @xcite . this may not only lose customer trust, but also violate some data regulations or laws, e.g., gdpr @xcite . the privacy-enhancing technologies (pets) @xcite @xcite have been widely studied to ensure the data privacy in the machine learning, which mainly include two foundations of theory as following. first, the cryptographic protocols @xcite can help to securely train the model with the private data (in encrypted format), and the privacy of data depends on the hard mathematical problems @xcite . although the cryptographic protocolbased schemes provide good data privacy, these also arouse high computational overheads due to the computation on encrypted data and other complicated building blocks. second, differential privacy (dp) @xcite provides a lightweight way to protect the data against the adversaries with arbitrary information during the training, which can obtain quantifiable privacy guarantees. for example, the widely used dp-sgd @xcite ensures the privacy of training data sample by clipping the gradients and adding dp noise (e.g., gaussian mechanism) with the model updates. the introduction of dp noise enables the limited effect of one individual data on the trained model (and thus achieving the privacy guarantee). additionally, another category of work is to add dp noise into the dataset following the method of dp synthetic data release and then train a model on such private data @xcite ). yet, the differential privacy-based learning schemes could cause great accuracy loss. alternatively, a private learning scheme called instance encoding @xcite has been proposed to obtain both privacy and utility for model training, which encodes the private data into "encrypted" data via mixup @xcite . while the privacy is claimed to be guaranteed by the encoding scheme, the data utility can be maintained by mixup scheme, only causing minor accuracy loss. however, it has been shown that such instance encoding scheme cannot provide strong privacy guarantee as cryptographic protocols (mohassel and rindal, 2018) or differential privacy @xcite against privacy attacks empiri-cally @xcite . that is, well-designed privacy attacks @xcite can break the instance encoding scheme to reconstruct the original data from the encoded data with high success rates. to address the privacy issue, we improve the texthide with differential privacy and prove the improved scheme ensures theoretical privacy guarantee under the differential privacy framework. besides, the experimental results validate the performance of proposed scheme. ## preliminaries of differential privacy as one main category of privacy-enhancing technologies, differential privacy (dp) @xcite has been widely used as a de facto standard notion in protecting individual's data privacy for data collection and analysis @xcite , especially in machine learning applications @xcite . the principle of the differential privacy @xcite is that an individual's data point x in one dataset d will not arouse significant change to the outcome of a randomized mechanism or algorithm applied to the d. thus, the attacker cannot make difference with such a specific data point x by observing the outputs of d by the randomized mechanism, which thus provides deniability for the existence of x (ensuring data privacy). formally, to define individual's privacy, we first define the neighboring datasets, i.e., d, d ′ ∈ d are the neighbors if they only differs in one data point, denoted as d ∼ d ′ . then we define the dp notation as following: definition 1 (differential privacy @xcite ). for any two neighboring datasets, d, d ′ ∈ d, a randomized mechanism m is said to be (ϵ, δ)-differentially private if it satisfies the following equation: in this work, we will utilize the laplace and gaussian mechanisms to guarantee (ϵ, δ)-dp. the laplace mechanism @xcite ) adds the noise from laplace distribution with mean zero and scale parameter b, denoted as lap(b) with theorem 2 (gaussian mechanism @xcite ). given any function f : @xmath0, the gaussian mechanism is defined as m g (d, f, ϵ) = f (d) + n , where n is the random noise drawn from gaussian distribution n (0, σ 2 i d ) with σ ≥ ∆f 2 ln (1.25/δ)/ϵ. ∆f is the ℓ 2 sensitivity of function f , i.e., ℓ 2 = sup d∼d ′ ||f (d)f (d ′ || 2 . guassian mechanism satisfies (ϵ, δ)-dp. ## dp instance encoding given a training batch of data samples of size m @xmath1, y 1 ), (x 2 , y 2 ), • • • , (x i , y i )}, @xmath2, m ], which is randomly sampled from the training set. texthide will first encode every sample into a feature vector of dimension size d by a pretrained feature extractor ϕ(•), i.e., v @xmath3. then we can get the corresponding batch of encoded feature vectors b @xmath4, y 1 ), (v 2 , y 2 ), • • • , (v n , y n )}. for original instance encoding, texthide would mixup such set of size k vectors to generate private encoded vectors as training data per equation 1 . to address the privacy issue, we apply the differential private mechanism to such mixup process. algorithm 1 demonstrates the details. ## experimental evaluation for experiments, we would like to evaluate both utility and privacy of the proposed scheme as the following: 1) utility of the private instance encoding scheme, i.e., the performance (accuracy) of model trained on the private dataset; 2) privacy guarantee of the scheme against reconstruction attacks, i.e., the attack success rate (the percentage of reconstructed private vectors). ## conclusion & future work in this paper, we facilitate the instance encoding scheme with differential privacy. we have theoretically proven that the revised instance encoding with dp mechanism could provide good privacy guarantee under differential privacy framework. experimental results have shown that the proposed differentially private scheme can obtain good utility for downstream learning tasks, e.g., text classification. besides, we also evaluate the proposed dp scheme against privacy attacks and the results show that the scheme can ensure the privacy of dataset while presenting with attacks. for the future work, we would like to further revise current dp instance encoding with another differential privacy notion, i.e., rényi differential privacy @xcite , which generalizes the concept of differential privacy based on the rényi divergence. that is, revising the instance encoding scheme with rényi dp would derive a tighter privacy bound and thus achieve better privacyprotection. besides, another potential direction is to rescale the text vectors (generated by language feature extractor model) to a lower dimension vector by an extra mlp model or auto-encoder @xcite . we can utilize composition theorem @xcite in dp to theoretically find a better guarantee for various downstream tasks.