Reconstruction Attack on Instance Encoding for Language Understanding (2021)

Abstract: A private learning scheme TextHide was recently proposed to protect the private text data during the training phase via so-called instance encoding. We propose a novel reconstruction attack to break TextHide by recovering the private training data, and thus unveil the privacy risks of instance encoding. We have experimentally validated the effectiveness of the reconstruction attack with two commonly-used datasets for sentence classification. Our attack would advance the development of privacy preserving machine learning in the context of natural language processing.

## introduction with the development of deep learning technologies, a large number of applications in various domains (e.g., image classification and nlp) have been greatly promoted with significantly improved performance. however, this also arouses serious privacy concerns since a large portion of the training data are usually collected from individuals. for instance, the diagnosis systems in hospitals or healthcare institutions will be trained on the patients' private data, such as medical history @xcite , and radiology medical images @xcite . in addition, it has been reported that the input keyboard prediction model can be trained with the users' data on mobile devices @xcite , and the assisted composing function for emails/texts can be trained with users' personal messages @xcite . there have been various works on protecting users' data privacy during the training, which are categorized into two main types: 1) composing cryptographic protocols for securely training the data @xcite @xcite which result in high computational and communication costs in general; 2) leveraging the differential privacy techniques @xcite @xcite to prevent the information leakage which typically cause significant accuracy loss. despite the above demerits, both types of methods can ensure provable privacy guarantees for the training data. this also raises the question: are there any private learning schemes which can preserve both accuracy and efficiency? to this end, there are several techniques @xcite which privately train the model via the so-called instance encoding scheme, by encoding the local data into a somewhat "encrypted" (encoded) data with a mixup scheme @xcite , and directly training the model on the encoded data. data privacy is claimed to be well preserved through the encoding method while only causing minor accuracy loss with the merit of the mixup scheme. in this paper, we study the privacy risks of the instance encoding scheme, and show that the instance encoding cannot provide sufficient privacy protection as the conventional cryptographic techniques against well-designed attacks. specifically, we design a reconstruction attack to recover the original data from the privately encoded data. we focus on the instance encoding in language understanding, i.e., texthide @xcite as the state-of-the-art technique. ## texthide the texthide @xcite aims to protect the private text data under the federated learning setting. first, the input text is pre-processed with a bert transformer encoder to output the corresponding text representation. then, for "encryption", texthide will apply the instance encoding to mix up the original text representation with some randomly selected text (representations), which will be fed into the training model of various downstream language understanding tasks, e.g., classification, and question answering. formally, given the input text x i with the label y i , we denote the text representation as e @xmath0, where φ(•) is a pre-tuned bert model. the private instance encoded data e i can be generated as below: furthermore, texthide can utilize another dataset x public (usually a large public corpus, e.g., wikipedia) for mixup, where such mixup works similar to a random oracle in the cryptography domain. 1 specifically, texthide will mix up about one half k/2 public data with the private original data, then equation 1 is updated as: in practice, given the original training dataset (denoted as x), each data (x i , y i ) ∈ x will be encoded for n times (usually equal to the number of training epochs). ## attack preliminaries privacy-enhancing schemes. as mentioned before, both cryptographic protocols and differential privacy can provide provable privacy guarantees for protecting the private data. on the one hand, for cryptographic solutions, the data is usually protected by the encryption schemes, e.g., fully homomorphic encryption (fhe) @xcite , where the security of schemes depends on some hard mathematical problems. normally, to prove the security of the encryption scheme, we need to formulate a security game, e.g., ind-cpa @xcite , where an attacker with repeating many operations polynomially (w.r.t. the size of the security parameter) cannot do better than randomly guessing. it should be noted that the newly proposed instance encoding schemes are claimed to work as the encryption scheme for privacy protection @xcite , but fail to provide such provable security guarantees. on the other hand, differential privacy @xcite @xcite can statistically protect the individual information from being identified (i.e., against identification attacks @xcite ) by injecting wellcalibrated noise to the original values. for example, differential privacy can help to defend against socalled membership inference attacks @xcite in the machine learning such that an attacker cannot determine whether any specific individual information is in the dataset or not. privacy attacks. the attacks on the data privacy in ml training are generally referred to membership inference attacks @xcite @xcite @xcite , where an adversary can know whether a given data points was used to train the model or not. in addition, model inversion attacks @xcite @xcite can reconstruct a group of representative data points from the training set, e.g., utilizing gradients @xcite . our attack on texthide works closely as the reconstruction attack @xcite , which aims to reconstruct the original data/information from the protected data (privately encoded data). note that carlini et al. @xcite attacks the instance encoding on images while we extend this method to the language understanding domain. attack setting. we assume that the attacker have full knowledge of the public dataset x public and the embedding model for downstream ml tasks. besides, we assume that the attacker can obtain the private dataset (but unaware of the specific data for the training). note that we need to consider the worst case (attacker) to evaluate the vulnerabilities of the privacy-enhancing schemes. that is, the strong knowledge (e.g., embedding model and private training dataset) can be accessed by a skilled attacker armed with any background knowledge. for instance, such private training dataset can be machine-generated. specifically, if the dataset involves personal conversations, then the attacker can utilize some language models to generate a large set of commonly-used dialogs as the private training dataset. the attacker can also leverage some advanced inference attacks, e.g., side-channel or public essays to derive some sentences. ## results and analysis ## discussion privacy preserving machine learning (ppml) has been popular in industries under more and more restrictive data actions or regulations, e.g., general data protection regulation (gdpr) in european union. ppml could help the corporations improve business continuity while machine learning2 1 2 3 2 5 2 7 2 9 2 based services deal with large amounts personal data/information, including text data-based applications such as the keyboard input prediction @xcite . private instance encoding (e.g., tex-thide) has been proposed to address privacy risks in such application scenarios. however, weak privacy guarantees provided by texthide (e.g., against our proposed attack) may leak the personal information, and also violate privacy regulations and laws. this would cause severe sanctions and lose enterprise reputation from their customers. as depicted earlier, a well-designed privacyenhancing scheme must ensure provable privacy guarantee, and show its performance on data protection. since texthide is based on such mixup encoding method, it would be possible to apply differential privacy @xcite to the mixup encoding and thus to show similar indistinguishability of the privately encoded instances. this can defend against our reconstruction attacks to some extent (at least reducing the information disclosure). another possible defense method is to filter sensitive data from the training data. however, this might degrade the model performance. it is also worth noting that the intrinsic property of dnn model (i.e., memorization) can also be utilized to extract/recover training data from model itself, especially for language models @xcite . such works are orthogonal with our proposed attack since we focus more on the encoded data. nevertheless, our attack can be integrated with such attacks to be more powerful on instance encoding schemes. ## conclusion we proposed a novel reconstruction attack on a recent private learning scheme, texthide in the nlp domain. we have experimentally shown that such scheme cannot provide rigorous privacy guarantee even though it obtains good accuracy.