Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption (2022)

Abstract: Embeddings, which compress information in raw text into semantics-preserving low-dimensional vectors, have been widely adopted for their efficacy. However, recent research has shown that embeddings can potentially leak private information about sensitive attributes of the text, and in some cases, can be inverted to recover the original input text. To address these growing privacy challenges, we propose a privatization mechanism for embeddings based on homomorphic encryption, to prevent potential leakage of any piece of information in the process of text classification. In particular, our method performs text classification on the encryption of embeddings from state-of-the-art models like BERT, supported by an efficient GPU implementation of CKKS encryption scheme. We show that our method offers encrypted protection of BERT embeddings, while largely preserving their utility on downstream text classification tasks.

## introduction in recent years, the increasingly wide adoption of vector-based representations of text such as bert, elmo, and gpt @xcite @xcite , has called attention to the privacy ramifications of embedding models. for example, @xcite ; @xcite show that sensitive information such as the authors' gender and age can be partially recovered from an embedded representation of text. @xcite report that bert-based sentence embeddings can be inverted to recover up to 50%-70% of the input words. previously proposed solutions such as differential privacy based on perturbation/noise @xcite , require manually controlling the noise injected into embeddings, to control the privacyutility trade-off to a level suitable for each downstream task. in this work, we propose a privacy solution based on approximate homomorphic encryption, which is able to achieve little to no accuracy loss of bert embeddings on text classification foot_0 , while ensuring a desired level of encrypted protection, i.e. 128-bit security. homomorphic encryption (he) is a cryptographic primitive that serves computations over encrypted data without any decryption process. while previous works have focused on homomorphic computation where the inputs are numerical data, in applications such as privacy-preserving machine learning algorithms @xcite , logistic regression @xcite , and neural network inference @xcite , they have rarely been applied to unstructured data such as text. recent works in this direction include @xcite , who conduct sentiment classification over encrypted word embeddings using rnn. however, they use a simple embedding layer which maps words in a dictionary to real-valued vectors, and model training is only supported on plaintext. the most closely related work to ours is @xcite , a homomorphic encryption based method for privacy preserving text classification built on fasttext @xcite . we next describe our approach, focusing on our distinctions from privft: we introduce a new method for text classification on encrypted data. the crux is to operate a simple downstream classifier on encryptions of semantically rich vector representations (i.e. bert embeddings). by using rich input representations, our method significantly outperforms privft, while the use of a simple downstream classifier on encrypted data makes our method much more practical. importantly, by leveraging pretrained embeddings from models such as bert, a state-of-the-art in many nlp tasks, our method enables the training of a strong classifier in encrypted state within hours. as such, our method is well positioned to take full advantage of the recent trends in nlp, that rely on the language understanding capability of increasingly larger pre-trained language models @xcite . we develop an efficient gpu implementation of ckks which greatly improves computation speed. while privft also provides a gpu implementation of ckks, their implementation lacks the bootstrapping operation of ckks. inevitably, this limits the multiplicative depth of privft, and it makes the method less scalable. it also results in the use of less secure ckks parameters which have roughly 80-bit security level. in contrast, our gpu implementation includes the bootstrapping operation, which allows unlimited number of multiplications. this enables us to use a higher degree polynomial approximation (which is key to achieving a high downstream accuracy), and more secure ckks parameters (128-bit security level foot_1 ). moreover, with practicality in mind, we improved the implementation in terms of communication cost. more precisely, we introduce a practical implementation of ckks to significantly reduce the size of ciphertexts by more than 7.4× compared to the rudimentary implementation. we experimentally validate our approach on text classification datasets, showing that it offers encrypted protection of embedding vectors, while maintaining utility competitive to plaintext on downstream classification tasks. additionally, we compare our method with privft on homomorphic training on encrypted data, showing it outperforms privft, with much improved training efficiency. ## method we focus on the scenario in which the user directly applies the privacy mechanism to the output embeddings from a neural text encoder, before passing it on to a service provider for usage in a downstream task. this is also referred to as a local privacy setting @xcite . the privatization procedure m priv in the local privacy setting can be defined as follows: ## results we report the results of our logistic regression experiments in we next perform sentence embedding inversion experiments on snips. in ## conclusion we propose a privatization mechanism based on homomorphic encryption which, by leveraging bert pre-trained embeddings, enables efficient training of an he logistic regression classifier with little to no loss of downstream utility. while our method compares favorably to noise-based ldp and privft, we also note that there are some limitations. since he based models require higher computation costs compared to plaintext models, the challenge remains to adopt more complex models, such as neural networks, as downstream classifiers. nevertheless, the privacy benefits and efficiency of our method makes it a suitable candidate for scenarios with real-world privacy concerns.