THE - X : Privacy-Preserving Transformer Inference with Homomorphic Encryption (2022)

Abstract: As more and more pre-trained language models adopt on-cloud deployment, the privacy issues grow quickly, mainly for the exposure of plain-text user data (e.g., search history, medical record, bank account). Privacy-preserving inference of transformer models is on the demand of cloud service users. To protect privacy, it is an attractive choice to compute only with ciphertext in homomorphic encryption (HE). However, enabling pre-trained models inference on ciphertext data is difficult due to the complex computations in transformer blocks, which are not supported by current HE tools yet. In this work, we introduce THE-X, an approximation approach for transformers, which enables privacy-preserving inference of pre-trained models developed by popular frameworks. THE-X proposes a workflow to deal with complex computation in transformer networks, including all the non-polynomial functions like GELU, softmax, and LayerNorm. Experiments reveal our proposed THE-X can enable transformer inference on encrypted data for different downstream tasks, all with negligible performance drop but enjoying the theory-guaranteed privacy-preserving advantage.

## introduction accompanying the revolution of pre-trained models in many nlp applications, such as sentiment analysis @xcite , question answering @xcite , information retrieval @xcite , and text generation @xcite , many related technologies have been deployed on the cloud to process user data from personal customers, small businesses, and large enterprises by industrial service providers. however, the convenience of the on-cloud pretraining technology also comes with a series of contribution during internship at msra. u the corresponding author is jianxin @xmath0.edu.cn>. privacy challenges due to the sensitive nature of user data. for example, the input text or even text vector representations in user requests can leak private information, which may cause the specific user to be identified @xcite . this lack of privacy guarantees may impede privacy-conscious users from releasing their data to service providers. thus, service providers may suffer from the deficiency of evolving models with user data. besides, unintended data disclosure and other privacy breaches may result in litigation, fines, and reputation damages for service providers. these concerns spark our proposal of the-x, to enable privacy-preserving inference of transformer. specifically, we identify two challenges for the privacy-preserving inference of pre-trained models. the first challenge is how to protect users' plain text data from access by third-party service providers. (e.g., the clinic record or shopping history). prior work has applied differential privacy (dp) @xcite and its variants to address similar privatization issues -originally for statistical databases and more recently for dl @xcite and nlp @xcite @xcite @xcite . however, this solution may suffer from eavesdropping attackers. a handful of research @xcite demonstrated it possible to recover raw data from gradient leakage. also, privacy protection could never be theory-guaranteed. the second challenge is the performance concern, recent works like tex-thide @xcite and @xcite leverages the federated learning @xcite to train model on encrypted data, at cost of considerable performance dropping. focusing on the privacy of training data, they have not fully explored privacy-preserving inference. to solve the concerns above, we depict one practice of privacy-preserving inference in figure 1 , where a fine-tuned language model could be converted into the cloud service mode with the-x, and process users' data with its eyes blind. during inference, the content of the user query is anonymous to the transformer model. the results of computation are also ciphertext, which only can be decrypted by the user's private key. in addition, we need a theory-guaranteed encryption solution like the homomorphic encryption (he) @xcite to convince both service providers and users of the privacy security in production scenarios. the semantic security of he is guaranteed by lattice-based cryptography, and the he computation results on ciphertext could be decrypted to the same results in plaintext, preventing performance reduction cost. the basic idea of homomorphic encryption is to perform computations on encrypted data without first decrypting it, which could fully ensure privacy in cloud-serving scenarios. it allows user data to be encrypted and out-sourced to commercial cloud environments for processing. however, due to the complex operations (e.g., gelu activation) in transformer-based models, the popular partially homomorphic encryption solution, which only supports addition or multiplication, can not easily be adapted into scenarios of pre-trained models. based on he transformer backend @xcite , we designed a series of approximation components to fulfill the whole inference pipeline of the mainstream transformer backbone. we evaluate the-x for berttiny on the glue benchmark @xcite and the conll2003 task @xcite . our results show that the-x can achieve the privacy-preserving inference with the averaged performance reduction of only 1.49%. our contributions include: ## the-x: formal description there are two core ideas in the-x. the first one is to incorporate the user device into the he inference, and the second is using "simplified computation" to approximate the non-polynomial functions. in the following, we will describe how to enable homomorphic encryption of transformer-based models with the-x. ## experiments in this section, we design both sequence-level and token-level tasks to evaluate the approximation performance of our the-x solution. we also discuss several identified factors which greatly affect approximation workflow. ## conclusions we present the-x, a practical approach to enable pre-trained transformer models to infer under homomorphic encryption. it requires several approximation components to replace the original operations in the transformer model. it imposes a slight burden in terms of performance cost but enjoys the full advantage of homomorphic encryption -the theory-guaranteed user privacy. we see this as a first step in combing homomorphic encryption to address emerging privacy issues in pre-trained models. we hope our work motivates further research, including better approximation solutions on different nlp applications.