Original: MPCF ORMER :

Original: F AST ,

Original:  PERFORMANT AND PRIVATE T RANSFORMER INFERENCE WITH MPC
Revised: MPCF ORMER : F AST , PERFORMANT AND PRIVATE T RANSFORMER INFERENCE WITH MPC

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Enabling private inference is crucial for many cloud inference services that are based on Transformer models.
Revised: Enabling private inference is crucial for many cloud inference services that are based on Transformer models.

Original: However, existing private inference solutionsfor Transformers can increase the inference latency by more than 60 × or significantly compromise thequality of inference results.
Revised: However, existing private inference solutions can increase the inference latency by more than 60 × or significantly compromise the inference quality.

Original: In this paper, we design the framework MPCF ORMER  using secure multi-party computation (MPC) and 
Revised: In this paper, we design the framework MPCF ORMER as a practical solution, using Secure Multi-Party Computation (MPC) and Knowledge

Original: Knowledge Distillation (KD).
Revised: Distillation (KD).

Original: It can be used in tandem with many specifically designed MPC-friendly approximations and trained Transformer models.

Original:  MPCF ORMER significantly speeds up Transformermodel inference in MPC settings while achieving similar ML performance to the input model.We evaluate MPCF ORMER with various settings in MPC.
Revised: Through extensive evaluations, we show that MPCF ORMER significantly speeds up Transformer inference in MPC settings while achieving similar ML performance to the input model.

Original: On the IMDb dataset, we achieve similar performance to BERT BASE , while being 5.3 × faster.
Revised: On the IMDb dataset, it achieves similar performance to BERT BASE , while being 5.3 × faster.

Original: On the GLUE benchmark, we achieve 97% performance of BERT BASE with a 2.2 × speedup.
Revised: On the GLUE benchmark, it achieves 97% performance of BERT BASE with a 2.2 × speedup.

Original: We show that MPCF ORMER remains effective with different trained Transformer weights such as ROBERTA BASE and larger models including BERT Large .
Revised: MPCF ORMER remains effective with different trained Transformer weights such as ROBERTA BASE and larger models including BERT Large .

Original: In particular, we achieve similar performance to BERT LARGE , while being 5.93 × faster on the IMDb dataset.
Revised: Code is available at https://github.com/MccRee177/MPCFormer .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Pre-trained Transformer models can adapt to various downstream tasks with high performance, then be deployed as model inference services (Bommasani et al., 2021;
Revised: Pre-trained Transformer models can be easily fine-tuned on various downstream tasks with high performance and have been widely developed as model inference services (Bommasani et al., 2021;

Original: Feng et al., 2020; Yang et al., 2019b; Clark et al., 2020).
Revised: Feng et al., 2020; Yang et al., 2019b; Clark et al., 2020).

Original: However, such model inference services can pose privacy concerns.
Revised: However, these model inference services can pose privacy concerns.

Original: For instance, GitHub Copilot, a code-generating engine adapted from pre-trained GPT weights (Chen et al., 2021; Brown et al., 2020), requires either users to reveal their code prompts to the service provider for code generation, or the service provider to release the Copilot’s trained weights, which are business proprietary, to users.
Revised: For instance, GitHub Copilot, a code-generating engine adapted from pre-trained GPT weights, requires either users to reveal their code prompts to the service provider, or the service provider to release the Copilot’s trained weights, which are business proprietary, to users (Chen et al., 2021; Brown et al., 2020).

Original: Secure Multi-Party Computation (MPC) offers a promising solution by keeping data and model weights private during inference (Evans et al., 2018).
Revised: Secure Multi-Party Computation (MPC) offers a promising solution by keeping data and model weights private during inference (Evans et al., 2018).

Original: However, the vanilla Transformer inferencecomputation in MPC is unacceptably slow.
Revised: However, the vanilla Transformer inference in MPC is unacceptably slow.

Original: For instance, BERT BASE  takes < 1 second without MPC, but ∼ 60 seconds with MPC (Figure 2).
Revised: For instance, BERT BASE inference takes < 1 second without MPC, but ∼ 60 seconds with MPC (Figure 2).

Original: Prior work on convolutional neural networks (CNNs) has shown that by replacing computational operations with  faster approximations (we call these MPCfriendly approximations), the inference process in MPC can be accelerated (Mohassel & Zhang, 2017).
Revised: An intuitive way to accelerate MPC inference replaces computational operations with their faster approximations and retrains the approximated model, which has been adopted on convolutional neural networks (CNNs)

Original: However, such a simple replacement strategy drastically decreases the models’ quality (Chou et al., 2018). 
Revised: (Chou et al., 2018). Unfortunately, adapting this solution to Transformers drastically decreases the model’s performance (§ 5).

Original: In this paper, we take the first step to tackle the research problem: How to perform privacypreserving Transformer model inference in MPC, while remaining fast and performant? In particular, we present a solution for performing privacy-preserving Transformer model inference using
Revised: In this paper, we take the first step to pursue privacy-preserving Transformer model inference in MPC , while remaining fast and performant . We take inspiration from the approximation approachand attribute the performance degradation to two challenges.

Original: MPC. Our solution is simple and effective, and allows for various MPC-friendly approximations and
Revised: First, many MPC-friendly approximations toughen model training. For example, quadratic functions cause the gradient explosion problem in deep neural networks (Mishra et al., 2020).

Original: Transformer weights to be used.
Revised: Second, downstream datasets used for

Original: We explore a new two-stage approach to making transformer inference fast in MPC. Drawing insights from existing private inference methods for CNNs, we first validate that using MPC-friendly approximations can also bring speedups to  Transformermodels. More concretely, we benchmark the transformer inference process with an MPC system (Knott et al., 2021) and find that the major bottlenecks are the GeLU and Softmax functions (Wang et al., 2022).
Revised: Transformer fine-tuning usually contain insufficient data to retrain an approximated Transformer with common task objectives (Zhang & Sabuncu, 2018; Hinton et al., 2012). To address these two challenges, we resort to the knowledge distillation (KD) framework.

Original: Replacing them with off-theshelf MPC-friendly approximations significantly speeds up the whole process (See § 4.2.1).
Revised: KD can ease the model training by matching intermediate representations between the teacher and the student model (Romero et al., 2014); this intermediate supervision can alleviate the gradient explosion

Original: Pre-trainedTransformer
Revised: Pre-trainedTransformer

Original: Weights , e.g. ,
Revised: Weights , e.g. ,

Original: GPTTeacher:
Revised: GPTTeacher:

Original: Fine-tuned
Revised: Fine-tuned

Original: Transformer
Revised: Transformer

Original: Down-stream Datasets
Revised: Down-stream Datasets

Original: Inference
Revised: Inference

Original: MPCFormer
Revised: MPCFormer

Original: Student:
Revised: Student:

Original: Approximated
Revised: Approximated

Original: Transformer
Revised: Transformer

Original: Stage1: Plugging- in MPC-friendly
Revised: Stage1: Plugging- in MPC-friendly

Original: Approximations
Revised: Approximations

Original: Stage2:
Revised: Stage2:

Original: KnowledgeDistillation
Revised: KnowledgeDistillation

Original: Distilled
Revised: Distilled

Original: Approximated
Revised: Approximated

Original: Transformer
Revised: Transformer

Original: MPC Engine, e.g. , Crypten
Revised: MPC Engine, e.g. , Crypten

Original: [Figure]
Revised: [Figure]

Original: Service
Revised: Service

Original: Provider
Revised: Provider

Original: User
Revised: User

Original: Private Data
Revised: Private Data

Original: MPC Engine
Revised: MPC Engine

Original: Secret
Revised: Secret

Original: Sharing

Original: In the second stage, we seek to make the fast approximated Transformer performant.

Original: Unlike existing methods, we validate that simply training the fast approximated architecture is not performant (§ 5).

Original: There are two possible causes: (1) many MPC-friendly approximations make models harder to train.

Original: For instance, quadratic functions are fast in MPC but they cause the gradient explosion problem in deep neural networks (Mishra et al., 2020).

Original: (2) Downstream datasets usually contain a small amount of data, which is insufficient for task-specific objectives such as the cross entropy loss to train a good model Zhang & Sabuncu (2018); Hinton et al. To address these two causes, we resort to the knowledge distillation (KD) framework. Firstly, KD can ease the model training by matching intermediate representations between the teacher and the student model (Romero et al., 2014). Specifically, prior work has shown that intermediate supervision can alleviate the gradient explosion problem (Lee et al., 2015).
Revised: Sharing problem (Lee et al., 2015).

Original: In our use case, we formulate the input Transformer model as the teacher, the approximated Transformer model as the student, and provide layer-wise distillation.

Original: In addition, prior work has shown that KD  is data-efficient (Touvron et al., 2021). We empirically validate that this property allows the approximated Transformermodel to learn well on small downstream datasets (§ 5.3).
Revised: At the same time, the KD objective is data-efficient and allows training an approximated Transformer on small downstream datasets (Touvron et al., 2021).

Original: Our approach.
Revised: Our approach and contributions.

Original: In this paper, we build MPCF ORMER , an easy-to-adopt framework for fast, performant, and private Transformer inference.
Revised: In this paper, we build MPCF ORMER , an easy-to-adopt framework for privacy-preserving Transformer inference.

Original: MPCF ORMER is compatible with many MPC-friendly approximations and  trained Transformer models.
Revised: MPCF ORMER takes in an MPC-friendly approximation and a trained Transformer.

Revised: It returns a Transformer with low inference latency in MPC and high ML performance simultaneously.

Original: It first replaces bottleneck functions in the input Transformer model with the given MPC-friendly approximations.
Revised: To do so, MPCF ORMER first replaces bottleneck functions in the input Transformer model with the given MPC-friendly approximations.

Original: The resulting approximated Transformer model has a faster inference speed in the MPC setting.
Revised: The resulting approximated Transformer model has a faster inference speed in MPC.

Original: MPCF ORMER then applies knowledge distillation to  the approximated Transformer model, by using  the  input performant Transformer model as the teacher. The intermediate supervision and the data efficient property allows the approximated Transformer model to learn well with downstream datasets.
Revised: Next, it applies knowledge distillation to train the approximated Transformer with high performance, using teacher guidance from the original input Transformer.

Original: Finally, the model provider can use the distilled approximated Transformer on top of an MPC engine, e.g., Crypten for private model inference service, achieving high inference speed and high ML performance simultaneously.
Revised: Finally, the model provider can use the distilled approximated Transformer on top of an MPC engine, e.g., CrypTen, for private model inference service.

Original: The overall workflow of MPCF ORMER is shown in Figure 1.Our contributions.
Revised: The overall workflow of MPCF ORMER is shown in Figure 1.

Original: Our contributions are three-fold. We propose MPCF ORMER , a two-stage framework that enables fast and performant private Transformer model inference with MPC by allowing many MPC-friendly approximations and trained Transformer models to be plugged in. We improve Transformer inference speed by leveraging our framework with an MPC system, by plugging in MPC-friendly approximations and trained Transformer models.
Revised: We implement MPCF ORMER on an MPC system (Knott et al., 2021), with various MPC-friendly approximations.

Original: In the process, we  design a new and faster MPC-friendly approximation to the Softmax function.
Revised: In the process, we also design a new and faster MPC-friendly approximation to the Softmax function.

Original: We extensively evaluate the framework with plugged-in approximations and trained Transformers in the MPC setting.
Revised: We extensively evaluate our implementation with various Transformer models.

Original: On the IMDb benchmark, we achieve similar ML performance to BERT BASE with  5.3 × speedup.
Revised: On the IMDb benchmark, MPCF ORMER achieves similar ML performance to BERT BASE with a 5.3 × speedup.

Original: We achieve similar ML performance to BERT LARGE with  5.9 × speedup.
Revised: It achieves similar ML performance to BERT LARGE with a 5.9 × speedup.

Revised: On the

Original: On the GLUE benchmark, we achieve 97% performance of BERT BASE with  2.2 × speedup.
Revised: GLUE benchmark, it achieves 97% performance of BERT BASE with a 2.2 × speedup.

Original: MPCF ORMER is also effective when plugged in with different trained Transformer models, e.g., RoBERTa BASE .
Revised: MPCF ORMER is also effective when given different trained Transformer models, e.g., RoBERTa BASE .

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this section, we first describe the Transformer model.
Revised: In this section, we first describe the Transformer model.

Original: Then we describe how functions in Transformer models can be implemented in MPC, and point out performance bottlenecks.
Revised: Then we describe how functions in Transformer models can be implemented in MPC, and analyze performance bottlenecks.

Original: 2. T RANSFORMER MODELS
Revised: 2. T RANSFORMER MODELS

Original: An n -layer Transformer model consists of three components: (1) The embedding layer.
Revised: An n -layer Transformer model consists of three components: (1) The embedding layer.

Original: (2) A stack of n Transformer layers.
Revised: (2) A stack of n Transformer layers.

Original: (3) The prediction layer.
Revised: (3) The prediction layer.

Original: The embedding layer maps a token (e.g. a word or an image patch) to a hidden representation (Devlin et al., 2018; Dosovitskiy et al., 2020).
Revised: The embedding layer maps a token (e.g. a word or an image patch) to a hidden representation (Devlin et al., 2018; Dosovitskiy et al., 2020).

Original: One Transformer layer consists of an attention module and several matrix multiplication modules (Bahdanau et al., 2014).
Revised: One Transformer layer consists of an attention module and several matrix multiplication modules (Bahdanau et al., 2014).

Original: The prediction layer maps the output of the last Transformer layer to a task output ( e.g., a probability distribution for a classification task).
Revised: The prediction layer maps the output of the last Transformer layer to a task output ( e.g., a probability distribution for a classification task).

Original: A partial illustration of a Transformer model can be found in Figure 3.
Revised: A partial illustration of a Transformer model can be found in Figure 3.

Original: 2. T RANSFORMER MODELS IN MPC
Revised: 2. T RANSFORMER MODELS IN MPC

Original: The Transformer model inference process can be formulated as a 2-Parties Computation (2PC).
Revised: The Transformer model inference process can be formulated as a 2-Parties Computation (2PC).

Original: In 2PC, the user party inputs the data, and the model provider party inputs the Transformer model.
Revised: In 2PC, the user party inputs the data, and the model provider party inputs the Transformer model.

Original: They jointly compute an inference result.
Revised: They jointly compute an inference result.

Original: Throughout the entire inference process, 2PC guarantees both parties only know information about their own inputs and the result (Yang et al., 2019a).
Revised: Throughout the entire inference process, 2PC guarantees both parties only know information about their own inputs and the result (Yang et al., 2019a).

Original: We describe the secret sharing scheme as a mean to preserve privacy during the inference process (Damg ˚ ard et al., 2012;
Revised: We describe the secret sharing scheme as a mean to preserve privacy during the inference process (Damg˚ard et al., 2012;

Original: Goldreich et al., 2019).
Revised: Goldreich et al., 2019).

Original: Assuming that the user provides a number x as its input, the secret sharing scheme splits x into two numbers, x 1 and x 2 .
Revised: Assuming that the user provides a number x as its input, the secret sharing scheme splits x into two numbers, x 1 and x 2 .

Original: It then lets the user party hold x 1 and distributes x 2 to the model provider party.
Revised: It then lets the user party hold x 1 and distributes x 2 to the model provider party.

Original: There are two properties of x 1 and x 2 .
Revised: There are two properties of x 1 and x 2 .

Original: First, either x 1 or x 2 alone contains no information about x .
Revised: First, either x 1 or x 2 alone contains no information about x .

Original: This property allows the user to hide the actual value x from the model provider.
Revised: This property allows the user to hide the actual value x from the model provider.

Original: Second, together they reconstruct x .
Revised: Second, together they reconstruct x .

Original: For instance, xand x 2 add up to x : x = x 1 + x 2 .
Revised: For instance, xand x 2 add up to x : x = x 1 + x 2 .

Original: The second property allows joint computation.
Revised: The second property allows joint computation.

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: We take multiplication via Beaver triple as a joint computation example (Beaver, 1991).
Revised: We take multiplication via Beaver triple as a joint computation example (Beaver, 1991).

Original: In multiplication, the user party provides x and the model provider provides y , and they secret share x and y .
Revised: In multiplication, the user party provides x and the model provider provides y , and they secret share x and y .

Original: Thus, the user gets x 1 and y 1 ; the model provider gets x 2 , and y 2 .
Revised: Thus, the user gets x 1 and y 1 ; the model provider gets x 2 , and y 2 .

Original: Beaver triple assumes a triple c =
Revised: Beaver triple assumes a triple c =

Original: ab has been generated 1 .
Revised: ab has been generated 2 .

Original: The triple is also secret shared so that the user party gets c 1 , a 1 , b 1 , and the model provider gets c 2 , a 2 , b 2 .
Revised: The triple is also secret shared so that the user party gets c 1 , a 1 , b 1 , and the model provider gets c 2 , a 2 , b 2 .

Original: The user first computes ϵ 1 = x 1 − a 1 , δ 1 = y 1 − b 1 locally.
Revised: The user first computes ϵ 1 = x 1 − a 1 , δ 1 = y 1 − b 1 locally.

Original: The model provider similarly computes ϵ 2 = x 2 − a 2 , δ 2 = y 2 − b 2 locally.
Revised: The model provider similarly computes ϵ 2 = x 2 − a 2 , δ 2 = y 2 − b 2 locally.

Original: They communicate these four numbers and reconstruct ϵ
Revised: They communicate these four numbers and reconstruct ϵ

Original: =
Revised: =

Original: ϵ 1 + ϵ 2 , δ =
Revised: ϵ 1 + ϵ 2 , δ =

Original: δ 1 + δ 2 .
Revised: δ 1 + δ 2 .

Original: the user then use these two values to compute r 1 = c 1 + ϵb 1 + δa 1 + δϵ .
Revised: the user then use these two values to compute r 1 = c 1 + ϵb 1 + δa 1 + δϵ .

Original: The model provider computes r 

Revised: The model provider computes r

Original: =
Revised: =

Original: c
Revised: c

Original: 2 + ϵb 2 + δa 2 .
Revised: 2 + ϵb 2 + δa 2 .

Original: At this point, the multiplication result xy can be reconstructed by xy = r 1 + r 2 .
Revised: At this point, the multiplication result xy can be reconstructed by xy = r 1 + r 2 .

Original: There are two important observations in the multiplication example: (1) it does not leak information to the other party.
Revised: There are two important observations in the multiplication example: (1) it does not leak information to the other party.

Original: For instance, the user does not send x 1 to the model party.
Revised: For instance, the user does not send x 1 to the model party.

Original: Instead, it sends ϵ 1 = x 1 − a 1 where a 1 is a random mask; (2) it requires one extra round of communication compared to the multiplication without MPC.
Revised: Instead, it sends ϵ 1 = x 1 − a 1 where a 1 is a random mask; (2) it requires one extra round of communication compared to the multiplication without MPC.

Original: This partially explains why vanilla Transformer models are slow in MPC.
Revised: This partially explains why vanilla Transformer models are slow in MPC.

Original: In particular, functions in Transformers (e.g., nonlinear activation) can be mainly implemented by three routines 2 , i.e., addition, multiplication, and comparison.
Revised: In particular, functions in Transformers (e.g., nonlinear activation) can be mainly implemented by three routines 3 , i.e., addition, multiplication, and comparison.

Original: Any computational operations composed by these routines would result in extra complexity in MPC 3 .
Revised: Any computational operations composed by these routines would result in extra complexity in MPC 4 .

Original: Empirically, we show these complexities by running BERT BASE (Figure 2) and reporting communication statistics in Table 1 with a secret-sharing-based MPC system (Knott et al., 2021).
Revised: Empirically, we show these complexities by running BERT BASE (Figure 2) and reporting communication statistics in Table 1 with a secret-sharing-based MPC system (Knott et al., 2021).

Original: We observe that GeLU functions and Softmax functions in Transformer layers are the major sources of bottlenecks.
Revised: We observe that GeLU functions and Softmax functions in Transformer layers are the major sources of bottlenecks, which echoes findings in a concurrent study (Wang et al., 2022).

Original: GeLU( x ) = x × 12 (cid:104) 1 + erf (cid:16) x √ 2 (cid:17)(cid:105) is slow because the Gaussian Error function erf( · ) is evaluated by a high order Taylor expansion, which requires many multiplication routines.
Revised: GeLU( x ) = x × 12 (cid:104) 1 + erf (cid:16) x √ 2 (cid:17)(cid:105) is slow because the Gaussian Error function erf( · ) is evaluated by a high order Taylor expansion, which requires many multiplication routines.

Original: Softmax( x i ) =
Revised: Softmax( x i ) =

Original: exp( x i ) (cid:80) j exp( x j ) is slow because (1) The exponential function is evaluated by several iterations of squaring, which requires many multiplication routines; (2) the maximum operation over x is required for numerical stability (Paszke et al., 2019), which requires comparison routines.
Revised: [Equation] is slow because (1) The exponential function is evaluated by several iterations of squaring, which requires many multiplication routines; (2) the maximum operation over x is required for numerical stability (Paszke et al., 2019), which requires comparison routines.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: MPC.
Revised: MPC.

Original: Secure Multi-party Computation (MPC) enables joint computation between parties while keeping inputs private.
Revised: Secure Multi-party Computation (MPC) enables joint computation between parties while keeping inputs private.

Original: The privacy feature and rich support of systems have made it suitable for
Revised: The privacy feature and rich support of systems have made it suitable for

Original: Transformer inference (Mohassel & Zhang, 2017; Liu et al., 2017; Mohassel & Rindal, 2018; Riazi et al., 2018; Juvekar et al., 2018; Wagh et al., 2019; Mishra et al., 2020; Knott et al., 2021).
Revised: Transformer inference (Mohassel & Zhang, 2017; Liu et al., 2017; Mohassel & Rindal, 2018; Riazi et al., 2018; Juvekar et al., 2018; Wagh et al., 2019; Mishra et al., 2020; Knott et al., 2021).

Original: In this paper, we do not aim to implement a new MPC system.
Revised: In this paper, we do not aim to implement a new MPC system.

Original: Rather, we aim to develop an algorithmic solution to speed up Transformer inference that can be portable across many MPC systems.
Revised: Rather, we aim to develop an algorithmic solution to speed up Transformer inference that can be portable across many MPC systems.

Original: Transformer models.
Revised: Transformer models.

Original: Transformer models have achieved great success in language understanding
Revised: Transformer models have achieved great success in language understanding

Original: Yang et al.
Revised: Yang et al.

Original: (2019b)
Revised: (2019b)

Original: ; Lan et al.
Revised: ; Lan et al.

Original: Raffel et al.
Revised: Raffel et al.

Original: ; Clark et al.
Revised: ; Clark et al.

Original: (2020), vision understanding
Revised: (2020), vision understanding

Original: Dosovitskiy et al.
Revised: Dosovitskiy et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: ; Radford et al.
Revised: ; Radford et al.

Original: (2021), and beyond (Sharir et al., 2021).
Revised: (2021), and beyond (Sharir et al., 2021).

Original: In particular, the two-stage training strategy for Transformer models has been shown to be effective in extensive settings and has become the domincated paradigm (Liu et al., 2019;
Revised: In particular, the two-stage training strategy for Transformer models has been shown to be effective in extensive settings and has become the domincated paradigm (Liu et al., 2019;

Original: Radford et al., 2018;
Revised: Radford et al., 2018;

Original: Turc et al., 2019).
Revised: Turc et al., 2019).

Original: In this training strategy, Transformer models are first pre-trained on a large dataset for general understanding, and then fine-tuned on a small downstream dataset to learn task-specific features.
Revised: In this training strategy, Transformer models are first pre-trained on a large dataset for general understanding, and then fine-tuned on a small downstream dataset to learn task-specific features.

Original: In this work, we consider this paradigm as the default setting, where we assume that model providers use pre-trained Transformer weights from elsewhere, and only have downstream data.
Revised: In this work, we consider this paradigm as the default setting, where we assume that model providers use pre-trained Transformer weights from elsewhere, and only have downstream data.

Original: MPC-friendly approximations.
Revised: MPC-friendly approximations.

Original: Existing research has developed MPC-friendly approximations to speed up CNN computation in MPC.
Revised: Existing research has developed MPC-friendly approximations to speed up CNN computation in MPC.

Original: Chou et al.
Revised: Chou et al.

Original: (2018) develops an optimization framework that minimizes the approximation error of order 2 polynomial to ReLU:
Revised: (2018) develops an optimization framework that minimizes the approximation error of order 2 polynomial to ReLU:

Original: ReLU( x ) = 0 .
Revised: ReLU( x ) = 0 .

Original: × x 2 + 0 .
Revised: × x 2 + 0 .

Original: × x + 0 .
Revised: × x + 0 .

Original: This introduces a significant accuracy drop because the quadratic activation causes the Gradient Descent (GD) algorithm to diverge.
Revised: This introduces a significant accuracy drop because the quadratic activation causes the Gradient Descent (GD) algorithm to diverge.

Original: Mishra et al.
Revised: Mishra et al.

Original: (2020) alleviates this problem by using a set of carefully designed heuristics along with Neural Architecture Search (NAS).
Revised: (2020) alleviates this problem by using a set of carefully designed heuristics along with Neural Architecture Search (NAS).

Original: Mohassel &
Revised: Mohassel &

Original: Zhang (2017) proposes an approximation to softmax by replacing exponential with ReLU functions.
Revised: Zhang (2017) proposes an approximation to softmax by replacing exponential with ReLU functions.

Original: We do not focus on developing heuristics for a single pair of bottleneck functions and approximations.
Revised: We do not focus on developing heuristics for a single pair of bottleneck functions and approximations.

Original: Rather, we focus on developing a general framework that can consistently output a performant
Revised: Rather, we focus on developing a general framework that can consistently output a performant

Original: Transformer model with various approximations.
Revised: Transformer model with various approximations.

Original: Knowledge Distillation (KD).
Revised: Knowledge Distillation (KD).

Original: KD transfers knowledge from the teacher model to the student model by matching their hidden representations (Hinton et al., 2006).
Revised: KD transfers knowledge from the teacher model to the student model by matching their hidden representations (Hinton et al., 2006).

Original: Several research has designed effective objectives for Transformer models (Sanh et al., 2019;
Revised: Several research has designed effective objectives for Transformer models (Sanh et al., 2019;

Original: Jiao et al., 2019; Dosovitskiy et al., 2020) such as matching the attention matrices.
Revised: Jiao et al., 2019; Dosovitskiy et al., 2020) such as matching the attention matrices.

Original: In particular, Sanh et al.
Revised: In particular, Sanh et al.

Original: (2019) and Jiao et al.
Revised: (2019) and Jiao et al.

Original: (2019) have a different goal than us and train on the pre-training dataset as well.
Revised: (2019) have a different goal than us and train on the pre-training dataset as well.

Original: However, we share the same assumption on the model providers’ side — they only have downstream datasets.
Revised: However, we share the same assumption on the model providers’ side — they only have downstream datasets.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we present the MPCF ORMER framework.
Revised: In this section, we present the MPCF ORMER framework.

Original: MPCF ORMER allows the model provider to convert its Transformer model to a faster and performant one for private inference service.
Revised: MPCF ORMER allows the model provider to convert its Transformer model to a faster and performant one for private inference service.

Original: In 4.1, we introduce the workflow of MPCF ORMER (Figure 1), followed by the details of each step in 4.2.[Table] [Figure]
Revised: In 4.1, we introduce the workflow of MPCF ORMER (Figure 1), followed by the details of each step in 4.2.

Original: 4. H IGH - LEVEL WORKFLOW
Revised: 4. H IGH - LEVEL WORKFLOW

Original: In the inference service, a model provider holds a Transformer model T , and the user holds data X .
Revised: In the inference service, a model provider holds a Transformer model T , and the user holds data X .

Original: They reach to an agreement on an MPC system to perform private inference.
Revised: They reach to an agreement on an MPC system to perform private inference.

Original: In §2, we illustrate that using T to perform the inference in the MPC system is slow.
Revised: In §2, we illustrate that using T to perform the inference in the MPC system is slow.

Original: Instead of using T , the model provider can use MPCF ORMER to generate a more suited one S .
Revised: Instead of using T , the model provider can use MPCF ORMER to generate a more suited one S .

Original: S runs much faster than T in the MPC setting while having similar ML performance compared to T .
Revised: S runs much faster than T in the MPC setting while having similar ML performance compared to T .

Original: To use MPCF ORMER , the model provider needs to provide the trained Transformer model T , the downstream dataset D , and MPC-friendly approximations A .
Revised: To use MPCF ORMER , the model provider needs to provide the trained Transformer model T , the downstream dataset D , and MPC-friendly approximations A .

Original: These MPC-friendly approximations
Revised: These MPC-friendly approximations

Original: A need to be fast in MPC and will be used to replace bottleneck functions in T .
Revised: A need to be fast in MPC and will be used to replace bottleneck functions in T .

Original: The workflow can be concisely described as:
Revised: The workflow can be concisely described as:

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: 4. MPCF ORMER
Revised: 4. MPCF ORMER

Original: MPCF ORMER is a two-stage framework as shown in Figure 3.
Revised: MPCF ORMER is a two-stage framework as shown in Figure 3.

Original: The first stage leverages A and T to construct an MPC-friendly Transformer architecture S ′ , which achieves fast inference in the MPC system.
Revised: The first stage leverages A and T to construct an MPC-friendly Transformer architecture S ′ , which achieves fast inference in the MPC system.

Original: The second stage applies knowledge distillation (KD) to S ′ to learn the output model S , which is fast in MPC and preserves the high performance of T .
Revised: The second stage applies knowledge distillation (KD) to S ′ to learn the output model S , which is fast in MPC and preserves the high performance of T .

Original: 4.2. S TAGE 1: A PPROXIMATION
Revised: 4.2. S TAGE 1: A PPROXIMATION

Original: In the first stage, MPCF ORMER replaces bottleneck functions in T with given A to construct a
Revised: In the first stage, MPCF ORMER replaces bottleneck functions in T with given A to construct a

Original: MPC-friendly Transformer architecture S ′ (Figure 3).
Revised: MPC-friendly Transformer architecture S ′ (Figure 3).

Original: Below we show how we construct A for our experiments i.e., using the MPC system Knott et al.
Revised: Below we show how we construct A for our experiments i.e., using the MPC system Knott et al.

Original: (2021).In §2, we identify the bottleneck to be
Revised: (2021).In §2, we identify the bottleneck to be

Original: GeLU and Softmax functions.
Revised: GeLU and Softmax functions.

Original: We thus construct A for these two functions.
Revised: We thus construct A for these two functions.

Original: Approximating GeLU.
Revised: Approximating GeLU.

Original: Analysis in § 2 shows that multiplication in MPC requires extra communication.
Revised: Analysis in § 2 shows that multiplication in MPC requires extra communication.

Original: Thus, quadratics are the fastest nonlinear activation in MPC.
Revised: Thus, quadratics are the fastest nonlinear activation in MPC.

Original: Since GeLU and ReLU functions share similar function values, we simply take the quadratics designed for the ReLU function (§3) to approximate the GeLU function: GeLU ( x ) ≈ 0 .
Revised: Since GeLU and ReLU functions share similar function values, we simply take the quadratics designed for the ReLU function (§3) to approximate the GeLU function: GeLU ( x ) ≈ 0 .

Original: 125 x 2 + 0 .
Revised: 125 x 2 + 0 .

Original: 25 x + 0 .
Revised: 25 x + 0 .

Original: We denote this approximation as “Quad”.
Revised: We denote this approximation as “Quad”.

Original: Approximating Softmax.
Revised: Approximating Softmax.

Original: Prior works in CNNs have developed an MPC-friendly approximation to Softmax functions (Mohassel & Zhang, 2017):
Revised: Prior works in CNNs have developed an MPC-friendly approximation to Softmax functions (Mohassel & Zhang, 2017):

Original: [Equation]
Revised: [Equation]

Original: We validate that this has a faster inference speed than the Softmax function in our setting (Figure 4).
Revised: We validate that this has a faster inference speed than the Softmax function in our setting (Figure 4).

Original: We denote this approximation as “2ReLU”.
Revised: We denote this approximation as “2ReLU”.

Original: However, this is not yet satisfactory.
Revised: However, this is not yet satisfactory.

Original: Analysis in § shows that evaluating the ReLU function requires heavy use of comparison routines, which is very[Figure] expensive.
Revised: Analysis in § shows that evaluating the ReLU function requires heavy use of comparison routines, which is very expensive.

Original: Thus, we propose a more aggressive approximation for the Softmax by replacing the
Revised: Thus, we propose a more aggressive approximation for the Softmax by replacing the

Original: ReLU in Eq.
Revised: ReLU in Eq.

Original: 2 with a quadratic function:
Revised: 2 with a quadratic function:

Original: [Equation]
Revised: [Equation]

Original: We denote this as “2Quad”.
Revised: We denote this as “2Quad”.

Original: Importantly, “2Quad“ and Softmax functions differ a lot by numerical values, while prior works argue that similarity in numerical values is crucial to the model’s performance (Chou et al., 2018).
Revised: Importantly, “2Quad“ and Softmax functions differ a lot by numerical values, while prior works argue that similarity in numerical values is crucial to the model’s performance (Chou et al., 2018).

Original: We are able to use this aggressive approximation because our next distillation stage is effective enough to bridge the performance gap.
Revised: We are able to use this aggressive approximation because our next distillation stage is effective enough to bridge the performance gap.

Original: Figure 4 shows the comparison between the running time of the original GeLU and softmax function against their approximations.
Revised: Figure 4 shows the comparison between the running time of the original GeLU and softmax function against their approximations.

Original: In particular, 2Quad has a much faster inference speed than 2ReLU.
Revised: In particular, 2Quad has a much faster inference speed than 2ReLU.

Original: 4.2. S TAGE 2: D ISTILLATION
Revised: 4.2. S TAGE 2: D ISTILLATION

Original: In the second stage, we use KD to make the fast approximated Transformer model S ′ performant.
Revised: In the second stage, we use KD to make the fast approximated Transformer model S ′ performant.

Original: The benefits of KD are two-fold.
Revised: The benefits of KD are two-fold.

Original: First, it allows us to use more aggressive approximations such as “2Quad“, which leads to higher speedups.
Revised: First, it allows us to use more aggressive approximations such as “2Quad“, which leads to higher speedups.

Original: Second, its data efficiency allows us to effectively learn a good S with the small downstream datasets.
Revised: Second, its data efficiency allows us to effectively learn a good S with the small downstream datasets.

Original: Concretely, we conduct layer-wise distillationto transfer knowledge from the input model T to S ′ by matching representation at the following four positions: (1) the embedding layer, (2) the attention matrix in each Transformer layer, (3) the hidden states after each Transformer layer, and (4) the final prediction layer.
Revised: Concretely, we conduct layer-wise distillation

Original: These four positions have been shown to store meaningful information in previous works (Hinton et al., 2015; Jiao et al., 2019; Clark et al., 2019). We use the Mean Square Error (MSE) loss to match the representations between T and S for all positions. We follow the learning procedure of Jiao et al. (2019) to first distill the embedding and Transformer layers (including the attention matrix and the hidden states) and then distill the prediction layer.
Revised: [Figure]

Original: Student initialization.
Revised: Student initialization.

Original: An important component of knowledge distillation is the initialization of the student model (Sanh et al., 2019).
Revised: An important component of knowledge distillation is the initialization of the student model (Sanh et al., 2019).

Original: Taking the advantage that S ′ and T share the same architecture, we initialize S ′ using weights in T .
Revised: Taking the advantage that S ′ and T share the same architecture, we initialize S ′ using weights in T .

Original: We find that this outperforms random weight initialization, especially on smaller datasets (§5.3).
Revised: We find that this outperforms random weight initialization, especially on smaller datasets (§5.3).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We design the MPCF ORMER framework to be compatible with many MPC-friendly approximations and trained Transformer models, so that model providers can conveniently plug in MPC-friendly approximations based on their MPC systems.
Revised: We design the MPCF ORMER framework to be compatible with many MPC-friendly approximations and trained Transformer models, so that model providers can conveniently plug in MPC-friendly approximations based on their MPC systems.

Original: Thus, we evaluate MPCF ORMER with different MPCfriendly approximations under (1) Different datasets ( § 5.1), and (2) Different models (especially larger models) ( § 5.2).
Revised: Thus, we evaluate MPCF ORMER with different MPCfriendly approximations under (1) Different datasets ( § 5.1), and (2) Different models (especially larger models) ( § 5.2).

Original: In the ablation study, we study (1) the effect of student initialization and, (2) the effect of the number of training examples in the distillation stage.
Revised: In the ablation study, we study (1) the effect of student initialization and, (2) the effect of the number of training examples in the distillation stage.

Original: Experimental setup.
Revised: Experimental setup.

Original: We use two P3.2x AWS instances to simulate the inference service scenarios (one P3.2x for the model provider, and one for the user).
Revised: We use two P3.2x AWS instances to simulate the inference service scenarios (one P3.2x for the model provider, and one for the user).

Original: Each instance is equipped with one Tesla
Revised: Each instance is equipped with one Tesla

Original: V100 GPU, and 10GbE Ethernet bandwidth.
Revised: V100 GPU, and 10GbE Ethernet bandwidth.

Original: We place instances in the same placement group to guarantee a 10GbE bandwidth in AWS.
Revised: We place instances in the same placement group to guarantee a 10GbE bandwidth in AWS.

Original: Time breakdown is measured with Crypten, which implements secret sharing with semi-honest adversaries assumption (Section §2) (Knott et al., 2021).
Revised: Time breakdown is measured with CrypTen, which implements secret sharing with semi-honest adversaries assumption (Section §2) (Knott et al., 2021).

Original: We train and evaluate models based on HuggingFace (Wolf et al., 2019).
Revised: We train and evaluate models based on HuggingFace (Wolf et al., 2019).

Original: In particular, we find that the implementation of 2Quad requires careful use of HuggingFace source code (§A.2).
Revised: In particular, we find that the implementation of 2Quad requires careful use of HuggingFace source code (§A.2).

Original: Baselines.
Revised: Baselines.

Original: We identify three important properties during Transformer inference in § 1: speed, performance, and privacy.
Revised: We identify three important properties during Transformer inference in § 1: speed, performance, and privacy.

Original: In our workflow, privacy has been guaranteed by using MPC systems.
Revised: In our workflow, privacy has been guaranteed by using MPC systems.

Original: Thus, we evaluate MPCF ORMER by the other two aspects: speed and performance.
Revised: Thus, we evaluate MPCF ORMER by the other two aspects: speed and performance.

Original: Concretely,we require S to run faster than T while matching the performance of T .
Revised: Concretely, S shall run faster than T while matching the performance of T .

Original: Since there is a limited amount of work on Transformer inference in MPC, we seek a baseline from CNN literature.
Revised: Since there is a limited amount of work on Transformer inference in MPC, we seek a baseline from CNN literature.

Original: In particular, we choose the training strategy in (Chou et al., 2018) and denote it as MPCF ORMER w/o { d } .
Revised: In particular, we choose the training strategy in (Chou et al., 2018) and denote it as MPCF ORMER w/o { d } .

Original: MPCF ORMER
Revised: MPCF ORMER

Original: w/o { d } also constructs the approximated model S ′ but trains S ′ on D with the taskspecific objective, i.e., without distillation.
Revised: w/o { d } also constructs the approximated model S ′ but trains S ′ on D with the taskspecific objective, i.e., without distillation.

Original: We note that S ′ is initialized with weights in T , i.e., with different functions, whose effect has not been studied.
Revised: We note that S ′ is initialized with weights in T , i.e., with different functions, whose effect has not been studied.

Original: We thus propose a second baseline MPCF ORMER w/o { p,d } , which trains S ′ on D without distillation, and random weight initialization.
Revised: We thus propose a second baseline MPC- F ORMER w/o { p,d } , which trains S ′ on D without distillation, and random weight initialization.

Original: Below we compare the performance of MPCF ORMER with MPCF ORMER w/o { p,d } and MPCF ORMER w/o { d } with the same speedups.
Revised: Below we compare the performance of MPCF ORMER with MPCF ORMER w/o { p,d } and MPCF ORMER w/o { d } with the same speedups.

Original: We denote the output model of our framework with BERT BASE , Roberta-base, and BERT LARGE as
Revised: We denote the output model of our framework with BERT BASE , Roberta-base, and BERT LARGE as

Original: MPCBert-B, MPCRoberta-B, and MPCBert-L for short.
Revised: MPCBert-B, MPCRoberta-B, and MPCBert-L for short.

Original: 5. C OMPARISON WITH B ASELINES ON D IFFERENT B ENCHMARKS
Revised: 5. C OMPARISON WITH B ASELINES ON D IFFERENT B ENCHMARKS

Original: Settings.
Revised: Settings.

Original: In this section, we evaluate our MPCFormer framework with different approximations and compare it with baselines on the IMDb dataset and the GLUE benchmark (Maas et al., 2011;
Revised: In this section, we evaluate our MPCFormer framework with different approximations and compare it with baselines on the IMDb dataset and the GLUE benchmark (Maas et al., 2011;

Original: Wang et al., 2018).
Revised: Wang et al., 2018).

Original: For all experiments in this section, we use BERT BASE as the base model.
Revised: For all experiments in this section, we use BERT BASE as the base model.

Original: According to the dataset statistics, we use a sequence length of 512 for the IMDb dataset and a sequence length of 128 for GLUE datasets.
Revised: According to the dataset statistics, we use a sequence length of 512 for the IMDb dataset and a sequence length of 128 for GLUE datasets.

Original: We note that a longer sequence length generally reflects a higher speedup because the Softmax functions can be sped up more.
Revised: We note that a longer sequence length generally reflects a higher speedup because the Softmax functions can be sped up more.

Original: Baselines are trained with learning rates tuned from 1e-6, 5e-6, 1e-5, and 1e-4, the number of epochs from 10, 30, and 100, the batch size 32 for IMDB, batch sizes 64 and 256 for GLUE.
Revised: Baselines are trained with learning rates tuned from 1e-6, 5e-6, 1e-5, and 1e-4, the number of epochs from 10, 30, and 100, the batch size 32 for IMDB, batch sizes 64 and 256 for GLUE.

Original: MPCBert-B is trained with learning rate 5efor embedding and Transformer layer distillation, and 1e-5 for prediction layer distillation.
Revised: MPCBert-B is trained with learning rate 5efor embedding and Transformer layer distillation, and 1e-5 for prediction layer distillation.

Original: Further details on hyper-parameters tuning can be found in A.4.
Revised: Further details on hyper-parameters tuning can be found in A.4.

Original:  We show the accuracy and speedup on the IMDb dataset in Table 2.
Revised: [Table] [Table] We show the accuracy and speedup on the IMDb dataset in Table 3.

Original: MPCBert-B achieves 5.26 × speedup with “Quad+2Quad” approximation with almost no accuracy drop.
Revised: MPCBert-B achieves 5.26 × speedup with “Quad+2Quad” approximation with almost no accuracy drop.

Original: In addition, we note that this holds for not only the fastest “Quad+2Quad“ approximation but other approximations ranging from 1.24 × to 2.65 × speedups.
Revised: In addition, we note that this holds for not only the fastest “Quad+2Quad“ approximation but other approximations rang-

Original: Baselines have inferior performance for all approximations.
Revised: [Table]

Original: For example, both baselines have at least an accuracy drop of 6.8% with 5.26 × speedup.
Revised: For example, both baselines have at least an accuracy drop of 6.8% with 5.26 × speedup.

Original: Interestingly, MPCBert-B w/o { d } has moderate accuracy drop for “GeLU+2ReLU” approximation with 1.76 × speedup.
Revised: Interestingly, MPCBert-B w/o { d } has moderate accuracy drop for “GeLU+2ReLU” approximation with 1.76 × speedup.

Original: However, it does not preserve accuracy with other approximations.
Revised: However, it does not preserve accuracy with other approximations.

Original: In contrast,
Revised: In contrast,

Original: MPCF ORMER consistently preserves accuracy with various kinds of approximation.
Revised: MPCF ORMER consistently preserves accuracy with various kinds of approximation.

Original: To validate the observation with more datasets, We evaluate MPCBert-B on the Glue benchmark (8 datasets) (Wang et al., 2018).
Revised: To validate the observation with more datasets, We evaluate MPCBert-B on the Glue benchmark (8 datasets) (Wang et al., 2018).

Original: As shown in Table 3, MPCBert-B achieves 1.93 × speedup with 98% performance of BERT BASE on the GLUE benchmark, and 2.2 × speedup with 97% performance of BERT BASE .
Revised: As shown in Table 4, MPCBert-B achieves 1.93 × speedup with 98% performance of BERT BASE on the GLUE benchmark, and 2.2 × speedup with 97% performance of BERT BASE .

Original: Both baselines introduce severe performance drop, i.e., 19.5 average score drop for
Revised: Both baselines introduce severe performance drop, i.e., 19.5 average score drop for

Original: MPCBert-B w/o { d } and 26.2 average score drop for MPCBert-B w/o { p,d } with 1.93 × speedup.
Revised: MPCBert-B w/o { d } and 26.2 average score drop for MPCBert-B w/o { p,d } with 1.93 × speedup.

Original: Interest- ingly, we observe that the baseline MPCBert-B w/o { d } consistently outperforms MPCBert-B w/o { p,d } .
Revised: Interest- ingly, we observe that the baseline MPCBert-B w/o { d } consistently outperforms MPCBert-B w/o { p,d } .

Original: This indicates that using weights in T as initialization benefits the training of S ′ with a task-specific objective, using a 12-layer Transformer backbone.
Revised: This indicates that using weights in T as initialization benefits the training of S ′ with a task-specific objective, using a 12-layer Transformer backbone.

Original: Additionally, we evaluate MPCF ORMER with more approximations using a subset of the GLUE benchmark.
Revised: Additionally, we evaluate MPCF ORMER with more approximations using a subset of the GLUE benchmark.

Original: Results are shown in the right part of
Revised: Results are shown in the right part of

Original: Table 2.
Revised: Table 3.

Original: We observe similar patterns as in the IMDb dataset.
Revised: We observe similar patterns as in the IMDb dataset.

Original: The baseline MPCBert-B w/o { d } performs well in “GeLU+2Quad‘ and “GeLU+2ReLU“ approximations, but MPCF ORMER achieves high ML performance consistently under all approximations.
Revised: The baseline MPCBert-B w/o { d } performs well in “GeLU+2Quad‘ and “GeLU+2ReLU“ approximations, but MPCF ORMER achieves high ML performance consistently under all approximations.

Original: 5. M ORE COMPARISONS WITH DIFFERENT MODELS
Revised: 5. M ORE COMPARISONS WITH DIFFERENT MODELS

Original: We evaluate MPCF ORMER with trained Transformer models other than BERT BASE , i.e.,
Revised: We evaluate MPCF ORMER with trained Transformer models other than BERT BASE , i.e.,

Original: ROBERTA BASE model (12 layers) (Liu et al., 2019), and in particular a larger BERT LARGE model [Table] [Table] [Table]
Revised: ROBERTA BASE model (12 layers) (Liu et al., 2019), and in particular a larger BERT LARGE model (24 layers).

Original: (24 layers).

Original: Results are shown in Table 4.
Revised: Results are shown in Table 5.

Original: MPCRoberta-B preserves 98% average score of the input model ROBERTA BASE , and outperforms baselines by large margins.
Revised: MPCRoberta-B preserves 98% average score of the input model ROBERTA BASE , and outperforms baselines by large margins.

Original: Comparing the performance of baselines, we again observe that initialization with weights in T helps training with S ′ , i
Revised: Comparing the performance of baselines, we again observe that initialization with weights in T helps training with S ′ , i.e., MPCRoberta-B w/o { d } performs better than MPCRoberta-B w/o { p,d } .

Original: [Table]
Revised: [Table]

Original: e., MPCRoberta-B w/o { d } performs better than MPCRoberta-B w/o { p,d } .
Revised: To show our method can scale to different model sizes

Original: [Table]
Revised: [Table]

Original: To show our method can scale to different model sizes, we evaluate MPCF ORMER with a larger model BERT LARGE (24-layer).
Revised: we evaluate MPCF ORMER with a larger model BERT LARGE (24-layer).

Original: On the IMDb dataset, BERT BASE achieves 95.0% accuracy.
Revised: On the IMDb dataset, BERT BASE achieves 95.0% accu- racy.

Original: MPCBert-L achieves 5.9 × speedup with 94.5% accuracy, while baselines MPCBert-L w/o { p,d } achieves 87.1% accuracy, and MPCBert-L w/o { d } achieves 50.0% accuracy.
Revised: MPCBert-L achieves 5.9 × speedup with 94.5% accuracy, while baselines MPCBert-L w/o { p,d } achieves 87.1% accuracy, and MPCBert-L w/o { d } achieves 50.0% accuracy.

Original: We further select four datasets from the GLUE benchmark, where Bert-L* noticeably outperforms Bert-B*(MNLI, QNLI,
Revised: We further select four datasets from the GLUE benchmark, where Bert-L* noticeably outperforms Bert-B*(MNLI, QNLI,

Original: CoLA, and RTE).
Revised: CoLA, and RTE).

Original: Compared with Bert-B* in table 3, Bert-L* increases the average score from 82.8 to 85.3.
Revised: Compared with Bert-B* in table 4, Bert-L* increases the average score from 82.8 to 85.3.

Original: MPCF ORMER increases the average score from 81.5 to 84.6.
Revised: MPCF ORMER increases the average score from 81.5 to 84.6.

Original: This indicates that
Revised: This indicates that

Original: MPCF ORMER can scale with the size of T .
Revised: MPCF ORMER can scale with the size of T .

Original: On the other hand, baselines do not scale with the input model: MPCBert-L w/o { p,d } decreases the average score from 60.1 to 59.3; MPCBert-L w/o { d } decreases the average score from 68.5 to 43.5.
Revised: On the other hand, baselines do not scale with the input model: MPCBert-L w/o { p,d } decreases the average score from 60.1 to 59.3; MPCBert-L w/o { d } decreases the average score from 68.5 to 43.5.

Original: In particular, we observe that initializing S ′ with weights in T without distillation harms performance when the model is larger.
Revised: In particular, we observe that initializing S ′ with weights in T without distillation harms performance when the model is larger.

Original: 5. A BLATION STUDY
Revised: 5. A BLATION STUDY

Original: [Figure]
Revised: [Figure]

Original: The first question we study is the effect of different student initialization.
Revised: The first question we study is the effect of different student initialization.

Original: This is of interest as we do not have a general knowledge of whether initializing with weights in T will still benefit the training of S ′ after aggressive approximations.
Revised: This is of interest as we do not have a general knowledge of whether initializing with weights in T will still benefit the training of S ′ after aggressive approximations.

Original: We design an experiment with random initialization (denoted as MPCBert-B r in the table).
Revised: We design an experiment with random initialization (denoted as MPCBert-B r in the table).

Original: We train MPCBert-B r with 10 × more epochs than MPCBert-B r and confirm that its distillation objective has converged.
Revised: We train MPCBert-B r with 10 × more epochs than MPCBert-B r and confirm that its distillation objective has converged.

Original: We observe that on larger datasets(QNLI and SST-2), the gap between using different initialization is small, but on smaller datasets(STS-B, MRPC, and RTE), initializing with the weights in T is better.
Revised: We observe that on larger datasets(QNLI and SST-2), the gap between using different initialization is small, but on smaller datasets(STS-B, MRPC, and RTE), initializing with the weights in T is better.

Original: The second question we study is the effect of the number of training examples in the distillation stage.
Revised: The second question we study is the effect of the number of training examples in the distillation stage.

Original: We perform studies on two small (RTE, MRPC) and two medium (SST-2, QNLI) datasets in the
Revised: We perform studies on two small (RTE, MRPC) and two medium (SST-2, QNLI) datasets in the

Original: GLUE benchmark (Figure 5).
Revised: GLUE benchmark (Figure 5).

Original: We find that roughly 5% of the small datasets and 2% of the medium datasets provides enough data to learn a good S .
Revised: We find that roughly 5% of the small datasets and 2% of the medium datasets provides enough data to learn a good S .

Original: This shows that KD in our setting is efficient enough to learn a good S in downstream tasks (using GLUE datasets as representatives).
Revised: This shows that KD in our setting is efficient enough to learn a good S in downstream tasks (using GLUE datasets as representatives).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: L IMITATION AND FUTURE DIRECTION
Revised: L IMITATION AND FUTURE DIRECTION

Original: We recognize two limitations in our paper.
Revised: We recognize two limitations in our paper.

Original: First, our speedups and performance are tested on a single MPC system.
Revised: First, our speedups and performance are tested on a single MPC system.

Original: We leave theoretical analysis or empirical study on more MPC systems as future work.
Revised: We leave theoretical analysis or empirical study on more MPC systems as future work.

Original: Second, in our design, T and S only differ by functions, i.e., they have the same model size.
Revised: Second, in our design, T and S only differ by functions, i.e., they have the same model size.

Original: We leave extension toour framework to a smaller student model as future work.
Revised: We leave extension to a smaller student model as future work.

Original:  CONCLUSION
Revised:  CONCLUSION

Original: In this paper, we propose a framework to achieve fast and performant private Transformer model inference with MPC.
Revised: In this paper, we propose a framework to achieve fast and performant private Transformer model inference with MPC.

Original: Evaluations show that it is compatible with various MPC-friendly approximations and trained Transformer models.
Revised: Evaluations show that it is compatible with various MPC-friendly approximations and trained Transformer models.

Original: We suggest two directions of interest: (1) Theoretical or empirical analysis on more MPC systems, and (2) extension on the problem formulation to allow a smaller size of S . 
Revised: We suggest two directions of interest: (1) Theoretical or empirical analysis on more MPC systems, and (2) extension on the problem formulation to allow a smaller size of S . A CKNOWLEDGEMENT

Revised: The authors thank Qirong Ho for allocating the computing resources.

Revised: This research was supported by NSF IIS1563887, NSF CCF1629559, NSF IIS1617583, NGA HM04762010002, NSF

Revised: IIS1955532, NSF CNS2008248, NSF IIS2123952, and NSF BCS2040381.

Original: A A PPENDIX A. A CONCRETE SYSTEM IMPLEMENTATION OF MPC: C RYPTEN
Revised: A A PPENDIX A. A CONCRETE SYSTEM IMPLEMENTATION OF MPC: C RYP T EN

Original: In this section, we provide how a concrete MPC system (Crypten) implements routines and functions for Transformer models in detail (Knott et al., 2021).
Revised: In this section, we provide how a concrete MPC system (CrypTen) implements routines and functions for Transformer models in detail (Knott et al., 2021).

Original: We provide a portion of details here to help describe the complexity of Transformer inference in MPC.
Revised: We provide a portion of details here to help describe the complexity of Transformer inference in MPC.

Original: A more complete system overview and privacy proof are available in the Crypten paper.
Revised: A more complete system overview and privacy proof are available in the CrypTen paper.

Original: Threat model.
Revised: Threat model.

Original: Crypten follows Evans et al.
Revised: CrypTen follows Evans et al.

Original: (2018) to assume that parties are semi-honest.
Revised: (2018) to assume that parties are semi-honest.

Original: Under this assumption, parties are honest that they will follow the system protocols.
Revised: Under this assumption, parties are honest that they will follow the system protocols.

Original: However, each party is also curious (i.e., semi-honest), meaning it will try to infer the information about others’ data based on the values it receives.
Revised: However, each party is also curious (i.e., semi-honest), meaning it will try to infer the information about others’ data based on the values it receives.

Original: Secret shares Crypten uses secret shares to implement private computation.
Revised: Secret shares CrypTen uses secret shares to implement private computation.

Original: A floating point value x f is first scaled to an integer x 4 , then secretly shared with both parties.
Revised: A floating point value x f is first scaled to an integer x 5 , then secretly shared with both parties.

Original: Secret shares are of type arithmetic or binary .
Revised: Secret shares are of type arithmetic or binary .

Original: The arithmetic secret shares [ x ] = { [ x ] 1 , [ x ] 2 } is a set of two numbers, where the first party holds [ x ] 1 , and the second holds [ x ] 2 .
Revised: The arithmetic secret shares [ x ] = { [ x ] 1 , [ x ] 2 } is a set of two numbers, where the first party holds [ x ] 1 , and the second holds [ x ] 2 .

Original: They are constructed with a pair of zero-sum random maskings (Cramer et al., 2005)
Revised: They are constructed with a pair of zerosum random maskings (Cramer et al., 2005)

Original: so that [ x ] 1 + [ x ] 2 = x .
Revised: so that [ x ] 1 + [ x ] 2 = x .

Original: Binary shares ⟨ x ⟩ are formed by arithmetic secret shares of bits in x , so that the bitwise xor ⟨ x ⟩ 1 ⊕ ⟨ x ⟩ 1 = x .
Revised: Binary shares ⟨ x ⟩ are formed by arithmetic secret shares of bits in x , so that the bitwise xor ⟨ x ⟩ 1 ⊕ ⟨ x ⟩ 1 = x .

Original: Routines and functions In arithmetic secret shares, to privately evaluate addition [ x ] + [ y ] , both parties simply compute: [ x ] i +[ y ] i individually.
Revised: Routines and functions In arithmetic secret shares, to privately evaluate addition [ x ] + [ y ] , both parties simply compute: [ x ] i +[ y ] i individually.

Original: Multiplication [ x ][ y ] is evaluated by using a Beaver triple generated off-line ([ c ] , [ a ] , [ b ]) where c = ab (Beaver, 1991).
Revised: Multiplication [ x ][ y ] is evaluated by using a Beaver triple generated off-line ([ c ] , [ a ] , [ b ]) where c = ab (Beaver, 1991).

Original: Both parties compute and reveal intermediate values [ ϵ ] =
Revised: Both parties compute and reveal intermediate values [ ϵ ] =

Original: [ x ] − [ a ] and [ δ ] =
Revised: [ x ] − [ a ] and [ δ ] =

Original: [ y ] − [ b ] .
Revised: [ y ] − [ b ] .

Original: The final result is computed by [ x ][ y ] =
Revised: The final result is computed by [ x ][ y ] =

Original: [ c ]+ ϵ [ b ]+[ a ] δ + ϵδ .
Revised: [ c ]+ ϵ [ b ]+[ a ] δ + ϵδ .

Original: Linear functions such as matrix multiplication can be evaluated by using additions and multiplications.
Revised: Linear functions such as matrix multiplication can be evaluated by using additions and multiplications.

Original: Non-linear functions are evaluated by numerical approximations using additions and multiplications, such as Taylor expansion.
Revised: Non-linear functions are evaluated by numerical approximations using additions and multiplications, such as Taylor expansion.

Original: Comparison requires conversions between arithmetic and binary secret shares.
Revised: Comparison requires conversions between arithmetic and binary secret shares.

Original: Conversion from [ x ] to ⟨ x ⟩ first creates binary secret shares ⟨ [ x ] i ⟩ from arithmetic secret share [ x ] i ( i = 0 , 1 ), then computes ⟨ x ⟩ = ⟨ [ x ] 1 ⟩ + ⟨ [ x ] 2 ⟩ using an adder circuit.
Revised: Conversion from [ x ] to ⟨ x ⟩ first creates binary secret shares ⟨ [ x ] i ⟩ from arithmetic secret share [ x ] i ( i = 0 , 1 ), then computes ⟨ x ⟩ = ⟨ [ x ] 1 ⟩ + ⟨ [ x ] 2 ⟩ using an adder circuit.

Original: Conversion from ⟨ x ⟩ to [ x ] is done by: [ x ] = (cid:80)
Revised: Conversion from ⟨ x ⟩ to [ x ] is done by: [ x ] = (cid:80)

Original: Bb =1 2 b [ ⟨ x ⟩ ( b ) ] , where ⟨ x ⟩ ( b ) is the b
Revised: Bb =1 2 b [ ⟨ x ⟩ ( b ) ] , where ⟨ x ⟩ ( b ) is the b

Original: th bit of ⟨ x ⟩ .
Revised: th bit of ⟨ x ⟩ .

Original: The comparison function [ z < 0] is then evaluated by: (1) convert [ z ] to ⟨ z ⟩ (2) compute the sign bit ⟨ b ⟩ = ⟨ z ⟩ >> ( L − 1) .
Revised: The comparison function [ z < 0] is then evaluated by: (1) convert [ z ] to ⟨ z ⟩ (2) compute the sign bit ⟨ b ⟩ = ⟨ z ⟩ >> ( L − 1) .

Original: (3) Convert ⟨ b ⟩ to [ b ] .
Revised: (3) Convert ⟨ b ⟩ to [ b ] .

Original: We study on the standard setting where each tensor is represented in 64 bits (i.e. L=64).
Revised: We study on the standard setting where each tensor is represented in 64 bits (i.e. L=64).

Original: Each multiplication requires one round of communication for revealing the intermediate values ϵ, δ .
Revised: Each multiplication requires one round of communication for revealing the intermediate values ϵ, δ .

Original: Each conversion from [ x ] to ⟨ x ⟩ requires log 2 L = 6 rounds of communications for the adder circuit; each conversion from ⟨ x ⟩ to [ x ] requires one round for generating [ ⟨ x ⟩ ( b ) ] .
Revised: Each conversion from [ x ] to ⟨ x ⟩ requires log 2 L = 6 rounds of communications for the adder circuit; each conversion from ⟨ x ⟩ to [ x ] requires one round for generating [ ⟨ x ⟩ ( b ) ] .

Original: Thus, each comparison requires 7 rounds of communication.
Revised: Thus, each comparison requires 7 rounds of communication.

Original: Each max( · ) between N elements requires O (log 2 ( N )) rounds of communications, assuming a tree-reduction algorithm.
Revised: Each max( · ) between N elements requires O (log 2 ( N )) rounds of communications, assuming a tree-reduction algorithm.

Original: We provide a simple addition example here.
Revised: We provide a simple addition example here.

Original: The scaling factor and ring size Q are set to small for ease of understanding.
Revised: The scaling factor and ring size Q are set to small for ease of understanding.

Original: [Table]
Revised: [Table]

Original: In addition, we provide a more complete breakdown in terms of communication and computer load for Figure 2 for a holistic view of execution pattern in the MPC setting.
Revised: In addition, we provide a more complete breakdown in terms of communication and computer load for Figure 2 for a holistic view of execution pattern in the MPC setting.

Original: [Table]
Revised: [Table]

Original: In particular, computation only takes 21% of the running time and the communication takes 79% of the running time.
Revised: In particular, computation only takes 21% of the running time and the communication takes 79% of the running time.

Original: Consequently, the number of floating point operations (FLOP), a popular estimator for the running time in plain-text Transformer inference, is no longer accurate in the MPC setting.
Revised: Consequently, the number of floating point operations (FLOP), a popular estimator for the running time in plain-text Transformer inference, is no longer accurate in the MPC setting.

Original: The MPC system we use in the paper uses All-reduce to implement intermediate communication, where both parties have the same communication load.
Revised: The MPC system we use in the paper uses All-reduce to implement intermediate communication, where both parties have the same communication load.

Original: And they have similar computation load (see the multiplication example, where both parties are computing the same function locally with their own secret shares).
Revised: And they have similar computation load (see the multiplication example, where both parties are computing the same function locally with their own secret shares).

Original: Thus, the time breakdown is similar for both parties.
Revised: Thus, the time breakdown is similar for both parties.

Original: In the above table, we report the statistics from the model provider.
Revised: In the above table, we report the statistics from the model provider.

Original: A. 2Q UAD IMPLEMENTATION DETAILS
Revised: A. 2Q UAD IMPLEMENTATION DETAILS

Original: We note that, implementing “2Quad“ to replace the softmax requires attention to the effect brought by the masking operation.
Revised: We note that, implementing “2Quad“ to replace the softmax requires attention to the effect brought by the masking operation.

Original: For example, the default implementation by Huggingface Wolf et al.
Revised: For example, the default implementation by Huggingface Wolf et al.

Original: (2019) would result in an exploding problem due to masking.
Revised: (2019) would result in an exploding problem due to masking.

Original: Therefore, we would need to do a different version of the implementation of masking.
Revised: Therefore, we would need to do a different version of the implementation of masking.

Original: We describe it in detail below.
Revised: We describe it in detail below.

Original: The default attention implementation by Huggingface is
Revised: The default attention implementation by Huggingface is

Original: [Equation]
Revised: [Equation]

Original: If we directly replace the e x with ( x + c ) 2 as in 2Quad approximation, where x = QK T √ d k + M { 0 , − inf } will explode when being masked, causing a problem in the forward pass.
Revised: If we directly replace the e x with ( x + c ) 2 as in 2Quad approximation, where x = QK T √ d k + M { 0 , − inf } will explode when being masked, causing a problem in the forward pass.

Original: To solve this problem, we could simply change the implementation of masking from “adding a zero or negative infinite number in the exponent” to “multiplying one or zero to the exponential function”.
Revised: To solve this problem, we could simply change the implementation of masking from “adding a zero or negative infinite number in the exponent” to “multiplying one or zero to the exponential function”.

Original: That is,
Revised: That is,

Original: [Equation]
Revised: [Equation]

Original: It’s just a different implementation of the same masking purpose but avoids exploding at the masking positions.
Revised: It’s just a different implementation of the same masking purpose but avoids exploding at the masking positions.

Original: In our experiments, we empirically tried c = 5 and it worked pretty well, indicating the choice of the constant c could be flexible.
Revised: In our experiments, we empirically tried c = 5 and it worked pretty well, indicating the choice of the constant c could be flexible.

Original: A. R OBUSTNESS OF THE STUDENT MODEL
Revised: A. R OBUSTNESS OF THE STUDENT MODEL

Original: Some approximations may increase the local Lipschitz constants, which decreases the robustness.
Revised: Some approximations may increase the local Lipschitz constants, which decreases the robustness.

Original: We applied some empirical text adversarial attacks to evaluate the adversarial robustness of the
Revised: We applied some empirical text adversarial attacks to evaluate the adversarial robustness of the

Original: BERT-base model before and after approximations (Zeng et al., 2020).
Revised: BERT-base model before and after approximations (Zeng et al., 2020).

Original: As shown in Table 8, the student model has a moderate increase in terms of attack success rate (ASR) over the three scorebased attacks.
Revised: As shown in Table 9, the student model has a moderate increase in terms of attack success rate (ASR) over the three scorebased attacks.

Original: But the student model has a lower ASR with the gradient-based attack HotFlip.
Revised: But the student model has a lower ASR with the gradient-based attack HotFlip.

Original: Considering these results, the effect on robustness by the approximations are empirically moderate.
Revised: Considering these results, the effect on robustness by the approximations are empirically moderate.

Original: [Table]
Revised: [Table]

Original: A. HYPER - PARAMETER CHOICE
Revised: H

Revised: [Table] [Table] CHOICE

Revised: For baselines

Original: For baselines, We study the effect of hyper-parameters by running a grid search over the STS-B dataset 5 , with learning rate from [1e-6, 5e-6, 1e-5, 5e-5, 1e-4, 5e-4], batch size from [256, 128, 64, 32, 16], epoch from [3, 10, 30, 50, 80, 100, 200].
Revised: [Table] We study the effect of hyper-parameters by running a grid search over the STS-B dataset [Table] with learning rate from [1e-6, 5e-6, 1e-5, 5e-5, 1e-4, 5e-4], batch size from [256, 128, 64, 32, 16], epoch from [3, 10, 30, 50, 80, 100, 200].

Original: We show the grid search results with BERT BASE in figure 6, 7, and a smaller grid search for BERT Large and ROBERTA BASE in Figure 8, 9.
Revised: We show the grid search results with BERT BASE in figure 6, [Table] and a smaller grid search for BERT Large and ROBERTA BASE in Figure 8, 9.

Original: We empirically discover that the learning rates from 1e-6, 5e-6, 1e-5, batch size from 64 and 256, epoch from 10, 100 give good performance.
Revised: We empirically discover that the learning rates from 1e-6, 5e-6, 1e-5, batch size from 64 and 256, epoch from 10, 100 give good performance.

Original: To let baselines explore more hyper-parameters, we use learning rate from [1e-6, 5e-6, 1e-5, 1e-4], batch size from [64, 256], epochs from [10, 30, 100] for all Glue datasets.
Revised: To let baselines explore more hyper-parameters, we use learning rate from [1e-6, 5e-6, 1e-5, 1e-4], batch size from [64, 256], epochs from [10, 30, 100] for all Glue datasets.

Original: Since we use sequence length 512 for IMDB dataset, we use batch size 32 to fit into our 16GB Tesla V100 GPU.
Revised: Since we use sequence length 512 for IMDB dataset, we use batch size 32 to fit into our 16GB Tesla V100 GPU.

Original: We also empirically discover that (1) MPCBert-B w/o { d } (best 0.43) can not scale up when the base model scales to BERT Large i.e., MPCBert-L w/o { d } (best 0.08).
Revised: We also empirically discover that (1) MPCBert-B w/o { d } (best 0.43) can not scale up when the base model scales to BERT Large i.e., MPCBert-L w/o { d } (best 0.08).

Original: (2) baseline benefits from using the pre-trained weights, i.e., MPCBert-B w/o { d } (best 0.42) performs better than MPCBert-B w/o { p, d } (best 0.23).
Revised: (2) baseline benefits from using the pre-trained weights, i.e., MPCBert-B w/o { d } (best 0.42) performs better than MPCBert-B w/o { p, d } (best 0.23).

Original: (3) MPCFormer w/o { d } benefits when the base model becomes better, i.e., MPCRoberta-B w/o { d } (best 0.62) performs better than MPCBert-B w/o { d } (best 0.42).
Revised: (3) MPCFormer w/o { d } benefits when the base model becomes better, i.e., MPCRoberta-B w/o { d } (best 0.62) performs better than MPCBert-B w/o { d } (best 0.42).

Original: For MPCF ORMER , we decide the number of epochs according to the MSE loss for embedding and
Revised: For MPCF ORMER , we decide the number of epochs according to the MSE loss for embedding and

Original: Transformer layer distillation, 5 epochs for prediction layer distillation, and batch size 8 for small datasets (CoLA, MRPC, RTE) and 32 for larger ones (MNLI, QQP, SST2, STS-B).
Revised: Transformer layer distillation, 5 epochs for prediction layer distillation, and batch size 8 for small datasets (CoLA, MRPC, RTE) and 32 for larger ones (MNLI, QQP, SST2, STS-B).

Original: We minimize the hyper-parameter tuning for MPCF ORMER , since we would like the performance to be an expectation for future researchers using MPCF ORMER , who prefer not to tune hyper-parameters.
Revised: We minimize the hyper-parameter tuning for MPCF ORMER , since we would like the performance to be an expectation for future researchers using MPCF ORMER , who prefer not to tune hyper-parameters.

Original: Specifically, we use 5 epochs for MNLI, 5 epochs for QQP, 10 epochs for QNLI, 10 epochs for SST-2, 20 epochs for MRPC, 30 epochs for IMDB 50 epochs for STS-B, 50 epochs for CoLA, 50 epoches for RTE, for the embedding and Transformer layer distillation stage.
Revised: Specifically, we use 5 epochs for MNLI, 5 epochs for QQP, 10 epochs for QNLI, 10 epochs for SST-2, 20 epochs for MRPC, 30 epochs for IMDB 50 epochs for STS-B, 50 epochs for CoLA, 50 epoches for RTE, for the embedding and Transformer layer distillation stage.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]