E FFICIENT REPRESENTATIONS
FOR PRIVACY - PRESERVING INFERENCE
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep neural networks have a wide range of applications across multiple domains such as computer vision and medicine.
In many cases, the input of a model at inference time can consist of sensitive user data, which raises questions concerning the level of privacy and trust guaranteed by such services.
Much existing work has leveraged homomorphic encryption (HE) schemes that enable computation on encrypted data to achieve private inference for multilayer perceptrons and CNNs.
An early work along this direction was CryptoNets, which takes 250 seconds for one MNIST inference.
The main limitation of such approaches is that of compute, which is due to the costly nature of the NTT (number theoretic transform) operations that constitute HE operations.
Others have proposed the use of model pruning and efﬁcient data representations to reduce the number of HE operations required.
In this paper, we focus on improving upon existing work by proposing changes to the representations of intermediate tensors during CNN inference.
We construct and evaluate private CNNs on the MNIST and CIFAR-10 datasets, and achieve over a two-fold reduction in the number of operations used for inferences of the CryptoNets architecture.
I NTRODUCTION
In recent years, deep neural networks have achieved state-of-the-art accuracy for tasks such as image recognition.
They have been deployed in a range of sectors, powering a wide variety of applications such as recommendation systems, medical diagnosis, and content ﬁltering.
Machine Learning as a
Service (MLaaS) is a framework in which cloud services apply machine learning algorithms on usersupplied data to produce an inference result which is then returned to the user.
Cloud systems are an attractive platform for deploying pretrained models due to the relatively low cost and the availability of remote servers.
However, the data has to be decrypted before inference, which allows a serverside adversary to have access to the user’s information.
Homomorphic encryption (HE), can be applied to enable inference to be performed on encrypted data, enabling the result to be delivered to the user without risk of the server accessing the original data or the inference result.
C RYPTO N ETS (Gilad-Bachrach et al., 2016) was the ﬁrst application of HE to secure neural network inference, and leveraged the YASHE’ scheme to perform MNIST classiﬁcations.
C RYPTO N ETS suffers from a high number of homomorphic operations (HOPs), with a single MNIST inference requiring ∼ 290 ,homomorphic multiplications and ∼ 250 seconds of inference latency.
Subsequent works such as
F ASTER C RYPTO N ETS
(Chou et al., 2018) used neural network surgery and a faster encryption scheme to reduce the inference latency of C RYPTO N ETS .
Later works utilised ciphertext rotations as opposed to the SIMD packing scheme, enabling convolutional and fully connected layers to be computed using much fewer HOPs (Juvekar et al., 2018;
Mishra et al., 2020).
This has been shown to reduce the inference latency of MNIST models by more than an order of magnitude, bringing conﬁdence that private inference can be practical.
L O L A (Brutzkus et al., 2019) proposed novel representations for intermediate tensors and their MNIST model requires only 2.2 seconds for one inference.
One drawback of their representations is poorly scalability to harder datasets such as CIFAR-10 due to the limited number of slots per ciphertext acting as a barrier to the size of tensors that are practical.
The limitated set of operations supported by HE schemes prevents the secure computation of nonpolynomial activation functions which impedes model training due to the problem of exploding gradients (Chou et al., 2018).
To address this, others have proposed the use of secure multi-party computation to enable secure computation of non-polynomial activations using multiple parties (Juvekar et al., 2018; Mishra et al., 2020).
Despite enabling use of popular nonpolynomial activations such as ReLU, relying on multiparty computation incurs large amounts of data transfer between parties and the requirement for the parties involved to be online and have feasibly fast data transfer rates.
For example, GAZELLE (Juvekar et al., 2018) requires ∼ 1 GB of data transfer per inference for their CIFAR-10 model, and DELPHI (Mishra et al., 2020) requires ∼ 2 GB per inference with a
ResNet-32 model.
Single-party approaches often choose to approximate the ReLU activation using a second-degree polynomial (Gilad-Bachrach et al., 2016;
Chou et al., 2018; Brutzkus et al., 2019).
In this work, we introduce a framework for secure inference on secure CNNs, designed to reduce the number of HOPs required per inference whilst preserving prediction accuracy.
Our contributions can be summarised as follows:
• We integrate the convolution-packing method from L O L A with the fast matrix-vector product method introduced by Halevi and Shoup (Halevi & Shoup, 2019) and utilised by Juvekar et al.
(2018) in their multi-party computation framework.
Intermediate convolutions are converted into fully-connected layers and computed as matrix-vector products.
We show that utilising the Halevi-Shoup method allows the use of rotations and ciphertext packing to scale better compared with the representations in L O L A , when applied to larger convolutional layers.
We perform a more detailed investigation on the scalability of the methods used in L O L A to larger models and show that they are signiﬁcantly outperformed by our proposed method.
• We compare our framework against L O L A by constructing models for MNIST and CIFAR10.
Our main evaluation criteria is the number of HOPs required per inference for a model.
With the
same
layer
parameters
as
L O L A
, we are able to obtain over a two-fold reduction in the number of
HOPs
per inference.
Our
CIFAR
-10 model achieves similar accuracy to that of L O
L
A
’s
but uses far fewer
operations.
B ACKGROUND AND PREREQUISITES 2. T HREAT MODEL
Our threat model concerns that of the machine learning as a service paradigm (MLaaS), in which the user ﬁrst sends data to a server, which then performs machine learning inference on the received data using some model.
The inference result is then delivered back to the user.
For example, consider an online machine learning service which claims to detect the probability of a person having COVIDfrom an audio recording of their cough.
Suppose that Alice decides to send a recording of her cough to this service, in the hopes of receiving a diagnosis.
There are two key threats in this scenario: (i) the risk of an adversary eavesdropping on the data transmission, and (ii) the risk of the MLaaS provider performing unauthorised access on the user’s data – in this case, the recording produced by
Alice.
The ﬁrst threat can be mitigated using standard cryptographic protocols.
However, the second risk is harder to address, especially if the user data is decrypted before inference Bae et al.
The use of HE mitigates both risks.
The data is encrypted using HE, which is sufﬁcient to prevent an adversary from eavesdropping.
In addition, the provider is only able to perform computations on the encrypted data and will output the inference result without being able to decrypt.
2. H OMOMORPHIC OPERATIONS
Several recent HE schemes such as BFV (Brakerski & Vaikuntanathan, 2011) and CKKS (Cheon et al., 2017) are based on the RLWE problem and support SIMD ciphertext operations.
On a high level, such schemes establish a mapping between real vectors and a plaintext space.
The plaintext space is usually the polynomial ring R =
Z [ X ] / ( X N + 1) .
In particular, this is a cyclotomic polynomial ring R =
Z [ X ] / (Φ M ( X )) where Φ M ( X ) is the M -th cyclotomic polynomial and
M = 2 N is a power of two.
The decoding operation maps an element in R to a vector that is either real or complex, depending on the scheme used.
The encoding operation performs the reverse.
Plaintext polynomials are encrypted into ciphertext polynomials using a public key .
The operations of addition and multiplication can be performed over ciphertexts using an evaluation key .
Since each ciphertext corresponds to a vector of real (or complex) values, a single homomorphic operation between two ciphertexts constitutes an element-wise operation between two vectors.
In addition, such schemes support rotations of the slots within a ciphertext, with the use of Galois automorphisms.
2. F LATTENED CONVOLUTIONS
Consider the convolution of an image I with a ﬁlter f .
For simplicity, assume that both the image and ﬁlter are square, the vertical and horizontal strides of the ﬁlter are equal.
Let I ∈ R d in × d in × c in , and f ∈ R k × k × c in .
Denote the stride as s and padding as p .
Now, the output feature map J is such that J ∈ R d out × d out where
[Equation]
A full convolutional layer that outputs c out feature maps will require a convolution of the input image with each of the c out ﬁlters.
Consider the vector v obtained by ﬂattening each output feature map row-wise.
This can be expressed as a matrix-vector product of the form v = A · w ∈ R d 2 out · c out where
A ∈ R d 2 out · c out × d 2 in · c in and w is the ﬂattened representation of I .
2. F AST CONVOLUTION
The ﬁrst convolutional layer in a CNN can be represented using convolution-packing (Brutzkus et al., 2019).
The convolution of an input image f with a ﬁlter g of width w , height h and depth d is
[Equation]
Observe that the parallelism inherent in this computation enables it to be vectorized as
[Equation]
x = h − 1 (cid:88) y =
d − 1 (cid:88)
z
= g
[ x, y, z ] · F ( x,y,z ) , (2) where F ( x,y,z ) is a matrix such that F ( x,y,z ) ij = f
[ i + x, j + y, z ] .
For an input image I ∈ R c in × d in × d in feature maps and kernel of window size k × k , the input image is represented as k 2 · c in vectors v 1 , . .
. , v k 2 · c in , where v i contains all elements convolved with the i -th value in the ﬁlter.
Denote corresponding ciphertexts as ct 1 , . .
, ct k 2 · c in .
The process of producing the j -th output feature map is now reduced to ciphertext-plaintext multiplication of each ct i with the i -th value in the j -th ﬁlter.
In total, the process requires k 2 · c in ciphertext-scalar multiplications per output feature map, leading to a total of k 2 · c in · c out multiplications.
2. M ATRIX - VECTOR MULTIPLICATION
Multiplying a plaintext weight matrix of A ∈ R m × n with a ciphertext vector can be achieved naively by ﬁrst performing m ciphertext multiplications of the vector with each row of A .
Then for each product ciphertext, we can compute the sum of all elements within it by applying a rotateand-sum procedure (Halevi & Shoup, 2019): we ﬁrst rotate the ciphertext by N/ 2 slots and add it to the original.
Then the procedure is repeated for N/ 4 slots, N/ 8 and so on, until the sum of the ciphertext resides within all slots of the ciphertext.
The resulting dot products can be summed together.
This basic approach requires O ( m log n ) rotations.
Halevi & Shoup (2019) introduced a more efﬁcient method of computing the encrypted matrixvector product A · v for square A .
GAZELLE (Juvekar et al., 2018) extended the approach to support rectangular A ∈ R m × n .
The method works by decomposing A into its m diagonals, denoted { d 1 , d 2 , . . .
d m } such that d i =
[ A i, 0 , A
i +1 , 1 , . . .
, A i + n − 1 ,n − 1 ] .
Note that all row positions are in modulo m .
Then each d i is rotated i positions to align values belonging to the same row of
A into the same column(s), and ﬁnally each rotated diagonal is multiplied with with corresponding rotations of v .
The ciphertexts are summed, and the last stage is to apply a rotate-and-sum procedure to the resulting ciphertext.
Overall, this procedure requires O ( m ) multiplications and O ( m +log 2 n ) rotations.
We propose an improved variant of this approach for our framework described in section 3.1.
M ETHOD
In this section, we present our method for achieving privacy-preserving CNN inference with low numbers of HOPs.
In summary, we adopt the fast convolution method from L O L A but compute the intermediate convolutional layers in a network using the Halevi-Shoup (HS) matrix-vector product method instead.
This enables large convolutions to be performed in far fewer ciphertext rotations than L O L A ’s approach of computing a rotate-and-sum procedure for each row in the weight matrix.
In section 3.1 we explain the approach we use.
In section 3.2, we perform an analysis to show the improvements made by our modiﬁcations compared with the approach from L O L A .
In section 4, we apply our approach to models for the MNIST and CIFAR-10 datasets.
3. P ERFORMING INTERMEDIATE CONVOLUTIONS
In Section 2.3 we state that a convolution can be ﬂattened and represented as a matrix-vector product
A · w ∈ R d 2 out · c out where A ∈ R d 2 out · c out × d 2 in · c in and w is the ﬂattened input.
This product can be computed efﬁciently using the Halevi-Shoup method.
However, neither the original method (Halevi & Shoup, 2019) nor GAZELLE (Juvekar et al., 2018) discuss the constraints the ciphertext slot count N imposes in this context.
In Proposition 1 we show that certain constraints must be applied on the sizes of the input and output of the convolution.
In particular, we must have N ≥ d 2 in · c in + d 2 out · c out − 1 , unless d out and c out are both powers of 2.
Proposition 1.
Let N denote the ciphertext slot count, and n = d 2 in · c in be the size of the convolution input, and m =
d 2 out · c out be the size of the output.
The basic Halevi-Shoup method, which takes m − 1 + (cid:100) log 2 (cid:0) m + n − 1 m (cid:1)
[Equation]
rotations, requires the condition that N ≥ m + n − 1 .
If this does not hold, but it is the case that m = 2 l , 0 ≤ l ≤ log 2 N , then m − 1 + log 2 Nm rotations are required.
Proof.
Applying the Halevi-Shoup technique requires m ciphertext diagonals
d (1) , . . .
, d ( m ) of length n to be extracted, rotated and summed.
Note that d ( i ) j
= A i + j,j .
Now, if N ≥ m + n − 1 then
N is sufﬁciently large for all rotations to be performed without wrapping around the ciphertext.
If
N < m + n − 1 , then wrap-around will occur for at least one of the diagonal ciphertexts during rotation.
For any slot d ( i ) j , the rotation by i positions will shift the slot to position k
= i + j ( mod N ) .
The slot d ( i ) j is from the row of A with index r
= i + j ( mod m ) , and so must be shifted into an index that is equivalent to r in modulo m , in order for the rotate-and-sum algorithm to be used.
If
N is an integer multiple of m , then we see k ≡ r ( mod m ) indeed holds.
Otherwise, wrap-around will cause the diagonals to be misaligned when summed together.
Since N is a power of 2, the requirement that m divides N is satisﬁed whenever d out and c out are also powers of 2 such that 2 log 2 d out + log 2 c out ≤ log 2 N .
Based on Proposition 1, we utilise a procedure to ensure that the method can be applied for all choices of ( d in , d out , c in , c out ) where d 2 out · c out ≤ N : if the condition that N ≥ m + n − 1 does not hold, then we ‘round’ m to the closest power of 2 not less than itself, and add corresponding rows ﬁlled with 0’s to the weight matrix.
This procedure is illustrated below for the simple case where
N = 4 , m = 3 and n = 4 .
Notice that the padding allows the diagonals to align correctly after rotation.
-Shoup method that utilises a padding method.
3. C OMPARISON WITH L O L A
Firstly, we follow the terminology used by Brutzkus et al.
and deﬁne dense , sparse , stacked and interleaved representations for a vector v ∈ R n .
• Sparse : v is represented with n ciphertexts, where the i -th ciphertext contains the i -th element of v in all its slots.
• Dense : v is represented as a single ciphertext where the ﬁrst n slots correspond to the values in v .
Stacked : v is represented as a single ciphertext that contains as many copies of v as the ciphertext slot count permits.
• Interleaved : v is represented as a single ciphertext similar to the dense representationhowever, the slots are shufﬂed by some permutation.
L O L A proposes a dense-vector row-major matrix-vector product method (which we call L O L Adense) that maps the input vector in the dense representation to an output vector in the sparse representation.
Let A ∈ R m × n .
The dense-vector row-major method computes Av 1 = v 2 via a multiplication per row of A and summing the elements inside each product vector using the rotateand-sum procedure.
This is the basic approach described in Section 2.5, except the dot products are not summed together.
They also propose a stacked-vector row-major method (which we call L O L A -stacked) that requires input vector in the stacked representation and provides the output in the interleaved representation.
To obtain the stacked representation, they ﬁrst pack k copies of v into a single ciphertext where k = N/δ ( n ) and δ ( n ) = 2 (cid:100) log 2 n (cid:101) is the smallest power of 2 greater than or equal to n .
The stacked vector is then multiplied with corresponding stacked rows of A , before rotate-and-sum is applied.
The main drawback of this method is its reliance on the ability to pack many copies of v into a ciphertext, which requires N to be much larger than n .
For large convolutions, this is hard to achieve.
We derive the number of operations this method takes.
m
[Figure]
Remark 1.
The stacked-vector row-major method proposed by Brutzkus et al.
(2019) requires (cid:100) mk (cid:101)
( k + (cid:100) log 2 n (cid:101) − 1) + (cid:100) mk (cid:101) − 1 rotations and (cid:100) mk (cid:101) multiplications.
Proof.
The stacking in Brutzkus et al.
(2019) is done is using k − 1 rotations and additions.
Then the point-wise multiplication requires a single SIMD multiplication between the ciphertext and k
(stacked) rows of A .
Finally, (cid:100) log 2 n (cid:101) rotations and additions are performed to compute the k elements of the product vector.
Since N is not always sufﬁciently large to pack m copies of n , this procedure has to be performed (cid:100) mk (cid:101) times.
To produce all elements of the product vector, (cid:100) mk (cid:101) ( k + (cid:100) log 2 n (cid:101) − 1) rotations and (cid:100) mk (cid:101) multiplications are performed, before (cid:100) mk (cid:101)
− 1 rotations are performed to bring the (cid:100) mk (cid:101) ciphertexts into a single one.
Note that the produced ciphertext(s) contain the product elements in an interleaved format, i.e. a permutation.
Suppose we are passing a 4096-length representation into a fully connected layer to map to a 64length embedding, and let N = 16384 .
L O L A -dense would require 64 · log 2 4096 =
768 rotations and 64 multiplications; L O L A -stacked would require 1023 rotations and 64 multiplications.
Now, the Halevi-Shoup product approach would require only 64 + log 2 8192 =
77 rotations and 64 multiplications.
Since rotations are the most expensive operation, we compare the number of rotations required by each of the three methods in Figure 2.
It can be shown that for any n > 1 , L O L A -stacked uses fewer rotations than L O L A -dense.
However,
L O L A -dense can be used to compute two layers instead of one.
For large inputs and output layers (relative to N ), however, both methods require signiﬁcantly more rotations than HS.
L O L A -stacked relies on m · δ ( n ) /N being small whereas L O L A -dense relies on log 2 n being small.
With HS, even if n = N , log 2 n is insigniﬁcant compared to m .
In general, we believe that having the number of rotations be linear to only m is beneﬁcial since neural networks typically down-sample or pool the data to produce denser, higher-level representations, and so can expect n ≥ m generally.
It should be noted that there are exceptions - such as bottleneck layers.
E XPERIMENTS
We conduct experiments on the MNIST and CIFAR-10 datasets.
We ﬁrst apply our approach to the
C RYPTO N ETS architecture used in L O L A , to create a model C RYPTO N ETS -HS.
The same architecture was shown to achieve close 98.95% accuracy by Gilad-Bachrach et al.
and we observe similar performance using their training parameters.
Training is conducted using TensorFlow (Abadi et al., 2015).
The architecture is then converted into a sequence of homomorphic operations.
We use the SEAL library (Chen et al., 2017) for this.
For reference, the original C RYPTO N ETS architecture is shown in Figure 3.
L O L A uses a combination of their stacked and interleaved representations for their intermediate convolutions.
We opt to use the efﬁcient approach instead.
To reduce consumption of instruction depth, linear layers without activations between them are composed together.
For instance, each convolution-pooling block of C RYPTO N ETS -HS is a single linear layer.
This is done to enable our baseline to match the architecture in L O L A .
It should be noted that omitting activations can reduce representational power and made not be ideal in practice.
The models are implemented using operations provided by the SEAL library, and inference is performed on a standard desktop processor.
We utilise only a single thread.
We measure the number of homomorphic operations required per single inference, as well as model test accuracy.
Using the proposed improvements we also construct models ME and CE, for MNIST and CIFAR10, respectively.
The proposed architectures have reduced memory requirements and layer parameters that are better suited towards the HS method:
• Model ME has similar test accuracy to C RYPTO N ETS -HS but is designed in consideration of the way we are computing the layers.
We note that applying the HS method in computing layer 6 of C RYPTO N ETS -HS requires setting m = 100 and n = 845 .
Speciﬁcally, we reduce the kernel size of the ﬁrst convolution from 5 × 5 to 3 × 3 , and the size of the ﬁrst dense layer from 100 to 32 .
To account for the reduced representational power of the ﬁrst convolution, the stride is reduced from 2 to 1 .
This notably does not add any homomorphic operations to the computation.
We are able to achieve 98.7% test accuracy using ME after 100 epochs of training with the Adam optimiser.
• Model CE is larger, and requires more operations to compute.
We initially construct a model with a second convolutional layer with an output tensor of size 8 × 8 with 64 channels, and then follow the approach from Lu et al.
and use SVD to factorise this layer into a smaller sub-convolution followed by a 1 × 1 convolution which is efﬁcient to compute using the convolution-packing method described in L O L A .
The smaller sub-convolution is computed using the HS method.
The ﬁnal trained model has a test accuracy of 73 .
[Table]
R ESULTS AND DISCUSSION
Table 2 shows a breakdown of the operations for the models discussed so far.
In homomorphic encryption applied to neural networks, the most expensive homomorphic operation is rotation, with a worst case time complexity of performing both a number theoretic transform (NTT) and an inverse
NTT on vectors of length N .
We observe that C RYPTO N ETS -HS requires a total of 122 rotations, as shown in Table 2a.
L O L A -MNIST requires a total of 380 rotations 1 for the same architecture.
We notice that at both the convolution-pooling blocks in the C RYPTO N ETS architecture, the HS implementation has fewer rotations than L O L A .
The reduction in rotations is a direct result of applying the HS method to compute ﬂattened convolutions, in which the number of rotations scales with the size of the output tensor, rather than the size of the input. L O L A uses the stacked-vector row-major method for the ﬁrst convolution, and the dense-vector row-major method for the second convolution. In Section 3.2 we discussed that both of these scale poorly compared to the HS method.
Now, the optimised ME architecture requires only 56 rotations per inference, but still achieves a high test accuracy of 98.7%, which is close to the 98 .
95% test accuracy achieved by L O L A -MNIST, suggesting that careful selection of layer sizes can lead to great reductions in inference latency.
We do note that C RYPTO N ETS -HS requires more multiplications than L O L A -MNIST
(245 vs. 148).
This is due to the stacked-vector row-major approach reducing the number of multiplications needed.
However, the reduction in rotations exceeds the increase in multiplications.
[Table] [Table] In terms of latency, the C RYPTO N ETS -HS model requires 2.7 seconds, whereas ME requires only 0.97 seconds.
For reference, L O L A -MNIST requires 2.2 seconds per inference; however, they utilise 8 cores on a server CPU whereas our homomorphic operations are run on a single core of a desktop processor.
For the CE model, we provide a model with sightly lower test accuracy ( 73 .
1% vs. 74 . 1% )
but requiring less than 10% the number of rotations than LoLa’s CIFAR-10 model.
This due to both ensuring that intermediate tensor sizes ﬁt well into the number of available slots, and also the use of the HS method.
The number of operations for each layer is shown in 2b.
It should be noted that a sizeable portion of this reduction comes from using a smaller model.
The exact model from L O L A was not implemented due to its high memory requirements. However, we expect that utilising our approach for L O L A ’s CIFAR-10 architecture would still signiﬁcantly reduce the inference latency.
The main takeaway from the results is that the application of the HS method to the framework proposed by L O L A can signiﬁcantly reduce the number of rotations, which are the most computationally expensive HE operation.
In addition, our models ME and CE are more efﬁcient than the ones proposed by L O L A whilst achieving similar accuracy.
C ONCLUSION
Privacy-preserving inference using homomorphic encryption is largely constrained by the computational requirements of the operations.
We propose improvements over L O L A to achieve lower latencies computing intermediate convolutions, resulting in over a two-fold reduction in the number of rotations for the same MNIST architecture.
It is clear that further improvements can made along this direction, especially in the topic of automatically selecting suitable layer parameters to set the trade-off between inference latency and model accuracy (Lou et al., 2020).