Volley Revolver: A Novel Matrix-Encoding Method for Privacy-Preserving Neural Networks (Inference)
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In this work, we present a novel matrix-encoding method that is particularlyconvenient for neural networks to make predictions in a privacy-preserving mannerusing homomorphic encryption.
Based on this encoding method, we implement aconvolutional neural network for handwritten image classiﬁcation over encryption.
For two matrices A and B to perform homomorphic multiplication, the main ideabehind it, in a simple version, is to encrypt matrix A and the transpose of matrix
B into two ciphertexts respectively.
With additional operations, the homomorphicmatrix multiplication can be calculated over encrypted matrices efﬁciently.
Forthe convolution operation, we in advance span each convolution kernel to a matrixspace of the same size as the input image so as to generate several ciphertexts, each of which is later used together with the ciphertext encrypting input images for calculating some of the ﬁnal convolution results.
We accumulate all these intermediate results and thus complete the convolution operation.
In a public cloud with 40 vCPUs, our convolutional neural network implementationon the MNIST testing dataset takes ∼ 287 seconds to compute ten likelihoods of32 encrypted images of size 28 × 28 simultaneously.
The data owner only needs toupload one ciphertext ( ∼ 19 .
8 MB) encrypting these 32 images to the public cloud.
Introduction
Machine learning applied in some speciﬁc domains such as health and ﬁnance should preserve privacywhile processing private or conﬁdential data to make accurate predictions.
In this study, we focus onprivacy-preserving neural network inference, which aims to outsource a well-trained inference modelto a cloud service in order to make predictions on private data.
For this purpose, the data shouldbe encrypted ﬁrst and then sent to the cloud service that should not be capable of having access tothe raw data.
Compared to other cryptology technologies such as Secure Multi-Party Computation,
Homomorphic Encryption (HE) provides the most stringent security for this task.
Combining HE with Convolutional Neural Networks (CNN) inference has been receiving moreand more attention in recent years since Gilad-Bachrach et al.
[6] proposed a framework called
Cryptonets .
Cryptonets applies neural networks to make accurate inferences on encrypted datawith high throughput.
Chanranne et al.
[2] extended this work to deeper CNN using a differentunderlying software library called HElib [7] and leveraged batch normalization and training processto develop better quality polynomial approximations of the ReLU function for stability and accuracy.
Chou et al.
[4] developed a pruning and quantization approach with other deep-learning optimizationtechniques and presented a method for encrypted neural networks inference, Faster CryptoNets .
Brutzkus et al.
[1] developed new encoding methods other than the one used in Cryptonets forrepresenting data and presented the Low-Latency CryptoNets ( LoLa ) solution.
Jiang et al.
[9] proposed an efﬁcient evaluation strategy for secure outsourced matrix multiplication with the help ofa novel matrix-encoding method.
Contributions In this study, our contributions are in three main parts:
We introduce a novel data-encoding method for matrix multiplications on encrypted matrices,
Volley Revolver , which can be used to multiply matrices of arbitrary shape efﬁciently.2.
We propose a feasible evaluation strategy for convolution operation, by devising an efﬁcienthomomorphic algorithm to sum some intermediate results of convolution operations.
We develop some simulated operations on the packed ciphertext encrypting an image dataset
as if there were multiple virtual ciphertexts inhabiting it, which provides a compelling new
perspective of viewing the dataset as a three-dimensional structure.
Preliminaries
Let “ ⊕ ” and “ ⊗ ” denote the component-wise addition and multiplication respectively betweenciphertexts encrypting matrices and the ciphertext ct .P
the encryption of a matrix P .
Let I ( m )
[ i ][ j ]represent the single pixel of the j -th element in the i -th row of the m -th image from the dataset.
Homomorphic Encryption Homomorphic Encryption is one kind of encryption but has its characteristic in that over an HE system operations on encrypted data generate ciphertexts encryptingthe right results of corresponding operations on plaintext without decrypting the data nor requiringaccess to the secret key.
Since Gentry [5] presented the ﬁrst fully homomorphic encryption scheme,tackling the over three decades problem, much progress has been made on an efﬁcient data encodingscheme for the application of machine learning to HE.
Cheon et al.
[3] constructed an HE scheme(CKKS) that can deal with this technique problem efﬁciently, coming up with a new procedurecalled rescaling for approximate arithmetic in order to manage the magnitude of plaintext.
Theiropen-source library, HEAAN , like other HE libraries also supports the Single Instruction Multiple Data(aka SIMD) manner [11] to encrypt multiple values into a single ciphertext.
Given the security parameter, HEAAN outputs a secret key sk , a public key pk , and other public keysused for operations such as rotation.
For simplicity, we will ignore the rescale operation anddeem the following operations to deal with the magnitude of plaintext automatedly.
HEAAN has thefollowing functions to support the HE scheme:
Enc pk ( m ): For the public key pk and a message vector m , HEAAN encrypts the message minto a ciphertext ct .2.
Dec sk ( ct ): Using the secret key, this algorithm returns the message vector encrypted by theciphertext ct .3.
Add ( ct 1 , ct 2 ):
This operation returns a new ciphertext that encrypts the message Dec sk ( ct 1 )
⊕ Dec sk ( ct 2 ).
Mul ( ct 1 , ct 2 ):
This procedure returns a new ciphertext that encrypts the message Dec sk ( ct 1 )⊗ Dec sk ( ct 2 ).5.
Rot ( ct , l ): This procedure generates a ciphertext encrypting a new plaintext vector obtainedby rotating the the original message vector m encrypted by ct to the left by l positions.
Database Encoding Method For brevity, we assume that the training dataset has n samples withf features and that the number of slots in a single ciphertext is at least n × f .
A training dataset isusually organized into a matrix Z each row of which represents an example.
Kim et al.
[10] proposean efﬁcient database encoding method to encrypt this matrix into a single ciphertext in a row-by-rowmanner.
They provide two basic but important shifting operations by shifting 1 and f positionsrespectively: the incomplete column shifting and the row shifting.
The matrix obtained from matrix
Z by the incomplete column shifting operation is shown as follows:
Z =  
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
f ] . [Equation] [Equation] [Equation]
[Equation]
[ n ][
[Equation]
z [ n ][ f ]   incomplete column shifting (cid:55)−−−−−−−−−−−−−−→   z
[1][2] z
z [2][1] z [2][2] z
z [3][1] .
z [ n ][2] z [ n ][3] . . .
z [1][1]   .
Han et al.
[8] summarize another two procedures, SumRowVec and SumColVec , to compute thesummation of each row and column respectively.
The results of two procedures on Z are as follows:
SumRowVec( Z
[Equation]
 
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
z [
[Equation]
][ f ] (cid:80) ni =
[Equation]
z [ i ][1] [Equation] [ i ][2] [Equation]
(cid:80)
[Equation]
[Equation]
][ f ] .
[Equation]
[Equation]
[Equation]
[Equation]
(cid:80) ni
[Equation]
[ i ][1] (cid:80) ni
[Equation]
z [ i ][2]
[Equation]
[Equation]
(cid:80) ni =
[Equation] z [ i ][ f ]   , SumColVec( Z ) =
  (cid:80) fj
[Equation]
z [1][ j ] (cid:80)
[Equation]
We propose a new useful procedure called SumForConv to facilitate convolution operation for everyimage. Below we illustrate the result of SumForConv on Z taking the example that n and f are both4 and the kernel size is 3 × 3 :
[Equation]
[Equation]
[Equation]
] 0  0  0  , where s [ i ][ j ] = (cid:80) i +2 p
=
i (cid:80) j +2 q =
j z [ p ][ q ] for 1 ≤ i, j ≤ 2 .
In the convolutional layer, SumForConv can helpto compute some partial results of convolution operation for an image simultaneously.
Technical details
We introduce a novel matrix-encoding method called Volley Revolver , which is particularlysuitable for secure matrix multiplication.
The basic idea is to place each semantically-completeinformation (such as an example in a dataset) into the corresponding row of a matrix and encryptthis matrix into a single ciphertext.
When applying it to private neural networks, Volley Revolverputs the whole weights of every neural node into the corresponding row of a matrix, organizes all thenodes from the same layer into this matrix, and encrypts this matrix into a single ciphertext.
3.1 Encoding Method for Matrix Multiplication
Suppose that we are given an m × n matrix A and a n × p matrix B and suppose to compute the matrix
C of size m × p , which is the matrix product A · B with the element C [ i ][ j ] = (cid:80) nk
=1 a [ i ][ k ] × b [ k ][ j ] :
A =   a [1][1] a [1][2] . . .
a [1][ n ] a [2][1] a [2][2] . . .
a [2][ n ] .
a [ m ][1] a [ n ][2] . . .
a [ m ][ n ]   , B =   b [1][1] b [1][2] . . .
b [1][ p ] b [2][1] b [2][2] . . .
b [2][ p ] .
b [ n ][1] b [ n ][2] . . .
b [ n ][ p ]   .
For simplicity, we assume that each of the three matrices A , B and C could be encrypted intoa single ciphertext.
We also make the assumption that m is greater than p , m > p .
We willnot illustrate the other cases where m ≤ p , which is similar to this one.
When it comes to thehomomorphic matrix multiplication, Volley Revolver encodes matrix A directly but encodesthe padding form of the transpose of matrix B , by using two row-ordering encoding maps.
Formatrix A , we adopt the same encoding method that [9] did by the encoding map τ a : A
(cid:55)→ ¯ A =( a [1+( k/n )][1+( k % n )] )
≤ k<m × n .
For matrix B , we design a very different encoding method from [9] 104 for Volley Revolver : we transpose the matrix B ﬁrst and then extend the resulting matrix in the vertical direction to the size m × n .
Therefore Volley Revolver adopts the encoding mapτ b : B (cid:55)→ ¯ B = ( b [1+( k % n )]
[1+(( k/n )% p )] )
≤ k<m × n , obtaining the matrix from mapping τ b on B :
 b [1][1] b [1][2] . . .
b [1][ p ] b [2][1] b [2][2] . . .
b [2][ p ] .
b [ n ][1] b [ n ][2] . . .
b [ n ][ p ]   τ b (cid:55)−→   b [1][1] b [2][1] . . .
b [ n ][1] b [1][2] b [2][2] . . .
b [ n ][2] .
b [1][ p ] b [2][ p ] . . . b [ n ][ p ] b [1][1] b [2][1] . . .
b [ n ][1] .
b [1][1+( m − 1)% p ] b [2][1+( m − 1)% p ] . . .
b [ n ][1+( m − 1)% p ]   .
Homomorphic Matrix Multiplication We report an efﬁcient evaluation algorithm for homomorphicmatrix multiplication.
This algorithm uses a ciphertext ct .R
encrypting zeros or a given value suchas the weight bias of a fully-connected layer as an accumulator and an operation RowShifter toperform a speciﬁc kind of row shifting on the encrypted matrix ¯ B .
RowShifter pops up the ﬁrst rowof ¯ B and appends another corresponding already existing row of ¯ B :  b [1][1] b [2][1] . . .
b [ n ][1] b [1][2] b [2][2] . . .
b [ n ][2] .
b [1][ p ] b [2][ p ] . . . b [ n ][ p ] b [1][1] b [2][1] . . .
b [ n ][1] .
b [1][ r ] b [2][ r ] . . .
b [ n ][ r ] 
 RowShifter ( ¯ B ) (
cid:55)−−−−−−−−−→   b [1][2] b [2][2] . . .
b [ n ][2] .
b [1][ p ] b [2][ p ] . . . b [ n ][ p ] b [1][1] b [2][1] . . .
b [ n ][1] .
b [1][ r ] b [2][ r ] . . .
b [ n ][ r ] b [1][( r +1)% p ] b [2][( r +1)% p ] . . .
b [ n ][( r +1)% p ]   .
For two ciphertexts ct .A and ct .
¯ B , the algorithm for homomorphic matrix multiplication has piterations.
For the k -th iteration where 0 ≤ k < p there are the following four steps:
Step 1.
This step uses RowShifter on ct .
¯ B to generate a new ciphertext ct .[-¯ B 
-]
¯ B 1 and then computes the homomorphic multiplication between ciphertexts ct .A and ct .
¯ B 1 to get the resultingproduct
ct .A
¯ B 1 .
When k = 0 , in this case RowShifter just return a copy of the ciphertext ct .
¯ B .
Step 2.
In this step, the public cloud applies SumColVec on ct .A
¯ B 1 to collect the summation ofthe data in each row of A ¯ B 1 for some intermediate results, and obtain the ciphertext ct .D .
Step 3.
This step designs a special matrix F to generate a ciphertext ct .F for ﬁltering out theredundancy element in D by one multiplication
Mul ( ct .F , ct .D ), resulting the ciphertext ct .D 1 .
Step 4.
The ciphertext ct .R is then used to accumulate the intermediate ciphertext ct .D 1 .
The algorithm will repeat Steps 1 to 4 for p times and ﬁnally aggregates all the intermediate ciphertexts, returning the ciphertext ct .C .
Algorithm 1 shows how to perform our homomorphic matrixmultiplication.
Figure 1 describes a simple case for Algorithm 1 where m = 2 , n = 4 and p = 2 .
The calculation process of this method, especially for the simple case where m = p , is intuitivelysimilar to a special kind of revolver that can ﬁre multiple bullets at once (The ﬁrst matrix A issettled still while the second matrix B is revolved).
That is why we term our encoding method" Volley Revolver ".
In the real-world cases where m mod p = 0 , the operation RowShifter canbe reduced to only need one rotation RowShifter =
Rot ( ct , n ) , which is much more efﬁcient andshould thus be adopted whenever possible.
Corresponding to the neural networks, we can set thenumber of neural nodes for each fully-connected layer to be a power of two to achieve this goal.
3.2 Homomorphic Convolution Operation
In this subsection, we ﬁrst introduce a novel but impractical algorithm to calculate the convolutionoperation for a single grayscale image of size h × w based on the assumption that this single imagecan happen to be encrypted into a single ciphertext without vacant slots left, meaning the number Nof slots in a packed ciphertext chance to be N = h × w . We then illustrate how to use this method tocompute the convolution operation of several images of any size at the same time for a convolutionallayer after these images have been encrypted into a ciphertext and been viewed as several virtual
Input A B A ∈ R m × n
B ∈ R n × p
B Volley Revolver Encoding
¯ B ∈ R × n Set do 4: ct T ←
5: ct
T ← Mul Set (cid:46) n 8: do 9: F i i idx n
←10:
11: ct
T ← Mul pk
F ct
T )
12: ct
C
Add
13: end for 14: return ct .C · a 0 a 1 a 2 ab
b 2 b 4 ba 4 a 5 a 6 ab 1 b 3 b 5 b× b 0 ba 0 a 1 a 2 ab 2 ba 4 a 5 a 6 ab 4 bb 6 b        ⊗ a 0 a 1 a 2 ab 0 b 2 b 4 ba 4 a 5 a 6 ab 1 b 3 b 5 bc
= a 0 · b 0 + a 1 · b 2 + a 2 · b 4 + a 3 · b 6 c 0 c 0 c 0 cc 3 c 3 c 3 cc 3 = a 4 · b 1 + a 5 · b 3 + a 6 · b 5 + a 7 · bc 0
c 3 ⊗ a 0 a 1 a 2 ab 1 b 3 b 5 ba 4 a 5 a 6 ab 0 b 2 b 4 bc 1 = a 0 · b 1 + a 1 · b 3 + a 2 · b 5 + a 3 · b 7 c 1 c 1 c 1 cc 2 c 2 c 2 cc 2 = a 4 · b 0 + a 5 · b 2 + a 6 · b 4 + a 7 · b0 c 1 c 2  × b 0 ba 0 a 1 a 2 ab 2 ba 4 a 5 a 6 ab 4 bb 6 b⊕ c 0  0 c 1 0 c 3 c 2  c 0 c 1 c 2 c 3 c 0 cc 2 c
Encrypt Encoding
[Figure]
ciphertexts inhabiting this real ciphertext.
For simplicity, we assume that the image is grayscale andthat the image dataset can be encrypted into a single ciphertext.
An impractical algorithm Given a grayscale image I of size h × w and a kernel K of size k × kwith its bias k 0 such that h and w are both greater than k , based on the assumption that this imagecan happen to be encrypted into a ciphertext ct .I
with no more or less vacant slots, we present anefﬁcient algorithm to compute the convolution operation.
We set the stride size to the usual defaultvalue (1 , 1) and adopt no padding technique in this algorithm.
Before the algorithm starts, the kernel K should be called by an operation that we term
Kernelspanner to in advance generate k 2 ciphertexts for most cases where h ≥ 2 · k − 1 andw ≥ 2 · k − 1 , each of which encrypts a matrix P i for 1 ≤ i ≤ k 2 , using a map to span the k × k 149 kernel to a h × w matrix space.
For a simple example that h = 4 , w = 4 and k = 2 , Kernelspanner generates 4 ciphertexts and the kernel bias k 0 will be used to generate a ciphertext:(cid:20) k
[Equation]
k
[Equation]

 ,   0 kk 0 kk 0 kk 0 kk 
 ,      kkkk 2 kkkk 4     ,
[Equation] [Equation]
[Equation] [Equation]
[Equation]
     0 kk 0 kk     
 .
Our impractical homomorphic algorithm for convolution operation also needs a ciphertext ct .R
toaccumulate the intermediate ciphertexts, which should be initially encrypted by the kernel bias k 0 .
This algorithm requires k × k iterations and the i -th iteration consists of the following four steps for1 ≤ i ≤ k 2 :
Step 1.
For ciphertexts ct .I
and ct .P
i , this step computes their multiplication and returns theciphertext
ct .IP
i =
Mul ( ct .I, ct .P i ) .
Step 2.
To aggregate the values of some blocks of size k × k , this step applies the procedure
SumForConv on the ciphertext ct .IP
i , obtaining the ciphertext ct .D .
Step 3.
The public cloud generates a ciphertext encrypting a specially-designed matrix in order toﬁlter out the garbage data in ct .D by one multiplication, obtaining a ciphertext ct .
¯ D .
Step 4.
In this step, the homomorphic convolution-operation algorithm updates the accumulatorciphertext ct .R
by homomorphically adding ct .
¯ D to it, namely ct .R =
Add ( ct .R, ct .
¯ D ) .
Note that Steps 1–3 in this algorithm can be computed in parallel with k × k threads.
We describehow to compute homomorphic convolution operation in Algorithm 2 in detail.
Figure 2 describes asimple case for the algorithm where h = 3 , w = 4 and k = 3 {+.i 
+}
{+i 12 i 13 ii 
+}
{+i 22 i 23 ii 31 i 32 i 33 ik 1 k 2 kk 4 k 5 kk 7 k 8 kkk 
+}
k
{+i 11 i 12 i 13 ii 21 i 22 i 23 ii 31 i 32 i 33 i⊗ k 1 k 2 k k 4 k 5 k k 7 k 8 k c           c 
+}
{+= i 11 · k 1 + ··· + i 33 · ki 11 i 12 i 13 ii 
+}
i 22 i 23 ii 31 i 32 i 33 i⊗ k 1 k 2 k k 4 k 5 k k 7 k 8 k c          c 2 = i 12 · k 1 + ··· + i 34 · k
[Figure]
w
[Equation]
and k = Next, we will show how to make this impractical homomorphic algorithm work efﬁciently in realworld cases.
Set
← do 4: for for 7:
9: for i := 0 to k − 1 do 10: for j := 0 to k − 1 do 11: ct
T ←
I
S i
12: ct
T ← SumForConv ( ct
T )
(cid:46) Design a matrix to ﬁlter out the redundant values 13:
Set F ←
14: for hth := 0 to h − 1 do 15: for wth := 0 to w − 1 do 16: if ( wth − i ) mod k = 0 and wth + k ≤ w and 17: ( hth − j mod k
=
and hth + k ≤ h then 18:
F [ hth ][ wth ] ←19: end if 20: end for 21: end for 22: ct .T
← Mul ( Enc pk ( F
, ct .T )
(cid:46)
To accumulate the intermediate results 23: ct .I
s ← Add ( ct .I
s , ct .T )
24: end for 25: end for 26: return ct .I
s
Encoding Method for Convolution Operation For simplicity, we assume that the dataset X ∈ 169 R m × f can be encrypted into a single ciphertext ct .X ,
m is a power of two, all the images aregrayscale and have the size h × w .
Volley Revolver encodes the dataset as a matrix using thedatabase encoding method [10] and deals with any CNN layer with a single formation.
In most cases,h × w < f , if this happened, zero columns could be used for padding.
Volley Revolver extendsthis database encoding method [10] with some additional operations to view the dataset matrix X asa three-dimensional structure.
Algorithm 2 is a feasible and efﬁcient way to calculate the secure convolution operation in an HEdomain.
However, its working-environment assumption that the size of an image is exactly the lengthof the plaintext, which rarely happens, is too strict to make it a practical algorithm, leaving thisalgorithm directly useless.
In addition, Algorithm 2 can only deal with one image at a time due tothe assumption that a single ciphertext only encrypts only one image, which is too inefﬁcient forreal-world applications.
To solve these problems, Volley Revolver performs some simulated operations on the ciphertextct .X to treat the two-dimensional dataset as a three-dimensional structure.
These simulated operations together could simulate the ﬁrst continual space of the same size as an image of each row of thematrix encrypted in a real ciphertext as a virtual ciphertext that can perform all the HE operations.
Moreover, the number of plaintext slots is usually set to a large number and hence a single ciphertextcould encrypt several images.
For example, the ciphertext encrypting the dataset X ∈ R m × f could be used to simulate m virtual ciphertexts vct i for 1 ≤ i ≤ m , as shown below:
Enc  
I (1)[1][1]
I (1)[1][2] . . .
I (1)[ h ][ w ] 0 . . .
I (2)[1][1]
I (2)[1][2] . . .
I (2)[ h ][ w ] 0 . . ....
I ( m ) [1][1]
I ( m ) I ( m )
[ h ][ w ] 0 . . .
 −→ 
 vEnc  
I (1)[1][1] ... I (1)[1]
[ w ] ...
I (1)[ h ][1] ... I (1)
[ h ][ w ]   0 . .
vEnc  
I ( m ) [1][1] ... I ( m ) [1][ w ] ...
I ( m )
[ h ][1] ... I ( m )
[ h ][ w ]   0 . . .
 .
Similar to an HE ciphertext, a virtual ciphertext has virtual HE operations: vEnc , vDec , vAdd ,vMul , vRescale , vBootstrapping and vRot .
Except for vRot , others can be all inherited from thecorresponding HE operations.
The HE operations, Add , Mul , Rescale and Bootstrapping , resultin the same corresponding virtual operations: vAdd , vMul , vRescale and vBootstrapping .
Thevirtual rotation operation vRot is much different from other virtual operations: it needs two rotationoperations over the real ciphertext.
We only need to simulate the rotation operation on these virtualciphertexts to complete the simulation.
The virtual rotation operation vRot ( ct , r ) , to rotate all thevirtual ciphertexts dwelling in the real ciphertext ct to the left by r positions, has the followingsimulation result:
[Equation]
 
[Equation]
To bring all the pieces together, we can use Algorithm 2 to perform convolution operations forseveral images in parallel based on the simulation virtual ciphertexts.
The most efﬁcient part ofthese simulated operations is that a sequence of operations on a real ciphertext results in the samecorresponding operations on the multiple virtual ciphertexts, which would sufﬁce the real-worldapplications.
Privacy-preserving CNN Inference
Limitations on applying CNN to HE Homomorphic Encryption cannot directly compute functions such as the ReLU activation function.
We use Octave to generate a degree-three polynomialby the least square method and just initialize all the activation layers with this polynomial, leavingthe training process to determine the coefﬁcients of polynomials for every activation layer.
Othercomputation operations, such as matrix multiplication in the fully-connected layer and convolutionoperation in the convolutional layer, can also be performed by the algorithms we proposed above.
Neural Networks Architecture We adopt the same CNN architecture as [9] but with some differenthyperparameters.
Our encoding method Volley Revolver can be used to build convolutional neuralnetworks as deep as it needs.
However, in this case, the computation time will therefore increaseand bootstrapping will have to be used to refresh the ciphertext, resulting in more time-consuming.
Table 1 gives a description of our neural networks architecture on the MNIST dataset.
Experimental Results
We use C++ to implement our homomorphic CNN inference.
Our complete source code is publiclyavailable at https://anonymous.4open.science/r/HE-CNNinfer-106F/ .
[Table]
Database We evaluate our implementation of the homomorphic CNN model on the MNIST datasetto each time calculate ten likelihoods for 32 encrypted images of handwritten digits.
The MNISTdatabase includes a training dataset of 60 000 images and a testing dataset of 10 000, each image ofwhich is of size 28 × 28 .
For such an image, each pixel is represented by a 256-level grayscale andeach image depicts a digit from zero to nine and is labeled with it.
Building a model in the clear
In order to build a homomorphic model, we follow the normalapproach for the machine-learning training in the clear — except that we replace the normal ReLUfunction with a polynomial approximation: we (1) train our CNN model described in Table 1 withthe MNIST training dataset being normalized into domain [0 , 1] , and then we (2) implement thewell-trained resulting CNN model from step (1) using the HE library and HE programming.
For step (1) we adopt the highly customizable library keras with Tensorflow , which provides uswith a simple framework for deﬁning our own model layers such as the activation layer to enact thepolynomial activation function.
After many attempts to obtain a decent CNN model, we ﬁnally get a
CNN model that could reach a precision of 98 .
66% on the testing dataset.
We store the weights ofthis model into a CSV ﬁle for the future use.
In step (2) we use the HE programming to implementthe CNN model, accessing its weights from the CSV ﬁle generated by step (1).
We normalize the
MNIST training dataset by dividing each pixel by the ﬂoating-point constant 255.
Classifying encrypted inputs We implement our homomorphic CNN inference with the library
HEAAN by [3].
Note that before encrypting the testing dataset of images, we also normalize the
MNIST testing dataset by dividing each pixel by the ﬂoating-point constant 255, just like the normalprocedure on the training dataset in the clear.
Parameters .
We follow the notation of [10] and set the HE scheme parameters for our implementment:∆ = 2 45 and ∆ c = 2 20 ; slots = 32768 ; logQ = 1200 and logN = 16 to achieve a security levelof 80 − bits.
(see [8, 9] for more details on these parameters).
Result .
We evaluate the performance of our implementation on the MNIST testing dataset of 10000 images.
Since in this case Volley Revolver encoding method can only deal with 32 MNISTimages at one time, we thus partition the 10 000 MNIST testing images into 313 blocks with thelast block being padded zeros to make it full.
We then test the homomorphic CNN inference onthese 313 ciphertexts and ﬁnally obtain a classiﬁcation accuracy of 98 .
The processing of eachciphertext outputs 32 digits with the highest probability of each image, and it takes ∼ 287 seconds ona cloud server with 40 vCPUs.
There is a slight difference in the accuracy between the clear and theencryption, which is due to the fact that the accuracy under the ciphertext is not the same as that underthe plaintext.
In order to save the modulus, a TensorFlow Lite model could be used to reduce theaccuracy in the clear from ﬂoat 32 to ﬂoat 16.
The data owner only uploads 1 ciphertext ( ∼ 19 .
MB)encrypting these 32 images to the public cloud while the model provider has to send 52 ciphertexts( ∼ 1 GB) encrypting the weights of the well-trained model to the public cloud.
Conclusion
The encoding method we proposed in this work, Volley Revolver , is particularly tailored forprivacy-preserving neural networks.
There is a good chance that it can be used to assist the privateneural networks training, in which case for the backpropagation algorithm of the fully-connectedlayer the ﬁrst matrix A is revolved while the second matrix B is settled to be still.
Checklist
For all authors...(
a ) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the
ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[Yes](c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes](d) Did you discuss whether
and how consent was obtained from people whose data you’reusing/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
A Appendix
Optionally include extra information (complete proofs, additional experiments and plots) in theappendix.
This section will often be part of the supplemental material.