CryptoGCN: Fast and Scalable Homomorphically Encrypted Graph Convolutional Network Inference
Abstract
Recently cloud-based graph convolutional network (GCN) has demonstrated great success and potential in many privacy-sensitive applications such as personal healthcare and financial systems.
Despite its high inference accuracy and performance on the cloud, maintaining data privacy in GCN inference, which is of paramount importance to these practical applications, remains largely unexplored.
In this paper, we take an initial attempt towards this and develop CryptoGCN –a homomorphic encryption (HE) based GCN inference framework.
A key to the success of our approach is to reduce the tremendous computational overhead for HE operations, which can be orders of magnitude higher than its counterparts in the plaintext space.
To this end, we develop a solution that can effectively take advantage of the sparsity of matrix operations in GCN inference to significantly reduce the encrypted computational overhead.
Specifically, we propose a novel A djacency M atrixA ware (AMA) data formatting method along with the AMA assisted patterned sparse matrix partitioning, to exploit the complex graph structure and perform efficient matrix-matrix multiplication in HE computation. In this way, the number of HE operations can be significantly reduced.
We also develop a co-optimization framework that can explore the trade-offs among the accuracy, security level, and computational overhead by judicious pruning and polynomial approximation of activation modules in GCNs.
Based on the NTU-XVIEW skeleton joint dataset, i.e., the largest dataset evaluated homomorphically by far as we are aware of, our experimental results demonstrate that CryptoGCN outperforms state-of-the-art solutions in terms of the latency and number of homomorphic operations, i e., achieving as much as a 3.10 × speedup on latency and reduces the total Homomorphic Operation Count (HOC) by 77.4% with a small accuracy loss of 1-1.5 % .
Our code is publicly available at https://github.com/ranran0523/CryptoGCN .
Introduction
Graph Convolutional Neural Networks [22] (GCNs) have recently emerged in machine learning and demonstrated superior performance in various privacy-sensitive applications such as human action recognition [34, 37], financial recommendation system [36], and autonomous driving [24].
While it has been increasingly popular to deploy machine learning services on the cloud, the cloud environment raises critical concerns for GCN-based privacy-sensitive services, since graph data usually contain a considerable amount of sensitive information.
Recently, Homomorphic Encryption (HE) based private-preserving machine learning (PPML) has emerged to be an effective way to protect the privacy of clients.
Using HE, a client can encrypt the data and send it to the cloud.
The cloud server directly operates on the encrypted data and sends the encrypted results back to the client, which can then be decrypted and used.
As the data are encrypted throughout the entire process when it is out of the client’s control, data privacy is greatly enhanced.
The challenge, however, is to deal with the tremendously increased computational cost associated with the HE operations (e.g., rotations, multiplications, additions), which is orders of magnitude higher than that in the plaintext space [12, 30, 19].
There exists a flurry of work that aims to alleviate the computation overhead of HE-based PPML [2, 10, 12, 21, 26]. However, these solutions are all focused on the traditional Convolutional Neural Network (CNN) models and they are ineffective for GCNs because of a major difference of computation pattern between GCN and CNN.
That is, each graph convolution layer in the deep GCN model would involve a unique adjacency matrix multiplication operation to incorporate the graphic information during the convolution.
The additional matrix multiplications in GCN can significantly increase the required HE operations.
As our profiling results in Figure 1a demonstrate, for a typical 64-channelGCN layer with matrix size 25 × 25 , the matrix multiplication can lead to a nearly 49 × increase in terms of homomorphic operation count (HOC) in the worst case.
How to deal with the significantly increased HE operations in the GCN inferencing process presents a primary challenge in our approach.
Furthermore, current HE frameworks, such as the Cheon-Kim-Kim-Song (CKKS) scheme [6] used by this work, are normally built on the so-called Leveled HE (LHE) [1] scheme, which has a limit on the maximum number of concatenated homomorphic multiplication operations.
The increased multiplication operations in GCNs increase the multiplication depths of the total computation circuit, leading to the requirement of larger encryption parameters (e.g. increased polynomial degree and primes) to maintain the same security levels.
As shown in Figure 1c, the choice of parameters for HE schemes not only affects the security level of the encryption but also has profound impacts on the latency of HE operations.
How to judiciously choose the HE parameter to optimize the security, computational cost, and latency is also critical for effective and efficient GCN inferencing.
One intuitive approach to dealing with the adjacency matrix multiplication operation at each graph convolution layer is to employ the state-of-the-art HE matrix multiplication methods (e.g. [7, 18]).
However, these solutions can be problematic for deep GCN models because of increasing the total multiplication depths, e.g. by 3 [7] and 6 [18], which leads to a higher polynomial degree for encryption to maintain at least the same security level.
From the comparison in Figure 1b, with no optimization, these results would surely be inferior to the method in [13], which we used in our paper as the baseline results for comparison.
Instead, we take advantage of the sparsity in the adjacency matrix, which is very common, especially for applications based on a class of popular GCNs–Spatial
Temporal GCN (ST-GCN), and can dramatically reduce the number of HE operations.
In this paper, we have made the following contributions.
First, we develop an approach that can effectively take advantage of the sparsity of matrix operations in GCN inferencing that can significantly reduce the computational overhead.
As shown above, for GCN inferencing, the required matrix-matrix multiplication can lead to significantly increased HE operations.
In the meantime, the matrix operation for GCN inferencing exhibits strong sparsity features, which can be exploited to reduce the computational overhead.
To this end, we develop a novel GCN data formatting method, i.e., A djacency M atrix A ware (AMA) data formatting method to support the associated multi-channel multi-batch convolution and matrix-matrix multiplications, which can exploit the single instruction multiple data (SIMD) structures in HE computation and thus greatly reduce the HE operations.
Second, we also study how to better manage the HE computation numbers and levels for GCN inferencing by judiciously pruning and approximation of the activation module in GCN and settings of HE parameters.
We develop a co-optimization framework that can help to explore the tradeoffs among security level, inference accuracy, and inference latency.
Third, we conduct extensive experiments based on the NTU-XVIEW [33] skeleton joint [35, 14, 3], dataset.
Our experimental results show that the AMA data formatting achieves a latency speedup of up to 3.10 × , and the Activation Prune achieves as much as 2.29 × speedup for latency.
To our best knowledge, this is the first work that builds the HE-based PPML pipeline for GCN-based models with HOC decrease by as much as 77.44% compared to previous benchmarks. [Figure]
Background and Related Work
Spatial-Temporal Graph Convolution Network.
The Spatial-Temporal Graph Convolution Network (ST-GCN) [37] contains two types of convolutions, spatial and temporal convolution, designed to extract spatial and temporal information from the input graph data respectively.
The spatial convolution is described by Equation 1.
[Equation]
in which j indicates different sets of graph connections separated by a partition strategy to better extract the spatial information.
X is the input graph data.
W j ∈ R C in × C out is a set of filter parameters to transform the input tensor from channel C in to channel C out .
˜
D j is the degree matrix.
˜
A j is the adjacency matrix with self-loop.
The XW j is implemented by a 2D-convolution with kernel size 1 × 1 , then multiplied with the normalized adjacency matrix ˜ D
j − 12 ˜
A j ˜ D
j − 12 , and then the resulted data is summed up.
The Temporal Convolution is a convolution with the same graph node data from different time slices.
Thus, it can extract information on different graph nodes from the temporal domain.
CKKS Homomorphic Encryption Scheme.
The Cheon-Kim-Kim-Song (CKKS) scheme [6], which is based on the hardness of ring learning with errors (RLWE) problem, is a leveled homomorphic encryption scheme that allows arithmetic operations on encrypted data over fixed-point numbers.
CKKS provides configurable precision by taking the encryption noise [23] as natural error introduced in the approximation computations and through dropping the least significant bits of computations via the rescaling of ciphertext.
The supported homomorphic operations includes the ciphertext addition Add( ct 1 , ct 2 ) ,
ciphertext multiplication CMult( ct 1 , ct 2 ) , scalar multiplication PMult(ct , pt) , rotation Rot(ct , k) and rescaling Rescale(ct) .
The scalar multiplication is to multiply a ciphertext with plaintext.
The rotation is to apply Galois automorphisms of the cyclotomic extension to the plaintext polynomials in encrypted form resulting in a cyclic shift of the slot vector.
For instance, Rot ( ct, k ) transforms an encryption of ( v 0 , ..., v N/ 2 − 1 ) into an encryption of ( v k , ..., v N/ 2 − 1 , v 0 , ..., v k − 1 ) .
Compared with unencrypted computations, CKKS introduces a significant runtime and memory overhead.
The use of packing, also referred to as batching, allows to pack of multiple data values into one ciphertext so the encrypted computations can be done in a Single Instruction Multiple Data (SIMD) manner.
We use this property to improve the amortized overhead in this paper.
The security level [4] of the CKKS encryption scheme is measured in bits.
With λ = 128 , it will take around 2 128 operations to break the encryption.
Throughout this paper, we assume that N is the cyclotomic polynomial degree.
The CKKS is a leveled HE.
The level of a ciphertext ( l ) is defined as the number of successive multiplications that can be performed to the ciphertext without bootstrapping.
The level is decreased by one through the rescaling operation after each homomorphic multiplication.
If the level becomes zero, bootstrapping is needed to make this zero-level ciphertext a higher-level ciphertext to enable further homomorphic operations.
In our work, we optimize the GCN network to have lower depth and select proper parameters to avoid the costly bootstrapping procedure.
Related Work.
CryptoNets [12] is the first work that demonstrates the feasibility of building PPML by HE.
CryproNets evaluates a 5-layer NN with MNIST dataset and can perform inference on 59000 pictures within one hour.
However, the long inference latency makes it hard to be applied to large-scale models and datasets.
A more recent work, SHE [26], preserves and translates ReLU and Max Pooling operations by Boolean operations and implemented by TFHE [8], and achieved state-ofthe-art inference accuracy.
However, SHE also has high inference latency issues, where making a prediction on a CIFAR-10 size image requires 2258 seconds.
Many MPC [31, 20, 28, 15, 25, 29] combined two-party computation protocols [38] with HE framework offer low inference latency.
However, they suffer from high communication overhead in terms of data transfer.
For example, DeepSecure [31] needs to exchange 722GB of data between the client and the server for only a 5-layer CNN inference on one MNIST image.
Another approach [9] employs the sparsity location vector to avoid unnecessary computations when performing sparse matrix multiplication.
To this end, they developed a PIR protocol to extract the required non-zero elements from the ciphertexts in their multi-party computation (MPC) setting, which is not applicable in our HE environment at all since decryption at the inference stage is not possible.
The studies by LoLa [2], CHET [10], and HEAR [21] target to use the natural ciphertext packing technique to place multiple values from network nodes in the same ciphertext to perform HE operation in a single instruction multiple data (SIMD) way.
However, in their frameworks, the data representation in the ciphertext is not optimized for GCN-based models so a large amount of HE operations (multiplication and addition) are needed.
Methodology
In this section, we first present the threat model assumption for this work.
We then discuss technique details of our proposed CryptoGCN –a CKKS-based homomorphic encryption framework tailored for fast and scalable GCN inference on encrypted graph tensor.
In particular, we focus on two key components to significantly reduce the number of HE operations: 1) adjacency matrix-aware (AMA) data formatting dedicated to simplifying GCN’s matrix-matrix multiplications without involving ciphertext rotation; 2) non-linear activation pruning to reduce the multiplicative depth, resulting in trade-offs between security level and inference latency with low accuracy loss.
Threat Model.
We assume the cloud-based machine learning service, of which a well-trained graph convolutional network model with plaintext weights, is hosted in a cloud server.
A client can upload private and sensitive data to the cloud for obtaining the online inference service.
The cloud server is semi-honest (e.g. honest but curious).
To ensure the confidentiality of clients’ data against such a cloud server, a client will encrypt the data by HE and send it to the cloud for performing encrypted inference without decrypting the data or accessing the client’s private key.
The client then can decrypt the returned encrypted inference outcome from the cloud using the private key.
In this work, we focus on encrypting graph node features, and the normalized adjacency matrix (often sparse and the same for different graph inputs) is assumed as plaintext.
3.1 AMA Data Formatting and Matrix-Matrix Multiplication
Since HE operations can be performed on encrypted vectors by taking advantage of the SIMD architecture for parallel computing, the input tensor should be well placed into a “big vector container" by a certain format which we call “data formatting".
The state-of-the-art row-major format [10, 21] concatenates a ciphertext row by row and facilitates the dot-product computation that is essential to convolution or matrix-based operations.
However, as we shall show later, it cannot efficiently support
GCNs’ matrix-matrix multiplication that would occur multiple times in a multi-channel GCN layer, because of the extensive ciphertext rotation, addition, and multiplication.
The mini-batch inference over multiple GCN layers would further escalate HE overhead and
Adjacency Matrix-Aware (AMA) Data Formatting.
We use the skeleton-based action recognition graph tensor data as an example to better illustrate our proposed AMA data formatting: assuming the size of an input graph tensor is B × C × T × J , where B is the batch size of mini-batch inference (e.g. B = 1 for a single input inference), C , T and J are the number of input channels, video frames, and joints, respectively.
The first step is to permute the tensor as J columns, with each column of size C × B × T .
Then each column is flattened to C 1D vectors, with each vector of size B × T .
For each of such 1D vectors, zeros are padded to make its length equal to the smallest power-of-two integer greater than B × T .
The C zero-padded 1D vectors are concatenated and then further stacked to a plaintext vector via a channel-interleaving manner until the space of a plaintext vector can be fully exploited, e.g. the length reaches half of the polynomial degree N .
Finally, we encrypt such a plaintext as a fully-packed ciphertext ct k , k ∈ J .
The detailed process is presented in Algorithmand Fig. 2(a).
AMA Data Format-Aided Matrix-Matrix Multiplication.
Once the AMA data formatted ciphertext is created, the next step would be to apply it to simplify and speed up the matrix-matrix multiplication introduced by adjacency matrix and convolution operations.
Recall Eq.
1 in Section 2, a typical ST-GCN layer’s computation involves three consecutive PMult(ct , pt) : spatial convolution ( 1 × 1 plaintext kernel), matrix-matrix multiplication with normalized adjacency matrix J × J in plaintext, and the temporal convolution along the dimension T ( K × 1 plaintext kernel).
Considering the property of 1 × 1 spatial convolutional kernel, we can easily merge this into the adjacency matrix and formulate a new plaintext matrix to reduce one multiplicative depth of PMult .
Then we can perform the matrix-matrix multiplication based on the new J × J plaintext matrix A .
Note that each graph tensor input channel could contain one such matrix.
For row-major formatted ciphertext, as Fig. 2(c) shows (bottom), even with state-of-the-art diagonal encoding method [13], such matrix-matrix multiplication would still involve 2 J − 1 ciphertext rotations ( Rot ).
Since the final outcome is the sum of each rotated ciphertext ( Rot(ct,k) ) multiplied by the corresponding diagonal encoded vector ( Pt k ) from the plaintext matrix A , it also brings extra
PMult and Add .
In contrast to the row-major data format, our AMA data format can significantly reduce the amount of these HE operations.
As Fig. 2(c) (top) demonstrates, first, we decompose the J × J plaintext matrix A into a series of patterned sparse matrices
A i –each A i contains at most one valid element in each column, and A = (cid:80) mi =1 A i , m ≤ J .
Second, we simply multiply the column-wise fully-packed ciphertext (due to the AMA data formatting) with the valid element of the corresponding column in A i , and then sum the m intermediate column-wise ciphertext to obtain a final ciphertext:
[Equation]
Where a i k k represents the single valid element in column k of A i .
Since the process does not require any Rot , except the simple column-wise PMult with a single plaintext value and final summation, the
HE operations can be greatly decreased compared with the row-major format.
To be specific, as the example in Fig. 2(c) shows, the sparse adjacency 4 × 4 matrix A in a GCN consists of 8 non-zero elements ( a 11 , a 13 , a 14 , a 23 , a 32 , a 41 , a 42 , a 44 ) and 8 zeros, while the dense encrypted feature matrix that needs to be multiplied with A has numbers 1 to 16.
For AMA format, as shown in the top part of Fig. 2(c), due to the sparsity of A , A can be easily decomposed into two submatrices A 1 + Awhose column only contains a single non-zero value (e.g. a 11 in column 1 of A 1 , a 41 in column 1 of A 2 ). Then the output ciphertext can be quickly calculated by simply multiplying a constant value in a column of A
i
( i = 1 , 2) with the corresponding column-wise packed ciphertext in the dense feature matrix, and then sum such column-wise multiplied results from A 1 and A 2 .
In this way, the rotation is eliminated.
The sparsity of A determines how many submatrices need to be decomposed, and how many PMults are needed.
The sparser (e.g. a 11 , a 13 become zero)
A is, then the less numbers of
PMult ( ct i k , a i k k ) will be.
Thus our AMA format takes advantage of A ′ s sparsity in practical GCN applications to reduce HOCs.
In contrast, the row-major format presented in the bottom part of Fig.
2(c), cannot utilize this sparsity for computation overhead reduction due to using a diagonal encoding method to form multiple plaintexts to be multiplied with a corresponding rotated ciphertext.
Theoretical Analysis of HE Operation Reduction.
We analytically compare the number of HE operations needed for matrix-matrix multiplication between our AMA and row-major data formats.
To ensure the evaluation generality, our analysis is conducted by assuming a mini-batch inference of multi-channel graph tensor input ( B × C × T × J ) on a typical ST-GCN layer which consists of C input channels and C output channels with 1 × 1 convolution kernels.
The J × J matrix A in
Eq. 2 can be decomposed into J sub-matrices if A is dense in the worst case.
For a fair comparison, we assume the space of a ciphertext– T × J is fully exploited in both methods.
As shown in Fig.(a) and (b), a ciphertext in row-major and AMA format contains one channel data of size T × J , and multiple-channel multiple-batch data of size B × T × c
( c = J/B, c ∈ N + ), respectively.
This results in the same total amount of ciphertext– B × C in both methods.
The implementations of an T J B×C (a) Input data 1 2 3 4 5 6 79 10 11 12 13 14 15(b) ... ... ... Batch B
Ch1 Batch
ChC
Row 2 Ch1 Row Row 3 Row... ... .. Row ChC Row
Row 3Row15 16 1 2 3 4 5 6 7 8 9 10 11 12 1314 15 16 1 2 3 4 5 6 7 8 9 10 11 123 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2= + = + 1 2 3 4 5 6 79 10 11 12 13 14 15 16 ...
(c) Multiply in AMA format Multiply in Row-major format ... Input Data (ct) example matrix-matrix multiplication using the two methods are presented in Fig.
2 (c).
Since AMA formatting increases processing parallelism in SIMD by packing multiple-channel and multi-batch data into one ciphertext, it involves the rotation of the outcomes of matrix-matrix multiplications from different channels and then summing up them to obtain the result for an output channel (Fig.(b)).
A ciphertext including data from more channels (a larger C ) requires more rotations.
To further reduce the rotation overhead, we leverage the multi-channel convolutional technique and baby-step strategy from HEAR [21] 1 .
We also apply them to row-major formatting in evaluation 5.
However, the improvement is limited since a row-major formatted ciphertext can only contain single-channel data.
[Table] [Table] Table 1 reports the detailed breakdown of HE operation numbers for each method (see Appendix A. for detailed proof).
Overall, AMA data format requires almost half of PMult and Add operations of that of row-major format, since row-major format needs to compute 2 J − 1 versions of a ciphertext due to rotations in the matrix-matrix multiplication.
The number of rotations increases proportionally as the number of channels C that can be included in a ciphertext increase for both methods.
However, it is much less than that of row-major.
The rotation number difference between AMA and row-major format can be further enlarged when only the batch size B increases.
This is because the AMA formatted ciphertext contains data from more batches, resulting in the reduction of rotation times ( JC − BC ).
In contrast, the rotation amount of the row-major format grows as B increases.
This means that our AMA data formatting performs better when the batch size increases.
Moreover, since
A is often a sparse matrix in practical ST-GCNs, the number of decomposed matrices m can be much smaller than J for matrix-matrix multiplication, indicating better efficiency than Table 1 which is obtained from a dense matrix.
We further validate these observations in Section 5.
1: Input: X k = ( x 1 , x 2 , . . .
, x C ) ∈ R C
× B × T 2: Output: ct k
3.2 Activation Pruning
Applying the Leveled Homomorphic Encryption scheme like CKKS for private inference of a deep network is challenging because of the limited modulus bits for the encrypted ciphertext.
This means that one LHE encrypted ciphertext has a limited rescale level.
For computation like batch normalization, it could be easily absorbed in the adjacent linear computation like convolution and matrix multiplication [12] since batch normalization [16] is a fixed-parameter based linear transformation in inference.
However, the nonlinear ReLU activation computation must be replaced by polynomial approximate activation (PAA).
However, even a simple degree-2 PAA would consume 2 rescale levels.
Given that each GCN layer can contain one or more ReLU activation for accuracy purposes, more rescale levels would be needed for deep GCNs.
This increases the polynomial degree to achieve a certain security level, leading to higher computation overhead and inference latency.
To achieve the trade-off between inference latency, security level, and accuracy, we investigated the state-of-the-art solution–fine-grained channel-wise PAA for ReLU in SAFENet [27], which is to replace the ReLU activation in the same layer with different polynomial degrees.
However, the method has a major limitation when applied to LHE-based private inference.
Given a ciphertext could contain data from multiple channels in our AMA data formatting, a mask vector that chooses different PAAs must be utilized to process a ciphertext.
As a result, this would consume an extra rescale level and offset the benefit of reduction from the multiplicative level in ReLU.
Therefore, we propose to identify and prune ReLU activations based on the fact that removing ReLU activation of some layers in deep networks leads to marginal accuracy loss [17]. For our Leveled HE schemes
CKKS, computation overhead from activation is not linearly changed with activation counts different from some existing approaches like CryptoNAS [11] and Delphi [28].
Thus, our target is not to prune as many activations as possible.
Instead, we need to take the total multiplication depth, model accuracy, latency, and security level all into consideration in our optimization process.
We name such a technique Activation Pruning (AP).
As Algorithm 2 shows, we first replace all the ReLU activations with a 2-degree polynomial activation ax 2 + bx + c , in which ( a, b, c ) is updated during the training process, and train this model’s accuracy to a desired level as the baseline.
Second, we rank the activation layers according to the accuracy, search the architecture by increasing the number of pruned activation layers and fine-tune the new model to recover the accuracy.
A model can be selected based on the accuracy and number of needed rescale levels to trade off between security level and latency.
Experiment Methodology
HE parameter setting.
We have two sets of encryption parameters for the experiments without
Activation Prune and with Activation Prune.
For both settings, we choose the scaling factor ∆ = 2to maintain the inference accuracy.
For each level, it will consume 33 bits of ciphertext modulus Q .
For experiments without AP setting, it requires 21 levels for the whole network architecture.
Thus, we set Q = 740 , and the polynomial degree N should be set to 2 15 to make the security level ≥ 80 bits.
For experiments with the AP setting, the total level is 19 or 17.
Therefore, we set Q = 680 orand the polynomial degree N = 2 14 to achieve at least 80-bit security level.
Dataset.
NTU-RGB+D [33] is the current largest dataset with 3D joint annotations for human action recognition tasks.
It contains 56,880 action clips in 60 action classes.
The annotations contain the 3D information ( X, Y, Z ) of 25 joints for each subject in the skeleton sequences.
We choose one benchmark NTU-cross-View (NTU-XView) as the dataset for our evaluation because this benchmark is a representative human skeleton joints dataset.
It contains 37,920 and 18,960 clips for training and evaluation, respectively.
For better evaluation, we use 256 frames from the video clip as our input data.
Thus, the input tensor size 2 × 3 × 256 × 25 contain the 25 skeleton-joint information ( X, Y, Z ) for 2 person in a video has 256 frames
Network Architecture.
ST-GCN [37] is the state-of-the-art GCN architecture in human action recognition tasks, which combines the GCN and CNN to better extract the spatial and temporal information than the previous models.
In our experiments, we use a stack of 3 ST-GCN layers with a global average pooling layer and a fully-connected layer and study one small net and one large net: 64-STGCN-3 and 128-STGCN-3, where the first number stands for the channel number of the first
ST-GCN layer.
Table 2 summarizes the network architecture.
One ST-GCN layer is composed of one Spatial Conv layer and one Temporal Conv layer.
Following the stacked 3 ST-GCN layers are a global average pooling layer and a fully-connected layer. We use Stochastic Gradient Descent (SGD) [Table]
[Table] [Table] optimizer with a mini-batch size of 64, a momentum of 0.9, and a weight decay of e − 4 to train the model for 200 epochs.
The initial learning rate was set to 0.01 with a decay of 0.1.
Evaluation Setup.
Our experiments are conducted on a machine with AMD Ryzen Threadripper PRO 3975WX with a single thread setting.
We are using Microsoft SEAL version 3.7.2 [32] to implement a RNS-variant of CKKS [5] scheme.
To perform the temporal convolution, we leverage the baby-step strategy [21] to do multi-channel convolution.
The Global Avg Pooling layer and fully-connected layer have a small impact on total latency so we just apply a straightforward implementation to compute these two layers.
Results and Analysis 5.1 Activation Prune Ablation Study
We first analyze the effect of our Activation Prune (AP) technique in our framework through an ablation study.
To evaluate the performance, we optimize our neural network architecture by pruning the activation layers.
After the algorithm optimization, we have two types of variants, i.e., 1 AP and 2 AP, where the numbers denote how many activation layers have been pruned.
Then, we compare the performance of these optimized architectures on the model inference accuracy and HE inference latency.
The results are shown in Table 3.
Despite the architectures having different widths (channel numbers), the original unoptimized architectures (w/o AP) have the best accuracy but also the highest latency.
To deal with the multiplicative depth of the unoptimized architectures, we have to set the polynomial degree in the encryption parameter to 2 15 , resulting in a tremendous latency increase.
By applying AP, the 1 AP (one activation layer pruned) architecture only has a small accuracy loss (1.1-1.5%), while the latency has been greatly improved by about 2.3 × due to two levels has been saved.
This is because these saved two levels allow us to trade off the polynomial degree with a security level decrease and achieve a sweet point.
We can use 2 14 as our polynomial degree for the optimized depths with an 80-bit (previous 128-bit) security level.
If we try to prune one more activation layer (2 AP), the accuracy loss (3.95-4.04%) is larger than the 1 AP variant.
Besides, the latency speedup is limited because we can not reduce the polynomial degree further to have at least an 80-bit security level.
5.2 AMA Format Effectiveness
To evaluate the effectiveness of the AMA format method, we compare its performance against the row-major formatting with the same encryption parameter setting.
As the AP improves the latency for every HE operation, we evaluate the performance of two data formatting methods on AP-optimized architecture.
Then, we compare our AMA format with the row-major format in different batch size settings. [Table] [Table]
5.2.1 Compare with row-major format
As described in Table [Table] our AMA format improves the inference latency by 1.59 × for 64-STGCNarchitectures and 1.97 × for 128-STGCN-3 architecture.
Table 5 is a breakdown of HE operations for 64-STGCN-3.
From the table, we see that the AMA format has only 31 .
3% of PMult and Add, compared with the row-major format.
The theoretical complexity is presented in Appendix A.2.
Here are two reasons for the reduction of operations.
First, the AMA format uses fewer ciphertexts than the row-major format.
The row-major format does not fully utilize the ciphertext space as the feature map has a size of 256 × 25 .
In row-major format, the feature map is first converted into a 1D vector with a size of 6400; then zero padding is applied to the right end of this vector to make the size equal to 8192.
Therefore, in the resulting encrypted ciphertext, 1792 slots have been wasted.
Our AMA format, fully utilizes the ciphertext space, because 256 is a power-of-two number.
None of the slots in the ciphertext is wasted.
Second, compared to row-major formatting, our AMA formatting ciphertext could perform matrixmatrix multiplication with fewer HE operations.
For the matrix used in our proposed network, one row-major format ciphertext should perform 19 × multiplications for one output channel.
However, one AMA formatting ciphertext only needs to perform 3 × multiplications for one output channel.
A similar reason holds for the Add operation.
5.2.2 Different batch size settings
We analyze our data formatting method in different batch size settings.
As described in Table 6, with the batch size increasing, the latency of the row-major format increases linearly as they do not have any parallelism for a mini-batch setting.
However, our AMA format allows processing a mini-batch of data on the same ciphertext, which allows the parallelism for a mini-batch setting.
Besides, the number of rotations for multi-channel convolution decreases when the batch size increases and the number of other HE operations are linearly increasing, resulting in a higher speedup up to 3.1 × for average latency with the batch size increasing.
5.3 Computation Complexity Evaluation
Table 7 compares CryptoGCN against the state-of-the-art privacy-preserving neural network frameworks (i.e., CHET [10] and Fast-HEAR [21]).
The previous frameworks and ours were implemented in different environments (different CPUs and number of threads).
For a fair comparison, we evaluate the numbers of the required homomorphic operations for 64-STGCN-3 on the same dataset, which are independent of the hardware and software configurations.
Similar to our work, CHET and Fast-HEAR utilize the ciphertext packing technique to reduce HE computation complexity.
They used the row-major format as the data representation for the feature maps.
The main difference between Fast-HEAR and CHET is that Fast-HEAR leverage the non-valid [Table] [Table] space in ciphertext after down-sampling (avg pooling layer) in the CNN model such that Fast-HEAR could have less Homomorphic operation count (HOC).
Neither CHET nor Fast-HEAR is optimized for GCN-based models and the unique matrix multiplication mechanism could significantly increase the HOC.
When batch size equals 1, compared to
CHET and Fast-HEAR, our AMA formatted ciphertext better utilizes the sparsity of the matrix and significantly reduces the multiply and addition operations.
Specifically, when performing matrix multiplication combined with a multi-channel convolution, the AMA format avoids performing an inner loop for matrix multiplication and hence reduces the amount of PMult and Add operations by 52.5-66.2%.
Furthermore, we pack the graph data into ciphertexts to maximize the use of the slots and prune one activation layer from the original architecture, which reduces the number of CMult by 40-50%.
With the batch size increasing, CryptoGCN could reduce 77.4% of total HOC compared to prior work.
The detailed computation complexity formula is in Appendix A.2.
Conclusion
Homomorphic encryption has become an effective way to build private-preserving machine learning thanks to the great development in HE scheme research.
In this paper, we build a fast and scalable LHE-based private preserving inference framework optimized for GCN models by our novel AMA data formatting and model architecture optimization strategy.
To the best of our knowledge, this is the first framework supporting private inference for a large skeleton joint data with a 40 × size of previous work Fast-HEAR and a 3 × deep neural network model.
Our solution shows highly promising results for enhancing privacy-preserving inference on GCN models in a cost-effective manner.
In the future, we could leverage the multi-threading technique to further reduce the latency.
We would like to extend the encrypted data to both graph node features and adjacency matrices, as these matrices may also contain a part of sensitive information.
By encrypting graph node features and adjacency matrices, the client’s privacy could be fully guaranteed  Acknowledgement
The authors would like to thank the anonymous reviewers for their suggestions for improving this paper.
This work was partially supported by the National Science Foundation (NSF) under Award
CCF- and Award CCF-2006748.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We clearly present the theoretical analysis and compare our contribution by ablation study in the result part (b) Did you describe the limitations of your work?
[Yes] See section(c)
Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See section 3. (b) Did you include complete proofs of all theoretical results?
[Yes] See section A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See in section (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See in section(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We report the average latency with same random seed setting (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See in section4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] Yes, we cite all the creators in the papar (b) Did you mention the license of the assets?
[Yes] SEAL has MIT license.
ST-GCN repo has BSD-2-Clause license.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Appendix A.1 Theoretical Complexity Comparison in Matrix-Matrix Multiplication
We continue to explain the theoretical results presented in Table 1.
The assumption has been [-illustratedin 3.
-]illustrated in 3.
Row-major format:
For one ciphertext, it represent the data from one channel.
In order to multiply with A dense matrix
J × J .
We have to rotate each ciphertext with 2 J − 2 times.
Total Rotation =
B × C × (2 J − 2)
As we need to get C output channel data, all the existing BC ciphertext need to do 2 J − 2 times multiplications.
Total PMult = C × B
× C × ( 2 J −
Then, we need to get same BC ciphertext as the output channel is also C .
We just sum up all the ciphertext.
Total Add = C
× B × C × ( 2 J − 2) − B × C
AMA format:
The rotation operation is only used to sum up all the input channel data, because the ciphertext with
AMA format contains J/B channels data.
For each ciphertext, it need to rotate (J/B-1) times.
Total Rotation =
B × C × ( J/B − 1)
To get C output channels data, each ciphertext need to perform J times PMult
Total PMult = B × C × J × C
Then, we need to get same B × C ciphertext as the output channel is also C .
We just sum up all the ciphertext.
Total PMult = B × C × J × C − B × C
A.2 Compact complexity across Layers and other methods
We use the following notations: Number of samples N , Data size N · C · T · J , Batch size B ,
Channels on one ciphertext in AMA U = R / 2 · N · T · B , Number of ciphertexts in AMA N a = J · C / U ,
Number of ciphertexts in row-major format N r = C · N · B , Output channel O , polynomial degree
R , Spatial-conv layers S p , Temporal-conv layers T e , Activation layers A , Kernel size K , Valid matrix elements number V , Diagonal decomposition number D , Number of output classes C s .
The complexity comparison across layers between AMA format and row-major format is in Table 8.
The total complexity comparison with other SOTA methods is in Table 9.
[Table]
[Table]
[Table]
a · O ·
[Table]
− N a ) ·
[Table]
[Table]
[Table]
[Table]
/U ·
[Table]
J
[Table]
s
[Table]
[Table]
[Table]
[Table]
[Table]