PolyKervNets: Activation-free Neural Networks For Efficient Private Inference
Anonymous
Abstract —With the advent of cloud computing, machine learn-
ing as a service (MLaaS) has become a growing phenomenon with the potential to address many real-world problems.
In an untrusted cloud environment, privacy concerns of users is a major impediment to the adoption of MLaaS. To alleviate these privacy issues and preserve data confidentiality, several private inference (PI) protocols have been proposed in recent years based on cryptographic tools like Fully Homomorphic
Encryption (FHE) and Secure Multiparty Computation (MPC).
Deep neural networks (DNN) have been the architecture of choice in most MLaaS deployments.
One of the core challenges in developing PI protocols for DNN inference is the substantial costs involved in implementing non-linear activation layers such as Rectified Linear Unit (ReLU).
This has spawned a search for accurate, but efficient approximations of the ReLU function and neural architectures that operate on a stringent ReLU budget.
While these methods improve efficiency and ensure data confidentiality, they often come at a significant cost to prediction accuracy.
In this work, we propose a DNN architecture based on polynomial kervolution called PolyKervNet (PKN), which completely eliminates the need for non-linear activation and max pooling layers.
PolyKervNets are both FHE and MPCfriendly - they enable FHE-based encrypted inference without any approximations and improve the latency on MPC-based PI protocols without any use of garbled circuits.
We demonstrate that it is possible to redesign standard convolutional neural networks (CNN) architectures such as ResNet-18 and VGG-16 with polynomial kervolution and achieve approximately 30 × improvement in latency of MPC-based PI with minimal loss in accuracy on many image classification tasks.
I. I NTRODUCTION
In today’s cloud computing era, most major cloud service providers (CSP) offer machine learning (ML) as a service (MLaaS) to enable customers gain valuable insights from their data, without the need for investing in resources and expertise to build or run predictive models on their own.
Deep neural networks (DNN) have become the de-facto standard for ML tasks, especially in the areas of computer vision and natural language understanding.
Ever since the introduction of AlexNet [1] in 2012, which achieved state-of-the-art (SOTA) performance on a large-scale image classification task using a DNN model with 60 million parameters and tens of layers, the ML community has developed several deeper and larger DNN models.
While SOTA vision models such as convolutional neural networks (CNN) and vision transformers have millions of parameters and hundreds of layers, SOTA language models such as GPT-3 have billions of parameters.
These humongous models require extensive computational and memory resources, serving as a driving force for outsourcing/offloading ML computations to the cloud.
Thus, the MLaaS model has the potential to solve many real-world problems in a cost-effective way across a range of industries including finance, healthcare, and logistics.
For example, healthcare providers can exploit the MLaaS framework to accurately diagnose diseases based on patient data such as medical images and electronic health records.
The key roadblock in the widespread adoption of MLaaS is the sensitive nature of customer data and the privacy concerns associated with the release of such sensitive information to untrusted CSPs.
Apart from end-user privacy concerns, regulations such as General Data Protection
Regulation (GDPR) also place stringent data confidentiality requirements, thereby hindering the growth of MLaaS.
Cryptographic tools such as homomorphic encryption (HE) and secure multi-party computation (MPC) offer a solution to the problem of privacy-preserving machine learning.
While these confidential computing tools can be applied for both model learning and inference, they come at a tremendous computational and/or communication cost, often rendering the training task unviable.
Therefore, most of the effort in privacy-preserving machine learning has focused on private inference (PI), where the goal is to apply a pre-trained ML model on “encrypted” data and perform inference in the encrypted domain (see Figure 1).
In this context, encryption refers to the fact that the service provider never sees the private data of customers.
Fundamental challenges in the efficient implementation of PI protocols are the large depth of the underlying neural networks and the complexity introduced by the presence of non-linear (activation and pooling) layers in the DNN architecture.
It is well-known that deeper networks have exponentially larger expressive power and better generalization ability compared to shallow architectures [32].
Furthermore, in the absence of non-linear activation layers, the entire DNN can be compressed into a single linear layer, severely degrading the accuracy of the resulting model.
Thus, both large depth and non-linearity are essential to the unparalleled success of
DNN models and compromising on them can cause substantial loss in accuracy of DNN models.
However, adding more layers and non-linearities stymie the possibility of designing efficient PI protocols .
This is because non-linear operations are orders of magnitude more computationally expensive in the case of both HE and MPC.
Solving this accuracy vs. efficiency trade-off has been a subject of active research and the proposed solutions broadly fall into two categories.
One approach is to develop accurate and efficient approximations for non-linear activation functions commonly used in DNNs such as Rectified Linear Unit (ReLU) [2]–[4].
Alternatively, neural architecture search (NAS) algorithms have been used to identify architectures that can work accurately within a constrained non-linearity (e.g. ReLU) budget [5]–[7].
While the first approach typically leads to significant accuracy degradation, the latter method caps the computational complexity, while maintaining accuracy at comparable levels.
In this work, we propose a different strategy to solve the accuracy vs. efficiency conundrum in PI protocols.
Specifically, we leverage the idea of kervolution (kernel convolution) proposed in the computer vision literature to design DNNs that are more conducive to encrypted inference.
Since kervolution operators are based on patch-wise kernel functions (as opposed to point-wise non-linearity in traditional CNNs), they can capture higher order interactions between features without increasing the number of parameters.
Furthermore, kervolution also avoids the need for non-linear activation and max pooling layers - an ideal scenario for private inference.
Therefore, the main contributions of this work are as follows:
• We are the first to identify the synergy between the requirements of private inference protocols and the characteristics of kervolutional neural networks [8], leading to the design of more accurate and efficient DNNs amenable to private inference.
• We modify the well-known CNN architectures (e.g., ResNet-18, VGG-16, etc.) to incorporate polynomial kervolution operators in lieu of traditional convolution, ReLU, and maxpooling layers.
The resulting PolyKervNets (PKNs) lead to a minimal loss in accuracy compared to the baseline CNN architectures, when evaluated on four image classification datasets: CIFAR10, CIFAR-100, MNIST, and Chest X-ray dataset for tuberculosis detection.
However, we show that even this minimal loss in accuracy can be mitigated by designing shallower residual networks with polynomial kervolution.
Finally, we leverage PolyKervNets to implement private inference based on secure multiparty computation (using the Delphi protocol [21]) on VGG-16 and ResNet-18 architectures.
The results show that the proposed approach minimizes latency by approximately 30 × in both cases.
We also show preliminary results for FHE-based private inference (using the TENSEAL library [28]) based on a simple 3-layer CNN architecture, which achieves better accuracy and lower latency compared to commonly used ReLU approximations.
II. R ELATED W ORK A. Private Inference
In a typical private inference pipeline, the client encrypts the raw data using his public key and sends the ciphertexts to the untrusted cloud.
It is assumed that the model owner already has access to a pre-trained ML model.
When the model owner also serves as the CSP, the pre-trained ML model is directly applied to the encrypted data and the encrypted inference results are sent back to the client for decryption using the secret key (as shown in Figure 1).
In a more general case where the model owner is a separate entity (different from the
CSP), the model architecture and encrypted model parameters (encrypted using client’s public key) are sent to the cloud and the inference takes place in the encrypted domain.
There are two possible ways in which private inference can be achieved - fully homomorphic encryption (FHE) and secure MPC.
When minimal communication between the client and CSP is desired, FHE is the only feasible solution.
This approach has been enabled by the development of many
FHE cryptosystems such as BFV [9], [10], BGV [11], CKKS [12], and TFHE [13] over the past decade.
CryptoNets [2] was one of the earliest works to demonstrate that encrypted inference using FHE is possible.
This was followed by other implementations such as Faster CryptoNets [14], Low
Latency CrptoNets [15], Shift-accumulation based leveled HE (SHE) [16], and CKKS with bootstrapping [17].
Apart from
SHE that uses gate-level HE operations based on TFHE, all the other implementations encounter difficulties in dealing with non-linear activation functions such as ReLU and max pooling layers.
The typical work around is to replace ReLU with square activation [2] or other polynomial approximations [3], [4] and utilize sum pooling in the place of max pooling.
As noted in [17], these choices lead to unacceptable trade-off between accuracy and efficiency.
An alternative approach is to rely on secure MPC or a combination of HE and MPC.
Often, a two-party protocol (2PC) between the client and the CSP is used for this purpose.
The key difference between the methods that fall under this umbrella is the manner in which the linear layers in DNNs are computed.
While Gazelle [18] and SAFENet [19] use HE, MiniONN [20], DELPHI [21], CryptoNAS [5], and DeepReduce [7] rely on arithmetic secret sharing.
As with FHE-based methods, the primary bottleneck in the MPC approach is the computation of non-linear layers, which is usually achieved using garbled circuits.
Since garbled circuits are several orders of magnitude more expensive to compute than secret sharing protocols, non-linear computations take up the lion’s share of the overall computational and/or communication load.
This has led to the development of techniques that either perform ReLU dropping after training [7] or design neural network architectures that operate within a given ReLU budget.
Hybrid methods that perform selective
ReLU dropping as well as ReLU approximation have also been proposed [19].
In contrast to the above techniques, our goal is to completely remove all non-linear activation functions and almost all the pooling layers from the DNN architecture.
This is facilitated by the use of kervolution operators, which was first proposed in [8].
To the best of our knowledge, our work is the first attempt to achieve efficient private inference through total elimination of activation and max pooling layers without compromising on accuracy.
B. Kervolutional Neural Networks
Existing CNN models primarily rely on the activation layers to introduce point-wise non-linearity into the ML model.
In [8], it was argued that CNN architectures can be made more expressive if the linear convolution layer is replaced with patch-wise non-linearity using the kernel trick.
This approach was referred to as kernel convolution (or kervolution) and has the potential to improve generalizability and model capacity, while retaining the advantages of simple linear convolution such as speed and number of parameters.
Three kernel types were proposed in [8] for use in the kervolutional layer:
Norm Kervolution, Polynomial Kervolution and Gaussian
Kervolution.
While both polynomial and Gaussian kernels were found to achieve good recognition accuracy in [8],
Gaussian kervolution requires computing an exponential function, which is expensive in the encrypted domain.
Hence, we utilize only polynomial kervolution (PolyKerv) in this work because of its better suitability to PI protocols.
Polynomial Kervolution : The standard convolution operator takes an input x ∈ R n (vectorized version of a 2-dimensional matrix) and filter weights w ∈ R n and outputs:
[Equation]
where b represents the bias.
In contrast, the polynomial kervolution operator applies patch-wise non-linearity and outputs:
[Equation]
where d p ( d p ∈ Z + ) is the degree of polynomial and c p ( c p ∈ R + ) is the balance factor.
Intuitively, the polynomial degree extends the feature space to d p dimensions and the balance factor is able to balance the non-linear terms.
Thus, the polynomial kernel learns linear terms just like the standard convolutional operator, but is also able to output nonlinear terms, which can be considered as an approximation of an activation layer’s output.
In other words, polynomial kervolution subsumes the functions of both the standard convolution operator and the non-linear activation layer, while making the activation learnable and more expressive per use.
Though the idea of kervolution was introduced in [8], it was primarily used as a replacement for the standard convolution operator in CNNs, leaving the rest of the architecture (including activation and pooling layers) intact.
The possibility of removing ReLU and replacing the max pooling layers with average pooling was left as a tantalising teaser in a single experiment, where the LeNet-5 architecture was modified to achieve good performance on the MNIST dataset.
In this work, we push this idea to the extreme and attempt to design DNNs using only polynomial kervolution and completely removing all ReLU layers.
We believe that the use of kervolutional neural networks has not taken off in the broader computer vision community because of its poor cost-benefit trade-off.
Replacing convolution with kervolution and retaining ReLUs increases the computational cost (for plaintext inference), while providing only marginal improvement in accuracy.
Removing the ReLUs affects the stability of the training process and erases even the marginal accuracy gains.
Hence, there is little incentive to use kervolutional neural networks for inference on plaintext data.
However, the trade-offs are completely different in the private inference scenario. For private inference, removing ReLUs provides huge gains in latency and marginal reduction in accuracy becomes acceptable. III. P ROPOSED M ETHOD : P OLY K ERV N ETS (PKN S )
The goal of this work is to design DNNs using polynomial kervolution layers that do not require non-linear activation functions and hence, more conducive for private inference.
The challenge with designing such networks is the difficulty in training (optimizing their parameters) using traditional gradient descent techniques because of vanishing/exploding gradients.
Since residual networks (ResNet)
[22] are known to be easier to optimize, we start with ResNet as the baseline architecture and design PolyKervResNets (PKRs) , which are composed of kervolution-based residual blocks.
A. Input to the Residual Blocks
A typical ResNet architecture consists of four residual blocks.
However, before the input is fed to these residual blocks, it passes through a 7 × 7 convolution layer (with 64 channels and a stride of 2 ), followed by ReLU activation and a 3 × 3 max pooling layer (with stride of 2 ).
In the PKR architecture, we modify the 7 × 7 convolutional layer into a 3 × 3 polynomial kervolution layer and remove the ReLU and max pooling layers which follows it (see Figure 2).
This produces a 64 -channel output that is a non-linear function of the input.
We skip the
ReLU and pooling layers because the kervolution layer already introduces the required non-linearity.
Consequently, the size of the inputs to each residual block in a PKR is twice the size of the input to the corresponding residual block in a vanilla
ResNet architecture.
B. Residual Block Design
Since our eventual target is to design DNNs that enable efficient private inference, we start with the ResNet-18 architecture.
In ResNet-18, each residual block has two identical sub-blocks and each sub-block has the following structure: a 3 × 3 convolution layer, followed by ReLU activation, and another 3 × 3 convolution layer.
The output of this second convolution layer is added to the output of an identity function (input to the block) and this combined output is passed through another ReLU activation.
For the block, the first convolution has a stride of 2 which downsizes the input size by a factor of 2.
The final output of the second sub-block serves as the input to the next residual block.
In the proposed PolyKervResNet- (PKR-18), we eliminate all the ReLU activation layers within a residual sub-block and replace both the 3 × 3 convolution filters with 3 × 3 polynomial kervolution filters.
This is illustrated in Fig 2, where L represents the number of residual blocks.
There is no change in the number of filters/channels in any of the residual blocks.
Note that it is also possible to design the residual sub-block in the PolyKervResNet such that only the first convolution layer is replaced with polynomial kervolution, the second 3 × 3 convolution layer remains unaltered, and the final ReLU function is eliminated.
We refer to this variation as MixedPolyKervResNet (MPKR) .
C. Output of the Residual Blocks
Similar to vanilla ResNet, the output of the final residual block in a PKR is passed through an average pooling layer before it is flattened and presented to a fully connected (FC) layer.
Since the PKR is trained on plaintext data, a softmax function is applied to the output of the FC layer and the network is trained using the standard cross-entropy loss.
During private inference, the encrypted logit values from the final FC layer can be sent back to the client, who can decrypt them to perform inference (assign the sample to the class with the highest logit value).
D. Other CNN Architectures
To further establish the versatility and applicability of PolyKervNets, we replace all the convolution layers in other wellknown CNN architectures such as VGG-16 [23], AlexNet [1] and LeNet-5 [24] with polynomial kervolution layers.
We also remove all the non-linear activation layers in these networks and replace the max pooling layers with average pooling layers.
The resulting architectures are referred to as PolyKervVGG-16 (PKV-16), PolyKervAlexNet (PKA8) and PolyKervLeNet-5 (PKL-5) , respectively.
Originally, all the above CNN architectures have three fully connected (FC) layers at the end, of which the first two FC layers are followed by activation layers and the last FC layer followed by a softmax layer.
However, since our goal is to remove all the activation layers, we use only one FC layer followed by the softmax layer in our PolyKervNets (see Figure 3).
IV. E XPERIMENTAL R ESULTS
We demonstrate the effectiveness of PolyKervNets through experiments on four datasets (CIFAR-10, CIFAR-100, TB
Chest X-ray, and MNIST)
using four standard CNN architectures (ResNet-18, VGG-16, AlexNet, and LeNet-5).
The proposed approach is benchmarked against five baseline methods: ReLU - this is original CNN architecture, DeepReduce - architecture obtained using the ReLU reduction method proposed in [7],
Square - ReLU replaced by square activation [2], PolyApprox-1 - ReLU replaced by quadratic polynomial approximation proposed in [3], and PolyApprox-2 - ReLU replaced by quadratic polynomial approximation proposed in [4].
The evaluation metrics used for benchmarking are accuracy and inference time/latency .
Note that the inference time in PI (especially FHE-based) protocols is highly dependent on the multiplicative depth of the ML model, which represents the maximum number of multiplicative operations (gates) required along any of its circuit paths.
Hence, we also report the multiplicative depth of various models along with the number of parameters in the model, which is coarse proxy for model capacity.
A. Datasets
CIFAR10 and CIFAR100 : CIFAR10 and CIFAR100 [26] are standard natural image classification datasets used in the literature.
CIFAR10 contains 60000 imagesimages for training and 10000 images for inference.
It consists of 10 classes and each class has an average of 5000 train images and 1000 test images.
CIFAR100 also contains the same total number of images, with the only difference being that the 60000 images in CIFAR100 come from 100 classes at an average of 600 images per class.
Both
CIFAR10 and CIFAR100 contain RGB images of size 32 × 32 .
TB Chest X-ray : In a bid to demonstrate the effectiveness of the proposed approach on a more real-world image classification task, we evaluate it on the tuberculosis (TB)
Chest X-ray dataset [27].
This dataset has 4200 X-ray (grayscale) images of size 512 × 512 drawn from two classes.
The normal class contains 3500 images, while the tuberculosis class contains 700 images.
MNIST :
To demonstrate the working of a PolyKervNet for encrypted inference using the CKKS FHE scheme, we employed the much simpler MNIST dataset, which contains 60000 28 × 28 gray-scale images representing the 10 digits.
B. Pre-processing and Training Details
Our primary goal is to benchmark the accuracy and latency of
PolyKervNets relative to the baseline CNN architectures.
We would like to emphasize that the objective is not to achieve state-of-the-art accuracy on specific datasets.
Therefore, we do not focus on searching for the best training settings for each architecture and dataset.
Instead, we use common preprocessing steps, hyperparameter settings, and optimization algorithms across all architectures for a fair comparison.
CIFAR10 and CIFAR100 : The CIFAR10 and CIFAR datasets have predefined train and test splits within the
Pytorch dataset module.
The input images were kept the same ( 32 × 32 ) and a batch size of 128 was used.
We then employed horizontal flips and random rotation as data augmentation strategies for these two datasets.
The whole train and test sets were also normalized before going in as inputs to the architectures.
TB Chest X-ray : Since there were no predefined train-test splits for this dataset, we implemented a custom dataset loader that randomly selects 3700 images for training and retains 500 images as the test set, while maintaining the original class proportions of the dataset.
The images were downsized to 64 × 64 for faster training.
The random crop augmentation was leveraged to extract a 56 × 56 patch from the downsized images and these cropped patches were resized back to a size of 64 × 64 .
A batch size of 32 was set for this task in order to have more training batches, while also reducing training time.
MNIST :
No augmentation was applied for this dataset because the task is very simple.
We chose a batch size offor this dataset.
Training Details : All the DNN architectures were trained using the Adam optimizer with a learning rate of 1e-3 with a scheduler set to decrease the learning rate by 10 everysteps.
For polynomial kervolution, we used polynomial degree d p = 2 to ensure a fair comparison with square activation [2] and the quadratic polynomial ReLU approximations used in [4] and [3].
The balance factor c p was set to 1 in our experiments.
Each network is then trained for 200 epochs on
CIFAR10 and CIFAR100, and 50 epochs on the TB Chest
X-Ray datasets.
All our experiments in the plaintext domain were run on the following computing environment:
• System used: NVIDIA QUADRO RTX•
Operating System: Ubuntu 21.
• GPU memory: 24 GB GDDR
For private inference experiments based on for PolyKervNets (PKNs) as well as other baseline methods, we used the following computing environment:
• System used: Intel Xeon(R)
Silver 4215 CPU • Base Frequency: 2.50 GHz • RAM: 32 GB • PI protocols: TenSEAL CKKS [28], Delphi [21] [Table]
C. Plaintext Evaluation
Accuracy of ResNet-18 models on different datasets : We first evaluate the accuracy of different ResNet-18 models, obtained using the five baseline methods and
the proposed
PolyKervResNet (PKR-18), on the TB Chest X-ray, CIFAR10 and CIFAR-100 datasets.
As emphasized earlier, we did not extensively search for the most optimal training settings for each dataset.
Consequently, the accuracy achieved by the vanilla ResNet-18 model on CIFAR-10 (CIFAR-100) dataset was only 91 .
While these accuracy values are reasonably high, they do not match the state-of-the-art accuracy values reported in the literature.
For instance, there are pre-trained ResNet-18 models available on the Internet that can achieve
[Table] % accuracy on CIFAR- [Table] and
[Table]
top-1 accuracy on CIFAR-100 2 .
However, it is difficult in practice to replicate the exact training settings used to train such models.
To enable fair comparison, it is essential to have a common training setting across different baseline methods and CNN architectures.
Hence, we do not focus on optimizing the accuracy of each individual model and rather base our analysis on the relative accuracy trends.
From Table I, we observe that all models have high test accuracy on the TB X-ray dataset, with PKR-18 having the same accuracy of 99 . 2% as the original (ReLU) and
ReLU-constrained (DeepReduce) models.
On the CIFARand CIFAR-100 datasets, the ReLU and DeepReduce models had almost comparable accuracy, which was marginally higher than the PKR-18 model.
However, the PKR-18 model had much better accuracy compared to square activation and marginally better accuracy compared to the polynomial
ReLU approximations.
This is because deep models with square activation often fail to converge well due to gradient explosion, indicating that square activation is not good enough to replace the ReLU function, especially in the case of complex classification tasks.
This result is consistent with the observations reported in [4] on square activation.
While the other polynomial approximations fare better, they still suffer from the same issues as square activation, albeit to a much lesser extent.
Thus, rigid quadratic polynomial approximations are not effective replacements for the
ReLU activation functions and the proposed PolyKervNets consistently outperform networks designed with such approximations.
The small accuracy degradation suffered by the PKR-18 model (in comparison to ReLU and DeepReduce) is primarily due to accumulation of errors during training, which is typically aggravated by the depth of the network.
One way to mitigate this problem is to reduce the depth of the network, which can be easily achieved in a ResNet architecture by decreasing the number of residual blocks.
The PKR-18 architecture uses the same number of residual blocks as ResNet-18 ( L = 4 ).
The model obtained by removing one residual block (PKR-14) has better accuracy compared to PKR-18 on CIFAR-10 as well as lower multiplicative depth and smaller number of parameters (see Table II).
Further removing another residual block (PKR-10) again leads to performance improvement and resulting accuracy becomes almost equal to that of vanilla
ResNet-18 with all the non-linear layers intact.
Reducing the number of residual blocks to a value less than two did not yield any accuracy improvement, which can be explained by the excessive loss of model capacity.
Thus, for the
CIFAR-10 dataset, L = 2 represents the optimal number of residual blocks in a PolyKervResNet because it offers the best compromise between model capacity (or expressivity) and ease of training.
We also observed that the MPKRarchitecture, where the second convolution layer in each residual block is not replaced with polynomial kervolution, also provides very good accuracy.
Robustness to other CNN Architectures : The accuracy results for the various CNN architectures on the test split of the CIFAR-10 dataset are summarized in Figure 4 and Table
II.
There are three key pointers in these results:
• The PolyKerv versions of VGG-16, AlexNet, and LeNet5 (referred to as PKV-16, PKA, and PKL-5, respectively) have slightly lower accuracy compared to their original (ReLU) counterparts but significantly higher accuracy compared to the rigid polynomial approximations of ReLU.
At a high level, these results indicate the proposed idea of replacing convolution, ReLU, and max-pooling layers with polynomial kervolution and average pooling is generic and can be applied to most base CNN architectures, without much loss in accuracy.
• Figure 4 also shows that irrespective of the base architecture, polynomial kervolution achieves better trade-off between multiplicative depth and accuracy compared to polynomial approximations of ReLU.
This again demonstrates the superiority of polynomial kervolution over ReLU approximations.
• Finally, among the various architectures, ResNet-10 (with two residual blocks) achieves the highest accuracy, even though it has a lower depth compared to ResNet-14, VGG-16, and ResNet-18.
While the poor performance of shallow architectures like LeNet-5 and AlexNet are due to their small model capacity, the marginally lower accuracy of deeper architectures (ResNet-14, VGG-16,
[Table]
[Table] and ResNet-18) is due to the difficulty in training these networks in the absence of ReLU activations.
For the selected dataset, ResNet-10 provides the best compromise between these two phenomena.
Consequently, PKR10 (which uses ResNet-10 architecture with polynomial kervolution) achieves the best accuracy on CIFAR-10, while having low multiplicative depth and fewer number of parameters.
The accuracy results for the various DNN architectures on the test split of the CIFAR-100 dataset are summarized in Table
III.
Due to the larger number of classes, we apply only the
VGG-16 and ResNet-18 architectures on this dataset as the smaller architectures did not have sufficient model capacity to achieve good performance.
Similar to the earlier results, the
ReLU-constrained (DeepReduce [7]) ResNet-18 and VGGmodels had almost the same test accuracy on CIFAR-100 as the corresponding vanilla models (which use ReLU activation without any constraints).
However, these two methods are not amenable to FHE-based private inference because of the presence of ReLU functions.
On the other hand, the FHEfriendly models with square activation [2] and polynomial approximations of ReLU [3], [4] have significantly less accuracy on CIFAR-100.
In contrast, the proposed PolyKervNets (PKR-18 and PKV-16) are FHE-friendly and exhibit little compromise on accuracy, even though PKV-16 uses fewer number of parameters (due to the removal of two FC layers) compared to other VGG-16 models. [Table]
D. Private Inference Evaluation
MPC : We use the Delphi [21] library to implement the MPC-based private inference protocol.
Delphi is in fact a hybrid cryptographic protocol that uses a mix of 2PC and HE primitives for private ML inference.
Delphi uses HE operations to implement the linear (convolution) layers, Beaver’s multiplication triplets [31] for polynomial approximations and square activation, and Garbled Circuits (GC) [30] for non-linear computations such as ReLU.
The original Delphi work [21] also includes neural architecture search (NAS) to identify an optimal architecture, where some
ReLU functions are replaced with polynomial approximations in order to reduce the amount of GC computations of ReLU.
However, in this work the NAS is performed using the
DeepReduce method [7], which has been proven to be more effective than the original Delphi NAS method.
We evaluated the inference time of the proposed PolyKerv method as well as other baselines on the CIFAR-10 and
CIFAR-100 datasets using the VGG-16 and ResNetarchitectures, respectively.
The accuracy and inference time of these models for 2PC-based private inference are summarized in Table IV.
The vanilla version of both
VGG-16 and ResNet-18 implemented using the Delphi protocol consumed about 12 seconds and 17 seconds for secure inference on the CIFAR-10 and CIFAR-100 datasets, respectively.
The DeepReduce method was able to achieve 1 . 9 × faster inference than the original VGG-16 (applied to CIFAR-10) and 1 . 8 × faster inference than the original
ResNet-18 (applied on CIFAR-100) without compromising on accuracy.
We have also tested vanilla and PKN variants of ResNet-18 and VGG-16 on the TB-Chest X-ray dataset.
Similar to the CIFAR datasets, the accuracy of PKN variants is marginally lower than the corresponding vanilla models, but with a significant reduction in latency.
Note that it is possible to further reduce the ReLU budget and obtain even faster inference using the DeepReduce method, but this comes at the cost of significant loss in accuracy.
As reported in [7], among models that required less than a second for inference, the best achieved accuracy using the ResNetarchitecture on CIFAR-100 dataset was around 68% .
This is despite the fact that the original ResNet-18 model achieving an accuracy of 74 .
46% on CIFAR-100.
Due to differences in the training settings, the corresponding accuracy of the original ResNet-18 model in our work on the same dataset is only 72 . 57% .
Hence, we can expect ReLU-constrained
ResNet-18 models with sub-second inference latency trained using our settings to have an accuracy of less than 68% on CIFAR-100.
Among the ReLU approximation methods, the square activation has the best latency improvement (approximately 30 × over the original architecture) for both the experiments (VGG-16 on CIFAR-10 and ResNet-18 on
CIFAR-100).
Unfortunately, square activation also leads to large degradation in accuracy.
The PolyApprox-2 method of [4] achieves a much better accuracy-latency trade-off.
The proposed PolyKervNets clearly outperform all the above approximation methods and achieves both very good accuracy and large improvements in latency.
For instance, complete private inference using the PKV-16 model can be obtained in 0 .
38 seconds, making it almost 32 × faster than the original
VGG-16 architecture, while still achieving greater than 90% accuracy on CIFAR-10.
Similarly, the PKR-18 model requires only 0 .
59 seconds for MPC-based PI on the CIFARdataset, while achieving an accuracy of approximately 71% .
This is at least 3% absolute improvement in top-1 accuracy on
CIFAR-100 over the comparable DeepReduce models having similar inference latency (despite the sub-optimal training settings used in this work, which results in lower accuracy of the original ResNet-18 model).
Thus, the proposed PolyKervNets have better accuracy-latency trade-off than state-of-the-art ReLU reduction/approximation methods.
For our experiments using DeepReDuce, we used a ReLU budget that gives the best performance while still having a fair improvement on latency. For ResNet-18 model on CIFARdataset, the DeepReduce model reported in our paper had a ReLU count of 248 K, an accuracy of 72 . 53% , and a latency of around 9 .
5 s. When the ReLU count is reduced to 98 K, the accuracy degrades to 67 .
84% and the latency reduces to around 3 .
4 s.
In contrast, the PKR-18 model had an accuracy of 70 .
97% (based on the same dataset) and a latency of 0 .
6 s.
Thus, PKR-18 achieves almost 3% higher accuracy than
DeepReduce, while being close to 6 × faster.
Since a more detailed investigation of the accuracy vs. latency trade-off of
DeepReduce has already been reported in [7], we have not repeated this experiment in this work.
The proposed PKNs achieve a latency of around 0 .
5 seconds.
However, it must be noted that the maximum image size used in our evaluations was 64 × 64 (TB Chest X-ray dataset).
For real-world applications, higher resolution images (minimum
[Table]
of 224 × 224 ) need to be processed, which will increase the latency.
But this latency increase can be mitigated through parallelization of computations across multiple GPUs/CPUs.
Thus, the reduced latency of PKNs indeed brings them closer to practical feasibility.
FHE : We use the TenSEAL [28] library for implementation of FHE-based private inference.
TenSEAL is a HE library for performing homomorphic encryption operations on tensors built on top of Microsoft SEAL [29].
The key advantage of this library is the ease of use through a Python API, while preserving efficiency by implementing most of its operations using C++.
However, a major limitation is that it allows the use of only a single convolution layer (which is a consequence of the specific ciphertext packing strategy enabled in this library).
Hence, we were unable to implement any of the standard DNN architectures used in this work using TenSEAL.
Instead, we report only preliminary results for FHE-based inference using a custom CNN-3 architecture recommended by [28] and evaluate it on the MNIST dataset.
The custom CNN-3 architecture is adapted into a PolyKervNet (PKN-3), which uses a 7 × 7 × 4 polynomial kervolution layer with a stride of 3 .
This layer takes in a 28 × 28 ×
MNIST image as input and outputs a tensor of size 8 × 8 × 4 .
This tensor is flattened to produce a 256-dimensional vector before passing through a fully connected layer that outputs the 10-dimensional vector.
The PKN-3 architecture does not involve any activation or pooling layers.
The Adam optimizer with a learning rate of 1e-3 was used for training before encrypting the data to verify its private inference (PI) performance.
Table V shows the performance of the above architectures during encrypted inference.
Though the competing architectures had comparable classification accuracy on the MNIST dataset during the encrypted inference, the custom PKNhas the least latency of 783.4 milliseconds per encrypted inference.
Moreover, the multiplicative depth of the custom PKN-3 is two less than the other architectures with polynomial ReLU approximations.
This experiment again confirms that polynomial approximations of ReLU are good enough for shallow networks designed for simpler tasks and the real benefits of PolyKervNets (PKNs) can be realized only in the case of deeper networks required for more complex real-world tasks.
V. A BLATION S TUDIES A. Impact of c p and d p hyperparameters
While the core experiments focused on demonstrating the ability of PolyKervNets to preserve accuracy and improve latency, there are other factors which impact the performance of the proposed methods.
Firstly, we attempted to study the impact of the balance factor c p on the PolyKervNet (PKN) model performance.
The accuracy of various PKN architectures on the CIFAR10 dataset using different values of c p are shown in Figure 5.
The balance factor c p has a significant impact on the accuracy of each model using a d p of 2.
However, c p = 0 .
5 , c p = 0 .
75 and c p = 1 gave the best accuracy and most stable performance for most architectures.
Setting c p = 0 could work well in smaller models and on simpler datasets, but does not work in the case of deeper models or on complex datasets.
Using c p > 1 could potentially enlarge each layer’s output, and either explode the gradients or increase the loss, which results to a degradation in the mode performance.
Using 0 .
5 ≤ c p ≤ 1 stabilizes the performance depending on the architecture and ensures a good accuracy when using a degree ( d p ) of 2.
Next, we attempted to vary the polynomial degree d p used in
PolyKervNets.
Using a d p value of 3 or higher in PKN architectures exposes model training to the problem of exploding or vanishing gradients.
Consequently, models with d p = 3 failed to learn the tasks well and hence, the results are not reported in this work.
Besides that, using a higher polynomial degree will increase multiplicative depth, thereby increasing latency. Hence, we believe that using a sufficient number of layers with a d p value of 2 is good enough for most tasks. [Table] B. PKRs vs. MPKNs
We performed a comprehensive study of MixedPolyKervNets and the results are summarized in Table VI.
MixedPolyKervNets work well with the ResNet architecture, having marginally lower accuracy and lower latency compared to
PolyKervNets.
However, mixing convolutions and polynomial kervolutions does not work well with the VGG-16 architecture, indicating that residual connections play an important role in stabilizing the training of PolyKervNets.
C. Impact of removing convolution layers
We also tried to reduce the depth of the network by reducing the number of convolution layers within each residual sub-block (from two to one).
We refer to this architecture as PolyKervResNet-Small (PKR-S).
Note that in our earlier experiments, the depth was reduced by decreasing the number of residual blocks instead of modifying the number of convolution layers within each block.
Though the PKR-S architecture has the same multiplicative depth as PKR-10, the performance of the PKR-S model is still considerably lower than that of PKR-10.
This justifies our original design of eliminating residual blocks as a whole to reduce depth, instead of reducing the number of convolutions within each sub-block
[Table] [Table] D. Training using Knowledge Distillation Instead of training from scratch, we tried to perform distillation using a pre-trained ResNet-18 model 3 as the teacher.
Since this pre-trained model had a higher accuracy ( 93 . 6 % on CIFAR-10) compared to our vanilla ResNetmodel trained from scratch ( 91 . 89 % on the same dataset), all the PKN variants distilled from this teacher had better accuracy compared to the corresponding models trained from scratch (see Table VII). For example, the distilled
PKN-18 model had an accuracy of 93 . 41 % compared to the same architecture trained from scratch ( 91 . 1 %).
This shows that distillation from a well-trained teacher model is an effective way of improving the accuracy of PolyKervNets. Unfortunately, we found that distillation had little impact on the sensitivity of the training process to other hyperparameters such as d p .
VI. C ONCLUSION
In this work, we attempted to design deep neural networks that are more suitable for private inference.
Often, the presence of non-linear activation and pooling layers present the greatest stumbling block in achieving efficient private inference.
In contrast to prevailing approaches that approximate non-linear layers such as ReLU with polynomial functions or remove as many non-linear computations as possible through neural architecture search, we propose a completely different approach based on polynomial kervolution.
The proposed approach completely eliminates the need for activation layers and almost all pooling layers and leads to DNNs with low multiplicative depth, but good recognition accuracy.
While PolyKervNets are more amenable to private inference, it is important to note that training PolyKervNets is more unstable than traditional CNN models.
This necessitates a careful selection of hyperparameters.
On the issue of training stability, the key takeaways from this study are:
• While very shallow PolyKervNets are easy to train, the training stability decreases as the network depth increases.
This is reason why PKR-10 provides better accuracy than PKR-14 and PKR-18.
For the same reason, we have not been very successful (thus far) in training even deeper architectures such as ResNet-50.
• PolyKervNets with residual connections are relatively easier to train compared to other architectures such as VGG-16.
Therefore, we recommend ResNet or models with residual connections as the core architecture for designing PolyKervNets.
• Across various architectures, c p = 1 and d p =consistently give good performance.
Experiments on four datasets prove the efficacy of the proposed PolyKervNets based on both FHE and MPC techniques.
Our experiments also show that reducing the multiplicative depth of neural networks is not a straightforward task of removing layers arbitrarily or using coarser approximations.
Instead it is better to closely follow well-tested CNN architectures and modify them carefully to achieve better trade-off between accuracy and efficiency.
We are currently investigating the implementation of the proposed approach using stateof-the-art platforms like Cheetah (for MPC) [33] and HEAAN (for RNS-CKKS FHE) [34].