nGraphHE2: A HighThroughput Framework for Neural Network Inference on Encrypted Data. In previous work, Boemer et al. introduced nGraphHE, an extension to the Intel nGraph deep learning (DL) compiler, that enables data scientists to deploy models with popular frameworks such as TensorFlow and PyTorch with minimal code changes. However, the class of supported models was limited to relatively shallow networks with polynomial activations. Here, we introduce nGraphHE2, which extends nGraphHE to enable privacypreserving inference on standard, pretrained models using their native activation functions and number fields (typically real numbers). The proposed framework leverages the CKKS scheme, whose support for real numbers is friendly to data science, and a clientaided model using a twoparty approach to compute activation functions.  :107,first present CKKSspecific optimizations, enabling a 3x88x runtime speedup for scalar encoding, and doubling the throughput through a novel use of CKKS plaintext packing into complex numbers. Second, we optimize ciphertextplaintext addition and multiplication, yielding 2.6x4.2x runtime speedup. Third, we exploit two graphlevel optimizations: lazy rescaling and depthaware encoding, which allow us to significantly improve performance.  :164,these optimizations enable stateoftheart throughput of 1,998 imagess on the CryptoNets network. Using the clientaided model, we also present homomorphic evaluation of (to our knowledge) the largest network to date, namely, pretrained MobileNetV2 models on the ImageNet dataset, with 60.4percent82.7percent top1top5 accuracy and an amortized runtime of 381 msimage.

SHE: A Fast and Accurate Deep Neural Network for Encrypted Data Homomorphic Encryption (HE) is one of the most promising security solutions to emerging Machine Learning as a Service (MLaaS). LeveledHE (LHE)enabled Convolutional Neural Networks (LHECNNs) are proposed to implement MLaaS to avoid large bootstrapping overhead. However, prior LHECNNs have to pay significant computing overhead but achieve only low inference accuracy, due to their polynomial approximation activations and poolings. Stacking many polynomial approximation activation layers in a network greatly reduces inference accuracy, since the polynomial approximation activation errors lead to a low distortion of the output distribution of the next batch normalization layer. So the polynomial approximation activations and poolings have become the obstacle to a fast and accurate LHECNN model.  :110,this paper, we propose a Shiftaccumulationbased LHEenabled deep neural network (SHE) for fast and accurate inferences on encrypted data. We use the binaryoperationfriendly Leveled Fast Homomorphic Encryption over Torus (LTFHE) encryption scheme to implement ReLU activations and max poolings. We also adopt the logarithmic quantization to accelerate inferences by replacing expensive LTFHE multiplications with cheap LTFHE shifts. We propose a mixed bitwidth accumulator to accelerate accumulations. Since the LTFHE ReLU activations, max poolings, shifts and accumulations have small multiplicative depth overhead, SHE can implement much deeper network architectures with more convolutional and activation layers. Our experimental results show SHE achieves the stateoftheart inference accuracy and reduces the inference latency by 76.21%  94.23% over prior LHECNNs on MNIST and CIFAR10. The source code of SHE is available at :110,this https URL.