L EARNING TO L INEARIZE D EEP N EURAL N ETWORKS FOR S ECURE AND E FFICIENT P RIVATE I NFERENCE
Anonymous authors
Paper under double-blind review
A BSTRACT
The large number of ReLU non-linearity operations in existing deep neural networks makes them ill-suited for latency-efficient private inference (PI).
Existing techniques to reduce ReLU operations often involve manual effort and sacrifice significant accuracy.
In this paper, we first present a novel measure of non-linearity layers’ ReLU sensitivity, enabling mitigation of the time-consuming manual efforts in identifying the same.
Based on this sensitivity, we then present
SENet, a three-stage training method that for a given ReLU budget, automatically assigns per-layer ReLU counts, decides the ReLU locations for each layer’s activation map, and trains a model with significantly fewer ReLUs to potentially yield latency and communication efficient PI.
Experimental evaluations with multiple models on various datasets show SENet’s superior performance both in terms of reduced ReLUs and improved classification accuracy compared to existing alternatives.
In particular, SENet can yield models that require up to ∼ 2 × fewer ReLUs while yielding similar accuracy.
For a similar ReLU budget SENet can yield models with ∼ 2 .
32% improved classification accuracy, evaluated on CIFAR-100.
I NTRODUCTION
With the recent proliferation of several AI-driven client-server applications including image analysis Litjens et al.
object detection, speech recognition Hinton et al.
(2012), and voice assistance services, the demand for machine learning inference as a service (MLaaS) has grown.
Simultaneously, the emergence of privacy concerns from both the users and model developers has made private inference (PI) an important aspect of MLaaS. In PI the service provider retains the proprietary models in the cloud where the inference is performed on the client’s encrypted data ( ciphertexts ), thus preserving both model Kundu et al.
(2021b)
and data-privacy Yin et al.
Existing PI methods rely on various cryptographic protocols, including homomorphic encryption (HE) Brakerski & Vaikuntanathan (2014); Gentry (2009) and additive secret sharing (ASS)
Goldreich et al.
for the linear operations in the convolutional and fully connected (FC) layers.
For example, popular methods like Gazelle
Juvekar et al.
(2018), DELPHIMishra et al.
(2020), and
Cheetah Reagen et al.
(2021) use HE while MiniONN Liu et al.
(2017) and CryptoNAS Ghodsi et al.
(2020) use SS.
For performing the non-linear ReLU operations, the PI methods generally use Yao’s
Garbled Circuits (GC) Yao (1986).
However, GCs demands orders of magnitude higher latency and communication than the PI of linear operations, making latency efficient PI an exceedingly difficult task.
In contrast, standard inference latency is dominated by the linear operations Kundu et al.
(2022b) and is significantly lower than that of PI.
This has motivated the unique problem of reducing the number of ReLU non-linearity operations to reduce the communication and latency overhead of PI.
In particular, recent literature has leveraged neural architecture search (NAS) to optimize both the number and placement of ReLUs Ghodsi et al.
Cho et al.
However, these methods often cost significant accuracy drop, particularly [Table] when the ReLU budget is low.
For example, with a ReLU budget of 86k, CryptoNAS costs ∼ 9% accuracy compared to the model with all ReLUs (AR) present.
DeepReDuce Jha et al.
(2021) used a careful multi-stage optimization and provided reduced accuracy drop of ∼ 3% at similar ReLU budgets.
However, DeepReDuce heavily relies on manual effort for precise removal of ReLU layers, making this strategy exceedingly difficult, particularly, for models with many layers.
A portion of these accuracy drops can be attributed to the fact that these approaches are constrained to remove ReLUs at a higher granularity of layers and channels rather than at the pixel level.
Only very recently, a contemporary work Cho et al.
(2022) proposed l 1 -regularized pixel level ReLU reduction.
However, such approaches are extremely hyperparameter sensitive and often does not guarantee meeting a specific ReLU budget.
Moreover, the large number of training iterations required for improved accuracy may not be suitable for compute limited servers Mishra et al.
Our contributions.
Our contribution is three-fold.
We first empirically demonstrate the relation between a layer’s sensitivity towards pruning and its associated ReLU sensitivity.
Based on our observations, we introduce an automated layer-wise ReLU sensitivity evaluation strategy and propose SENet, a three stage training process to yield secure and efficient networks for PI that guarantees meeting a target ReLU budget without any hyperparameter-dependent iterative training.
In particular, for a given global ReLU budget we first determine a sensitivity-driven layer-wise non-linearity (ReLU) unit budget.
Given this budget, we then present a layer-wise ReLU allocation mask search.
For each layer, we evaluate a binary mask tensor with the size of corresponding activation map for which a 1 or 0 signifies the presence or absence of a ReLU unit, respectively.
Finally, we use the trained mask to create a partial ReLU (PR) model with ReLU present only at fixed parts of the non-linearity layers, and fine-tune it via distillation from an iso-architecture trained AR model.
Importantly, we support ReLU masks allocation both at the granularity of individual pixels and activation channels.
To further reduce both linear and non-linear (ReLU) layer compute cost, we extend our approach to SENet++.
SENet++ uses a single training loop to train a model of different channel dropout rates (DRs)
d r ( d r ≤ 1 . 0 ) of the weight tensor, where each d r yields a sub-model with a MAC-ReLU budget smaller than or same as that of the original one.
In particular, inspired by the idea of ordered dropout Horvath et al.
(2021), we train a PR model with multiple dropout rates Horvath et al.
(2021), where each dropout rate corresponds to a scaled channel sub model having number of channels per layer
∝ the d r .
This essentially allows the server to yield multiple sub models for different compute requirements that too via a single training loop, without costly memory footprint.
Table 1 compares the important characteristics of our methods with existing alternatives.
We conduct extensive experiments and ablations on various models including variants of ResNet,
Wide Residual Networks, and VGG on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet datasets.
Experimental results show that, SENet can yield SOTA accuracy-ReLU trade-off with an improved accuracy of up to ∼ 2 .
32% for similar ReLU budgets.
SENet++ ( d r = 0 . 5 ) can further improve the MAC and ReLU cost of SENet, with additional saving of 4 × and ∼ 2 × , respectively.
P RELIMINARIES AND R ELATED W ORK
Cryptographic primitives.
We briefly describe the relevant cryptographic primitives in this section.
Additive secret sharing.
Given an element x , an ASS of x is the pair ( ⟨ x ⟩ 1 , ⟨ x ⟩ 2 ) =
( x − r, r ) , where r is a random element and x = ⟨ x ⟩ 1 + ⟨ x ⟩ 2 .
Since r is random, the value x cannot be revealed by a single share, so that the value x is hidden.
Homomorphic encryption.
HE Gentry (2009) is a public key encryption scheme that supports homomorphic operations on the ciphertexts.
Here, an encryption function E generates the ciphertext t of a plaintext m where t = E ( m, pk ) , and a decryption function D obtains the plaintext m via m = D ( t, sk ) , where pk and sk are corresponding public and secret key, respectively.
In PI, the results of linear operations can be obtained homomorphically through m 1 ◦ m
=
D
( t 1 ⋆ t 2 , sk ) , where ◦ represents a linear operation, ⋆ is its corresponding homomorphic operation, t 1 and t 2 are the ciphertexts of m 1 and m 2 , respectively.
Garbled circuits.
GC Yao (1986) allows two parties to jointly compute a Boolean function f over their private inputs without revealing their inputs to each other.
The Boolean function f is represented as a Boolean circuit C .
Here, a garbler creates an encoded Boolean circuit ˜ C and a set of input-correspondent labels through a procedure Garble ( C ) to send ˜ C and the labels to the other party who acts as an evaluator.
The evaluator further sends the output label upon evaluation via Eval ( ˜ C ) .
Finally, the garbler decrypts the labels to get the plain results to share with the evaluator.
Private inference.
Similar to Mishra et al.
(2020), in this paper, we focus on a semi-honest clientserver PI scenario where a client, holding private data, intend to use inference service from a server having a private model.
Specifically, the semi-honest parties strictly follow the protocol but try to reveal their collaborator’s private data by inspecting the information they received.
On the other hand, a malicious client could deviate from the protocol.
To defend against various threats existing cryptographic protocols Mishra et al.
(2020); Ghodsi et al. Lou et al.
(2021) rely on the popular online-offline topology Mishra et al.
(2020), where the client data independent component is pre-computed in the offline phase Juvekar et al.
; Mishra et al.
(2020); Ghodsi et al.
; Lehmkuhl et al.
For the linear operations, DELPHI Mishra et al.
and MiniONN Liu et al.
(2017) move the heavy primitives in HE and SS to offline enabling fast linear operations during PI online stage.
However, the compute heavy Eval ( ˜ C ) of GC keeps the ReLU cost high even at the online stage.
ReLU reduction for efficient PI.
Existing works use model designing with reduced ReLU counts via either search for efficient models Mishra et al.
(2020); Ghodsi et al.
Lou et al.
Cho et al.
(2021) or manual re-design from an existing model Jha et al.
In particular,
SAFENet Lou et al.
(2021) enables more fine-grained channel-wise substitution and mixed-precision activation approximation.
CryptoNAS Ghodsi et al.
re-designs the neural architectures through evolutionary NAS techniques to minimize ReLU operations.
Sphynx Cho et al.
(2021) further improves the search by leveraging differentiable macro-search NAS Liu et al.
(2018a) in yielding efficient PI models.
DeepReDuce Jha et al.
(2021), on the other hand, reduced ReLU models via a manual effort of finding and dropping redundant ReLU layers starting from an existing standard model.
Finally, a recent work Cho et al.
(2022) leveraged l 1 -regularization to remove ReLU at the pixel level to yield SOTA accuracy vs non-linearity trade-off. However, the extreme hyperparameter dependence of such methods often provide sub-optimal solution and does not necessarily guarantee to meet a target ReLU budget.
Moreover, a resource limited server Mishra et al. (2020) may not afford costly iterative training Cho et al.
(2022) in reducing the ReLU count.
M OTIVATIONAL S TUDY : R ELATION BETWEEN R E LU IMPORTANCE AND P RUNING S ENSITIVITY
Existing work to find importance of a ReLU layer Jha et al.
(2021), requires manual effort and is extremely time consuming.
In contrast, model pruning literature Lee et al.
; Kundu et al.
(2021a)
leveraged various metrics to efficiently identify a layer’s sensitivity towards a target pruning ratio.
In particular, a layer’s pruning sensitivity can be quantitatively defined as the accuracy reduction caused by pruning a certain ratio of parameters from it Ding et al.
In particular, recent literature leveraged sparse learning Ding et al.
; Kundu et al.
(2021a)
and used a trained sparse model to evaluate the sensitivity of a layer l ( η θ l ) as the ratio [Table] zero [Table]
Despite significant progress in weight pruning sensitivity, due to absence of any trainable parameter in ReLU layer, its sensitivity for a given ReLU budget is yet to be explored .
We hypothesize that there maybe a correlation between a layer’s pruning sensitivity Kundu et al.
(2021a)
and the importance of ReLU and have conducted the following experiments to explore this relation.
Let us assume an L -layer DNN model Φ parameterized by Θ ∈ R m that learns a function f Φ , where m represents the total number of model parameters.
The goal of DNN parameter pruning is to identify and remove the unimportant parameters from a
DNN and yield a reduced parameter model that has comparable performance to the baseline unpruned model.
As part of the pruning process for a given parameter density d , each parameter is associated with an auxiliary indicator variable c belonging to a mask tensor c ∈ { 0 , 1 } m such that only those θ remain non-zero whose corresponding c = 1 .
With these notations, we can formulate the training optimization as
[Equation]
Where L ( . ) represents the loss function.
In general, for image classification tasks, we define L ( . ) to be the cross-entropy (CE) loss.
We used a sparse learning framework Kundu et al.
(2021a) to train a ResNet18 on CIFAR-100 for a target d = 0 .
1 and computed the pruning sensitivity of each layer.
In particular, as shown in Fig. 2, earlier layers have higher pruning sensitivity than later ones.
This means that to achieve close to baseline performance, the model trains later layers’ parameters towards zero more than those of earlier layers.
We then compared this trend with that of the importance of different ReLU layers as defined in Jha et al.
In particular, for each basic block stage, we created a different ResNet18 keeping ReLUs associated with that stage while replacing all other ReLUs with identity layers, trained the network, and measured the resulting test accuracy.
We performed the same operation to evaluate the accuracy with ReLU only after the first CONV layer.
The ReLU stages of higher importance are those which yielded the highest accuracy.
We then normalized the importance of a ReLU stage with accuracy Acc as the ratio ( Acc − Acc min ) /
( Acc max − Acc min ) .
Here Acc max and Acc min corresponds to the accuracy of models with all and no ReLUs, respectively.
As depicted in Figure 2, the results show that the ReLU importance and parameter pruning sensitivity of a layer are inversely correlated.
This inverse correlation may imply that a pruned layer can afford to have more zero-valued weights when the associated ReLU layer forces most of the computed activation values to zero.
SEN ET T RAINING M ETHODOLOGY
As highlighted earlier, for large number of ReLU layers L r , the manual evaluation and analysis of the candidate architectures become inefficient and time consuming.
[-3 η α ← evalActSens( Θ ,d ) 4 for l ← 0 to L r do η α l ← η α l (cid:80) Li =
 η α i × ai 6 end 7 initVals ( r remain ,r total , r final ) 8 while r total < r do for l ← 0 to L dor lcur ← assignReluProportion( r remain ,η α l , a )r lfinal ,r total ← assignUpdateRelu( r lfinal ,r lcur ,r total )end 13 end 14 r remove ← r total − r 15 while r remove > 0 do for l ← 0 to L dor lcur ← removeReluProportion( r del ,η α l , a )r lfinal ,r remove ← removeUpdateRelu( r lfinal ,r lcur ,r remove )end 20 end 21 return r final-]Moreover, the manual assignment of ReLU at the pixel level, becomes even more intractable because the number of pixels that must be considered, explode.
To that end, we now present SENet, a three-stage automated ReLU trimming strategy that can yield models for a given reduced ReLU budget.
4. S ENSITIVITY A NALYSIS
Inspired by our observations in Section 3, we define the ReLU sensitivity of a layer l as
[Equation]
It is important to emphasize that unlike ReLU importance, ReLU sensitivity does not require training many candidate models.
However, η θ l can only be evaluated for a specific d .
We empirically observe that d > 0 .
3 tends to yield uniform sensitivity across layers due to a large parameter budget.
In contrast, ultra low density d < 0 . 1 , costs non-negligible accuracy drops Liu et al.
(2018b)
; Kundu et al.
(2022a).
Based on these observations, we propose to quantify ReLU sensitivity with a proxy density of d = 0 .
Data: Global ReLU budget r , model parameters Θ , model parameter proxy density d , 1 number of ReLU layers L r , active ReLU indicator a ∈ { 1 } Lr 2 Output: Per-layer
# ReLU count.
3 η α ← evalActSens( Θ ,d ) 4 for l ← 0 to L r do η
{+α l ← η α l (cid:80) Li =
 η α i × ai 6 end 7 initVals ( r remain ,r total , r final ) 8 while r total < r do for l ← 0 to L dor lcur ← assignReluProportion( r remain ,η α l , a )r lfinal ,r total ← assignUpdateRelu( r lfinal ,r lcur ,r total )end 13 end 14 r remove ← r total − r 15 while r remove > 0 do for l ← 0 to L dor lcur ← removeReluProportion( r del ,η α l , a )r lfinal ,r remove ← removeUpdateRelu( r lfinal ,r lcur ,r remove )end 20 end 21 return r final+} Moreover, to avoid the compute heavy pruning process, we leverage the idea of sensitivity evaluation before training Lee et al.
On a sampled mini batch from training data D , the sensitivity of the j th connection with associated indication variable and vector as c j and e j , can be evaluated as,
[Equation]
c represents the containing all indicator variables.
The ∂ L ∂c j is an infinitesimal version of ∆ L j measures change in c j from 1 → 1 − δ and can be computed using one forward pass for all j at once.
We normalize the connection sensitivities, rank them, and identify the top d-fraction of connections.
We then define the layer sensitivity η Θ l as the fraction of connections of each layer that are in the top d-faction.
For a given global ReLU budget r , we then assign the # ReLU for each layer proportional to its normalized ReLU sensitivity.
The details are shown in Algorithm 1 (Fig.as point 1 ⃝ ).
4. R E LU M ASK I DENTIFICATION
After layer-wise #ReLU allocation, we next identify the ReLU locations in each layer’s activation map.
In particular, for a non-linear layer l , we assign a mask tensor M l ∈ { 0 , 1 } h l × w l × c l , where h l , w l , and c l represents the height, width, and the number of channels in the activation map. For a layer l , we initialize M with r lfinal assigned 1’s with random locations.
Then we perform a distillation-based training of the PR model performing ReLU ops only at the locations of the masks with 1, while distilling knowledge from an AR model of the same architecture (see Fig. 3, point 2 ⃝ ).
At the end of each epoch, for each layer l , we rank the top- r lfinal locations based on the highest absolute difference between the PR and AR model’s post ReLU activation output (averaged over all the mini-batches) for that layer, and update the M l with 1’s at these locations.
This, on average, de-emphasizes the locations where the post ReLU activations in both the PR and AR models are positive.
We terminate mask evaluation once the ReLU mask 1 evaluation reaches the maximum mask training epochs or when the normalized hamming distance between masks generated after two consecutive epochs is below a certain pre-defined ϵ value. Notably, there has been significant research in identifying important trainable parameters Savarese et al. ; Kusupati et al.
through various proxies including magnitude, gradient, Hessian, however, due tensor has non-zeros irregularly placed.
This can be easily extended to generation of structured mask, by allowing the assignment and removal of mask values at the granularity of channels instead of activation scalar Kundu et al. (2021a). to absence of any trainable parameter in the ReLU layer, such methods can’t be deployed in identifying important ReLU units of a layer. Channel-wise ReLU mask identification.
The mask identification technique described above, creates ReLU masks in an irregular way.
To support a lower level of granularity where the ReLU removal happens ”channel-wise”, we now present a simple yet effective extension of the mask identification.
For a layer l , we first translate the total non-zero ReLU counts to total non-zero ReLU channels as r lc = ⌈ r lfinal h l w l ⌉ .
We then follow the same procedure as irregular mask identification, however, only keep top- r lc channels as non-zero.
4. M AXIMIZING A CTIVATION S IMILARITY VIA D ISTILLATION
Once the mask for each layer is frozen, we start our final training phase in which we maximize the similarity between activation functions of our PR and AR models, see Fig.
3, point 3 ⃝ .
In particular, we initialize a PR model with the weights and mask of best PR model of stage 2 and allow only the parameters to train.
We train the PR model with distillation via KL-divergence loss Hinton et al.
(2015) from a pre-trained AR along with a CE-loss.
Moreover, we introduce an AR-PR post-ReLU activation mismatch (PRAM) penalty into the loss function.
This loss drives the PR model to have activation maps that are similar to that of the AR model.
More formally, let Ψ mpr and Ψ mar represent the m th pair of vectorized post-ReLU activation maps of same layer for Φ pr and Φ ar , respectively.
Our loss function for the fine-tuning phase is given as
[Equation]
loss
[Equation] PRAM loss [Equation] 4) where σ represents the softmax function with ρ being its temperature.
λ balances the importance between the CE and KL divergence loss components, and β is the weight for the PRAM loss.
Similar to Zagoruyko & Komodakis (2016a), we use the l 2 -norm of the normalized activation maps to compute this loss.
4. SEN ET ++: S UPPORT FOR O RDERED C HANNEL D ROPPING
To yield further compute-communication benefits, we now present an extension of SENet, namely
SENet++, that can perform the ReLU reduction while also supporting inference with reduced model sizes.
In particular, we leverage the idea of ordered dropout (OD) Horvath et al.
(2021) to simultaneously train multiple sub models with different fractions of channels.
The OD method is parameterized by a candidate dropout set D r with dropout rate values d r ∈ (0 , 1] .
At a selected d r for any layer l , the model uses a d r -sub model with only the channels with indices { 0 , 1 , ..., ⌈ d r · C l ⌉ − 1 } active, effectively pruning the remaining {⌈ d r · C l ⌉ , ..., C l − 1 } channels.
Hence, during training, the selection of a d r -sub model with d r < 1 . 0 ∈ D r , is a form of channel pruning, while d r = 1 .trains the full model.
For each minibatch of data, we perform a forward pass once for each value of d r in D r , accumulating the loss.
We then perform a backward pass in which the model parameters are updated based on the gradients computed on the accumulated loss.
We first train an AR model with a dropout set D r .
For the ReLU budget evaluation, we consider only the model with d r = 1 . 0 , and finalize the mask by following the methods in Sections 4.1 and 4.2.
During the maximizing of activation similarity stage, we fine tune the PR model supporting the same set D r as that of the AR model.
In particular, the loss function for the fine tuning is the same as 4, for d r = 1 . 0 .
For d r < 1 . 0 , we exclude the PRAM loss because we empirically observed that adding the PRAM loss for each sub model on average does not improve accuracy.
During inference, SENet++ models can be dynamically switched to support reduced channel widths, allowing both ReLUs and MACs to reduce compared to the baseline model.
E XPERIMENTS 5. E XPERIMENTAL S ETUP Models and Datasets.
To evaluate the efficacy of the SENet yielded models, we performed extensive experiments on three popular datasets, CIFAR-10, CIFAR-100 Krizhevsky et al.
(2009), Tiny-ImageNet Hansen (2015), and ImageNet 2 with three different model variants, namely ResNet (ResNet18, ResNet34)
He et al.
(2016), wide residual network (WRN228) Zagoruyko & Komodakis (2016b), and VGG (VGG16) Simonyan & Zisserman (2014).
We used PyTorch API to define and train our models on an Nvidia RTX 2080 Ti GPU. [Table] [Figure]
Training Hyperparameters.
We performed standard data augmentation (horizontal flip and random crop with reflective padding) and the SGD optimizer for all training.
We trained the baseline all-ReLU model for 240, 120, and 60 epochs for CIFAR, Tiny-ImageNet, and ImageNet respectively, with a starting learning rate (LR) of 0.05 that decays by a factor of 0.1 at the 62.5%, 75%, and 87.5% training epochs completion points.
For all the training we used an weight decay coefficient of 5 × 10 − 4 .
For a target ReLU budget, we performed the mask evaluation for 150, 100, and 30 epochs, respectively, for the three dataset types with the ϵ set to 0 .
05 , meaning the training prematurely terminates when less than 5% of the total #ReLU masks change their positions.
Finally, we performed the post-ReLU activation similarity improvement for 180, 120, and 50 epochs, for CIFAR, Tiny-ImageNet, and ImageNet respectively.
Also, unless stated otherwise, we use λ = 0 . 9 , and β = 1000 for the loss described in Eq.
Further details of our training hyper-parameter choices are provided in the Appendix. In
Table 5, we report the accuracy averaged over three runs.
5. SEN ET R ESULTS
As shown in Table 5, SENet yields models that have higher accuracy than existing alternatives by [Table] [Table] a significant margin while often requiring fewer ReLUs.
For example, at a small ReLU budget of ≤ 100 k, our models yield up to 4 .
15% and 7 . 8% higher accuracy, on CIFAR-10 and CIFAR-100, respectively.
At a ReLU budget of ≤ 500 k, our improvement is up to 0 .
50% and 2 .
38% , respectively, on the two datasets.
We further evaluate the communication saving due to the nonlinearity reduction by taking the per ReLU communication cost mentioned in Table 2.
In particular, the communication saving reported in the 8 th column of Table 5 is computed as the ratio of communication costs associated with an AR model to that of the corresponding PR model with reduced ReLUs. We did not report any saving for the custom models,
[Table]
as they do not have any corresponding AR baseline model.
On Tiny-ImageNet, SENet models can provide up to 0 .
3% higher performance while requiring 3 .
× fewer ReLUs (Table 3).
More importantly, even for high resolution dataset like ImageNet, SENet models can yield close to the baseline performance, depicting the efficacy of our proposed training method.
Results with activation channel level ReLU reduction. As shown in Table 4, while trimming ReLUs at a higher granularity of activation channel level, SENet models suffers a little more drop in accuracy compared to that at pixel level. For example, at a ReLU budget of 240k, channel level
ReLU removal yields accuracy of 79 .
3% compared to 79 .
81% of pixel-level.
However, compared to existing alternatives, SENet can achieve improved performance of up to 1 .
85% for similar ReLUs.
For SENet++, we performed experiments with D r = [0 . 5 , 1 . 0] , meaning each training loop can yield models with two different channel dropout rates.
The 0 .
5 -sub model enjoys a ∼ 4 × MACs reduction compared to the full model.
Moreover, as shown in Fig. 4, the 0 .
5 -sub model also requires significantly less # ReLUs due to reduced model size.
In particular, the smaller models have # ReLUs reduced by a factor of 2 .
05 × , 2 . 08 × , and 1 . 88 × on CIFAR-10, CIFAR-100, and Tiny-ImageNet, respectively, compared to the PR full models, averaged over four experiments with different ReLU budgets for each dataset.
Lastly, the similar performance of the SENet and SENet++ models at d r = 1 . 0 with similar ReLU budgets, clearly depicts the ability of SENet++ to yield multiple sub models without sacrificing any accuracy for the full model .
5. A NALYSIS OF L INEAR AND R E LU I NFERENCE L ATENCY
Table 2 shows the GC-based online ReLU operation latency is ∼ 343 × higher than one linear operation (multiply and accumulate), making the ReLU operation latency the dominant latency component.
Inspired by this observation, we quantify the online PI latency as that of the N ReLU operations for a model with ReLU budget of N .
In particular, based on this evaluation, Fig. 5(a) shows the superiority of SENet++ of up to ∼ 9 .
6 × ( ∼ 1 . 92 × ) reduced online ReLU latency on CIFAR- (CIFAR-100).
With negligibly less accuracy this latency improvement can be up to ∼ 21 × .
Furthermore, as for d r < 1 . 0 , SENet++ requires fewer MACs, the linear operation latency can also be reduced significantly as demonstrated in Fig. 5(b).
5. A BLATION S TUDIES
[Table]
Importance of ReLU sensitivity.
For a given ReLU budget, to understand the importance of layer-wise ReLU sensitivity evaluations, we conducted experiments with evenly allocated ReLUs.
Specifically, for
ResNet18, for a ReLU budget of 25% as that of the original model, we randomly removed 75% ReLUs from each PR layer with identity elements to create the ReLU mask and trained the PR model with this mask.
We further trained two other PR ResNet18 with similar and lower # ReLU budgets with the per-layer ReLUs assigned following the proposed sensitivity.
As shown in Table 6, the sensitivity driven PR models can yield significantly improved performance of ∼ 5 .
76% for similar ReLU budget, demonstrating the importance of proposed ReLU sensitivity.
Choice of the hyperparameter λ and β .
To determine the influence of the AR teacher’s influence on the PR model’s learning, we conducted the final stage distillation with λ ∈ [0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9] and β ∈ [100 , 300 , 500 , 700 , 1000] .
As shown in Fig.
6, the performance of the student PR model improves with the increasing influence of the teacher both in terms of high λ and β values.
However, we also observe, the performance improvement tend to saturate at β ≈ 1000 .
Note, we keep λ = 0 .
5 and β = 1000 , for β and λ ablation, respectively.
C ONCLUSIONS
In this paper, we introduced the notion of ReLU sensitivity for non-linear layers of a DNN model.
Based on this notion, we present an automated ReLU allocation and training algorithm for models with limited ReLU budgets targeting latency and communication efficient PI.
The resulting networks can achieve similar to SOTA accuracy while significantly reducing the # ReLUs by up to 9 .
6 × on
CIFAR-10, enabling dramatic reduction of the latency and communication costs of PI.
A A PPENDIX A. T RAINING H YPERPARAMETERS
The training hyperparameter details for each stage is provided in table 7. Also, for the KLdivergence loss in stage 2 and 3, we fix the temperature value ρ = 4 . 0 .
To evaluate η θ , we a set of 1000 randomly selected training image samples. Table 8 details the loss function used for different OD rate values for the SENet++ (at the fine-tuning stage of the PR model i.e. training stage 3). Note, the training stage 1 where the AR model gets trained, we use standard CE loss for all the sub-models with different OD rate values.
The detailed fine-tune training algorithm is provided in 2. Note, for SENet, the algorithm remains same with D r = [1 . 0] , meaning supporting only the full model with all the channels present 3 . [Table]
[Table]
Algorithm 2: SENet++ Fine Tune Training Stage
[Table]
PR , mini-batch size B , learned ReLU mask from stage 2 Π , OD set D r .
1 , Output: Trained PR model with # ReLU count r .
2 Θ PR ← applyModelWeight( Θ AR
Π ) 3 for i ← 0 to to ep do for j ← 0 to n B do for dropout rate d r in sorted D r do sampleSBN ( d r ) //sample
the BN corresponding to d r
L CE ← computeCELoss( X 0: B , Y 0:
B , Y pr 0: B , Π )
{+L KL ← computeKLLoss( f Φ (Θ PR ) , f Φ (Θ AR ) , ρ, Π ) if d r == 1 . 
+}
then
L PRAM ← computePRAMLoss( f Φ (Θ PR ) , f Φ (Θ AR ) , Π )
L ← (1 − λ )
L CE + λ L KL + β 2 L PRAM else
L ← (1 − λ )
L CE + λ L KL end accumulateGrad
( L ) end updateParam ( Θ PR , ∇ L ) end 19 end
A. M ODEL L ATENCY E STIMATION
Similar to the existing literature Jha et al.
Cho et al.
; Lou et al.
(2021), we assume the popular PI framework described in Delphi Mishra et al.
(2020), and leverage their reported per ReLU operation latency to estimate the total online cryptographic private inference latency.
In particular, assuming sequential execution, the total latency can be estimated as
[Equation]
where N MAC and N ReLU corresponds to the total number of MAC and ReLU operations required to perform a single forward pass.
Per ReLU cipher text execution wall-clock time is taken from [4] and the execution of 1000 ReLUs takes proportional time as reported in Cho et al.
Thus similar to Cho et al.
; Jha et al.
, we extrapolated per-ReLU wall clock time to extract total ReLU latency for cryptographic inference.
Now, as cryptographic ReLU latency is around 343 × costlier than that for linear ops, similar to earlier literature Jha et al.
Cho et al.
Lou et al.
(2021), for comparable N MAC and N ReLU we can approximate the total latency with that of the non-linear latency.
A. M ORE RESULTS AND A NALYSIS
Layer-wise ReLU Sensitivity.
Fig.
7(a) shows the ReLU sensitivity per layer.
As elaborated in the Section 3, the ReLU sensitivity is more in the later layers, making our ReLU sensitivity follow similar trend as ReLU importance in DeepReDuce Jha et al.
Fig. 7(b) shows the layer wise ReLU count for different ReLU budget, allocation driven by sensitivity (Fig. 7(a)).
With the increasing count of ReLU budget, the assignment of ReLU happens more aggressively at the later layers, compared to the earlier ones.
SENet++: Training with More Than 2 Dropout Rates.
In the original manuscript we showed results with two dropout rates ( D r = [0 . 5 , 1 . 0] ), while training with different dropouts, and yield models with different channel width factors.
We now show results with D r = [0 .
In particular, Fig. 8 and 9 show the results with models yielded via training on four different d r choices.
As shown in the Fig. 8(a) and 9(a), the performance of the models at d r = 1 . 0 and d r = 0 .
5 , are similar for both two and four d r choices, making SENet++ an efficient algorithm in yielding multiple reduced FLOPs/ReLU models.
Moreover, the effective ReLU reduction remains proportional to the corresponding ordered dropout rate (OD rate) (Fig. 8(b) and 9(b)).
Fig.8(c) shows the effective FLOPs reduction for the CONV layers while performing inference at a reduced
OD rate model selection.