Paper: Optimizing Layerwise Polynomial Approximation for Efficient Private Inference on Fully Homomorphically Encryption: A Dynamic Programming Approach (ICLR-2024)

Reviewer comment (Experiments):
The authors only evaluate the proposed method on ReLU. However, other activation functions, e.g., Swish, Sigmoid, GeLU, are not evaluated.

Author response:
**Simulations for other activations**: The optimized approximation method we propose is adaptable to other activation functions, as outlined in Theorem 2.2. Given that this theorem establishes the minimization of mean squared error (MSE) by the derived polynomial, we can theoretically regard it as the optimal polynomial for minimizing inference errors. Furthermore, the ReLU function, in contrast to other activation functions, is non-differentiable, which can result in substantial errors. 

Also, our findings show that when approximating ReLU using the weighted least squares method, the MSE is higher than other differentiable activation functions (e.g., Swish, Sigmod, GELU). See the table below which presents the minimized for each activation where $\mu=0$ and $\sigma^2=1$. The minimized MSE can be evaluated by the second equation in Theorem 2.2.

| Degree | ReLU | GELU | Swish | Sigmoid |
|:---:|:---:|:---:|:---:|:---:|
| 0 | 0.34085 | 0.34564 | 0.31308 | 0.04338 |
| 1 | 0.09085 | 0.09564 | 0.06308 | 0.00069 |
| 2 | 0.00612 | 0.00113 | 0.00154 | 0.00069 |
| 3 | 0.00612 | 0.00113 | 0.00154 | 0.00004 |

The efficacy of our method in approximating the challenging ReLU function supports that it should perform effectively in approximating other activation functions as well.

Reviewer comment (Presentation):
Some visualization of (1) the ReLU function, (2) the proposed approximation, and (3) other baselines will be helpful to understand the effect of the proposed approximation.

Author response:
**Visualization for the proposed approximate polynomial**: In the final version of the paper, we will include a graph comparing the characteristics of the existing minimax polynomial with those of the proposed polynomial. This will aid in better understanding the differences between the two.

Reviewer comment (Experiments):
Only a simple dataset, i.e., Cifar-10, is demonstrated. Larger datasets are important as they may impact the distribution of the activation functions as well as the required approximation accuracy.

Author response:
**Simulation for various dataset**: We agree that the various simulations for many datasets are important. However, the challenges for large dataset in FHE-based PI still need further exploration. Pre-trained models that classify such datasets tend to have more intricate functions requiring optimization. For instance, non-linear functions such as max-pooling involve complex multivariate functions, and there hasn't been a suggested method to effectively approximate these functions in encrypted data for use in DNN inference. Moreover, due to current time latency issues, attempts to classify extensive datasets like TinyImageNet without retraining are restricted to operating on the final layers with encrypted data using FHE [1].

[1] D. Kim et al, "Optimized Privacy-Preserving CNN Inference With Fully Homomorphic Encryption." IEEE Transactions on Information Forensics and Security 18 (2023): 2175-2187.

Reviewer comment (Experiments):
I just have one major concern. There exist some work use 2-degree polynomial approximation [1,2] with ignorable accuracy loss. How does the proposed method compared to other advanced low-degree polynomial approximation techniques? As a low-degree polynomial approximation would outperforms the results in this work.  

Reference:  

[1] Park, Jaiyoung, et al. "AESPA: Accuracy preserving low-degree polynomial activation for fast private inference." arXiv preprint arXiv:2201.06699 (2022).  
[2] Peng, Hongwu, et al. "PASNet: Polynomial Architecture Search Framework for Two-party Computation-based Secure Neural Network Deployment." 2023 60th ACM/IEEE Design Automation Conference (DAC). IEEE, 2023.

Author response:
Thank you for recognizing that the polynomial approximation problem is a crucial aspect in reducing latency for private inference (PI).

Many studies employ low-degree polynomials as activation functions in PI using multi-party computation (MPC) protocols, allowing data sharing among multiple parties. Both of the studies you presented are MPC-based works which achieved high performance using low-degree polynomials. In contrast, our work focuses solely on encrypted data inferences using fully homomorphic encryption (FHE). FHE-based PI enables a server to perform homomorphic computations by itself without requiring additional communication costs.

Within the realm of PI, MPC-based methodologies, similar to our study, tackle similar challenges and hold significance owing to unique advantages, like precise computation of non-arithmetic functions such as ReLU. Yet, directly contrasting these fields presents hurdles. Each prioritizes different optimization facets: FHE-based PI focuses on minimizing computation time for overall inference, whereas MPC-based PI grapples with substantial impediments in communication costs during computations. Despite recent efforts in MPC-based approaches employing low-degree polynomials for activation functions, their drawback lies in the substantial consumption of several gigabytes of communication costs [1]. Previous studies lack comprehensive quantitative assessments between FHE-based and MPC-based PI. Depending on the context, MPC-based methodologies might prove beneficial, while FHE-based approaches could offer distinctive advantages. Acknowledging the significance of both technologies is pivotal, given their representation as critical domains in continual progression.

[1] P. Mishra et al, "Delphi: A Cryptographic Inference System for Neural Networks," in Proc. Workshop Privacy-Preserving Mach. Learn. Pract., Nov. 2020.

Reviewer comment (Experiments):
**Complexity and scalability of the proposed approach:** An essential concern regarding the presented solution lies in its complexity, an aspect left unexplored in the paper. Given that the method determines the appropriate polynomial degree for each layer, its computational complexity varies depending on the network's depth. This adaptability makes it increasingly impractical for deeper networks, such as ResNet101 and ResNet152. In contrast, the complexity of PI-specific manual ReLU pruning depends solely on the number of stages within the networks and remains independent of the total number of layers.

Author response:
**Complexity and scalability of the proposed approach**: We agree that the computational complexity of our method depends on the network’s depth. In Section 3.2, page 7, we provide the complexity of the proposed dynamic programming algorithm with big O notation.
However, once we obtain the set of the polynomial degrees from the proposed algorithm, we can perform the inference on all test datasets without additional computations for determining polynomials. Furthermore, the operations in the proposed algorithm involve operations between plaintexts, not ciphertexts. Therefore, we can efficiently obtain the optimal set of the polynomial degrees within a short timeframe. Consequently, obtaining the optimal polynomials for a given backbone network is expected to pose no significant burden when performing inference on the dataset.

Reviewer comment (Experiments):
**Lack of comprehensive empirical evaluation:** The experimental evaluation in the paper is limited to CIFAR-10 using ResNet20 and ResNet32 networks. As per the standard practice in PI [1, 2, 3], an evaluation on CIFAR-100 and TinyImageNet datasets with networks such as ResNet18 should have been included to validate the effectiveness of the proposed solution on complex datasets.

Author response:
**Simulation for complex dataset**: Experiments could be conducted on the CIFAR-100 dataset to further demonstrate the effectiveness of our algorithm. We check the performance of our optimizing method by classifying plaintext CIFAR-100 on ResNet-32.

| Method | Max. Degree | Max. Depth | Original Acc. | Approx. Acc. |
|:---:|:---:|:---:|:---:|:---:|
| Previous [11] | 6,075 | 14 | 69.50% | 69.43% |
| Proposed | 511 | 9 | 69.49% | 68.94%* |

(*The accuracy of the model using the proposed approximated polynomials is obtained in plaintext.)

Our technique significantly reduced depth consumption while maintaining the performance of the pre-trained model. Since the maximum depth consumption is reduced from 14 to 9, we can anticipate that the computation time for bootstrapping and ReLU evaluation will also be significantly reduced in ciphertext CIFAR-100 classification using the proposed method. Given the limited discussion timeframe, demonstrating other outcomes such as ciphertext experiments might be challenging at present. However, efforts will be made to sufficiently supplement the experimental content before the final revision deadline.

FHE-based PI still presents several unresolved issues necessitating further exploration, prompting us to propose future work for classifying larger datasets. Pre-trained models capable of classifying large datasets often contain more complex functions requiring optimization. For instance, non-arithmetic functions like max-pooling involve multivariate functions, and there hasn't been a proposed approach to effectively approximate these in encrypted data for inference in DNNs. Additionally, due to the current time latency issues, instances of classifying large datasets like TinyImageNet without retraining are limited to performing on the last few layers with FHE'ed data [12].

Reviewer comment (Experiments):
**Comparison with prior work:** To demonstrate the superiority of the HE-only solution over a hybrid approach (HE + MPC) [1, 2, 3], a comparative analysis of the end-to-end PI runtime is required.

Author response:
**Comparison between previous works with hybrid approach and our work**: Both approaches, the hybrid approach (allowing MPC) and the FHE-only approach, share the common goal of performing privacy-preserving inference. However, directly comparing these domains is impracticable due to differences in their focus areas: these approaches deal with *distinct domains* and optimize *different bottlenecks*.

*Distinct domain.* We kindly ask for consideration regarding the fact that research in these two domains: hybrid and FHE-only, is conducted for distinct environmental aspects. If the client can participate in computations alongside the server during PI service provision, employing a hybrid approach is desirable. However, situations may arise, due to communication constraints (such as the client being intermittently offline) or for security reasons concerning the server's computational process, where client involvement during inference is not permitted. In such cases, the hybrid model becomes unfeasible, necessitating the exclusive use of an FHE-only model. Therefore, the enhancement of FHE-only models a crucial research focus on some scenarios. This is why our current line of work has been shaped: to improve upon FHE-only models by comparing and contrasting them with their counterparts.

*Different bottleneck.* Each approach prioritizes different optimization aspects: FHE-only PI focuses on minimizing computation time for overall inference, while hybrid PI addresses significant challenges related to communication costs during computations. Despite recent MPC-based attempts to use low-degree polynomials for activation functions, their drawback lies in their consumption of several gigabytes of communication cost [7]. Due to these inherent and critical distinctions, quantitatively comparing these challenges is challenging. To our knowledge, there lacks proper papers that fairly compare FHE-based PI and MPC-based PI. 

Moreover, hardware accelerators have effectively addressed the inference time, previously a bottleneck in FHE-only approaches. Several studies [8-10] demonstrate the practicality of executing homomorphic operations and inference on DNNs using the RNS-CKKS scheme with FHE’ed data. Our research compares the effects of reducing the degree of the approximating polynomial in situations where the backbone network mandates the direct utilization of given pre-trained parameters without retraining, contrasting this with conventional minimax approximations [11,12]. Directly comparing the overall end-to-end PI runtime with other hybrid works seems difficult. However, with the ongoing advancements in hardware accelerators and related technologies, integrating them with our layerwise optimization gives ample reason to anticipate a further reduction in the overall runtime.

Summing up, the PI researches on hybrid approach and FHE only approach exhibit distinct differences, each serving specific environments and application domains. Consequently, independent research is imperative in both areas to accommodate their unique applicability.

[7] P. Mishra et al, "Delphi: A Cryptographic Inference System for Neural Networks," in Proc. Workshop Privacy-Preserving Mach. Learn. Pract., Nov. 2020.

[8] S. Kim et al, "BTS: An Accelerator for Bootstrappable Fully Homomorphic Encryption," Proceedings of the 49th Annual International Symposium on Computer Architecture. 2022.

[9] SU. Yang, et al, "FPGA-based Hardware Accelerator for Leveled Ring-LWE Fully Homomorphic Encryption," IEEE Access, 2020.

[10] H. Lee et al, "A 2.7~13.3 uJ/boot/slot Flexible RNS-CKKS Processor in 28nm CMOS Technology for FHE-based Privacy-Preserving 
Computing," IEEE ISSCC, San Francisco, CA, USA, Feb. 2024, accepted for presentation. 

[11] E. Lee et al, "Low-Complexity Deep Convolutional Neural Networks on Fully Homomorphic Encryption Using Multiplexed 
Convolutions," ICML, 2022.

[12] D. Kim et al, "Optimized Privacy-Preserving CNN Inference With Fully Homomorphic Encryption." IEEE Transactions on Information Forensics and Security 18 (2023): 2175-2187, 2023.

Reviewer comment (Experiments):
**Question:** Given that Table 1 indicates the need for a high-degree polynomial in the initial layer, is it possible to solely utilize the identity connection in these layers? Previous research on PI-specific ReLU optimization has shown that the initial ReLUs are not critical and can be eliminated without significant performance degradation. Is this applicable here?

Author response:
**Reply for the last question**: We note that the optimized polynomial degrees would depend on the datasets, the trained models, and the retraining process. While not directly within the scope of PI research, analogous studies in the field of quantization–specifically, those focusing on quantizing weights and activations without retraining the model–have indicated the need for higher bit-precision in the initial layers during quantization [6]. This observation is consistent with our research, where we use higher-degree polynomials to approximate the initial ReLU functions. Additionally, if our method were to be employed in scenarios permitting retraining as in prior MPC-based work, the initial ReLU functions might be eliminated.

Reviewer comment (Theory):
Why is the "variance of the loss" a good surrogate for the classification accuracy? The challenge in encrypted domain inference is not sensitivity to small approximation errors. Contrarily, even a single large (unbounded) approximation error can screw up the entire inference process (as observed in the bit-flip attack on machine learning models). This is the reason we need some bound on the approximation error (leading to the minimax formulation).

Author response:
**Discussion of the proposed variance of the loss**: We provide an intuition to explain why the variance of the loss increment $Var(\Delta L)$ is a proper surrogate in our optimization. Because of $\Delta \mathcal{L}= \frac{\partial \mathcal{L}}{\partial a_{i,j}} \Delta a_{i,j}$, where $a_{i,j}$ is a random variable representing a polynomial approximation error (see page 5), we can set $E[\Delta \mathcal{L}]=\frac{\partial \mathcal{L}}{\partial a_{i,j}} E[\Delta a_{i,j}]$. Our empirical findings indicate that $E[\Delta a_{i,j}]\approx 0$, which leads to $E[\Delta \mathcal{L}]\approx 0$. Hence, we can claim that $Var(\Delta L) \approx E[(\Delta L)^2]$. 
The practical implication of this approximation is important. By focusing on minimizing $Var(\Delta L)$, we can effectively reduce the noise power on the loss function (i.e., maximizing the signal to noise ratio (SNR) of loss). 

As you mentioned, we fully agree that a single large approximation error can severely compromise the total inference accuracy. This holds true even in the context of minimax approximation. Recognizing the impact of a single value on the overall inference, the configuration of the approximation range becomes crucial. Consequently, both prior research employing minimax approximation and our own study have considered the task of defining the approximation range [3,4].

In the study employing minimax approximation, the approximation range should be set such that all input values of each activation function fall within the approximation range. In our approach, employing the weighted least squares method, we aimed to reduce the size of regions where polynomial values diverge significantly. To achieve this, we scaled the obtained standard deviation $\sigma$ of the input distribution as an approximation weight, as detailed in the last part of Section 2.2, page 4.

Once an appropriate approximation range is established, the inference error can be considered as determined by the small errors in each layer. These small errors in each layer are expected to have varying impacts on the overall inference result. Through the discussion in Section 3.1, we argue that the diverse contributions of each layer directly correlate with the variance of the loss increment $Var(\Delta L)$. This discussion gains significance from the assumption that input values exist within appropriate approximation ranges, ensuring the absence of single large approximation errors in each layer.

Our simulation results further support the validity of our approximation range settings. The minimal difference in classification accuracy before and after approximating the ReLU function with a polynomial supports our approach (see the results of the proposed method in Table 2).

Another reason that the variance of the loss increment can be considered as a stable measure that indicates the classification accuracy is that the value $\Delta L$ is always positive. We can assume that the activation nodes are on the global minimum of the loss function. Then, the small errors of the activation function always result in positive change on the loss function. Therefore, the reduction of the variance $\Delta L$ leads the activation function to the global minimum, which gives better results in classification accuracy.

[3] J.-W. Lee et al, "Privacy-Preserving Machine Learning with Fully Homomorphic Encryption for Deep Neural Network," IEEE Access, 10:30039–30054, 2022b.

[4] E. Lee et al, "Low-Complexity Deep Convolutional Neural Networks on Fully Homomorphic Encryption Using Multiplexed Convolutions," ICML, 2022.

Reviewer comment (Novelty):
It is important to acknowledge the progress made on ReLU reduction in the field of MPC-based private inference because it is directly relevant to the problem considered in this work.

Author response:
**Comments about AutoReP**: Thank you for bringing the recent AutoReP study to our attention. We will acknowledge AutoReP and other pertinent research on MPC-based private inference in our paper. It's noteworthy that AutoReP introduces a method for selecting specific ReLU functions to replace with identical degree polynomials, which is technically distinct from our approach. In our work, due to the limitations of fully homomorphic encryption (FHE) in evaluating ReLU operations, we substitute all ReLU functions in nodes with fine-grained polynomials, which are optimized by dynamic programming approach. 
Moreover, AutoReP employs retraining to identify ReLU activation nodes for polynomial replacement. In contrast, our method does not require retraining, making it more suited for scenarios where retraining is not feasible.
Additionally, we would like to note that both AutoReP and our work are considered contemporaneous according to the ICLR reviewer guidelines (https://iclr.cc/Conferences/2024/ReviewerGuide) because AutoReP was posted on arXiv in August 2023, a month prior to our submission in September 2023.

Reviewer comment (Experiments):
Can the proposed dynamic programming approach be scaled one step further to find the optimal polynomial degree for each specific neuron?

Author response:
**Optimizing for each specific neuron**: In AutoReP, a more optimized approximation approach is adopted by considering the characteristics of each neuron individually, rather than uniformly applying the same polynomial across the entire layer. While the RNS-CKKS scheme also allows applying different polynomials to each neuron within a single ciphertext, the RNS-CKKS scheme used in our FHE-based technology follows a single instruction multiple data (SIMD) approach, where multiple data are encoded into a single ciphertext for homomorphic operations. Consequently, even if different polynomials are applied to the values within a single ciphertext, the depth consumption of the ciphertext depends on the maximum degree among those polynomials. In FHE-based operations, the most crucial factor influencing computation time is the depth consumption of ciphertexts. Therefore, applying different polynomials to each neuron may not result in significant differences in overall computation time.

However, our proposed dynamic programming technique can work independently of the SIMD structure, making it applicable to MPC-based schemes as well. By assessing the input distribution for each neuron and formulating an optimization problem with optimization variables for each degree (i.e., $d_{i,j}$ for $j$th node in the $i$th layer), we can optimize the polynomials tailored to each individual neuron.

Reviewer comment (Evaluation):
The results indicate that there is orders of magnitude decrease in the maximum polynomial degree (see Table 3), which is surprisingly not reflected in the maximum depth as well as the overall inference time. There should be a more in-depth analysis of how the maximum polynomial degree impacts the inference time because that forms the core motivation for this work.

Author response:
**The impact of degrees and depths on the inference time**: In FHE-based schemes, the total inference time is closely associated with the depth consumption of a polynomial rather than the degree. The depth consumption is almost proportional to the logarithm of the polynomial degree (depth consumption = $\lceil \log_2 (d+1) \rceil$, where $d$ is the degree of the polynomial).

The analysis of the depth consumption and its relation to the computation time for the polynomials proposed in each layer is presented in Section 4. As illustrated in Figure 1, page 7, as the depth $\delta$ consumed by each polynomial in a layer decreases, both bootstrapping and polynomial evaluation times decrease (blue graph). Furthermore, experimental results using the moduli chain management method proposed in Section 4 confirm that as the depth of the approximated polynomials decreases, the computation time decreases even more (orange graph).
By reducing depth consumption, we can effectively reduce the overall inference time. We provide detailed analysis on this aspect in Section 4.