Original: H YPHEN : A H YBRID P ACKING M ETHOD AND O PTIMIZATIONS FOR H OMOMORPHIC E NCRYPTION B ASED N EURAL N ETWORK
Revised: H YPHEN : A H YBRID P ACKING M ETHOD AND O PTIMIZATIONS FOR H OMOMORPHIC E NCRYPTION B ASED N EURAL N ETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Private Inference (PI) enables users to enjoy secure AI inference services while companies comply with regulations.
Revised: Private Inference (PI) enables users to enjoy secure AI inference services while companies comply with regulations.

Original: Fully Homomorphic Encryption (FHE) based Convolutional Neural Network (CNN) inference is promising as users can offload the whole computation process to the server while protecting the privacy of sensitive data.
Revised: Fully Homomorphic Encryption (FHE) based Convolutional Neural Network (CNN) inference is promising as users can offload the whole computation process to the server while protecting the privacy of sensitive data.

Original: Recent advances in AI research have enabled HE-friendly
Revised: Recent advances in AI research have enabled HE-friendly

Original: deep
Revised: deep

Original: CNN like ResNet.
Revised: CNN like ResNet.

Original: However, FHE-based CNN (HCNN) suffers from high computational overhead.
Revised: However, FHE-based CNN (HCNN) suffers from high computational overhead.

Original: Prior HCNN approaches rely on dense packing techniques that aggregate as many channels into the ciphertext to reduce element-wise operations like multiplication and bootstrapping.
Revised: Prior HCNN approaches rely on dense packing techniques that aggregate as many channels into the ciphertext to reduce element-wise operations like multiplication and bootstrapping.

Original: However, these approaches require performing an excessive amount of homomorphic rotations to accumulate channels and maintain dense data organization, which takes up most of the runtime.
Revised: However, these approaches require performing an excessive amount of homomorphic rotations to accumulate channels and maintain dense data organization, which takes up most of the runtime.

Original: To overcome this limitation, we present HyPHEN, a deep HCNN implementation that drastically reduces the number of homomorphic rotations.
Revised: To overcome this limitation, we present HyPHEN, a deep HCNN implementation that drastically reduces the number of homomorphic rotations.

Original: HyPHEN utilizes a novel convolution algorithm, RAConv, utilizing replication-based data organization, which leads to a significant reduction in rotation count.
Revised: HyPHEN leverages two convolution algorithms, CAConv and RAConv. Alternating between two convolution algorithms leads to a significant reduction in rotation count.

Original: Furthermore, we propose hybrid gap packing method for HyPHEN, which gathers sparse convolution results into a dense data organization with a marginal increase in the number of rotations.
Revised: Furthermore, we propose hybrid gap packing method for HyPHEN, which gathers sparse convolution results into a dense data organization with a marginal increase in the number of rotations.

Original: HyPHEN explores the trade-off between the computational costs of rotations and other operations, and finds the optimal point minimizing the execution time.
Revised: HyPHEN explores the trade-off between the computational costs of rotations and other operations, and finds the optimal point minimizing the execution time.

Original: With these optimizations, HyPHEN takes 3.8-4.9 × less execution time than the state-of-the-art HCNN implementation and brings the runtimes of ResNet  inference down to 1.38-14.86s using a GPU-accelerated HEAAN library.
Revised: With these optimizations, HyPHEN takes 3.4-4.4 × less execution time than the state-of-the-art HCNN implementation and brings the runtimes of ResNet on CIFAR10 inference down to 1.44-13.37s using a GPU-accelerated HEAAN library.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Private inference (PI) has recently gained the spotlight in the MLaaS domain as cloud companies should comply with privacy regulations such as GDPR Regulation (2016) and HIPAA Act (1996).
Revised: Private inference (PI) has recently gained the spotlight in the MLaaS domain as cloud companies should comply with privacy regulations such as GDPR Regulation (2016) and HIPAA Act (1996).

Original: PI enables inference services at the cloud server while protecting the privacy of the client and the intellectual properties of the service provider.
Revised: PI enables inference services at the cloud server while protecting the privacy of the client and the intellectual properties of the service provider.

Original: For instance, hospitals can provide a private medical diagnosis of diseases, and security companies can provide private surveillance systems without accessing client’s sensitive data (Kumar et al., 2020; Bowditch et al., 2020).
Revised: For instance, hospitals can provide a private medical diagnosis of diseases, and security companies can provide private surveillance systems without accessing client’s sensitive data (Kumar et al., 2020; Bowditch et al., 2020).

Original: PI can be achieved using various cryptographic primitives (Gentry, 2009; Yao, 1982; Costan &
Revised: PI can be achieved using various cryptographic primitives (Gentry, 2009; Yao, 1982; Costan &

Original: Devadas, 2016).
Revised: Devadas, 2016).

Original: Fully Homomorphic Encryption (FHE), which is a set of cryptographic schemes that can directly evaluate a rich set of functions on encrypted data, is especially suited for PI.
Revised: Fully Homomorphic Encryption (FHE), which is a set of cryptographic schemes that can directly evaluate a rich set of functions on encrypted data, is especially suited for PI.

Original: FHEbased PI solution uniquely features 1) full offloading of the computation process to the server, 2) succinct data communication requirement, and 3) non-disclosure of any information about the model except the inference result.
Revised: FHEbased PI solution uniquely features 1) full offloading of the computation process to the server, 2) succinct data communication requirement, and 3) non-disclosure of any information about the model except the inference result.

Original: Such benefits have driven researchers to investigate convolutional neural network (CNN) PI implementations using FHE (Gilad-Bachrach et al., 2016; Brutzkus et al., 2019;
Revised: Such benefits have driven researchers to investigate convolutional neural network (CNN) PI implementations using FHE (Gilad-Bachrach et al., 2016; Brutzkus et al., 2019;

Original: Dathathri et al., 2020; Lee et al., 2022a).
Revised: Dathathri et al., 2020; Lee et al., 2022a; Aharoni et al., 2020).

Original: To implement CNN using FHE, activation functions should be replaced with polynomials as FHE only supports arithmetic operations of addition and multiplication.
Revised: To implement CNN using FHE, activation functions should be replaced with polynomials as FHE only supports arithmetic operations of addition and multiplication.

Original: Given the constraint, two classes of polynomial activation functions have been proposed: (i) low-degree polynomials (Gilad-Bachrach et al., 2016; Chabanne et al., 2017) replacing the activation functions in training neural networks, and (ii) more precise high-degree approximation of ReLU (Lee et al., 2021) that replaces RELU during
Revised: Given the constraint, two classes of polynomial activation functions have been proposed: (i) low-degree polynomials (Gilad-Bachrach et al., 2016; Chabanne et al., 2017) replacing the activation functions in training neural networks, and (ii) more precise high-degree approximation of ReLU (Lee et al., 2021) that replaces RELU during

Original: PI without additional training.
Revised: PI without additional training.

Original: However, both approaches lack practicality; low-degree polynomials are not applicable to deep neural networks and high-degree approximation significantly degrades the runtime of PI.
Revised: However, both approaches lack practicality; low-degree polynomials are not applicable to deep neural networks and high-degree approximation significantly degrades the runtime of PI.

Original: Recently, Park et al.
Revised: Recently, Park et al.

Original: (2022) showed that deep homomorphic CNNs (HCNNs) can be trained with low-degree polynomials even for complex image datasets with their proposal, AESPA, which utilizes orthogonal polynomial bases and fuses activation functions with batch normalization (BN) to turn them into second-degree polynomials.
Revised: (2022) showed that deep homomorphic CNNs (HCNNs) can be trained with low-degree polynomials even for complex image datasets with their proposal, AESPA, which utilizes orthogonal polynomial bases and fuses activation functions with batch normalization (BN) to turn them into second-degree polynomials.

Original: AESPA does not sacrifice runtime nor accuracy unlike prior approaches, thus we employ AESPA in our work.
Revised: AESPA does not sacrifice runtime nor accuracy unlike prior approaches, thus we employ AESPA in our work.

Original: Another line of research lies in implementing an efficient convolution algorithm in FHE.
Revised: Another line of research lies in implementing an efficient convolution algorithm in FHE.

Original: Gazelle (Juvekar et al., 2018) proposed a convolution algorithm that can compute a single Conv layer on FHE.
Revised: Gazelle (Juvekar et al., 2018) proposed a convolution algorithm that can compute a single Conv layer on FHE.

Original: However, Gazelle’s method cannot be directly applied to continuous convolutions as it requires adjusting arrangement of data by re-encrypting ciphertexts after every Conv layer.
Revised: However, Gazelle’s method cannot be directly applied to continuous convolutions as it requires adjusting arrangement of data by re-encrypting ciphertexts after every Conv layer.

Original: Lee et al.
Revised: Lee et al.

Original: (2022a) modified Gazelle’s convolution by densely mapping data into a ciphertext before entering the next
Revised: (2022a) modified Gazelle’s convolution by densely mapping data into a ciphertext before entering the next

Original: Conv layer.
Revised: Conv layer.

Original: However, the current state of HCNN is far from being practical.
Revised: However, the current state of HCNN is far from being practical.

Original: Using the convolution algorithm of Lee et al.
Revised: Using the convolution algorithm of Lee et al.

Original: (2022a)
Revised: (2022a)

Original: and
Revised: and

Original: approximated ReLU, inference times of ResNet20 CIFARare 1653/201s using a single/64 threads in our CPU environment.
Revised: approximated ReLU, inference times of ResNet20 CIFARare 1662/174s using a single/64 threads in our CPU environment.

Original: Despite the unique advantages of
Revised: Despite the unique advantages of

Original: FHE-based PI, the huge runtime overhead prevents FHE from being the go-to solution for PI.
Revised: FHE-based PI, the huge runtime overhead prevents FHE from being the go-to solution for PI.

Original: We propose Hy brid P acking method and optimizations for H omomophic E ncryption-based neural
Revised: We propose Hy brid P acking method and optimizations for H omomophic E ncryption-based neural

Original: N etwork (HyPHEN), which mitigates the huge overhead of HCNN with an optimized convolution algorithm and packing method.
Revised: N etwork (HyPHEN), which mitigates the huge overhead of HCNN with an optimized convolution algorithm and packing method.

Original: We observe that after AEPSA is applied, rotation operations in
Revised: We observe that after AEPSA is applied, rotation operations in

Original: HCNN take up the majority of the runtime (See Appendix A) and most of the rotations (92-99%) are spent to implement the sum of channels within the same ciphertext and maintain data organization.
Revised: HCNN take up the majority of the runtime (See Appendix A) and most of the rotations (92-99%) are spent to implement the sum of channels within the same ciphertext and maintain data organization.

Original: We design a novel convolution algorithm named RAConv that does not require rotations to accumulate channels.
Revised: We design a novel convolution algorithm named RAConv that does not require rotations to accumulate channels.

Original: In addition, based on the observation that maintaining a single data organization necessitates massive unnecessary rotations, we design RAConv to take the new data organization based on the replication of the images.
Revised: In addition, based on the observation that maintaining a single data organization necessitates massive unnecessary rotations, we design RAConv to take the new data organization based on the replication of the images.

Original: By alternating between two data organizations, we remove rotations priorly required to adjust the data organization.
Revised: By alternating between two data organizations, we remove rotations priorly required to adjust the data organization.

Original: HyPHEN also includes a novel Hybrid
Revised: HyPHEN also includes a novel Hybrid

Original: Packing (HP) method that effectively handles the gap arising from strided convolution (Section 3.2).
Revised: Packing (HP) method that effectively handles the gap arising from strided convolution (Section 3.2).

Original: With additional optimizations, HyPHEN achieves 41.1s and 1.4s of execution time in ResNet20 for the CIFAR-10 dataset on CPU and GPU, respectively.
Revised: HyPHEN achieves 39.6s and 1.44s of runtime in ResNet20 for the CIFAR-10 dataset on CPU and GPU, respectively.

Original: The key contributions of the paper are as follows:
Revised: The key contributions of the paper are as follows:

Original: • We propose a replication-based convolution method, RAConv, that can effectively reduce two types of unnecessary rotations which are the major bottleneck in HCNN.
Revised: • We propose a replication-based convolution method, RAConv, that can effectively reduce two types of unnecessary rotations which are the major bottleneck in HCNN.

Original: • We propose a novel hybrid packing (HP) method that can utilize the entire slots of a ciphertext with a marginal increase in the number of rotations.
Revised: • We propose a novel hybrid packing (HP) method that can utilize the entire slots of a ciphertext with a marginal increase in the number of rotations.

Original: • Our experiments show that our HCNN implementation with HyPHEN improves the inference latency by 3.8-4.9 × over prior state-of-the-art HCNNs for ResNet on CIFAR-10.
Revised: • Our experiments show that our HCNN implementation with HyPHEN improves the inference latency by 3.4-4.4 × over prior state-of-the-art HCNNs for ResNet on CIFAR-10.

Original:  B ACKGROUND 2. F ULLY H OMOMORPHIC E NCRYPTION
Revised:  B ACKGROUND 2. F ULLY H OMOMORPHIC E NCRYPTION

Original: FHE is a set of public key encryption schemes that can perform computation on encrypted data.
Revised: FHE is a set of public key encryption schemes that can perform computation on encrypted data.

Original: Among several popular FHE schemes, RNS-CKKS (Cheon et al., 2018) has been broadly adopted in the PI domain as it supports fixed-point numbers and slot batching .
Revised: Among several popular FHE schemes, RNS-CKKS (Cheon et al., 2018) has been broadly adopted in the PI domain as it supports fixed-point numbers and slot batching .

Original: A plaintext in RNS-CKKS is an unencrypted degree- N polynomial in a cyclotomic polynomial ring, R Q = Z Q
Revised: A plaintext in RNS-CKKS is an unencrypted degree- N polynomial in a cyclotomic polynomial ring, R Q = Z Q

Original: [ X ] / ( X N +1) .
Revised: [ X ] / ( X N +1) .

Original: A plaintext maps to a message which is a vector of N/ 2 real (or complex) numbers.
Revised: A plaintext maps to a message which is a vector of N/ 2 real (or complex) numbers.

Original: Thus a single plaintext batches N/ 2 slots , which can store complex or real numbers.
Revised: Thus a single plaintext batches N/ 2 slots , which can store complex or real numbers.

Original: CKKS encrypts a plaintext into a ciphertext in R 2 Q .
Revised: CKKS encrypts a plaintext into a ciphertext in R 2 Q .

Original: Q is a ring modulus which is represented by a set of prime modulus obtained from the Chinese Remainder Theorem (CRT) as (cid:81) li =

Revised: Q is a ring modulus which is represented by a set of prime modulus obtained from the Chinese Remainder Theorem (CRT) as (cid:81) li =

Original: q i (1 ≤ l ≤ L ) .
Revised: q i (1 ≤ l ≤ L ) .

Original: L and l denote the initial and current level of a ciphertext.
Revised: L and l denote the initial and current level of a ciphertext.

Original: The level is an HE-specific resource that determines the number of multiplications applicable to a given ciphertext.
Revised: The level is an HE-specific resource that determines the number of multiplications applicable to a given ciphertext.

Original: We also denote the associated level of ring modulus using subscript as Q L or Q l .
Revised: We also denote the associated level of ring modulus using subscript as Q L or Q l .

Original: We denote the plaintext and ciphertext of a message a as 〈 a 〉 and [ a ].
Revised: We denote the plaintext and ciphertext of a message a as 〈 a 〉 and [ a ].

Original: HE operations of addition, multiplication, and rotation can be described as follows:
Revised: HE operations of addition, multiplication, and rotation can be described as follows:

Original: • HE.Eval([ a ],[ b ], f l ) =
Revised: • HE.Eval([ a ],[ b ], f l ) =

Original: HE.Eval([ a ], 〈 b 〉 , f l ) =
Revised: HE.Eval([ a ], 〈 b 〉 , f l ) =

Original: [ f l ( a , b )] • HE.Rotate([ a ],r) =
Revised: [ f l ( a , b )] • HE.Rotate([ a ],r) =

Original: [rot( a ,r)]
Revised: [rot( a ,r)]

Original: f l denotes linear operations, either Hadamard addition or multiplication.
Revised: f l denotes linear operations, either Hadamard addition or multiplication.

Original: rot( a ,r) represents cyclically shifting vector a by r to the left.
Revised: rot( a ,r) represents cyclically shifting vector a by r to the left.

Original: Unlike addition and rotation, multiplication in RNS-CKKS requires additional rescale operation, which consumes a level by dividing ct ∈ R Q l into
Revised: Unlike addition and rotation, multiplication in RNS-CKKS requires additional rescale operation, which consumes a level by dividing ct ∈ R Q l into

Original: ct ′ ∈
Revised: ct ′ ∈

Original: R Q l − 1 .
Revised: R Q l − 1 .

Original: If a ciphertext has no level left after a series of multiplications, bootstrapping (Bossuat et al., 2022) is needed to reconcile the levels and allow further operation.
Revised: If a ciphertext has no level left after a series of multiplications, bootstrapping (Bossuat et al., 2022) is needed to reconcile the levels and allow further operation.

Original: Bootstrapping, the most costly operation in HE, consists of multiple HE operations including rescale operations.
Revised: Bootstrapping, the most costly operation in HE, consists of multiple HE operations including rescale operations.

Original: After bootstrapping, the level of the resulting ciphertext becomes L ′ =(
Revised: After bootstrapping, the level of the resulting ciphertext becomes L ′ =(

Original: L − L b ) where L b is the depth of rescale operations in the bootstrapping circuit.
Revised: L − L b ) where L b is the depth of rescale operations in the bootstrapping circuit.

Original: As it is beneficial to perform many operations before bootstrapping, L should be sufficiently larger than L b .
Revised: As it is beneficial to perform many operations before bootstrapping, L should be sufficiently larger than L b .

Original: However, large L decreases the security level , which should be high enough to tolerate cryptographic attacks.
Revised: However, large L decreases the security level , which should be high enough to tolerate cryptographic attacks.

Original: The security level is roughly proportional to N/L .
Revised: The security level is roughly proportional to N/L .

Original: Considering the security requirement of HE, large L requires large N ( ≥ 2 15 ).
Revised: Considering the security requirement of HE, large L requires large N ( ≥ 2 15 ).

Original: Thus prior works on
Revised: Thus prior works on

Original: FHE (Bossuat et al., 2021; Jung et al., 2021; Lee et al., 2022b) target N = 2 

Revised: FHE (Bossuat et al., 2021; Jung et al., 2021; Lee et al., 2022b) target N = 2

Original: to 2 17 .
Revised: to 2 17 .

Original: Table 1 shows the execution time of HE operations on a system specified in Section 4.1.
Revised: Table 1 shows the execution time of HE operations on a system specified in Section 4.1.

Original: We measured the execution time of each operation at the initial level (max level) of a ciphertext and thus the execution time may decrease for ciphertexts with lower levels.
Revised: We measured the execution time of each operation at the initial level (max level) of a ciphertext and thus the execution time may decrease for ciphertexts with lower levels.

Original: Bootstrapping consumes over two orders of magnitude longer runtime than other operations, but boostrapping does not occur as frequently as others.
Revised: Bootstrapping consumes over two orders of magnitude longer runtime than other operations, but boostrapping does not occur as frequently as others.

Original: Except for bootstrapping, Rotate and MulCt are the most time-consuming operations in HE, which is due to the expensive key-switching procedure.
Revised: Except for bootstrapping, Rotate and MulCt are the most time-consuming operations in HE, which is due to the expensive key-switching procedure.

Original: [Table]
Revised: [Table]

Original: 2. C ONVOLUTION ON H OMOMORPHIC E NCRYPTION
Revised: 2. C ONVOLUTION ON H OMOMORPHIC E NCRYPTION

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: In this subsection, we describe previous convolution algorithms for FHE.
Revised: In this subsection, we describe previous convolution algorithms for FHE.

Original: We represent input and output tensors with tuples { w i , h i , c i } and { w o , h o , c o } , and Conv layers with { f w , f h ,
Revised: We represent input and output tensors with tuples { w i , h i , c i } and { w o , h o , c o } , and Conv layers with { f w , f h ,

Original: c i , c o } .
Revised: c i , c o } .

Original: We denote the stride of the convolution as s and assume padding = 1 for simplicity.
Revised: We denote the stride of the convolution as s and assume padding = 1 for simplicity.

Original: Gazelle (Juvekar et al., 2018) proposed efficient SISO (single-input and single-output channel) convolution algorithms on HE.
Revised: Gazelle (Juvekar et al., 2018) proposed efficient SISO (single-input and single-output channel) convolution algorithms on HE.

Original: Figures 1b, 1c show SISO convolutions for s = 1 , 2 .
Revised: Figures 1b, 1c show SISO convolutions for s = 1 , 2 .

Original: Filter elements are separated into f w f h plaintexts.
Revised: Filter elements are separated into f w f h plaintexts.

Original: Each slot in i -th plaintext stores k i or 0 (punctured) depending on whether k i is involved in the computation of the output pixel at the same slot.
Revised: Each slot in i -th plaintext stores k i or 0 (punctured) depending on whether k i is involved in the computation of the output pixel at the same slot.

Original: SISO operates as follows: 1) rotate an encrypted input image with different indexes according to plaintext filter, 2) perform Hadamard multiplication, and 3) accumulate the multiplication results to obtain the output.
Revised: SISO operates as follows: 1) rotate an encrypted input image with different indexes according to plaintext filter, 2) perform Hadamard multiplication, and 3) accumulate the multiplication results to obtain the output.

Original: Alternatively, if we prepare the filter plaintexts inversely rotated and directly multiply them with the input ciphertext, the rotations can be performed after MulPt operations (Zhang et al., 2021), which we dub lazy-SISO .
Revised: Alternatively, if we prepare the filter plaintexts inversely rotated and directly multiply them with the input ciphertext, the rotations can be performed after MulPt operations (Zhang et al., 2021), which we dub lazy-SISO .

Original: We selectively use SISO and lazy-SISO to minimize rotations for convolution.
Revised: We selectively use SISO and lazy-SISO to minimize rotations for convolution.

Revised: Detailed explanation of lazy-SISO is provided in Appendix E

Original: In more general cases of multiple channels, convolution on HE is performed in a SIMD manner.
Revised: In more general cases of multiple channels, convolution on HE is performed in a SIMD manner.

Original: If the size of a channel image is smaller than the number of slots in a ciphertext, multiple channels can be placed in a ciphertext.
Revised: If the size of a channel image is smaller than the number of slots in a ciphertext, multiple channels can be placed in a ciphertext.

Original: For example, if slot = 2 15 and an input channel is sized w i h i = 32 × 32 as in the input image of the CIFAR-10 dataset, slotw i h i =
Revised: For example, if slot = 2 15 and an input channel is sized w i h i = 32 × 32 as in the input image of the CIFAR-10 dataset, slotw i h i =

Original: 32 channels can be placed in a single ciphertext in an aligned manner (i.e. channel-aligned ).
Revised: 32 channels can be placed in a single ciphertext in an aligned manner (i.e. channel-aligned ).

Original: Then, the process of convolution for a channel-aligned input ciphertext storing multiple channels can be described as follows.
Revised: Then, the process of convolution for a channel-aligned input ciphertext storing multiple channels can be described as follows.

Original: Suppose c i = slotw
Revised: Suppose c i = slotw

Original: i h i .
Revised: i h i .

Original: First,
Revised: First,

Original: SISO is performed on c i input channels in a SIMD manner (see Figure 2c), which produces c i c o convolution outputs MK
Revised: SISO is performed on c i input channels in a SIMD manner (see Figure 2c), which produces c i c o convolution outputs MK

Original: ( i,j ) ( 1 ≤ i ≤ c i , 1 ≤ j ≤ c o ).
Revised: ( i,j ) ( 1 ≤ i ≤ c i , 1 ≤ j ≤ c o ).

Original: To compute the result for the k -th output channel, (cid:80) c i i
Revised: To compute the result for the k -th output channel, (cid:80) c i i

Original: =1 MK ( i,k ) is accumulated by RaS (Rotate and Sum) , which is repeated until all the output channels are acquired.
Revised: =1 MK ( i,k ) is accumulated by RaS (Rotate and Sum) , which is repeated until all the output channels are acquired.

Original: Finally, the ciphertexts packed with output channels are realigned to match the next layer’s input alignment by IR (Image Realign) .
Revised: Finally, the ciphertexts packed with output channels are realigned to match the next layer’s input alignment by IR (Image Realign) .

Original: Throughout this paper, we refer to this convolution that takes a channel-aligned ciphertext as the input as channel-aligned convolution (CAConv) .
Revised: Throughout this paper, we refer to this convolution that takes a channel-aligned ciphertext as the input as channel-aligned convolution (CAConv) .

Original: CAConv can be further optimized for the case where the input tensor is not large enough to fill all the slots in a ciphertext; slotw i h
Revised: CAConv can be further optimized for the case where the input tensor is not large enough to fill all the slots in a ciphertext; slot w i h

Original: i c i repeated copies of the input tensor are placed in a ciphertext, then slotw i
Revised: i c i repeated copies of the input tensor are placed in a ciphertext, then slotw i

Original: h
Revised: h

Original: i c i output channels can be computed together in a single ciphertext (Lee et al., 2022a).
Revised: i c i output channels can be computed together in a single ciphertext (Lee et al., 2022a).

Original: Strided convolution ( s > 1 ) using SISO algorithm creates a gap between valid values (see Figure 1c).
Revised: Strided convolution ( s > 1 ) using SISO algorithm creates a gap between valid values (see Figure 1c).

Original: A ciphertext with a gap underutilizes its slots, leading to throughput degradation.
Revised: A ciphertext with a gap underutilizes its slots, leading to throughput degradation.

Original: While
Revised: While

Original: Juvekar et al.
Revised: Juvekar et al.

Original: (2018) can remove the gap by a client-aided re-encryption process, non-interactive PI shall remove the gap through masking and rotation, which incur additional rotation overhead and also consume more levels.
Revised: (2018) can remove the gap by a client-aided re-encryption process, non-interactive PI shall remove the gap through masking and rotation, which incur additional rotation overhead and also consume more levels.

Original: Lee et al.
Revised: Lee et al.

Original: (2022a) proposed a multiplexed packing method that can be combined with CAConv ( MP-CAConv ) to mitigate the overheads.
Revised: (2022a) proposed a multiplexed packing method that can be combined with CAConv ( MP-CAConv ) to mitigate the overheads.

Original: In the IR stage of MP-CAConv, multiplexed packing fills the gap with other channels (see Figure 3b), which we refer to as the repacking process .
Revised: In the IR stage of MP-CAConv, multiplexed packing fills the gap with other channels (see Figure 3b), which we refer to as the repacking process .

Original: Other than the repacking process, MP-CAConv is very similar to CAConv.
Revised: Other than the repacking process, MP-CAConv is very similar to CAConv.

Original: In
Revised: In

Original: MP-CAConv, IR collectively refers to realigning and repackaging process.
Revised: MP-CAConv, IR collectively refers to realigning and repacking process.

Revised: Tile tensoring based convolution proposed in (Aharoni et al., 2020) is a yet another convolution algorithm for FHE.

Revised: While tile tensoring based convolution can be efficient alternative to SISObased convolution when the image size is sufficiently high, our paper mainly focuses on SISO-based convolution, which can be applied more broadly.

Original: In RNS-CKKS, the hoisting optimization allows multiple rotations to share common sub-operations composing a rotation operation.
Revised: In RNS-CKKS, the hoisting optimization allows multiple rotations to share common sub-operations composing a rotation operation.

Original: First-level hoisting shares the front decomposition sub-operations when rotating a single ciphertext multiple times with different indexes.
Revised: First-level hoisting shares the front decomposition sub-operations when rotating a single ciphertext multiple times with different indexes.

Original: Second-level hoisting shares the rear ModDown sub-operations when rotating and summing up multiple ciphertexts.
Revised: Second-level hoisting shares the rear ModDown sub-operations when rotating and summing up multiple ciphertexts.

Original: We refer the reader to Bossuat et al.
Revised: We refer the reader to Bossuat et al.

Original: (2021) for further details of the hoisting optimization.
Revised: (2021) for further details of the hoisting optimization.

Original: First-level hoisting can be applied to SISO, whereas second-level hoisting can be applied to lazy-SISO.
Revised: First-level hoisting can be applied to SISO, whereas second-level hoisting can be applied to lazy-SISO.

Original: 2. P OLYNOMIAL A CTIVATION F UNCTION ON H OMOMORPHIC E NCRYPTION
Revised: 2. P OLYNOMIAL A CTIVATION F UNCTION ON H OMOMORPHIC E NCRYPTION

Original: Non-linear activation functions, such as ReLU, cannot be used in HCNN.
Revised: Non-linear activation functions, such as ReLU, cannot be used in HCNN.

Original: They must be replaced with polynomial functions as HE only supports addition and multiplication operations.
Revised: They must be replaced with polynomial functions as HE only supports addition and multiplication operations.

Original: To directly replace ReLU with approximate polynomials, approximation error should be negligible over a wide range to retain the original accuracy of a CNN model.
Revised: To directly replace ReLU with approximate polynomials, approximation error should be negligible over a wide range to retain the original accuracy of a CNN model.

Original: Lee et al.
Revised: Lee et al.

Original: (2021) designed a precise approximation of ReLU having an l 1 norm approximation error less than 2 − 13 in the range of [-50, 50], obtained by a composition of { 15, 15, 27 } degree polynomials.
Revised: (2021) designed a precise approximation of ReLU having an l 1 norm approximation error less than 2 − 13 in the range of [-50, 50], obtained by a composition of { 15, 15, 27 } degree polynomials.

Original: This approximation-based approach has a benefit that it can be applied to pretrained neural networks.
Revised: This approximation-based approach has a benefit that it can be applied to pretrained neural networks.

Original: However, evaluation of highdegree polynomials imposes a significant runtime overhead in HCNN inference.
Revised: However, evaluation of highdegree polynomials imposes a significant runtime overhead in HCNN inference.

Original: Another approach is to train neural networks with low-degree polynomial activations as in Ishiyama et al.
Revised: Another approach is to train neural networks with low-degree polynomial activations as in Ishiyama et al.

Original: (2020); Chabanne et al.
Revised: (2020); Chabanne et al.

Original: ; Obla et al.
Revised: ; Obla et al.

Original: ; Hesamifard et al.
Revised: ; Hesamifard et al.

Original: ; Thaine et al.
Revised: ; Thaine et al.

Original: While this approach requires retraining, operational cost is much cheaper compared to high-degree polynomials.
Revised: While this approach requires retraining, operational cost is much cheaper compared to high-degree polynomials.

Original: Recently AESPA (Park et al., 2022) demonstrated that CNN trained with low-degree polynomials can achieve equivalent accuracy to ReLU-based networks across various CNN networks and image datasets.
Revised: Recently AESPA (Park et al., 2022) demonstrated that CNN trained with low-degree polynomials can achieve equivalent accuracy to ReLU-based networks across various CNN networks and image datasets.

Original: AESPA replaces batch normalization (BN) and ReLU with the composition of orthogonal basis polynomials and basis-wise BN as follows:
Revised: AESPA replaces batch normalization (BN) and ReLU with the composition of orthogonal basis polynomials and basis-wise BN as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, h i are the orthogonal bases, σ and µ are the mean and variance computed from BN, and γ and β are trainable parameters.
Revised: Here, h i are the orthogonal bases, σ and µ are the mean and variance computed from BN, and γ and β are trainable parameters.

Original: For d = 2 , AESPA turns into a second-degree polynomial with different coefficients for each channel on inference.
Revised: For d = 2 , AESPA turns into a second-degree polynomial with different coefficients for each channel on inference.

Original: We adopt AESPA in this paper, which leads to better runtime performance.
Revised: We adopt AESPA in this paper, which leads to better runtime performance.

Original: 2. T HREAT M ODEL
Revised: 2. T HREAT M ODEL

Original: We adopt the same threat model as prior PI works.
Revised: We adopt the same threat model as prior PI works.

Original: A client sends encrypted data to an untrusted server.
Revised: A client sends encrypted data to an untrusted server.

Original: The server performs CNN inference using HE operations and returns inference results to the client.
Revised: The server performs CNN inference using HE operations and returns inference results to the client.

Original: The client decrypts the resulting ciphertext to obtain the private result.
Revised: The client decrypts the resulting ciphertext to obtain the private result.

Original: The server only holds the client’s public keys and cannot decrypt any intermediate ciphertexts in the inference process.
Revised: The server only holds the client’s public keys and cannot decrypt any intermediate ciphertexts in the inference process.

Original: The client does not know any information about the processing at the server other than the result.
Revised: The client does not know any information about the processing at the server other than the result.

Original:  M ETHOD 3. R EPLICATION -A LIGNED C ONVOLUTION
Revised:  M ETHOD 3. R EPLICATION -A LIGNED C ONVOLUTION

Original: The main performance bottleneck of CAConv is the massive number of rotations.
Revised: The main performance bottleneck of CAConv is the massive number of rotations.

Original: CAConv requires an enormous number of rotations to implement the sum of the channels within the same ciphertext (RaS) and the relocation of the channels (IR) to match the next layer’s input representation.
Revised: CAConv requires an enormous number of rotations to implement the sum of the channels within the same ciphertext (RaS) and the relocation of the channels (IR) to match the next layer’s input representation.

Original: These rotations take up most of the time in CAConv.
Revised: These rotations take up most of the time in CAConv.

Original: For example, when N = 2 16 , rotations for RaS and IR account for 49% and 43% of the total number of rotations in ResNet20, respectively.
Revised: For example, when N = 2 16 , rotations for RaS and IR account for 49% and 43% of the total number of rotations in ResNet20, respectively.

Original: Furthermore,
Revised: Furthermore,

Original: IR consumes an additional level for masking to extract the values.M
Revised: IR consumes an additional level for masking to extract the values.

Original: [Equation]
Revised: [Figure]

Original:  M [Equation]
Revised: [Figure] M (2) M (

Original: ) c o c i ∗
Revised: ) c o c i ∗

Original: [Equation] [Equation] [Equation] o c i f w f h
Revised: K (

Original: [Equation]
Revised: [Table]

Original: ) MK (2 , 1) MK ([Equation] (cid:80) −→ N (1) N (2) (a) RAConv with SISO M (1) M (1) M (2) M (2)
Revised: 1) K (

Original: c o c i ∗
Revised: [Table] ) K [Table]

Original: K (1 , 1) K (1 , 2)
Revised: [Table] N (2) [Figure] M (1) M (1)

Original:  K (2 , 1) K (2 , 2) c
Revised: M (2) M (2) c o c i ∗ K (1 , 1) K (1 , 2) K (2 , 1) K (2 , 2) c

Original: o c i f
Revised: o c i f

Original: w f h
Revised: w f h

Original: SISO l −−−−→ MK (1 , 1) MK (1 , 2) MK (1 , 1) MK (1 , 2) MK (1 , 1) MK (1 , 2)RaS −−−→ N (1) N (2) (
Revised: SISO l −−−−→ MK (1 , 1) MK (1 , 2) MK (1 , 1) MK (1 , 2) MK (1 , 1) MK (1 , 2)

Original: [Figure]
Revised: [Figure]

Original: lazy
Revised: −−−→

Original: [Figure]
Revised: [Figure]

Original: SISO
Revised: (1) N (2) [Figure]

Original: M (1) M (2) c i ∗
Revised: M (1) M (2) c i ∗

Original: K (1 , 1) K (2 , 1)
Revised: K ( [Figure] [Figure]

Original:  K (1 , 2) K (2 , 2) c i c o f w f h
Revised: [Figure] K (1 , 2) K (2 , 2) c i c o f w f h

Original: SISO −−−−→ MK (1 , 1) MK (2 , 1)
Revised: [Figure] −−−−→ MK (1 , 1) MK (2 , 1)

Original: MK (1 , 2) MK (2 , 2) rotateandadd
Revised: MK (1 , 2) MK (2 , 2) rotateandadd

Original: RaS −−−→ N (1) N (1)
Revised: RaS −−−→ N (1) N (1)

Original: N (2) N (2)
Revised: N (2) N (2)

Original: [Figure]
Revised: [Figure]

Original: To mitigate the performance bottleneck caused by rotations in CAConv, we propose ReplicationAligned Convolution (RAConv) .
Revised: To mitigate the performance bottleneck caused by rotations in CAConv, we design ReplicationAligned Convolution (RAConv) to receive the alternative data representation.

Original: In CAConv, the output ciphertext of RaS contains replications of the channel sum (i.e. replication-aligned ) as shown in Figure 2c.
Revised: In CAConv, the output ciphertext of RaS contains replications of the channel sum (i.e. replication-aligned ) as shown in Figure 2c.

Original: RAConv receives replicationaligned ciphertexts as input and skips IR and RaS stages.
Revised: RAConv receives replication-aligned ciphertexts as input and skips IR and RaS stages.

Original: Figure 2a shows an example of performing RAConv.
Revised: Figure 2a shows an example of performing RAConv.

Original: RAConv takes c i input ciphertexts each filled with replications of a single input channel and weight plaintexts aligned in output channel order.
Revised: RAConv takes c i input ciphertexts each filled with replications of a single input channel and weight plaintexts aligned in output channel order.

Original: RAConv operates as follows: 1) perform parallel SISO, which outputs c i ciphertexts where the i -th ciphertext contains MK ( i,j ) for all j values, and 2) accumulate the ciphertexts by simple HE additions.
Revised: RAConv operates as follows: 1) perform parallel SISO, which outputs c i ciphertexts where the i -th ciphertext contains MK ( i,j ) for all j values, and 2) accumulate the ciphertexts by simple HE additions.

Original: SISO in RAConv actually increases the number of rotations because c i input ciphertexts require c
Revised: SISO in RAConv actually increases the number of rotations because c i input ciphertexts require c

Original: i ( f w f h − 1) rotations during parallel SISO.
Revised: i ( f w f h − 1) rotations during parallel SISO.

Original: Instead, we utilize lazy-SISO with RAConv, which requires much fewer ( f w f h − 1) rotations for SISO.
Revised: Instead, we utilize lazy-SISO with RAConv, which requires much fewer ( f w f h − 1) rotations for SISO.

Original: RAConv produces a densely-packed channel-aligned ciphertext that complies with the CAConv input data organization, so we alternate between RAConv and CAConv.
Revised: RAConv produces a densely-packed channel-aligned ciphertext that complies with the CAConv input data organization, so we alternate between RAConv and CAConv.

Original: RAConv-CAConv chain halves the RaS rotations previously required in two CAConvs to c i log ( c o ) .
Revised: RAConv-CAConv chain halves the RaS rotations previously required in two CAConvs to c i log ( c o ) .

Original: 3. H YBRID P ACKING

Original: SISO convolution suffers from low slot utilization in ciphertexts due to two reasons.

Original: First, strided convolution creates a gap between valid values (see Figure 3a).

Original: Second, the small size of input tensors creates empty slots in ciphertexts.

Original: Due to the security requirement of FHE, the number of

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: ) c (2) ## ## c (3) c (4) ## ##
Revised: ) c (2) ## ## c (3) c (4) ## ##

Original: Mask and Broadcast
Revised: Mask and Broadcast

Original: Broadcast −−−−−−−→ c (1) c (2) c (1) c (2) c (3) c (4) c (3) c (4) (a) CAConv with hybrid packing (HP-CAConv) a (1)1 a (2)1 a (1)1 a (2) a (3)1 a (4)1 a (3)1 a (4)
Revised: Broadcast −−−−−−−→ c (1) c (2) c (1) c (2) c (3) c (4) c (3) c (4) (a) CAConv with hybrid packing (HP-CAConv) a (1)1 a (2)1 a (1)1 a (2) a (3)1 a (4)1 a (3)1 a (4)

Original: RAConv −−−−−−→ b (1 − 8 , 1) b (9 − 16 , 1) b (1 − 8 , 17) b (9 − 16 , 17) b (17 − 24 , 1) b (25 − 32 , 1) b (17 − 24 , 17) b (25 − 32 , 17)
Revised: RAConv −−−−−−→ b (1 − 8 , 1) b (9 − 16 , 1) b (1 − 8 , 17) b (9 − 16 , 17) b (17 − 24 , 1) b (25 − 32 , 1) b (17 − 24 , 17) b (25 − 32 , 17)

Original: RaS
Revised: RaS

Original: RaS −−−→ c (1) ## c (17) ## ## ## ## ##
Revised: RaS −−−→ c (1) ## c (17) ## ## ## ## ##

Original: Mask and Broadcast
Revised: Mask and Broadcast

Original: Broadcast −−−−−−−→ c (1) c (1) c (17) c (17) c (1) c (1) c (17) c (17) (b) RAConv with hybrid packing (HP-RAConv) slots in a ciphertext is often larger than the size of an input tensor.
Revised: Broadcast −−−−−−−→ c (1) c (1) c (17) c (17) c (1) c (1) c (17) c (17) (b) RAConv with hybrid packing (HP-RAConv) 3. H YBRID P ACKING SISO convolution suffers from low slot utilization in ciphertexts due to two reasons.

Revised: First, strided convolution creates a gap between valid values (see Figure 3a).

Revised: Second, the small size of input tensors creates empty slots in ciphertexts.

Revised: Due to the security requirement of FHE, the number of slots in a ciphertext is often larger than the size of an input tensor.

Original: Underutilization of slots in SISO leads to severe throughput degradation in HCNN.
Revised: Underutilization of slots in SISO leads to severe throughput degradation in HCNN.

Original: Prior state-of-the-art HCNN implementation (MP-CAConv) mitigates underutilization of slots using multiplexed packing and input repetition (see Section 2.2).
Revised: Prior state-of-the-art HCNN implementation (MP-CAConv) mitigates underutilization of slots using multiplexed packing and input repetition (see Section 2.2).

Original: However, they cause a lot of additional rotation overhead to adjust the data organization.
Revised: However, they cause a lot of additional rotation overhead to adjust the data organization.

Original: In MP-CAConv, RaS operation is used to accumulate SISO results in a ciphertext.
Revised: In MP-CAConv, RaS operation is used to accumulate SISO results in a ciphertext.

Original: Under input repetition, RaS operation returns an output channel at the slots where an input ciphertext stores the first channel of an input tensor.
Revised: Under input repetition, RaS operation returns an output channel at the slots where an input ciphertext stores the first channel of an input tensor.

Original: Multiplexed channels are also accumulated through RaS operation, thus values constituting an output channel exist only in non-gap slots, as shown in Figure 3a.
Revised: Multiplexed channels are also accumulated through RaS operation, thus values constituting an output channel exist only in non-gap slots, as shown in Figure 3a.

Original: To restore the data organization of MP-CAConv, invalid values are masked off and empty slots are filled with other channels through IR process.
Revised: To restore the data organization of MP-CAConv, invalid values are masked off and empty slots are filled with other channels through IR process.

Original: In the presence of input repetition, IR spends O ( c o ) rotations to relocate output channels and slotw
Revised: In the presence of input repetition, IR spends O ( c o ) rotations to relocate output channels and slotw

Original: o h
Revised: o h

Original: o c
Revised: o c

Original: o additional rotations for generating input repetition.
Revised: o additional rotations for generating input repetition.

Original: To reduce the relocating overhead between convolutional layers, we propose a novel image packing method, hybrid packing (HP) .
Revised: To reduce the relocating overhead between convolutional layers, we propose a novel image packing method, hybrid packing (HP) .

Original: HP fills the gap with channel duplicates of multiple channels (See Figure 3d).
Revised: HP fills the gap with channel duplicates of multiple channels (See Figure 3d).

Original: We design HP based on two key observations.
Revised: We design HP based on two key observations.

Original: First, applying convolution over a duplicate-packed ciphertext (Figure 3c) produces a multiplex-packed output ciphertext as in Figure 3b.
Revised: First, applying convolution over a duplicate-packed ciphertext (Figure 3c) produces a multiplex-packed output ciphertext as in Figure 3b.

Original: Second, converting a void-packed ciphertext into a duplicate-packed ciphertext requires fewer rotations than converting it into a multiplex-packed ciphertext.
Revised: Second, converting a void-packed ciphertext into a duplicate-packed ciphertext requires fewer rotations than converting it into a multiplex-packed ciphertext.

Original: Duplicate packing only needs
Revised: Duplicate packing only needs

Original: O(log( gap size )) rotations while multiplexed packing requires O( gap size ) rotations.
Revised: O(log( gap size )) rotations while multiplexed packing requires O( gap size ) rotations.

Original: HP is a hybrid of duplicate packing and multiplexed packing.
Revised: HP is a hybrid of duplicate packing and multiplexed packing.

Original: We represent a hybrid-packed ciphertext by a pair of numbers, the number of multiplexed channels m , and the number of duplicates d .
Revised: We represent a hybrid-packed ciphertext by a pair of numbers, the number of multiplexed channels m , and the number of duplicates d .

Original: For example, Figure 3d shows ( m, d ) =
Revised: For example, Figure 3d shows ( m, d ) =

Original: (2 , 2) HP.
Revised: (2 , 2) HP.

Original: Packing of the ciphertext switches between two
Revised: Packing of the ciphertext switches between two

Original: HP settings while processing CAConv and RAConv as shown in Figure 4.
Revised: HP settings while processing CAConv and RAConv as shown in Figure 4.

Original: We denote HP parameter ( m, d ) of the input and output ciphertext as ( m in , d in ) and ( m out , d out ).
Revised: We denote HP parameter ( m, d ) of the input and output ciphertext as ( m in , d in ) and ( m out , d out ).

Original: Input repetition is no longer required as HP with larger d in can be used instead.
Revised: Input repetition is no longer required as HP with larger d in can be used instead.

Original: Duplicates of HP produce different output channels within the gap (See c (1) and c (17) in Figure 4b)
Revised: Duplicates of HP produce different output channels within the gap (See c (1) and c (17) in Figure 4b)

Original: Then, the IR process adjusts the output ciphertext’s organization from ( m out , d out ) =
Revised: Then, the IR process adjusts the output ciphertext’s organization from ( m out , d out ) =

Original: ( d in , 1) to ( d in , m in ) , which only requires O (log m in ) rotations, which fill the gaps.
Revised: ( d in , 1) to ( d in , m in ) , which only requires O (log m in ) Method ct in ct out SISO

Original: After performing a series of CAConv and RAConv, the HP organization of the output ciphertext returns to the initial ( m i n, d i n ) .
Revised: RaS IR (Lee et al., 2022a) ⌈ w i h i c i n ⌉ ⌈ w o h o c o n ⌉ ct in ( f w f h − 1)

Revised: w i h

Revised: i c i c

Revised: o n log c

Revised: i c o + log n w o h

Revised: o c

Revised: o

Revised: HP-CAConv w

Revised: i

Revised: h

Revised: i c i d in n c o d in ct min ( f w f h − 1) ct out log c

Revised: i ct in ct out log m in

Revised: HP-

Revised: [Table]

Original: The complete procedures of RAConv and CAConv with HP are described more in detail in Appendix C.
Revised: c i m in w i h i c o m in n ct min ( f [Table]

Original: Compared to MP-CAConv, HP significantly reduces the rotations in RaS and IR.
Revised: [Table]

Original: The rotation complexities of MP-CAConv (Lee et al., 2022a) and our hybrid-packed convolutions are shown in Ta-
Revised: in ct

Original: [Table]
Revised: [Table]

Revised: in rotations, which fill the gaps.

Original: ble 2.
Revised: After performing a series of CAConv and RAConv, the HP organization of the output ciphertext returns to the initial ( m i n, d i n ) . The complete procedures of RAConv and CAConv with HP are described more in detail in Appendix H. Compared to MP-CAConv, HP significantly reduces the rotations in RaS and IR. The rotation complexities of MP-CAConv (Lee et al., 2022a) and our hybrid-packed convolutions are shown in Table 2.

Original: For both HP convolution methods, the product of the numbers of input and output ciphertexts remains constant ( ct in · ct out =
Revised: For both HP convolution methods, the product of the numbers of input and output ciphertexts remains constant ( ct in · ct out =

Original: w i h
Revised: w i h

Original: i c
Revised: i c

Original: i c o n ).
Revised: i c o n ).

Original: Compared to MP-CAConv, the number of rotations for
Revised: Compared to MP-CAConv, the number of rotations for

Original: RaS is reduced by about ct in times for both HP-CAConv and HP-RAConv.
Revised: RaS is reduced by about ct in times for both HP-CAConv and HP-RAConv.

Original: IR stage of HP repacks the gap with duplicates.
Revised: IR stage of HP repacks the gap with duplicates.

Original: Rotation decreases from c o of MP-CAConv to mere log m in rotations per output ciphertext.
Revised: Rotation decreases from c o of MP-CAConv to mere log m in rotations per output ciphertext.

Original: HP convolutions require more rotations than MP-CAConv for SISO; ct in of MP-CAConv is always smaller than or equal to ct min of HP-CAConv or HP-RAConv.
Revised: HP convolutions require more rotations than MP-CAConv for SISO; ct in of MP-CAConv is always smaller than or equal to ct min of HP-CAConv or HP-RAConv.

Original: Nevertheless, hoisting can be applied to SISO which reduces the significance of the rotation cost of SISO, and the reduction of rotations in RaS and IR overwhelms the increase in SISO.
Revised: Nevertheless, hoisting can be applied to SISO which reduces the significance of the rotation cost of SISO, and the reduction of rotations in RaS and IR overwhelms the increase in SISO.

Original: All things considered, HP reduces the overall number of rotations required for convolutions.
Revised: All things considered, HP reduces the overall number of rotations required for convolutions.

Original: We can also explore various combinations for the ( m, d ) pair to minimize the total number of rotations.
Revised: We can also explore various combinations for the ( m, d ) pair to minimize the total number of rotations.

Original: The choice of ( m, d ) decides ct in and ct out values and creates a trade-off between SISO, RaS, and IR costs, and also affects the number of ciphertexts we have to perform bootstrapping with.
Revised: The choice of ( m, d ) decides ct in and ct out values and creates a trade-off between SISO, RaS, and IR costs, and also affects the number of ciphertexts we have to perform bootstrapping with.

Original: We provide an in-depth performance analysis with regard to the choice of ( m, d ) in Section 4.2.
Revised: We provide an in-depth performance analysis with regard to the choice of ( m, d ) in Section 4.2.

Original: 3. T HE R ES N ET A RCHITECTURE ON H Y PHEN
Revised: 3. T HE R ES N ET A RCHITECTURE ON H Y PHEN

Original: [Figure]
Revised: [Figure]

Original: HyPHEN combines RAConv and HP to build the entire CNN model.
Revised: HyPHEN combines RAConv and HP to build the entire CNN model.

Original: Figure 5 shows the basic block of ResNet implemented on HyPHEN.
Revised: Figure 5 shows the basic block of ResNet implemented on HyPHEN.

Original: There are three more considerations when deciding the order of operations.
Revised: There are three more considerations when deciding the order of operations.

Original: First, bootstrapping is cheaper when placed after RAConv, and not CAConv, because the number of ciphertexts is smaller at the moment.
Revised: First, bootstrapping is cheaper when placed after RAConv, and not CAConv, because the number of ciphertexts is smaller at the moment.

Original: Second, to match the level between the shortcut path and the main CAConv-RAConv path, bootstrapping should be placed either before residual connections diverge or after they converge.
Revised: Second, to match the level between the shortcut path and the main CAConv-RAConv path, bootstrapping should be placed either before residual connections diverge or after they converge.

Original: Last, it is beneficial to perform convolutional layers at the lowest level possible.
Revised: Last, it is beneficial to perform convolutional layers at the lowest level possible.

Original: The complexity of FHE operations such as rotation is proportional to the level l of ciphertext.
Revised: The complexity of FHE operations such as rotation is proportional to the level l of ciphertext.

Original: Therefore, the lower the calculated level, the smaller the computational cost.
Revised: Therefore, the lower the calculated level, the smaller the computational cost.

Original: All things put together, our ResNet basic block implementation consumes a total of 6 levels.
Revised: All things put together, our ResNet basic block implementation consumes a total of 6 levels.

Original: The level consumption of each layer is represented in the parenthesis of each block.
Revised: The level consumption of each layer is represented in the parenthesis of each block.

Original: CAConv and
Revised: CAConv and

Original: RAConv use HP and consume one level for each SISO and IR.
Revised: RAConv use HP and consume one level for each SISO and IR.

Original: Activation uses AESPA and consumes one level.
Revised: Activation uses AESPA and consumes one level.

Original: AESPA is a quadratic polynomial having different coefficients for each channel while training.
Revised: AESPA is a quadratic polynomial having different coefficients for each channel while training.

Original: During inference, we fuse the coefficients into nearby layers, then the activation becomes a simple square function x 2 .
Revised: During inference, we fuse the coefficients into nearby layers, then the activation becomes a simple square function x 2 .

Original: We set the initial ciphertext level L to six and perform bootstrapping when the level becomes zero.
Revised: We set the initial ciphertext level L to six and perform bootstrapping when the level becomes zero.

Original:  E VALUATION 4. E XPERIMENTAL S ETUP
Revised:  E VALUATION 4. E XPERIMENTAL S ETUP

Original: We ran HCNN inference on CPU and GPU environments using the RNS-CKKS library, HEAAN.
Revised: We ran HCNN inference on CPU and GPU environments using the RNS-CKKS library, HEAAN.

Original: CPU instance is equipped with AMD EPYC 7452 running at 2.35GHz (64 cores) and 480GB
Revised: CPU instance is equipped with AMD EPYC 7452 running at 2.35GHz (64 cores) and 480GB

Original: DRAM.
Revised: DRAM.

Original: GPU experiments are conducted at the same system with an additional NVIDIA A100 GPU with 80GB HBM.
Revised: GPU experiments are conducted at the same system with an additional NVIDIA A100 GPU with 80GB HBM.

Original: Our HCNN inference experiments use the CIFAR-10dataset (Krizhevsky et al., 2009), consisting of 50,000 training and 10,000 test images.
Revised: Our HCNN inference experiments use the CIFAR-10 (Krizhevsky et al., 2009).

Original: We evaluate ResNet20/32/44/18 trained with AESPA on PyTorch and applied the fusion technique to all the networks.
Revised: We evaluate ResNet20/32/44/18 trained with AESPA on PyTorch and applied the fusion technique to all the networks.

Original: Our RNS-CKKS parameters satisfy 128-bit security level (Cheon et al., 2019) with polynomial degree N = 2 16 and hamming weight 192.
Revised: Our RNS-CKKS parameters satisfy 128-bit security level (Cheon et al., 2019) with polynomial degree N = 2 16 and hamming weight 192.

Original: Multiplication and bootstrapping primes each occupy 48 bits and over 56 bits, respectively.
Revised: Multiplication and bootstrapping primes each occupy 48 bits and over 56 bits, respectively.

Original: The bootstrapping implementation consumes 17 levels in our setup.
Revised: The bootstrapping implementation consumeslevels in our setup.

Original: 4. O PTIMAL P OINT
Revised: 4. O PTIMAL P OINT

Original: We explore the optimal ( m, d ) pair for HP to minimize the latency of ResNet 20/18.
Revised: We explore the optimal ( m, d ) pair for HP to minimize the latency of ResNet 20/18.

Original: Table 3 shows the operation count of rotation and bootstrapping, which are major contributors to the runtime.
Revised: Table 3 shows the operation count of rotation and bootstrapping, which are major contributors to the runtime.

Original: Rotation is categorized into SISO and non-SISO.
Revised: Rotation is categorized into SISO and non-SISO.

Original: Only SISO rotations can be optimized with hoisting.
Revised: Only SISO rotations can be optimized with hoisting.

Original: Table 3 presents an instance of MP-CAConv and three (or two) instances of our HP-based convolution.
Revised: Table 3 presents an instance of MP-CAConv and three (or two) instances of our HP-based convolution.

Original: Instances of HP-based convolution consist of settings with minimal bootstrapping, minimal rotations, and optimal latency.
Revised: Instances of HP-based convolution consist of settings with minimal bootstrapping, minimal rotations, and optimal latency.

Original: In ResNet20, The optimal point corresponds to the minimal bootstrapping.
Revised: In ResNet20, The optimal point corresponds to the minimal bootstrapping.

Original: In ResNet18, the amount of rotations hikes following the channel increment.
Revised: In ResNet18, the amount of rotations hikes following the channel increment.

Original: The optimal point sacrifices bootstrapping counts to play fewer rotations.
Revised: The optimal point sacrifices bootstrapping counts to play fewer rotations.

Original: We opt for these two settings to evaluate in the following section.
Revised: We opt for these two settings to evaluate in the following section.

Original: A more thorough parameter study on GPU is presented in Appendix B.
Revised: A more thorough parameter study on GPU is presented in Appendix B.

Original: [Table]
Revised: [Table]

Original: 4. S ENSITIVITY S TUDY
Revised: 4. S ENSITIVITY S TUDY

Original: [Figure]
Revised: [Figure]

Revised: MP

Revised: CAConv (Lee et al

Original:  In Figure 6, three activation function & convolution algorithm pairs are evaluated.
Revised: , 2022a) HyPHEN AESPA with our convolution method used in the experiments are explained in Table In Figure 6, three activation function & convolution algorithm pairs are evaluated.

Original: Set1 follows the implementation of Lee et al.
Revised: Set1 follows the implementation of Lee et al.

Original: (2022a), using AppReLU with MP-CAConv.
Revised: (2022a), using AppReLU with MP-CAConv.

Original: Set spends most of the time for bootstrapping in
Revised: Set spends most of the time for bootstrapping in

Original: ResNet20 inference.
Revised: ResNet20 inference.

Original: Set2 employs the square activation of AESPA along with MP-CAConv.
Revised: Set2 employs the square activation of AESPA along with MP-CAConv.

Original: AESPA reduces the number of levels consumed by activation functions, thus requiring much less time for bootstrapping.
Revised: AESPA reduces the number of levels consumed by activation functions, thus requiring much less time for bootstrapping.

Original: Set2 experiences 5.8 × and 2.8 × reduction in bootstrapping time in ResNet20 and ResNet18, respectively.
Revised: Set2 experiences 5.1 × and 2.9 × reduction in bootstrapping time in ResNet20 and ResNet18, respectively.

Original: For both networks, most of the runtime is spent on
Revised: For both networks, most of the runtime is spent on

Original: CAConv in Set2 .
Revised: CAConv in Set2 .

Original: By introducing HyPHEN in Set3 , the execution times spent for convolutional layers are reduced by 3.3 × and 4.5 × in ResNet20 and
Revised: By introducing HyPHEN in Set3 , the execution times spent for convolutional layers are reduced by 3.1 × and 5.0 × in ResNet20 and

Original: ResNet18, respectively.
Revised: ResNet18, respectively.

Original: The impact of HyPHEN is amplified for more complex networks like ResNet18; HyPHEN resolves the problem of MP-CAConv showing a superlinear increase in the number of rotations as the number of channels increases.
Revised: The impact of HyPHEN is amplified for more complex networks like ResNet18; HyPHEN resolves the problem of MP-CAConv showing a superlinear increase in the number of rotations as the number of channels increases.

Original: Overall, HyPHEN achieves 3.8-4.9 × speedup compared to the prior state-of-the-art implementation of Lee et al.
Revised: Overall, HyPHEN achieves 3.4-4.4 × speedup compared to the prior state-of-the-art implementation of Lee et al.

Original: (2022a)
Revised: (2022a)

Original: ( Set1 ).
Revised: ( Set1 ).

Original: 4. E XECUTION TIME
Revised: 4. E XECUTION TIME

Original: Table 4 shows the runtimes of various ResNet instances on CIFAR-10.
Revised: Table 4 shows the runtimes of various ResNet instances on CIFAR-10.

Original: We measured the execution time of running inference with a single CIFAR-10 image in our CPU/GPU environments.
Revised: We measured the execution time of running inference with a single CIFAR-10 image in our CPU/GPU environments.

Original: Our
Revised: Our

Original: ResNet20/32/44 implementations on GPU take merely a few seconds to complete.
Revised: ResNet20/32/44 implementations on GPU take merely a few seconds to complete.

Original: Furthermore,
Revised: Furthermore,

Original: We demonstrate running ResNet18 for the first time.
Revised: We demonstrate running ResNet18 for the first time.

Original: As ResNet18 has 4 × more channels than
Revised: As ResNet18 has 4 × more channels than

Original: ResNet20/32/44, execution time largely depends on the convolutional layer.
Revised: ResNet20/32/44, execution time largely depends on the convolutional layer.

Original: Table 4 again demonstrates that RAConv effectively reduces the overall runtime of the convolutional layer, as our operation count analysis in Table 
Revised: Table 4 again demonstrates that RAConv effectively reduces the overall runtime of the conv layer, as our operation count analysis in Table 3. Detailed comparison with Lee et al.

Revised: (2022a) is provided in Appendix A

Original: [Table]
Revised: [Table]

Original: 4. A CCURACY
Revised: A CCURACY

Original: [Table]
Revised: [Table]

Original: In Table 5, we measured inference accuracies for CIFAR-10 images running ResNet model on our RNS-CKKS-based implementation.
Revised: In Table 5, we measured inference accuracies for CIFAR-10 images running ResNet model on our RNS-CKKS-based implementation.

Original: Though we found an error below the second decimal place at the classifier result, we did not observe any deterioration in the accuracy of ResNet20/32/44.
Revised: Though we found an error below the second decimal place at the classifier result, we did not observe any deterioration in the accuracy of ResNet20/32/44.

Original: ResNet18 shows -0.08% degradation in accuracy, which is smaller compared to the accuracy drop in Lee et al.
Revised: ResNet18 shows -0.08% degradation in accuracy, which is smaller compared to the accuracy drop in Lee et al.

Original: (2022a).
Revised: (2022a).

Original: The difference in accuracy drop can be explained by whether the original network is executed as is or an approximation has been made.
Revised: The difference in accuracy drop can be explained by whether the original network is executed as is or an approximation has been made.

Original:  L IMITATIONS
Revised:  L IMITATIONS

Original: ResNet and CIFAR-10 dataset are the only model and dataset tested in the paper.
Revised: ResNet networks are the only models addressed in the paper.

Original: Broader experiments toward deeper modelsand more realistic dataset such as Liu et al.
Revised: Broader experiments toward various models such as Liu et al.

Original: ; Russakovsky et al.

Original: (2015) would show the practicality of FHE-based PI more clearly.
Revised: (2022) would show the practicality of FHE-based PI more clearly.

Original: However, as the memory capacity may be constrained while evaluating larger benchmarks on FHE, a further investigation of memory-efficient algorithms should precede expanding the scope of FHE-based PI.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we proposed an efficient convolution algorithm RAConv, and a novel packing method Hybrid Packing.
Revised: In this paper, we proposed an efficient convolution algorithm RAConv, and a novel packing method Hybrid Packing.

Original: We showed HyPHEN, FHE-based ResNet architecture implementation applying proposed optimizations.
Revised: We showed HyPHEN, FHE-based ResNet architecture implementation applying proposed optimizations.

Original: Our experiments on real machine show 3.8-4.9 × lower latency for ResNet20/32/44/18 compared to Lee et al.
Revised: Our experiments on real machine show 3.4-4.4 × lower latency for ResNet20/32/44/18 compared to Lee et al.

Original: (2022a).
Revised: (2022a).

Original: Using GPU 
Revised: Using GPU acceleration, HyPHEN demonstrates 1.44/2.21s/2.98s/13.37s execution time for running ResNet20/32/44/18 on CIFARdataset.

Original: A B OTTLENECK A NALYSIS
Revised: A B OTTLENECK A NALYSIS

Original: Figure 7 shows the runtime analysis of the convolutional layer, which is conducted on actual layer instances of ResNet20.
Revised: Figure 7 shows the runtime analysis of the convolutional layer, which is conducted on actual layer instances of ResNet20.

Original: In Lee et al.
Revised: In Lee et al.

Original: (2022a), rotation accounts for 83-94% of the total convolution time, which is reduced to 46-77% with our method.
Revised: (2022a), rotation accounts for 83-94% of the total convolution time, which is reduced to 46-77% with our method.

Original: The optimized convolutional layer in HyPHEN leads to a smaller ratio of rotation in the total execution time.
Revised: The optimized convolutional layer in HyPHEN leads to a smaller ratio of rotation in the total execution time.

Original: [Figure]{+Table 6 shows runtime breakdown of each operation and detailed comparison with (Lee et al., 2022a) in our CPU environment. As we set both implementations to use AESPA with the same HE parameter set, the speedup is solely due to the different packing schemes. In ResNet20, our implementation shows slight increase in activation function runtime. However, latency improvements in convolution layer leads to the 1 . 
+}
Revised: {+Table 6 shows runtime breakdown of each operation and detailed comparison with (Lee et al., 2022a) in our CPU environment. As we set both implementations to use AESPA with the same HE parameter set, the speedup is solely due to the different packing schemes. In ResNet20, our implementation shows slight increase in activation function runtime. However, latency improvements in convolution layer leads to the 1 . 
+}

Revised: × lower total execution time.

Revised: Similarly in ResNet18, our implementation reports increased runtime in activation function and bootstrapping time, but 5 .

Revised: 77 × lower convolution time again leads to 3 .

Revised: 33 × lower total execution execution time

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Original: B P ARAMETER S TUDY
Revised: B P ARAMETER S TUDY

Original: We present a parameter study to explore the optimal HP setting which minimizes latency.
Revised: We present a parameter study to explore the optimal HP setting which minimizes latency.

Original: Tableand 7 show the rotation and bootstrapping counts with varying (m, d) available in ResNet20 and 18 and the execution time running the network on GPU.
Revised: Tableand 8 show the rotation and bootstrapping counts with varying (m, d) available in ResNet20 and 18 and the execution time running the network on GPU.

Original: We only represent the ( m, d ) pair of CAConv, as m and d are exchanged at RAConv.
Revised: We only represent the ( m, d ) pair of CAConv, as m and d are exchanged at RAConv.

Original: In ResNet20, we start with ( m, d ) =
Revised: In ResNet20, we start with ( m, d ) =

Original: (1 , 2) to remove input repetition as the size of the input tensor in the first layer (32 × 32 × 16) is smaller than the ciphertext slots ( 2 15 ).
Revised: (1 , 2) to remove input repetition as the size of the input tensor in the first layer (32 × 32 × 16) is smaller than the ciphertext slots ( 2 15 ).

Original: Nevertheless, Larger d has not been considered as it leads to more bootstrapping, as shown in our proposed architecture (See Figure 5).
Revised: Nevertheless, Larger d has not been considered as it leads to more bootstrapping, as shown in our proposed architecture (See Figure 5).

Original: As the input ciphertexts go through the downsampling layer, m · d gets quadrupled and the size of the intermediate tensor gets halved.
Revised: As the input ciphertexts go through the downsampling layer, m · d gets quadrupled and the size of the intermediate tensor gets halved.

Original: In ResNet20, HP that doubles d every downsampling layer yields optimal performance, which reduces rotation without increasing bootstrap.
Revised: In ResNet20, HP that doubles d every downsampling layer yields optimal performance, which reduces rotation without increasing bootstrap.

Original: In ResNet18, the impact of bootstrapping increment is often smaller than the impact of rotation decrement.
Revised: In ResNet18, the impact of bootstrapping increment is often smaller than the impact of rotation decrement.

Original: The optimal HP setting requires 15 more bootstrappings and 10687 fewer rotations than the minimum bootstrapping HP setting
Revised: The optimal HP setting requires 15 more bootstrappings and 10687 fewer rotations than the minimum bootstrapping HP setting

Original: [Table] 
Revised: [Table] [Table]

Revised: C R ES N ET ARCHITECTURE AND PARAMETERS

Revised: Figure 8 presents model architecture of modifed ResNet20 used in HCNN evaluation.

Revised: Tableand 10 shows parameters used in convolution layer of ResNet20/32/44/18.

Revised: All the parameters ( c i , c o , w i , h i , w o , h o , f w , f h , s ) are determined following the origianl ResNet paper (He et al., 2016).

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: D T RAINING D ETAILS

Revised: Models used in this paper is all trained using PyTorch (Paszke et al., 2019).

Revised: For ResNet18 and 20, our training settings are mostly identical to the AESPA; To be specific, networks are trained for 200 epochs using SGD optimizer, 0.1 initial learning rate, 100 batch size, 0.0005 weight decay and 0.9 momentum, and cosine annealing scheduler.

Revised: We also use soft labels as in (Park et al., 2022) to get higher accuracy.

Revised: For ResNet32 and 44, we use knowledge distillation (Hinton et al., 2015) to enhance the accuracy, using pretrained ResNet32/44 with 93.4% and 94.1% accuracies as teacher models.

Revised: As the FC layer of the student and teacher network is identical, teacher’s FC layer is directly reused in student network.

Revised: We trained the student networks by minimizing l 2 loss ( L kd =

Revised: ∥

Revised: f t − f s ∥ 22 ).

Revised: ResNet32 and 44 are trained for 200 epochs using SGD optimizer, 0.

Revised: weight decay and 0.9 momentum.

Revised: We use 0.05 initial learning rate and learning rate scheduler decays learning rate on epochs 150, 180, 200 by 0.1.

Revised: E L AZY -SISO

Revised: Unlike the original SISO convolution which rotates input ciphertexts before multiplying with filter plaintexts, lazy-SISO proposed in (Zhang et al., 2021) uses inversely rotated filter plaintexts to multiply with input ciphertexts.

Revised: The actual process of lazy-SISO when c i , c o = 1 is depicted in

Revised: Figure 9.

Revised: After multiplying with filter plaintext, postponed rotation is performed to accumulate intermediate ciphertexts.

Revised: During this process, multiple ciphertexts sharing the same rotation index are grouped to be accumulated first and then rotated, reducing the amount of rotations.

Revised: (e.g. c i ciphertexts are grouped in Figure 2b ).

Revised: Lazy-SISO is beneficial when input channels, which are to be accumulated, are distributed in different ciphertexts as in RAConv.

Revised: F M EMORY REQUIREMENT

Revised: Memory requirement for HCNN depends on FHE parameters and data representations (packing schemes).

Revised: In FHE, expansion of data size occurs while encryption and encoding procedure.

Revised: Resulting ciphertexts and plaintexts are typically orders of magnitude larger than messages.

Revised: The tableshows the actual size of the ciphertext, plaintext and evaluation key on three FHE parameter settings.

Revised: dnum denotes RNS-decomposition number introduced in (Han & Ki, 2019).

Revised: Given N, the degree of a cyclotomic polynomial ring, large dnum increases L, max level.

Revised: ParamSet1 is used in Set in Figure 6 to reproduce Lee et al.

Revised: (2022a).

Revised: As Set1 uses approximated ReLU for activation,

Revised: ParamSet1 adopts maximum dnum to have L = 16 .

Revised: ParamSet2 is set with the smallest L among three and is only used in Set2 on ResNet18.

Revised: ParamSet3 is the parameter set used to evaluate

Revised: [Table]

Original: {+b 1 b 1 b0 b 1 b 1 b 1 0 b 1 b 1 b⊕ 0 0 0 
+}
Revised: {+b 1 b 1 b0 b 1 b 1 b 1 0 b 1 b 1 b⊕ 0 0 0 
+}

Revised: b 2 b 2 b 2 bb 2 b

Revised: b

Revised: b

Revised: b

Revised: b

Revised: b

Original: {+2 b⊕ 0 0 0 
+}
Revised: {+2 b⊕ 0 0 0 
+}

Revised: b 3 b 3 b 3b 3 b 3 b 3 0 b 3 b 3 b 3··· = c 1 c 2 c 3 c 4 c 5 c 6 c 7 cc 9 c 10 c 11 c 12 c 13 c 14 c 15 c

Revised: [Figure]

Revised: HyPHEN and Set2 on ResNet20 which has L = 6 .

Revised: Some FHE operations such as MulCt, Rotate and Conjugate requires specific key switching procedure.

Revised: Eval key denotes public key used at this process.

Revised: The size of single Eval key is 2147, 206, 176 MB in ParamSet1, 2 and 3, respectively.

Revised: To support bootstrapping operation, one relinearization key for MulCt, one conjugate key androtation keys per rotation index are required.

Revised: We additionally load frequently used rotation keys to perform convolution.

Revised: For instance, We load 68 unique Eval keys in ResNet20 which take up 146, 14, 12 GB in ParamSet1, 2, and 3.

Revised: Once FHE parameter is determined, the packing scheme determines the number of ciphertext, plaintext to run each ResNet block.

Revised: Table 12 and 13 shows the required number and total memory size of ciphertexts and plaintexts.

Revised: We further explain the actual computation procedure to explain how the results are obtained in Appendix H. In SISO-based HCNN kernel, the size of filter plaintexts increases by factor of w i h i as each filter element is duplicated to the size of input image, which requires total f w f h w i h

Revised: i c i c o slots for weight plaintexts.

Revised: In consequence, weight plaintexts take up the majority of memory regardless of the ciphertext packing method.

Revised: In table 12, our implementation shows up to 14.75% memory overhead compared to Lee et al.

Revised: (2022a), which is primarily due to the increase in the number of ciphertexts and bias plaintext.

Revised: In Table 13, our implementation shows up to 36.8% memory overhead compared to Lee et al.

Revised: (2022a).

Revised: The larger memory overhead is caused by using different FHE parameters; In ResNet18, we use ParamSet3 for our implementation and ParamSet2 for Lee et al.

Revised: (2022a).

Revised: When using the same ParamSet3, overhead is reduced by up to 7%.

Original: [Table]
Revised: Unlike CPU, GPU memory capacity is more constrained by the current HBM technology to support high bandwidth. As the GPU memory is not capable of loading the weights of the entire model considering ResNet18, weight plaintexts should be loaded separately; while one stream computes the current ResNet block, another stream is overlapped to load next block’s weight plaintext. Through profiling the workloads of CPU and GPU activities through NVIDIA Nsight Systems (nvi, 2021), we find that all the copy stream is completed before the end of the compute stream, meaning that loading weight plaintexts does not affect overall execution time. As such, for running large neural networks, fine-grained multi-streaming can be applied to relieve memory capacity constraint.

Original: [Table] [Table] [Table]
Revised: We conducted additional experiments to evaluate the ResNet18 model on the ImageNet dataset.

Original: [Table]
Revised: The runtime is 81.85 seconds in our GPU environment. We slightly modified the first pooling layer to

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: average pool with kernel size 2 × 2 and stride 2. As shown in Figure 10, the network receives input ciphertexts processed with modified im2col. Original im2col would transform 224 × 224 × 3 images into 147 × 12544 matrix. As HCNN prefers images size to be exponential of 2, 12544 columns turns to 16384 columns.

Original: [Table] [Table] [Table]
Revised: We further split 16384 columns with stride 2 ( s avg in Figure 10) to perform average pooling without rotation, resulting 4096 columns.

Original: [Table]
Revised: We pack slot/ 4096 = 8 rows into a ciphertext.

Original: [Table] [Table] [Table] [Table]
Revised: As the total number of row is f h f w c i = 147 , the number of input ciphertext is ⌈ 147 / 8 ⌉ × 4 .

Original: [Table]
Revised: After the initial convolution layer, the number of intermediate ciphertexts becomes 64 and each ciphertext stores 8 channels with (m,d) = 1.

Original: [Table] [Table] [Table] [Table]
Revised: Table 14 shows the runtime of ResNet on ImageNet.

Original: [Table]
Revised: During the experiment, swap memory is used to make up the lack of CPU memory. Loading weight plaintexts from swap memory, which is in SSD, incurs runtime hikes as Tableshows large runtimes for Etc.

Original: [Table] [Table] [Table] [Table]
Revised: a 11 a 12 a 13 aa 21 a 22 a 23 aa 31 a 32 a 33 aa 41 a 42 a 43 a

Original: [Table]
Revised: Im 2 Col −−−−−→ ...

Original: [Table] [Table]
Revised: [Figure] [Figure]

Original: [Table] [Table] [Table] [Table]
Revised: ⊙

Original: [Table] [Table]
Revised: [Figure] [Figure]

Original: C C OMPLETE CONVOLUTION PROCEDURE OF HYBRID PACKED TENSOR
Revised: H C OMPLETE CONVOLUTION PROCEDURE OF HYBRID PACKED TENSOR

Original: Figure 8 and 9 show the complete procedure of CAConv and RAConv with HP shown briefly in Figure 4.
Revised: Figure 11 and 12 show the complete procedure of CAConv and RAConv with HP shown briefly in Figure 4.

Original: Both CAConv and RAConv perform the sequence of { SISO, RaS, IR } to the input ciphertexts.
Revised: Both CAConv and RAConv perform the sequence of { SISO, RaS, IR } to the input ciphertexts.

Original: As in Figure 2, a single superscript represents the channel of input images, and a superscript pair represents (input channel, output channel) of filters.
Revised: As in Figure 2, a single superscript represents the channel of input images, and a superscript pair represents (input channel, output channel) of filters.

Original: If multiple channels are stored in a box, we represent the list of channels using & or the range of channels using − .
Revised: If multiple channels are stored in a box, we represent the list of channels using & or the range of channels using − .

Original: We set c
Revised: We set c

Original: i , c o = 32 and w i , h i = 16 as the layer 2 of ResNet20.
Revised: i , c o = 32 and w i , h i = 16 as the layer 2 of ResNet20.

Original: In HP-CAConv (Figure 8), the HP setting of input ciphertext is ( m, d ) =
Revised: In HP-CAConv (Figure 11), the HP setting of input ciphertext is ( m, d ) =

Original: In HP-RAConv (Figure 9), the HP setting of input ciphertext is ( m, d ) =
Revised: In HP-RAConv (Figure 12), the HP setting of input ciphertext is ( m, d ) =

Original: M (1&17) ×··· M (16&32) ×∗ ## ## ## ## ## A 1 ··· A## ...
Revised: We use different brightness of color to fill the ciphertexts and plaintexts to reflect the actual computation process.

Original: ## A 211 ··· Aa (1)1 a (1)1 a (17) a (17) a (1)1 a (1)1 a (17) a (17) K (1 , 1 − 8) ··· K (16 , 1 − 8){+In Figure 11 and 12, the intermediate ciphertexts of the CAConv and input ciphertexts of the RAConv are c o 
+}
Revised: {+In Figure 11 and 12, the intermediate ciphertexts of the CAConv and input ciphertexts of the RAConv are c o 
+}

Original: K (1 , 9 − 16) ··· K (16 , 9 − 16) K (1 , 17 − 24) ··· K (16 , 17 − 24)
Revised: and c i 4 times larger than input ciphertext of the CAConv, respec-

Original: K (1 , 25 − 32) ···
Revised: [Table] [Table]

Original: K (16 , 25 − 32)
Revised: tively.

Original: K (1 , 1 − 8) ··· K (16 , 1 − 8) K (1 , 9 − 16) ··· K (16 , 9 − 16) ··· ··· ···
Revised: The CAConv, activation and RAConv in a ResNet block is processed at once to mitigate the huge number of intermediate ciphertexts.

Original: K (1 , 25 − 32) ···
Revised: To avoid an increase of memory footprint, the operation on the input ciphertext continues until the ciphertext shrinks again.

Original: K (16 , 25 − 32) K (1&17 , 1 − 4) ··· K (16&32 , 1 − 4) K (1&17 , 5 − 8) ···
Revised: Thus, the tuple of operations { SISO, RaS, IR, Square, SISO l } are processed to an input ciphertext and then accumulated.

Original: K (16&32 , 5 − 8) ··· ··· ··· K (1&17 , 29 − 32) ··· K (16&32 , 29 − 32) ci /c o /
Revised: Blocks colored with high brightness show actual working set, which means c o 4 times larger intermediate ciphertexts (colored with low brightness) are irrelevant with the peak memory consumption.

Original: F···
Revised: [Table] M [Table]

Original: F ...
Revised: [Table]

Original: F···
Revised: ∗ ##

Original: Fk (1 , 1)k (1 , 2)k (17 , 1)k (17 , 2)k (1 , 3)k (1 , 4)k (17 , 3)k (17 , 4)w i h i w i h i f
Revised: [Table]

Original: w f h
Revised: [Table]

Original: SISO −−−−→ MK (1&17 , 1 − 4) ··· MK (16&32 , 1 − 4) MK (1&17 , 5 − 8) ··· MK (16&32 , 5 − 8) ··· ··· ··· MK (1&17 , 29 − 32) ··· MK (16&32 , 29 − 32)
Revised: [Table]

Original: RaS RaS −−−→
Revised: [Table] − [Table]

Original: MK (1 − 32 , 1 − 4) ···
Revised: MK (1 − 32 , 1 − 4) ···

Original: MK (1 − 32 , 5 − 8) ··· ··· ···
Revised: MK (1 − 32 , 5 − 8) ··· ··· ···

Original: MK (1 − 32 , 29 − 32) ··· b (1 − 16 , 1) b (1 − 16 , 2) b (17 − 32 , 1) b (17 − 32 , 2) b (1 − 16 , 3) b (1 − 16 , 4) b (17 − 32 , 3) b (17 − 32 , 4)
Revised: MK (1 − 32 , 29 − 32) ··· b (1 − 16 , 1) b (1 − 16 , 2) b (17 − 32 , 1) b (17 − 32 , 2) b (1 − 16 , 3) b (1 − 16 , 4) b (17 − 32 , 3) b (17 − 32 , 4)

Original: RaS
Revised: RaS

Original: RaS −−−→ c (1) c (2) ## ## c (3) c (4) ## ##
Revised: RaS −−−→ c (1) c (2) ## ## c (3) c (4) ## ##

Original: Mask and Broadcast
Revised: Mask and Broadcast

Original: Broadcast −−−−−−−→ c (1) c (2) c (1) c (2) c (3) c (4) c (3) c (4) repack −−−−→
Revised: Broadcast −−−−−−−→ c (1) c (2) c (1) c (2) c (3) c (4) c (3) c (4) repack −−−−→

Original: N (1 − 4) ×···
Revised: N (1 − 4) ×···

Original: N (5 − 8) ×··· ··· ···
Revised: N (5 − 8) ×··· ··· ···

Original: N (29 − 32) ×···
Revised: N (29 − 32) ×···

Original: M (1 − 4) ×···
Revised: M (1 − 4) ×···

Original: M (5 − 8) ×··· ··· ···
Revised: M (5 − 8) ×··· ··· ···

Original: M (29 − 32) ×··· c o /c i /∗
Revised: M (29 − 32) ×··· c o /c i /∗

Original: K (1 − 4 , 1&17) ···
Revised: K (1 − 4 , 1&17) ···

Original: K (1 − 4 , 16&32)
Revised: K (1 − 4 , 16&32)

Original: K (5 − 8 , 1&17) ···
Revised: K (5 − 8 , 1&17) ···

Original: K (5 − 8 ,16&32)
Revised: K (5 − 8 ,

Original: K (9 − 12 , 1&17) ··· K (9 − 12 , 16&32) K (29 − 32 , 1&17) ··· K (29 − 32 , 16&32) K (1 − 4 , 1&17) ··· K (1 − 4 , 16&32)
Revised: [Table]

Original: K(5 − 8 , 1&17) ··· K (5 − 8 , 16&32) K (9 − 12 ,1&17) ··· K (9 − 12 , 16&32) K (29 − 32 , 1&17) ··· K (29 − 32 , 16&32)
Revised: K (9 − 12 ,

Original: K (1 − 4 , 1&17) ··· K (1 − 4 , 16&32) K (5 − 8 , 1&17) ··· K (5 − 8 , 16&32)
Revised: [Table]

Original: K (9 − 12 , 1&17) ··· K (9 − 12 ,16&32) K (29 − 32 , 1&17) ··· K (29 − 32 , 16&32) c o /c
Revised: 17) ··· K (9 − 12 ,

Original: i /
Revised: [Table]

Original: A 1 ··· ··· A... A 241 ··· ··· Aw i h i a (1)1 a (2)1 a (1)1 a (2) a (3)1 a (4)1 a (3)1 a (4) F 1 ··· F ... F 1 ··· F     w i h
Revised: K (29 − 32 ,

Original: i k (1 , 1) 1 , 1 k (2 , 1) 1 , 1 k (1 , 17) 1 ,k (2 , 17) 1 ,k (3 , 1) 1 , 1 k (4 , 1) 1 , 1 k (3 , 17) 1 ,k (4 , 17) 1 ,

Original: SISO l −−−−→

Original: MK (1 − 32 , 1&17) ··· MK (1 − 32 , 1&17) ··· MK (1 − 32 , 1&

Original: [Figure]
Revised: [Table]

Original: [Figure] [Figure] (1 − [Figure]
Revised: 17) ··· K (29 − 32 ,

Original: [Figure]

Original: [Figure] − [Figure] [Figure]
Revised: [Table]

Original: (1 − [Figure] [Figure] [Figure]
Revised: 32) c o /c

Original: −

Original: [Figure]

Original: b ( [Figure] − [Figure]
Revised: i /

Original: [Figure] [Figure] − [Figure]
Revised: [Table]

Original: ) b (

Original: [Figure] − [Figure] [Figure]
Revised: [Table]

Original: −

Original: [Figure]
Revised: [Figure]

Original: (1) ## c (

Original: [Figure]

Original: [Figure]