  Homomorphic Encryption (HE), allowing computations on encrypted data
(ciphertext) without decrypting it first, enables secure but prohibitively slow
Neural Network (HENN) inference for privacy-preserving applications in clouds.
To reduce HENN inference latency, one approach is to pack multiple messages
into a single ciphertext in order to reduce the number of ciphertexts and
support massive parallelism of Homomorphic Multiply-Add (HMA) operations
between ciphertexts. However, different ciphertext packing schemes have to be
designed for different convolution layers and each of them introduces overheads
that are far more expensive than HMA operations. In this paper, we propose a
low-rank factorization method called FFConv to unify convolution and ciphertext
packing. To our knowledge, FFConv is the first work that is capable of
accelerating the overheads induced by different ciphertext packing schemes
simultaneously, without incurring a significant increase in noise budget.
Compared to prior art LoLa and Falcon, our method reduces the inference latency
by up to 87% and 12%, respectively, with comparable accuracy on MNIST and
CIFAR-10.
