Paper:
When applying machine learning to sensitive data one has to balance between accuracy, information leakage, and computational-complexity. Recent studies have shown that Homomorphic Encryption (HE) can be used for protecting against information leakage while applying neural networks. However, this comes with the cost of limiting the kind of neural networks that can be used (and hence the accuracy) and with latency of the order of several minutes even for relatively simple networks. In this study we improve on previous results both in the kind of networks that can be applied and in terms of the latency. Most of the improvement is achieved by novel ways to represent the data to make better use of the capabilities of the encryption scheme.

Review:
The paper proposes improvements on the area of neural network inference with homomorphically encrypted data, where a major drawback in current applications is the high computational cost.
The paper makes several minor contributions and, in my opinion, it fails to provide an unified message to the reader. The arguably independent contributions are:
1. the use of the faster SAEL 3.2.1 over CryptoNet — not really an innovation per se
2. flexible representation of message space — the main contribution
3. a much more complex network than CryptoNet for an experiment on CIFAR10 — minor contribution/application of 2
4. using pre-trained model for embedding images before encryption — minor contribution
I think the authors should refocus this work on point 2 and 3 only. 1 could simply be a note in the paper, as not a a real contribution and 4 may be even excluded as it goes toward a different direction.
I will mainly focus on the central contribution of the paper — the data representation — which in my opinion has the potential to progress this area further. Unfortunately, the current quality of presentation is suboptimal. The choices on architecture and intermediate representations of LoLa in Section 4 are hard to follow, and it is not clear to me when the authors are making a choice or satisfying a constraint. That is, for example when we aim to vs. we have to use one representation in place of another? Since this is the main contribution, I suggest the author to help the presentation with diagrams and formulae in this Section. Each component should be presented in modular fashion. In fact, this is what the authors did in Section 3. Yet, towards reuse of these concepts for machine learning applications, the authors could present their improvements as applied to *neural network layers*. E.g. responding to answers such as: what are the most efficient data representation in input and output if we want to compute a convolutional layer? It would be also interesting to break down cost requirements for each intermediate computation, to highlight bottlenecks and compare alternative choices.
Incidentally, LoLa-Conv improvements appear to be simply due to the use of a conv vs. dense layer as input.
Section 6 explains the use of a pre-trained AlexNet to embed data for learning a linear model (HE-aware) for Caltech 101. In the context of application of HE, the idea is novel to my knowledge, although it is a rather common practice in deep learning. The number reported in this section have no baseline to compare with and therefore it’s hard to evaluate their significance.
A missing reference is [A].
Minors
* section 4: “this saves a processing step which saves"
* typos in section 6: “a networks”, “in medical imagine"
* footnote 4: “convinient"
[A] TAPAS: Tricks to Accelerate (encrypted) Prediction As a Service.