Paper: Ditto: Quantization-Aware Secure Inference of Transformers upon MPC (ICLR-2024)

Reviewer comment (Evaluation):
The paper acknowledges that both Ditto and MPCFormer exhibit noticeable utility drops in Bert tasks when employing ReLU approximation for Softmax. They offer Quad approximation for GeLU to maintain a balance between utility and efficiency, but this limitation may constrain the applicability of the framework for tasks where such approximations are not tolerable.

Author response:
We first note that **the approximation of GeLU is not coupled to Ditto.** In Section 5.2, we present the efficiency of Ditto without approximations, which leads to about $2\times$ communication reduction than prior works. For scenarios where this approximation is not applicable, we can resort to using the standard GeLU or investigating more accurate approximations.

Nevertheless, our comprehensive evaluations, which include models like Bert and GPT, reveal that **the approximated GeLU offers a more favorable tradeoff between utility and efficiency.** As a result, we currently believe that it is feasible to incorporate this approximation into Ditto.

Reviewer comment (Writing):
The paper in general is hard to read and require additional proof-reading. I would recommend making the paper to be easier to read by highlighting important concepts, introducing figures that support main results, and describing contributions and future work.

Author response:
We thank the detailed review of the paper writing. We will highlight the important concepts in quantization-aware secure inference in the final version.

To summarize, we propose Ditto to bridge the gap between plaintext quantization and MPC-based secure inference. The gap mainly results from two aspects: 

1. (Section 4.3) **Type conversions are difficult in MPC**: MPC encodes **floating-point plaintext** into **fixed-point ciphertexts** (over rings). Typically, the ciphertexts are encoded over a uniform ring of fixed size (like $2^\ell$). In this case, although plaintexts can be quantized into low-bit representation (e.g., from fp16 to int8), the ciphertexts remain the same size (i.e., $\ell$-bit fixed-point numbers). **To incorporate low-bit quantization into MPC, we need to allow ciphertext conversion among different rings. This is why we propose two novel type conversion MPC primitives.** Compared to SOTA type conversion primitives, our protocol incurs much lower communication overhead. Table below lists the theoretical communication complexity against SOTA works (cast from $\mathbb{Z}\_{2^\ell}$ to $\mathbb{Z}\_{2^{\ell'}}$. $\lambda$ is the security parameter, 128 in SIRNN[2]).

|||Round|Size (bits)|
|:--:|:--:|:--:|:--:|
|Downcast|SecureML[1]|0|0|
||SIRNN[2]|0|0|
||Ditto|0|0|
|Upcast|SIRNN[2]|$\log\ell+2$|$\lambda(\ell+1)+13\ell+\ell'$|
||Baccarini et al.[3]|$\log\ell+4$|$2(\ell+\ell')+\log\ell+2$|
||Ditto|3|$2\ell+\ell'$|

2. (Section 4.2) **Dynamic quantization is expensive in MPC**: Standard plaintext quantization requires additional operations like `clamp` and `min/max` during runtime to achieve low-bit quantization (e.g., int8). These operations involve non-linear comparisons that are cheap in plaintext yet quite expensive in MPC. **As a result, we propose an MPC-friendly quantization scheme to improve efficiency.** Besides, we implement the quantization-aware distillation to retain the model utility.

Ditto effectively bridges the aforementioned gaps. **By reducing the size of ciphertexts, we are able to mitigate communication overhead, resulting in improved efficiency against SOTA works.**

Reviewer comment (Experiments):
Lack of comparison to the latest related work.

Author response:
We thank the reviewer MwvF for the positive feedback on "creative solutions to a key challenge in quantization-aware secure inference.". It is encouraging for us to further narrow the gap between the two entirely different worlds of plaintext quantization and MPC-based secure inference.

We then answer the specific question below.

> Lack of comparison to the latest related work.

Thank you for mentioning Iron. We first present a short comparison against Iron. In general, Ditto is orders of magnitude faster than Iron on Bert (base and large) models with input sequence length of 128.

|       |     Bert-base     |                     |    Bert-large     |                      |
| :---: | :---------------: | :-----------------: | :---------------: | :------------------: |
| Iron  | $\approx$ 30 min  |                     | $\approx$ 100 min |                      |
| Ditto | $\approx$ 0.5 min | $\uparrow 60\times$ |  $\approx$ 1 min  | $\uparrow 100\times$ |

The reason we do not compare Ditto with Iron is because the settings of these two works are not the same. As stated in Section 2, Iron is based on 2PC (2-party computation) setting, while Ditto is 3PC, thus leading to an unfair comparison. 

We note that we indeed comprehensively compare with the SOTA works. MPCFormer (ICLR 2023) and PUMA (2023.09) are two latest 3PC works of the same setting as Ditto.

Reviewer comment (Experiments):
How would the proposed DITTO be compared with Iron [1]?  

[1] Hao, Meng, et al. "Iron: Private inference on transformers." Advances in Neural Information Processing Systems 35 (2022): 15718-15731.

Author response:
We thank the reviewer MwvF for the positive feedback on "creative solutions to a key challenge in quantization-aware secure inference.". It is encouraging for us to further narrow the gap between the two entirely different worlds of plaintext quantization and MPC-based secure inference.

We then answer the specific question below.

> Lack of comparison to the latest related work.

Thank you for mentioning Iron. We first present a short comparison against Iron. In general, Ditto is orders of magnitude faster than Iron on Bert (base and large) models with input sequence length of 128.

|       |     Bert-base     |                     |    Bert-large     |                      |
| :---: | :---------------: | :-----------------: | :---------------: | :------------------: |
| Iron  | $\approx$ 30 min  |                     | $\approx$ 100 min |                      |
| Ditto | $\approx$ 0.5 min | $\uparrow 60\times$ |  $\approx$ 1 min  | $\uparrow 100\times$ |

The reason we do not compare Ditto with Iron is because the settings of these two works are not the same. As stated in Section 2, Iron is based on 2PC (2-party computation) setting, while Ditto is 3PC, thus leading to an unfair comparison. 

We note that we indeed comprehensively compare with the SOTA works. MPCFormer (ICLR 2023) and PUMA (2023.09) are two latest 3PC works of the same setting as Ditto.

Reviewer comment (Novelty):
Distillation is widely used in MPC-based secure inference works.

Author response:
We clarify that our main contribution is proposing MPC-friendly quantization and quantization-aware distillation, which effectively bridges the gap between plaintext quantization and MPC-based inference.
Different from prior works that leverage distillation, we further incorporate MPC-friendly quantization into distillation, which is not trivial. **This involves converting the computations in the student model into fixed-point arithmetic (illustrated in Figure 5, Appendix A.7) and utilizing them as inputs to the distillation process.** With quantization-aware distillation, we can enhance efficiency while maintaining a negligible utility drop.

Reviewer comment (Novelty):
It seems limited contributions of MPC protocols.

Author response:
Due to page limitation, we are unable to provide a detailed description in Section 4.3.1. The **formulated protocol construction**, **correctness analysis** and **security proofs** are presented in Appendix A.2~A.4.
Besides, we emphasize that the Downcast and UpCast are different from prior works and exhibit better performance, which is the key to quantization-aware secure inference. We believe that the following explanations will help demonstrate the contribution of MPC protocols.

Reviewer comment (Theory):
Does the Downcast protocol have a probabilistic error? What is the difference compared with the truncation of SecureML?

Author response:
To summarize, there are distinct differences between Downcast and the truncation method in SecureML. **Downcast introduces a probabilistic error solely in the least significant bit (LSB), whereas SecureML introduces probabilistic errors in both the LSB and MSB.** The impact of LSB error is negligible, while the MSB error may cause significant deviations.

As shown in Equation 2, Appendix A.3,

$x'=x/2^t$ 

$=((x_0+x_1+x_2)\mod 2^\ell)/2^t$

$=x_0/2^t+x_1/2^t+x_2/2^t-w\cdot 2^{\ell-t}+w'$

$=(x_0/2^t+x_1/2^t+x_2/2^t-w\cdot 2^{\ell-t}+w')\mod 2^{\ell'}$

$=(x_0/2^t\mod 2^{\ell'})+(x_1/2^t\mod 2^{\ell'})+(x_2/2^t\mod 2^{\ell'})-(w\cdot 2^{\ell-t}\mod 2^{\ell'})+w'\mod 2^{\ell'}$

there are two potential wraps, i.e., $w$ and $w'$. $w$ leads to the MSB error, while $w'$ leads to the LSB error. In Ditto, since we have $\ell−t=64−(18−8)=54$ and $\ell'=32$, we can get $w \cdot 2^{\ell−t}\mod 2^{\ell'}=0$. That is, we can implicitly erase the MSB error in Ditto. While in SecureML, the potential wrap happens with probability and may lead to significant errors.

Reviewer comment (Reproducibility):
In Upcast, what distribution is $r$ sampled from? How to ensure the input is positive?

Author response:
The randomness $r$ is a random $\ell$-bit integer, uniformly sampled from $\mathbb{Z}_{2^{\ell}}$ to ensure perfect security during the mask-and-open process of $x$. That is, $r \in [-2^{\ell-1}, 2^{\ell-1}-1]$.

We have a trick that restricts the range of $x\in[-2^{\ell-2},2^{\ell-2}-1]$. As a result, $x+2^{\ell-2}\in[0,2^{\ell-1}-1]$, thus ensuring that the input to Upcast is positive. 
After the conversion, the bias term $2^{\ell-2}$ can be conveniently subtracted to eliminate its influence.
This approach allows us to achieve performance improvements at the sacrifice of one bit without compromising the overall security of the system.
The correctness analysis is provided in Appendix A.3.

Reviewer comment (Experiments):
Could you provide the theoretical or experimental advantages of the proposed Downcast and Upcast protocols compared with SOTA?

Author response:
Consider casting from $\mathbb{Z}\_{2^\ell}$ to $\mathbb{Z}\_{2^{\ell'}}$. $\lambda$ denotes the computational security parameter, typically 128 in SIRNN[2].

The table below demonstrates that Ditto surpasses previous works in terms of the upcast protocol, excelling in both communication round and size. These two factors are crucial in measuring the efficiency of protocols. **Notably, Upcast in Ditto is constant-round (3 rounds), which is much better than SOTA works ($\mathcal{O}(\log\ell)$).**

In terms of downcast, **Ditto incurs no communication like prior works, while avoiding MSB error**, which was a drawback in the SecureML approach.

|||Round|Size (bits)|
|:------:|:------------------:|:-------------:|:---------------------------------:|
|Downcast|SecureML[1]|0|0|
||SIRNN[2]|0|0|
||Ditto|0|0|
|Upcast|SIRNN[2]| $\log \ell + 2$ | $\lambda (\ell+1) + 13\ell + \ell'$ |
||Baccarini et al.[3]| $\log \ell + 4$ |  $2(\ell + \ell') + \log \ell + 2$  |
||Ditto|3|$2\ell + \ell'$|