Paper: Confidential-DPproof: Confidential Proof of Differentially Private Training (ICLR-2024)

Reviewer comment (Experiments):
Even with these simplifications, gradient computation + clipping can still take over a second per sample in higher feature dimensions. However, as the authors point out in the related work, even the problem of computationally feasible non-private proof of learning is still unresolved. This would be an important avenue for future work in order for this auditing protocol to be applied to modern models with hundreds of billions of parameters trained on massive datasets.

Author response:
> **This would be an important avenue for future work in order for this auditing protocol to be applied to modern models with hundreds of billions of parameters trained on massive datasets.**

We agree with the reviewer that an interesting future direction is to audit the privacy of complex models for example through advancing research in cryptography. Therefore, we added the following in Section 6: 

``Finally, our framework naturally benefits from future advances in cryptography, so that extending to massive models and datasets would be an interesting future direction.”

Reviewer comment (Reproducibility):
The auditor needs to know many implementation details, e.g. clip threshold, and number of iterations. Also, the protocol is specifically designed for DP-SGD. It seems that we need to design different protocols for different algorithms, even if we only make minor adjustments to the algorithm.

Author response:
> **The auditor needs to know many implementation details, e.g. clip threshold, and number of iterations.** 

Yes, as written at the beginning of Section 3, the prover and auditor agree on the DP-SGD training algorithm and specific values for its hyperparameters. This is also the case in all privacy auditing approaches and does not impose any overhead. These hyperparameters would typically be set by the prover to achieve high utility and shared with the auditor to verify the claimed privacy guarantees $(\varepsilon,\delta)$.

> **Also, the protocol is specifically designed for DP-SGD. It seems that we need to design different protocols for different algorithms, even if we only make minor adjustments to the algorithm.**

Our protocol, coupled with the unbiased randomness sampling protocol we propose, can support any randomized polynomial-time computable function and thus any (tractable) DP mechanism. In particular, our protocol is designed to attest the correctness of the key and common building blocks of DP algorithms: 1) data subsampling; 2) gradient computation; 3) sensitivity analysis;  4) noise addition; and 5) model update. Therefore one does not need to design new protocols for a new DP algorithm; however, extra implementation in ZK is needed to support them. 
Note that our approach is also decoupled from the privacy accounting technique used. For example, our framework is compatible with recent advanced results for DP  in the hidden state model [Jiayuan & Shokri, NeurIPS 2022; Altschuler & Talwar, NeurIPS 2022] as our protocol keeps the entire trajectory (i.e., intermediate model updates) hidden from the auditor and can help to improve the privacy by adding hidden states into the privacy accounting technique.

We added this discussion in Section 6.

[Ye & Shokri, NeurIPS 2022] Jiayuan Ye, and Reza Shokri. "Differentially private learning needs hidden state (or much faster convergence)." NeurIPS 2022

[Altschuler & Talwar, NeurIPS 2022] Jason M. Altschuler, Kunal Talwar: “Privacy of Noisy Stochastic Gradient Descent: More Iterations without More Privacy Loss.” NeurIPS 2022

Reviewer comment (Theory):
Many steps in the DP-SGD algorithm need to be proved in Phase 3. If one step is missing, for example, the auditor forgets to let the prover verify step vi, the total number of iterations, how would it affect validity of the privacy audit claims made by the auditor?

Author response:
> **Many steps in the DP-SGD algorithm need to be proved in Phase 3. If one step is missing, for example, the auditor forgets to let the prover verify step vi, the total number of iterations, how would it affect validity of the privacy audit claims made by the auditor?**

We would like to stress that rather than verifying each step separately, our protocol allows the prover to generate a single proof to show that every step in the DP-SGD is computed correctly. This is reflected in step 3 and 4 of Algorithm 3 where the DP-SGD algorithm is encoded as a public circuit. So if the prover skips a step in the algorithm, it will not be able to produce a valid proof and the auditor will be able to tell. This follows from the soundness property of the underlying ZKP framework. 
We clarified this at the beginning of Section 3 by saying:

``Prover and auditor run our zero-knowledge protocol and provide a certificate for the claimed privacy guarantee $(\varepsilon,\delta)$. We represent the whole procedure as one single public circuit, allowing the prover to generate one single proof demonstrating to the auditor that it correctly executed all of the steps of the DP-SGD algorithm (see Algorithm 1).”

Reviewer comment (Experiments):
The proposed cryptographic approach does not scale to large models trained with DP-SGD.

Author response:
> **The proposed cryptographic approach does not scale to large models trained with DP-SGD.**

We proposed the first zero-knowledge protocol which enables auditors to directly verify the privacy guarantees of a DP-SGD training run, while protecting the confidentiality of both their model and training dataset.  We agree with you that our protocol, similar to any other cryptographic protocols, comes with some overhead but we want to emphasize that such overhead allows us to achieve strong confidentiality, which could not be achieved otherwise. 

To reduce this overhead, instead of verifying the training of complex models on raw data which calls for heavy cryptographic machinery, we proposed a co-design of a DP training algorithm with a customized zero-knowledge (ZK) protocol in which we incorporate feature extractors to enable DP-SGD training of simpler models, thus being more efficient. Note that it has been recently shown that public feature extractors enable simpler models to achieve accuracy-privacy tradeoffs that are competitive with the state-of-the-art training of a complex model on a raw dataset with DP-SGD. Therefore, training simpler models is desirable in DP literature due to 1) the computational overhead imposed by DP algorithms themselves and 2) the fact that the utility cost of DP-SGD increases with the number of parameters. For example, as shown in [Figure 2, Tramer and Boneh, ICLR2021], privately training a simple logistic regression on features obtained by ScatterNet gives much better accuracy-privacy tradeoffs than privately training a CNN on raw data or even on extracted features.

Finally, the overhead of our approach can be further reduced as research in cryptography continues, which is out of the scope of this paper.

[Tramer and Boneh, ICLR2021] Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). In The International Conference on Learning Representations (ICLR), 2021

Reviewer comment (Theory):
It seems to me that the protocol only attempts to verify that every step of the DP-SGD algorithm is executed correctly as claimed, and the certified privacy parameters are simply derived based on the verified $\sigma$ and subsampling level, which is an upper bound on the actual privacy guarantee (as stated in the conclusion). Thus, when the certified upper bound exceeds the claimed value, we do not know whether there is a privacy failure. Note that even with 100% correct execution of DP-SGD, privacy failure may still exist due to other issues like finite precision computation of floats [1]. Thus, verifying all steps are executed correctly is not sufficient to audit privacy claims, and a privacy lower bound should still be necessary.

Author response:
> **It seems to me that the protocol only attempts to verify that every step of the DP-SGD algorithm is executed correctly as claimed, and the certified privacy parameters are simply derived based on the verified $\sigma$ and subsampling level, which is an upper bound on the actual privacy guarantee (as stated in the conclusion). Thus, when the certified upper bound exceeds the claimed value, we do not know whether there is a privacy failure. Compared to the privacy upper bound provided for the chosen $\sigma$, how accurate are the certified privacy guarantees?**


Thank you for the question. Your first point is correct. In differential privacy, the privacy guarantees are obtained using privacy accounting techniques that typically give an upper bound. This is the best we can have as it is usually infeasible to compute the exact privacy guarantees of an algorithm (except in very simple cases). Our protocol allows an auditor to verify the correct execution of DP-SGD performed by a prover while keeping all intermediate updates and even the final model hidden from the auditor. Once the correctness of the algorithm has been certified, the privacy guarantee can be computed and verified using any existing (public) privacy accounting technique. There is no need for cryptography here: the prover simply reveals the claimed privacy guarantee and indicates the accounting technique used to obtain it, and the auditor (or anyone) can verify that it is correct in the clear. As a consequence, to answer your last question, the certified privacy guarantees will exactly match the privacy upper bound given by privacy accounting.

We have clarified this in Section 4 and 6.

Reviewer comment (Novelty):
How does the approach compare to the recent work of [2]?

Author response:
> **How does the approach compare to the recent work of [2]? [2] Privacy Auditing with One (1) Training Run, Thomas Steinke, Milad Nasr, Matthew Jagielski, https://arxiv.org/abs/2305.08846**

The work of [2] tries to improve the computational costs of post hoc privacy auditing techniques that we cited in our paper but it still suffers from several limitations inherent to post hoc privacy auditing (described in the introduction and related work of our paper): in particular 1) establishing only lower bound on the privacy loss; and 2) the model and some data must be shared with the auditor to get a tighter lower bound on the privacy loss. Due to (1), an auditor using this post hoc approach can only hope to disprove a false claim of privacy as they cannot provide a certificate of the privacy claim made by the prover.
Instead, we propose a zero-knowledge proof (ZKP) protocol to proactively generate a certificate of privacy during training while preserving the confidentiality of all information. As highlighted by reviewer yrMg our proposal is 
  - ``a strong alternative to current methods for privacy auditing’’ which includes [2]; 
  - ``has the potential to change the paradigm for privacy auditing moving forward’’; and 
  - ``completely sidestep the need for optimal adversaries for verifying privacy guarantees’’.

Following your suggestion, we cited [2] in the related work of our paper.

Reviewer comment (Writing):
Page 5, line 10 (the description of phase 2): "Next, the auditor generates... and sends.. to the auditor". The second "auditor" should be "prover"?

Author response:
> **Typos:
Page 5, line 10 (the description of phase 2): "Next, the auditor generates... and sends.. to the auditor". The second "auditor" should be "prover"?
Page 6, line 2: we first generates -> generate**

Thanks for spotting these typos, we fixed them following your suggestions.

Reviewer comment (Writing):
Page 6, line 2: we first generates -> generate

Author response:
> **Typos:
Page 5, line 10 (the description of phase 2): "Next, the auditor generates... and sends.. to the auditor". The second "auditor" should be "prover"?
Page 6, line 2: we first generates -> generate**

Thanks for spotting these typos, we fixed them following your suggestions.

Reviewer comment (Reproducibility):
What are the hyper-parameters: $C, \sigma, T$ in your experiments, and what are the corresponding theoretical upper bounds on $\epsilon, \delta$?

Author response:
> **What are the hyper-parameters: $C, \sigma, T$ in your experiments, and what are the corresponding theoretical upper bounds on $\epsilon, \delta$?**

We added the values of the clipping norm $C$ and the noise multiplier $\sigma$ in Table 1 following the hyperparameter search done by [Tramer and Boneh, ICLR2021]:

|Dataset| Clipping norm $C$ | $\sigma$ | 
|-------|--------|--------|
| MNIST | 0.1  | 3  | 
| CIFAR-10 | 0.1 | 3.32 | 

We clarified the number of training iterations at the beginning of Section 4.2 as:

“We run in total 225 training iterations and 55 training iterations for CIFAR-10 and MNIST, respectively, and reported the accuracy in the left column of Figure 1 and the certified DP guarantees ($\varepsilon$) in the right column of Figure 1, which both change as a function of training iterations.”

We also specified the value of $\delta$ in the caption of Figure 1 as follows:

“The running time is computed based on the number of DP training iterations for various DP budgets $(\varepsilon,\delta=10^{-5})$.”

[Tramer and Boneh, ICLR2021] Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). In The International Conference on Learning Representations (ICLR), 2021.

Reviewer comment (Evaluation):
Compared to the privacy upper bound provided for the chosen $\sigma$, how accurate are the certified privacy guarantees?

Author response:
> **It seems to me that the protocol only attempts to verify that every step of the DP-SGD algorithm is executed correctly as claimed, and the certified privacy parameters are simply derived based on the verified $\sigma$ and subsampling level, which is an upper bound on the actual privacy guarantee (as stated in the conclusion). Thus, when the certified upper bound exceeds the claimed value, we do not know whether there is a privacy failure. Compared to the privacy upper bound provided for the chosen $\sigma$, how accurate are the certified privacy guarantees?**


Thank you for the question. Your first point is correct. In differential privacy, the privacy guarantees are obtained using privacy accounting techniques that typically give an upper bound. This is the best we can have as it is usually infeasible to compute the exact privacy guarantees of an algorithm (except in very simple cases). Our protocol allows an auditor to verify the correct execution of DP-SGD performed by a prover while keeping all intermediate updates and even the final model hidden from the auditor. Once the correctness of the algorithm has been certified, the privacy guarantee can be computed and verified using any existing (public) privacy accounting technique. There is no need for cryptography here: the prover simply reveals the claimed privacy guarantee and indicates the accounting technique used to obtain it, and the auditor (or anyone) can verify that it is correct in the clear. As a consequence, to answer your last question, the certified privacy guarantees will exactly match the privacy upper bound given by privacy accounting.

We have clarified this in Section 4 and 6.

Reviewer comment (Evaluation):
The results do not show certified level of $\delta$.

Author response:
> **The results do not show certified level of $\delta$.**

We do certify an $(\varepsilon, \delta)$-DP guarantee. As explained above, once we have certified that the DP-SGD algorithm has been executed correctly, the DP guarantee is obtained by running a (public) privacy accounting technique which typically takes as input an arbitrary value of $\delta$ and produces a value for $\varepsilon$. In practice, a popular choice of $\delta$ is to set it as the reciprocal of the size of the dataset, which can be easily verified by the auditor in our framework after the prover has committed to the training dataset.

For clarity, we added the value of $\delta$ of the certified guarantee in the caption of Figure 1 and the above discussion at the beginning of Section 4.

Reviewer comment (Evaluation):
The running time may vary across different machines and may not be a consistent measure of computational cost.

Author response:
> **The running time may vary across different machines and may not be a consistent measure of computational cost.**

Following your suggestion, we evaluated the running time of our framework using different machines and demonstrated that the trends and conclusions drawn from our results remain consistent and reliable across different machines.

In particular, we analyzed the running time of the basic building blocks including per-example gradient computation, noise sampling and model updates as a function of the number of data attributes across three different machines with different CPUs: ARM, intel, and AMD (m1.xlarge, m7i.2xlarge, m7a.2xlarge, all with 16GB of RAM). Results are shown in Table 3 and Table 4 in the Appendix B of our paper and below:

| #Attributes | ARM Gradient computation + clipping (s) | INTEL Gradient computation + clipping (s) | AMD Gradient computation + clipping (s) | ARM Model update (s) | INTEL Model update (s) INTEL | AMD Model update (s) | ARM Noise sampling | INTEL Noise sampling | AMD Noise sampling |
|-------------|-----------------------------------------|-------------------------------------------|-----------------------------------------|----------------------|------------------------------|----------------------|--------------------|----------------------|--------------------|
| 16          | 0.005                                   | 0.009                                     | 0.012                                   | 0.0003               | 0.0004                       | 0.0005               | 0.0230              | 0.0285               | 0.0344             |
| 32          | 0.012                                   | 0.018                                     | 0.025                                   | 0.0006               | 0.0014                       | 0.0017               | 0.0407              | 0.0505               | 0.0676             |
| 64          | 0.021                                   | 0.032                                     | 0.047                                   | 0.0011               | 0.0020                       | 0.0028               | 0.0754              | 0.1004               | 0.1320             |
| 128         | 0.051                                   | 0.052                                     | 0.095                                   | 0.0024               | 0.0051                       | 0.0077               | 0.1461              | 0.1995               | 0.2587             |
| 256         | 0.091                                   | 0.124                                     | 0.188                                   | 0.0098               | 0.0062                       | 0.0099               | 0.2713              | 0.3885               | 0.5352             |
| 512         | 0.182                                   | 0.248                                     | 0.375                                   | 0.0144               | 0.0187                       | 0.0250               | 0.5668              | 0.7625               | 1.0065             |
| 1024        | 0.324                                   | 0.533                                     | 0.747                                   | 0.0550               | 0.0275                       | 0.0467               | 1.0873              | 1.5874               | 2.0179             |
| 2048        | 0.719                                   | 1.048                                     | 1.489                                   | 0.0625               | 0.0689                       | 0.0934               | 2.1725              | 3.1763               | 4.0886             |
| 4096        | 1.462                                   | 2.121                                     | 2.984                                   | 0.1614               | 0.1498                       | 0.1944               | 4.5596              | 6.4705               | 8.0601             |
| 8192        | 3.068                                   | 4.196                                     | 5.944                                   | 0.3750               | 0.2844                       | 0.3867               | 9.3851              | 12.657               | 16.2481            |


We uploaded our code as supplementary material and considered standard experimental choices to ensure reproducibility. We are happy to perform any other experiment that you might have in mind.

Reviewer comment (Reproducibility):
How this framework to give guidance to correct DP-SGD implementation if the algorithm did not pass the privacy auditing?

Author response:
**How this framework to give guidance to correct DP-SGD implementation if the algorithm did not pass the privacy auditing?**


Thanks for the good suggestion. Our framework can be extended to support identifying the cause of failure (i.e., which part of DP-SGD was not computed correctly) by generating separate proofs for each step of the DP-SGD. This comes at additional costs such as proving the consistency of intermediate values between proofs of each step. Given these complexities, we believe that a careful and thorough exploration is needed and we leave this to future work.