Paper: Distributed DPHelmet: Differentially Private Non-interactive Convex Blind Averaging (ICLR-2024)

Reviewer comment (Novelty):
There have been several existing works on the combination of differential-privacy and secure aggregation. Essentially you can turn local DP into a central DP via secure aggregation. In particular, [1] (which is already included in the submission) gives an implementation of distributed DP mechanism that has matching utility with central DP. In Table 1, you claim that DP-FL has noise scale of $O(1/m\sqrt{n})$, which could be too large.

Author response:
We have updated the table in the updated submission, and have addressed the weaknesses of our original submission's Table 1 in a comment to reviewer 91JF.

Kairouz et al. (2021) match the utility and privacy bound of centralized DP-SGD training (in our experiments: DP-FL with 1 user) but require too many MPC invocations as they noise and average the gradient in each iteration and not the parameter output once after training (as we do). It also remains open whether their MPC protocol is able to scale to millions of users as Bell et al. (2020) does. Thus, we excluded this bound in Table 1 due to this huge scalability issue with the number of iterations and users.

Reviewer comment (Novelty):
If I understand correctly, 1 round of secure aggregation is enough because the average of local SVM models converges to global SVM optimal. This seems to limit the use case, because in practice we do not often use SVM in FL. Indeed we observe good performance in experiments, but it may also come from the use of pretrained feature extractor. More justification of the importance of SVM learning is highly appreciated.

Author response:
Our results show strong utility and privacy bounds for scalable distributed SVM learning (with only 1 invocation of secure summation). Hence, we consider our results a valuable contribution to scenarios where SVM learning is sufficient. To the question, it is useful if foundation models [R1], hand-crafted features, or application-specific kernels are available.

Additionally, we show strong distributed learning privacy results (privacy amplification via blind averaging due to an output sensitivity bound) for learning a Softmax-SLP, which is often used for fine-tuning the last layer of large neural networks.

[R1] Rishi Bommasani et al. "On the opportunities and risks of foundation models." arXiv preprint (2021).

Reviewer comment (Presentation):
In Table 1, in the row of DP-FL, "− (O(M) rounds)", why do you have the "-"?

Author response:
In Table 1 we have noted with "-" that DP-FL does not have MPC invocations but regular distributed training rounds. We updated Table 1 to clarify this point. In the response to reviewer 91JF, we have discussed the changes in Table 1 in more detail.

Reviewer comment (Evaluation):
The scope is narrowed to problems that can be learnt efficiently with SVM/SLP, and it scales poorly with the number of classes as seen on CIFAR100.

Author response:
> The scope is narrowed to problems that can be learnt efficiently with SVM/SLP [...]

This focus on SVM/Softmax-SLP allows us to prove an output sensitivity bound, which enables privacy amplification via blind averaging (Lemma 6) – a strong property for distributed learning. Such privacy amplification via blind averaging is not known for non-convex models. We even show an output sensitivity bound for fine-tuning large models with retraining their last layer with Softmax.

Leveraging pre-training feature extractors has been proven promising for finding strong tradeoffs between expressivity, accuracy, and privacy guarantees ([R1], Tramèr and Boneh (2021), De et al. (2022)).

[R1] Rishi Bommasani et al. "On the opportunities and risks of foundation models." arXiv preprint (2021).

Reviewer comment (Theory):
The privacy results doesn't seem a big contribution from the mathematical point of view, and the proofs are a bit messy (see below).

Author response:
> The privacy results doesn't seem a big contribution from the mathematical point of view, and the proofs are a bit messy (see below)

In terms of theoretical contribution, we provide the first output sensitivity result for Softmax-SLP (Theorem 11), by showing $L$-Lipschitzness (Theorem 27) and $\beta$-smoothness (Theorem 28) for Softmax-SLP.

An output sensitivity bound is a sufficient condition for the privacy amplification via blind averaging (cf. Lemma 6). In other words, mechanisms with output sensitivity lead to privacy guarantees in a setting where the parties learn locally, but only add noise in the magnitude of centralized learning.

Reviewer comment (Evaluation):
The privacy results are only at the row level, and not at the user level. Overall, motivation and real-use cases for this privacy and communication setting could have been more developed.

Author response:
> The privacy results are only at the row level, and not at the user level. Overall, motivation and real-use cases for this privacy and communication setting could have been more developed

Our submission does present several flavors of person level privacy results, as we agree that person-level privacy is important. In use-cases where local aggregators collect data from people in their vicinity the row-based approach coincides with the person-level. In use-cases where all rows of a user belong to the same person, our submission does report user-level results: technically, we first showed guarantees for group-DP in Corollary 9 which then directly results in user level guarantees (cf. Corollary 10). We also evaluated this setting experimentally in Figure 5 (Appendix A) which demonstrates the applicability of our method to user-level privacy as long as there are enough users. In this evaluation, each user holds 50 data points, but the number (and exact training procedure) can differ from one user to another.

Reviewer comment (Experiments):
No heterogeneity is tackled, despite the fact it is likely to be an issue in real-use cases.

Author response:
> No heterogeneity is tackled, despite the fact it is likely to be an issue is real-use cases. Have you tried to do experiments with some heterogeneity and other datasets? Even 'fake' heterogeneity with just class unbalance would be a good complement

We tackled a strongly-biased non-iid scenario in Table 2 which represents one of the worst-case distributions of data points among users. Here each user has only exclusive access to one class.

We would like to emphasize that privacy-preserving learners have been shown to lead to discrimination against minority classes [R2].

[R2] Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. "Differential privacy has disparate impact on model accuracy." NeurIPS, 32, 2019.

Reviewer comment (Experiments):
Have you try to do experiments with some heterogeneity and other datasets? Even "fake" heterogeneity with just class unbalance would be a good complement.

Author response:
> No heterogeneity is tackled, despite the fact it is likely to be an issue is real-use cases. Have you tried to do experiments with some heterogeneity and other datasets? Even 'fake' heterogeneity with just class unbalance would be a good complement

We tackled a strongly-biased non-iid scenario in Table 2 which represents one of the worst-case distributions of data points among users. Here each user has only exclusive access to one class.

We would like to emphasize that privacy-preserving learners have been shown to lead to discrimination against minority classes [R2].

[R2] Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. "Differential privacy has disparate impact on model accuracy." NeurIPS, 32, 2019.

Reviewer comment (Theory):
I have quickly browsed the proofs, I saw that Lemma 24 is the usual technique of scaling by the sensitivity constant. The proof with developing the ratio seems unnecessarily difficult and p27 in the calculation of the line starting by "due to the Cauchy-Schwarz inequality", you put square on the norms, but there are not. Maybe going through the appendix and simplifying or highlighting the key points could help.

Author response:
> I have quickly browsed the proofs, I saw that Lemma 24 is the usual technique of scaling by the sensitivity constant. The proof with developing the ratio seems unnecessary difficult and p27 in the calculation of the line starting by "due to the Cauchy-Schwarz inequality", you put square on the norms, but there are not. Maybe going through the appendix and simplify or highlights the key points could help.

We use Lemma 24 to prove the bound in our group privacy variant (cf. Corollary 9), which we use to prove that user-level privacy holds with strong utility-privacy tradeoffs. We agree that the argumentation can be simplified by analyzing the sensitivity in the group privacy setting.

At p27 we take the square root on the norm to allow us to put squares on the norms.

Reviewer comment (Evaluation):
It is unclear whether the utilities for different methods in Table 1 are correct.

Author response:
We agree that our statements in Table 1 on the utility bounds were not helpful. We have replaced those with what we deem more relevant and to the point: whether or not blind averaging works, i.e. whether, if no noise is added, the approach converges to the model we would train centrally.

DP federated learning (DP-FL) and Jayaraman et al.'s gradient perturbation just train a global model which by definition is the same as the centralized one. For our SVM result, we have shown this convergence formally in Theorem 14. For our Softmax-SLP result, we do not have a formal proof, but have indicated it experimentally and so have Jayaraman et al. for their output perturbation result. The convergence rate of Jayaraman et al. (2018, Theorem 3.2) depends on the number of data points whereas we truly converge with the number of iterations (cf. Theorem 14).

We also divided Table 1 by the algorithms, i.e. SVM and Softmax-SLP, which have different use-cases and face different challenges. For more details on these differences see our answer for reviewer VsvY ("Softmax-SLP scales with number of classes" and "Non-trivial privacy results").

Reviewer comment (Reproducibility):
The computation and memory cost of the secure summation is unclear.

Author response:
The computation and memory costs of secure summation depend on the particular secure summation variant. If you use Bell et al. (2020) we provided the computation costs in the extended preliminaries in Appendix C.2 and for the example of DPHelmet in Section 6, "Computation costs". Most notably, in DPHelmet's SVM-SGD ($1000$ users) we only transmit data of $l \approx 100{,}000$ floating points.

We have not measured the memory costs of Bell et al.'s protocol.

Reviewer comment (Presentation):
In Table 1, where are the references for the DP-FL, Centralized training, and the utility for your proposed method?

Author response:
Centralized training refers to Algorithm 1 (SVM) and 3 (Softmax-SLP) respectively. As discussed in our introduction (cf. Section 1) and extended related work in Appendix D.1, DP-FL refers to a combination of federated learning (McMahan et al., 2017) with DP-SGD (Abadi et al., 2016). For the utility of our proposed method, see above "Utility bounds in Table 1".

Reviewer comment (Theory):
I find it is very hard to follow the main results in the current paper. For example, in Corollary 4, why do you have the constraint on $\epsilon$ and what is the meaning of adding random noise to a set of outputs and why do you need to define $I_d$?

Author response:
Corollary 4 directly follows from Lemma 3 where we rearranged the noise scale restriction to a restriction on the epsilon. We add $(p+1)\times |K|$-dimensional noise to a $|K|$-sized set of $(p+1)$-dimensional outputs. Formally, we add for each element in the $|K|$-sized set $(p+1)$-dimensional noise. We defined $I_d$ to prevent any confusion of our theorems.

Reviewer comment (Theory):
How will the number of local updates $M$ affect your privacy and utility guarantees?

Author response:
The privacy guarantees are independent of the number of local updates $M$ since we consider convex, Lipschitz, and $\beta$-smooth optimization problems. In Lemma 2, we recall Wu et al. (2018, Lemma 8) that show for a strongly convex optimization objective, e.g. SVM, LR, or Softmax-SLP, with a linear decreasing learning rate, that the noise scale is independent of the number of iterations M. 

Concerning utility, our convergence result of SVM-training in Theorem 14 shows that the average of locally trained SVM converges with rate $1/M$ to the best model for the combined local datasets $\mho$ where $M$ are the number of local iterations.

Reviewer comment (Theory):
How will the rescaling step in your algorithms (e.g., projected SGD) affect your utility guarantees?

Author response:
Projected SGD is a requirement for the convergence rate of $1/M$ (cf. Proof of Theorem 14 and in particular the result of Locoste-Julien et al., 2012) as well as for a useful bound on differential privacy (cf. Wu et al., 2017). For privacy, we need a bounded hypothesis space which is also reflected by the noise scale which depends on the radius of the hypothesis space $R$. With projected SGD we can enforce such a bounded hypothesis space.

Reviewer comment (Writing):
What is SimCLR, and how will it affect your results?

Author response:
The effect of SimCLR is illustrated in Figure 6 in Appendix C.3 and described there as well. In essence, it reduces the dimensionality of the image data via a deep neural network that has been trained on ImageNet. The prediction of SimCLR gives us a $6{,}144$d embeddings vector on which we perform SVM, Softmax-SLP, and DP-FL. This technique goes in hand with the privacy literature (Tramèr and Boneh (2021), De et al. (2022)) to boost the privacy-utility tradeoff.

Reviewer comment (Reproducibility):
How do you implement secure aggregation?

Author response:
The implementation of a concrete secure aggregation protocol is not in the scope of this work.
Also note, that our work does not require a specific secure summation technique and is agnostic to different secure summation implementations.

Reviewer comment (Writing):
The phrase "the size of each communication round" is insufficiently precise. Please define the "size of a round". There are probably less ambiguous formulations such as "the communication cost of a round", "the computation cost of a round", "the number of messages in every round", etc.

Author response:
We clarified all your points throughout our paper in an updated version.
- We agree, $O((\sum_i |D_i|)^{-1})$ is better.
- With "size of each communication round" we meant "communication cost of each round".
- Yes, M refers to the number of iterations. This means that DP-FL needs one communication round per training iteration.
- You are correct: the number of MPC invocations in Jayaraman et al. (2018, Theorem 3.6) scales with $\mathcal{O}(\log(nm))$ to reach their utility bound. Note that in our scheme we can train until convergence without sacrificing utility by stopping early. They have to stop early since their noise scales with the square root of the number of iterations (Theorem 3.4 in Jayaraman et al. (2018)). In their Section 3.2 above Theorem 3.4, they stated that they perform one MPC invocation per iteration.
- The parameter h in Algorithm 1 scales the smoothness of the Huber loss (cf. Definition 21).
- We are now more precise in Algorithm 1: we added $f^{(k)}_{m-1}$ as an additional input to the function "SGD".
- $1920$ is the number of iterations we used for DP-FL and is loosely adapted to the ones reported by Tramèr and Boneh (2021); it consists of $40$ epochs and $48$ iterations per epoch ($40\cdot48=1920$).

Reviewer comment (Writing):
In the statement "It does need a communication round per training iteration M." -> M is undefined and looks here like a variable representing a training round. From the use of M much later in the text, I guess you mean "Let M be the number of training iterations. It (the algorithm?) needs only one communication round per training iteration."

Author response:
We clarified all your points throughout our paper in an updated version.
- We agree, $O((\sum_i |D_i|)^{-1})$ is better.
- With "size of each communication round" we meant "communication cost of each round".
- Yes, M refers to the number of iterations. This means that DP-FL needs one communication round per training iteration.
- You are correct: the number of MPC invocations in Jayaraman et al. (2018, Theorem 3.6) scales with $\mathcal{O}(\log(nm))$ to reach their utility bound. Note that in our scheme we can train until convergence without sacrificing utility by stopping early. They have to stop early since their noise scales with the square root of the number of iterations (Theorem 3.4 in Jayaraman et al. (2018)). In their Section 3.2 above Theorem 3.4, they stated that they perform one MPC invocation per iteration.
- The parameter h in Algorithm 1 scales the smoothness of the Huber loss (cf. Definition 21).
- We are now more precise in Algorithm 1: we added $f^{(k)}_{m-1}$ as an additional input to the function "SGD".
- $1920$ is the number of iterations we used for DP-FL and is loosely adapted to the ones reported by Tramèr and Boneh (2021); it consists of $40$ epochs and $48$ iterations per epoch ($40\cdot48=1920$).

Reviewer comment (Evaluation):
In Table 1, where log(M) is used, looking in the cited paper suggests you probably mean the logarithm of the number of users rather than the number of training iterations, but even then this seems to represent the number of rounds per training iteration rather than the global number of MPC invocations.

Author response:
We clarified all your points throughout our paper in an updated version.
- We agree, $O((\sum_i |D_i|)^{-1})$ is better.
- With "size of each communication round" we meant "communication cost of each round".
- Yes, M refers to the number of iterations. This means that DP-FL needs one communication round per training iteration.
- You are correct: the number of MPC invocations in Jayaraman et al. (2018, Theorem 3.6) scales with $\mathcal{O}(\log(nm))$ to reach their utility bound. Note that in our scheme we can train until convergence without sacrificing utility by stopping early. They have to stop early since their noise scales with the square root of the number of iterations (Theorem 3.4 in Jayaraman et al. (2018)). In their Section 3.2 above Theorem 3.4, they stated that they perform one MPC invocation per iteration.
- The parameter h in Algorithm 1 scales the smoothness of the Huber loss (cf. Definition 21).
- We are now more precise in Algorithm 1: we added $f^{(k)}_{m-1}$ as an additional input to the function "SGD".
- $1920$ is the number of iterations we used for DP-FL and is loosely adapted to the ones reported by Tramèr and Boneh (2021); it consists of $40$ epochs and $48$ iterations per epoch ($40\cdot48=1920$).

Reviewer comment (Reproducibility):
In Algorithm 1, in every iteration, f_m^{(k)} is computed, but one would expect that f_m^{(k)} depends on f_{m-1}^{(k)}, i.e., the result of the previous iteration. This can't be seen in the code.

Author response:
We clarified all your points throughout our paper in an updated version.
- We agree, $O((\sum_i |D_i|)^{-1})$ is better.
- With "size of each communication round" we meant "communication cost of each round".
- Yes, M refers to the number of iterations. This means that DP-FL needs one communication round per training iteration.
- You are correct: the number of MPC invocations in Jayaraman et al. (2018, Theorem 3.6) scales with $\mathcal{O}(\log(nm))$ to reach their utility bound. Note that in our scheme we can train until convergence without sacrificing utility by stopping early. They have to stop early since their noise scales with the square root of the number of iterations (Theorem 3.4 in Jayaraman et al. (2018)). In their Section 3.2 above Theorem 3.4, they stated that they perform one MPC invocation per iteration.
- The parameter h in Algorithm 1 scales the smoothness of the Huber loss (cf. Definition 21).
- We are now more precise in Algorithm 1: we added $f^{(k)}_{m-1}$ as an additional input to the function "SGD".
- $1920$ is the number of iterations we used for DP-FL and is loosely adapted to the ones reported by Tramèr and Boneh (2021); it consists of $40$ epochs and $48$ iterations per epoch ($40\cdot48=1920$).

Reviewer comment (Experiments):
Just after Algorithm 1: How do you get to the specific number of "1920 rounds?"

Author response:
We clarified all your points throughout our paper in an updated version.
- We agree, $O((\sum_i |D_i|)^{-1})$ is better.
- With "size of each communication round" we meant "communication cost of each round".
- Yes, M refers to the number of iterations. This means that DP-FL needs one communication round per training iteration.
- You are correct: the number of MPC invocations in Jayaraman et al. (2018, Theorem 3.6) scales with $\mathcal{O}(\log(nm))$ to reach their utility bound. Note that in our scheme we can train until convergence without sacrificing utility by stopping early. They have to stop early since their noise scales with the square root of the number of iterations (Theorem 3.4 in Jayaraman et al. (2018)). In their Section 3.2 above Theorem 3.4, they stated that they perform one MPC invocation per iteration.
- The parameter h in Algorithm 1 scales the smoothness of the Huber loss (cf. Definition 21).
- We are now more precise in Algorithm 1: we added $f^{(k)}_{m-1}$ as an additional input to the function "SGD".
- $1920$ is the number of iterations we used for DP-FL and is loosely adapted to the ones reported by Tramèr and Boneh (2021); it consists of $40$ epochs and $48$ iterations per epoch ($40\cdot48=1920$).

Reviewer comment (Theory):
Just after Algorithm 1: How do you get to the specific number of "1920 rounds?" (more generally, I understand little of the provided argument here as the cited papers not always allow for finding easily the claim for which they are cited).

Author response:
We clarified all your points throughout our paper in an updated version.
- We agree, $O((\sum_i |D_i|)^{-1})$ is better.
- With "size of each communication round" we meant "communication cost of each round".
- Yes, M refers to the number of iterations. This means that DP-FL needs one communication round per training iteration.
- You are correct: the number of MPC invocations in Jayaraman et al. (2018, Theorem 3.6) scales with $\mathcal{O}(\log(nm))$ to reach their utility bound. Note that in our scheme we can train until convergence without sacrificing utility by stopping early. They have to stop early since their noise scales with the square root of the number of iterations (Theorem 3.4 in Jayaraman et al. (2018)). In their Section 3.2 above Theorem 3.4, they stated that they perform one MPC invocation per iteration.
- The parameter h in Algorithm 1 scales the smoothness of the Huber loss (cf. Definition 21).
- We are now more precise in Algorithm 1: we added $f^{(k)}_{m-1}$ as an additional input to the function "SGD".
- $1920$ is the number of iterations we used for DP-FL and is loosely adapted to the ones reported by Tramèr and Boneh (2021); it consists of $40$ epochs and $48$ iterations per epoch ($40\cdot48=1920$).

Reviewer comment (Experiments):
What evidence is there is the extent to which the proposed method is applicable to more than just a few simpler datasets satisfying some desirable properties (being balanced, having little noise, being separable, ...)?

Author response:
For SVMs, we show in Theorem 14 a general convergence statement for SVM for a large enough $\Lambda$. Our non-iid intuition in Figure 2 also demonstrates this on a highly unbalanced, non-separable dataset.

Currently, all our experiments speak in our favor as CIFAR-10 and CIFAR-100 datasets are highly unbalanced on a per-SVM basis (for SVMs, we use the one-vs-rest approach, where we train n\_classes SVMs and weigh each class against all others). For SVMs and Softmax-SLP, we analyzed in Table 2 how our method performs in a setting where the local training data sets are unbalanced and non-iid variants of the CIFAR-10 and CIFAR-100 datasets.

We would like to emphasize that privacy-preserving learners have been shown to lead to discrimination against minority classes [R2].

Good separability can often be offered by foundation models like SimCLR (Chen et al. 2020b) where the authors suggest fine-tuning a linear layer. These foundation models have gained increasing popularity and relevance in practice [R1].

We are not sure what you meant by "having little noise". Requiring little DP noise is not an issue in our experiments as we have strong results for $\varepsilon=0.1$ (cf. Figure 3) and even $\varepsilon=2 \cdot 10^{-5}$ (cf. Figure 5). This is due to our sensitivity bound which scales by $\frac{1}{nm}$.

[R1] Rishi Bommasani et al. "On the opportunities and risks of foundation models." arXiv preprint (2021).
[R2] Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. "Differential privacy has disparate impact on model accuracy." NeurIPS, 32, 2019.