Paper: Avoiding Pitfalls for Privacy Accounting of Subsampled Mechanisms under Composition (ICLR-2024)

Reviewer comment (Theory):
Regarding pitfall b), the provided claim that the pair of distributions \(P = q \cdot \mathcal{N}(1,\sigma^2) + (1-q) \cdot \mathcal{N}(0,\sigma^2)\) and \(Q = q \cdot \mathcal{N}(-1,\sigma^2) + (1-q) \cdot \mathcal{N}(0,\sigma^2)\) forms a dominating pair in the case of subsampling without replacement and substitute neighbourhood relation of datasets is not correct. This is believed to be true for Poisson subsampling in the substitute neighbourhood case, and can be shown similarly to the Rényi divergence case using analysis for Poisson subsampling and add/remove neighborhood relation (as given in Mironov et al. (2019)). The correct bound for hockey-stick divergence in the case of subsampling without replacement and substitute neighbourhood relation is given in Proposition 30 by Zhu et al. (2022). This problem is thus solved, and major numerical accounting software (autodp, PRV accountant, Opacus, Google DP library) use correct formulas.

Author response:
Proposition 30 gives the upper bound of the hockey-stick divergence for a single iteration. However, it is not a dominating pair of distributions since the order of the pair differs between alpha below or above 1. In Section 7 we show that the pair of datasets considered in Proposition 30 is not sufficient to give tight bounds under composition for general mechanisms. You are correct that Zhu et al. give a construction of a dominating pair of distributions in Corollary 32. However, there is no implementation of this construction to the best of our knowledge. 
Regarding implementations of accountants for the subsampled Gaussian mechanism: the “PRV accountant”/Opacus (and PLD Accountant) only computes the “remove” relation. The “autodp” and Google DP library correctly computes both the “add” and “remove” relations and takes the point-wise maximum, as in Theorem 11 of Zhu et al. As mentioned in our response to Reviewer LTUo we conjecture that in the case of the Gaussian mechanism, it is sufficient to consider the “remove” datasets for subsampling without replacement under the substitution relation. However, we are not aware of any proof and therefore we disagree with the claim that this problem is solved.

Reviewer comment (Theory):
The claim that "Poisson subsampling is sometimes assumed to have similar privacy guarantees compared to sampling without replacement" is not true, because Proposition 30 by Zhu et al. (2022) provides bounds for both. While the bounds have the same form, in one case the pair \((P,Q)\) is a dominating pair under the add/remove relation, and in the other under the substitute relation. The latter leads to higher \(\varepsilon\)-values. The bounds are in two parts, and one can determine a numerical dominating pair using methods such as those by Doroshenko et al. (2022).

Author response:
We agree that this claim is too strong as is and should be rephrased. The privacy parameters are not assumed to be identical. However, the privacy parameters are sometimes assumed to be close as is the case between the add/remove and substitution neighboring relations. We base this claim on the fact that accountants for Poisson subsampling are often used when it is not the implemented sampling scheme.

Reviewer comment (Novelty):
The authors did not provide a viable technical solution for dealing with the worst-case dataset problem under replacement DP.

Author response:
Determining the worst-case dataset under replacement is a difficult problem in general as we show in Section 7. The technique by Zhu et al. can be used to construct a dominating pair of distributions. See the review and response to Reviewer G1qb for more details. Under subsampling without replacement without composition, the privacy curve of the Gaussian and Laplace mechanisms is the same under the substitution neighborhood as under the remove relation for $\varepsilon \geq 0$. As we show in Sections 5 and 7, that is not the case under composition for the Laplace mechanism. We conjecture that the curve under the substitution and remove relations matches under composition for the Gaussian mechanism. Proving or disproving this is a direction for future work. Nonetheless, the goal of Section 7 is to highlight that accounting for worst-case datasets under substitution is unsolved and that care should be taken when using numerical accounting under the substitution neighboring relation.

Reviewer comment (Experiments):
While constructing the bad cases on lower dimensional datasets is sufficient to demonstrate the claims, it would be nice to include empirical results on more realistic datasets to show that these pitfalls can actually appear in practice.

Author response:
We agree that understanding the privacy loss for actual datasets is an important problem. It would be interesting to e.g. examine how privacy auditing techniques for DP-SGD are affected by various sampling schemes using standard benchmark datasets. It is unlikely that the gradients resemble the worst-case datasets for sampling without replacement in all iterations. That is, all gradients have maximum length and the same direction. It is also unlikely that they resemble the worst-case datasets for Poisson subsampling where all but one datapoint is the zero vector. As such, it is not immediately clear how the sampling schemes compare empirically. However, we consider this a separate problem from tight privacy accounting.

Reviewer comment (Evaluation):
For replacement DP, is there a good way to find a good approximation of privacy curves when the exact worst-case dataset is hard to obtain?

Author response:
Determining the worst-case dataset under replacement is a difficult problem in general as we show in Section 7. The technique by Zhu et al. can be used to construct a dominating pair of distributions. See the review and response to Reviewer G1qb for more details. Under subsampling without replacement without composition, the privacy curve of the Gaussian and Laplace mechanisms is the same under the substitution neighborhood as under the remove relation for $\varepsilon \geq 0$. As we show in Sections 5 and 7, that is not the case under composition for the Laplace mechanism. We conjecture that the curve under the substitution and remove relations matches under composition for the Gaussian mechanism. Proving or disproving this is a direction for future work. Nonetheless, the goal of Section 7 is to highlight that accounting for worst-case datasets under substitution is unsolved and that care should be taken when using numerical accounting under the substitution neighboring relation.