Paper: Correlated Noise Provably Beats Independent Noise for Differentially Private Learning (ICLR-2024)

Reviewer comment (Theory):
Parts of the text are very hard to follow. For example, the text introduces the limit with T going to \infty, without explaining how this can make sense for a bounded privacy budget (i.e., for a finite privacy budget and an infinite (large) number of iterations, an infinite (large) amount of noise should be added in every step to the gradient, making it hard to converge).

Author response:
**[Technical flaw] “... for a finite privacy budget and an infinite (large) number of iterations, an infinite (large) amount of noise should be added in every step to the gradient”**

This assessment is incorrect. We prove bounds in the streaming setting, where each data point is processed only once. Thus, the number $n$ of data points and the number of iterations $T$ are equal. To see why a finite amount of noise is sufficient even as $T \to \infty$, consider the uncorrelated noise case (DP-SGD): the privacy guarantee for all iterations can be obtained from parallel composition, so the noise multiplier is independent of $T$. Prior work has also studied this setting, e.g. [Chourasia et al. NeurIPS 2021](https://arxiv.org/pdf/2102.05855.pdf) and [Altschuler & Talwar, NeurIPS 2022](https://openreview.net/pdf?id=pDUYkwrx__w).

For the more general correlated noise setting, Theorem 1.1 gives a rigorous privacy guarantee that holds any $T$, including in the limit $T \to \infty$. Note that the only dependence of the noise multiplier on $T$ is via the sensitivity $\gamma_T(\boldsymbol{B})$.

Reviewer comment (Theory):
While Eq (1) defines f(\theta, z) as objective function, Eq (6) seems to treat f as the derivative of the objective function.

Author response:
**"While Eq (1) defines f(\theta, z) as objective function, Eq (6) seems to treat f as the derivative of the objective function."**

The notation is correct — the objective for mean estimation is $f(\theta; z) = z^2 / 2 - z\theta$:
$$
\frac{1}{2} (\theta - z)^2 = \frac{z^2}{2} - z\theta + \frac{\theta^2}{2} = f(\theta; z) + r(\theta).
$$

Reviewer comment (Theory):
The proof of theorem 2.1 cites Fourier analysis, but doesn't make explicit what is the derivation the authors have in mind. There is even no proof that the asymptotic optimality defined by Eq (4) exists (i.e., the limit converges), which is non-trivial as the more iterations are performed, the higher the amount of noise per iteration needs to be.

Author response:
**"The proof of theorem 2.1 cites Fourier analysis, but doesn't make explicit what is the derivation the authors have in mind."**

The main paper only gives a short proof sketch (which provides an idea of why the statement is true), while the full proof and all its technical details are given in Appendix B. 

The technical details are based on standard results in Fourier analysis that we recall in Appendix F (specifically, Theorem F.2). The key idea of the proof is conveyed in that it is based on Fourier analysis, but we strongly feel that technical details regarding the Fourier transform would be a distraction in a short proof sketch.

**"There is even no proof that the asymptotic optimality defined by Eq (4) exists (i.e., the limit converges), which is non-trivial as the more iterations are performed, the higher the amount of noise per iteration needs to be."**

The reviewer is mistaken as we pointed out above. The limit exists in all cases we consider in the theoretical statements. The argument goes as follows:
- For a finite noise variance, the quantity in Eq. (4) is well-defined. This requires the limiting sensitivity $\gamma_\infty(B)$ (defined in Eq. (5)) and the clip norm $G$ to both be finite; indeed, by Theorem 1.1, the required noise variance is $\gamma_\infty(B)^2 G^2 / (2 \rho)$.
- The sensitivity $\gamma_\infty(B)^2 = log(1/\nu)$ is finite for $\nu$-DP-FTRL (see Table 2).
- For mean estimation in Theorem 2.1, we take $G=1$ and for general strongly convex functions in Theorem 3.1, we take $G$ to be the Lipschitz constant of the loss function, which is assumed finite. Thus, the noise variance of DP-FTRL is finite and the limit in Eq. (4) is well-defined.
- For quadratic functions, we analyze the asymptotic suboptimality of noisy-FTRL (i.e. DP-FTRL without gradient clipping) in Theorem 2.2; this asymptotic quantity is always well-defined. To get a DP bound in this case, (a) we argue for privacy that the gradient norm after $T$ iterations is bounded with high probability, and (b) the suboptimality after $T$ iterations is upper bounded by the asymptotic suboptimality at the same noise variance. The full privacy-utility bound of this case is given in Theorem D.13 in Appendix D.4.

Reviewer comment (Writing):
The text is insufficiently explicit, leading the reader to incorrect assumptions on what is meant. For example, while most of the text just uses "differential privacy", Appendix A.2 suddenly says that instead of "neighboring datasets" as most machine learning literature considers, the current paper considers zero-out neighborhood, where two sequences of gradients are adjacent if they only differ in one gradient. This is clearly not the case if we are performing an optimization and are comparing the algorithm being run on neighboring datasets, in which case (almost) every gradient will be different.

Author response:
**"insufficiently explicit.. Neighboring datasets … zero-out neighborhood."**

Differential privacy is always defined with respect to a particular definition of neighboring datasets. The zero-out notion of neighborhood is standard in the DP-FTRL literature: e.g. [Kairouz et al. ICML 2021](https://arxiv.org/pdf/2103.00039.pdf) (Def. 1.1), [Denisov et al. NeurIPS 2022](https://arxiv.org/pdf/2202.08312.pdf)  (Thm. 4.1) or their numerous follow-ups [[1](https://arxiv.org/pdf/2211.06530.pdf), [2](https://arxiv.org/pdf/2306.08153.pdf), [3](https://openreview.net/forum?id=qCglMj6A4z)]. We stick to this notion of neighborhood consistently throughout the paper in order to be consistent with prior work on DP-FTRL. We will be more explicit about this in the revision.

Also note that Sec. 2.1.1 of the practical guide of [Ponomareva et al. JAIR 2023](https://arxiv.org/pdf/2303.00654.pdf) says: 

> [add-or-remove one record] and [zero-out one record] have comparable semantics for a fixed $\varepsilon$

Finally, here is a quick proof sketch about why the differential privacy guarantee holds in the learning setting. The reviewer’s specific concern that once a single gradient differs, the entire input stream afterward may differ is addressed by Theorem 4.1 of Denisov et al., which says that computing $CG+W$ with $G$ being a stream of adaptive gradients satisfies the same DP guarantees as $G$ being a stream differing in a single entry between adjacent datasets. The high-level idea is if $x$ is the sensitive example, we can split $G$ into $G_1 + G_2$, where $G_1$ is the contribution to $G$ of all examples except $x$, and $G_2$ is the contribution of $x$. We release all intermediate values, and we can assume the **adversary knows all examples except $x$ and which rounds they participate in**, so the adversary can compute $CG_1$ and subtract it from $CG+W$ to get $CG_2 + W$. $G_2$ will be 0 everywhere for both $D$ and $D’$ except for the single round that $x$ participates in, so this gives a reduction from gradient streams to streams differing in a single entry.

Reviewer comment (Theory):
As a result of the change in definition, Appendix B seems to be not really proving theorem 2.1 but a variant where "neighboring datasets" is replaced by "zero-out neighbors", which changes the meaning of the theorem.

Author response:
**"insufficiently explicit.. Neighboring datasets … zero-out neighborhood."**

Differential privacy is always defined with respect to a particular definition of neighboring datasets. The zero-out notion of neighborhood is standard in the DP-FTRL literature: e.g. [Kairouz et al. ICML 2021](https://arxiv.org/pdf/2103.00039.pdf) (Def. 1.1), [Denisov et al. NeurIPS 2022](https://arxiv.org/pdf/2202.08312.pdf)  (Thm. 4.1) or their numerous follow-ups [[1](https://arxiv.org/pdf/2211.06530.pdf), [2](https://arxiv.org/pdf/2306.08153.pdf), [3](https://openreview.net/forum?id=qCglMj6A4z)]. We stick to this notion of neighborhood consistently throughout the paper in order to be consistent with prior work on DP-FTRL. We will be more explicit about this in the revision.

Also note that Sec. 2.1.1 of the practical guide of [Ponomareva et al. JAIR 2023](https://arxiv.org/pdf/2303.00654.pdf) says: 

> [add-or-remove one record] and [zero-out one record] have comparable semantics for a fixed $\varepsilon$

Finally, here is a quick proof sketch about why the differential privacy guarantee holds in the learning setting. The reviewer’s specific concern that once a single gradient differs, the entire input stream afterward may differ is addressed by Theorem 4.1 of Denisov et al., which says that computing $CG+W$ with $G$ being a stream of adaptive gradients satisfies the same DP guarantees as $G$ being a stream differing in a single entry between adjacent datasets. The high-level idea is if $x$ is the sensitive example, we can split $G$ into $G_1 + G_2$, where $G_1$ is the contribution to $G$ of all examples except $x$, and $G_2$ is the contribution of $x$. We release all intermediate values, and we can assume the **adversary knows all examples except $x$ and which rounds they participate in**, so the adversary can compute $CG_1$ and subtract it from $CG+W$ to get $CG_2 + W$. $G_2$ will be 0 everywhere for both $D$ and $D’$ except for the single round that $x$ participates in, so this gives a reduction from gradient streams to streams differing in a single entry.

Reviewer comment (Theory):
Before Eq (3) the text says that B is a Toeplitz matrix. How shall we read B(\omega) after Eq (3)? B is not a function, and the righthandside of B(\omega)=... evaluates to a complex number rather than a real-valued matrix.

Author response:
**"Before Eq (3) the text says that B is a Toeplitz matrix. How shall we read B(\omega) after Eq (3)?"**

Note that matrices are denoted with a bold $\boldsymbol{B}$, while non-bold $B$ is the discrete-time Fourier transform $B: [-\pi, \pi] \to \mathbb{C}$.

Reviewer comment (Theory):
Does the series \beta_t need to satisfy any property to make the sums and limits converge?

Author response:
**"Does the series \beta_t need to satisfy any property to make the sums and limits converge?"**

All the limits are well-defined as long as $\beta_t$’s are square-summable (denoted as $\boldsymbol{\beta} \in \ell^2$ in Theorem C.14, for instance). Some statements need the stronger assumption that $\beta_t$’s are absolutely summable (e.g. the lower bound in Theorem 2.2 or Theorem C.17), and these are explicitly pointed out (see e.g. the discussion at the top of p.28 right after Theorem C.17).

Reviewer comment (Writing):
How is \gamma_T defined? The text says that one can infer Eq (4) from this definition. In Eq 4, how shall we read the superscript 2? Does it square \gamma_T(B) or does it square its argument B? (In the former case, many people would write (\gamma_T(B))^2 or \gamma_T^2(B), in the latter case, most people would write \gamma_T(B^2)).

Author response:
**"How is \gamma_T defined?"**

$\gamma_T(\boldsymbol{B})^2 = \max_{t} \Vert (\boldsymbol{B}^{-1})_{:,t} \Vert_2$ is the maximum column norm of $\boldsymbol{B}$. This is defined in the statement of Theorem 1.1. 

**"The text says that one can infer Eq (4) from this definition."**
Do you mean Eq. (5)? Please see Property C.1 or Theorem F.3 for the proof of Eq. (5).

**"In Eq 4, how shall we read the superscript 2?"**

There is no superscript in Eq. (4). Do you refer to Eq. (5)? 
$\gamma_T(B)^2$ refers to $\big(\gamma_T(B)\big)^2$. Squaring the argument would be written as $\gamma_T(B^2)$. We believe there is no ambiguity between the two.

Reviewer comment (Theory):
My main concern is about the privacy guarantee. The authors use a high probability bound to argue that most of the time, the stochastic gradients won't exceed the clipping norm. Consequently, $\nu$-DP-FTRL doesn't need any gradient clipping. But I'm not sure if this meets the standard DP definition, as sensitivity might be prohibitively high in rare cases. To me, the link between the high probability result and the standard definition isn't clear. Once this is addressed, this paper has very solid results.

Author response:
**"$\nu$-DP-FTRL doesn't need any gradient clipping … sensitivity might be prohibitively high"**

Great question! The key is that the statement "most of the time, the stochastic gradients won’t exceed the clipping norm" is made for the unclipped gradients. However, we still use the clipped gradients in the algorithm, and so the formal DP guarantee _always holds_ but the utility guarantee is conditional on the unclipped and clipped gradients being the same (i.e. the event $\mathcal{E}$). This is because DP-FTRL always includes gradient clipping as shown in Algorithm 1. 

Please see Theorem D.13 in Appendix D for the detailed theorem statement.

Reviewer comment (Theory):
I'm curious about the role of $\nu$ in $\hat \beta_t^\nu$. Since $\binom{1/2}{t}$ in $\hat \beta_t^\nu$ already decreases rapidly (roughly $1/t$ if I'm correct), is it necessary to add an additional damping term $(1-\nu)^t$? Also, $\nu$ is currently set to some small value in the experiments, so $(1-\nu)^t$ decays significantly slower than $\binom{1/2}{t}$. Thus, I wonder if $\nu$ can be dropped to save some tuning effort. How would removing $\nu$ (or setting $\nu=0$) affect the theoretical bound and the empirical performance?

Author response:
**"role of $\nu$ in $\hat \beta_t^\nu$"**

It turns out that ${½ \choose t} \approx t^{-3/2}$ (this can be seen from Stirling’s approximation). However, this is not enough for theory or practice, as we explain below.

Firstly, a correction to the reviewer’s comment. The damping term $(1-\nu)^t \le \exp(-\nu t)$ is an exponential decay, which is asymptotically _faster_ (not slower as the reviewer notes) than the polynomial $t^{-3/2}$ (although the latter is faster for $t$ small). 

**Theory**: We need both these terms to get the right rate in theory. Note that $\nu=0$ corresponds to the weights of [Fichtenberger et al. (2023)](https://arxiv.org/pdf/2202.11205.pdf). As shown in Table 2, its sensitivity grows as $\log(T)$ — this is unbounded as $T \to \infty$. Thus, its asymptotic suboptimality is unbounded too. The exponential damping term is sufficient to achieve near-optimality for linear regression.

**Experiments**: We compare to $\nu=0$ experimentally under the name “Optimal CC” (see Table 3 for details). Our experiments show that despite tuning a restart parameter for Optimal CC, it is around 3pp worse than $\nu$-DP-FTRL for all levels of privacy for CIFAR-10.
Together, both these results show that the $(1-\nu)^t$ damping term is essential both in theory and practice.

Hope we answered all your concerns. We would be happy to address further questions. Thank you for your time!

Reviewer comment (Theory):
Most of the results are for Noisy-SGD and Noisy-FTRL, which are not DP as written in the paper.

Author response:
**"Most of the results are for Noisy-SGD and Noisy-FTRL, which are not DP as written in the paper."**

Theorem 2.1 for mean estimation and Theorem 3.1 for general strongly convex functions both hold for DP-FTRL and satisfy full DP guarantees.

We give the main results for linear regression for noisy-FTRL for simplicity. The corresponding finite-time privacy-utility guarantees for DP-FTRL are given in Theorem D.13 of Appendix D, where DP-FTRL has a similar advantage over DP-SGD as Noisy-FTRL over Noisy-SGD (effective dimension dependence instead of dimension, etc.). Further, all our deep learning results compare our $\nu$-DP-FTRL to DP-SGD and prior works in a non-convex setting where all methods satisfy the stated privacy guarantees.

Reviewer comment (Theory):
This paper uses lots of assumptions, and it seems difficult to adopt the results in practice.

Author response:
**"This paper uses lots of assumptions, and it seems difficult to adopt the results in practice."**

We respectfully disagree with this assessment. Our theoretical guarantees have two strengths:
- This is the first rigorous evidence that DP-FTRL is provably better than DP-SGD. This explains empirical observations in prior works that lacked the theoretical backing.
- Next, our theory is a guiding principle for further algorithmic developments. $\nu$-DP-FTRL, which we derived from our theoretical analysis, is **readily applicable in practice** for private deep learning. 

We empirically demonstrate that $\nu$-DP-FTRL outperforms all previous efficient DP-FTRL instantiations (Table 1) and DP-SGD for private image classification and federated language modeling. Furthermore, the simple structure of our correlated noise, which avoids the need to solve an SDP, actually makes it even easier to deploy in practice than the DP-FTRL instantiations proposed by [Denisov et al. (NeurIPS 2022)](https://arxiv.org/pdf/2202.08312.pdf) and [Choquette-Choo et al. (ICML 2023)](https://arxiv.org/abs/2211.06530).

Thus, our theoretical analysis seamlessly leads to practical improvements.

Reviewer comment (Experiments):
I understand it is a theory paper, but it would be useful to add more experimental results as they show the practical applicability.

Author response:
**"...add more experimental results..."**

In Section 4, we compare our method on the benchmark tasks established in the prior literature ([Kairouz et al. ICML 2021](https://arxiv.org/pdf/2103.00039.pdf)), ([Choquette-Choo et al. ICML 2023](https://arxiv.org/pdf/2211.06530.pdf)), ([Choquette-Choo et al. NeurIPS 2023](https://arxiv.org/pdf/2306.08153.pdf), [Koloskova et al. NeurIPS 2023](https://openreview.net/forum?id=qCglMj6A4z)): example-level DP for image-classification on CIFAR-10 and user-level DP for language modeling (next word prediction) on Stack Overflow. Since the submission, we have updated our results on Stack Overflow to include all mechanisms and find that $\nu$-DP-FTRL outperforms all efficient baselines (listed in Table 3) and even matches the multi-epoch upper bound.

If the reviewer has specific experiments they would like to see, we would be happy to add them.

Reviewer comment (Writing):
What is $\mathcal{E}$ inside the expectation on page 7?

Author response:
**"What is $\mathcal{E}$ inside the expectation on page 7?"**

$\mathcal{E}$ denotes the event when the norm of the gradients is under the clip norm $G$ (taken as in Eq. (9)) for all $T$ iterations. On this event, all the clipped gradients coincide exactly with the unclipped ones. As discussed in part (a) at the top of page 7, we have that $\mathbb{P}(\mathcal{E}) \ge 1-p$. The privacy guarantee always holds but the utility bound only holds under the event $\mathcal{E}$ (this approach is standard in the literature, see e.g. [Varshney et al. COLT 2022](https://arxiv.org/pdf/2207.04686.pdf)). We refer to Theorem D.13 in Appendix D for a full mathematical statement.

Reviewer comment (Experiments):
How to choose $\nu$ in general?

Author response:
**"How to choose $\nu$ in general?"**

We treat the parameter $\nu$ as a hyperparameter which we tune together with the optimizer hyperparameters (e.g. learning rate). In our experimental comparisons, we tune a “restart” parameter for all the baselines (denoted as $\times S$ in the legend), so the overall hyperparameter tuning cost of $\nu$-DP-FTRL is the same as that of the baselines. With the tuning costs equalized, we still find that $\nu$-DP-FTRL outperforms all other efficient baselines by a large margin.