Paper: Private Overparameterized Linear Regression without Suffering in High Dimensions (ICLR-2024)

Reviewer comment (Theory):
The assumption that we know the covariance of the features $H$ or that we can somehow get an approximation from the public data is quite strong. I think it would make the paper much stronger if there was some version that makes a DP approximation of $H$.

Author response:
Note that we do not assume that our algorithm knows the covariance of the features $\mathbf{H}$. 
In addition, if we do not have a public dataset, we can use the isotropic noise $\mathbf{\Sigma}=\mathbf{I}$ in our method to still get excess risk bounds (see Corollary 4.3, Corollary 4.4) that depend on the eigenspectrum of the data covariance instead of the problem dimension as those in Varshney et al. (2022), Liu et al. (2023). The idea of using public data in the DP methods has been widely used in the DP literature, e.g., Kairouz et al. (2020), Yu et al, (2022), Zhou et al. (2021). Moreover, getting unlabeled public data is generally cheaper than labeled training data. It is a good idea to develop a DP approximate $\mathbf{H}$, and we leave it as our future work.

Reviewer comment (Theory):
I think one argument against the baseline method of Varshney et al. (2022) is weak: namely that you can get rid of the restriction on $\varepsilon$ by using DP-FTRL instead of the shuffling approach. This argument is mentioned few times. As far as I see, the condition on $\varepsilon$ for the shuffling approach of Varshney et al. comes from the analysis of Feldman et al. (2022) for shufflers, where the $(\varepsilon,\delta)$-bounds are obtained by bounding the $(\varepsilon,\delta)$-distance between certain binomial distributions. And the condition on $\varepsilon$ then comes from the analysis that they use to obtain convenient analytical bounds. I believe one could have for the shuffler $(\varepsilon,\delta)$-bounds with arbitrarily small $\varepsilon$'s, just the way you have DP bounds for DP-FTRL. My point is that this restriction is not inherent limitation of the shuffler approach, just of the particular analytical bounds, and it would be possible to get bounds with small $\varepsilon$'s.

Author response:
Firstly, we would like to clarify the restriction of $\epsilon$ in Varshney et al. (2022). On page 6, it should be $\epsilon\leq 1/\sqrt{N}$ instead of $\epsilon\leq \sqrt{N}$. Therefore, the  $\epsilon$ in Varshney et al. (2022) is very restrictive since they can only choose a very small $\epsilon$. Secondly, we believe that this restriction is essential when applying the shuffling result (Feldman et al. (2022)). The reason that they have this restriction is as follows. In their proofs, they first show that each step in their method is $(\epsilon_0,\delta_0)$-DP. Then they use the shuffling amplification result to show that their method is $(\epsilon,\delta)$-DP with $\epsilon=\epsilon_0/\sqrt{N}$. Note that this shuffling amplification result requires $\epsilon_0\leq 1$, and thus they have the restriction $\epsilon\leq 1/\sqrt{N}$. The good thing about this shuffling amplification result is that one only needs to add random noise with the variance at the order of $1/\epsilon_0^2=1/(\epsilon^2 N)$ at each step to achieve $(\epsilon,\delta)$-DP. As a result, one can significantly improve the utility guarantee due to the $1/N$ factor in variance for the random noise. In order to get rid of this restriction, one needs to choose a large $\epsilon_0$. However, according to Feldman et al. (2022) (comments under Theorem 3.1 and also in Appendix B), when $\epsilon_0$ is larger than the order of $\log N$, there is no amplification effect, resulting in $\epsilon=\epsilon_0$. As a result, one needs to add random noise with the variance at the order of $1/\epsilon_0^2=1/\epsilon^2$ at each iteration to achieve $(\epsilon,\delta)$-DP. This leads to a substantially weaker utility guarantee, as the beneficial $1/N$ factor in the noise’s variance no longer applies. Therefore, we believe this restriction is essential in the shuffling result to achieve a strong utility guarantee since it does not allow local privacy parameter $\epsilon_0$ to be large.

Reviewer comment (Writing):
There are typos in summation indices in several places, e.g. p. 3: 
$$
\sum_{k_1 < k \leq k_2} \mu_i v_i v_i^T
$$
and
$$
\sum_{j=0}^T g_i
$$
in several places at least on p. 5.

Author response:
Thanks for pointing out those typos and the bibliography issue, we will fix them in the revision.

Reviewer comment (Presentation):
Check the bibliography. Example: you give the Arxiv versions of Varshney et al. (2022) and Karwa and Vadhan (2018) as references. They have appeared in COLT and ITCS, respectively.

Author response:
Thanks for pointing out those typos and the bibliography issue, we will fix them in the revision.

Reviewer comment (Writing):
In the abstract and introduction, the authors claim that their convergence rates "improve on previous work on DP linear regression", even more so when using general covariance matrices in the Gaussian mechanism. However, these claims should be toned down as there is missing context. Specifically, the covariance matrix used in the Gaussian mechanism to achieve the best rates requires access to (unlabeled) public data, which is never mentioned in the abstract or introduction. The best rates stated in the introduction are achieved only with public data, not solely from the original data, and this should be explicitly mentioned.

Author response:
>(Major problem) access to (unlabeled) public data is not ever mentioned in the abstract or introduction.

We indeed mention in the abstract that “Furthermore, when **unlabeled public data** is available, we can design a better noise covariance matrix structure to improve the utility. ” We will also add comments about using unlabeled public data in the introduction section when we state our main contributions (the second bullet).

Reviewer comment (Novelty):
While the authors improve on rates from previous work, there is missing nuance. For example, Liu et al. (2023) aimed for algorithms robust to a small proportion of adversarial corruptions, and both Varshney et al. (2022) and Liu et al. (2023) assume less about the data distribution (except possibly Theorem 4.2, where comparisons are unclear). Additionally, the hidden constants imply that beyond requiring \(d > N\), \(N\) must be significantly large for the rates to actually improve over those with dependency on \(d\). These points do not invalidate the contribution but should be discussed for fair placement of the submission's contributions in the literature.

Author response:
>(Minor problem) a fair placement of the contributions of the submission in the literature

We agree with the reviewer that we need to have a fair placement of our work in the literature. It is true that our results and the results in Varshney et al. (2022), Liu et al. (2023) may be not directly comparable. For example, our results have an explicit dependence on the estimated residual $\ell$ while Varshney et al. (2022), Liu et al. (2023) get rid of this dependence by establishing a feasible upper bound for this term, assuming that the data distribution has a nice property, i.e., the condition number of the data covariance matrix is finite. 

However, we would also like to clarify that our results are at least more reasonable and meaningful in the high-dimensional or even infinite-dimensional setting. Our results, stated as a function directly with respect to the data covariance and noise covariance, can still provide nonvaucous excess risk bounds even when $d$ is very large or even infinite (see our Corollary 4.4 and Corollary 4.6). In contrast, the prior bounds in  Varshney et al. (2022), Liu et al. (2023)  have an explicit dependence on $d$, which clearly become vacuous in the high-dimensional or infinite-dimensional case. Therefore, we will revise our comments about the relationship of our work with that of Varshney et al. (2022), Liu et al. (2023) to ensure a fair placement of our work in the literature. For example, rather than claiming that our method outperforms previous method, we will state that our method provides the first problem- and algorithm- dependent excess risk bound for private linear regression.

Reviewer comment (Novelty):
The results, although interesting, seem somewhat incremental as they rely heavily on previous results, and it is unclear what extra technical difficulties were overcome. A stronger submission would better delineate the technical challenges of combining the several techniques used. The main contribution seems to be extending benign overfitting rates from Zou et al. (2021b) to the private setting via DP-SGD with adaptive gradient clipping from Liu et al. (2023), but implemented through DP-FTRL to avoid shuffling/random sampling. Since DP-FTRL can be written as a gradient update, the analysis closely follows Zou et al. (2021b), making it difficult to identify which aspects are novel and technically challenging.

Author response:
>technical difficulties and analytical tools

As you said, the reason that we propose to use DP-FTRL based method instead of the DP-SGD based method is to prevent the need to use shuffling/random sampling to get good dependency on privacy parameters. The main reason that we do not follow the DP-FTRL based analysis is that it cannot ensure us to achieve problem- and algorithm- dependent excess risk bound. Therefore, to achieve the desired excess risk bound, we instead resort on the fact that DP-FTRL can be written as a gradient update (Sec B.2.2) to actually more closely follow the analysis from Zou et al (2021b).  However, directly applying their analysis will lead to an unfavorable dependence on the problem dimension $d$ when the additive noise is introduced. Therefore, we have developed highly non-trivial new techniques to address this issue and get an excess risk bound that can get rid of the explicit dimension dependence. In the following, we summarize the key technique difficulties in our analysis and the new analytical tools we have developed to address these difficulties:

1. Although DP-FTRL can be written as a gradient update (Sec B.2.2), the additive noise at each iteration $\mathbf{\xi}$ (see equation (B.4)) is correlated with previous iterations. This is significantly different from DP-SGD, where independent random noise is added at each iteration. Therefore, we cannot simply factor out the additive noise using the independence of the additive noise like the one used in the DP-SGD analysis. This correlation will introduce extra errors (see the second equation under equation (B.14)), which are challenging to be upper bounded and have never been studied in the analysis of Zou et al (2021b). To overcome this difficulty, we propose to characterize the properties of the additive noise generated by the tree aggregation procedure, and develop new proof techniques (see Sec B.4) as well as new results (see Lemma B.12) to characterize these extra error terms (see equation (B.16) and Theorem B.5). These new techniques and results enable us to derive the new bias-variance decomposition expression for each iteration in our proposed DP-FTRL based method (Theorem B.5).

 2. Given the new bias-variance decomposition expression (Theorem B.5) for each iteration, we still cannot directly apply the proof techniques in Zou et al (2021b) to obtain the desired excess risk bound. The reason is that the extra errors introduced by the private mechanism depend on the general covariance matrix $\mathbf{\Sigma}$ and will affect the variance error (see Lemma B.6). As we discussed under Lemma B.6, the proof techniques in Zou et al (2021b) will make the variance error explode since the general covariance matrix $\mathbf{\Sigma}$ will no longer be well aligned with $\mathbf{H}$, a condition crucial for the proofs in in Zou et al (2021b)’s proofs. Therefore, to provide a tight upper bound on the variance error, we derive new results (Lemma B.7 and Lemma B.8) to handle the general covariance matrix $\mathbf{\Sigma}$ case. 

In summary, to obtain the excess risk bound of our proposed method, we need to develop new proof techniques that can handle the correlated noise and the variance error introduced by the general covariance matrix of the additive noise. We believe our new analytical tools could be of independent interest due to their effectiveness in addressing the challenges posed by correlated noise and general covariance matrices.

Reviewer comment (Theory):
The beginning of the proof of Theorem 4.1 (Sec B.1) focuses on showing that tree aggregation is private. However, tree aggregation for privately releasing partial sums is already well studied, and it is unclear why the analysis needs to be redone. The use of a general \(\Sigma\) matrix in the Gaussian mechanism should not necessitate a completely new analysis — adding noise \(\mathcal{N}(0, \Sigma)\) to \(x\) is equivalent to adding \(\mathcal{N}(0, I)\) to \(\Sigma^{-1/2} x\) and then post-processing by multiplying by \(\Sigma^{1/2}\). This confusion extends to other parts of the analysis, making it hard to distinguish between results from previous work and key technical insights from the authors.

Author response:
>why the analysis of tree aggregation needs to be redone.

It's important to note that the tree aggregation protocol used in our method is slightly different from those used in prior work (e.g., Guha Thakurta & Smith (2013),  Kairouz et al. (2021)). 
Specifically, our method proposes to add different amounts of noise to each node in the tree (see line 7 in Algorithm 1 and line 11 in Algorithm 3) while the previous work proposes to add the same amount of noise to each node in the tree. This is due to the adaptive clipping step (see line 6 in Algorithm 1) used in our method, leading to different norm bounds for different nodes. Therefore, we cannot directly apply existing results (such as those in Guha Thakurta & Smith (2013),  Kairouz et al. (2021) ) to provide the privacy guarantee of our tree aggregation protocol. We will clarify this in the revision.

Reviewer comment (Theory):
Question — Can the authors briefly mention the specific technical challenges they had to overcome? The analysis does not appear to simply follow the benign overfitting analysis from 2021 with extra privacy terms, but the appendix makes it hard to identify the key technical contributions. The tree aggregation privacy analysis is a good example to clarify whether a new proof was actually necessary.

Author response:
>technical difficulties and analytical tools

As you said, the reason that we propose to use DP-FTRL based method instead of the DP-SGD based method is to prevent the need to use shuffling/random sampling to get good dependency on privacy parameters. The main reason that we do not follow the DP-FTRL based analysis is that it cannot ensure us to achieve problem- and algorithm- dependent excess risk bound. Therefore, to achieve the desired excess risk bound, we instead resort on the fact that DP-FTRL can be written as a gradient update (Sec B.2.2) to actually more closely follow the analysis from Zou et al (2021b).  However, directly applying their analysis will lead to an unfavorable dependence on the problem dimension $d$ when the additive noise is introduced. Therefore, we have developed highly non-trivial new techniques to address this issue and get an excess risk bound that can get rid of the explicit dimension dependence. In the following, we summarize the key technique difficulties in our analysis and the new analytical tools we have developed to address these difficulties:

1. Although DP-FTRL can be written as a gradient update (Sec B.2.2), the additive noise at each iteration $\mathbf{\xi}$ (see equation (B.4)) is correlated with previous iterations. This is significantly different from DP-SGD, where independent random noise is added at each iteration. Therefore, we cannot simply factor out the additive noise using the independence of the additive noise like the one used in the DP-SGD analysis. This correlation will introduce extra errors (see the second equation under equation (B.14)), which are challenging to be upper bounded and have never been studied in the analysis of Zou et al (2021b). To overcome this difficulty, we propose to characterize the properties of the additive noise generated by the tree aggregation procedure, and develop new proof techniques (see Sec B.4) as well as new results (see Lemma B.12) to characterize these extra error terms (see equation (B.16) and Theorem B.5). These new techniques and results enable us to derive the new bias-variance decomposition expression for each iteration in our proposed DP-FTRL based method (Theorem B.5).

 2. Given the new bias-variance decomposition expression (Theorem B.5) for each iteration, we still cannot directly apply the proof techniques in Zou et al (2021b) to obtain the desired excess risk bound. The reason is that the extra errors introduced by the private mechanism depend on the general covariance matrix $\mathbf{\Sigma}$ and will affect the variance error (see Lemma B.6). As we discussed under Lemma B.6, the proof techniques in Zou et al (2021b) will make the variance error explode since the general covariance matrix $\mathbf{\Sigma}$ will no longer be well aligned with $\mathbf{H}$, a condition crucial for the proofs in in Zou et al (2021b)’s proofs. Therefore, to provide a tight upper bound on the variance error, we derive new results (Lemma B.7 and Lemma B.8) to handle the general covariance matrix $\mathbf{\Sigma}$ case. 

In summary, to obtain the excess risk bound of our proposed method, we need to develop new proof techniques that can handle the correlated noise and the variance error introduced by the general covariance matrix of the additive noise. We believe our new analytical tools could be of independent interest due to their effectiveness in addressing the challenges posed by correlated noise and general covariance matrices.

>why the analysis of tree aggregation needs to be redone.

It's important to note that the tree aggregation protocol used in our method is slightly different from those used in prior work (e.g., Guha Thakurta & Smith (2013),  Kairouz et al. (2021)). 
Specifically, our method proposes to add different amounts of noise to each node in the tree (see line 7 in Algorithm 1 and line 11 in Algorithm 3) while the previous work proposes to add the same amount of noise to each node in the tree. This is due to the adaptive clipping step (see line 6 in Algorithm 1) used in our method, leading to different norm bounds for different nodes. Therefore, we cannot directly apply existing results (such as those in Guha Thakurta & Smith (2013),  Kairouz et al. (2021) ) to provide the privacy guarantee of our tree aggregation protocol. We will clarify this in the revision.

Reviewer comment (Theory):
Question (Minor) — Theorem 4.2 specifies constants depending on \(\mathrm{Tr}(H)\), which is assumed to be unknown; is it standard for DP linear regression algorithm parameters to depend on the unknown matrix \(H\)?

Author response:
>Is it standard to have the algorithm parameters in DP linear regression depend on the unknown matrix

It is very standard in the DP literature to set algorithm parameters to some unknown values in order to derive utility guarantees (e.g., Varshney et al. (2022), Liu et al. (2023)) as long as the privacy guarantee (i.e., Theorem 4.1 in our paper) does not depend on any unknown values.

Reviewer comment (Theory):
Question (Minor) — The authors assume that \(\mathrm{Tr}(H)\) is no bigger than a constant. The reviewer's question about this assumption appears incomplete in the provided text.

Author response:
>assume that $\mathrm{Tr}(\mathbf{H})$ is no bigger than a constant

We can also present our results in terms of Tr(H). The reason that we choose to assume $\mathrm{Tr}(\mathbf{H})$ to be a constant is to simplify our final results, especially in the polynomial and exponential decay settings. Moreover, considering bounded $\mathrm{Tr}(\mathbf{H})$ is the necessary condition to enable the analysis in the infinity dimensional case.

Reviewer comment (Presentation):
Suggestion — Make it clear that the tighter convergence rates depend on a matrix estimated from public data, and mention this early (e.g., in the abstract), not only halfway through the paper.

Author response:
>(Major problem) access to (unlabeled) public data is not ever mentioned in the abstract or introduction.

We indeed mention in the abstract that “Furthermore, when **unlabeled public data** is available, we can design a better noise covariance matrix structure to improve the utility. ” We will also add comments about using unlabeled public data in the introduction section when we state our main contributions (the second bullet).

Reviewer comment (Theory):
The proposed designed noise covariance matrix is also limited and may be unpractical since it needs the unlabeled data first to generate an estimation of $\mathbf{H}$.

Author response:
The idea of using public data in the DP methods has been widely used in the literature, e.g., Kairouz et al. (2020), Yu et al. (2022), and Zhou et al. (2021). In addition, getting unlabeled public data is generally cheaper than labeled training data. Therefore, we do not think the proposed designed noise covariance matrix is limited and impractical. Additionally, if we do not have a public dataset, we can use the isotropic noise $\mathbf{\Sigma}=\mathbf{I}$. In this case, we also develop an problem- and algorithm- dependent analysis for the excess risk bound (see Corollary 4.3, Corollary 4.4), which are stated as the entire data covariance matrix instead of the problem dimension, thus can be potentially better than the existing works (Varshney et al. (2022), Liu et al. (2023)) that rely on the dimension explicitly. 

Moreover, we also want to emphasize that our excess risk bound is general, and can be employed for any noise covariance. Therefore, one can definitely seek to develop private algorithms using part of the training data points to estimate $\mathbf{H}$ (regardless of its estimation error) and then apply our general theoretical framework to get the excess risk bound accordingly. In fact,  as long as the estimate can well approximate the large eigenvalue of $\mathbf{H}$ (i.e., the estimate is better than $\mathbf{I}$ in terms of estimating $\mathbf{H}$), it will result in better utility guarantees compared to that obtained by using isotropic noise. We will add the discussion in the revision.

Reviewer comment (Writing):
The statement of Algorithm 3 is not clear. Binary representation $b_i$ is not clear in line 13. Also, Line 7 in algorithm 3 shows "$n_j$ is the last node in $\mathbf{p}$ that is a left child" and add noise when $n_i \in \mathbf{p}_j$, while Page 15 states "adding Gaussian noise to the nodes along this path from m to the first left child", which is contradict. $\mathbf{p}_j$ is a set and first left child is a node. Hope the author give more explanations.

Author response:
The binary representation mentioned in line 13 refers to expressing the number  $t$ as a binary number using $k$ bits, i.e., $\\{b_1,....,b_k\\}$, where $b_1$ is the most significant bit. We will add more explanations about the binary representation in the revision. There is no contradiction between line 7 in Algorithm 3 and Page 15. The set $\mathbf{p}_j$ stores nodes in a top-down sequence, from the root to the leaf. Since the first node $n_j$ stored in $\mathbf{p}_j$ is the last left child node in $\mathbf{p}$ (top-down order), it is equivalent to state that $\mathbf{p}_j$ stores the nodes from the leaf node $m_t$ to the first left child, i.e., $n_j$, along the path (down-top order). We will clarify this in the revision.

Reviewer comment (Writing):
Typo: Page 5, update rule should be \sum_{i=0}^t g_i instead of \sum_{j=0}^t.

Author response:
Thanks for pointing out this typo, we will fix it in the revision.

Reviewer comment (Experiments):
The authors should add experiments to support their theory.

Author response:
Our paper’s main contribution is on the theoretical side. However, we would like to add experiments in the revision.

Reviewer comment (Theory):
The author should add lower bounds.

Author response:
We believe that it is possible to have an algorithm-dependent lower bound that is also stated as a function of both data covariance and the covariance of the additive noise. When the covariance of the additive noise is $\mathbf{\Sigma}=\mathbf{I}$, we believe our upper and lower bounds can be tight up to logarithmic factors. In the more general case, there may exist a gap because we may need to handle the non-commute matrices  in the lower bound proofs, which may necessitate additional relaxations to address this challenge. The motivation of our paper is to develop a problem-dependent framework to study the private linear regression that can handle general data covariance and general non-isotropic additive noise to achieve DP. Our results can potentially lead to better noise designs for achieving DP and obtain better utility guarantees. Adding the lower bound to our paper is a plus but this will not affect the main contributions of our paper. We would like to add the discussions about the lower bound in the revision.