Paper: Non-asymptotic Analysis of Stochastic Gradient Descent under Local Differential Privacy Guarantee (ICLR-2024)

Reviewer comment (Presentation):
The presentation of the theoretical results needs to be improved and simplified. For e.g., Theorem 4 is very hard to parse and overloaded with too many symbols. I'd recommend deferring the full versions to the Appendix and presenting abridged versions in the main paper having only the important terms. Also, I'd have liked to see a remark or something explicitly comparing the dependence of the convergence bounds of the averaged iterate and last iterate w.r.t. \(n\) together (and therefore, explaining why the averaged iterate does better) rather than leaving it to the reader.

Author response:
Acknowledging the complexity of the theoretical expressions in our work, we appreciate the reviewer's observation regarding their intricacy for those less familiar with the field. The task of non-asymptotic analysis is notably challenging, particularly when it comes to deriving tractable theoretical results from complex estimation bounds. Such complexity often renders these results difficult to comprehend, especially in understanding the convergence behavior of estimators. Our work, however, provides a clear and accurate bound for the estimation process. It elucidates key aspects: the predominant rate of convergence, the diminishing influence of initial conditions, and how various factors—like dimensionality and the privacy budget—affect the theoretical bounds. This clarity in our results sheds significant light on areas previously obscured by complexity. To facilitate understanding, we have included detailed explanatory remarks. In response to the feedback, we will further simplify these expressions in the revised version to enhance accessibility for our readers. For a more comprehensive overview, we add **Table 1** in our revised version to summarize our results. This table will detail the leading convergence rates for each specific situation addressed in our research, offering a more comprehensive understanding of our contributions in comparison to existing work.

Reviewer comment (Theory):
Looking at Theorem 4, there is a term of \(||\theta_0 - \theta^{\ast}||^4\) whereas in Theorem 3, there is just \(||\theta_0 - \theta^{\ast}||^2\). If our initialization is very far from the optimum, this may make the averaged iterate have worse convergence than the last iterate. This point should be discussed in the paper.

Author response:
We appreciate the reviewer's concerns and apologize for any confusion our work may have caused. Indeed, there is a notable difference in how quickly initial conditions diminish in two scenarios: the averaged version and the last step estimation, particularly with smaller step sizes. As outlined in Remark 1 and Remark 2, the averaged version tends to 'forget' these initial values at a slower rate compared to the last step estimation. For a more comprehensive understanding of the differences in the convergence rate between these two methods, we direct readers to the detailed results and discussion in Paragraph 1 on Page 9 of our report. Please also see Table 1 for the summary of convergence rate in different situations.

Reviewer comment (Experiments):
I understand that this is a theoretical paper and so I don't want to complain too much about the scale of the experiments, but for this proposal to be more convincing and practically useful, the authors should consider performing some larger experiments -- beyond linear/logistic regression and definitely with larger \(d\).

Author response:
Thank you for your feedback. Our choice to utilize linear and logistic regression models is grounded in their widespread use and representativeness in regression and classification problems, respectively. These models also exemplify the two categories of loss functions our paper explores: strongly convex and non-strongly convex. Furthermore, we have conducted simulations with higher-dimensional data, the details of which are available in the supplementary materials.

Reviewer comment (Writing):
It'd be nice to also provide some intuitive explanation of why averaging helps instead of just math.

Author response:
Intuitively, our procedure involves introducing a noise term with a zero mean in each iteration. This addition leads to considerable variance when considering the last step as our estimator. However, by employing an averaging technique, we significantly reduce the variance induced by this noise. As a result, this averaging approach not only stabilizes our estimator but also endows it with a smaller variance.

Reviewer comment (Theory):
Is Condition 3 being used in Theorem 3? I don't see any \(C_1\) term in Theorem 3.

Author response:
We thank the reviewer for reading our paper carefully. Actually, Condition 3 is not used in the proof of Theorem 3. In this revision, we have corrected it.

Reviewer comment (Theory):
Why are there \(E\big(||\theta_0 - \theta^{\ast}||^2\big)\) and \(E\big(||\theta_0 - \theta^{\ast}||^4\big)\) terms in the results? These are deterministic, right? Or are you taking expectation w.r.t. something else here?

Author response:
We feel sorry for the misleading notation here. To clarify, the initial conditions referred to are fixed numbers.

Reviewer comment (Theory):
What are the constants \(C_{3,0}\) and \(C_{4,0}\) in Theorem 4? Overall this theorem needs to be simplified and cleaned up as I mentioned in Weaknesses.

Author response:
These constants are associated with the loss function, as detailed in the supplementary material on Page 4, Paragraph 2. We aim to present our results in a more simplified manner in the revised version.

Reviewer comment (Theory):
Can the results be extended to regular \((\varepsilon,\delta)\)-DP? Or is there a reason the authors are considering GDP?

Author response:
In practice, according to Section 2.4 in [1], we know $f$-DP is a generalization of $(\varepsilon, \delta)$-DP. Therefore, this result can be easily extended to the regular DP. Meanwhile, within the context of an online setting, we utilize the
parallel composition approach rather than relying on the composition property of the
GDP. In this case, incorporating the classical definition of LDP seems more appropriate. Theoretically, it’s possible to extend noise distribution beyond
Gaussian noise by imposing the assumptions on the boundedness of noise moments
when considering the general LDP. In our revised version, we are committed to thoroughly integrating classical LDP notation alongside GDP.

[1] Dong, Jinshuo, Aaron Roth, and Weijie J. Su. "Gaussian differential privacy." Journal of the Royal Statistical Society Series B: Statistical Methodology 84.1 (2022): 3-37.

Reviewer comment (Theory):
The paper does not provide any lower bounds so it is hard to evaluate the tightness of results.

Author response:
We agree with the reviewer on the importance of a lower bound in evaluating and understanding the estimator's behavior. However, the task of establishing a lower bound for parameter estimation in our specific context, and similar ones, is exceptionally challenging and remains an unresolved issue in the field. To our knowledge, there exists a substantial gap in the literature and available methodologies that directly address this challenge. Also, the existing literature [1, 2] conducting non-asymptotic analysis only provide upper bounds, which indirectly highlights the inherent challenges associated with establishing a lower bound.

Indeed, theoretical analysis in the field of LDP algorithms, particularly in terms of their convergence, is generally sparse. Our study distinguishes itself from existing studies in LDP, which primarily concentrates on the development of algorithms that achieve set privacy benchmarks, but often overlooks their convergence behaviors. In contrast, our study makes a significant contribution by not only examining but also quantifying the rate and manner in which these DP algorithms approach optimal solutions. This marks a significant leap forward in the realm of DP.

Reviewer comment (Novelty):
This paper lacks comprehensive comparisons to prior works. There are many works on stochastic gradient descent on local differential privacy guarantee. For example, Algorithm 4 in Wang, Teng, et al. "Local differential privacy for data collection and analysis." Neurocomputing 426 (2021): 114-133 and Algorithm 1 in Liu, Ruixuan, et al. "Fedsel: Federated sgd under local differential privacy with top-k dimension selection." Database Systems for Advanced Applications: 25th International Conference, DASFAA 2020, Jeju, South Korea, September 24–27, 2020, Proceedings, Part I 25. Springer International Publishing, 2020. From my point of view, this is not the first work on LDP-SGD, and therefore a comprehensive comparison is expected in this paper. I suggest the author adding a table including previous results and this work. Also, the upper bounds in the paper are unnecessarily long; for simplicity and better rendering I suggest keeping the leading asymptotic term.

Author response:
We thank the reviewer for providing additional work focusing on LDP. However, the papers you referenced address different contexts within the field of local differential privacy. For instance, 'Local differential privacy for data collection and analysis' focuses on mean estimation, contrasting with the general parameter estimation issue our study addresses. Similarly, 'Fedsel: Federated SGD under local differential privacy with top-k dimension selection' considers private
dimension selection mechanisms and adapts the gradient accumulation
technique to stabilize the learning process with noisy updates within a distributed framework, while our work is framed in an online parameter estimation context under LDP. Considering these different scenarios, making direct comparisons within a uniform framework or situation is difficult. Nonetheless, for a more lucid comparison, we have introduced a table in our revision. This table details the leading convergence rates for each specific situation addressed in our study, offering a more comprehensive understanding of our contributions in comparison to existing work.

Reviewer comment (Novelty):
The technical contribution is relatively weak as I could not find sufficient novelty in the proofing techniques.

Author response:
In our technical details of online SGD algorithm, we recognize that specialists in the field might note the structural similarities in proofs across various studies. Such similarities often stem from the use of standard methods and tools within the discipline. Specifically, in our convergence analysis of DP-SGD estimator, the proof structure mirrors that of non-private versions, a similarity we only discerned after completing our analysis. This resemblance arises from the inherent recursive nature of the SGD algorithm. 

Also, we wish to clarify that our work does not claim to introduce novel proof techniques. Instead, our significant contribution lies in adeptly applying established methodologies from the extensive literature to tackle the new and distinct challenges presented by online LDP problems. Our focus is on solving these novel problems using existing tools, rather than on the novelty of the proof techniques themselves. Indeed, the non-asymptotic analysis of online LDP is inherently challenging, and achieving tractable theoretical results was not feasible until we developed these theorems. 

Finally, in our exploration of the theoretical outcomes when $\alpha=1$, we found that the convergence of the algorithm hinges on specific conditions, which is different from the non-private cases. It demonstrates that even though the structures and tools of the proofs may be similar, the outcomes in private settings, as opposed to non-private ones, can differ significantly. This distinction not only highlights the uniqueness of our results but also reinforces the value of our contribution in extending non-asymptotic analysis to the private version of the online SGD algorithm.

Reviewer comment (Presentation):
The presentation of this work could be enhanced. Specifically, it appears there may be an issue with the visibility of equations 3 and 4, which I presume are the upper bounds in Theorems 3 and 4. This could potentially be a result of a compiling error.

Author response:
We express our sincere gratitude to the reviewer for identifying the editing error in the cross-reference links. Our main emphasis has been on the thorough derivation of theoretical results to ensure their accuracy. We acknowledge the importance of clear and precise presentation and commit to improving this aspect in the final manuscript.

Reviewer comment (Writing):
There is an obvious typo in the proof. On page 1 of supplementary material, “For the second term of (1), by Condition ??, we have…”, here Condition ?? should be Condition 2.

Author response:
Thank you for spotting the error. In this revision, following your suggestion, we have revised it.

Reviewer comment (Theory):
Why the authors particularly favor Polyak-Ruppert averaging estimator in the theoretical analysis? It would also be interesting to see the results of many other variants of SGD that give better performance.

Author response:
In fact, the Polyak-Ruppert averaging method greatly improves the stability and efficiency of stochastic approximation algorithms through the averaging of iterative sequences. This approach has been demonstrated to achieve information-theoretically optimal asymptotic variance. It does so particularly when employing learning rates characterized by slower decay, coupled with uniform averaging. Meanwhile, the Polyak-Ruppert averaging technique remains the most popular and user-friendly version of the online SGD algorithm. Its widespread use underscores its efficiency and straightforward application. While this method is our primary focus, we recognize the value of investigating other variants, such as the weighted sum approach, in our future research.

Reviewer comment (Theory):
I think authors may not provide a good explaination on their motivation on using GDP instead of classical LDP notation, see also Question 1.

Author response:
**W1 & Q1**: Thank you for your insights on the use of GDP over classical DP in our work. We recognize that classical DP is indeed a more general framework. As outlined in Section 2.4 of [1], $f$-DP, which includes GDP, is an extension of $(\varepsilon, \delta)$-DP, allowing us to easily extend our results to classical DP scenarios. It is theoretically feasible to expand our analysis to include noise distributions beyond Gaussian, which forms our ongoing research. 

Our choice of GDP in this work is driven by two main factors. Firstly, there is a growing interest in GDP, with numerous algorithms proposed that lack theoretical convergence guarantees. Our work aims to bridge this gap, providing a foundation for non-asymptotic analysis of these algorithms. Secondly, GDP demonstrates advantageous theoretical properties in asymptotic analysis, as detailed in [2], prompting us to offer a comprehensive non-asymptotic analysis for GDP in advance. In conclusion, while classical DP offers a broader context, our focus on GDP is intentional, aligning with the specific aims and scope of our current paper.

Reviewer comment (Novelty):
Both the results and proof techniques in this paper are similar to those in [1] in the non-private setting. It appears that the introduction of additional Gaussian noise does not pose significant challenges to the analysis. However, the authors do not explicitly compare their paper to [1] or other non-private research in both the proof and result sections.

Author response:
**W2**: In our technical details of online SGD algorithm, we recognize that specialists in the field might note the structural similarities in proofs across various studies. Such similarities often stem from the use of standard methods and tools within the discipline. Specifically, in our convergence analysis of DP-SGD estimator, the proof structure mirrors that of non-private versions, a similarity we only discerned after completing our analysis. This resemblance arises from the inherent recursive nature of the SGD algorithm. 

Also, we wish to clarify that our work does not claim to introduce novel proof techniques. Instead, our significant contribution lies in adeptly applying established methodologies from the extensive literature to tackle the new and distinct challenges presented by online LDP problems. Our focus is on solving these novel problems using existing tools, rather than on the novelty of the proof techniques themselves. Indeed, the non-asymptotic analysis of online LDP is inherently challenging, and achieving tractable theoretical results was not feasible until we developed these theorems.

Reviewer comment (Theory):
Why did the authors choose to employ GDP notation rather than LDP in the paper? As far as my understanding goes, GDP's primary advantage is its tighter composition guarantee. However, within the context of this paper, where each user's data passes through the algorithm only once, the notion of composition does not apply. Consequently, I am struggling to understand the underlying motivation and advantages of adopting GDP notation.

Author response:
**W1 & Q1**: Thank you for your insights on the use of GDP over classical DP in our work. We recognize that classical DP is indeed a more general framework. As outlined in Section 2.4 of [1], $f$-DP, which includes GDP, is an extension of $(\varepsilon, \delta)$-DP, allowing us to easily extend our results to classical DP scenarios. It is theoretically feasible to expand our analysis to include noise distributions beyond Gaussian, which forms our ongoing research. 

Our choice of GDP in this work is driven by two main factors. Firstly, there is a growing interest in GDP, with numerous algorithms proposed that lack theoretical convergence guarantees. Our work aims to bridge this gap, providing a foundation for non-asymptotic analysis of these algorithms. Secondly, GDP demonstrates advantageous theoretical properties in asymptotic analysis, as detailed in [2], prompting us to offer a comprehensive non-asymptotic analysis for GDP in advance. In conclusion, while classical DP offers a broader context, our focus on GDP is intentional, aligning with the specific aims and scope of our current paper.

Reviewer comment (Theory):
Concerning the theoretical results when $\alpha = 1$: In the convergence rate section of Remark 1, the authors assert that "For a scenario where $\alpha = 1$, convergence of the LDP-SGD estimator $\theta_n$ is not assured." Nevertheless, based on my knowledge, at least in the strongly convex setting, the convergence analysis of LDP-SGD with a step size $\theta_i = O(1/i)$ seems to be readily attainable by adapting the proof provided in [1]. Have I possibly overlooked any technical complexities?

Author response:
**Q2**: In our exploration of the theoretical outcomes when $\alpha=1$, we found that the convergence of the algorithm hinges on specific conditions. These are determined through mathematical derivation and involve factors such as the correlation between the eigenvalues of the loss function's Hessian matrix and the initial step size. Additionally, our simulation studies highlight that these conditions often go unmet in practical scenarios, leading to situations where convergence is not guaranteed.

This distinctive behavior in convergence under the condition $\alpha=1$ for private versus non-private cases further underscores an important aspect of our work. It demonstrates that even though the structures and tools of the proofs may be similar, the outcomes in private settings, as opposed to non-private ones, can differ significantly. This distinction not only highlights the uniqueness of our results but also reinforces the value of our contribution in extending non-asymptotic analysis to the private version of the online SGD algorithm.