Paper: General Stability Analysis for Zeroth-Order Optimization Algorithms (ICLR-2024)

Reviewer comment (Novelty):
It would be better if the author emphasized the new technique and idea used in this paper compared with the existing work by Nikolakakis et al. (2022).

Author response:
Thanks for your valuable comments. There exist  key differences between (Nikolakakis et al. 2022) and our work, which are summarized as below:

1) Objective function: **ZO-SGD with 2-point gradient estimator Vs. general ZO optimization.** Nikolakakis et al. (2022) merely considered the generalization bound of ZO-SGD with 2-point gradient estimator. In this paper, we established a unified generalization analysis framework  for  ZO-GD, ZO-SGD as well as ZO-SVRG algorithms with 1-point / 2-point / coordinate-wise gradient estimators, which cover the mainstream estimators in existing zeroth-order algorithms.

2) Conditions: **Gaussian distribution and non-convex condition Vs. general distribution and convex conditions**.  Nikolakakis et al. (2022) just considered the $u_k\sim\mathcal{N}\left(0, I_d\right)$ and non-convex loss function. In this paper, we derive the generalization bounds for more distributions (e.g.,uniform $u_k\sim \mathcal{U}\left(\sqrt{d+2} \mathbb{B}^d\right)$, $u_k\sim \mathcal{U}\big(\sqrt{d} \mathbb{S}^{d-1}\big)$ see **Appendix E**) and wider convexity conditions (strongly convex see **Theorem 3,6,9 and Appendix G, H, I**, convex see **Theorem 3,6,9 and Appendix G,H, I** and non-convex **Theorem 1-9**).

3)  **Filling the gap on the  generalization bounds of ZO-SVRG.** ZO-SVRG is a  stage-wise algorithm with multiplication and summation from previous stages, which is totally different from ZO-SGD. To the best of our knowledge, this paper is the first endeavor to investigate the ZO-SVRG's generalization behavior. Meanwhile, our derived estimations of ZO-SGD $\Big({O}\big(T^{\frac{\beta C}{\beta C + 1}}/n\big)\Big)$ are tighter than the related results $\Big({O}\big(T/n\big)\Big)$ in Nikolakakis et al. 2022.

We have polished the comparisons with the related works and highlighted our contributions (see **Appendix J**).

Reviewer comment (Theory):
One potential weakness of the paper is that the results assume that the objective function is smooth, which, in many important optimization problems (e.g., learning ReLU networks) is not true.

Author response:
Thanks for your constructive comments. Indeed, there exist several works on non-smooth settings for first-order optimization, see e.g., [Ref1][Ref2]. However, these researches are limited to the convex or weakly convex settings. 
Moreover, the absence of smooth condition posed challenges on bounding the approximation error induced by zeroth-order optimization. Our analysis framework established here can not be extended to the non-smooth setting directly.  
We are striving to address this issue in the future work.


We also added the related discussions in the revised paper, see **new Remark 3 at Appendix J**.



[Ref1] Bassily R, Feldman V, Guzmán C, et al. Stability of stochastic gradient descent on nonsmooth convex losses, NeurIPS, 2020.
 
[Ref2]Lei Y. Stability and generalization of stochastic optimization with nonconvex and nonsmooth problems, ICML, 2023.

Reviewer comment (Theory):
Could your results be extended to the case where the objective function is not smooth?

Author response:
Thanks for your constructive comments. Indeed, there exist several works on non-smooth settings for first-order optimization, see e.g., [Ref1][Ref2]. However, these researches are limited to the convex or weakly convex settings. 
Moreover, the absence of smooth condition posed challenges on bounding the approximation error induced by zeroth-order optimization. Our analysis framework established here can not be extended to the non-smooth setting directly.  
We are striving to address this issue in the future work.


We also added the related discussions in the revised paper, see **new Remark 3 at Appendix J**.



[Ref1] Bassily R, Feldman V, Guzmán C, et al. Stability of stochastic gradient descent on nonsmooth convex losses, NeurIPS, 2020.
 
[Ref2]Lei Y. Stability and generalization of stochastic optimization with nonconvex and nonsmooth problems, ICML, 2023.

Reviewer comment (Theory):
Are there any generalization bounds for this setting, even for first order algorithms?

Author response:
Thanks for your constructive comments. Indeed, there exist several works on non-smooth settings for first-order optimization, see e.g., [Ref1][Ref2]. However, these researches are limited to the convex or weakly convex settings. 
Moreover, the absence of smooth condition posed challenges on bounding the approximation error induced by zeroth-order optimization. Our analysis framework established here can not be extended to the non-smooth setting directly.  
We are striving to address this issue in the future work.


We also added the related discussions in the revised paper, see **new Remark 3 at Appendix J**.



[Ref1] Bassily R, Feldman V, Guzmán C, et al. Stability of stochastic gradient descent on nonsmooth convex losses, NeurIPS, 2020.
 
[Ref2]Lei Y. Stability and generalization of stochastic optimization with nonconvex and nonsmooth problems, ICML, 2023.

Reviewer comment (Experiments):
The paper could benefit from more extensive numerical experiments and a more detailed implementation section. Specifically, it should include a comparison of the performance of various zeroth-order estimators, providing empirical support for the favorable theoretical results associated with coordinate estimation.

Author response:
Thanks for your constructive comments. In the revised version, we have conducted more numerical experiments to verify the theoretical results, where the implementation details are also provided (see **Section 5.1 Experimental Setups**). 

Firstly, in **Section 5.2**, we compare the generalization errors of different ZO algorithms associated with the same gradient estimators. Experimental results in **Figure 1** show that ZO-SGD can achieve the competitive generalization performance compared with ZO-GD and ZO-SVRG.

Secondly, we compare the generalization errors of same ZO algorithms with  different gradient estimators in  **Section 5.3 and Appendix B**. Experimental results in **Figure 2** demonstrate that ZO algorithms with coordinate-wise gradient estimator usually have the lowest generalization error.

Reviewer comment (Reproducibility):
What is the zeroth-order estimator used in the experiments?

Author response:
In the experiments, we employ the coordinate-wise estimator  as the zeroth-order estimator.  Implementing code can be found in  the **Supplementary Material**, lines 45-59 of the algs.py file.

Reviewer comment (Experiments):
How do different zeroth-order estimators affect the optimization performance?

Author response:
Indeed, different zeroth-order estimators affect the optimization performance by their approximation errors and variances with respect to the true gradient.

Specifically, 1-point gradient estimator may suffer from high variance (the variance explodes as $\mu$ goes to 0). Despite of this drawback in theory, it is still a useful black-box optimization algorithm for certain applications, e.g., reinforcement learning [Ref3].
 
For 2-point gradient estimator, the variance is bounded under forward difference or central difference structure. Thus, the relevant 2-point gradient estimator is more stable than 1-point one [Ref4]. Normally, zeroth-order optimization algorithms with 2-point gradient estimators own comparable convergence rate with respect to their first-order counterparts but may suffer from the dimension penalty of order $d$ [Ref5][Ref6].
 
For the coordinate-wise gradient estimator, its variance often is zero from the  deterministic approximation procedure. The optimization performance of ZO algorithms with coordinate-wise gradient estimators is directly influenced by the approximation error. Moreover,  the approximation error can be  controlled by  $O(d\mu^2)$, where the smoothing parameter $\mu$  can be arbitrarily small [Ref7]. However, coordinate-wise gradient estimator needs to compute $O(d)$ function queries to achieve a more accurate approximation, which may not be applicable in high dimensional optimization problems.



[Ref3] T. Salimans et al. Evolution strategies as a scalable alternative to reinforcement learning.

[Ref4]A. D. Flaxman, A. T. Kalai, and H. B. McMahan, Online convex
optimization in the bandit setting: Gradient descent without a gradient,in Proc.16th Annual ACM-SIAM Symposium on Discrete algorithms(SODA), 2005.

[Ref5] Nesterov Y, Spokoiny V. Random gradient-free minimization of convex functions.Foundations of Computational Mathematics(Found Comut Math), 2017.

[Ref6] Ghadimi S, Lan G. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization(SIOPT), 2013.

[Ref7] Ji K, Wang Z, Zhou Y, et al. Improved zeroth-order variance reduced algorithms and analysis for nonconvex optimization. In International conference on machine learning(ICML), 2019.

Reviewer comment (Presentation):
Please indicate beta and C in Table 1.

Author response:
We apologize for not explicitly indicating the meanings of these two symbols in the main text. In the latest submitted PDF, we have included explanations for them.
Beta is a constant of smoothness and $C$ is a positive constant.

Reviewer comment (Reproducibility):
More implementation details are needed.

Author response:
Thanks for your constructive comments. In our latest submission, we have incorporated additional implementation details and new comparative experiments (see **Section 5 Numerical Experiments**). Detailed experimental discussions have been included to analyze the generalization performance of ZO-SGD, ZO-GD, and ZO-SVRG, incorporating scenarios (see **Subsection 5.2**). Furthermore, we have introduced comparisons of the generalization bounds and convergence properties for multiple estimators (see **Subsection 5.3 and Appendix B**). More details also can be found in A2 to Q2.

Reviewer comment (Writing):
Utilize parenthetical citations to enhance readability.

Author response:
Thanks. We have used the parenthetical citations to enhance the readability and further proofread the whole paper.

Reviewer comment (Theory):
For the ZO-SGD, ZO-GD and ZO-SVRG with one-point and two-point gradient estimation, the authors only presented the results for the non-convex setting. Is there any theoretical results for the general convex and strongly convex settings, like the results for the coordinate-wise gradient estimation? If the authors could add these theoretical results, it could make the theoretical contributions much more complete.

Author response:
Thanks for your constructive comments.convex and strongly convex conditions. 
Following your  valuable suggestions, we have added the corresponding generalization bounds  under convex and strongly convex conditions. For details, please refer to **Appendix (F.1, F.2, G.1, G.2, H.2)**.  We expect that the additional results can make our analysis more complete and improve its readability. 
Explicitly, the generalization bounds for ZO-SGD are  $O(T/n^{1-\frac{c}{d^2 L C \sqrt{n}}})$ and  $O(T/n^{1-\frac{c}{d^2 L C \sqrt{n}}})$  under one-point strongly convex condition and one-point convex conditions respectively. Meanwhile, generalization bounds for ZO-SGD are $O(T/n)$ under two-point strongly convex condition  and $O(T/n)$ under two-point convex condition. For ZO-GD, we present the generalization bounds $O(T^{C}/n)$ under one-point strongly convex condition, $O(T^{C}/n)$ under one-point convex condition, $O(T^{\beta C}/n)$ under two-point strongly convex condition, and $O(T^{\beta C}/n)$ under two-point convex condition. Similarly, ZO-SVRG exhibits generalization bounds $O(S^{3 C}/n)$ under one-point strongly convex condition, $O(S^{3 C}/n)$ under one-point convex condition, $O(S^{3 \beta C}/n)$ under two-point strongly convex condition, and $O(S^{3\beta C}/n)$ under two-point convex condition. Indeed, there are some preliminary discussions in Remark 1 in the original version.

In essential, the convexity property has impact  on the expansion coefficient $\eta$ involving in the generalization bounds of first-order optimization methods (see **Appendix (Lemma 6)**) . It is easy to deduce that $\eta_t=1-\frac{\alpha_t\beta\gamma}{\beta+\gamma}$ (strong convex), $\eta = 1$ (convex), and $\eta_t = 1 + \alpha_t\beta$ (non-convex). However, for the zeroth-order case, there often is additional increment induced by the procedures of one-point or two-point estimation (Ref1). We can verify  $\eta^{\prime}_t=1+\alpha_t\left(\frac{L}{\mu}\mathbb{E}\|u\|+\beta\mathbb{E}\left[\|u\|^2\right]\right)$ (one-point convex condition), $\eta^{\prime}_t=1+\alpha_t\mu\beta\mathbb{E}\left[\|u\|^2\right]$ (two-point convex condition),  $\eta^{\prime}_t=1-\frac{\alpha_t\beta\gamma}{\beta+\gamma}+\alpha_t\left(\frac{L}{\mu}\mathbb{E}\|u\|+\beta\mathbb{E}\left[\|u\|^2\right]\right)$ (one-point strongly convex condition), and $\eta^{\prime}=1-\frac{\alpha_t\beta\gamma}{\beta+\gamma}+\alpha_t\mu\beta\mathbb{E}\left[\|u\|^2\right]$ (two-point strongly convex condition). 

Due to the additional increments (e.g., $\eta^{\prime}$ may exceed $1$), multiplication process. 
the additional results for the general convex and strongly convex settings are analogous to the existing error bounds under non-convex conditions (e.g., **Theorems 1,2,4,5,7,8**).


[Ref1] K. Nikolakakis et al., Black-box generalization: Stability of zeroth-order learning. Advances in Neural Information Processing Systems (NeurIPS), 2022.