Original: Z EROTH -O RDER O PTIMIZATION WITH T RAJECTORYI NFORMED D ERIVATIVE E STIMATION
Revised: Z EROTH -O RDER O PTIMIZATION WITH T RAJECTORYI NFORMED D ERIVATIVE E STIMATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Zeroth-order (ZO) optimization, in which the derivative is unavailable, has recently succeeded in many important machine learning applications.
Revised: Zeroth-order (ZO) optimization, in which the derivative is unavailable, has recently succeeded in many important machine learning applications.

Original: Existing algorithms rely on finite difference (FD) methods for derivative estimation and gradient descent (GD)-based approaches for optimization.
Revised: Existing algorithms rely on finite difference (FD) methods for derivative estimation and gradient descent (GD)-based approaches for optimization.

Original: However, these algorithms suffer from query inefficiency because additional function queries are required for derivative estimation in their every GD update, which typically hinders their deployment in applications where every function query is expensive.
Revised: However, these algorithms suffer from query inefficiency because additional function queries are required for derivative estimation in their every GD update, which typically hinders their deployment in applications where every function query is expensive.

Original: To this end, we propose a trajectory-informed derivative estimation method which only uses the optimization trajectory (i.e., the history of function queries during optimization) and hence eliminates the need for additional function queries to estimate a derivative.
Revised: To this end, we propose a trajectory-informed derivative estimation method which only uses the optimization trajectory (i.e., the history of function queries during optimization) and hence eliminates the need for additional function queries to estimate a derivative.

Original: Moreover, based on our derivative estimation, we propose the technique of dynamic virtual updates , which allows us to reliably perform multiple steps of GD updates without reapplying derivative estimation.
Revised: Moreover, based on our derivative estimation, we propose the technique of dynamic virtual updates , which allows us to reliably perform multiple steps of GD updates without reapplying derivative estimation.

Original: Based on these two contributions, we introduce the zeroth-order optimization with trajectory-informed derivative estimation (Z O RD) algorithm for query-efficient ZO optimization.
Revised: Based on these two contributions, we introduce the zeroth-order optimization with trajectory-informed derivative estimation (Z O RD) algorithm for query-efficient ZO optimization.

Original: We theoretically demonstrate that our trajectory-informed derivative estimation and our Z O RD algorithm improve over existing approaches, which is then supported by our realworld experiments such as black-box adversarial attack, non-differentiable metric optimization and derivative-free reinforcement learning.
Revised: We theoretically demonstrate that our trajectory-informed derivative estimation and our Z O RD algorithm improve over existing approaches, which is then supported by our realworld experiments such as black-box adversarial attack, non-differentiable metric optimization and derivative-free reinforcement learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Zeroth-order (ZO) optimization, in which the objective function to be optimized is only accessible by querying, has received great attention in recent years due to its success in many applications, e.g., black-box adversarial attack (Ru et al., 2020), non-differentiable metric optimization (Hiranandani et al., 2021), and derivative-free reinforcement learning (Salimans et al., 2017).
Revised: Zeroth-order (ZO) optimization, in which the objective function to be optimized is only accessible by querying, has received great attention in recent years due to its success in many applications, e.g., black-box adversarial attack (Ru et al., 2020), non-differentiable metric optimization (Hiranandani et al., 2021), and derivative-free reinforcement learning (Salimans et al., 2017).

Original: In these problems, the derivative of objective function is either prohibitively costly to obtain or even non-existent, making it infeasible to directly apply standard derivative-based algorithms such as gradient descent (GD).
Revised: In these problems, the derivative of objective function is either prohibitively costly to obtain or even non-existent, making it infeasible to directly apply standard derivative-based algorithms such as gradient descent (GD).

Original: In this regard, existing works have proposed to estimate the derivative using the finite difference (FD) methods and then apply GD-based algorithms using the estimated derivative for ZO optimization (Nesterov and Spokoiny, 2017; Cheng et al., 2021).
Revised: In this regard, existing works have proposed to estimate the derivative using the finite difference (FD) methods and then apply GD-based algorithms using the estimated derivative for ZO optimization (Nesterov and Spokoiny, 2017; Cheng et al., 2021).

Original: These algorithms, which we refer to as
Revised: These algorithms, which we refer to as

Original: GD with estimated derivatives , have been the most widely applied approach to ZO optimization especially for problems with high-dimensional input spaces, because of their theoretically guaranteed convergence and competitive practical performance.
Revised: GD with estimated derivatives , have been the most widely applied approach to ZO optimization especially for problems with high-dimensional input spaces, because of their theoretically guaranteed convergence and competitive practical performance.

Original: Unfortunately, these algorithms suffer from query inefficiency, which hinders their real-world deployment especially in applications with expensive-to-query objective functions, e.g., black-box adversarial attack.
Revised: Unfortunately, these algorithms suffer from query inefficiency, which hinders their real-world deployment especially in applications with expensive-to-query objective functions, e.g., black-box adversarial attack.

Original: Specifically, one of the reasons for the query inefficiency of existing algorithms on GD with estimated derivatives is that in addition to the necessary queries (i.e., the query of every updated input) 1 , the
Revised: Specifically, one of the reasons for the query inefficiency of existing algorithms on GD with estimated derivatives is that in addition to the necessary queries (i.e., the query of every updated input) 1 , the

Original: FD methods applied in these algorithms require a large number of additional queries to accurately estimate the derivative at an input (Berahas et al., 2022).
Revised: FD methods applied in these algorithms require a large number of additional queries to accurately estimate the derivative at an input (Berahas et al., 2022).

Original: This naturally begs the question: Can we estimate a derivative without any additional query?
Revised: This naturally begs the question: Can we estimate a derivative without any additional query?

Original: A natural approach to achieve this is to leverage the optimization trajectory , which is inherently available as a result of the necessary queries and their observations, to predict the derivatives.
Revised: A natural approach to achieve this is to leverage the optimization trajectory , which is inherently available as a result of the necessary queries and their observations, to predict the derivatives.

Original: However, this requires a non-trivial method to simultaneously (a) predict a derivative using only the optimization trajectory (i.e., the history of updated inputs and their observations), and (b) quantify the uncertainty of this prediction to avoid using inaccurate predicted derivatives.
Revised: However, this requires a non-trivial method to simultaneously (a) predict a derivative using only the optimization trajectory (i.e., the history of updated inputs and their observations), and (b) quantify the uncertainty of this prediction to avoid using inaccurate predicted derivatives.

Original: Interestingly, the Gaussian process (GP) model (Rasmussen and Williams, 2006) satisfies both requirements and is hence a natural choice for such a derivative estimation.
Revised: Interestingly, the Gaussian process (GP) model (Rasmussen and Williams, 2006) satisfies both requirements and is hence a natural choice for such a derivative estimation.

Original: Specifically, under the commonly used assumption that the objective function is sampled from a GP (Srinivas et al., 2010), the derivative at any input in the domain follows a Gaussian distribution which, surprisingly, can be calculated using only the optimization trajectory.
Revised: Specifically, under the commonly used assumption that the objective function is sampled from a GP (Srinivas et al., 2010), the derivative at any input in the domain follows a Gaussian distribution which, surprisingly, can be calculated using only the optimization trajectory.

Original: This allows us to (a) employ the mean of this Gaussian distribution as the estimated derivative, and (b) use the covariance matrix of this Gaussian distribution to obtain a principled measure of the predictive uncertainty and the accuracy of this derivative estimation, which together constitute our trajectory-informed derivative estimation (Sec. 3.1).
Revised: This allows us to (a) employ the mean of this Gaussian distribution as the estimated derivative, and (b) use the covariance matrix of this Gaussian distribution to obtain a principled measure of the predictive uncertainty and the accuracy of this derivative estimation, which together constitute our trajectory-informed derivative estimation (Sec. 3.1).

Original: Another reason for the query inefficiency of the existing algorithms on GD with estimated derivatives is that every update in these algorithms requires reapplying derivative estimation and hence necessitates additional queries.
Revised: Another reason for the query inefficiency of the existing algorithms on GD with estimated derivatives is that every update in these algorithms requires reapplying derivative estimation and hence necessitates additional queries.

Original: This can preclude their adoption of a large number of GD updates since every update requires potentially expensive additional queries.
Revised: This can preclude their adoption of a large number of GD updates since every update requires potentially expensive additional queries.

Original: Therefore, another question arises: Can we perform multiple GD updates without reapplying derivative estimation and hence without any additional query?
Revised: Therefore, another question arises: Can we perform multiple GD updates without reapplying derivative estimation and hence without any additional query?

Original: To address this question, we propose a technique named dynamic virtual updates (Sec. 3.2).
Revised: To address this question, we propose a technique named dynamic virtual updates (Sec. 3.2).

Original: Specifically, thanks to the ability of our method to estimate the derivative at any input in the domain while only using existing optimization trajectory, we can apply multi-step
Revised: Specifically, thanks to the ability of our method to estimate the derivative at any input in the domain while only using existing optimization trajectory, we can apply multi-step

Original: GD updates without the need to reapply derivative estimation and hence without requiring any new query.
Revised: GD updates without the need to reapply derivative estimation and hence without requiring any new query.

Original: Moreover, we can dynamically determine the number of steps for these updates by inspecting the aforementioned predictive uncertainty at every step, such that we only perform an update if the uncertainty is small enough (which also indicates that the estimation error is small, see Sec. 4.1).
Revised: Moreover, we can dynamically determine the number of steps for these updates by inspecting the aforementioned predictive uncertainty at every step, such that we only perform an update if the uncertainty is small enough (which also indicates that the estimation error is small, see Sec. 4.1).

Original: By incorporating our aforementioned trajectory-informed derivative estimation and dynamic virtual updates into GD-based algorithms, we then introduce the zeroth-order optimization with trajectoryinformed derivative estimation (Z O RD) algorithm for query-efficient ZO optimization.
Revised: By incorporating our aforementioned trajectory-informed derivative estimation and dynamic virtual updates into GD-based algorithms, we then introduce the zeroth-order optimization with trajectoryinformed derivative estimation (Z O RD) algorithm for query-efficient ZO optimization.

Original: We theoretically bound the estimation error of our trajectory-informed derivative estimation and show that this estimation error is non-increasing in the entire domain as the number of queries is increased and can even be exponentially decreasing in some scenarios (Sec. 4.1).
Revised: We theoretically bound the estimation error of our trajectory-informed derivative estimation and show that this estimation error is non-increasing in the entire domain as the number of queries is increased and can even be exponentially decreasing in some scenarios (Sec. 4.1).

Original: Based on this, we prove the convergence of our Z O RD algorithm, which improves over the existing ZO optimization algorithms that rely on the FD methods for derivative estimation (Sec. 4.2).
Revised: Based on this, we prove the convergence of our Z O RD algorithm, which improves over the existing ZO optimization algorithms that rely on the FD methods for derivative estimation (Sec. 4.2).

Original: Lastly, we use extensive experiments, such as black-box adversarial attack, non-differentiable metric optimization, and derivative-free reinforcement learning, to demonstrate that (a) our trajectory-informed derivative estimation improves over the existing FD methods and that (b) our Z O RD algorithm consistently achieves improved query efficiency compared with previous ZO optimization algorithms (Sec. 5).
Revised: Lastly, we use extensive experiments, such as black-box adversarial attack, non-differentiable metric optimization, and derivative-free reinforcement learning, to demonstrate that (a) our trajectory-informed derivative estimation improves over the existing FD methods and that (b) our Z O RD algorithm consistently achieves improved query efficiency compared with previous ZO optimization algorithms (Sec. 5).

Original:  P RELIMINARIES 2. P ROBLEM S ETUP
Revised:  P RELIMINARIES 2. P ROBLEM S ETUP

Original: Throughout this paper, we use ∇ and ∂ x to denote, respectively, the total derivative (i.e., gradient) and partial derivative w.r.t the variable x .
Revised: Throughout this paper, we use ∇ and ∂ x to denote, respectively, the total derivative (i.e., gradient) and partial derivative w.r.t the variable x .

Original: We consider the minimization of a black-box objective function f : X → R , in which X ⊂ R
Revised: We consider the minimization of a black-box objective function f : X → R , in which X ⊂ R

Original: d is a convex subset of the d -dimensional domain:
Revised: d is a convex subset of the d -dimensional domain:

Original: [Equation]
Revised: [Equation]

Original: Since we consider ZO optimization, the derivative information is not accessible and instead, we are only allowed to query the inputs in X .
Revised: Since we consider ZO optimization, the derivative information is not accessible and instead, we are only allowed to query the inputs in X .

Original: For every queried input x ∈ X , we observe a corresponding noisy output of y ( x ) =
Revised: For every queried input x ∈ X , we observe a corresponding noisy output of y ( x ) =

Original: f
Revised: f

Original: ( x ) + ζ , in which ζ is a zero-mean Gaussian noise with a variance of σ 2 : ζ ∼ N (0 , σ 2 ) .
Revised: ( x ) + ζ , in which ζ is a zero-mean Gaussian noise with a variance of σ 2 : ζ ∼ N (0 , σ 2 ) .

Original: Besides, we adopt a common assumption on f which has already been widely used in the literature of Bayesian optimization (BO) (Srinivas et al., 2010; Kandasamy et al., 2018): we assume that f is sampled from a Gaussian process (GP).
Revised: Besides, we adopt a common assumption on f which has already been widely used in the literature of Bayesian optimization (BO) (Srinivas et al., 2010; Kandasamy et al., 2018): we assume that f is sampled from a Gaussian process (GP).

Original: A GP GP ( µ ( · ) , k ( · , · )) , which is characterized by a mean function µ ( · ) and a covariance function k ( · , · ) , is a stochastic process in which any finite subset of random variables follows a multi-variate Gaussian distribution (Rasmussen and Williams, 2006).
Revised: A GP GP ( µ ( · ) , k ( · , · )) , which is characterized by a mean function µ ( · ) and a covariance function k ( · , · ) , is a stochastic process in which any finite subset of random variables follows a multi-variate Gaussian distribution (Rasmussen and Williams, 2006).

Original: In addition, following the common practice of GP and BO, we assume w.l.o.g. that µ ( x ) = 

Revised: In addition, following the common practice of GP and BO, we assume w.l.o.g. that µ ( x ) =

Original: and k ( x , x ′ ) ≤ 1 ( ∀ x , x ′ ∈ X ) .
Revised: and k ( x , x ′ ) ≤ 1 ( ∀ x , x ′ ∈ X ) .

Original: We also assume that the kernel function k is differentiable, and that ∥ ∂ z ∂ z ′ k
Revised: We also assume that the kernel function k is differentiable, and that ∥ ∂ z ∂ z ′ k

Original: ( z , z ′ )
Revised: ( z , z ′ )

Original: | z = z ′ =
Revised: | z = z ′ =

Original: x ∥ 2 ≤ κ , ∀ x ∈ X for some κ > 0 .
Revised: x ∥ 2 ≤ κ , ∀ x ∈ X for some κ > 0 .

Original: This is satisfied by most commonly used kernels such as the squared exponential (SE) kernel (Rasmussen and Williams, 2006).
Revised: This is satisfied by most commonly used kernels such as the squared exponential (SE) kernel (Rasmussen and Williams, 2006).

Original: 1: Input: Objective function f : X → R , initialization x 0 , iteration number T , learning rates { η t } Tt =1 , projection function P X ( x ) 2: for iteration t = 1 , . .
Revised: 1: Input: Objective function f : X → R , initialization x 0 , iteration number T , learning rates { η t } Tt =1 , projection function P X ( x ) 2: for iteration t = 1 , . .

Original: , T do 3: g ( x t − 1 ) ≈ ∇ f ( x t − 1 ) with (2) 4: x t ← P X ( x t − 1 − η t − 1 g ( x t − 1 )) 5:
Revised: , T do 3: g ( x t − 1 ) ≈ ∇ f ( x t − 1 ) with (2) 4: x t ← P X ( x t − 1 − η t − 1 g ( x t − 1 )) 5:

Original: Query x t to yield y ( x t ) 6: end for 7: Return arg min x 1: T y ( x )
Revised: Query x t to yield y ( x t ) 6: end for 7: Return arg min x 1: T y ( x )

Original: Algorithm 2: Z O RD
Revised: Algorithm 2: Z O RD

Original: (Ours
Revised: (Ours

Original: 1: Input: In addition to the parameters in Algo. 1, set the steps of virtual updates { V t } Tt = 2: for iteration t = 1 , . .
Revised: 1: Input: In addition to the parameters in Algo. 1, set the steps of virtual updates { V t } Tt = 2: for iteration t = 1 , . .

Original: , T do 3: x t, 0 ← x t − 1 4: for iteration τ = 1 , . . .
Revised: , T do 3: x t, 0 ← x t − 1 4: for iteration τ = 1 , . . .

Original: , V t do 5: x t,τ ← P X ( x t,τ − 1 − η t,τ − 1 ∇ µ t − 1 ( x t,τ − 1 )) 6: end for 7:
Revised: , V t do 5: x t,τ ← P X ( x t,τ − 1 − η t,τ − 1 ∇ µ t − 1 ( x t,τ − 1 )) 6: end for 7:

Original: Query x t = x t,τ to yield y ( x t ) 8:
Revised: Query x t = x t,τ to yield y ( x t ) 8:

Original: Update (4) using optimization trajectory 9: end for 10: Return arg min x 1: T y ( x )
Revised: Update (4) using optimization trajectory 9: end for 10: Return arg min x 1: T y ( x )

Original: ZO O PTIMIZATION WITH E STIMATED D ERIVATIVES
Revised: ZO O PTIMIZATION WITH E STIMATED D ERIVATIVES

Original: To solve (1), GD with estimated derivatives (e.g., Algo. 1) has been developed (Flaxman et al., 2005; Ghadimi and Lan, 2013; Nesterov and Spokoiny, 2017; Liu et al., 2018a;b).
Revised: To solve (1), GD with estimated derivatives (e.g., Algo. 1) has been developed (Flaxman et al., 2005; Ghadimi and Lan, 2013; Nesterov and Spokoiny, 2017; Liu et al., 2018a;b).

Original: Particularly, these algorithms first‘ estimate the derivative of f (line 3 of Algo. 1) and then plug the estimated derivative into GD-based methods to obtain the next input for querying (lines 4-5 of Algo. 1).
Revised: Particularly, these algorithms first‘ estimate the derivative of f (line 3 of Algo. 1) and then plug the estimated derivative into GD-based methods to obtain the next input for querying (lines 4-5 of Algo. 1).

Original: In these algorithms, the derivative is typically estimated by averaging the finite difference approximation of the directional derivatives for f along certain directions, which we refer to as the finite difference (FD) method in this paper.
Revised: In these algorithms, the derivative is typically estimated by averaging the finite difference approximation of the directional derivatives for f along certain directions, which we refer to as the finite difference (FD) method in this paper.

Original: For example, given a parameter λ and directions { u i } ni =1 , the derivative ∇ f at any x ∈ X can be estimated by the following FD method (Berahas et al., 2022):
Revised: For example, given a parameter λ and directions { u i } ni =1 , the derivative ∇ f at any x ∈ X can be estimated by the following FD method (Berahas et al., 2022):

Original: [Equation]
Revised: [Equation]

Original: The directions { u i } ni =1 are usually sampled from the standard Gaussian distribution (Nesterov and
Revised: The directions { u i } ni =1 are usually sampled from the standard Gaussian distribution (Nesterov and

Original: Spokoiny, 2017) or uniformly from the unit sphere (Flaxman et al., 2005), or set as the standard basis vectors with 1 at one of its coordinates and 0 otherwise (Lian et al., 2016).
Revised: Spokoiny, 2017) or uniformly from the unit sphere (Flaxman et al., 2005), or set as the standard basis vectors with 1 at one of its coordinates and 0 otherwise (Lian et al., 2016).

Original: As mentioned before, existing FD methods typically require many additional queries (i.e., { x + λ u i } ni =1 ) to achieve an accurate derivative estimation in every iteration of Algo.
Revised: As mentioned before, existing FD methods typically require many additional queries (i.e., { x + λ u i } ni =1 ) to achieve an accurate derivative estimation in every iteration of Algo.

Original: 1 (Berahas et al., 2022), making existing ZO optimization algorithms (Flaxman et al., 2005; Nesterov and Spokoiny, 2017) query-inefficient.
Revised: 1 (Berahas et al., 2022), making existing ZO optimization algorithms (Flaxman et al., 2005; Nesterov and Spokoiny, 2017) query-inefficient.

Original:  ZO O PTIMIZATION WITH T RAJECTORY -I NFORMED D ERIVATIVE E STIMATION (Z O RD)
Revised:  ZO O PTIMIZATION WITH T RAJECTORY -I NFORMED D ERIVATIVE E STIMATION (Z O RD)

Original: To improve existing GD with estimated derivatives (e.g., Algo. 1), we propose the Z O RD algorithm
Revised: To improve existing GD with estimated derivatives (e.g., Algo. 1), we propose the Z O RD algorithm

Original: (Algo. 2), which achieves more query-efficient ZO optimization thanks to our two major contributions.
Revised: (Algo. 2), which achieves more query-efficient ZO optimization thanks to our two major contributions.

Original: Firstly, we propose a derived GP-based derivative estimation method which only uses the optimization trajectory and consequently does not require any additional query for derivative estimation (Sec. 3.1).
Revised: Firstly, we propose a derived GP-based derivative estimation method which only uses the optimization trajectory and consequently does not require any additional query for derivative estimation (Sec. 3.1).

Original: Secondly, thanks to the ability of our method to estimate the derivative at any input in the domain without any additional query and to measure the estimation error in a principled way, we develop the technique of dynamic virtual updates to further improve the query efficiency of our Z O RD (Sec. 3.2).
Revised: Secondly, thanks to the ability of our method to estimate the derivative at any input in the domain without any additional query and to measure the estimation error in a principled way, we develop the technique of dynamic virtual updates to further improve the query efficiency of our Z O RD (Sec. 3.2).

Original: 3. T RAJECTORY -I NFORMED D ERIVATIVE E STIMATION
Revised: 3. T RAJECTORY -I NFORMED D ERIVATIVE E STIMATION

Original: To begin with, note that if a function f follows a GP, then its derivative ∇ f also follows a GP (Rasmussen and Williams, 2006).
Revised: To begin with, note that if a function f follows a GP, then its derivative ∇ f also follows a GP (Rasmussen and Williams, 2006).

Original: This is formalized by our Lemma 1 below (proof in Appx.
Revised: This is formalized by our Lemma 1 below (proof in Appx.

Original: B.1):
Revised: B.1):

Original: Lemma 1 (Derived GP for Derivatives) .
Revised: Lemma 1 (Derived GP for Derivatives) .

Original: If a function f follows a GP: f ∼ GP (cid:0) µ ( · ) , σ 2 ( · , · ) (cid:1) , then
Revised: If a function f follows a GP: f ∼ GP (cid:0) µ ( · ) , σ 2 ( · , · ) (cid:1) , then

Original: [Equation]
Revised: [Equation]

Original: where ∂σ 2 ( · , · ) denotes the cross partial derivative w.r.t the first and second arguments of σ 2 ( · , · ) .
Revised: where ∂σ 2 ( · , · ) denotes the cross partial derivative w.r.t the first and second arguments of σ 2 ( · , · ) .

Original: Lemma 1 then provides us a principled way to estimate the derivative at any input in the domain.
Revised: Lemma 1 then provides us a principled way to estimate the derivative at any input in the domain.

Original: f Follows the Posterior GP.
Revised: f Follows the Posterior GP.

Original: As discussed in Sec. 2.1, we assume that f ∼ GP ( µ ( · ) , k ( · , · )) .
Revised: As discussed in Sec. 2.1, we assume that f ∼ GP ( µ ( · ) , k ( · , · )) .

Original: So, in every iteration t of our Algo. 2, conditioned on the current optimization trajectory D t − 1 ≜
Revised: So, in every iteration t of our Algo. 2, conditioned on the current optimization trajectory D t − 1 ≜

Original: { ( x τ , y τ ) } t − 1 τ =1 , f follows the posterior GP : f ∼ GP (cid:0) µ t − 1 ( · ) , σ 2 t − 1 ( · , · ) (cid:1) with the mean function µ t − 1 ( · ) and the covariance function σ 2 t − 1 ( · , · ) defined as below (Rasmussen and Williams, 2006):
Revised: { ( x τ , y τ ) } t − 1 τ =1 , f follows the posterior GP : f ∼ GP (cid:0) µ t − 1 ( · ) , σ 2 t − 1 ( · , · ) (cid:1) with the mean function µ t − 1 ( · ) and the covariance function σ 2 t − 1 ( · , · ) defined as below (Rasmussen and Williams, 2006):

Original: [Equation]
Revised: [Equation]

Original: where y ⊤ t − 1 ≜ [ y τ ] t − 1 τ =1 and k t − 1 ( x ) ⊤ ≜
Revised: where y ⊤ t − 1 ≜ [ y τ ] t − 1 τ =1 and k t − 1 ( x ) ⊤ ≜

Original: [ k ( x , x τ )]
Revised: [ k ( x , x τ )]

Original: t − 1 τ =1 are ( t − 1) -dimensional row vectors, and
Revised: t − 1 τ =1 are ( t − 1) -dimensional row vectors, and

Original: K t − 1 ≜ [ k ( x τ , x τ ′ )]
Revised: K t − 1 ≜ [ k ( x τ , x τ ′ )]

Original: t − 1 τ,τ ′ =1 is a ( t − 1) ×
Revised: t − 1 τ,τ ′ =1 is a ( t − 1) ×

Original: ( t − 1) -dimensional matrix.
Revised: ( t − 1) -dimensional matrix.

Original: Define σ 2 t − 1 ( x ) ≜ σ 2 t − 1 ( x , x ) , the posterior distribution at x is Gaussian with mean µ t − 1 ( x ) and variance σ 2 t − 1 ( x ) .
Revised: Define σ 2 t − 1 ( x ) ≜ σ 2 t − 1 ( x , x ) , the posterior distribution at x is Gaussian with mean µ t − 1 ( x ) and variance σ 2 t − 1 ( x ) .

Original: ∇ f Follows the Derived GP for Derivatives.
Revised: ∇ f Follows the Derived GP for Derivatives.

Original: Substituting (3) into Lemma 1, we have that
Revised: Substituting (3) into Lemma 1, we have that

Original: [Equation]
Revised: [Equation]

Original: in which the mean ∇ µ t − 1 ( x ) at x and the covariance ∂σ 2 t − 1 ( x , x ′ ) at x , x ′ are ∇
Revised: in which the mean ∇ µ t − 1 ( x ) at x and the covariance ∂σ 2 t − 1 ( x , x ′ ) at x , x ′ are ∇

Original: [Equation]
Revised: [Equation]

Original: in which ∂ z k t − 1 ( z ) ≜ [ ∂ z k ( z , x τ )]
Revised: in which ∂ z k t − 1 ( z ) ≜ [ ∂ z k ( z , x τ )]

Original: t − 1 τ =1 is a ( t − 1) ×
Revised: t − 1 τ =1 is a ( t − 1) ×

Original: d -dimensional matrix and ∂ z ∂ z ′ k
Revised: d -dimensional matrix and ∂ z ∂ z ′ k

Original: ( z , z ′ ) is a d × d -dimensional matrix.
Revised: ( z , z ′ ) is a d × d -dimensional matrix.

Original: Therefore, ∇ µ t − 1 ( x ) is a d -dimensional vector and ∂σ 2 t − 1 ( x , x ′ ) is a d × d -dimensional matrix.
Revised: Therefore, ∇ µ t − 1 ( x ) is a d -dimensional vector and ∂σ 2 t − 1 ( x , x ′ ) is a d × d -dimensional matrix.

Original: We refer to this GP (4) followed by ∇ f as the derived GP for derivatives .
Revised: We refer to this GP (4) followed by ∇ f as the derived GP for derivatives .

Original: So, define ∂σ 2 t − 1 ( x ) ≜ ∂σ 2 t − 1 ( x , x ) , we have that for any input x ∈ X , the derivative ∇ f ( x ) at x follows a d -dimensional Gaussian distribution: ∇ f ( x ) ∼ N ( ∇ µ t − 1 ( x ) , ∂σ 2 t − 1 ( x )) .
Revised: So, define ∂σ 2 t − 1 ( x ) ≜ ∂σ 2 t − 1 ( x , x ) , we have that for any input x ∈ X , the derivative ∇ f ( x ) at x follows a d -dimensional Gaussian distribution: ∇ f ( x ) ∼ N ( ∇ µ t − 1 ( x ) , ∂σ 2 t − 1 ( x )) .

Original: This allows us to (a) estimate the derivative ∇ f ( x ) at any input x ∈ X using the posterior mean ∇ µ t − 1 ( x ) of the derived GP for derivatives (4):
Revised: This allows us to (a) estimate the derivative ∇ f ( x ) at any input x ∈ X using the posterior mean ∇ µ t − 1 ( x ) of the derived GP for derivatives (4):

Original: [Equation]
Revised: [Equation]

Original: and (b) employ the posterior covariance matrix ∂σ 2 t − 1 ( x ) to obtain a principled measure of the uncertainty for this derivative estimation, which together constitute our novel derivative estimation.
Revised: and (b) employ the posterior covariance matrix ∂σ 2 t − 1 ( x ) to obtain a principled measure of the uncertainty for this derivative estimation, which together constitute our novel derivative estimation.

Original: Remarkably, our derivative estimation only makes use of the naturally available optimization trajectory
Revised: Remarkably, our derivative estimation only makes use of the naturally available optimization trajectory

Original: D t − 1 and does not need any additional query , which is in stark contrast to the existing FD methods (e.g., (2)) that require many additional queries for their derivative estimation.
Revised: D t − 1 and does not need any additional query , which is in stark contrast to the existing FD methods (e.g., (2)) that require many additional queries for their derivative estimation.

Original: Moreover, our principled measure of uncertainty allows us to perform dynamic virtual updates (Sec. 3.2) and theoretically guarantee the quality of our derivative estimation (Sec. 4.1).
Revised: Moreover, our principled measure of uncertainty allows us to perform dynamic virtual updates (Sec. 3.2) and theoretically guarantee the quality of our derivative estimation (Sec. 4.1).

Original: 3. D YNAMIC VIRTUAL UPDATES
Revised: 3. D YNAMIC VIRTUAL UPDATES

Original: Note that our derived GP-based derivative estimation (6) can estimate the derivative at any input x within the domain.
Revised: Note that our derived GP-based derivative estimation (6) can estimate the derivative at any input x within the domain.

Original: As a result, in every iteration t of our Z O RD algorithm, for a step τ ≥ 1 , after performing a GD update using the estimated derivative at x t,τ − 1 (i.e., ∇ µ t − 1 ( x t,τ − 1 ) ) to reach the input x t,τ (line 5 of Algo. 2), we can again estimate the derivative at x t,τ (i.e., ∇ µ t − 1 ( x t,τ ) ) and then perform another GD update to reach x t,τ +1 without requiring any additional query.
Revised: As a result, in every iteration t of our Z O RD algorithm, for a step τ ≥ 1 , after performing a GD update using the estimated derivative at x t,τ − 1 (i.e., ∇ µ t − 1 ( x t,τ − 1 ) ) to reach the input x t,τ (line 5 of Algo. 2), we can again estimate the derivative at x t,τ (i.e., ∇ µ t − 1 ( x t,τ ) ) and then perform another GD update to reach x t,τ +1 without requiring any additional query.

Original: This process can be repeated for multiple steps, and can further improve the query efficiency of our Z O RD.
Revised: This process can be repeated for multiple steps, and can further improve the query efficiency of our Z O RD.

Original: Formally, given the projection function P X ( x ) ≜ arg min z ∈X ∥ x − z ∥ 22 / 2 and learning rates { η t,τ } V t − 1 τ =0 , we perform the following virtual updates for V t steps (lines 4-6 of Algo. 2):
Revised: Formally, given the projection function P X ( x ) ≜ arg min z ∈X ∥ x − z ∥ 22 / 2 and learning rates { η t,τ } V t − 1 τ =0 , we perform the following virtual updates for V t steps (lines 4-6 of Algo. 2):

Original: [Equation]
Revised: [Equation]

Original: and then choose the last x t,V t to query (i.e., line 7 of Algo. 2).
Revised: and then choose the last x t,V t to query (i.e., line 7 of Algo. 2).

Original: Importantly, these multi-step virtual
Revised: Importantly, these multi-step virtual

Original: GD updates are only feasible in our Z O RD
Revised: GD updates are only feasible in our Z O RD

Original: (Algo. 2) because our derivative estimator (6) does not require any new query in all these steps , whereas the existing FD methods require additional queries to estimate the derivative in every step.
Revised: (Algo. 2) because our derivative estimator (6) does not require any new query in all these steps , whereas the existing FD methods require additional queries to estimate the derivative in every step.

Original: The number of steps for our virtual updates (i.e., V t ) induces an intriguing trade-off: An overly small
Revised: The number of steps for our virtual updates (i.e., V t ) induces an intriguing trade-off: An overly small

Original: V t may not be able to fully exploit the benefit of our derivative estimation (6) which is free from the requirement for additional queries, yet an excessively large V t may lead to the usage of inaccurate derivative estimations which can hurt the performance (validated in Appx. D.2).
Revised: V t may not be able to fully exploit the benefit of our derivative estimation (6) which is free from the requirement for additional queries, yet an excessively large V t may lead to the usage of inaccurate derivative estimations which can hurt the performance (validated in Appx. D.2).

Original: Remarkably, (4) allows us to dynamically choose V t by inspecting our principled measure of the predictive uncertainty
Revised: Remarkably, (4) allows us to dynamically choose V t by inspecting our principled measure of the predictive uncertainty

Original: (i.e., ∂σ 2 t − 1 ( x ) ) for every derivative estimation.
Revised: (i.e., ∂σ 2 t − 1 ( x ) ) for every derivative estimation.

Original: Specifically, after reaching the input x t,τ , we continue the virtual updates (to reach x t,τ +1 ) if our predictive uncertainty is small, i.e., if (cid:13)(cid:13)
Revised: Specifically, after reaching the input x t,τ , we continue the virtual updates (to reach x t,τ +1 ) if our predictive uncertainty is small, i.e., if (cid:13)(cid:13)

Original: ∂σ 2 t − 1 ( x t,τ ) (cid:13)(cid:13) 2 ≤ c where c is a confidence threshold; otherwise, we terminate the virtual updates and let V t =
Revised: ∂σ 2 t − 1 ( x t,τ ) (cid:13)(cid:13) 2 ≤ c where c is a confidence threshold; otherwise, we terminate the virtual updates and let V t =

Original: τ since the derivative estimation at x t,τ is likely unreliable.
Revised: τ since the derivative estimation at x t,τ is likely unreliable.

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: In this section, we present theoretical guarantees on the estimation error of our derivative estimation (Sec. 4.1) and the convergence of our Z O RD algorithm
Revised: In this section, we present theoretical guarantees on the estimation error of our derivative estimation (Sec. 4.1) and the convergence of our Z O RD algorithm

Original: (Sec. 4.2).
Revised: (Sec. 4.2).

Original: 4. D ERIVATIVE E STIMATION E RROR
Revised: 4. D ERIVATIVE E STIMATION E RROR

Original: To begin with, we derive a theoretical guarantee on the error of our derivative estimation at any x .
Revised: To begin with, we derive a theoretical guarantee on the error of our derivative estimation at any x .

Original: Theorem 1 (Derivative Estimation Error) .
Revised: Theorem 1 (Derivative Estimation Error) .

Original: Let δ ∈ (0 , 1) and β ≜ (cid:113) d
Revised: Let δ ∈ (0 , 1) and β ≜ (cid:113) d

Original: + 2( √ d + 1) ln(1 /δ ) .
Revised: + 2( √ d + 1) ln(1 /δ ) .

Original: For any x ∈ X and any t ≥ 1 , the following holds with probability of at least 1 − δ ,
Revised: For any x ∈ X and any t ≥ 1 , the following holds with probability of at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Thm. 1 (proof in Appx.
Revised: Thm. 1 (proof in Appx.

Original: B.2) has presented an upper bound on the error of our derivative estimation (6) at any x ∈ X in terms of (cid:13)(cid:13)
Revised: B.2) has presented an upper bound on the error of our derivative estimation (6) at any x ∈ X in terms of (cid:13)(cid:13)

Original: ∂σ 2 t ( x ) (cid:13)(cid:13) 2 , which is a measure of the uncertainty about our derivative estimation at x (Sec. 3.1).
Revised: ∂σ 2 t ( x ) (cid:13)(cid:13) 2 , which is a measure of the uncertainty about our derivative estimation at x (Sec. 3.1).

Original: This hence implies that the threshold c applied to our predictive uncertainty (cid:13)(cid:13)
Revised: This hence implies that the threshold c applied to our predictive uncertainty (cid:13)(cid:13)

Original: ∂σ 2 t ( x ) (cid:13)(cid:13) 2 (Sec. 3.2) also ensures that the derivative estimation error is small during our dynamic virtual updates.
Revised: ∂σ 2 t ( x ) (cid:13)(cid:13) 2 (Sec. 3.2) also ensures that the derivative estimation error is small during our dynamic virtual updates.

Original: Next, we show in the following theorem (proof in Appx.
Revised: Next, we show in the following theorem (proof in Appx.

Original: B.3) that our upper bound on the estimation error from Thm.
Revised: B.3) that our upper bound on the estimation error from Thm.

Original: 1 is non-increasing as the number of function queries is increased.
Revised: 1 is non-increasing as the number of function queries is increased.

Original: Theorem 2 (Non-Increasing Error) .
Revised: Theorem 2 (Non-Increasing Error) .

Original: For any x ∈ X and any t ≥ 1 , we have that
Revised: For any x ∈ X and any t ≥ 1 , we have that

Original: [Equation]
Revised: [Equation]

Original: Let δ ∈ (0 , 1) .
Revised: Let δ ∈ (0 , 1) .

Original: Define r ≜ max x ∈X ,t ≥ 1 (cid:13)(cid:13)
Revised: Define r ≜ max x ∈X ,t ≥ 1 (cid:13)(cid:13)

Original: ∂σ 2 t ( x ) (cid:13)(cid:13) 2 / (cid:13)(cid:13)
Revised: ∂σ 2 t ( x ) (cid:13)(cid:13) 2 / (cid:13)(cid:13)

Original: ∂σ 2 t − 1 ( x ) (cid:13)(cid:13) 2 , given the β in Thm.
Revised: ∂σ 2 t − 1 ( x ) (cid:13)(cid:13) 2 , given the β in Thm.

Original: 1 , we then have that r ∈
Revised: 1 , we then have that r ∈

Original: [1 / (1 + 1 /σ 2 ) , 1] , and that with probability of at least 1 − δ ,
Revised: [1 / (1 + 1 /σ 2 ) , 1] , and that with probability of at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Thm. 2 shows that our upper bound on the derivative estimation error (i.e., β (cid:13)(cid:13) ∂σ 2 t ( x ) (cid:13)(cid:13) 2 from Thm. 1) is guaranteed to be non-increasing in the entire domain as the number of function queries is increased.
Revised: Thm. 2 shows that our upper bound on the derivative estimation error (i.e., β (cid:13)(cid:13) ∂σ 2 t ( x ) (cid:13)(cid:13) 2 from Thm. 1) is guaranteed to be non-increasing in the entire domain as the number of function queries is increased.

Original: Moreover, in some situations (i.e., when r < 1 ), our upper bound on the estimation error is even exponentially decreasing.
Revised: Moreover, in some situations (i.e., when r < 1 ), our upper bound on the estimation error is even exponentially decreasing.

Original: Of note, r characterizes how fast the uncertainty about our derivative estimation (measured by (cid:13)(cid:13)
Revised: Of note, r characterizes how fast the uncertainty about our derivative estimation (measured by (cid:13)(cid:13)

Original: ∂σ 2 t ( x ) (cid:13)(cid:13) 2 ) is reduced across the domain.
Revised: ∂σ 2 t ( x ) (cid:13)(cid:13) 2 ) is reduced across the domain.

Original: Since GD-based algorithms usually perform a local search in a neighborhood (especially for the problems with high-dimensional input spaces), all the inputs within the local region are expected to be close to each other (measured by the kernel function k ).
Revised: Since GD-based algorithms usually perform a local search in a neighborhood (especially for the problems with high-dimensional input spaces), all the inputs within the local region are expected to be close to each other (measured by the kernel function k ).

Original: Moreover, as the objective function is usually smooth in the local region (i.e., its derivatives are continuous), reducing the uncertainty of the derivative at an input x t (i.e., by querying x t ) is also expected to decrease the uncertainty of the derivatives at the other inputs in the same local region (i.e., decrease (cid:13)(cid:13)
Revised: Moreover, as the objective function is usually smooth in the local region (i.e., its derivatives are continuous), reducing the uncertainty of the derivative at an input x t (i.e., by querying x t ) is also expected to decrease the uncertainty of the derivatives at the other inputs in the same local region (i.e., decrease (cid:13)(cid:13)

Original: ∂σ 2 t ( x ) (cid:13)(cid:13) 2 ).
Revised: ∂σ 2 t ( x ) (cid:13)(cid:13) 2 ).

Original: So, r < 1 is expected to be a reasonable condition that can be satisfied in practice.
Revised: So, r < 1 is expected to be a reasonable condition that can be satisfied in practice.

Original: This will also be corroborated by our empirical results (e.g., Figs. 1 and 2), which demonstrates that the error of our derivative estimation (6) is indeed reduced very fast.
Revised: This will also be corroborated by our empirical results (e.g., Figs. 1 and 2), which demonstrates that the error of our derivative estimation (6) is indeed reduced very fast.

Original: Our GP-based Method (6) vs. Existing FD Methods.
Revised: Our GP-based Method (6) vs. Existing FD Methods.

Original: Our derivative estimation method based on the derived GP (6) is superior to the traditional FD methods (e.g., (2)) in a number of major aspects.
Revised: Our derivative estimation method based on the derived GP (6) is superior to the traditional FD methods (e.g., (2)) in a number of major aspects.

Original: (a) Our method (6) is more favorable in problems with high-dimensional input spaces (i
Revised: (a) Our method (6) is more favorable in problems with high-dimensional input spaces (i

Original: e., when d
Revised: e., when d

Original: is
Revised: is

Original: large), because our derivative estimation error depends on d via O ( √ d ) , whereas the error of the existing FD methods scales as O ( d √ d ) (Berahas
Revised: large), because our derivative estimation error depends on d via O ( √ d ) , whereas the error of the existing FD methods scales as O ( d √ d ) (Berahas

Original: et
Revised: et

Original: al.
Revised: al.

Original: (b) Our derivative estimation error
Revised: (b) Our derivative estimation error

Original: is exponentially
Revised: is exponentially

Original: decreasing in some situations (i.e.
Revised: decreasing in some situations (i.e.

Original: when r <
Revised: when r <

Original: Thm
Revised: Thm

Original: , which is unachievable
Revised: , which is unachievable

Original: for the existing
Revised: for the existing

Original: FD methods since they can only attain
Revised: FD methods since they can only attain

Original: a
Revised: a

Original: polynomial rate of reduction (
Revised: polynomial rate of reduction (

Original: Berahas
Revised: Berahas

Original: et al., 2022)
Revised: et al., 2022)

Original: (c) Our method (6) does not need any additional query to
Revised: (c) Our method (6) does not need any additional query to

Original: estimate
Revised: estimate

Original: the derivative (but only requires
Revised: the derivative (but only requires

Original: the
Revised: the

Original: optimization trajectory), whereas the existing FD methods require additional queries for
Revised: optimization trajectory), whereas the existing FD methods require additional queries for

Original: every derivative
Revised: every derivative

Original: estimation.
Revised: estimation.

Original: (d)
Revised: (d)

Original: Our
Revised: Our

Original: method (6) is equipped with a principled measure of
Revised: method (6) is equipped with a principled measure of

Original: the predictive uncertainty and hence the estimation error for derivative estimation (i.e., via (cid:13)(cid:13)
Revised: the predictive uncertainty and hence the estimation error for derivative estimation (i.e., via (cid:13)(cid:13)

Original: ∂σ 2 t ( x ) (cid:13)(cid:13) 2 , Thm. 1), which is typically unavailable for the existing FD methods.
Revised: ∂σ 2 t ( x ) (cid:13)(cid:13) 2 , Thm. 1), which is typically unavailable for the existing FD methods.

Original: (e) Our method (6), unlike the existing
Revised: (e) Our method (6), unlike the existing

Original: FD methods, makes it possible to apply the technique of dynamic virtual updates (Sec. 3.2) thanks to its capability of estimating the derivative at any input in the domain without requiring any additional query and measuring the estimation error in a principled way (Thm. 1).
Revised: FD methods, makes it possible to apply the technique of dynamic virtual updates (Sec. 3.2) thanks to its capability of estimating the derivative at any input in the domain without requiring any additional query and measuring the estimation error in a principled way (Thm. 1).

Original: 4. C ONVERGENCE A NALYSIS
Revised: 4. C ONVERGENCE A NALYSIS

Original: To analyze the convergence of our Z O RD, besides our main assumption that f is sampled from a GP (Sec. 2.1), we assume that f is L c -Lipchitz continuous for L c > 0 .
Revised: To analyze the convergence of our Z O RD, besides our main assumption that f is sampled from a GP (Sec. 2.1), we assume that f is L c -Lipchitz continuous for L c > 0 .

Original: This is a mild assumption since it has been shown that a function f sampled from a GP is Lipchitz continuous with high probability for commonly used kernels, e.g., the SE kernel and Mat´ern kernel with ν > 2 (Srinivas et al., 2010).
Revised: This is a mild assumption since it has been shown that a function f sampled from a GP is Lipchitz continuous with high probability for commonly used kernels, e.g., the SE kernel and Mat´ern kernel with ν > 2 (Srinivas et al., 2010).

Original: We also assume that f is L s -Lipchitz smooth, which is commonly adopted in the analysis GD-based algorithms (J Reddi et al., 2016).
Revised: We also assume that f is L s -Lipchitz smooth, which is commonly adopted in the analysis GD-based algorithms (J Reddi et al., 2016).

Original: We aim to prove the convergence of our Z O RD for nonconvex f by analyzing how fast it converges to a stationary point (Ghadimi and Lan, 2013; Liu et al., 2018a).
Revised: We aim to prove the convergence of our Z O RD for nonconvex f by analyzing how fast it converges to a stationary point (Ghadimi and Lan, 2013; Liu et al., 2018a).

Original: Specifically, we follow the common practice of previous works (J Reddi et al., 2016;
Revised: Specifically, we follow the common practice of previous works (J Reddi et al., 2016;

Original: Liu et al., 2018b) to analyze the following derivative mapping:
Revised: Liu et al., 2018b) to analyze the following derivative mapping:

Original: [Equation]
Revised: [Equation]

Original: The convergence of our Z O RD is formally guaranteed by Thm.
Revised: The convergence of our Z O RD is formally guaranteed by Thm.

Original: 3 below (proof in Appx.
Revised: 3 below (proof in Appx.

Original: B.4).
Revised: B.4).

Original: Theorem 3 (Convergence of Z O RD) .
Revised: Theorem 3 (Convergence of Z O RD) .

Original: Let δ ∈ (0 , 1) .
Revised: Let δ ∈ (0 , 1) .

Original: Suppose our Z O RD
Revised: Suppose our Z O RD

Original: (Algo. 2) is run with
Revised: (Algo. 2) is run with

Original: V t = V and η t,τ = η ≤ 1 /L s for any t and τ .
Revised: V t = V and η t,τ = η ≤ 1 /L s for any t and τ .

Original: Then with probability of at least 1 − δ , when r < 1 , min t ≤ T
Revised: Then with probability of at least 1 − δ , when r < 1 , min t ≤ T

Original: [Equation]
Revised: [Equation]

Original: where α ≜ κ (cid:113) d + 2( √ d + 

Revised: where α ≜ κ (cid:113) d + 2( √ d +

Original: ) ln( V T/δ ) .
Revised: ) ln( V T/δ ) .

Original: When r = 1 , we instead have 2 = 

Revised: When r = 1 , we instead have 2 =

Original: α 

Revised: α

Original: + (2 L c + 1 /η )
Revised: + (2 L c + 1 /η )

Original: α .
Revised: α .

Original: In the upper bound of Thm. 3, the term 1 represents the convergence rate of (projected) GD when the true derivative is used and it asymptotically goes to 0 as T increases; the term 2 corresponds to the impact of the error of our derivative estimation (6) on the convergence.
Revised: In the upper bound of Thm. 3, the term 1 represents the convergence rate of (projected) GD when the true derivative is used and it asymptotically goes to 0 as T increases; the term 2 corresponds to the impact of the error of our derivative estimation (6) on the convergence.

Original: In situations where r <which is a reasonably achievable condition as we have discussed in Sec. 4.1, the term 2 will also asymptotically approach 0 .
Revised: In situations where r <which is a reasonably achievable condition as we have discussed in Sec. 4.1, the term 2 will also asymptotically approach 0 .

Original: This, remarkably, suggests that the impact of the derivative estimation error on the convergence vanishes asymptotically and our Z O RD algorithm is guaranteed to converge to a stationary point (i.e., min t ≤ T 1 V (cid:80) V − 1 τ =

Revised: This, remarkably, suggests that the impact of the derivative estimation error on the convergence vanishes asymptotically and our Z O RD algorithm is guaranteed to converge to a stationary point (i.e., min t ≤ T 1 V (cid:80) V − 1 τ =

Original: ∥ G t,τ ∥ 22 approaches 0 ) at the rate of O (1 /T ) when r < 1 .
Revised: ∥ G t,τ ∥ 22 approaches 0 ) at the rate of O (1 /T ) when r < 1 .

Original: This is unattainable by existing ZO optimization algorithms using FD-based derivative estimation (Nesterov and Spokoiny, 2017; Liu et al., 2018b), because these methods typically converge to a stationary point at the rate of O (1 /T + const . ) with a constant learning rate.
Revised: This is unattainable by existing ZO optimization algorithms using FD-based derivative estimation (Nesterov and Spokoiny, 2017; Liu et al., 2018b), because these methods typically converge to a stationary point at the rate of O (1 /T + const . ) with a constant learning rate.

Original: Even when r = 1 where the term 2 becomes a constant independent of T , our Thm. 3 is still superior to the convergence of these existing works because our result (Thm. 3) is based on the worst-case analysis whereas these works are typically based on the average-case analysis, i.e., their results only hold in expectation over the randomly sampled directions for derivative estimation.
Revised: Even when r = 1 where the term 2 becomes a constant independent of T , our Thm. 3 is still superior to the convergence of these existing works because our result (Thm. 3) is based on the worst-case analysis whereas these works are typically based on the average-case analysis, i.e., their results only hold in expectation over the randomly sampled directions for derivative estimation.

Original: This means that their convergence may become even worse when inappropriate directions are used, e.g., directions that are nearly orthogonal to the true derivative which commonly happens in high-dimensional input spaces.
Revised: This means that their convergence may become even worse when inappropriate directions are used, e.g., directions that are nearly orthogonal to the true derivative which commonly happens in high-dimensional input spaces.

Original: In addition, given a fixed T , our Z O RD enjoys a query complexity (i.e., the total number of required function queries in
Revised: In addition, given a fixed T , our Z O RD enjoys a query complexity (i.e., the total number of required function queries in

Original: T iterations) of O ( T ) , which significantly improves over the O ( nT ) of the existing works based on
Revised: T iterations) of O ( T ) , which significantly improves over the O ( nT ) of the existing works based on

Original: FD ( n in Sec. 2.2).
Revised: FD ( n in Sec. 2.2).

Original: The impacts of the number of steps of our virtual updates (i.e., V ) are partially reflected in Thm. 3.
Revised: The impacts of the number of steps of our virtual updates (i.e., V ) are partially reflected in Thm. 3.

Original: Specifically, a larger V improves the reduction rate of the term 1 because a larger number of virtual
Revised: Specifically, a larger V improves the reduction rate of the term 1 because a larger number of virtual

Original: GD updates (without requiring additional queries) will be applied in our Z O RD algorithm.
Revised: GD updates (without requiring additional queries) will be applied in our Z O RD algorithm.

Original: This is also unachievable by existing ZO optimization algorithms using FD-based derivative estimation since they require additional queries for the derivative estimation in their every GD update.
Revised: This is also unachievable by existing ZO optimization algorithms using FD-based derivative estimation since they require additional queries for the derivative estimation in their every GD update.

Original: Meanwhile, a larger V may also negatively impact the performance of our Z O RD since it may lead to the use of those estimated derivatives with large estimation errors (Sec. 3.2).
Revised: Meanwhile, a larger V may also negatively impact the performance of our Z O RD since it may lead to the use of those estimated derivatives with large estimation errors (Sec. 3.2).

Original: However, this negative impact has only been implicitly accounted for by the term 2 because this term comes from our Thm. 2, which is based on a worst-case analysis and gives an upper bound on the derivative estimation error for all inputs in the domain X .
Revised: However, this negative impact has only been implicitly accounted for by the term 2 because this term comes from our Thm. 2, which is based on a worst-case analysis and gives an upper bound on the derivative estimation error for all inputs in the domain X .

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: is better).
Revised: is better).

Original: Each curve is the mean ± standard error from five independent runs.
Revised: Each curve is the mean ± standard error from five independent runs.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we firstly empirically verify the efficacy of our derived GP-based derivative estimator (6) in Sec. 5.1, and then demonstrate that our Z O RD outperforms existing baseline methods for ZO optimization using synthetic experiments (Sec. 5.2) and real-world experiments (Secs. 5.3, 5.4).
Revised: In this section, we firstly empirically verify the efficacy of our derived GP-based derivative estimator (6) in Sec. 5.1, and then demonstrate that our Z O RD outperforms existing baseline methods for ZO optimization using synthetic experiments (Sec. 5.2) and real-world experiments (Secs. 5.3, 5.4).

Original: 5. DERIVATIVE ESTIMATION
Revised: 5. DERIVATIVE ESTIMATION

Original: Here we investigate the efficacy of our derivative estimator (6) based on the derived GP for derivatives (4).
Revised: Here we investigate the efficacy of our derivative estimator (6) based on the derived GP for derivatives (4).

Original: Specifically, we sample a function f (defined on a one-dimensional domain) from a GP using the SE kernel, and then use a set of randomly selected inputs as well as their noisy observations (as optimization trajectory) to calculate our derived GP for derivatives.
Revised: Specifically, we sample a function f (defined on a one-dimensional domain) from a GP using the SE kernel, and then use a set of randomly selected inputs as well as their noisy observations (as optimization trajectory) to calculate our derived GP for derivatives.

Original: The results (Fig. 1) illustrate a number of interesting insights.
Revised: The results (Fig. 1) illustrate a number of interesting insights.

Original: Firstly, in regions where (even only a few) function queries are performed (e.g., in the region of [ − 3 , 0] ), our estimated derivative (i.e., ∇ µ t − 1 ( x ) (6)) generally aligns with the groundtruth derivative (i.e., ∇ f ( x ) ) and our estimation uncertainty (i.e., characterized by (cid:13) (cid:13) ∂σ 2 t − 1 ( x ) (cid:13) (cid:13) 2 ) shrinks compared with other un-queried regions.
Revised: Firstly, in regions where (even only a few) function queries are performed (e.g., in the region of [ − 3 , 0] ), our estimated derivative (i.e., ∇ µ t − 1 ( x ) (6)) generally aligns with the groundtruth derivative (i.e., ∇ f ( x ) ) and our estimation uncertainty (i.e., characterized by (cid:13) (cid:13) ∂σ 2 t − 1 ( x ) (cid:13) (cid:13) 2 ) shrinks compared with other un-queried regions.

Original: These results hence demonstrate that our (4) is able to accurately estimate derivatives and reliably quantify the uncertainty of these estimations within the regions where function queries are performed.
Revised: These results hence demonstrate that our (4) is able to accurately estimate derivatives and reliably quantify the uncertainty of these estimations within the regions where function queries are performed.

Original: Secondly, as more input queries are collected (i.e., from left to right in Fig. 1), the uncertainty (cid:13)(cid:13)
Revised: Secondly, as more input queries are collected (i.e., from left to right in Fig. 1), the uncertainty (cid:13)(cid:13)

Original: ∂σ 2 t − 1 ( x ) (cid:13)(cid:13) 2 in the entire domain is decreased in general.
Revised: ∂σ 2 t − 1 ( x ) (cid:13)(cid:13) 2 in the entire domain is decreased in general.

Original: This provides an empirical justification for our Thm. 2 which guarantees non-increasing uncertainty and hence non-increasing estimation error.
Revised: This provides an empirical justification for our Thm. 2 which guarantees non-increasing uncertainty and hence non-increasing estimation error.

Original: Lastly, note that with onlyqueries (rightmost figure), our derivative estimator is already able to accurately estimate the derivative in the entire domain, which represents a remarkable reduction rate of our derivative estimation error.
Revised: Lastly, note that with onlyqueries (rightmost figure), our derivative estimator is already able to accurately estimate the derivative in the entire domain, which represents a remarkable reduction rate of our derivative estimation error.

Original: Next, we compare our derivative estimator (6) with the FD estimator (Sec. 2.2).
Revised: Next, we compare our derivative estimator (6) with the FD estimator (Sec. 2.2).

Original: Specifically, using the Ackley function with d = 10 (see Appx.
Revised: Specifically, using the Ackley function with d = 10 (see Appx.

Original: C.1), we firstly select an input x 

Revised: C.1), we firstly select an input x

Original: and then follow the
Revised: and then follow the

Original: FD method (2) to randomly sample n directions { u i } ni =1 from the standard Gaussian distribution, to construct input queries { x 0 + λ u i } n i =1 (see Sec. 2.2).
Revised: FD method (2) to randomly sample n directions { u i } ni =1 from the standard Gaussian distribution, to construct input queries { x 0 + λ u i } n i =1 (see Sec. 2.2).

Original: Next, these queries and their observations are ( a ) used as the optimization trajectory to apply our derivative estimator (6), and ( b ) used by the FD method to estimate the derivative following (2).
Revised: Next, these queries and their observations are ( a ) used as the optimization trajectory to apply our derivative estimator (6), and ( b ) used by the FD method to estimate the derivative following (2).

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: 2a (for two different values of λ ), in which for both our derived GP-based estimator (6) and the FD estimator, we measure the cosine similarity (larger is better) and Euclidean distance (smaller is better) between the estimated derivative and the true derivative at x 0 .
Revised: 2a (for two different values of λ ), in which for both our derived GP-based estimator (6) and the FD estimator, we measure the cosine similarity (larger is better) and Euclidean distance (smaller is better) between the estimated derivative and the true derivative at x 0 .

Original: The figures show that our derivative estimation error enjoys a faster rate of reduction compared with the FD method, which corroborates our theoretical insights
Revised: The figures show that our derivative estimation error enjoys a faster rate of reduction compared with the FD method, which corroborates our theoretical insights

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Metric
Revised: Metric

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: from Thm.
Revised: from Thm.

Original: 2 (Sec. 4.1) positing that our estimation error can be rapidly decreasing.
Revised: 2 (Sec. 4.1) positing that our estimation error can be rapidly decreasing.

Original: Subsequently, to further highlight our advantage of being able to exploit the optimization trajectory and hence to eliminate the need for additional function queries (Sec. 4.1), we perform another comparison where our derived GP-based estimator (6) only utilizes 20 queries from the optimization trajectory (sampled using the same method above) for derivative estimation.
Revised: Subsequently, to further highlight our advantage of being able to exploit the optimization trajectory and hence to eliminate the need for additional function queries (Sec. 4.1), we perform another comparison where our derived GP-based estimator (6) only utilizes 20 queries from the optimization trajectory (sampled using the same method above) for derivative estimation.

Original: The results (Fig. 2b) show that even with only these 20 queries (without any additional function query), our derivative estimator (6) achieves comparable or better estimation errors than FD using as many as 80 additional queries.
Revised: The results (Fig. 2b) show that even with only these 20 queries (without any additional function query), our derivative estimator (6) achieves comparable or better estimation errors than FD using as many as 80 additional queries.

Original: Overall, the results in Fig.
Revised: Overall, the results in Fig.

Original: 2 have provided empirical supports for the superiority of our derived GP-based derivative estimation (6), which substantiates our theoretical justifications in Sec. 4.1.
Revised: 2 have provided empirical supports for the superiority of our derived GP-based derivative estimation (6), which substantiates our theoretical justifications in Sec. 4.1.

Original: 5. S YNTHETIC E XPERIMENTS
Revised: 5. S YNTHETIC E XPERIMENTS

Original: Here we adopt the widely use Ackley and Levy functions with various dimensions (Eriksson et al., 2019) to show the superiority of our Z O RD.
Revised: Here we adopt the widely use Ackley and Levy functions with various dimensions (Eriksson et al., 2019) to show the superiority of our Z O RD.

Original: We compare Z O RD with a number of representative baselines for ZO optimization, e.g., RGF (Nesterov and Spokoiny, 2017) which uses FD for derivative estimation, PRGF (Cheng et al., 2021) which is a recent extension of RGF, GLD (Golovin et al., 2020) which is a recent ZO optimization algorithm based on direct search, and TuRBO (Eriksson et al., 2019) which is a highly performant Bayesian optimization (BO) algorithm.
Revised: We compare Z O RD with a number of representative baselines for ZO optimization, e.g., RGF (Nesterov and Spokoiny, 2017) which uses FD for derivative estimation, PRGF (Cheng et al., 2021) which is a recent extension of RGF, GLD (Golovin et al., 2020) which is a recent ZO optimization algorithm based on direct search, and TuRBO (Eriksson et al., 2019) which is a highly performant Bayesian optimization (BO) algorithm.

Original: We also evaluate the performance of a first-order optimization algorithm, i.e., GD with true derivatives.
Revised: We also evaluate the performance of a first-order optimization algorithm, i.e., GD with true derivatives.

Original: More details are in Appx.
Revised: More details are in Appx.

Original: C.1.
Revised: C.1.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: 3, where Z O RD outperforms all other ZO optimization algorithms.
Revised: 3, where Z O RD outperforms all other ZO optimization algorithms.

Original: Particularly, Z O RD considerably outperforms both RGF and PRGF, which can be attributed to our two major contributions.
Revised: Particularly, Z O RD considerably outperforms both RGF and PRGF, which can be attributed to our two major contributions.

Original: Firstly, our derivative estimator (6) used by Z O RD is more accurate and more query-efficient than the FD method adopted by RGF and PRGF, as theoretically justified in Sec.
Revised: Firstly, our derivative estimator (6) used by Z O RD is more accurate and more query-efficient than the FD method adopted by RGF and PRGF, as theoretically justified in Sec.

Original: 4.1 and empirically demonstrated in Sec. 5.1.
Revised: 4.1 and empirically demonstrated in Sec. 5.1.

Original: Secondly, our dynamic virtual updates (Sec. 3.2) can perform multi-step GD updates without requiring any additional query, which further improves the performance of Z O RD (validated in Appx.
Revised: Secondly, our dynamic virtual updates (Sec. 3.2) can perform multi-step GD updates without requiring any additional query, which further improves the performance of Z O RD (validated in Appx.

Original: D.2).
Revised: D.2).

Original: Moreover, Z O RD is the only ZO optimization algorithm that is able to converge to a comparable final performance to that of the GD with true derivatives in every figure of Fig. 3.
Revised: Moreover, Z O RD is the only ZO optimization algorithm that is able to converge to a comparable final performance to that of the GD with true derivatives in every figure of Fig. 3.

Original: 5. B LACK -B OX A DVERSARIAL A TTACK
Revised: 5. B LACK -B OX A DVERSARIAL A TTACK

Original: We further compare our Z O RD with other ZO optimization algorithms in the problem of black-box adversarial attack on images, which is one of the most important applications of ZO optimization in recent years.
Revised: We further compare our Z O RD with other ZO optimization algorithms in the problem of black-box adversarial attack on images, which is one of the most important applications of ZO optimization in recent years.

Original: In black-box adversarial attack (Ru et al., 2020), given a fully trained ML model and an image z , we intend to find (through only function queries) a small perturbation x to be added to z
Revised: In black-box adversarial attack (Ru et al., 2020), given a fully trained ML model and an image z , we intend to find (through only function queries) a small perturbation x to be added to z

Original: [Figure]
Revised: [Figure]

Original: such that the perturbed image z + x will be incorrectly classified by the ML model.
Revised: such that the perturbed image z + x will be incorrectly classified by the ML model.

Original: Following the practice from (Cheng et al., 2021), we randomly select an image from MNIST (Lecun et al., 1998) ( d = 28 × 28 ) or CIFAR-10 (Krizhevsky et al., 2009) ( d = 32 × 32 ), and aim to add a perturbation with an L ∞ constraint to make a trained deep neural network misclassify the image (more details in Appx.
Revised: Following the practice from (Cheng et al., 2021), we randomly select an image from MNIST (Lecun et al., 1998) ( d = 28 × 28 ) or CIFAR-10 (Krizhevsky et al., 2009) ( d = 32 × 32 ), and aim to add a perturbation with an L ∞ constraint to make a trained deep neural network misclassify the image (more details in Appx.

Original: C.2).
Revised: C.2).

Original: Tab. 1 summarizes the number of required queries to achieve a successful attack by different algorithms (see results on multiple images in Appx.
Revised: Tab. 1 summarizes the number of required queries to achieve a successful attack by different algorithms (see results on multiple images in Appx.

Original: D.3).
Revised: D.3).

Original: The results show that in such high-dimensional ZO optimization problems, our Z O RD again significantly outperforms the other algorithms since it requires a considerably smaller number of queries to achieve a successful attack.
Revised: The results show that in such high-dimensional ZO optimization problems, our Z O RD again significantly outperforms the other algorithms since it requires a considerably smaller number of queries to achieve a successful attack.

Original: Particularly, our Z O RD is substantially more query-efficient than RGF and PRGF which rely on the
Revised: Particularly, our Z O RD is substantially more query-efficient than RGF and PRGF which rely on the

Original: FD methods for derivative estimation, e.g., for CIFAR-10, the number of queries required by RGF and PRGF are 9 .
Revised: FD methods for derivative estimation, e.g., for CIFAR-10, the number of queries required by RGF and PRGF are 9 .

Original: 4 × and 10 .
Revised: 4 × and 10 .

Original: 8 × of that required by Z O RD.
Revised: 8 × of that required by Z O RD.

Original: This further verifies the advantages of our trajectory-informed derivative estimation (as justified theoretically in Sec.
Revised: This further verifies the advantages of our trajectory-informed derivative estimation (as justified theoretically in Sec.

Original: 4.1 and empirically in Sec. 5.1) and dynamic virtual updates (as demonstrated in Appx.
Revised: 4.1 and empirically in Sec. 5.1) and dynamic virtual updates (as demonstrated in Appx.

Original: D.2).
Revised: D.2).

Original: Remarkably, our Z O RD also outperforms BO (i.e., TuRBO-1/10 which correspond to two versions of the TuRBO algorithm (Eriksson et al., 2019)) which has been widely shown to be query-efficient in black-box adversarial attack (Ru et al., 2020).
Revised: Remarkably, our Z O RD also outperforms BO (i.e., TuRBO-1/10 which correspond to two versions of the TuRBO algorithm (Eriksson et al., 2019)) which has been widely shown to be query-efficient in black-box adversarial attack (Ru et al., 2020).

Original: Overall, these results showcase the ability of our Z O RD to advance the other
Revised: Overall, these results showcase the ability of our Z O RD to advance the other

Original: ZO optimization algorithms in challenging real-world ZO optimization problems.
Revised: ZO optimization algorithms in challenging real-world ZO optimization problems.

Original: 5. N ON - DIFFERENTIABLE M ETRIC O PTIMIZATION
Revised: 5. N ON - DIFFERENTIABLE M ETRIC O PTIMIZATION

Original: Non-differentiable metric optimization (Hiranandani et al., 2021; Huang et al., 2021), which has received a surging interest recently, can also be cast as a ZO optimization problem.
Revised: Non-differentiable metric optimization (Hiranandani et al., 2021; Huang et al., 2021), which has received a surging interest recently, can also be cast as a ZO optimization problem.

Original: We therefore use it to further demonstrate the superiority of our Z O RD to other ZO optimization algorithms.
Revised: We therefore use it to further demonstrate the superiority of our Z O RD to other ZO optimization algorithms.

Original: Specifically, we firstly train a multilayer perceptron (MLP) ( d = 2189 ) on the Covertype (Dua and Graff, 2017) dataset with the cross-entropy loss function.
Revised: Specifically, we firstly train a multilayer perceptron (MLP) ( d = 2189 ) on the Covertype (Dua and Graff, 2017) dataset with the cross-entropy loss function.

Original: Then, we use the same dataset to fine-tune this MLP model by exploiting ZO optimization algorithms to optimize a non-differentiable metric, such as precision, recall, F1 score and Jaccard index (see more details in Appx.
Revised: Then, we use the same dataset to fine-tune this MLP model by exploiting ZO optimization algorithms to optimize a non-differentiable metric, such as precision, recall, F1 score and Jaccard index (see more details in Appx.

Original: C.3).
Revised: C.3).

Original: Here we additionally compare with the evolutionary strategy (ES) which has been previously applied for non-differentiable metric optimization (Huang et al., 2021).
Revised: Here we additionally compare with the evolutionary strategy (ES) which has been previously applied for non-differentiable metric optimization (Huang et al., 2021).

Original: Fig.
Revised: Fig.

Original: 4 illustrates the percentage improvements achieved by different algorithms during the fine-tuning process (i.e., ( f ( x 0 ) − f ( x T )) × 100% /f ( x 0 ) ).
Revised: 4 illustrates the percentage improvements achieved by different algorithms during the fine-tuning process (i.e., ( f ( x 0 ) − f ( x T )) × 100% /f ( x 0 ) ).

Original: The results show that our Z O RD again consistently outperforms the other ZO optimization algorithms in terms of both the query efficiency and the final converged performance.
Revised: The results show that our Z O RD again consistently outperforms the other ZO optimization algorithms in terms of both the query efficiency and the final converged performance.

Original: These results therefore further substantiate the superiority of Z O RD in optimizing high-dimensional non-differentiable functions.
Revised: These results therefore further substantiate the superiority of Z O RD in optimizing high-dimensional non-differentiable functions.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have introduced the Z O RD algorithm, which achieves query-efficient ZO optimization through two major contributions.
Revised: We have introduced the Z O RD algorithm, which achieves query-efficient ZO optimization through two major contributions.

Original: Firstly, we have proposed a novel derived GP-based method (6) which only uses the optimization trajectory and hence eliminates the requirement for additional queries (Sec. 3.1) to estimate derivatives.
Revised: Firstly, we have proposed a novel derived GP-based method (6) which only uses the optimization trajectory and hence eliminates the requirement for additional queries (Sec. 3.1) to estimate derivatives.

Original: Secondly, we have introduced a novel technique, i.e., dynamic virtual updates, which is made possible by our GP-based derivative estimation, to further improve the performance of our Z O RD (Sec. 3.2).
Revised: Secondly, we have introduced a novel technique, i.e., dynamic virtual updates, which is made possible by our GP-based derivative estimation, to further improve the performance of our Z O RD (Sec. 3.2).

Original: Through theoretical justifications (Sec. 4) and empirical demonstrations (Sec. 5), we show that our derived GP-based derivative estimation improve over existing FD methods and that our Z O RD outperforms various ZO optimization baselines.
Revised: Through theoretical justifications (Sec. 4) and empirical demonstrations (Sec. 5), we show that our derived GP-based derivative estimation improve over existing FD methods and that our Z O RD outperforms various ZO optimization baselines.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: For our theoretical results we have discussed all our assumptions in Sec. 2.1 & Sec. 4.2, and provided our complete proofs in Appx.
Revised: For our theoretical results we have discussed all our assumptions in Sec. 2.1 & Sec. 4.2, and provided our complete proofs in Appx.

Original: B. For our empirical results, we have provided our detailed experimental settings in Appx.
Revised: B. For our empirical results, we have provided our detailed experimental settings in Appx.

Original: C and included our codes in the supplementary materials (i e., the zip file)
Revised: C and included our codes in the supplementary materials (i e., the zip file)

Original: A PPENDIX A R ELATED W ORK
Revised: A PPENDIX A R ELATED W ORK

Original: Various types of algorithms have been proposed in the literature to solve ZO optimization problems, e.g., direct search, Bayesian optimization (BO) and GD-based algorithms with estimated derivatives.
Revised: Various types of algorithms have been proposed in the literature to solve ZO optimization problems, e.g., direct search, Bayesian optimization (BO) and GD-based algorithms with estimated derivatives.

Original: Particularly, direct search, e.g., (Stich et al., 2013; Golovin et al., 2020), relies on the comparison of function values at different inputs for the updates, which can be query-inefficient in practice owing to its indirect utilization of function values.
Revised: Particularly, direct search, e.g., (Stich et al., 2013; Golovin et al., 2020), relies on the comparison of function values at different inputs for the updates, which can be query-inefficient in practice owing to its indirect utilization of function values.

Original: In contrast, Bayesian optimization (BO) directly utilizes the function values to model the objective function using a Gaussian process (GP) and iteratively selects the inputs to query by trading off sampling potentially optimal inputs (i.e., exploitation) and inputs that can improve the GP belief of the objective function over the entire input domain (i.e., exploration) (Chowdhury and Gopalan, 2017; Srinivas et al., 2010; Dai et al., 2019; 2020).
Revised: In contrast, Bayesian optimization (BO) directly utilizes the function values to model the objective function using a Gaussian process (GP) and iteratively selects the inputs to query by trading off sampling potentially optimal inputs (i.e., exploitation) and inputs that can improve the GP belief of the objective function over the entire input domain (i.e., exploration) (Chowdhury and Gopalan, 2017; Srinivas et al., 2010; Dai et al., 2019; 2020).

Original: However, in ZO optimization problems with high-dimensional input spaces, BO algorithms typically suffer from query inefficiency and large computational complexity (Rasmussen and Williams, 2006; Letham et al., 2020; Eriksson et al., 2019), which significantly hinders their real-world applications.
Revised: However, in ZO optimization problems with high-dimensional input spaces, BO algorithms typically suffer from query inefficiency and large computational complexity (Rasmussen and Williams, 2006; Letham et al., 2020; Eriksson et al., 2019), which significantly hinders their real-world applications.

Original: Therefore,
Revised: Therefore,

Original: GD-based algorithms with estimated derivatives, which inherit the advantage of GD-based algorithms in optimizing functions with high-dimensional input spaces, have been more widely applied in practice.
Revised: GD-based algorithms with estimated derivatives, which inherit the advantage of GD-based algorithms in optimizing functions with high-dimensional input spaces, have been more widely applied in practice.

Original: For these algorithms, the derivatives are commonly estimated using the finite difference (FD) approximation (which requires additional function queries) of the directional derivatives along selected directions, in which the directions can be randomly sampled unit vectors Flaxman et al.
Revised: For these algorithms, the derivatives are commonly estimated using the finite difference (FD) approximation (which requires additional function queries) of the directional derivatives along selected directions, in which the directions can be randomly sampled unit vectors Flaxman et al.

Original: (2005), Gaussian vectors (Nesterov and Spokoiny, 2017), or standard bases (Lian et al., 2016)
Revised: (2005), Gaussian vectors (Nesterov and Spokoiny, 2017), or standard bases (Lian et al., 2016)

Original: (Sec. 2.2).
Revised: (Sec. 2.2).

Original: More recently, some works have incorporated a time-dependent prior (i.e., the estimated derivative in the previous iteration) into existing FD methods to improve the quality of its derivative estimation (Ilyas et al., 2019;
Revised: More recently, some works have incorporated a time-dependent prior (i.e., the estimated derivative in the previous iteration) into existing FD methods to improve the quality of its derivative estimation (Ilyas et al., 2019;

Original: Meier et al., 2019; Cheng et al., 2021).
Revised: Meier et al., 2019; Cheng et al., 2021).

Original: Nevertheless, such a prior is also estimated by the FD method (i.e., in the previous iteration) and can hence be biased owing to the its estimation error, which may even lead to larger derivative estimation errors in practice due to compounding errors.
Revised: Nevertheless, such a prior is also estimated by the FD method (i.e., in the previous iteration) and can hence be biased owing to the its estimation error, which may even lead to larger derivative estimation errors in practice due to compounding errors.

Original: Another line of work has taken the surrogate derivatives from other sources to help reduce the derivative estimation error of existing FD methods (Maheswaranathan et al., 2019; Cheng et al., 2019).
Revised: Another line of work has taken the surrogate derivatives from other sources to help reduce the derivative estimation error of existing FD methods (Maheswaranathan et al., 2019; Cheng et al., 2019).

Original: However, these surrogate derivatives may generally be unavailable in practice.
Revised: However, these surrogate derivatives may generally be unavailable in practice.

Original: Importantly, these existing FD methods require additional function queries for every derivation estimation during optimization, which will significantly increase the query complexity of
Revised: Importantly, these existing FD methods require additional function queries for every derivation estimation during optimization, which will significantly increase the query complexity of

Original: ZO optimization algorithms which employ these FD methods for derivative estimation.
Revised: ZO optimization algorithms which employ these FD methods for derivative estimation.

Original: A PPENDIX B P ROOFS B. P ROOF OF L EMMA
Revised: A PPENDIX B P ROOFS B. P ROOF OF L EMMA

Original: According to Rasmussen and Williams (2006), if a function f follows from a Gaussian process, its derivative also follows a Gaussian process determined by its mean E [ · ] and covariance Cov( · , · ) , i.e.,
Revised: According to Rasmussen and Williams (2006), if a function f follows from a Gaussian process, its derivative also follows a Gaussian process determined by its mean E [ · ] and covariance Cov( · , · ) , i.e.,

Original: [Equation]
Revised: [Equation]

Original: So, to prove Lemma 1, we only need to derive the mean and the covariance of the Gaussian process above for a function f that is sampled from another Gaussian process, i.e., f ∼ GP ( µ ( · ) , σ 2 ( · , · )) .
Revised: So, to prove Lemma 1, we only need to derive the mean and the covariance of the Gaussian process above for a function f that is sampled from another Gaussian process, i.e., f ∼ GP ( µ ( · ) , σ 2 ( · , · )) .

Original: Specifically, for the mean E [ ∇ f ] , we have
Revised: Specifically, for the mean E [ ∇ f ] , we have

Original: [Equation]
Revised: [Equation]

Original: 10) where the first equality derives from the interchangeability of the expectation and derivative operation based on the Leibniz integral rule.
Revised: 10) where the first equality derives from the interchangeability of the expectation and derivative operation based on the Leibniz integral rule.

Original: The second equality comes from the fact that E [ f ] = µ .
Revised: The second equality comes from the fact that E [ f ] = µ .

Original: For the covariance Cov( ∇ f, ∇ f ) , we have
Revised: For the covariance Cov( ∇ f, ∇ f ) , we have

Original: [Equation]
Revised: [Equation]

Original: Notably, ( b ) and ( d ) also derive from the interchangeability of the expectation and derivative operation based on the Leibniz integral rule.
Revised: Notably, ( b ) and ( d ) also derive from the interchangeability of the expectation and derivative operation based on the Leibniz integral rule.

Original: Besides, ( e ) is obtained based on Cov( f, f ) =
Revised: Besides, ( e ) is obtained based on Cov( f, f ) =

Original: σ 2 ( · , · ) .
Revised: σ 2 ( · , · ) .

Original: This finally completes our proof.
Revised: This finally completes our proof.

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: To begin with, we introduce the following concentration inequality for standard multi-variate Gaussian distribution:
Revised: To begin with, we introduce the following concentration inequality for standard multi-variate Gaussian distribution:

Original: Lemma B.1 (Laurent and Massart (2000)) .
Revised: Lemma B.1 (Laurent and Massart (2000)) .

Original: Let ζ ∼ N ( 0 , I m ) and δ ∈ (0 , 1) then
Revised: Let ζ ∼ N ( 0 , I m ) and δ ∈ (0 , 1) then

Original: [Equation]
Revised: [Equation]

Original: Define ζ ≜ (cid:0) ∂σ 2 t ( x ) (cid:1) − 1 ( ∇ f ( x ) − ∇ µ t ( x )) , according to Lemma 1, we then have that ζ follows a standard multi-variate Gaussian distribution, i.e.,
Revised: Define ζ ≜ (cid:0) ∂σ 2 t ( x ) (cid:1) − 1 ( ∇ f ( x ) − ∇ µ t ( x )) , according to Lemma 1, we then have that ζ follows a standard multi-variate Gaussian distribution, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Let δ ∈ (0 , 1) .
Revised: Let δ ∈ (0 , 1) .

Original: By substituting the result above into Lemma B.1, the following holds with probability of at least 1 − δ :
Revised: By substituting the result above into Lemma B.1, the following holds with probability of at least 1 − δ :

Original: [Equation]
Revised: [Equation]

Original: 14) with β ≜ (cid:113) d + 2( √ d + 1) ln(1 /δ )
Revised: 14) with β ≜ (cid:113) d + 2( √ d + 1) ln(1 /δ )

Original: and the first inequality is from the Cauchy-Schwarz inequality, which completes our proof.
Revised: and the first inequality is from the Cauchy-Schwarz inequality, which completes our proof.

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: We first introduce the following lemmas.
Revised: We first introduce the following lemmas.

Original: Lemma B.2 (Chowdhury and Gopalan (2021)) .
Revised: Lemma B.2 (Chowdhury and Gopalan (2021)) .

Original: For any σ ∈ R and any matrix A , the following hold
Revised: For any σ ∈ R and any matrix A , the following hold

Original: [Equation]
Revised: [Equation]

Original: Lemma B.3 (Sherman-Morrison formula) .
Revised: Lemma B.3 (Sherman-Morrison formula) .

Original: For any invertible square matrix A and column vectors u , v , suppose A + uv ⊤ is invertible, then the following holds
Revised: For any invertible square matrix A and column vectors u , v , suppose A + uv ⊤ is invertible, then the following holds

Original: [Equation]
Revised: [Equation]

Original: Preparation.
Revised: Preparation.

Original: We then introduce some additional notations and representations for our proof of
Revised: We then introduce some additional notations and representations for our proof of

Original: Theorem 2.
Revised: Theorem 2.

Original: Following the common practice in (Chowdhury and Gopalan, 2021), we let the kernel k be defined by ψ ( x ) , i.e., k ( x , x ′ )
Revised: Following the common practice in (Chowdhury and Gopalan, 2021), we let the kernel k be defined by ψ ( x ) , i.e., k ( x , x ′ )

Original: =
Revised: =

Original: ψ ( x ) ⊤ ψ ( x ′ ) , and ϕ ( x ) ≜ ∇ ψ ( x ) .
Revised: ψ ( x ) ⊤ ψ ( x ′ ) , and ϕ ( x ) ≜ ∇ ψ ( x ) .

Original: We then further define the ( t × d ) -dimensional
Revised: We then further define the ( t × d ) -dimensional

Original: Jacobian matrix ϕ t ( x ) ≜
Revised: Jacobian matrix ϕ t ( x ) ≜

Original: [ ϕ ( x ) ⊤ ψ ( x τ )]
Revised: [ ϕ ( x ) ⊤ ψ ( x τ )]

Original: t τ =1 and Ψ t ≜ [ ψ ( x τ )]
Revised: t τ =1 and Ψ t ≜ [ ψ ( x τ )]

Original: t τ =1 .
Revised: t τ =1 .

Original: The matrix
Revised: The matrix

Original: K t and the covariance matrix ∂σ 2 t ( x ) defined on the optimization trajectory D t in our Sec.
Revised: K t and the covariance matrix ∂σ 2 t ( x ) defined on the optimization trajectory D t in our Sec.

Original: 3.1 can be reformulated as
Revised: 3.1 can be reformulated as

Original: [Equation]
Revised: [Equation]

Original: Based on the reformulation above, define V t ≜ Ψ t Ψ ⊤ t + σ 2 I , we can further reformulate ∂σ 2 t ( x ) as below
Revised: Based on the reformulation above, define V t ≜ Ψ t Ψ ⊤ t + σ 2 I , we can further reformulate ∂σ 2 t ( x ) as below

Original: [Equation]
Revised: [Equation]

Original: Note that ( b ) is obtained by exploiting the fact that K t =
Revised: Note that ( b ) is obtained by exploiting the fact that K t =

Original: Ψ ⊤ t Ψ t and ϕ t ( x ) =
Revised: Ψ ⊤ t Ψ t and ϕ t ( x ) =

Original: ϕ ( x ) ⊤ Ψ t .
Revised: ϕ ( x ) ⊤ Ψ t .

Original: In addition, ( d ) comes from Lemma B.2 by replacing the matrix A in Lemma B.2 with the matrix Ψ ⊤ t .
Revised: In addition, ( d ) comes from Lemma B.2 by replacing the matrix A in Lemma B.2 with the matrix Ψ ⊤ t .

Original: First Part.
Revised: First Part.

Original: We then prove the first half part of our Theorem 2, i.e., the following Lemma B.4.
Revised: We then prove the first half part of our Theorem 2, i.e., the following Lemma B.4.

Original: Lemma B.4 (Non-Increasing Variance Norm) .
Revised: Lemma B.4 (Non-Increasing Variance Norm) .

Original: For any x ∈ X and any t ≥ 1 , we have that
Revised: For any x ∈ X and any t ≥ 1 , we have that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Based on our additional notations and representations, we have
Revised: Based on our additional notations and representations, we have

Original: [Equation]
Revised: [Equation]

Original: Note that ( a ) follows from the aforementioned definition of V t and ( b ) comes from the fact that Ψ t Ψ ⊤ t =
Revised: Note that ( a ) follows from the aforementioned definition of V t and ( b ) comes from the fact that Ψ t Ψ ⊤ t =

Original: Ψ t − 1 Ψ ⊤ t − 1 + ψ ( x t ) ψ ( x t ) ⊤ .
Revised: Ψ t − 1 Ψ ⊤ t − 1 + ψ ( x t ) ψ ( x t ) ⊤ .

Original: Similarly, ( c ) uses the definition of V t − 1 .
Revised: Similarly, ( c ) uses the definition of V t − 1 .

Original: In addition, equality ( d ) derives from Lemma B.3 by letting A
Revised: In addition, equality ( d ) derives from Lemma B.3 by letting A

Original: = V t − 1 and u = v = ψ ( x t ) and ( e ) follows from the reformulation of ∂σ 2 t − 1 ( x ) in (18).
Revised: = V t − 1 and u = v = ψ ( x t ) and ( e ) follows from the reformulation of ∂σ 2 t − 1 ( x ) in (18).

Original: Finally, ( f ) derives from the positive semi-definite property of ϕ ( x ) ⊤ V − 1 t − 1 ψ ( x t ) ψ ( x t )
Revised: Finally, ( f ) derives from the positive semi-definite property of ϕ ( x ) ⊤ V − 1 t − 1 ψ ( x t ) ψ ( x t )

Original: ⊤ V − 1 t − 1 ϕ ( x ) as well as the fact that 1+ ψ ( x t )
Revised: ⊤ V − 1 t − 1 ϕ ( x ) as well as the fact that 1+ ψ ( x t )

Original: ⊤ V − 1 t − 1 ψ ( x t ) > 0 .
Revised: ⊤ V − 1 t − 1 ψ ( x t ) > 0 .

Original: That is, for any column vector
Revised: That is, for any column vector

Original: z we have that
Revised: z we have that

Original: [Equation]
Revised: [Equation]

Original: So, ϕ ( x ) ⊤ V − 1 t − 1 ψ ( x t ) ψ ( x t ) ⊤ V − 1 t − 1 ϕ ( x ) is positive semi-definite.
Revised: So, ϕ ( x ) ⊤ V − 1 t − 1 ψ ( x t ) ψ ( x t ) ⊤ V − 1 t − 1 ϕ ( x ) is positive semi-definite.

Original: Following a similar way, we are also able to verify that 1 + ψ ( x t )
Revised: Following a similar way, we are also able to verify that 1 + ψ ( x t )

Original: ⊤ V − 1 t − 1 ψ ( x t ) > 0 by showing that ψ ( x t ) ⊤ V − 1 t − 1 ψ ( x t ) ≥ 0 using the decomposition of V − 1 t − 1 from the Principle Component Analysis (PCA).
Revised: ⊤ V − 1 t − 1 ψ ( x t ) > 0 by showing that ψ ( x t ) ⊤ V − 1 t − 1 ψ ( x t ) ≥ 0 using the decomposition of V − 1 t − 1 from the Principle Component Analysis (PCA).

Original: Since ∂σ 2 t ( x ) ≼ σ 2 t − 1 ( x ) is equivalent to (cid:13)(cid:13)
Revised: Since ∂σ 2 t ( x ) ≼ σ 2 t − 1 ( x ) is equivalent to (cid:13)(cid:13)

Original: ∂σ 2 t ( x ) (cid:13)(cid:13) 2 ≤ (cid:13)(cid:13)
Revised: ∂σ 2 t ( x ) (cid:13)(cid:13) 2 ≤ (cid:13)(cid:13)

Original: ∂σ 2 t − 1 ( x ) (cid:13)(cid:13) 2 , we then complete the proof of first half part of our
Revised: ∂σ 2 t − 1 ( x ) (cid:13)(cid:13) 2 , we then complete the proof of first half part of our

Original: Theorem 2.
Revised: Theorem 2.

Original: Second Part.
Revised: Second Part.

Original: To prove the rest of our Theorem 2, we firstly introduce the following lemmas.
Revised: To prove the rest of our Theorem 2, we firstly introduce the following lemmas.

Original: Lemma B.5.
Revised: Lemma B.5.

Original: For any x ∈ X and any t ≥ 1 , the following holds
Revised: For any x ∈ X and any t ≥ 1 , the following holds

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For any column vector z , we have
Revised: For any column vector z , we have

Original: [Equation]
Revised: [Equation]

Original: The first equality comes from the intermediate result in (20).
Revised: The first equality comes from the intermediate result in (20).

Original: So, V t − V t − 1 is positive semi-definite,
Revised: So, V t − V t − 1 is positive semi-definite,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e., V t − 1 ≼ V t .
Revised: e., V t − 1 ≼ V t .

Original: This can also indicate that V − 1 t ≼ V − 1 t − 1 , which thus completes our proof.
Revised: This can also indicate that V − 1 t ≼ V − 1 t − 1 , which thus completes our proof.

Original: Lemma B.6 (Lower Bound of Variance Norm) .
Revised: Lemma B.6 (Lower Bound of Variance Norm) .

Original: For any x ∈ X and any t ≥ 1 , the following holds
Revised: For any x ∈ X and any t ≥ 1 , the following holds

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We firstly show that
Revised: We firstly show that

Original: [Equation]
Revised: [Equation]

Original: Note that ( a ) derives from the Cauchy-Schwarz inequality.
Revised: Note that ( a ) derives from the Cauchy-Schwarz inequality.

Original: As for ( b ) and ( c ) , they have exploited the fact
Revised: As for ( b ) and ( c ) , they have exploited the fact

Original: that (cid:16) V − 1 / 2 t
Revised: that (cid:16) V − 1 / 2 t

Original: [Equation]
Revised: [Equation]

Original: and ψ ( x )
Revised: and ψ ( x )

Original: [Equation]
Revised: [Equation]

Original: V
Revised: V

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: t is a row vector.
Revised: t is a row vector.

Original: In addition, ( e ) follows from Lemma B.5.
Revised: In addition, ( e ) follows from Lemma B.5.

Original: Finally, ( g ) results from V − 1 

Revised: Finally, ( g ) results from V − 1

Original: = I /σ
Revised: = I /σ

Original: and ( h ) derives from the assumption that k ( x , x ) ≤ 1 ( ∀ x ∈ X ) in Sec. 2.1.
Revised: and ( h ) derives from the assumption that k ( x , x ) ≤ 1 ( ∀ x ∈ X ) in Sec. 2.1.

Original: Alternatively, we can restate the result above as
Revised: Alternatively, we can restate the result above as

Original: [Equation]
Revised: [Equation]

Original: We then complete our proof on the first inequality in Lemma B.6 using the following inequality:
Revised: We then complete our proof on the first inequality in Lemma B.6 using the following inequality:

Original: [Equation]
Revised: [Equation]

Original: where ( a ) derives from (20) and ( c ) comes from the inversion of matrix product.
Revised: where ( a ) derives from (20) and ( c ) comes from the inversion of matrix product.

Original: Finally ( d ) follows from the result in (26) and ( e ) exploits the reformulation of ∂σ 2 t − 1 ( x ) .
Revised: Finally ( d ) follows from the result in (26) and ( e ) exploits the reformulation of ∂σ 2 t − 1 ( x ) .

Original: According to Lemma B.4 and Lemma B.6, the following holds for any x ∈ X and any t ≥ 1 ,
Revised: According to Lemma B.4 and Lemma B.6, the following holds for any x ∈ X and any t ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Based on the definition of r in our Theorem 2, we therefore also have
Revised: Based on the definition of r in our Theorem 2, we therefore also have

Original: [Equation]
Revised: [Equation]

Original: As a result, for every iteration t of our Algo. 2, we have
Revised: As a result, for every iteration t of our Algo. 2, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality derives from our assumption of ∥ ∂ z ∂
Revised: where the last inequality derives from our assumption of ∥ ∂ z ∂

Original: z ′ k
Revised: z ′ k

Original: ( z , z ′ )
Revised: ( z , z ′ )

Original: | z = z ′ =
Revised: | z = z ′ =

Original: x ∥ 2 ≤ κ ( ∀ x ∈ X ) in our Sec. 2.1.
Revised: x ∥ 2 ≤ κ ( ∀ x ∈ X ) in our Sec. 2.1.

Original: By subsisting the result above into our Theorem 1, we complete our proof of Theorem 2.
Revised: By substituting the result above into our Theorem 1, we complete our proof of Theorem 2.

Original: B
Revised: B. P ROOF OF T HEOREM

Original: P ROOF OF T HEOREM

Original: Preparation.
Revised: Preparation.

Original: Following the definition of the derivative mapping on the true derivative ∇ f ( x t,τ ) in (8), we defined the following derivative mapping on our estimated derivative ∇ µ t − 1 ( x t,τ ) :
Revised: Following the definition of the derivative mapping on the true derivative ∇ f ( x t,τ ) in (8), we defined the following derivative mapping on our estimated derivative ∇ µ t − 1 ( x t,τ ) :

Original: [Equation]
Revised: [Equation]

Original: By re-arranging it, we have the following update rule that has reformulated (7):
Revised: By re-arranging it, we have the following update rule that has reformulated (7):

Original: [Equation]
Revised: [Equation]

Original: Based on our definition of the derivative mappings in (31) and (8), we introduce the following lemmas:
Revised: Based on our definition of the derivative mappings in (31) and (8), we introduce the following lemmas:

Original: Lemma B.7 (General Projection Inequalities) .
Revised: Lemma B.7 (General Projection Inequalities) .

Original: Given P X ( x ) = arg min
Revised: Given P X ( x ) = arg min

Original: z ∈X ∥ x − z ∥ 22 / 2 and domain X , for any x , x ′ , we have
Revised: z ∈X ∥ x − z ∥ 22 / 2 and domain X , for any x , x ′ , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For (33), as P X ( x ′ ) ∈
Revised: For (33), as P X ( x ′ ) ∈

Original: X ( ∀ x ′ ) and P X ( x ) =
Revised: X ( ∀ x ′ ) and P X ( x ) =

Original: arg min
Revised: arg min

Original: z ∈X ∥ x − z ∥ 22 / 2 , we then naturally have (33).
Revised: z ∈X ∥ x − z ∥ 22 / 2 , we then naturally have (33).

Original: For (34), since P X ( x ) is the optimum of h ( z ) =
Revised: For (34), since P X ( x ) is the optimum of h ( z ) =

Original: ∥ x − z ∥ 22 / 2 , according to the optimality condition of the convex projection function h ( z ) within the domain z ∈ X (Boyd and Vandenberghe, 2014), we then have the following inequality for any P X ( x ′ ) ∈
Revised: ∥ x − z ∥ 22 / 2 , according to the optimality condition of the convex projection function h ( z ) within the domain z ∈ X (Boyd and Vandenberghe, 2014), we then have the following inequality for any P X ( x ′ ) ∈

Original: X :
Revised: X :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By taking ∇ h ( z ) =
Revised: By taking ∇ h ( z ) =

Original: z − x with z = P X ( x ) into the inequality above, we have
Revised: z − x with z = P X ( x ) into the inequality above, we have

Original: [Equation]
Revised: [Equation]

Original: By exchanging x and x ′ in the result above, we achieve the following similar result:
Revised: By exchanging x and x ′ in the result above, we achieve the following similar result:

Original: [Equation]
Revised: [Equation]

Original: By summing (36) and (37),
Revised: By summing (36) and (37),

Original: [Equation]
Revised: [Equation]

Original: Based on the Cauchy-Schwarz inequality, we finally achieve (34) using
Revised: Based on the Cauchy-Schwarz inequality, we finally achieve (34) using

Original: [Equation]
Revised: [Equation]

Original: where both sides need to be divided by ∥P X ( x ) − P X ( x ′ )
Revised: where both sides need to be divided by ∥P X ( x ) − P X ( x ′ )

Original: ∥ 2 to complete our proof.
Revised: ∥ 2 to complete our proof.

Original: Lemma B.8 (Inequalities for Derivative Mappings) .
Revised: Lemma B.8 (Inequalities for Derivative Mappings) .

Original: Given (31) and (8) , for every t and τ , we have
Revised: Given (31) and (8) , for every t and τ , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For (40), let (cid:98) x t,τ = x t,τ − η t,τ ∇ µ t − 1 ( x t,τ ) , we then have
Revised: For (40), let (cid:98) x t,τ = x t,τ − η t,τ ∇ µ t − 1 ( x t,τ ) , we then have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ( a ) results from the fact that x t,
Revised: where ( a ) results from the fact that x t,

Original: τ +

Revised: τ +

Original: = P X ( x t,τ − η t,τ ∇ µ t − 1 ( x t,τ ))
Revised: = P X ( x t,τ − η t,τ ∇ µ t − 1 ( x t,τ ))

Original: based on our (7) and ( b ) derives from the definition of (cid:98) G t,τ in (31).
Revised: based on our (7) and ( b ) derives from the definition of (cid:98) G t,τ in (31).

Original: In addition, ( c ) is based on the following result by substituting x = x t,τ and x ′ =
Revised: In addition, ( c ) is based on the following result by substituting x = x t,τ and x ′ =

Original: (cid:98) x t,τ into (38):
Revised: (cid:98) x t,τ into (38):

Original: [Equation]
Revised: [Equation]

Original: Finally, by dividing η 2 t,τ on the both sides of the last inequality in (43), we finish the proof for (40).
Revised: Finally, by dividing η 2 t,τ on the both sides of the last inequality in (43), we finish the proof for (40).

Original: For (41), following the same proof above, we can also obtain achieve the following inequality for the projected derivative G t,τ :
Revised: For (41), following the same proof above, we can also obtain achieve the following inequality for the projected derivative G t,τ :

Original: [Equation]
Revised: [Equation]

Original: We complete the proof for (41) by dividing ∥ G t,τ ∥ 2 on the both sides of the inequality above.
Revised: We complete the proof for (41) by dividing ∥ G t,τ ∥ 2 on the both sides of the inequality above.

Original: For (42), we have
Revised: For (42), we have

Original: [Equation]
Revised: [Equation]

Original: where ( a ) comes from the definition of (cid:98) G t,τ and G t,τ in (31) and (8), respectively.
Revised: where ( a ) comes from the definition of (cid:98) G t,τ and G t,τ in (31) and (8), respectively.

Original: In addition, ( c ) derives from (7) and (8).
Revised: In addition, ( c ) derives from (7) and (8).

Original: Finally, ( d ) results from (34).
Revised: Finally, ( d ) results from (34).

Original: Proof.
Revised: Proof.

Original: Since the objective function f is assumed to be L s -Lipschitz smooth (Sec. 4.2), we have the following inequality for any x t,τ ∈ X in our Z O RD algorithm:
Revised: Since the objective function f is assumed to be L s -Lipschitz smooth (Sec. 4.2), we have the following inequality for any x t,τ ∈ X in our Z O RD algorithm:

Original: [Equation]
Revised: [Equation]

Original: Let δ ′ ∈
Revised: Let δ ′ ∈

Original: Define β ≜ (cid:113) d + 2( √ d + 1) ln(1 /δ
Revised: Define β ≜ (cid:113) d + 2( √ d + 1) ln(1 /δ

Original: ′ ) , by substituting (32) into the inequality above, the following inequality holds with probability of at least 1 − δ ′ :
Revised: ′ ) , by substituting (32) into the inequality above, the following inequality holds with probability of at least 1 − δ ′ :

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: (48) where ( d ) derives from the Cauchy-Schwarz inequality and ( e ) follows from the Lemma B.7.
Revised: (48) where ( d ) derives from the Cauchy-Schwarz inequality and ( e ) follows from the Lemma B.7.

Original: Finally, ( f ) result from the bounded derivative estimation error in Theorem 2 and the fact that f is L c -Lipschitz continuous (i
Revised: Finally, ( f ) result from the bounded derivative estimation error in Theorem 2 and the fact that f is L c -Lipschitz continuous (i

Original: [Table]
Revised: [Table]

Original: e., ∥∇ f ( x ) ∥ 2 ≤ L c for any x ∈ X ) and η t,τ ≤ 1 /L s ( ∀ τ ) .
Revised: e., ∥∇ f ( x ) ∥ 2 ≤ L c for any x ∈ X ) and η t,τ ≤ 1 /L s ( ∀ τ ) .

Original: For every iteration t our Z O RD algorithm
Revised: For every iteration t our Z O RD algorithm

Original: [Table]
Revised: [Table]

Original: we in fact will apply the virtual updates (7) for V t times (see Algo. 2).
Revised: we in fact will apply the virtual updates (7) for V t times (see Algo. 2).

Original: Therefore, for probability ≥ 1 − V t δ ′ , we have
Revised: Therefore, for probability ≥ 1 − V t δ ′ , we have

Original: [Equation]
Revised: [Equation]

Original: where the first inequality results from (48) by re-arranging it and then sum it up over τ .
Revised: where the first inequality results from (48) by re-arranging it and then sum it up over τ .

Original: However
Revised: However

Original: [Equation]
Revised: [Equation]

Original: in order to prove the convergence of our Z O RD algorithm to a stationary point
Revised: in order to prove the convergence of our Z O RD algorithm to a stationary point

Original: [Equation]
Revised: [Equation]

Original: we need to consider the derivative mapping of G t,τ instead (refer to our Sec. 4.2).
Revised: we need to consider the derivative mapping of G t,τ instead (refer to our Sec. 4.2).

Original: So, for any τ , we propose the following inequality:
Revised: So, for any τ , we propose the following inequality:

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is from the Cauchy-Schwarz inequality and the second inequality comes from (42).
Revised: where the first inequality is from the Cauchy-Schwarz inequality and the second inequality comes from (42).

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: by taking the result above into (49)
Revised: by taking the result above into (49)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Then, substituting V t = V and η t,τ = η for any t, τ into the result above, the following inequality holds with probability of at least 1 − V Tδ ′ when r < 1 :
Revised: Then, substituting V t = V and η t,τ = η for any t, τ into the result above, the following inequality holds with probability of at least 1 − V Tδ ′ when r < 1 :

Original: [Equation]
Revised: [Equation]

Original: Note that ( b ) derives from the summation of the geometric sequence about r .
Revised: Note that ( b ) derives from the summation of the geometric sequence about r .

Original: When r = 1 , the following holds with probability of at least ≥ 1 − V Tδ ′ accordingly:
Revised: When r = 1 , the following holds with probability of at least ≥ 1 − V Tδ ′ accordingly:

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: let δ = V Tδ ′ ∈
Revised: let δ = V Tδ ′ ∈

Original: [Equation]
Revised: [Equation]

Original: the following holds with probability of at least 1 − δ ,
Revised: the following holds with probability of at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: where 1 and 2 can be defined as below with α ≜ κ (cid:113) d + 2( √ d + 1)
Revised: where 1 and 2 can be defined as below with α ≜ κ (cid:113) d + 2( √ d + 1)

Original: ln
Revised: ln

Original: ( V T/δ ) :
Revised: ( V T/δ ) :

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: A PPENDIX C E XPERIMENTAL S ETTINGS
Revised: A PPENDIX C E XPERIMENTAL S ETTINGS

Original: Among all our experiments in Sec. 5, to apply the derivative estimation in Sec.
Revised: Among all our experiments in Sec. 5, to apply the derivative estimation in Sec.

Original: 3.1 for every iteration t and every step τ of our Z O RD algorithm, we use the derived GP (4) based on the Mat´ern kernel with ν = 2 . 5 and fit this derived GP using 150 queries that achieves the smallest Euclidean distance with input x t,τ from the optimization trajectory.
Revised: 3.1 for every iteration t and every step τ of our Z O RD algorithm, we use the derived GP (4) based on the Mat´ern kernel with ν = 2 . 5 and fit this derived GP using 150 queries that achieves the smallest Euclidean distance with input x t,τ from the optimization trajectory.

Original: This is because we only need to model the objective function f in the vicinity of input x t,τ precisely rather than the entire domain, so as to achieve an accurate derivative estimation at input x t,τ .
Revised: This is because we only need to model the objective function f in the vicinity of input x t,τ precisely rather than the entire domain, so as to achieve an accurate derivative estimation at input x t,τ .

Original: As for the confidence threshold c of our dynamic virtual updates (Sec. 3.2), we use c = 0 .
Revised: As for the confidence threshold c of our dynamic virtual updates (Sec. 3.2), we use c = 0 .

Original: 35 to realize a good trade-off between query efficiency and accurate derivative estimation in practice.
Revised: 35 to realize a good trade-off between query efficiency and accurate derivative estimation in practice.

Original: In addition, among all our experiments in Sec. 5, we consistently use n = 10 , λ = 0 . 01 and directions { u i } ni =1 that are randomly sampled from a unit sphere for the derivative estimation of the FD method (2) applied in the RGF and PRGF algorithm.
Revised: In addition, among all our experiments in Sec. 5, we consistently use n = 10 , λ = 0 . 01 and directions { u i } ni =1 that are randomly sampled from a unit sphere for the derivative estimation of the FD method (2) applied in the RGF and PRGF algorithm.

Original: Moreover, following the common practice of (Berahas et al., 2022; Cheng et al., 2021), we conduct orthogonalization on these randomly selected directions via the Gram-Schmidt procedure.
Revised: Moreover, following the common practice of (Berahas et al., 2022; Cheng et al., 2021), we conduct orthogonalization on these randomly selected directions via the Gram-Schmidt procedure.

Original: As for the ES algorithm (e.g., the one applied in (Salimans et al., 2017)), we apply the same n , λ and { u i } ni =1 in RGF and PRGF for their update in every iteration.
Revised: As for the ES algorithm (e.g., the one applied in (Salimans et al., 2017)), we apply the same n , λ and { u i } ni =1 in RGF and PRGF for their update in every iteration.

Original: Following the practice used in (Eriksson et al., 2019), for all our experiments, we firstly re-scale the input domains into [0 , 10] d to ease the optimization and then re-scale the updated inputs back to the original domains for querying.
Revised: Following the practice used in (Eriksson et al., 2019), for all our experiments, we firstly re-scale the input domains into [0 , 10] d to ease the optimization and then re-scale the updated inputs back to the original domains for querying.

Original: C. S YNTHETIC E XPERIMENTS
Revised: C. S YNTHETIC E XPERIMENTS

Original: Let input x =
Revised: Let input x =

Original: [ x i ] di =1 , the Ackley and Levy function applied in our synthetic experiments are given below,
Revised: [ x i ] di =1 , the Ackley and Levy function applied in our synthetic experiments are given below,

Original: [Equation]
Revised: [Equation]

Original: where w i =1 + ( x  i  − 1) / 4 for any i = 1 , · · · , d and both functions achieve their minimum (i.e., min f ( x ) = 0 ) at x ∗ =0 .
Revised: d (cid:88) i = x 2 i   − exp(1 d d (cid:88) i = cos (2 πx i )) + 20 + exp(1) , ( Ackley ) f ( x ) = sin 2 ( πw 1 ) + d − 1 (cid:88) i =

Revised: ( w i − 1) 2 (cid:2) 1 + 10 sin 2 ( πw i + 1) (cid:3) + ( w d − 1) 2 (cid:2) 1 + sin 2 (2 πw d ) (cid:3) ( Levy ) (56) where w i =

Revised: + ( x i − 1) / 4 for any i = 1 , · · · , d , Ackley function achieves its minimum (i.e., min f ( x ) = 0 ) at x ∗ = 0 , and Levy function achieves its minimum (i.e., min f ( x ) = 0 ) at x ∗

Revised: = 1 .

Original: Note that the Ackley and Levy function for the synthetic experiments in our Sec.
Revised: Note that the Ackley and Levy function for the synthetic experiments in our Sec.

Original: 5.2 are defined within the domain [ − 20 , 20] d and [ − 7 . 5 , 7 . 5] d , respectively.
Revised: 5.2 are defined within the domain [ − 20 , 20] d and [ − 7 . 5 , 7 . 5] d , respectively.

Original: To give a better understanding of these two synthetic functions, we provide a 3D illustration of these two synthetic functions with d = 2 in our Fig. 5.
Revised: To give a better understanding of these two synthetic functions, we provide a 3D illustration of these two synthetic functions with d = 2 in our Fig. 5.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, these two synthetic functions are highly nonconvex and therefore have local minimums within their domains.
Revised: 5, these two synthetic functions are highly nonconvex and therefore have local minimums within their domains.

Original: [Table]

Original: To compare our Z O RD algorithm with other ZO/FO optimization baselines in Sec.
Revised: To compare our Z O RD algorithm with other ZO/FO optimization baselines in Sec.

Original: 5.2, we firstly employ TuRBO of 300 queries to find a good initialization for all other ZO/FO optimization algorithms in Fig.
Revised: 5.2, we firstly employ TuRBO of 300 queries to find a good initialization for all other ZO/FO optimization algorithms in Fig.

Original: 3 because of the nonconvexity of these two synthetic functions as shown in Fig.
Revised: 3 because of the nonconvexity of these two synthetic functions as shown in Fig.

Original: We then apply these ZO/FO optimization algorithms with a query budget of 200 for d = 20 , 40 , and a query budget of 400 for d = 100 to compare their query efficiency.
Revised: We then apply these ZO/FO optimization algorithms with a query budget of 200 for d = 20 , 40 , and a query budget of 400 for d = 100 to compare their query efficiency.

Original: We use the same Adam optimizer (Kingma and Ba, 2015) with a learning rate of 0.1 and exponential decay rates of 0.9, 0.
Revised: We use the same Adam optimizer (Kingma and Ba, 2015) with a learning rate of 0.1 and exponential decay rates of 0.9, 0.

Original: for RGF, PRGF, GD, and our Z O RD algorithm, for faster convergence compared with standard GD.
Revised: for RGF, PRGF, GD, and our Z O RD algorithm, for faster convergence compared with standard GD.

Original: C. B LACK -B OX A DVERSARIAL A TTACK
Revised: C. B LACK -B OX A DVERSARIAL A TTACK

Original: For the black-box adversarial attack experiment on the MNIST dataset, we use the same fully trained deep neural networks from (Cheng et al., 2021) and adopt a L ∞ constraint of ∥ x ∥ ∞ ≤ 0 .
Revised: For the black-box adversarial attack experiment on the MNIST dataset, we use the same fully trained deep neural networks from (Cheng et al., 2021) and adopt a L ∞ constraint of ∥ x ∥ ∞ ≤ 0 .

Original: 3 on the input perturbation x .
Revised: 3 on the input perturbation x .

Original: For the black-box adversarial attack experiment on the CIFAR-10 dataset, we fully train a ResNet-

Revised: For the black-box adversarial attack experiment on the CIFAR-10 dataset, we fully train a ResNet-

Original: (He et al., 2016) on CIFAR-10 using stochastic gradient descend (SGD) with a cosine annealed learning rate from 0.1 to 0, a momentum of 0.9 and a weight decay of 5 × 10 − 4 for 200 epochs, and adopt a L ∞ constraint of ∥ x ∥ ∞ ≤ 0 .
Revised: (He et al., 2016) on CIFAR-10 using stochastic gradient descend (SGD) with a cosine annealed learning rate from 0.1 to 0, a momentum of 0.9 and a weight decay of 5 × 10 − 4 for 200 epochs, and adopt a L ∞ constraint of ∥ x ∥ ∞ ≤ 0 .

Original: 2 on the input perturbation x .
Revised: 2 on the input perturbation x .

Original: Note that we use the same loss function as (Cheng et al., 2021) for these two experiments.
Revised: Note that we use the same loss function as (Cheng et al., 2021) for these two experiments.

Original: Meanwhile, to apply RGF,
Revised: Meanwhile, to apply RGF,

Revised: [Figure]

Revised: [Table]

Original:  PRGF and our Z O RD, we adopt Adam optimizer with the same learning rate of 0.5 and the same exponential decay rates of 0.9, 0.999.
Revised: | S | [Table] | A |      d PRGF and our Z O RD, we adopt Adam optimizer with the same learning rate of 0.5 and the same exponential decay rates of 0.9, 0.999.

Original: C. N ON -D IFFERENTIABLE M ETRIC O PTIMIZATION
Revised: C. N ON -D IFFERENTIABLE M ETRIC O PTIMIZATION

Original: The Covertype dataset used in Sec.
Revised: The Covertype dataset used in Sec.

Original: 5.4 is a classification dataset consisting of 581,012 samples from 7 different categories.
Revised: 5.4 is a classification dataset consisting of 581,012 samples from 7 different categories.

Original: Each sample from this dataset is a 54-dimensional vector of integers.
Revised: Each sample from this dataset is a 54-dimensional vector of integers.

Original: In this experiment, we randomly split the dataset into training and test sets with each containing 290, samples.
Revised: In this experiment, we randomly split the dataset into training and test sets with each containing 290, samples.

Original: The MLP classifier applied in Sec.
Revised: The MLP classifier applied in Sec.

Original: 5.4 consists of 2 layers with 30 and 14 hidden neurons respectively, leading to 2189 parameters in total (i.e., d = 2189 ).
Revised: 5.4 consists of 2 layers with 30 and 14 hidden neurons respectively, leading to 2189 parameters in total (i.e., d = 2189 ).

Original: We first train this MLP classifier on the training dataset of Covertype using the L-BFGS algorithm with the cross-entropy loss function for 300 epochs, and then apply ZO optimization algorithms to fine-tune our trained MLP directly on the non-differentiable metrics (i.e., using these metrics as the new loss functions), including precision, recall, F1 score and Jaccard index.
Revised: We first train this MLP classifier on the training dataset of Covertype using the L-BFGS algorithm with the cross-entropy loss function for 300 epochs, and then apply ZO optimization algorithms to fine-tune our trained MLP directly on the non-differentiable metrics (i.e., using these metrics as the new loss functions), including precision, recall, F1 score and Jaccard index.

Original: To obtain the results of ES, RGF, PRGF and our Z O RD algorithm in Sec.
Revised: To obtain the results of ES, RGF, PRGF and our Z O RD algorithm in Sec.

Original: 5.4, we apply the same Adam optimizer with a learning rate of 0.2 (for precision and recall) or 0.01 (for F1 score and Jaccard index) and exponential decay rates of 0.9, 0.999.
Revised: 5.4, we apply the same Adam optimizer with a learning rate of 0.2 (for precision and recall) or 0.01 (for F1 score and Jaccard index) and exponential decay rates of 0.9, 0.999.

Original: Note that we do not compare with TuRBO in this experiment because of its high input dimension (i.e., d = 2189 )
Revised: Note that standard BO algorithms (including TuRBO) fail to achieve any percentage improvements (i.e., achieving 0% in the y -axis of Fig. 4) in this experiment according to our five independent runs, which is likely due to their aggressive exploration in the input domain of such a high dimension.

Original: that TuRBO fails to deal with (Eriksson et al., 2019).
Revised: In light of this, we do not include them in our comparison since all other methods are able to achieve certain improvements.

Original: C. D ERIVATIVE -F REE R EINFORCEMENT L EARNING
Revised: C. D ERIVATIVE -F REE R EINFORCEMENT L EARNING

Original: Our derivative-free RL experiments aim to learn controllers (which outputs policies) that maximize the rewards/return for several environments in the OpenAI Gym (Brockman et al., 2016) without using true derivatives.
Revised: Our derivative-free RL experiments aim to learn controllers (which outputs policies) that maximize the rewards/return for several environments in the OpenAI Gym (Brockman et al., 2016) without using true derivatives.

Original: Specifically, we need to optimize the parameters (i.e., x ) of our neural network (MLP) controller with 2 hidden layers, where each hidden layer has 10 hidden neurons and one bias term.
Revised: Specifically, we need to optimize the parameters (i.e., x ) of our neural network (MLP) controller with 2 hidden layers, where each hidden layer has 10 hidden neurons and one bias term.

Original: We adopt a L ∞ constraint of ∥ x ∥ ∞ ≤ 1 on the parameters x .
Revised: We adopt a L ∞ constraint of ∥ x ∥ ∞ ≤ 1 on the parameters x .

Original: We use a softmax output layer for the policies that deal with discrete action spaces, and a tanh output layer for the policies that deal with continuous action spaces.
Revised: We use a softmax output layer for the policies that deal with discrete action spaces, and a tanh output layer for the policies that deal with continuous action spaces.

Original: The dimension of neural network parameters (represented as a columnvector) d is determined by the dimensions of both the observation | S | and the action space | A | of an environment, as detailed in Tab. 2. In order to search for policies that are robust to different random state initializations, we use the vectorized API of OpenAI Gym, and our observed function value y ( x ) given the network parameters
Revised: The dimension of neural network parameters (represented as a column

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 100 # Total Queries 0.
Revised: 100 # Total Queries 0.

Original: C o s i ne
Revised: C o s i ne

Original: S i m il a r i t y
Revised: S i m il a r i t y

Original: SE
Revised: SE

Original: E u c li dean D i s t an c e E u c li dean D i s t an c e
Revised: E u c li dean D i s t an c e E u c li dean D i s t an c e

Original: E u c li dean D i s t an c e E u c li dean D i s t an c e
Revised: E u c li dean D i s t an c e E u c li dean D i s t an c e

Original: FD GP
Revised: FD GP

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: vector) d is determined by the dimensions of both the observation | S | and the action space | A | of an environment, as detailed in Tab. 2.

Original: [Figure] x is an averaged return of 32 parallel environments.
Revised: In order to search for policies that are robust to different random state initializations, we use the vectorized API of OpenAI Gym, and our observed function value y ( x ) given the network parameters x is an averaged return of 32 parallel environments.

Original: We also fix the seed of OpenAI Gym for all queries, which ensures that we are evaluating on a fixed set of 32 state initializations and that our results can be reproduced.
Revised: We also fix the seed of OpenAI Gym for all queries, which ensures that we are evaluating on a fixed set of 32 state initializations and that our results can be reproduced.

Original: We first initialize a sample of 500 points from a Latin Hypercube (McKay et al., 1979) to find a good initial input, and then proceed to apply ZO optimization algorithms (i.e.,
Revised: We first initialize a sample of 500 points from a Latin Hypercube (McKay et al., 1979) to find a good initial input, and then proceed to apply ZO optimization algorithms (i.e.,

Original: ES, RGF, PRGF, and our Z O RD) with the same query budget of 1000 on this initial input.
Revised: ES, RGF, PRGF, and our Z O RD) with the same query budget of 1000 on this initial input.

Original: For all these ZO optimization algorithms, we employ the same Adam optimizer with a learning rate of 1. and exponential decay rates of 0.9, 0.999.
Revised: For all these ZO optimization algorithms, we employ the same Adam optimizer with a learning rate of 1. and exponential decay rates of 0.9, 0.999.

Original: Considering the prohibitive noise in RL experiments, we use 300 queries from the optimization trajectory that has the smallest Euclidean distance with an input needing to be updated.
Revised: Considering the prohibitive noise in RL experiments, we use 300 queries from the optimization trajectory that has the smallest Euclidean distance with an input needing to be updated.

Original: Of note, we conduct 10 trials in total where each trial differs from each other by both the OpenAI Gym seed and the Latin Hypercube initializations.
Revised: Of note, we conduct 10 trials in total where each trial differs from each other by both the OpenAI Gym seed and the Latin Hypercube initializations.

Original: A PPENDIX D M ORE R ESULTS D. M ORE R ESULTS ON D ERIVATIVE E STIMATION
Revised: A PPENDIX D M ORE R ESULTS D. M ORE R ESULTS ON D ERIVATIVE E STIMATION

Original: Besides the comparison in Fig.
Revised: Besides the comparison in Fig.

Original: 2, we provide additional comparison between our derived GP-based estimator (6) and the FD estimator (2) under various input dimensions in Fig. 6(a) and various kernels in Fig.
Revised: 2, we provide additional comparison between our derived GP-based estimator (6) and the FD estimator (2) under various input dimensions in Fig. 6(a) and various kernels in Fig.

Original: 6(b) using the Ackley function.
Revised: 6(b) using the Ackley function.

Original: We adopt the same setting in Sec. 5.2.
Revised: We adopt the same setting in Sec. 5.2.

Original: Interestingly, Fig.
Revised: Interestingly, Fig.

Original: 6(a)(b) show that under various input dimensions and GP kernels, our derived GP-based estimator (6) is still able to achieve faster reduction rates compared with the FD estimator.
Revised: 6(a)(b) show that under various input dimensions and GP kernels, our derived GP-based estimator (6) is still able to achieve faster reduction rates compared with the FD estimator.

Original: Of note, all the function queries applied in our derived GP-based estimator is from the optimization trajectory whereas the FD estimator requires additional function queries for its derivative estimation.
Revised: Of note, all the function queries applied in our derived GP-based estimator is from the optimization trajectory whereas the FD estimator requires additional function queries for its derivative estimation.

Original: So, Fig.
Revised: So, Fig.

Original: 6(a)(b) also show that our derived GP method is still able to achieve improved query efficiency for accurate derivative estimation than FD method under various input dimensions and GP kernels because our method avoids the requirement of additional queries for derivative estimation.
Revised: 6(a)(b) also show that our derived GP method is still able to achieve improved query efficiency for accurate derivative estimation than FD method under various input dimensions and GP kernels because our method avoids the requirement of additional queries for derivative estimation.

Original: Interestingly, the objective function (i.e., the Ackley function) is not truly sampled from the GPs based on these kernels.
Revised: Interestingly, the objective function (i.e., the Ackley function) is not truly sampled from the GPs based on these kernels.

Original: This therefore means that though we have assumed that we need the prior knowledge about the GP in which the objective function is sampled from (Sec. 2.1), such an assumption does not really need to be satisfied for our derived GP-based method to achieve accurate derivative estimation in practice.
Revised: This therefore means that though we have assumed that we need the prior knowledge about the GP in which the objective function is sampled from (Sec. 2.1), such an assumption does not really need to be satisfied for our derived GP-based method to achieve accurate derivative estimation in practice.

Original: More interestingly, we notice that Mat´ern( ν = 0 . 5 ) and SE kernel will achieve slightly worse derivativeestimation, indicating that the choice of GP kernels may impact the quality of our derived GP-based derivative estimation.
Revised: More interestingly, we notice that Mat ´ ern( ν = 0 . 5 ) and SE kernel will achieve slightly worse derivative

Original: However, in practice, our derived GP method based on Mat´ern( ν = 2 . 5 ) kernel, which has been widely adopted in our experiments, is already able to provide us with good derivative estimation for ZO optimization as confirmed by the results in our other experiments.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: estimation the of GP kernels GP estimation. However, in practice, our derived GP method based on Mat ´ ern( ν =

Original: success rate (within the range of [0 , 1] ) achieved after this number of queries, respectively.
Revised: 5 ) kernel which has been widely adopted in our experiments is already able to provide us with good derivative estimation for ZO optimization as confirmed by the results in our experiments

Original: D. M ORE R ESULTS ON S YNTHETIC E XPERIMENTS
Revised: D. M ORE R ESULTS ON S YNTHETIC E XPERIMENTS

Original: Here, we empirically investigate the impacts of the dynamic virtual updates (Sec. 3.2) on our Z O RD algorithm.
Revised: Here, we empirically investigate the impacts of the dynamic virtual updates (Sec. 3.2) on our Z O RD algorithm.

Original: In particular, we apply the same setting in Appx.
Revised: In particular, we apply the same setting in Appx.

Original: C.1 to optimize the Ackley and Levy function with d = 40 under various confidence thresholds c for our dynamic virtual updates.
Revised: C.1 to optimize the Ackley and Levy function with d = 40 under various confidence thresholds c for our dynamic virtual updates.

Original: Fig.illustrates the results.
Revised: Fig.illustrates the results.

Original: As shown in both Fig. 7(a) and (b), our Z O RD algorithm
Revised: As shown in both Fig. 7(a) and (b), our Z O RD algorithm

Original: using the technique of dynamic virtual updates (i.e., c > 0 ) can consistently achieve improved query efficiency compared with the one not using the technique of dynamic virtual updates (i.e., c = 0 ).
Revised: using the technique of dynamic virtual updates (i.e., c > 0 ) can consistently achieve improved query efficiency compared with the one not using the technique of dynamic virtual updates (i.e., c = 0 ).

Original: This indicates the essence of dynamic virtual updates in helping improve the query efficiency of our Z O RD algorithm.
Revised: This indicates the essence of dynamic virtual updates in helping improve the query efficiency of our Z O RD algorithm.

Original: Such a result actually corroborates our theoretical insights about virtual updates (Sec. 4.2).
Revised: Such a result actually corroborates our theoretical insights about virtual updates (Sec. 4.2).

Original: Remarkably, our Z O RD algorithm without the technique of dynamic virtual updates (i.e., c = 0 ) is still able to achieve both improved query efficiency and better converged performance compared with RGF and
Revised: Remarkably, our Z O RD algorithm without the technique of dynamic virtual updates (i.e., c = 0 ) is still able to achieve both improved query efficiency and better converged performance compared with RGF and

Original: PRGF, which further verifies the superiority of our derived GP-based derivative estimation.
Revised: PRGF, which further verifies the superiority of our derived GP-based derivative estimation.

Original: More interestingly, both Fig.
Revised: More interestingly, both Fig.

Original: 7(a) and Fig.
Revised: 7(a) and Fig.

Original: 7(b) have verified that there indeed exists a trade-off for the confidence threshold c as we have discussed in Sec.
Revised: 7(b) have verified that there indeed exists a trade-off for the confidence threshold c as we have discussed in Sec.

Original: 3.2: The confidence threshold c can not be overly small or excessively large in order to achieve the best query efficiency of our Z O RD algorithm, e.g., c = 0 .
Revised: 3.2: The confidence threshold c can not be overly small or excessively large in order to achieve the best query efficiency of our Z O RD algorithm, e.g., c = 0 .

Original: 3 for Ackley ( d = 40 ) and c = 0 .
Revised: 3 for Ackley ( d = 40 ) and c = 0 .

Original: 4 for Levy ( d = 40 ).
Revised: 4 for Levy ( d = 40 ).

Original: [Table] R [Table] ON B LACK -B OX A DVERSARIAL A TTACK
Revised: D. M ORE R ESULTS ON B LACK -B OX A DVERSARIAL A TTACK

Original: Besides the comparison in our Sec.5.
Revised: Besides the comparison in our Sec.

Original: [Table] we also compare the success rate achieved by different ZO optimization algorithms on the 15 images selected from MNIST or CIFAR-10 in Fig. 8.
Revised: 5.3, we also compare the success rate achieved by different ZO optimization algorithms on the 15 images selected from MNIST or CIFAR-10 in Fig. 8.

Original: Note that we adopt the same settings in Appx.
Revised: Note that we adopt the same settings in Appx.

Original: C.2 for this comparison.
Revised: C.2 for this comparison.

Original: Considering the large computational complexity of TuRBO-1/10 algorithm for hard-to-attack images 3 which is usually undesirable in practice [Table] we drop the comparison with them in this experiment.
Revised: Considering the large computational complexity of TuRBO-1/10 algorithm for hard-to-attack images 3 which is usually undesirable in practice, we drop the comparison with them in this experiment.

Original: Fig.
Revised: Fig.

Original: 8 shows that under the same query budget [Table] our Z O RD algorithm is able to achieve considerably improved success rate over other
Revised: 8 shows that under the same query budget, our Z O RD algorithm is able to achieve considerably improved success rate over other

Original: ZO optimization algorithms.
Revised: ZO optimization algorithms.

Original: These results therefore further support the superior query efficiency of our Z O RD algorithm in real-world challenging problems
Revised: These results therefore further support the superior query efficiency of our Z O RD algorithm in real-world challenging problems.

Original: . D. M ORE R ESULTS FOR D ERIVATIVE -F REE R EINFORCEMENT L EARNING
Revised: [Figure] [Table]

Original: Recent years have also witnessed a surging interest in derivative-free reinforcement learning (Salimans et al., 2017; Qian and Yu, 2021), where ZO optimization algorithms are widely applied.
Revised: Recent years have also witnessed a surging interest in derivative-free reinforcement learning (Salimans et al., 2017; Qian and Yu, 2021), where ZO optimization algorithms are widely applied.

Original: In light of this, we also demonstrate the superiority of our Z O RD algorithm in the problem of derivative-free reinforcement learning.
Revised: In light of this, we also demonstrate the superiority of our Z O RD algorithm in the problem of derivative-free reinforcement learning.

Original: Specifically, we adopt the setting in Sec.
Revised: Specifically, we adopt the setting in Sec.

Original: C.4 to experiment in different RL environments.
Revised: C.4 to experiment in different RL environments.

Original: Tab. 3 summarizes the comparison among different ZO optimization algorithms under the same query budget of 1000.
Revised: Tab. 3 summarizes the comparison among different ZO optimization algorithms under the same query budget of 1000.

Original: As BO algorithms usually suffer from the prohibitive computational complexity for a large T (Rasmussen and Williams, 2006) and GLD has never been applied in RL, we mainly compare our Z O RD algorithm with ES, RGF and PRGF, which also belongs to the same type of ZO optimization algorithm:
Revised: As BO algorithms usually suffer from the prohibitive computational complexity for a large T (Rasmussen and Williams, 2006) and GLD has never been applied in RL, we mainly compare our Z O RD algorithm with ES, RGF and PRGF, which also belongs to the same type of ZO optimization algorithm:

Original: GD with estimated derivative.
Revised: GD with estimated derivative.

Original: Remarkably, Tab. 3 shows that under the same query budget, our Z O RD algorithm can consistently enjoy improved performance (i.e., highest rewards) than the other ZO optimization algorithms in different RL environments.
Revised: Remarkably, Tab. 3 shows that under the same query budget, our Z O RD algorithm can consistently enjoy improved performance (i.e., highest rewards) than the other ZO optimization algorithms in different RL environments.

Original: This further supports the superiority of our Z O RD algorithm to other FD-based ZO optimization algorithms.
Revised: This further supports the superiority of our Z O RD algorithm to other FD-based ZO optimization algorithms.

Original: D. D ERIVED GP IN Z O RD VS . S TANDARD GP IN BO
Revised: D. D ERIVED GP IN Z O RD VS . S TANDARD GP IN BO

Original: In contrast to using the GP to model the objective function within the entire domain for global exploration in BO, our derived GP in Z O RD will be applied to estimate the derivative of the objective function for local exploitation by GD as shown in Sec. 3.1.
Revised: In contrast to using the GP to model the objective function within the entire domain for global exploration in BO, our derived GP in Z O RD will be applied to estimate the derivative of the objective function for local exploitation by GD as shown in Sec. 3.1.

Original: As GD typically optimizes in a local region, our derived GP only needs to estimate the derivative locally , which is known to be much simpler than modeling the objective function within the entire domain in BO especially for objective functions in high-dimensional input spaces.
Revised: As GD typically optimizes in a local region, our derived GP only needs to estimate the derivative locally , which is known to be much simpler than modeling the objective function within the entire domain in BO especially for objective functions in high-dimensional input spaces.

Original: In light of this, the derived GP for derivative estimation (4) in our Z O RD algorithm advances the standard GP in BO in the following aspects: 
Revised: In light of this, the derived GP for derivative estimation (4) in our Z O RD algorithm advances the standard GP in BO in the following aspects: −  5 − F un c t i on V a l ue n =−  5 −   n =−  5 −   n =−  5 −   n =  − D e r

Revised: i v a t

Original:  Improved Query Efficiency for Estimation.
Revised: i v e  −     −     − Function Queries [Figure] Improved Query Efficiency for Estimation.

Original: The derived GP in our Z O RD algorithm requires fewer function queries to provide accurate derivative estimation.
Revised: The derived GP in our Z O RD algorithm requires fewer function queries to provide accurate derivative estimation.

Original: We provide a visual example in Fig.
Revised: We provide a visual example in Fig.

Original: 9, in which we sample a one-dimensional function f from a GP prior GP (0 , k ( x, x )) using the standard SE kernel and then randomly select the same number of queries from the input domain of [ − 6 , 6] and [0 , 3] for standard GP and our derived GP, respectively.
Revised: 9, in which we sample a one-dimensional function f from a GP prior GP (0 , k ( x, x )) using the standard SE kernel and then randomly select the same number of queries from the input domain of [ − 6 , 6] and [0 , 3] for standard GP and our derived GP, respectively.

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 9, function in a local region (i.e., x ∈ [0 , 3] ) is usually smoother than its counterpart in the entire domain (i.e., x ∈ [ − 6 , 6] ).As a result, with only −  5 −
Revised: 9, function in a local region (i.e., x ∈ [0 , 3] ) is usually smoother than its counterpart in the entire domain (i.e., x ∈ [ − 6 , 6] ).

Original: F un c t i on V a l ue n =−  5 −   n =−  5 −   n =−  5 −   n =  −

Original: D e r

Original: i v a t

Original: i v e  −     −     −

Original: Function Queries [Figure] of this objective function whereas standard GP requires more than 8 function queries to model this objective function accurately in the entire domain.
Revised: As a result, with only 4 function queries, our derived GP can already provide accurate estimation to the derivative of this objective function whereas standard GP requires more than 8 function queries to model this objective function accurately in the entire domain.

Original: Reduced Computational Complexity.
Revised: Reduced Computational Complexity.

Original: Comparing (3) and (5), both the derived GP for derivative estimation in our Z O RD algorithm and the standard GP in BO enjoy a computational complexity of O ( n 3 ) with n function queries.
Revised: Comparing (3) and (5), both the derived GP for derivative estimation in our Z O RD algorithm and the standard GP in BO enjoy a computational complexity of O ( n 3 ) with n function queries.

Original: However, as a consequence of the improved query efficiency of our derived GP, it is able to require fewer function queries (i
Revised: However, as a consequence of the improved query efficiency of our derived GP, it is able to require fewer function queries (i

Original: e., smaller n ) for accurate derivative estimation 4 and hence can enjoy a reduced computational complexity in practice especially when a large number of queries (e
Revised: e., smaller n ) for accurate derivative estimation 4 and hence can enjoy a reduced computational complexity in practice especially when a large number of queries (e

Original: g., n > 1000 ) are applied to the standard GP in BO.
Revised: g., n > 1000 ) are applied to the standard GP in BO