Original: G RADIENTLESS D ESCENT : H IGH -D IMENSIONAL Z EROTH -O RDER O PTIMIZATION
Revised: G RADIENTLESS D ESCENT : H IGH -D IMENSIONAL Z EROTH -O RDER O PTIMIZATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Zeroth-order optimization is the process of minimizing an objective f ( x ) , given oracle access to evaluations at adaptively chosen inputs x .
Revised: Zeroth-order optimization is the process of minimizing an objective f ( x ) , given oracle access to evaluations at adaptively chosen inputs x .

Original: In this paper, we present two simple yet powerful GradientLess Descent (GLD) algorithms that do not rely on an underlying gradient estimate and are numerically stable.
Revised: In this paper, we present two simple yet powerful GradientLess Descent (GLD) algorithms that do not rely on an underlying gradient estimate and are numerically stable.

Original: We analyze our algorithm from a novel geometric perspective and present a novel analysis that shows convergence within an (cid:15) -ball of the optimum in O ( kQ log( n ) log( R/(cid:15) ))
Revised: We analyze our algorithm from a novel geometric perspective and present a novel analysis that shows convergence within an (cid:15) -ball of the optimum in O ( kQ log( n ) log( R/(cid:15) ))

Original: evaluations, for any monotone transform of a smooth and strongly convex objective with latent dimension k < n , where the input dimension is n , R is the diameter of the input space and Q is the condition number.
Revised: evaluations, for any monotone transform of a smooth and strongly convex objective with latent dimension k < n , where the input dimension is n , R is the diameter of the input space and Q is the condition number.

Original: Our rates are the ﬁrst of its kind to be both 1) poly-logarithmically dependent on dimensionality and 2) invariant under monotone transformations.
Revised: Our rates are the ﬁrst of its kind to be both 1) poly-logarithmically dependent on dimensionality and 2) invariant under monotone transformations.

Original: We further leverage our geometric perspective to show that our analysis is optimal.
Revised: We further leverage our geometric perspective to show that our analysis is optimal.

Original: Both monotone invariance and its ability to utilize a low latent dimensionality are key to the empirical success of our algorithms, as demonstrated on BBOB and MuJoCo benchmarks.
Revised: Both monotone invariance and its ability to utilize a low latent dimensionality are key to the empirical success of our algorithms, as demonstrated on BBOB and MuJoCo benchmarks

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: We consider the problem of zeroth-order optimization (also known as gradient-free optimization, or bandit optimization), where our goal is to minimize an objective function f : R n → R with as few evaluations of f ( x ) as possible.
Revised: We consider the problem of zeroth-order optimization (also known as gradient-free optimization, or bandit optimization), where our goal is to minimize an objective function f : R n → R with as few evaluations of f ( x ) as possible.

Original: For many practical and interesting objective functions, gradients are difﬁcult to compute and there is still a need for zeroth-order optimization in applications such as reinforcement learning (Mania et al., 2018; Salimans et al., 2017; Choromanski et al., 2018), attacking neural networks (Chen et al., 2017; Papernot et al., 2017), hyperparameter tuning of deep networks (Snoek et al., 2012), and network control (Liu et al., 2017).
Revised: For many practical and interesting objective functions, gradients are difﬁcult to compute and there is still a need for zeroth-order optimization in applications such as reinforcement learning (Mania et al., 2018; Salimans et al., 2017; Choromanski et al., 2018), attacking neural networks (Chen et al., 2017; Papernot et al., 2017), hyperparameter tuning of deep networks (Snoek et al., 2012), and network control (Liu et al., 2017).

Original: The standard approach to zeroth-order optimization is, ironically, to estimate the gradients from function values and apply a ﬁrst-order optimization algorithm (Flaxman et al., 2005).
Revised: The standard approach to zeroth-order optimization is, ironically, to estimate the gradients from function values and apply a ﬁrst-order optimization algorithm (Flaxman et al., 2005).

Original: Nesterov &
Revised: Nesterov &

Original: Spokoiny (2011) analyze this class of algorithms as gradient descent on a Gaussian smoothing of the objective and gives an accelerated O ( n √ Q log(( LR 2 + F ) /(cid:15) )) iteration complexity for an LLipschitz convex function with condition number Q and R = (cid:107) x 0 − x ∗ (cid:107) and F = f ( x 0 ) − f ( x ∗ ) .
Revised: Spokoiny (2011) analyze this class of algorithms as gradient descent on a Gaussian smoothing of the objective and gives an accelerated O ( n √ Q log(( LR 2 + F ) /(cid:15) )) iteration complexity for an LLipschitz convex function with condition number Q and R = (cid:107) x 0 − x ∗ (cid:107) and F = f ( x 0 ) − f ( x ∗ ) .

Original: They propose a two-point evaluation scheme that constructs gradient estimates from the difference between function values at two points that are close to each other.
Revised: They propose a two-point evaluation scheme that constructs gradient estimates from the difference between function values at two points that are close to each other.

Original: This scheme was extended by (Duchi et al., 2015) for stochastic settings, by (Ghadimi & Lan, 2013) for nonconvex settings, and by (Shamir, 2017) for non-smooth and non-Euclidean norm settings.
Revised: This scheme was extended by (Duchi et al., 2015) for stochastic settings, by (Ghadimi & Lan, 2013) for nonconvex settings, and by (Shamir, 2017) for non-smooth and non-Euclidean norm settings.

Original: Since then, ﬁrst-order techniques such as variance reduction (Liu et al., 2018), conditional gradients (Balasubramanian & Ghadimi, 2018), and diagonal preconditioning (Mania et al., 2018) have been successfully adopted in this setting.
Revised: Since then, ﬁrst-order techniques such as variance reduction (Liu et al., 2018), conditional gradients (Balasubramanian & Ghadimi, 2018), and diagonal preconditioning (Mania et al., 2018) have been successfully adopted in this setting.

Original: This class of algorithms are also known as stochastic search, random search, or (natural) evolutionary strategies and have been augmented with a variety of heuristics, such as the popular
Revised: This class of algorithms are also known as stochastic search, random search, or (natural) evolutionary strategies and have been augmented with a variety of heuristics, such as the popular

Original: CMA-ES (Auger & Hansen, 2005).
Revised: CMA-ES (Auger & Hansen, 2005).

Original: These algorithms, however, suffer from high variance due to non-robust local minima or highly non-smooth objectives, which are common in the ﬁelds of deep learning and reinforcement learning.
Revised: These algorithms, however, suffer from high variance due to non-robust local minima or highly non-smooth objectives, which are common in the ﬁelds of deep learning and reinforcement learn- ing.

Original: Mania et al.
Revised: Mania et al.

Original: (2018) notes that gradient variance increases as training progresses due to higher variance in the objective functions, since often parameters must be tuned precisely to achieve reasonable models.
Revised: (2018) notes that gradient variance increases as training progresses due to higher variance in the objective functions, since often parameters must be tuned precisely to achieve reasonable models.

Original: Therefore, some attention has shifted into direct search algorithms that usually ﬁnds a descent direction u and moves to x + δu , where the step size is not scaled by the function difference.
Revised: Therefore, some attention has shifted into direct search algorithms that usually ﬁnds a descent direction u and moves to x + δu , where the step size is not scaled by the function difference.

Original: The ﬁrst approaches for direct search were based on deterministic approaches with a positive spanning set and date back to the 1950s (Brooks, 1958).
Revised: The ﬁrst approaches for direct search were based on deterministic approaches with a positive spanning set and date back to the 1950s (Brooks, 1958).

Original: Only recently have theoretical bounds surfaced, with Gratton et al.
Revised: Only recently have theoretical bounds surfaced, with Gratton et al.

Original: (2015) giving an iteration complexity that is a large polynomial of n and Dodangeh & Vicente (2016) giving an improved O ( n 2 L 2 /(cid:15) ) .
Revised: (2015) giving an iteration complexity that is a large polynomial of n and Dodangeh & Vicente (2016) giving an improved O ( n 2 L 2 /(cid:15) ) .

Original: Stochastic approaches tend to have better complexities:
Revised: Stochastic approaches tend to have better complexities:

Original: Stich et al.
Revised: Stich et al.

Original: (2013) uses line search to give a O ( nQ log( F/(cid:15) )) iteration complexity for convex functions with condition number Q and most recently, Gorbunov et al.
Revised: (2013) uses line search to give a O ( nQ log( F/(cid:15) )) iteration complexity for convex functions with condition number Q and most recently, Gorbunov et al.

Original: (2019) uses importance sampling to give a O ( n ¯ Q log( F/(cid:15) )) complexity for convex functions with average condition number ¯ Q , assuming access to sampling probabilities.
Revised: (2019) uses importance sampling to give a O ( n ¯ Q log( F/(cid:15) )) complexity for convex functions with average condition number ¯ Q , assuming access to sampling probabilities.

Original: Stich et al.
Revised: Stich et al.

Original: (2013) notes that direct search algorithms are invariant under monotone transforms of the objective, a property that might explain their robustness in high-variance settings.
Revised: (2013) notes that direct search algorithms are invariant under monotone transforms of the objective, a property that might explain their robustness in high-variance settings.

Original: In general, zeroth order optimization suffers an at least linear dependence on input dimension n and recent works have tried to address this limitation when n is large but f ( x ) admits a low-dimensional structure.
Revised: In general, zeroth order optimization suffers an at least linear dependence on input dimension n and recent works have tried to address this limitation when n is large but f ( x ) admits a low-dimensional structure.

Original: Some papers assume that f ( x ) depends only on k coordinates and Wang et al.
Revised: Some papers assume that f ( x ) depends only on k coordinates and Wang et al.

Original: (2017) applies Lasso to ﬁnd the important set of coordinates, whereas Balasubramanian & Ghadimi (2018) simply change the step size to achieve an O ( k (log( n ) /(cid:15) ) 2 ) iteration complexity.
Revised: (2017) applies Lasso to ﬁnd the important set of coordinates, whereas Balasubramanian & Ghadimi (2018) simply change the step size to achieve an O ( k (log( n ) /(cid:15) ) 2 ) iteration complexity.

Original: Other papers assume more generally that f ( x ) =
Revised: Other papers assume more generally that f ( x ) =

Original: g ( P A x ) only depends on a k -dimensional subspace given by the range of P A and Djolonga et al.
Revised: g ( P A x ) only depends on a k -dimensional subspace given by the range of P A and Djolonga et al.

Original: (2013) apply low-rank approximation to ﬁnd the low-dimensional subspace while Wang et al.
Revised: (2013) apply low-rank approximation to ﬁnd the low-dimensional subspace while Wang et al.

Original: (2013) use random embeddings.
Revised: (2013) use random embeddings.

Original: Hazan et al.
Revised: Hazan et al.

Original: (2017) assume that f ( x ) is a sparse collection of k -degree monomials on the Boolean hypercube and apply sparse recovery to achieve a O ( n k ) runtime bound.
Revised: (2017) assume that f ( x ) is a sparse collection of k -degree monomials on the Boolean hypercube and apply sparse recovery to achieve a O ( n k ) runtime bound.

Original: We will show that under the case that f ( x ) =
Revised: We will show that under the case that f ( x ) =

Original: g ( P A x ) , our algorithm will inherently pick up any low-dimensional structure in f ( x ) and achieve a convergence rate that depends on k log( n ) .
Revised: g ( P A x ) , our algorithm will inherently pick up any low-dimensional structure in f ( x ) and achieve a convergence rate that depends on k log( n ) .

Original: This initial convergence rate survives, even if we perturb f ( x )
Revised: This initial convergence rate survives, even if we perturb f ( x )

Original: =
Revised: =

Original: g ( P A x ) + h ( x ) , so long as h ( x ) is sufﬁciently small.
Revised: g ( P A x ) + h ( x ) , so long as h ( x ) is sufﬁciently small.

Original: We will not cover the whole variety of black-box optimization methods, such as Bayesian optimization or genetic algorithms.
Revised: We will not cover the whole variety of black-box optimization methods, such as Bayesian optimization or genetic algorithms.

Original: In general, these methods attempt to solve a broader problem (e.g. multiple optima), have weaker theoretical guarantees and may require substantial computation at each step: e.g. Bayesian optimization generally has theoretical iteration complexities that grow exponentially in dimension, and CMA-ES lacks provable complexity bounds beyond convex quadratic functions.
Revised: In general, these methods attempt to solve a broader problem (e.g. multiple optima), have weaker theoretical guarantees and may require substantial computation at each step: e.g. Bayesian optimization generally has theoretical iteration complexities that grow exponentially in dimension, and CMA-ES lacks provable complexity bounds beyond convex quadratic functions.

Original: In addition to the slow runtime and weaker guarantees, Bayesian optimization assumes the success of an inner optimization loop of the acquisition function.
Revised: In addition to the slow runtime and weaker guarantees, Bayesian optimization assumes the success of an inner optimization loop of the acquisition function.

Original: This inner optimization is often implemented with many iterations of a simpler zeroth-order methods, justifying the need to understand gradient-less descent algorithms within its own context.
Revised: This inner optimization is often implemented with many iterations of a simpler zeroth-order methods, justifying the need to understand gradient-less descent algorithms within its own context.

Original: 1. O UR CONTRIBUTIONS
Revised: 1. O UR CONTRIBUTIONS

Original: In this paper, we present GradientLess Descent (GLD), a class of truly gradient-free algorithms (also known as direct search algorithms) that are parameter free and provably fast.
Revised: In this paper, we present GradientLess Descent (GLD), a class of truly gradient-free algorithms (also known as direct search algorithms) that are parameter free and provably fast.

Original: Our algorithms are based on a simple intuition: for well-conditioned functions, if we start from a point and take a small step in a randomly chosen direction, there is a signiﬁcant probability that we will reduce the objective function value.
Revised: Our algorithms are based on a simple intuition: for well-conditioned functions, if we start from a point and take a small step in a randomly chosen direction, there is a signiﬁcant probability that we will reduce the objective function value.

Original: We present a novel analysis that relies on facts in high dimensional geometry and can thus be viewed as a geometric analysis of gradient-free algorithms, recovering the standard convergence rates and step sizes.
Revised: We present a novel analysis that relies on facts in high dimensional geometry and can thus be viewed as a geometric analysis of gradient-free algorithms, recovering the standard convergence rates and step sizes.

Original: Speciﬁcally, we show that if the step size is on the order of O ( 1 √ n ) , we can guarantee an expected decrease of 1 − Ω( 1 n ) in the optimality gap, based on geometric properties of the sublevel sets of a smooth and strongly convex function.
Revised: Speciﬁcally, we show that if the step size is on the order of O ( 1 √ n ) , we can guarantee an expected decrease of 1 − Ω( 1 n ) in the optimality gap, based on geometric properties of the sublevel sets of a smooth and strongly convex function.

Original: Our results are invariant under monotone transformations of the objective function, thus our convergence results also hold for a large class of non-convex functions that are a subclass of quasi-convex functions.
Revised: Our results are invariant under monotone transformations of the objective function, thus our convergence results also hold for a large class of non-convex functions that are a subclass of quasi-convex functions.

Original: Speciﬁcally, note that monotone transformations of convex functions are not necessarily convex.
Revised: Speciﬁcally, note that monotone transformations of convex functions are not necessarily convex.

Original: However, a monotone transformation of a convex function is always quasi-convex .
Revised: However, a monotone transformation of a convex function is always quasi-convex .

Original: The maximization of quasi-concave utility functions, which is equivalent to the minimization of quasiconvex functions, is an important topic of study in economics (e.g. Arrow & Enthoven (1961)). 
Revised: The maximization of quasi-concave utility functions, which is equivalent to the minimization of quasiconvex functions, is an important topic of study in economics (e.g. Arrow & Enthoven (1961)). [Table] [Table] This paper (GLD)

Original:  Intuition suggests that the step-size dependence on dimensionality can be improved when f ( x ) admits a low-dimensional structure.
Revised: [Table] Intuition suggests that the step-size dependence on dimensionality can be improved when f ( x ) admits a low-dimensional structure.

Original: With a careful choice of sampling distribution we can show that if f ( x ) =
Revised: With a careful choice of sampling distribution we can show that if f ( x ) =

Original: g ( P A x ) , where P A is a rank k matrix, then our step size can be on the order of O ( 1 √ k ) as our optimization behavior is preserved under projections.
Revised: g ( P A x ) , where P A is a rank k matrix, then our step size can be on the order of O ( 1 √ k ) as our optimization behavior is preserved under projections.

Original: We call this property afﬁne-invariance and show that the number of function evaluations needed for convergence depends logarithmically on n .
Revised: We call this property afﬁne-invariance and show that the number of function evaluations needed for convergence depends logarithmically on n .

Original: Unlike most previous algorithms in the high-dimensional setting, no expensive sparse recovery [Table] This paper (GLD) [Table]
Revised: Unlike most previous algorithms in the high-dimensional setting, no expensive sparse recovery or subspace ﬁnding methods are needed.

Original: or subspace ﬁnding methods are needed.

Original: Furthermore, by novel perturbation arguments, we show that our fast convergence rates are robust and holds even under the more realistic assumption when f ( x ) =
Revised: Furthermore, by novel perturbation arguments, we show that our fast convergence rates are robust and holds even under the more realistic assumption when f ( x ) =

Original: g ( P A x ) + h ( x ) with h ( x ) being sufﬁciently small.
Revised: g ( P A x ) + h ( x ) with h ( x ) being sufﬁciently small.

Original: Theorem 1 (Convergence of GLD: Informal Restatement of Theorem 7 and Theorem 14) .
Revised: Theorem 1 (Convergence of GLD: Informal Restatement of Theorem 7 and Theorem 14) .

Original: Let f ( x ) be any monotone transform of a convex function with condition number Q and R = (cid:107) x 0 − x ∗ (cid:107) .
Revised: Let f ( x ) be any monotone transform of a convex function with condition number Q and R = (cid:107) x 0 − x ∗ (cid:107) .

Original: Let y be a sample from an appropriate distribution centered at x .
Revised: Let y be a sample from an appropriate distribution centered at x .

Original: Then, with constant probability,
Revised: Then, with constant probability,

Original: [Equation]
Revised: [Equation]

Original: Therefore, we can ﬁnd x T such that (cid:107) x
Revised: Therefore, we can ﬁnd x T such that (cid:107) x

Original: T − x ∗ (cid:107) ≤ (cid:15) after T = (cid:101) O
Revised: T − x ∗ (cid:107) ≤ (cid:15) after T = (cid:101) O

Original: ( nQ log( R/(cid:15) )) function evaluations.
Revised: ( nQ log( R/(cid:15) )) function evaluations.

Original: Furthermore, for functions f ( x ) =
Revised: Furthermore, for functions f ( x ) =

Original: g ( P A x ) + h ( x ) with rank k matrix P A and sufﬁciently small h ( x ) , we only require (cid:101)
Revised: g ( P A x ) + h ( x ) with rank k matrix P A and sufﬁciently small h ( x ) , we only require (cid:101)

Original: O
Revised: O

Original: ( kQ log( n ) log( R/(cid:15) )) evaluations.
Revised: ( kQ log( n ) log( R/(cid:15) )) evaluations.

Original: Another advantage of our non-standard geometric analysis is that it allows us to deduce that our rates are optimal with a matching lower bound (up to logarithmic factors), presenting theoretical evidence that gradient-free inherently requires Ω( nQ ) function evaluations to converge.
Revised: Another advantage of our non-standard geometric analysis is that it allows us to deduce that our rates are optimal with a matching lower bound (up to logarithmic factors), presenting theoretical evidence that gradient-free inherently requires Ω( nQ ) function evaluations to converge.

Original: While gradient-estimation algorithms can achieve a better theoretical iteration complexity of O ( n √ Q ) , they lack the monotone and afﬁne invariance properties.
Revised: While gradient-estimation algorithms can achieve a better theoretical iteration complexity of O ( n √ Q ) , they lack the monotone and afﬁne invariance properties.

Original: Empirically, we see that invariance properties are important to successful optimization, as validated by experiments on synthetic BBOB and
Revised: Empirically, we see that invariance properties are important to successful optimization, as validated by experiments on synthetic BBOB and

Original: MuJoCo benchmarks that show the competitiveness of GLD against standard optimization procedures.
Revised: MuJoCo benchmarks that show the competitiveness of GLD against standard optimization procedures.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We ﬁrst deﬁne a few notations for the rest of the paper.
Revised: We ﬁrst deﬁne a few notations for the rest of the paper.

Original: Let X be a compact subset of R n and let (cid:107) · (cid:107) denote the Euclidean norm.
Revised: Let X be a compact subset of R n and let (cid:107) · (cid:107) denote the Euclidean norm.

Original: The diameter of X , denoted (cid:107)X(cid:107) =
Revised: The diameter of X , denoted (cid:107)X(cid:107) =

Original: max x,x (cid:48) ∈X (cid:107) x − x (cid:48) (cid:107) , is the maximum distance between elements in X .
Revised: max x,x (cid:48) ∈X (cid:107) x − x (cid:48) (cid:107) , is the maximum distance between elements in X .

Original: Let f : X → R be a real-valued function which attains its minimum at x ∗ .
Revised: Let f : X → R be a real-valued function which attains its minimum at x ∗ .

Original: We use f ( X ) =
Revised: We use f ( X ) =

Original: { f ( x ) : x ∈ X} to denote the image of f on a subset X of R n , and B ( c, r ) = { x ∈ R n : (cid:107) c − x (cid:107) ≤ r } to denote the ball of radius r centered at c .
Revised: { f ( x ) : x ∈ X} to denote the image of f on a subset X of R n , and B ( c, r ) = { x ∈ R n : (cid:107) c − x (cid:107) ≤ r } to denote the ball of radius r centered at c .

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: The level set of f at point x ∈ X is L c ( f ) =
Revised: The level set of f at point x ∈ X is L c ( f ) =

Original: { y ∈ X : f ( y ) =
Revised: { y ∈ X : f ( y ) =

Original: f ( x ) } .
Revised: f ( x ) } .

Original: The sub-level set of f at point x ∈ X is L ↓ c ( f ) =
Revised: The sub-level set of f at point x ∈ X is L ↓ c ( f ) =

Original: { y ∈ X : f ( y ) ≤ f ( x ) } .
Revised: { y ∈ X : f ( y ) ≤ f ( x ) } .

Original: When the function f is clear from the context, we omit it.
Revised: When the function f is clear from the context, we omit it.

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: We say that f is α -strongly convex for α > 0 if f ( y ) ≥ f ( x ) + (cid:104)∇ f ( x ) , y − x (cid:105) + α 2 (cid:107) y − x (cid:107) 2 for all x, y ∈ X and β -smooth for β > 0 if f ( y ) ≤
Revised: We say that f is α -strongly convex for α > 0 if f ( y ) ≥ f ( x ) + (cid:104)∇ f ( x ) , y − x (cid:105) + α 2 (cid:107) y − x (cid:107) 2 for all x, y ∈ X and β -smooth for β > 0 if f ( y ) ≤

Original: f ( x ) + (cid:104)∇ f ( x ) , y − x (cid:105) + β 2 (cid:107)
Revised: f ( x ) + (cid:104)∇ f ( x ) , y − x (cid:105) + β 2 (cid:107)

Original: y − x
Revised: y − x

Original: (cid:107)for all x, y ∈ X .
Revised: (cid:107)for all x, y ∈ X .

Original: Deﬁnition 4.
Revised: Deﬁnition 4.

Original: We say that g ◦ f is a monotone transformation of f if g : f ( X ) → R is a monotonically (and strictly) increasing function.
Revised: We say that g ◦ f is a monotone transformation of f if g : f ( X ) → R is a monotonically (and strictly) increasing function.

Original: Monotone transformations preserve the level sets of a function in the sense that L x ( f ) =
Revised: Monotone transformations preserve the level sets of a function in the sense that L x ( f ) =

Original: L x ( g ◦ f ) .
Revised: L x ( g ◦ f ) .

Original: Because our algorithms depend only on the level set properties, our results generalize to any monotone transformation of a strongly convex and strongly smooth function.
Revised: Because our algorithms depend only on the level set properties, our results generalize to any monotone transformation of a strongly convex and strongly smooth function.

Original: This leads to our extended notion of condition number.
Revised: This leads to our extended notion of condition number.

Original: Deﬁnition 5.
Revised: Deﬁnition 5.

Original: A function f has condition number Q ≥ 1 if it is the minimum ratio β/α over all functions g such that f is a monotone transformation of g and g is α -strongly convex and β smooth.
Revised: A function f has condition number Q ≥ 1 if it is the minimum ratio β/α over all functions g such that f is a monotone transformation of g and g is α -strongly convex and β smooth.

Original: When we work with low rank extensions of f , we only care about the condition number of f within a rank k subspace.
Revised: When we work with low rank extensions of f , we only care about the condition number of f within a rank k subspace.

Original: Indeed, if f only varies along a rank k subspace, then it has a strong convexity value of 0 , making its condition number undeﬁned.
Revised: Indeed, if f only varies along a rank k subspace, then it has a strong convexity value of 0 , making its condition number undeﬁned.

Original: If f is α -strongly convex and β -smooth, then its
Revised: If f is α -strongly convex and β -smooth, then its

Original: Hessian matrix always has eigenvalues bounded between α and β .
Revised: Hessian matrix always has eigenvalues bounded between α and β .

Original: Therefore, we need a notion of a projected condition number.
Revised: Therefore, we need a notion of a projected condition number.

Original: Let A ∈ R d × k be some orthonormal matrix and let P A = AA (cid:62) be the projection matrix onto the column space of A .
Revised: Let A ∈ R d × k be some orthonormal matrix and let P A = AA (cid:62) be the projection matrix onto the column space of A .

Original: Deﬁnition 6.
Revised: Deﬁnition 6.

Original: For some orthonormal A ∈ R d × k with d > k , a function f has condition number restricted to A , Q ( A ) ≥ 1 , if it is the minimum ratio β/α over all functions g such that f is a monotone transformation of g and h ( y )
Revised: For some orthonormal A ∈ R d × k with d > k , a function f has condition number restricted to A , Q ( A ) ≥ 1 , if it is the minimum ratio β/α over all functions g such that f is a monotone transformation of g and h ( y )

Original: =
Revised: =

Original: g ( A y ) is α -strongly convex and β smooth.
Revised: g ( A y ) is α -strongly convex and β smooth.

Original:  A NALYSIS OF D ESCENT S TEPS
Revised:  A NALYSIS OF D ESCENT S TEPS

Original: The GLD template can be summarized as follows: given a sampling distribution D , we start at xand in iteration t , we choose a scalar radii r t and we sample y t from a distribution r t D centered around x t , where r t provides the scaling of D .
Revised: The GLD template can be summarized as follows: given a sampling distribution D , we start at xand in iteration t , we choose a scalar radii r t and we sample y t from a distribution r t D centered around x t , where r t provides the scaling of D .

Original: Then, if f ( x t +1 ) < x t , we update x t +

Revised: Then, if f ( x t +1 ) < x t , we update x t +

Original: =
Revised: =

Original: y t ; otherwise, we set x t +1 = x t .
Revised: y t ; otherwise, we set x t +1 = x t .

Original: The analysis of GLD follows from the main observation that the sublevel set of a monotone transformation of a strongly convex and strongly smooth function contains a ball of sufﬁciently large radius tangent to the level set (Lemma 15).
Revised: The analysis of GLD follows from the main observation that the sublevel set of a monotone transformation of a strongly convex and strongly smooth function contains a ball of sufﬁciently large radius tangent to the level set (Lemma 15).

Original: In this section, we show that this property, combined with facts of high-dimensional geometry, implies that moving in a random direction from any point has a good chance of signiﬁcantly improving the objective.
Revised: In this section, we show that this property, combined with facts of high-dimensional geometry, implies that moving in a random direction from any point has a good chance of signiﬁcantly improving the objective.

Original: As we mentioned before, the key to fast convergence is the careful choice of step sizes, which we describe in Theorem 7.
Revised: As we mentioned before, the key to fast convergence is the careful choice of step sizes, which we describe in Theorem 7.

Original: The intuition here is that we would like to take as large steps as possible while keeping the probability of improving the objective function reasonably high, so by insights in high-dimensional geometry, we choose a step size of Θ(1 / √ n ) .
Revised: The intuition here is that we would like to take as large steps as possible while keeping the probability of improving the objective function reasonably high, so by insights in high-dimensional geometry, we choose a step size of Θ(1 / √ n ) .

Original: Also, we show that if f ( x ) admits a latent rank-
Revised: Also, we show that if f ( x ) admits a latent rank-

Original: k structure, then this step size can be increased to Θ(1 / √ k ) and is therefore only dependent on the latent dimensionality of f ( x ) , allowing for fast high-dimensional optimization.
Revised: k structure, then this step size can be increased to Θ(1 / √ k ) and is therefore only dependent on the latent dimensionality of f ( x ) , allowing for fast high-dimensional optimization.

Original: Lastly, our geometric understanding allows us to show that our convergence rates are optimal with a matching lower bound.
Revised: Lastly, our geometric understanding allows us to show that our convergence rates are optimal with a matching lower bound.

Original: Without loss of generality, this section assumes that f ( x ) is strongly convex and smooth with condition number Q .
Revised: Without loss of generality, this section assumes that f ( x ) is strongly convex and smooth with condition number Q .

Original: 3. S TEP S IZE
Revised: 3. S TEP S IZE

Original: Theorem 7.
Revised: Theorem 7.

Original: For any x such that 35 Q (cid:107) x − x ∗ (cid:107) ∈
Revised: For any x such that 35 Q (cid:107) x − x ∗ (cid:107) ∈

Original: [ C 1 , C 2 ] , we can ﬁnd integers 0 ≤ k 1 , k 2 < log C 2 Csuch that if r = 2 k 1 C 1 or r = 2 − k 2 C 2 , then a random sample y from uniform distribution over
Revised: [ C 1 , C 2 ] , we can ﬁnd integers 0 ≤ k 1 , k 2 < log C 2 Csuch that if r = 2 k 1 C 1 or r = 2 − k 2 C 2 , then a random sample y from uniform distribution over

Original: B x = B ( x, r √ n ) satisﬁes
Revised: B x = B ( x, r √ n ) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: with probability at least 14 .
Revised: with probability at least 14 .

Original: Proving the above theorem requires the following lemma about the intersection of balls in high dimensions and it is proved in the appendix.
Revised: Proving the above theorem requires the following lemma about the intersection of balls in high dimensions and it is proved in the appendix.

Original: Lemma 8.
Revised: Lemma 8.

Original: Let B 1 and B 2 be two balls in R n of radii r 1 and r 2 respectively.
Revised: Let B 1 and B 2 be two balls in R n of radii r 1 and r 2 respectively.

Original: Let (cid:96) be the distance between the centers.
Revised: Let (cid:96) be the distance between the centers.

Original: If r 1 ∈ [ (cid:96) 2 √ n , (cid:96) √ n ] and r 2 ≥ (cid:96) − (cid:96) 4 n , then vol ( B 1 ∩ B 2 ) ≥ c n vol ( B 1 ) , where c n is a dimension-dependent constant that is lower bounded by 1 4 at n = 1 .
Revised: If r 1 ∈ [ (cid:96) 2 √ n , (cid:96) √ n ] and r 2 ≥ (cid:96) − (cid:96) 4 n , then [Equation] where c n is a dimension-dependent constant that is lower bounded by 1 4 at n = 1 .

Original: 3. G AUSSIAN S AMPLING AND L OW R ANK S TRUCTUREA
Revised: 3. G AUSSIAN S AMPLING AND L OW R ANK S TRUCTURE

Original:  direct application of Lemma 8 seems to imply that uniform sampling of a high-dimensional ball is necessary.
Revised: A direct application of Lemma 8 seems to imply that uniform sampling of a high-dimensional ball is necessary.

Original: Upon further inspection, this can be easily replaced with a much simpler Gaussian sampling procedure that concentrates the mass close to the surface to the ball.
Revised: Upon further inspection, this can be easily replaced with a much simpler Gaussian sampling procedure that concentrates the mass close to the surface to the ball.

Original: This procedure lends itself to better analysis when f ( x ) admits a latent low-dimensional structure since any afﬁne projection of a Gaussian is still Gaussian.
Revised: This procedure lends itself to better analysis when f ( x ) admits a latent low-dimensional structure since any afﬁne projection of a Gaussian is still Gaussian.

Original: Lemma 9.
Revised: Lemma 9.

Original: Let B 1 and B 2 be two balls in R n of radii r 1 and r 2 respectively.
Revised: Let B 1 and B 2 be two balls in R n of radii r 1 and r 2 respectively.

Original: Let (cid:96) be the distance between the centers.
Revised: Let (cid:96) be the distance between the centers.

Original: If r 1 ∈ [ (cid:96) 2 √ n , (cid:96) √ n ] and r 2 ≥ (cid:96) − (cid:96)n and X =
Revised: If r 1 ∈ [ (cid:96) 2 √ n , (cid:96) √ n ] and r 2 ≥ (cid:96) − (cid:96)n and X =

Original: ( X 1 , ..., X n ) are independent
Revised: ( X 1 , ..., X n ) are independent

Original: Gaussians with mean centered at the center of B 1 and variance r 21 n , then
Revised: Gaussians with mean centered at the center of B 1 and variance r 21 n , then

Original: [Equation]
Revised: [Equation]

Original: where c is a dimension-independent constant.
Revised: where c is a dimension-independent constant.

Original: Assume that there exists some rank k projection matrix P A such that f ( x )
Revised: Assume that there exists some rank k projection matrix P A such that f ( x )

Original: =
Revised: =

Original: g ( P A x ) , where k is much smaller than n .
Revised: g ( P A x ) , where k is much smaller than n .

Original: Because Gaussians projected on a k -dimensional subspace are still Gaussians, we show that our algorithm has a dimension dependence on k .
Revised: Because Gaussians projected on a k -dimensional subspace are still Gaussians, we show that our algorithm has a dimension dependence on k .

Original: We let Q g ( A ) be the condition number of g restricted to the subspace
Revised: We let Q g ( A ) be the condition number of g restricted to the subspace

Original: A that drives the dominant changes in f ( x ) .
Revised: A that drives the dominant changes in f ( x ) .

Original: Theorem 10.
Revised: Theorem 10.

Original: Let f ( x )
Revised: Let f ( x )

Original: =
Revised: =

Original: g ( P A x ) for some unknown rank k matrix P A with k < n and suppose 35 Q (cid:107) P A ( x − x ∗ ) (cid:107) ∈
Revised: g ( P A x ) for some unknown rank k matrix P A with k < n and suppose 35 Q (cid:107) P A ( x − x ∗ ) (cid:107) ∈

Original: [ C 1 , C 2 ] for some numbers C 1 , C 2 ∈ R + .
Revised: [ C 1 , C 2 ] for some numbers C 1 , C 2 ∈ R + .

Original: Then, there exist integers 0 ≤ k 1 , k 2 < log C 2 C 1 such that if r = 2 k 1 C 1 or r = 2 − k 2 C 2 , then a random sample y from a Gaussian distribution N ( x, r 2 k I ) satisﬁes
Revised: Then, there exist integers 0 ≤ k 1 , k 2 < log C 2 C 1 such that if r = 2 k 1 C 1 or r = 2 − k 2 C 2 , then a random sample y from a Gaussian distribution N ( x, r 2 k I ) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: with constant probability.
Revised: with constant probability.

Original: Note that the speed-up in progress is due to the fact that we can now tolerate the larger sampling radius of Ω(1 / √ k ) , while maintaining a high probability of making progress.
Revised: Note that the speed-up in progress is due to the fact that we can now tolerate the larger sampling radius of Ω(1 / √ k ) , while maintaining a high probability of making progress.

Original: If k is unknown, we can simply use binary search to ﬁnd the correct radius with an extra factor of log( n ) in our runtime.
Revised: If k is unknown, we can simply use binary search to ﬁnd the correct radius with an extra factor of log( n ) in our runtime.

Original: The low-rank assumption is too restrictive to be realistic; however, our fast rates still hold, at least for the early stages of the optimization, even if we assume that f ( x ) =
Revised: The low-rank assumption is too restrictive to be realistic; however, our fast rates still hold, at least for the early stages of the optimization, even if we assume that f ( x ) =

Original: g ( P A x ) + h ( x ) and | h ( x ) | ≤ δ is a full-rank function that is bounded by δ .
Revised: g ( P A x ) + h ( x ) and | h ( x ) | ≤ δ is a full-rank function that is bounded by δ .

Original: In this setting, we can show that convergence remains fast, at least until the optimality gap approaches δ .
Revised: In this setting, we can show that convergence remains fast, at least until the optimality gap approaches δ .

Original: Theorem 11.
Revised: Theorem 11.

Original: Let f ( x )
Revised: Let f ( x )

Original: =
Revised: =

Original: g ( P A x ) + h ( x ) for some unknown rank k matrix P A with k < n where g, h are convex and | h | ≤ δ .
Revised: g ( P A x ) + h ( x ) for some unknown rank k matrix P A with k < n where g, h are convex and | h | ≤ δ .

Original: Suppose 35 Q (cid:107) P A x − z ∗ (cid:107) ∈
Revised: Suppose 35 Q (cid:107) P A x − z ∗ (cid:107) ∈

Original: [ C 1 , C 2 ] for some numbers C 1 , C 2 ∈ R + where z ∗ minimizes g ( z ) .
Revised: [ C 1 , C 2 ] for some numbers C 1 , C 2 ∈ R + where z ∗ minimizes g ( z ) .

Original: Then, there exist integers 0 ≤ k 1 , k 2 < log C 2 C 1 such that if r = 2 k 1 C 1 or r = 2 − k 2 C 2 , then a random sample y from a Gaussian distribution N ( x, r 2 k I ) satisﬁes
Revised: Then, there exist integers 0 ≤ k 1 , k 2 < log C 2 C 1 such that if r = 2 k 1 C 1 or r = 2 − k 2 C 2 , then a random sample y from a Gaussian distribution N ( x, r 2 k I ) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: with constant probability whenever f ( x ) − f ( x ∗ ) ≥ 60 δkQ g ( A ) .
Revised: with constant probability whenever f ( x ) − f ( x ∗ ) ≥ 60 δkQ g ( A ) .

Original: 3. L OWER B OUNDS
Revised: 3. L OWER B OUNDS

Original: We show that our upper bounds given in the previous section are tight up to logarithmic factors for any symmetric sampling distribution D .
Revised: We show that our upper bounds given in the previous section are tight up to logarithmic factors for any symmetric sampling distribution D .

Original: These lower bounds are easily derived from our geometric perspective as we show that a sampling distribution with a large radius gives an extremely low probability of intersection with the desired sub-level set.
Revised: These lower bounds are easily derived from our geometric perspective as we show that a sampling distribution with a large radius gives an extremely low probability of intersection with the desired sub-level set.

Original: Therefore, while gradient-approximation algorithms can be accelerated to achieve a runtime that depends on the square-root of the condition number Q , gradient-less methods that rely on random sampling are likely unable to be accelerated according to our lower bound.
Revised: Therefore, while gradient-approximation algorithms can be accelerated to achieve a runtime that depends on the square-root of the condition number Q , gradient-less methods that rely on random sampling are likely unable to be accelerated according to our lower bound.

Original: However, we emphasize that monotone invariance allows these results to apply to a broader class of objective functions, beyond smooth and convex, so the results can be useful in practice despite the seemingly worse theoretical bounds.
Revised: However, we emphasize that monotone invariance allows these results to apply to a broader class of objective functions, beyond smooth and convex, so the results can be useful in practice despite the seemingly worse theoretical bounds.

Revised: Input: function: f : R n → R , T ∈

Revised: Z + : number iterations, x 0 : point,

Revised: D : search radius 1 Set K = log( R/r ) 2 for t = 0 , . .

Original: Theorem 12.
Revised: , T do Ball Sampling Trial i : for k = 0, ..., K do Set r i,k = 2 − k R .

Revised: Sample v i,k ∼ r

Revised: i,k D .

Revised: end

Revised: Update: x t +1 =

Revised: arg

Revised: min k (cid:110) f

Revised: ( y ) (cid:12)(cid:12)(cid:12) y

Revised: = x t , y = x t + v i,k (cid:111) 9 end 10 return x t

Revised: Theorem 12.

Original: Let y =
Revised: Let y =

Original: x
Revised: x

Original: + v , where v is a random sample from r D for some radius r > 0 and D is standard Gaussian or any rotationally symmetric distribution.
Revised: + v , where v is a random sample from r D for some radius r > 0 and D is standard Gaussian or any rotationally symmetric distribution.

Original: Then, there exist a region X with positive measure such that for any x ∈ X ,
Revised: Then, there exist a region X with positive measure such that for any x ∈ X ,

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 −1 poly ( nQ ) . do
Revised: with probability at least 1 −

Original: Ball Sampling i : do

Original: Set
Revised: [Equation]

Original: − k D .
Revised: ( nQ ) .

Original:  G RADIENTLESS A LGORITHMS
Revised:  G RADIENTLESS A LGORITHMS

Original: In this section, we present two algorithms that follow the same Gradientless Descent (GLD) template: GLD-Search and GLD-Fast, with the latter being an optimized version of the former when an upper bound on the condition number of a function is known.
Revised: In this section, we present two algorithms that follow the same Gradientless Descent (GLD) template: GLD-Search and GLD-Fast, with the latter being an optimized version of the former when an upper bound on the condition number of a function is known.

Original: For both algorithms, since they are monotone-invariant, we appeal to the previous section to derive fast convergence rates for any monotone transform of convex f
Revised: For both algorithms, since they are monotone-invariant, we appeal to the previous section to derive fast convergence rates for any monotone transform of convex f

Original: ( x ) with good condition number.
Revised: ( x ) with good condition number.

Original: We show the efﬁcacy of both algorithms experimentally in the Experiments section.
Revised: We show the efﬁcacy of both algorithms experimentally in the Experiments section.

Original: 4. G RADIENTLESS D ESCENT WITH B INARY S EARCH
Revised: 4. G RADIENTLESS D ESCENT WITH B INARY S EARCH

Original: Although the sampling distribution D is ﬁxed, we have a choice of radii for each iteration of the algorithm.
Revised: Although the sampling distribution D is ﬁxed, we have a choice of radii for each iteration of the algorithm.

Original: We can apply a binary search procedure to ensure progress.
Revised: We can apply a binary search procedure to ensure progress.

Original: The most straightforward version of our algorithm is thus with a naive binary sweep across an interval in [ r, R ] that is unchanged throughout the algorithm.
Revised: The most straightforward version of our algorithm is thus with a naive binary sweep across an interval in [ r, R ] that is unchanged throughout the algorithm.

Original: This allows us to give convergence guarantees without previous knowledge of the condition number at a cost of an extra factor of log( n/(cid:15) ) .
Revised: This allows us to give convergence guarantees without previous knowledge of the condition number at a cost of an extra factor of log( n/(cid:15) ) .

Original: Theorem 13.
Revised: Theorem 13.

Original: Let x 0 be any starting point and f a blackbox function with condition number Q .
Revised: Let x 0 be any starting point and f a blackbox function with condition number Q .

Original: Running Algorithm 1 with r = (cid:15) √ n , R = (cid:107)X(cid:107) and D = N (0 , I ) as a standard Gaussian returns a point x T such that (cid:107) x
Revised: Running Algorithm 1 with r = (cid:15) √ n , R = (cid:107)X(cid:107) and D = N (0 , I ) as a standard Gaussian returns a point x T such that (cid:107) x

Original: T − x ∗ (cid:107) ≤ 2 Q 3 / 2 (cid:15) after O ( nQ log( n (cid:107)X(cid:107) /(cid:15) ) 2 ) function evaluations with high probability.
Revised: T − x ∗ (cid:107) ≤ 2 Q 3 / 2 (cid:15) after O ( nQ log( n (cid:107)X(cid:107) /(cid:15) ) 2 ) function evaluations with high probability.

Original: Furthermore, if f ( x ) =
Revised: Furthermore, if f ( x ) =

Original: g ( P A x ) admits a low-rank structure with P A a rank k matrix, then we only require O ( kQ g ( A ) log( n (cid:107)X(cid:107) /(cid:15) ) 2 ) function evaluations to guarantee (cid:107) P A ( x T − x ∗ ) (cid:107) ≤ (cid:15) .
Revised: g ( P A x ) admits a low-rank structure with P A a rank k matrix, then we only require O ( kQ g ( A ) log( n (cid:107)X(cid:107) /(cid:15) ) 2 ) function evaluations to guarantee (cid:107) P A ( x T − x ∗ ) (cid:107) ≤ (cid:15) .

Original: This holds analogously even if f ( x ) =
Revised: This holds analogously even if f ( x ) =

Original: g ( P A x ) + h ( x ) is almost low-rank where | h | ≤ δ and (cid:15) >
Revised: g ( P A x ) + h ( x ) is almost low-rank where | h | ≤ δ and (cid:15) >

Original: 60 δkQ g ( A ) .
Revised: 60 δkQ g ( A ) .

Original: 4. G RADIENTLESS D ESCENT WITH F AST B INARY S EARCH
Revised: 4. G RADIENTLESS D ESCENT WITH F AST B INARY S EARCH

Original: GLD-Search (Algorithm 1) uses a naive lower and upper bound for the search radius (cid:107) x
Revised: GLD-Search (Algorithm 1) uses a naive lower and upper bound for the search radius (cid:107) x

Original: t − x ∗ (cid:107) , which incurs an extra factor of log(1 /(cid:15) ) in the runtime bound.
Revised: t − x ∗ (cid:107) , which incurs an extra factor of log(1 /(cid:15) ) in the runtime bound.

Original: In GLD-Fast, we remove this extra factor dependence on log(1 /(cid:15) ) by drastically reducing the range of the binary search.
Revised: In GLD-Fast, we remove this extra factor dependence on log(1 /(cid:15) ) by drastically reducing the range of the binary search.

Original: This is done by exploiting the assumption that f has a good condition number upper bound ˆ Q and by slowly halﬁng the diameter of the search space every few iterations since we expect x t → x ∗ as t → ∞ .
Revised: This is done by exploiting the assumption that f has a good condition number upper bound ˆ Q and by slowly halﬁng the diameter of the search space every few iterations since we expect x t → x ∗ as t → ∞ .

Original: Theorem 14.

Original: Let x 0 be any starting point and f a blackbox function with condition number upper bounded by Q .

Original: Running Algorithm 2 with suitable parameters returns a point x T such that f ( x T ) − f ( x ∗ ) ≤ (cid:15) after O ( nQ log 2 ( Q ) log( (cid:107)X(cid:107) /(cid:15) ))

Original: function evaluations with high probability.

Original: Furthermore, if f ( x ) =

Original: g ( P A x ) admits a low-rank structure with P A a rank k matrix, then we only require O ( kQ g ( A ) log( n ) log 2 ( Q g ( A )) log( (cid:107)X(cid:107) /(cid:15) ))

Original: function evaluations to guarantee (cid:107) P A ( x T − x ∗ ) (cid:107) ≤ (cid:15) .

Original: This holds analogously even if f ( x ) =

Original: g ( P A x ) + h ( x ) is almost low-rank where | h | ≤ δ and (cid:15) >

Original: 60 δkQ g ( A ) .

Original: Input: function f : R n → R , T ∈
Revised: Input: function f : R n → R , T ∈

Original: Z + : number of iterations, x 0 : starting point,
Revised: Z + : number of iterations, x 0 : starting point,

Original: D : sampling distribution, R : diameter of search space, Q : condition number bound 1 Set K = log(4 √ Q ) ,
Revised: D : sampling distribution, R : diameter of search space, Q : condition number bound 1 Set K = log(4 √ Q ) ,

Original: H = nQ log( Q ) 2 for t = 1 , . .
Revised: H = nQ log( Q ) 2 for t = 1 , . .

Original: , T do
Revised: , T do

Original: Set R =
Revised: Set R =

Original: R/ 2 when T ≡ 0 mod H (every H iterations).
Revised: R/ 2 when T ≡ 0 mod H (every H iterations).

Original: Ball Sampling Trial i : for k
Revised: Ball Sampling Trial i : for k

Original: = -K, ..., 0, ...,
Revised: = -K, ..., 0, ...,

Original: K do
Revised: K do

Original: Set r i,k = 2 − k R .
Revised: Set r i,k = 2 − k R .

Original: Sample v i,k ∼ r
Revised: Sample v i,k ∼ r

Original: i,k D .
Revised: i,k D .

Original: end
Revised: end

Original: Update: x t +1 = arg
Revised: Update: x t +1 = arg

Original: min i (cid:110) f
Revised: min i (cid:110) f

Original: ( y ) (cid:12)(cid:12)(cid:12) y
Revised: ( y ) (cid:12)(cid:12)(cid:12) y

Original: = x t , y = x t + v i (cid:111) 10 end 11 return x t
Revised: = x t , y = x t + v i (cid:111) 10 end 11 return x t

Revised: Theorem 14.

Revised: Let x 0 be any starting point and f a blackbox function with condition number upper bounded by Q .

Original: E XPERIMENTS
Revised: Running Algorithm 2 with suitable parameters returns a point x T such that f ( x T ) − f ( x ∗ ) ≤ (cid:15) after O ( nQ log 2 ( Q ) log( (cid:107)X(cid:107) /(cid:15) )) function evaluations with high probability. Furthermore, if f ( x ) = g ( P A x ) admits a low-rank structure with P A a rank k matrix, then we only require O ( kQ g ( A ) log( n ) log 2 ( Q g ( A )) log( (cid:107)X(cid:107) /(cid:15) ))

Revised: function evaluations to guarantee (cid:107) P A ( x T − x ∗ ) (cid:107) ≤ (cid:15) .

Revised: This holds analogously even if f ( x ) =

Revised: g ( P A x ) + h ( x ) is almost low-rank where | h | ≤ δ and (cid:15) >

Revised: 60 δkQ g ( A ) .

Revised:  E XPERIMENTS

Original: We tested GLD algorithms on a simple class of objective functions and compare it to Accelerated
Revised: We tested GLD algorithms on a simple class of objective functions and compare it to Accelerated

Original: Random Search (ARS) by Nesterov & Spokoiny (2011), which has linear convergence guarantees on strongly convex and strongly smooth functions.
Revised: Random Search (ARS) by Nesterov & Spokoiny (2011), which has linear convergence guarantees on strongly convex and strongly smooth functions.

Original: To our knowledge, ARS makes the weakest assumption among the zeroth-order algorithms that have linear convergence guarantees and perform only a constant order of operations per iteration.
Revised: To our knowledge, ARS makes the weakest assumption among the zeroth-order algorithms that have linear convergence guarantees and perform only a constant order of operations per iteration.

Original: Our main conclusion is that GLD-Fast is comparable to ARS and tends to achieve a reasonably low error much faster than ARS in high dimensions ( ≥ 50 ).
Revised: Our main conclusion is that GLD-Fast is comparable to ARS and tends to achieve a reasonably low error much faster than ARS in high dimensions ( ≥ 50 ).

Original: In low dimensions, GLD-Search is competitive with GLD-Fast and ARS though it requires no information about the function.
Revised: In low dimensions, GLD-Search is competitive with GLD-Fast and ARS though it requires no information about the function.

Original: We let H α,β,n ∈ R n × n be a diagonal matrix with its i -th diagonal equal to α + ( β − α )
Revised: We let H α,β,n ∈ R n × n be a diagonal matrix with its i -th diagonal equal to α + ( β − α )

Original: i − 1 n − 1 .
Revised: i − 1 n − 1 .

Original: In simple words, its diagonal elements form an evenly space sequence of numbers from α to β .
Revised: In simple words, its diagonal elements form an evenly space sequence of numbers from α to β .

Original: Our objective function is then f α,β,n : R n → R as f α,β,n ( x ) = 12 x (cid:62) H α,β,n x , which is α -strongly convex and β -strongly smooth.
Revised: Our objective function is then f α,β,n : R n → R as f α,β,n ( x ) = 12 x (cid:62) H α,β,n x , which is α -strongly convex and β -strongly smooth.

Original: We always use the same starting point x = 1 √ n (1 , . .
Revised: We always use the same starting point x = 1 √ n (1 , . .

Original: . , 1) , which requires (cid:107) X (cid:107) =
Revised: . , 1) , which requires (cid:107) X (cid:107) =

Original: √ Q for our algorithms.
Revised: √ Q for our algorithms.

Original: We plot the optimality gap f ( b t ) − f ( x ∗ ) against the number of function evaluations, where b t is the best point observed so far after t evaluations.
Revised: We plot the optimality gap f ( b t ) − f ( x ∗ ) against the number of function evaluations, where b t is the best point observed so far after t evaluations.

Original: Although all tested algorithms are stochastic, they have a low variance on the objective functions that we use; hence we average the results over 10 runs and omit the error bars in the plots.
Revised: Although all tested algorithms are stochastic, they have a low variance on the objective functions that we use; hence we average the results over 10 runs and omit the error bars in the plots.

Original: We ran experiments on f 1 , 8 ,n with imperfect curvature information ˆ α and ˆ β
Revised: We ran experiments on f 1 , 8 ,n with imperfect curvature information ˆ α and ˆ β

Original: (see Figure 3 in appendix).
Revised: (see Figure 3 in appendix).

Original: GLD-Search is independent of the condition number.
Revised: GLD-Search is independent of the condition number.

Original: GLD-Fast takes only one parameter, which is the upper bound on the condition number; if approximation factor is z , then we pass 8 z as the upper bound.
Revised: GLD-Fast takes only one parameter, which is the upper bound on the condition number; if approximation factor is z , then we pass 8 z as the upper bound.

Original: ARS requires both strong convexity and smoothness parameters.
Revised: ARS requires both strong convexity and smoothness parameters.

Original: We test three different distributions of the approximation error; when the approximation factor is z , then ARS-alpha gets ( α/z, β ) , ARS-beta gets ( α, zβ ), and ARS-even gets ( α/ √ z, √ zβ ) as input.
Revised: We test three different distributions of the approximation error; when the approximation factor is z , then ARS-alpha gets ( α/z, β ) , ARS-beta gets ( α, zβ ), and ARS-even gets ( α/ √ z, √ zβ ) as input.

Original: GLD-Fast is more robust and faster than ARS when the condition number is over-approximated.
Revised: GLD-Fast is more robust and faster than ARS when the condition number is over-approximated.

Original: When the condition number is underestimated, GLD-Fast still steadily converges.
Revised: When the condition number is underestimated, GLD-Fast still steadily converges.

Revised: [Figure]

Original: 5. M ONOTONE T RANSFORMATIONS
Revised: 5. M ONOTONE T RANSFORMATIONS

Original: In Figure 1, we ran experiments on f 1 , 8 ,n for different settings of dimensionality n, and its monotone transformation with g ( y ) =
Revised: In Figure [Figure] we ran experiments on f 1 , 8 ,n for different settings of dimensionality n

Original:  − exp( −√ y ) .
Revised: [Figure] and its monotone transformation with g ( y ) = − exp( −√ y ) .

Original: For this experiment, we assume a perfect oracle for the strong convexity and smoothness parameters of f .
Revised: For this experiment [Figure] we assume a perfect oracle for the strong convexity and smoothness parameters of f .

Original: The convergence of GLD is totally unaffected by the monotone transformation.
Revised: The convergence of GLD is totally unaffected by the monotone transformation.

Original: For the low-dimension cases of a transformed function (bottom half of the ﬁgure), we note that there are inﬂection points in the convergence curve of ARS.
Revised: For the low-dimension cases of a transformed function (bottom half of the ﬁgure), we note that there are inﬂection points in the convergence curve of ARS.

Original: This means that ARS initially struggles to gain momentum and then struggles to stop the momentum when it gets close to the optimum.
Revised: This means that ARS initially struggles to gain momentum and then struggles to stop the momentum when it gets close to the optimum.

Original: Another observation is that unlike ARS that needs to build up momentum, GLD-Fast starts from a large radius and therefore achieves a reasonably low error much faster than ARS, especially in higher dimensions.[Figure]
Revised: Another observation is that unlike ARS that needs to build up momentum, GLD-Fast starts from a large radius and therefore achieves a reasonably low error much faster than ARS, especially in higher dimensions.

Original: 5. BBOB B ENCHMARKS
Revised: 5. BBOB B ENCHMARKS

Original: To show that practicality of GLD on practical and non-convex settings, we also test GLD algorithms on a variety of BlackBox Optimization Benchmarking (BBOB) functions (Hansen et al., 2009).
Revised: To show that practicality of GLD on practical and non-convex settings, we also test GLD algorithms on a variety of BlackBox Optimization Benchmarking (BBOB) functions (Hansen et al., 2009).

Original: For each function, the optima is known and we use the log optimality gap as a measure of competance.
Revised: For each function, the optima is known and we use the log optimality gap as a measure of competance.

Original: Because each function can exhibit varying forms of non-smoothness and convexity, all algorithms are ran with a smoothness constant of 10 and a strong convexity constant of 0.1.
Revised: Because each function can exhibit varying forms of non-smoothness and convexity, all algorithms are ran with a smoothness constant of 10 and a strong convexity constant of 0.1.

Original: All other setup details are same as before, such as using a ﬁxed starting point.
Revised: All other setup details are same as before, such as using a ﬁxed starting point.

Original: The plots, given in Appendix C, underscore the superior performance of GLD algorithms on various
Revised: The plots, given in Appendix C, underscore the superior performance of GLD algorithms on various

Original: BBOB functions, demonstrating that GLD can successfully optimize a diverse set of functions even without explicit knowledge of condition number.
Revised: BBOB functions, demonstrating that GLD can successfully optimize a diverse set of functions even without explicit knowledge of condition number.

Original: We note that BBOB functions are far from convex and smooth, many exhibiting high conditioning, multi-modal valleys, and weak global structure.
Revised: We note that BBOB functions are far from convex and smooth, many exhibiting high conditioning, multi-modal valleys, and weak global structure.

Original: Due to our radius search produce, our algorithm appears more robust to non-ideal settings with non-convexity and ill conditioning.
Revised: Due to our radius search produce, our algorithm appears more robust to non-ideal settings with non-convexity and ill conditioning.

Original: As expected, we note that GLD-Fast tend to outperform GLDSearch, especially as the dimension increases, matching our theoretical understanding of GLD.
Revised: As expected, we note that GLD-Fast tend to outperform GLDSearch, especially as the dimension increases, matching our theoretical understanding of GLD.

Original: 5. M UJOCO C ONTROL B ENCHMARKS AND A FFINE T RANSFORMATIONS
Revised: 5. M UJOCO C ONTROL B ENCHMARKS AND A FFINE T RANSFORMATIONS

Original: We also ran experiments on the Mujoco benchmarks with varying architectures, both linear and nonlinear.
Revised: We also ran experiments on the Mujoco benchmarks with varying architectures, both linear and nonlinear.

Original: This demonstrates the viability of our approach even in the non-convex, high dimensional setting.
Revised: This demonstrates the viability of our approach even in the non-convex, high dimensional setting.

Original: We note that however, unlike e.g. ES which uses all queries to form a gradient direction, our algorithm removes queries which produce less reward than using the current arg-max, which can be an information handicap.
Revised: We note that however, unlike e.g. ES which uses all queries to form a gradient direction, our algorithm removes queries which produce less reward than using the current arg-max, which can be an information handicap.

Original: Nevertheless, we see that our algorithm still achieves competitive performance on the maximum reward.
Revised: Nevertheless, we see that our algorithm still achieves competitive performance on the maximum reward.

Original: We used a horizon of 1000 for all experiments.
Revised: We used a horizon of 1000 for all experiments.

Original: We further tested the afﬁne invariance of GLD on the policy parameters from using Gaussian ball sampling, under the HalfCheetah benchmark by projecting the state s of the MDP with linear policy to a higher dimensional state Ws , using a matrix multiplication with an orthonormal W .
Revised: We further tested the afﬁne invariance of GLD on the policy parameters from using Gaussian ball sampling, under the HalfCheetah benchmark by projecting the state s of the MDP with linear policy to a higher dimensional state Ws , using a matrix multiplication with an orthonormal W .

Original: Specifically, in this setting, for a linear policy parametrized by matrix K , the objective function is thus
Revised: Specifically, in this setting, for a linear policy parametrized by matrix K , the objective function is thus

Original: J ( KW ) where π K ( Ws ) = KWs .
Revised: J ( KW ) where π K ( Ws ) = KWs .

Original: Note that when projecting into a high dimension, there is a slowdown factor of log d new d old where d new , d old are the new high dimension and previous base dimension, respectively, due to the binary search in our algorithm on a higher dimensional space.
Revised: Note that when projecting into a high dimension, there is a slowdown factor of log d new d old where d new , d old are the new high dimension and previous base dimension, respectively, due to the binary search in our algorithm on a higher dimensional space.

Original: For our HalfCheetah case, we projected the 17 base dimension to a 200-length dimension, which suggests that the slowdown factor is a factor log 20017 ≈ 3 . 5 .
Revised: For our HalfCheetah case, we projected the 17 base dimension to a 200-length dimension, which suggests that the slowdown factor is a factor log 20017 ≈ 3 . 5 .

Original: This can be shown in our plots in the appendix (Figure 15).
Revised: This can be shown in our plots in the appendix (Figure 15).

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced GLD, a robust zeroth-order optimization algorithm that is simple, efﬁcient [Table] and we show strong theoretical convergence bounds via our novel geometric analysis.
Revised: We introduced GLD, a robust zeroth-order optimization algorithm that is simple, efﬁcient, and we show strong theoretical convergence bounds via our novel geometric analysis.

Original: As demonstrated by our experiments on BBOB and MuJoCo benchmarks, GLD performs very robustly even in the non-convex setting and its monotone and afﬁne invariance properties give theoretical insight on its practical efﬁciency.
Revised: As demonstrated by our experiments on BBOB and MuJoCo benchmarks, GLD performs very robustly even in the non-convex setting and its monotone and afﬁne invariance properties give theoretical insight on its practical efﬁciency.

Original: GLD is very ﬂexible and allows easy modiﬁcations.
Revised: GLD is very ﬂexible and allows easy modiﬁcations.

Original: For example, it could use momentum terms to keep moving in the same direction that improved the objective [Table] or sample from adaptively chosen ellipsoids similarly to adaptive gradient methods.
Revised: For example, it could use momentum terms to keep moving in the same direction that improved the objective, or sample from adaptively chosen ellipsoids similarly to adaptive gradient methods.

Original: (Duchi et al., 2011; McMahan & Streeter, 2010).
Revised: (Duchi et al., 2011; McMahan & Streeter, 2010).

Original: Just as one may decay or adaptively vary learning rates for gradient descent, one might use a similar change the distribution from which the ball-sampling radii are chosen, perhaps shrinking the minimum radius as the algorithm progresses, or concentrating more probability mass on smaller radii.
Revised: Just as one may decay or adaptively vary learning rates for gradient descent, one might use a similar change the distribution from which the ball-sampling radii are chosen, perhaps shrinking the minimum radius as the algorithm progresses, or concentrating more probability mass on smaller radii.

Original: Likewise, GLD could be combined with random restarts or other restart policies developed for gradient descent.
Revised: Likewise, GLD could be combined with random restarts or other restart policies developed for gradient descent.

Original: Analogously to adaptive per–coordinate learning rates Duchi et al.
Revised: Analogously to adaptive per–coordinate learning rates Duchi et al.

Original: ; McMahan & Streeter (2010), one could adaptively change the shape of the balls being sampled into ellipsoids with various length-scale factors.
Revised: ; McMahan & Streeter (2010), one could adaptively change the shape of the balls being sampled into ellipsoids with various length-scale factors.

Original: Arbitrary combinations of the above variants are also possible
Revised: Arbitrary combinations of the above variants are also possible.

Original: A P ROOFS OF S ECTION
Revised: A P ROOFS OF S ECTION

Original: Lemma 15.
Revised: Lemma 15.

Original: If h has condition number Q , then for all x ∈ X , there is a ball of radius Q − 1 (cid:107) x − x ∗ (cid:107) that is tangent at x and inside the sublevel set L ↓ x ( h ) .
Revised: If h has condition number Q , then for all x ∈ X , there is a ball of radius Q − 1 (cid:107) x − x ∗ (cid:107) that is tangent at x and inside the sublevel set L ↓ x ( h ) .

Original: Proof.
Revised: Proof.

Original: Write h
Revised: Write h

Original: = g ◦ f such that f is α -strongly convex and β -smooth for some β
Revised: = g ◦ f such that f is α -strongly convex and β -smooth for some β

Original: = Qα and g is monotonically increasing.
Revised: = Qα and g is monotonically increasing.

Original: From the smoothness assumption, we have for any s ,
Revised: From the smoothness assumption, we have for any s ,

Original: [Equation]
Revised: [Equation]

Original: Consider the ball B = B ( x − 1 β ∇ f ( x ) , 1 β (cid:107)∇ f ( x ) (cid:107) ) .
Revised: Consider the ball B = B ( x − 1 β ∇ f ( x ) , 1 β (cid:107)∇ f ( x ) (cid:107) ) .

Original: For any y ∈ B , the above inequality implies f ( y ) ≤ f ( x ) .
Revised: For any y ∈ B , the above inequality implies f ( y ) ≤ f ( x ) .

Original: Hence, when we apply g on both sides, we still have h ( y ) ≤
Revised: Hence, when we apply g on both sides, we still have h ( y ) ≤

Original: h ( x ) for all y ∈ B .
Revised: h ( x ) for all y ∈ B .

Original: Therefore, B ⊆ L ↓ h ( y ) .
Revised: Therefore, B ⊆ L ↓ h ( y ) .

Original: By strong convexity, (cid:107)∇ f ( x ) (cid:107) ≥ α (cid:107) x − x ∗ (cid:107) .
Revised: By strong convexity, (cid:107)∇ f ( x ) (cid:107) ≥ α (cid:107) x − x ∗ (cid:107) .

Original: It follows that the radius of B is at least αβ (cid:107) x − x ∗ (cid:107) .
Revised: It follows that the radius of B is at least αβ (cid:107) x − x ∗ (cid:107) .

Original: Proof of Lemma 8.
Revised: Proof of Lemma 8.

Original: Without loss of generality, consider the unit distance case where (cid:96) = 1 .
Revised: Without loss of generality, consider the unit distance case where (cid:96) = 1 .

Original: Furthermore, it sufﬁces to prove for the smallest possible radius r 2 = 1 − 14 n .
Revised: Furthermore, it sufﬁces to prove for the smallest possible radius r 2 = 1 − 14 n .

Original: Since | r 1 − r 2 | ≤ (cid:96) ≤ r 1 + r 2 , the intersection B 1 ∩ B 2 is composed of two hyperspherical caps glued end to end.
Revised: Since | r 1 − r 2 | ≤ (cid:96) ≤ r 1 + r 2 , the intersection B 1 ∩ B 2 is composed of two hyperspherical caps glued end to end.

Original: We lower bound vol ( B 1 ∩ B 2 ) by the volume of the cap C 1 of B 1 that is contained in the intersection.
Revised: We lower bound vol ( B 1 ∩ B 2 ) by the volume of the cap C 1 of B 1 that is contained in the intersection.

Original: Consider the triangle with sides r 1 , r 2 and (cid:96) .
Revised: Consider the triangle with sides r 1 , r 2 and (cid:96) .

Original: From classic geometry, the height of
Revised: From classic geometry, the height of

Original: C 1 is
Revised: C 1 is

Original: [Equation]
Revised: [Equation]

Original: The volume of a spherical cap is Li (2011),
Revised: The volume of a spherical cap is Li (2011),

Original: [Equation]
Revised: [Equation]

Original: where I is the regularized incomplete beta function deﬁned as
Revised: where I is the regularized incomplete beta function deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where x ∈ [0 , 1] and a, b ∈ (0 , ∞ ) .
Revised: where x ∈ [0 , 1] and a, b ∈ (0 , ∞ ) .

Original: Note that for any ﬁxed a and b , I x ( a, b ) is increasing in x .
Revised: Note that for any ﬁxed a and b , I x ( a, b ) is increasing in x .

Original: Hence, in order to obtain a lower bound on vol ( C 1 ) , we want to lower bound 1 − c 21 r 21 or equivalently, upper bound c 21 r 21 .
Revised: Hence, in order to obtain a lower bound on vol ( C 1 ) , we want to lower bound 1 − c 21 r 21 or equivalently, upper bound c 21 r 21 .

Original: Write r 1 = α 2 √ n for some α ∈ [1 , 2] .
Revised: Write r 1 = α 2 √ n for some α ∈ [1 , 2] .

Original: From Eq.
Revised: From Eq.

Original: [Equation]
Revised: [Equation]

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Since g ( α ) = 8 α + 

Revised: Since g ( α ) = 8 α +

Original: α is convex in [1 , 2] , g ( α ) ≤ max( g (1) , g (2))
Revised: α is convex in [1 , 2] , g ( α ) ≤ max( g (1) , g (2))

Original: =
Revised: =

Original: It follows that c 1 r 1 ≤ 1 16 √ n (cid:0) 12 − 1 n (cid:1) ≤ 3 4 √ n .
Revised: It follows that c 1 r 1 ≤ 1 16 √ n (cid:0) 12 − 1 n (cid:1) ≤ 3 4 √ n .

Original: So, 1 − c 21 r 21 ≥ 1 − 916 n .
Revised: So, 1 − c 21 r 21 ≥ 1 − 916 n .

Original: To complete the proof, note that
Revised: To complete the proof, note that

Original: V n := I 1 − 916 n ( n +12 , 12 ) is increasing in n , and V 1 = 14 .
Revised: V n := I 1 − 916 n ( n +12 , 12 ) is increasing in n , and V 1 = 14 .

Original: As n goes to inﬁnity, this value converges to 1 as B 1 ⊂ B 2 .
Revised: As n goes to inﬁnity, this value converges to 1 as B 1 ⊂ B 2 .

Original: Proof of Lemma 7.
Revised: Proof of Lemma 7.

Original: Let ν = 1 5 nQ .
Revised: Let ν = 1 5 nQ .

Original: Let q
Revised: Let q

Original: =
Revised: =

Original: (1 − ν ) x + νx ∗ .
Revised: (1 − ν ) x + νx ∗ .

Original: Let B q = B ( c q , r q ) be a ball that has q on its surface, lies inside L ↓ q , and has radius r q = Q − 1 (cid:107) x − x ∗ (cid:107) .
Revised: Let B q = B ( c q , r q ) be a ball that has q on its surface, lies inside L ↓ q , and has radius r q = Q − 1 (cid:107) x − x ∗ (cid:107) .

Original: Lemma 15 guarantees its existence.
Revised: Lemma 15 guarantees its existence.

Original: Suppose that
Revised: Suppose that

Original: [Equation]
Revised: [Equation]

Original: and that a random sample y from B x belongs to B q , which happens with probability at least 14 .
Revised: and that a random sample y from B x belongs to B q , which happens with probability at least 14 .

Original: Then, our guarantee follows by
Revised: Then, our guarantee follows by

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst line follows from Lemma 15 and second line from convexity of f .
Revised: where the ﬁrst line follows from Lemma 15 and second line from convexity of f .

Original: Therefore, it now sufﬁces to prove Eq. 2.
Revised: Therefore, it now sufﬁces to prove Eq. 2.

Original: To do so, we will apply Lemma 8 after showing that the radius of B x and B q are in the proper ranges.
Revised: To do so, we will apply Lemma 8 after showing that the radius of B x and B q are in the proper ranges.

Original: Let (cid:96) =
Revised: Let (cid:96) =

Original: (cid:107) x − c q (cid:107) and note that
Revised: (cid:107) x − c q (cid:107) and note that

Original: [Equation]
Revised: [Equation]

Original: Since x is outside of B q , we also have
Revised: Since x is outside of B q , we also have

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: In the log 2 space, our choice of k 1 is equivalent to starting from log 2 C 1 and sweeping through the range [log 2 C 1 , log 2 C 2 ] at the interval of size 1.
Revised: In the log 2 space, our choice of k 1 is equivalent to starting from log 2 C 1 and sweeping through the range [log 2 C 1 , log 2 C 2 ] at the interval of size 1.

Original: This is guaranteed to ﬁnd a point between (cid:96) 2 and (cid:96) , which is also an interval of size 1.
Revised: This is guaranteed to ﬁnd a point between (cid:96) 2 and (cid:96) , which is also an interval of size 1.

Original: Therefore, there exists a k 1 satisfying the theorem statement, and similarly, we can prove the existence of k 2 .
Revised: Therefore, there exists a k 1 satisfying the theorem statement, and similarly, we can prove the existence of k 2 .

Original: Finally, it remains to show that r q ≥ (1 − 1 / (4 n )) (cid:96).
Revised: Finally, it remains to show that r q ≥ (1 − 1 / (4 n )) (cid:96).

Original: From Eq.
Revised: From Eq.

Original: (3), it sufﬁces to show that (cid:107) x − q (cid:107) ≤ (cid:96) 4 n or equivalently ν (cid:107) x − x ∗ (cid:107) ≤ (cid:96) 4 n .
Revised: (3), it sufﬁces to show that (cid:107) x − q (cid:107) ≤ (cid:96) 4 n or equivalently ν (cid:107) x − x ∗ (cid:107) ≤ (cid:96) 4 n .

Original: From Eq.
Revised: From Eq.

Original: [Equation]
Revised: [Equation]

Original: For any Q, n ≥ 1 , 1 − ν ≥ 45 .
Revised: For any Q, n ≥ 1 , 1 − ν ≥ 45 .

Original: So,
Revised: So,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and the proof is complete.
Revised: and the proof is complete.

Original: Proof of Lemma 9.
Revised: Proof of Lemma 9.

Original: Without loss of generality, let (cid:96) = 1 and B 2 is centered at the origin with radius r 

Revised: Without loss of generality, let (cid:96) = 1 and B 2 is centered at the origin with radius r

Original: and B 1 is centered at e 1 = (1 , 0 , ..., 0) .
Revised: and B 1 is centered at e 1 = (1 , 0 , ..., 0) .

Original: Then, we simply want to show that
Revised: Then, we simply want to show that

Original: [Equation]
Revised: [Equation]

Original: By Markov’s inequality, we see that (cid:80) ni
Revised: By Markov’s inequality, we see that (cid:80) ni

Original: =2 X 2 i ≤ 2 r 21 = 2 /n with probability at most 1 / 2 .
Revised: =2 X 2 i ≤ 2 r 21 = 2 /n with probability at most 1 / 2 .

Original: And since X 1 is independent and r 2 ≥ 1 − 1 /n , it sufﬁces to show that
Revised: And since X 1 is independent and r 2 ≥ 1 − 1 /n , it sufﬁces to show that

Original: [Equation]
Revised: [Equation]

Original: Since X 1 has standard deviation at least r 1 / √ n ≥ 1 / (2 n ) , we see that the probability of deviating at least a few standard deviation below is at least a constant.
Revised: Since X 1 has standard deviation at least r 1 / √ n ≥ 1 / (2 n ) , we see that the probability of deviating at least a few standard deviation below is at least a constant.

Original: Proof of Theorem 10.
Revised: Proof of Theorem 10.

Original: We can consider the projection of all points onto the column space of A and since the Gaussian sampling process is preserved, our proof follows from applying Theorem 7 restricted onto the k -dimensional subspace and using Lemma 9 in place of Lemma 8.
Revised: We can consider the projection of all points onto the column space of A and since the Gaussian sampling process is preserved, our proof follows from applying Theorem 7 restricted onto the k -dimensional subspace and using Lemma 9 in place of Lemma 8.

Original: Proof of Theorem 11.
Revised: Proof of Theorem 11.

Original: By the boundedness of h , since f ( x ) − f ( x ∗ ) ≥ 60 δkQ g ( A ) , we see that g ( P A x ) − g
Revised: By the boundedness of h , since f ( x ) − f ( x ∗ ) ≥ 60 δkQ g ( A ) , we see that g ( P A x ) − g

Original: ( x ∗ ) ≥ 60 δkQ g ( A ) − 2 δ > 0 .
Revised: ( x ∗ ) ≥ 60 δkQ g ( A ) − 2 δ > 0 .

Original: By Lemma 9, we see that if we sample from a Gaussian distribution y ∼ N ( x, r 2 k I ) , then if z ∗ is the minimum of g ( x ) restricted to the column space of A , then
Revised: By Lemma 9, we see that if we sample from a Gaussian distribution y ∼ N ( x, r 2 k I ) , then if z ∗ is the minimum of g ( x ) restricted to the column space of A , then

Original: [Equation]
Revised: [Equation]

Original: with constant probability.
Revised: with constant probability.

Original: By boundedness on h , we know that h ( y ) ≤ h ( x )+2 δ .
Revised: By boundedness on h , we know that h ( y ) ≤ h ( x )+2 δ .

Original: Furthermore, this also implies that g ( P A x ∗ ) ≤ g ( z ∗ ) + 2 δ .
Revised: Furthermore, this also implies that g ( P A x ∗ ) ≤ g ( z ∗ ) + 2 δ .

Original: Therefore, we know that the decrease is at least
Revised: Therefore, we know that the decrease is at least

Original: [Equation]
Revised: [Equation]

Original: 1 − 5 kQ g ( A ) (cid:19) + 6 δ
Revised: 1 − 5 kQ g ( A ) (cid:19) + 6 δ

Original: Since f ( x ) − f ( x ∗ ) ≥ 10 δkQ g ( A ) , we conclude that ( f ( x ) − f ( x ∗ )) (cid:16) 1 − 1 5 kQ g ( A ) (cid:17) + 6 δ ≤ ( f ( x ) − f ( x ∗ )) (cid:16) 1 − 1 10 kQ g ( A ) (cid:17) and our proof is complete.
Revised: Since f ( x ) − f ( x ∗ ) ≥ 10 δkQ g ( A ) , we conclude that ( f ( x ) − f ( x ∗ )) (cid:16) 1 − 1 5 kQ g ( A ) (cid:17) + 6 δ ≤ ( f ( x ) − f ( x ∗ )) (cid:16) 1 − 1 10 kQ g ( A ) (cid:17) and our proof is complete.

Original: Proof of Theorem 12.
Revised: Proof of Theorem 12.

Original: Our main proof strategy is to show that progress can only be made with a radius size of O ( (cid:112) log( nQ ) / ( nQ )) ; larger radii cannot ﬁnd descent directions with high probability.
Revised: Our main proof strategy is to show that progress can only be made with a radius size of O ( (cid:112) log( nQ ) / ( nQ )) ; larger radii cannot ﬁnd descent directions with high probability.

Original: Consider a simple ellipsoid function f ( x ) =
Revised: Consider a simple ellipsoid function f ( x ) =

Original: x (cid:62) Dx , where D is a diagonal matrix and D 11 ≤
Revised: x (cid:62) Dx , where D is a diagonal matrix and D 11 ≤

Original: D 22 ≤ ... ≤ D nn , where WLOG we let D 11 = 1 and D ii =
Revised: D 22 ≤ ... ≤ D nn , where WLOG we let D 11 = 1 and D ii =

Original: Q for i > 1 .
Revised: Q for i > 1 .

Original: The optima is x ∗ =with f ( x ∗ ) = 0 .
Revised: The optima is x ∗ =with f ( x ∗ ) = 0 .

Original: Consider the region X = { x = ( x 1 , x 2 , ..., x n ) | 1 ≥ x 1 ≥ 0 .
Revised: Consider the region X = { x = ( x 1 , x 2 , ..., x n ) | 1 ≥ x 1 ≥ 0 .

Original: 9 , | x i | ≤ 0 .
Revised: 9 , | x i | ≤ 0 .

Original: 1 / ( Q √ n ) } .
Revised: 1 / ( Q √ n ) } .

Original: Then, if we let v ∼ N (0 , I ) be a standard Gaussian vector, then for some radius r , we see that the probability of ﬁnding a descent direction is:
Revised: Then, if we let v ∼ N (0 , I ) be a standard Gaussian vector, then for some radius r , we see that the probability of ﬁnding a descent direction is:

Original: [Equation]
Revised: [Equation]

Original: By standard concentration bounds for sub-exponential variables, we have
Revised: By standard concentration bounds for sub-exponential variables, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, with exponentially high probability, (cid:80) i> 1 X 2 i ≥ n/ 2 .
Revised: Therefore, with exponentially high probability, (cid:80) i> 1 X 2 i ≥ n/ 2 .

Original: Also, since | x i | ≤ 0 .
Revised: Also, since | x i | ≤ 0 .

Original: 1 / ( Q √ n ) ,
Revised: 1 / ( Q √ n ) ,

Original: Chernoff bounds give:
Revised: Chernoff bounds give:

Original: [Equation]
Revised: [Equation]

Original: Therefore, with probability at least 1 − 1 / ( nQ ) 3 , | (cid:80) i> 1 v i X i | ≤ (cid:112) log( nQ ) /Q .
Revised: Therefore, with probability at least 1 − 1 / ( nQ ) 3 , | (cid:80) i> 1 v i X i | ≤ (cid:112) log( nQ ) /Q .

Original: If Qrn ≥ Ω( (cid:112) log( nQ )) , then we have
Revised: If Qrn ≥ Ω( (cid:112) log( nQ )) , then we have

Original: [Equation]
Revised: [Equation]

Original: We conclude that the probability of descent is upper bounded by Pr (cid:104) v [Equation] X [Equation]
Revised: We conclude that the probability of descent is upper bounded by Pr (cid:104) v 1 X 1 ≤ − Ω( (cid:112) log( nQ ))

Original: ≤ − Ω

Original: [Equation]

Original: log( nQ ))

Original: (cid:105) .
Revised: (cid:105) .

Original: This probability is exactly Φ( − l ) , where Φ is the cumulative density of a standard normal and l = Ω( (cid:112) log( nQ )) .
Revised: This probability is exactly Φ( − l ) , where Φ is the cumulative density of a standard normal and l = Ω( (cid:112) log( nQ )) .

Original: By a naive upper bound, we see that
Revised: By a naive upper bound, we see that

Original: [Equation]
Revised: [Equation]

Original: Since l = Ω( (cid:112) log( nQ )) , we conclude thatTherefore, we conclude that with probability at least 1 − 1 /poly ( nQ ) , we have f ( y ) − f ( x ∗ ) ≥ f ( x ) − f ( x ∗ ) .
Revised: Since l = Ω( (cid:112) log( nQ )) , we conclude that with probability at least 1 − 1 /poly ( nQ ) , we have f ( y ) − f ( x ∗ ) ≥ f ( x ) − f ( x ∗ ) .

Original: Otherwise, we are in the case that Qrn ≤ O ( (cid:112) log( nQ )) .
Revised: Otherwise, we are in the case that Qrn ≤ O ( (cid:112) log( nQ )) .

Original: Arguing simiarly as before, with high probability, our objective function and each coordinate can change by at most O ( (cid:112) log( nQ ) /
Revised: Arguing simiarly as before, with high probability, our objective function and each coordinate can change by at most O ( (cid:112) log( nQ ) /

Original: ( Qn )) .
Revised: ( Qn )) .

Original: Next, we extend our proof to any symmetric distribution D .
Revised: Next, we extend our proof to any symmetric distribution D .

Original: Since D is rotationally symmetric, if we parametrize v = ( r, θ ) is polar-coordinates, then the p.d.f.
Revised: Since D is rotationally symmetric, if we parametrize v = ( r, θ ) is polar-coordinates, then the p.d.f.

Original: of any scaling of D must take the form p ( v ) =
Revised: of any scaling of D must take the form p ( v ) =

Original: p r ( r ) u
Revised: p r ( r ) u

Original: ( θ ) , where u ( θ ) induces the uniform distribution over the unit sphere.
Revised: ( θ ) , where u ( θ ) induces the uniform distribution over the unit sphere.

Original: Therefore, if
Revised: Therefore, if

Original: Y is a random variable that follows D , then we may write Y = Rv/ (cid:107) v (cid:107) , where R is a random scalar with p.d.f p r ( r ) and v is a standard Gaussian vector and R, X are independent.
Revised: Y is a random variable that follows D , then we may write Y = Rv/ (cid:107) v (cid:107) , where R is a random scalar with p.d.f p r ( r ) and v is a standard Gaussian vector and R, X are independent.

Original: As previously argued, (cid:107) v (cid:107) ∈ [0 . 5 n, 1 . 5 n ] with exponentially high probability.
Revised: As previously argued, (cid:107) v (cid:107) ∈ [0 . 5 n, 1 . 5 n ] with exponentially high probability.

Original: Therefore, if R ≥ Ω( (cid:112) log( nQ ) /Q ) , the same arguments will imply that Y is a descent direction with polynomially small probability.
Revised: Therefore, if R ≥ Ω( (cid:112) log( nQ ) /Q ) , the same arguments will imply that Y is a descent direction with polynomially small probability.

Original: Thus, when Y is a descent direction, it must be that R ≤ Ω( (cid:112) log( nQ ) /Q ) and as argued previously, our lower bound follows similarly.
Revised: Thus, when Y is a descent direction, it must be that R ≤ Ω( (cid:112) log( nQ ) /Q ) and as argued previously, our lower bound follows similarly.

Original: B P ROOFS OF S ECTION
Revised: B P ROOFS OF S ECTION

Original: Proof of Theorem 13.
Revised: Proof of Theorem 13.

Original: By the Gaussian version of Theorem 7 (full rank version of Theorem 10), as long as our binary search sweeps between minimum search radius r ≤ [Equation] and maximum search radius of the diameter of the whole space R = (cid:107)X(cid:107) , the objective value will decrease multiplicatively by 1 − 1 5 nQ in each iteration with constant probability.
Revised: By the Gaussian version of Theorem 7 (full rank version of Theorem 10), as long as our binary search sweeps between minimum search radius r ≤ 3 5 Q √ n (cid:107) x − x ∗ (cid:107) and maximum search radius of the diameter of the whole space R = (cid:107)X(cid:107) , the objective value will decrease multiplicatively by 1 − 1 5 nQ in each iteration with constant probability.

Original: Therefore, if (cid:107) x t − x ∗ (cid:107) ≥ 2 Q(cid:15) and we set r = (cid:15) √ n and R = (cid:107)X(cid:107) , then with high probability, we expect f ( x T ) − f ( x ∗ ) ≤ βQ 2 (cid:15) 2 after T
Revised: Therefore, if (cid:107) x t − x ∗ (cid:107) ≥ 2 Q(cid:15) and we set r = (cid:15) √ n and R = (cid:107)X(cid:107) , then with high probability, we expect f ( x T ) − f ( x ∗ ) ≤ βQ 2 (cid:15) 2 after T

Original: =
Revised: =

Original: O
Revised: O

Original: ( nQ log( (cid:107)X(cid:107) / ( Q(cid:15) )))
Revised: ( nQ log( (cid:107)X(cid:107) / ( Q(cid:15) )))

Original: iterations, where we note that F = f ( x 0 )
Revised: iterations, where we note that F = f ( x 0 )

Original: − f ( x ∗ ) ≤ β (cid:107)X(cid:107) 2 by smoothness.
Revised: − f ( x ∗ ) ≤ β (cid:107)X(cid:107) 2 by smoothness.

Original: Otherwise, if there exists some x t such that (cid:107) x
Revised: Otherwise, if there exists some x t such that (cid:107) x

Original: t − x ∗ (cid:107) ≤ 2 Q(cid:15) , then f ( x T ) − f ( x ∗ ) ≤ f ( x t ) − f ( x ∗ ) ≤ 4 βQ 2 (cid:15) 2 .
Revised: t − x ∗ (cid:107) ≤ 2 Q(cid:15) , then f ( x T ) − f ( x ∗ ) ≤ f ( x t ) − f ( x ∗ ) ≤ 4 βQ 2 (cid:15) 2 .

Original: Therefore, by strong convexity, we conclude that in either case, (cid:107) x T − x ∗ (cid:107) ≤ 2 Q 3 / 2 (cid:15) .
Revised: Therefore, by strong convexity, we conclude that in either case, (cid:107) x T − x ∗ (cid:107) ≤ 2 Q 3 / 2 (cid:15) .

Original: Finally note that each iteration uses a binary search that requires
Revised: Finally note that each iteration uses a binary search that requires

Original: O (log( R/r ))
Revised: O (log( R/r ))

Original: =
Revised: =

Original: O (log( n (cid:107)X(cid:107) /(cid:15) ))
Revised: O (log( n (cid:107)X(cid:107) /(cid:15) ))

Original: function evaluations.
Revised: function evaluations.

Original: Therefore, by combining these bounds, we derive our result.
Revised: Therefore, by combining these bounds, we derive our result.

Original: The low-rank result follows from applying Theorem 10 and Theorem 11 instead.
Revised: The low-rank result follows from applying Theorem 10 and Theorem 11 instead.

Original: Proof of Theorem 14.
Revised: Proof of Theorem 14.

Original: Let
Revised: Let

Original: H =
Revised: H =

Original: O ( nQ log( Q )) be the number of iterations between successive radius halving and we initialize R = (cid:107)X(cid:107) and half R every H iterations.
Revised: O ( nQ log( Q )) be the number of iterations between successive radius halving and we initialize R = (cid:107)X(cid:107) and half R every H iterations.

Original: We call the iterations between two halving steps an epoch.
Revised: We call the iterations between two halving steps an epoch.

Original: We claim that (cid:107) x i − x 0 (cid:107) ≤ R for all iterations and proceed with induction on the epoch number.
Revised: We claim that (cid:107) x i − x 0 (cid:107) ≤ R for all iterations and proceed with induction on the epoch number.

Original: The base case is trivial.
Revised: The base case is trivial.

Original: Assume that (cid:107) x
Revised: Assume that (cid:107) x

Original: i − x 0 (cid:107) ≤ R for all iterations in the previous epoch and let iteration i s be the start of the epoch and iteration i s + H be the end of the epoch.
Revised: i − x 0 (cid:107) ≤ R for all iterations in the previous epoch and let iteration i s be the start of the epoch and iteration i s + H be the end of the epoch.

Original: Then, since (cid:107) x i s − x ∗ (cid:107) ≤ R , we see that f ( x i s ) − f ( x ∗ ) ≤ βR 2 by smoothness.
Revised: Then, since (cid:107) x i s − x ∗ (cid:107) ≤ R , we see that f ( x i s ) − f ( x ∗ ) ≤ βR 2 by smoothness.

Original: If R 4 √ Q ≤ (cid:107) x i − x ∗ (cid:107) ≤ 4 √ QR for all i in the previous epoch, then by the Gaussian version of Theorem 7 (Theorem 10), since we do a binary sweep from R 4 Q to 4 √ QR , we can choose D accordingly so that we are guaranteed that our objective value will decrease multiplicatively by 1 − 1 5 nQ with constant probability at a cost of O (log( Q ))
Revised: If R 4 √ Q ≤ (cid:107) x i − x ∗ (cid:107) ≤ 4 √ QR for all i in the previous epoch, then by the Gaussian version of Theorem 7 (Theorem 10), since we do a binary sweep from R 4 Q to 4 √ QR , we can choose D accordingly so that we are guaranteed that our objective value will decrease multiplicatively by 1 − 1 5 nQ with constant probability at a cost of O (log( Q ))

Original: function evaluations per iteration.
Revised: function evaluations per iteration.

Original: This implies that with high probability, after O ( nQ log( Q )) iterations, we conclude
Revised: This implies that with high probability, after O ( nQ log( Q )) iterations, we conclude

Original: [Equation]
Revised: [Equation]

Original: Otherwise, there exists some 1 ≤ j ≤ H such that (cid:107) x
Revised: Otherwise, there exists some 1 ≤ j ≤ H such that (cid:107) x

Original: i s + j − x ∗ (cid:107) ≥ 4 √ QR or (cid:107)x i
Revised: i s + j − x ∗ (cid:107) ≥ 4 √ QR or (cid:107)

Original: [Equation] + j [Equation]
Revised: x i s + j − x ∗ (cid:107) ≤ R 4 √ Q .

Original: If it is the former, then by strong convexity, f ( x i s + j ) − f ( x ∗ ) ≥ α (cid:107) x i s + j − x ∗ (cid:107) 2 ≥ 2 βR 2 , which contradicts the fact that f ( x i s ) − f ( x ∗ ) ≤ βR 2 by smoothness.
Revised: If it is the former, then by strong convexity, f ( x i s + j ) − f ( x ∗ ) ≥ α (cid:107) x i s + j − x ∗ (cid:107) 2 ≥ 2 βR 2 , which contradicts the fact that f ( x i s ) − f ( x ∗ ) ≤ βR 2 by smoothness.

Original: If it is the latter, then by smoothness, we reach the same conclusion:
Revised: If it is the latter, then by smoothness, we reach the same conclusion:

Original: [Equation]
Revised: [Equation]

Original: Therefore, by strong convexity, we have
Revised: Therefore, by strong convexity, we have

Original: [Equation]
Revised: [Equation]

Original: And our induction is complete.
Revised: And our induction is complete.

Original: Therefore, we conclude that after log( (cid:107)X(cid:107) /(cid:15) )
Revised: Therefore, we conclude that after log( (cid:107)X(cid:107) /(cid:15) )

Original: epochs, we have (cid:107) x
Revised: epochs, we have (cid:107) x

Original: T − x ∗ (cid:107) ≤ (cid:15) .
Revised: T − x ∗ (cid:107) ≤ (cid:15) .

Original: Each epoch has H iterations, each with O (log( Q )) function evaluations and so our result follows.
Revised: Each epoch has H iterations, each with O (log( Q )) function evaluations and so our result follows.

Original: The low-rank result follows from applying Theorem 10 and Theorem 11 instead.
Revised: The low-rank result follows from applying Theorem 10 and Theorem 11 instead.

Original: However, note that since we do not know the latent dimension k , we must extend the binary search to incur an extra log( n ) factor in the binary search cost.
Revised: However, note that since we do not know the latent dimension k , we must extend the binary search to incur an extra log( n ) factor in the binary search cost.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: C

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: L o g O p t i m a li t y G a p
Revised: L o g O p t i m a li t y G a p

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: L o g O p t i m a li t y G a p
Revised: O p t i m a li t y G a p

Original: [Figure]
Revised: [Figure]

Original: BBOB Ellipsoidal Function
Revised: Ellipsoidal Function

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: L o g O p t i m a li t y G a p
Revised: L o g O p t i m a li t y G a p

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]