Original: I MPROVING G RADIENT E STIMATION IN E VOLUTIONARY S TRATEGIES W ITH P AST D ESCENT D IRECTIONS
Revised: I MPROVING G RADIENT E STIMATION IN E VOLUTIONARY S TRATEGIES W ITH P AST D ESCENT D IRECTIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Evolutionary Strategies (ES) are known to be an effective black-box optimization technique for deep neural networks when the true gradients cannot be computed, such as in Reinforcement Learning.
Revised: Evolutionary Strategies (ES) are known to be an effective black-box optimization technique for deep neural networks when the true gradients cannot be computed, such as in Reinforcement Learning.

Original: We continue a recent line of research that uses surrogate gradients to improve the gradient estimation of ES.
Revised: We continue a recent line of research that uses surrogate gradients to improve the gradient estimation of ES.

Original: We propose a novel method to optimally incorporate surrogate gradient information.
Revised: We propose a novel method to optimally incorporate surrogate gradient information.

Original: Our approach, unlike previous work, needs no information about the quality of the surrogate gradients and is always guaranteed to ﬁnd a descent direction that is better than the surrogate gradient.
Revised: Our approach, unlike previous work, needs no information about the quality of the surrogate gradients and is always guaranteed to ﬁnd a descent direction that is better than the surrogate gradient.

Original: This allows to iteratively use the previous gradient estimate as surrogate gradient for the current search point.
Revised: This allows to iteratively use the previous gradient estimate as surrogate gradient for the current search point.

Original: We theoretically prove that this yields fast convergence to the true gradient for linear functions and show under simplifying assumptions that it signiﬁcantly improves gradient estimates for general functions.
Revised: We theoretically prove that this yields fast convergence to the true gradient for linear functions and show under simplifying assumptions that it signiﬁcantly improves gradient estimates for general functions.

Original: Finally, we evaluate our approach empirically on MNIST and reinforcement learning tasks and show that it considerably improves the gradient estimation of ES at no extra computational cost.
Revised: Finally, we evaluate our approach empirically on MNIST and reinforcement learning tasks and show that it considerably improves the gradient estimation of ES at no extra computational cost.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Evolutionary Strategies (ES) (1; 2; 3) are a black-box optimization technique, that estimate the gradient of some objective function with respect to the parameters by evaluating parameter perturbations in random directions.
Revised: Evolutionary Strategies (ES) (1; 2; 3) are a black-box optimization technique, that estimate the gradient of some objective function with respect to the parameters by evaluating parameter perturbations in random directions.

Original: The beneﬁts of using ES in Reinforcement Learning (RL) were exhibited in (4).
Revised: The beneﬁts of using ES in Reinforcement Learning (RL) were exhibited in (4).

Original: ES approaches are highly parallelizable and account for robust learning, while having decent data-efﬁciency.
Revised: ES approaches are highly parallelizable and account for robust learning, while having decent data-efﬁciency.

Original: Moreover, black-box optimization techniques like ES do not require propagation of gradients, are tolerant to long time horizons, and do not suffer from sparse reward distributions (4).
Revised: Moreover, black-box optimization techniques like ES do not require propagation of gradients, are tolerant to long time horizons, and do not suffer from sparse reward distributions (4).

Original: This lead to a successful application of ES in variety of different RL settings (5; 6; 7; 8).
Revised: This lead to a successful application of ES in variety of different RL settings (5; 6; 7; 8).

Original: Applications of ES outside RL include for example meta learning (9).
Revised: Applications of ES outside RL include for example meta learning (9).

Original: In many scenarios, the true gradient is impossible to compute, however surrogate gradients are available.
Revised: In many scenarios, the true gradient is impossible to compute, however surrogate gradients are available.

Original: Here, we use the term surrogate gradients for directions that are correlated but usually not equal to the true gradient, e.g. they might be biased or unbiased approximations of the gradient.
Revised: Here, we use the term surrogate gradients for directions that are correlated but usually not equal to the true gradient, e.g. they might be biased or unbiased approximations of the gradient.

Original: Such scenarios include models with discrete stochastic variables (10), learned models in RL like
Revised: Such scenarios include models with discrete stochastic variables (10), learned models in RL like

Original: Q-learning (11), truncated backpropagation through time (12) and feedback alignment (13), see (14) for a detailed exhibition.
Revised: Q-learning (11), truncated backpropagation through time (12) and feedback alignment (13), see (14) for a detailed exhibition.

Original: If surrogate gradients are available, it is beneﬁcial to preferentially sample parameter perturbations from the subspace deﬁned by these directions (14).
Revised: If surrogate gradients are available, it is beneﬁcial to preferentially sample parameter perturbations from the subspace deﬁned by these directions (14).

Original: The proposed algorithm (14) requires knowing in advance the quality of the surrogate gradient, does not always provide a descent direction that is better than the surrogate gradient, and it remains open how to obtain such surrogate gradients in general settings.
Revised: The proposed algorithm (14) requires knowing in advance the quality of the surrogate gradient, does not always provide a descent direction that is better than the surrogate gradient, and it remains open how to obtain such surrogate gradients in general settings.

Original: In deep learning in general, experimental evidence has established that higher order derivatives are usually "well behaved", in which case gradients of consecutive parameter updates correlate and applying momentum speeds up convergence (15; 16; 17).
Revised: In deep learning in general, experimental evidence has established that higher order derivatives are usually "well behaved", in which case gradients of consecutive parameter updates correlate and applying momentum speeds up convergence (15; 16; 17).

Original: These observations suggest that past update directions are promising candidates for surrogate gradients.
Revised: These observations suggest that past update directions are promising candidates for surrogate gradients.

Original: In this work, we extend the line of research of (14).
Revised: In this work, we extend the line of research of (14).

Original: Our contribution is threefold:
Revised: Our contribution is threefold:

Original: • First, we show theoretically how to optimally combine the surrogate gradient directions with random search directions.
Revised: • First, we show theoretically how to optimally combine the surrogate gradient directions with random search directions.

Original: More precisely, our approach computes the direction of the subspace spanned by the evaluated search directions that is most aligned with the true gradient.
Revised: More precisely, our approach computes the direction of the subspace spanned by the evaluated search directions that is most aligned with the true gradient.

Original: Our gradient estimator does not need to know the quality of the surrogate gradients
Revised: Our gradient estimator does not need to know the quality of the surrogate gradients

Original: and always provides a descent direction that is more aligned with the true gradient than the surrogate gradient.
Revised: and always provides a descent direction that is more aligned with the true gradient than the surrogate gradient.

Original: • Second, above properties of our gradient estimator allow us to iteratively use the last update direction as a surrogate gradient for our gradient estimator.
Revised: • Second, above properties of our gradient estimator allow us to iteratively use the last update direction as a surrogate gradient for our gradient estimator.

Original: Repeatedly using the last update direction as a surrogate gradient will aggregate information about the gradient over time and results in improved gradient estimates.
Revised: Repeatedly using the last update direction as a surrogate gradient will aggregate information about the gradient over time and results in improved gradient estimates.

Original: In order to demonstrate how the gradient estimate improves over time, we prove fast convergence to the true gradient for linear functions and show, that under simplifying assumptions, it offers an improvement over ES that depends on the Hessian for general functions.
Revised: In order to demonstrate how the gradient estimate improves over time, we prove fast convergence to the true gradient for linear functions and show, that under simplifying assumptions, it offers an improvement over ES that depends on the Hessian for general functions.

Original: Third, we validate experimentally that these results transfer to practice, that is, the proposed approach computes more accurate gradients than standard ES.
Revised: Third, we validate experimentally that these results transfer to practice, that is, the proposed approach computes more accurate gradients than standard ES.

Revised: We observe that our algorithm considerably improves gradient estimation on the MNIST task compared to standard ES and that it improves convergence speed and performance on the tested Roboschool reinforcement learning environments

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Evolutionary strategies (1; 2; 3) are black box optimization techniques that approximate the gradient by sampling ﬁnite differences in random directions in parameter space.
Revised: Evolutionary strategies (1; 2; 3) are black box optimization techniques that approximate the gradient by sampling ﬁnite differences in random directions in parameter space.

Original: Promising potential of ES for the optimization of neural networks used for RL was demonstrated in (4).
Revised: Promising potential of ES for the optimization of neural networks used for RL was demonstrated in (4).

Original: They showed that ES gives rise to efﬁcient training despite the noisy gradient estimates that are generated from a much smaller number of samples than the dimensionality of parameter space.
Revised: They showed that ES gives rise to efﬁcient training despite the noisy gradient estimates that are generated from a much smaller number of samples than the dimensionality of parameter space.

Original: This placed ES on a prominent spot in the RL tool kit (5; 6; 7; 8).
Revised: This placed ES on a prominent spot in the RL tool kit (5; 6; 7; 8).

Original: The history of descent directions was previously used to adapt the search distribution in covariance matrix adaptation ES (CMA-ES) (18).
Revised: The history of descent directions was previously used to adapt the search distribution in covariance matrix adaptation ES (CMA-ES) (18).

Original: CMA-ES constructs a second-order model of the underlying objective function and samples search directions and adapts step size according to it.
Revised: CMA-ES constructs a second-order model of the underlying objective function and samples search directions and adapts step size according to it.

Original: However, maintaining the full covariance matrix makes the algorithm quadratic in the number of parameters, and thus impractical for high-dimensional spaces.
Revised: However, maintaining the full covariance matrix makes the algorithm quadratic in the number of parameters, and thus impractical for high-dimensional spaces.

Original: Linear time approximations of CMA-ES like diagonal approximations of the covariance matrix (19) often do not work well. E.g. even for linear functions the  gradient estimates will not converge to the true gradient, instead the step-size of the descent direction is arbitrarily increased.
Revised: Linear time approximations of CMA-ES like diagonal approximations of the covariance matrix (19) often do not work well, in the sense that their gradient estimates do not converge to the true gradient even if the true gradient does not change over time.

Original: Our approach differs as we simply improve the gradient estimation and then feed the gradient estimate to a ﬁrst-order optimization algorithm.
Revised: Our approach differs as we simply improve the gradient estimation and then feed the gradient estimate to a ﬁrst-order optimization algorithm.

Original: Our work is inspired by the line of research of (14), where surrogate gradient directions are used to improve gradient estimations by ’elongating’ the search space along these directions.
Revised: Our work is inspired by the line of research of (14), where surrogate gradient directions are used to improve gradient estimations by ’elongating’ the search space along these directions.

Original: That approach has two shortcomings.
Revised: That approach has two shortcomings.

Original: First, the bias of the surrogate gradients needs to be known to adapt the covariance matrix.
Revised: First, the bias of the surrogate gradients needs to be known to adapt the covariance matrix.

Original: Second, once the bias of the surrogate gradient is too small, the algorithm will not ﬁnd a better descent direction than the surrogate gradient.
Revised: Second, once the bias of the surrogate gradient is too small, the algorithm will not ﬁnd a better descent direction than the surrogate gradient.

Original: Another related area of research investigates how to use momentum for the optimization of deep neural networks.
Revised: Another related area of research investigates how to use momentum for the optimization of deep neural networks.

Original: Applying different kinds of momentum has become one of the standard tools in current deep learning and it has been shown to speed-up learning in a very wide range of tasks (20; 16; 17).
Revised: Applying different kinds of momentum has become one of the standard tools in current deep learning and it has been shown to speed-up learning in a very wide range of tasks (20; 16; 17).

Original: This hints, that for many problems the higher-order terms in deep learning models are "well-behaved" and thus, the gradients do not change too drastically after parameter updates.
Revised: This hints, that for many problems the higher-order terms in deep learning models are "well-behaved" and thus, the gradients do not change too drastically after parameter updates.

Original: While these approaches use momentum for parameter updates, our approach can be seen as a form of momentum when sampling directions from the search space of ES.
Revised: While these approaches use momentum for parameter updates, our approach can be seen as a form of momentum when sampling directions from the search space of ES.

Original:  G RADIENT E STIMATION
Revised:  G RADIENT E STIMATION

Original: We aim at minimizing a function f : R n → R by steepest descent.
Revised: We aim at minimizing a function f : R n → R by steepest descent.

Original: In scenarios where the gradient (cid:53) f does not exist or is inefﬁcient to compute, we are interested in obtaining some estimate of the (smoothed) gradient of f that provides a good parameter update direction.
Revised: In scenarios where the gradient (cid:53) f does not exist or is inefﬁcient to compute, we are interested in obtaining some estimate of the (smoothed) gradient of f that provides a good parameter update direction.

Original: 3. T HE ES G RADIENT E STIMATOR
Revised: 3. T HE ES G RADIENT E STIMATOR

Original: ES considers the function f
Revised: ES considers the function f

Original: σ that is obtained by Gaussian smoothing
Revised: σ that is obtained by Gaussian smoothing

Original: [Equation]
Revised: [Equation]

Original: where σ is a parameter modulating the size of the smoothing area and N ( 0 , I) is the n -dimensional
Revised: where σ is a parameter modulating the size of the smoothing area and N ( 0 , I) is the n -dimensional

Original: Gaussian distribution with 0 being the all 0 vector and I being
Revised: Gaussian distribution with 0 being the all 0 vector and I being

Original: the n -dimensional identity matrix.
Revised: the n -dimensional identity matrix.

Original: The gradient of f σ with respect to parameters θ is given by
Revised: The gradient of f σ with respect to parameters θ is given by

Original: [Equation]
Revised: [Equation]

Original: which can be sampled by a Monte Carlo estimator, see (5).
Revised: which can be sampled by a Monte Carlo estimator, see (5).

Original: Often antithetic sampling is used, as it reduces variance (5).
Revised: Often antithetic sampling is used, as it reduces variance (5).

Original: The antithetic ES gradient estimator using P samples is given by
Revised: The antithetic ES gradient estimator using P samples is given by

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) i are independently sampled from N ( 0 , I) for i ∈ { 1 , . .
Revised: where (cid:15) i are independently sampled from N ( 0 , I) for i ∈ { 1 , . .

Original: . , P } .
Revised: . , P } .

Original: This gradient estimator has been shown to be effective in RL settings (4).
Revised: This gradient estimator has been shown to be effective in RL settings (4).

Original: 3. O UR 1 -S TEP G RADIENT E STIMATOR
Revised: 3. O UR O NE S TEP G RADIENT E STIMATOR

Original: We ﬁrst give some intuition before presenting our gradient estimator formally.
Revised: We ﬁrst give some intuition before presenting our gradient estimator formally.

Original: Given one surrogate gradient direction ζ , our one step gradient estimator applies the following sampling strategy.
Revised: Given one surrogate gradient direction ζ , our one step gradient estimator applies the following sampling strategy.

Original: First, it estimates how much the gradient points into the direction of ζ by antithetically evaluating f in the direction of ζ .
Revised: First, it estimates how much the gradient points into the direction of ζ by antithetically evaluating f in the direction of ζ .

Original: Second, it estimates the part of (cid:53)
Revised: Second, it estimates the part of (cid:53)

Original: f that is orthogonal to ζ by evaluating random,  orthogonal search directions that are orthogonal to ζ .
Revised: f that is orthogonal to ζ by evaluating random, pairwise orthogonal search directions that are orthogonal to ζ .

Original: In this way, our estimator detects the optimal lengths of the parameter update step intothe both  surrogate direction and the evaluated orthogonal directions (e.g. if ζ and (cid:53) f are parallel, the update step is parallel to ζ , and if they are orthogonal the step into direction ζ has length 0 ).
Revised: In this way, our estimator detects the optimal lengths of the parameter update step into both the surrogate direction and the evaluated orthogonal directions (e.g. if ζ and (cid:53) f are parallel, the update step is parallel to ζ , and if they are orthogonal the step into direction ζ has length 0 ).

Original: Additionally, if the surrogate direction and the gradient are not perfectly aligned, then the gradient estimate almost surely improves over the surrogate direction due to the contribution from the evaluated directions orthogonal to ζ .
Revised: Additionally, if the surrogate direction and the gradient are not perfectly aligned, then the gradient estimate almost surely improves over the surrogate direction due to the contribution from the evaluated directions orthogonal to ζ .

Original: In the following we deﬁne our estimator formally and prove that the estimated direction possesses best possible alignment with the gradient that can be achieved with our sampling scheme.
Revised: In the following we deﬁne our estimator formally and prove that the estimated direction possesses best possible alignment with the gradient that can be achieved with our sampling scheme.

Original: We assume that k  orthogonal surrogate gradient directions ζ 1 , . . .
Revised: We assume that k pairwise orthogonal surrogate gradient directions ζ 1 , . . .

Original: , ζ k are given to our estimator.
Revised: , ζ k are given to our estimator.

Original: Denote by R ζ the subspace of R n that is spanned by the ζ i , and by R ⊥ ζ the subspace that is orthogonal to R ζ .
Revised: Denote by R ζ the subspace of R n that is spanned by the ζ i , and by R ⊥ ζ the subspace that is orthogonal to R ζ .

Original: Further, for vectors v and (cid:53) f , we denote by ˆ v and ˆ (cid:53)
Revised: Further, for vectors v and (cid:53) f , we denote by ˆ v and ˆ (cid:53)

Original: f the normalized vector v (cid:107) v (cid:107) and (cid:53) f (cid:107)(cid:53) f (cid:107) , respectively.
Revised: f the normalized vector v (cid:107) v (cid:107) and (cid:53) f (cid:107)(cid:53) f (cid:107) , respectively.

Original: Let ˆ (cid:15) 1 , . . .
Revised: Let ˆ (cid:15) 1 , . . .

Original: , ˆ (cid:15) P be random orthogonal unit vectors from R ⊥ ζ .
Revised: , ˆ (cid:15) P be random orthogonal unit vectors from R ⊥ ζ .

Original: Then, our estimator is deﬁned as
Revised: Then, our estimator is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: We write (cid:53) f
Revised: We write (cid:53) f

Original: = (cid:53) f (cid:107) ζ + (cid:53)
Revised: = (cid:53) f (cid:107) ζ + (cid:53)

Original: f ⊥ ζ , where (cid:53) f (cid:107) ζ and (cid:53)
Revised: f ⊥ ζ , where (cid:53) f (cid:107) ζ and (cid:53)

Original: f ⊥ ζ are the projections of (cid:53) f
Revised: f ⊥ ζ are the projections of (cid:53) f

Original: on R ζ and R ⊥ ζ , respectively.
Revised: on R ζ and R ⊥ ζ , respectively.

Original: In essence, the ﬁrst sum in (2) computes (cid:53) f (cid:107) ζ by assessing the quality of each surrogate gradient direction, and the second sum estimates (cid:53)
Revised: In essence, the ﬁrst sum in (2) computes (cid:53) f (cid:107) ζ by assessing the quality of each surrogate gradient direction, and the second sum estimates (cid:53)

Original: f ⊥ ζ similar to an orthogonalized antithetic ES gradient estimator, that samples directions from R ⊥ ζ , see (5).
Revised: f ⊥ ζ similar to an orthogonalized antithetic ES gradient estimator, that samples directions from R ⊥ ζ , see (5).

Original: We remark that we require  orthogonal unit directions ˆ (cid:15)
Revised: We remark that we require pairwise orthogonal unit directions ˆ (cid:15)

Original: i for the optimality proof.
Revised: i for the optimality proof.

Original: Due to the orthogonality of the directions, no normalization factor like the 1 /P factor in (1) is required in (2).
Revised: Due to the orthogonality of the directions, no normalization factor like the 1 /P factor in (1) is required in (2).

Original: In practice, the dimensionality n is often much larger than P .
Revised: In practice, the dimensionality n is often much larger than P .

Original: Then, this is nearly identical to sampling the (cid:15) i s from a N ( 0 , I) distribution, because in high-dimensional space the norm of (cid:15) i ∼ N ( 0 , I) is highly concentrated around 

Revised: Then, sampling pairwise orthogonal unit vecotrs (cid:15) i is nearly identical to sampling the (cid:15) i s from a N ( 0 , I) distribution, because in high-dimensional space the norm of (cid:15) i ∼ N ( 0 , I) is highly concentrated around

Original: and the cosine of two such random vectors is highly concentrated around 0 .
Revised: and the cosine of two such random vectors is highly concentrated around 0 .

Original: For the sake of analysis, we assume that f is differentiable and we assume equality for the following ﬁrst order approximation 
Revised: For the sake of analysis, we assume that f is differentiable and that the second order approximation f ( x + [Equation] x ) + [Equation]

Revised: [Equation]

Revised: f (

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: H ( x )

Revised: [Equation]

Revised: , where H ( x ) denotes the Hessian matrix of f at x , is exact.

Revised: This assumption implies that

Revised: [Equation]

Revised: because the even terms cancel for antithetic sampling.

Revised: The following proposition and theorems provide theoretical understanding, how our gradient estimation scheme improves gradient estimation in this smooth, noise-free setting.

Original: In the following, we will omit the θ in (cid:53)
Revised: In the following, we will omit the θ in (cid:53)

Original: f ( θ ) .
Revised: f ( θ ) .

Original: Our ﬁrst theorem states that g our computes the direction in the subspace spanned by ζ 1 , . . .
Revised: Our ﬁrst proposition states that g ⊥ computes the direction in the subspace spanned by ζ 1 , . . .

Original: , ζ k , (cid:15) 1 , . . .
Revised: , ζ k , (cid:15) 1 , . . .

Original: , (cid:15) P that is most aligned with (cid:53) f .
Revised: , (cid:15) P that is most aligned with (cid:53) f .

Original: Theorem 1 (Optimality of g our ) .
Revised: Proposition 1 (Optimality of g ⊥ ) .

Original: Let ζ 1 , . .
Revised: Let ζ 1 , . .

Original: , ζ k , (cid:15) 1 , . . .
Revised: , ζ k , (cid:15) 1 , . . .

Original: , (cid:15) P be  orthogonal vectors in R n .
Revised: , (cid:15) P be pairwise orthogonal vectors in R n .

Original: Then, g our = (cid:80) ki =1 (cid:104)(cid:53) f, ˆζ i (cid:105)
Revised: Then, g ⊥ = [Equation] , ˆ

Original: ˆ ζ i +(cid:80) Pi =1 (cid:104)(cid:53)
Revised: [Equation] (cid:105) [Equation] +

Original:  f, ˆ (cid:15) i (cid:105) ˆ (cid:15)
Revised: [Equation] f, ˆ (cid:15) i (cid:105) ˆ (cid:15)

Original: i computes the projection of (cid:53) f on the subspace spanned by ζ 1 , . . .
Revised: i computes the projection of (cid:53) f on the subspace spanned by ζ 1 , . . .

Original: , ζ k , (cid:15) 1 , . . .
Revised: , ζ k , (cid:15) 1 , . . .

Original: , (cid:15) P .
Revised: , (cid:15) P .

Original: Especially, (cid:15) =
Revised: Especially, (cid:15) =

Original: g our is the vector of that subspace that maximizes the cosine (cid:104) ˆ (cid:53) f, ˆ (cid:15) (cid:105) between (cid:53) f and (cid:15) .
Revised: g ⊥ is the vector of that subspace that maximizes the cosine (cid:104) ˆ (cid:53) f, ˆ (cid:15) (cid:105) between (cid:53) f and (cid:15) .

Original: Moreover, the squared cosine between g our and (cid:53) f is given by 
Revised: Moreover, the squared cosine between g ⊥ and (cid:53) f is given by [Equation] , ˆ

Original: [Equation]
Revised: [Equation]

Original: We remark that when evaluating (cid:104)(cid:53) f, v i (cid:105) for arbitrary directions v i , no information about search directions orthogonal to the subspace spanned by the v i s is obtained.
Revised: We remark that when evaluating (cid:104)(cid:53) f, v i (cid:105) for arbitrary directions v i , no information about search directions orthogonal to the subspace spanned by the v i s is obtained.

Original: Therefore, one can only hope for ﬁnding the best approximation of (cid:53)
Revised: Therefore, one can only hope for ﬁnding the best approximation of (cid:53)

Original: f lying within the subspace spanned by the v i s, which is accomplished by g our .
Revised: f lying within the subspace spanned by the v i s, which is accomplished by g ⊥ .

Original: The proof of Theorem 1 follows easily from the Cauchy-Schwarz inequality and is given in the appendix.
Revised: The proof of Proposition 1 follows easily from the Cauchy-Schwarz inequality and is given in the appendix.

Original: 3. I TERATIVE A PPLICATION OF OUR E STIMATOR
Revised: 3. I TERATIVE G RADIENT E STIMATION U SING P AST D ESCENT D IRECTIONS

Original: We suppose that the last gradient estimate is more aligned with the  gradient than a random direction. Thus, we propose to use the gradient estimate of the last time step as surrogate direction for the current time step. 
Revised: Our gradient estimation algorithm iteratively applies the one step gradient estimator g ⊥ by using the gradient estimate of the last time step as surrogate direction for the current time step. Therefore, our algorithm relies on the assumption that gradients are correlated from one time step to the next.

Revised: This assumption is justiﬁed since it is one of the reasons momentum-based optimizers (15; 16; 17) are successful in deep learning.

Original: When iteratively applying our 1 -step gradient estimator with the last estimateas surrogate direction, then at any time step the estimator improves over the surrogate direction. Therefore, the  gradient estimate accumulates information about the gradient and becomes more aligned with  the gradient  over time.
Revised: We also explicitly test this assumption experimentally, see Figure 1a. The quality of the gradient estimate at any time step, depends on the quality of the surrogate gradient, which might be restrained because (a) the previous gradient estimate might be very noisy, and (b) the gradient changes with parameter updates. Our algorithm efﬁciently tackles problem (a), since it improves the gradient estimate over the surrogate gradient at any time step.

Original: For linear functions, the estimate converges to  the true gradient, because the gradient does not change over time.
Revised: In order to theoretically quantify this, we ﬁrst analyse the case where the gradient does not change with parameter updates, i.e. if a linear function is optimized.

Revised: In this setting, we show that our algorithm needs a small factor more than n (dimension) samples to align with the true gradient, see Theorem 1, which is close to optimal, because at least n samples are required to determine the true gradient.

Original: In the following, we rigorously analyze the convergence rate of this iterative gradient estimation process for linear functions. For general functions,  the gradient changes over time due to the parameter updates. 
Revised: Next, we incorporate problem (b) into our analysis by considering general, non-linear functions, for which the gradient changes with parameter updates. We show under some simplifying assumptions that, also in this case, our algorithm builds up an improved gradient estimate over time, see Theorem 2.

Original: In that setting, we determine the convergence value assuming some simplifying assumptions.
Revised: We ﬁrst need some notation.

Original: Denote by θ t the search point at time t  and by ζ t the parameter update stepthat is applied at time t , that is, θ t [-+
-] =
Revised: Denote by θ t the search point, by (cid:53) f t = (cid:53) f ( θ t ) the gradient and by ζ t the parameter update step at time t , that is, θ t [-+
-]+1 =

Original: θ t + ζ t .
Revised: θ t + ζ t .

Original: For the analysis, we assume that in order to obtain an estimate for the gradient (cid:53) f ( θ t) , we compute g our by evaluating the last update direction ζ t − 1  and one random direction (cid:15) t orthogonal to ζ t − 1 :
Revised: The iterative gradient estimation algorithm obtains the gradient estimate ζ t by computing g ⊥ with the last update direction ζ t − 1 as surrogate gradient and P new random directions ˆ (cid:15) i .

Original: [Equation] Then, Equation 3 of Theorem 1 turns into
Revised: Formally, let ˆ (cid:15) 1 t , . . .

Original: [Equation] We remark that restricting the analysis to one direction ˆ (cid:15)  t is no restriction at all, because the right hand side of (2) can be obtained by choosing (cid:15) t
Revised: , ˆ (cid:15) P t be pairwise orthogonal unit directions chosen uniformly from the unit sphere, that is, they are conditioned to be pairwise orthogonal and are marginally uniformly distributed.

Original:  = (cid:80) Pi =1 (cid:104)(cid:53)
Revised: By deﬁning (cid:15) t = (cid:80) Pi =1 (cid:104)(cid:53)

Original: f t , ˆ (cid:15) i (cid:105) ˆ (cid:15) i and 
Revised: f t , ˆ (cid:15) it (cid:105) ˆ (cid:15) it , and setting ζ t =

Original: ζ
Revised: g ⊥ , we obtain

Original: [Equation]
Revised: [Equation]

Original: (cid:80) ki = 
Revised: (5) where we used (cid:15) t = (cid:107) (cid:15) t (cid:107) 2 / (cid:107) (cid:15) t (cid:107) · ˆ (cid:15) t = (cid:104)(cid:53) f t , (cid:15) t (cid:105) / (cid:107) (cid:15) t (cid:107) · ˆ (cid:15) t = (cid:104)(cid:53) f, ˆ (cid:15) t (cid:105) ˆ (cid:15) t . Then, Equation 4 of

Revised: Proposition 1 turns into

Original: [Equation]
Revised: [Equation]

Original: (cid:104)(cid:53) f t , ˆ ζ i (cid:105) ˆ ζ i . The next theorem quantiﬁes how fast the cosine between ζ t and (cid:53) f t converges to 1 , iff is a linear function, that is, (cid:53) f t does not change over time.
Revised: The next theorem quantiﬁes how fast the cosine between ζ t and (cid:53) f t converges to 1 , if (cid:53) f t does not change over time.

Original: Theorem 2 (Convergence rate for linear functions) .
Revised: Theorem 1 (Convergence rate for linear functions) .

Original: Let c ≈ 1 be  the constant such that the expected cosine squared between two random vectors in an N − 1 -dimensional space is cN − 1 . Let X t = (cid:104) ˆ (cid:53) f, ˆ ζ t (cid:105) be the random variable that denotes the cosine between ζ t and (cid:53) f t at time t .
Revised: Let ζ t be iteratively computed, as in Equation (5) , using the past update direction and P pairwise orthogonal random directions and let X t = (cid:104) ˆ (cid:53) f, ˆ ζ t (cid:105) be the random variable that denotes the cosine between ζ t and (cid:53) f t at time t .

Original: Then, the expected drift of X 2 t is E [ X 2 t − X 2 t − 1 | X t − 1 =
Revised: Then, the expected drift of X 2 t is E [ X 2 t − X 2 t − 1 | X t − 1 =

Original: x
Revised: x

Original: t − 1 ]
Revised: t − 1 ]

Original: = (1 − x 2 t − 1 ) cN − 1 .
Revised: = (1 − x 2 t − 1 ) PN − 1 .

Original: Moreover, let (cid:15) > 0 and deﬁne T to be the ﬁrst point in time t with X 2 t ≥ 1 − δ .
Revised: Moreover, let (cid:15) > 0 and deﬁne T to be the ﬁrst point in time t with X 2 t ≥ 1 − δ .

Original: It holds
Revised: It holds

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst bound E [ T ] ≤ N − 1 c 1 − δδ is tight for δ close to 1 and follows by an additive drift theorem, while the second bound E [ T ] ≤ N − 1 c (1 + ln(1 /δ )) is tight for δ close to 0 and follows by a variable drift theorem, see appendix. 
Revised: The ﬁrst bound E [ T ] ≤ N − 1 P 1 − δδ is tight for δ close to 1 and follows by an additive drift theorem, while the second bound E [ T ] ≤ N − 1 P (1 + ln(1 /δ )) is tight for δ close to 0 and follows by a variable drift theorem, see appendix. We remark that in a smooth and noise-free setting, where one can sample the true directional derivative with Equation (3), a cosine squared of 1 − δ can be reached by sampling (1 − δ ) N random orthogonal directions, see Proposition 2 in the appendix. Since our algorithm evaluates P + 1 directions per time step, it requires approximately min { 1 /δ, 1+ln(1 /δ )

Revised: 1 − δ } times more samples to reach the same alignment.

Original: Naturally, the linear case is not the most interesting one.
Revised: Naturally, the linear case is not the most interesting one.

Original: However, it is hard to rigorously analyse the case of general f , because it is unpredictable how the gradient (cid:53) f t differs from (cid:53)
Revised: However, it is hard to rigorously analyse the case of general f , because it is unpredictable how the gradient (cid:53) f t differs from (cid:53)

Original: f t − 1 .
Revised: f t − 1 .

Original: Note that (cid:53) f t − (cid:53)
Revised: Note that (cid:53) f t − (cid:53)

Original: f t − 

Revised: f t −

Original: ≈ Hζ t − 1 , where H is the Hessian matrix of f at θ t − 1 .
Revised: ≈ Hζ t − 1 , where H is the Hessian matrix of f at θ t − 1 .

Original: We deﬁne α t = (cid:104) ˆ (cid:53)
Revised: We deﬁne α t = (cid:104) ˆ (cid:53)

Original: f t , ˆ (cid:53)
Revised: f t , ˆ (cid:53)

Original: f t − 1 (cid:105) and write ˆ (cid:53) f t = α t ˆ (cid:53)
Revised: f t − 1 (cid:105) and write ˆ (cid:53) f t = α t ˆ (cid:53)

Original: f t − 1 + (cid:53) f ⊥
Revised: f t − 1 + (cid:53) f ⊥

Original: where (cid:53) f ⊥ is orthogonal to ˆ (cid:53) f t and has squared norm 1 − α 2 t .
Revised: where (cid:53) f ⊥ is orthogonal to ˆ (cid:53) f t and has squared norm 1 − α 2 t .

Original: Then, the ﬁrst term of (5) is equal to
Revised: Then, the ﬁrst term of (6) is equal to

Original: [Equation]
Revised: [Equation]

Original: In the following, we assume that (cid:53) f ⊥ is a direction orthogonal to (cid:53) f t − 1 chosen uniformly at random.
Revised: In the following, we assume that (cid:53) f ⊥ is a direction orthogonal to (cid:53) f t − 1 chosen uniformly at random.

Original: Though, this assumption is not entirely true, it allows to get a grasp on the approximate cosine that our estimator is going to converge to.
Revised: Though, this assumption is not entirely true, it allows to get a grasp on the approximate cosine that our estimator is going to converge to.

Original: Theorem 3.
Revised: Theorem 2.

Original: Let c be deﬁned as in Theorem 2, and let X t − 1 be the  cosine between two vectors ˆ (cid:53) f t −and ζ t− 1 .
Revised: Let ζ t be iteratively computed using the past update direction and P pairwise orthogonal random directions, see Equation (5) , and let X t = (cid:104) ˆ (cid:53) f, ˆ ζ t (cid:105) be the random variable that denotes the cosine between ζ t and (cid:53) f t at time t .

Original: Further, let 1 ≥ α t ≥ 0 and ˆ (cid:53) f t = α t ˆ (cid:53)
Revised: Further, write ˆ (cid:53) f t = α t ˆ (cid:53)

Original: f t − 1 + (cid:53) f ⊥ , where  (cid:53) f ⊥ is a random vector orthogonal to ˆ (cid:53) f t − 1with norm (cid:112) 1 − α 2 t .
Revised: f t − 1 + (cid:53) f ⊥ , where 1 ≥ α t ≥ 0 and assume that (cid:53) f ⊥ has a random direction orthogonal to ˆ (cid:53) f t − 1 .

Original: Choose ζ t according to Equation (4) and deﬁne X t to be the cosine between ˆ (cid:53) f t and ˆ ζ t .
Revised: Choose ζ t according to Equation (5) and deﬁne X t to be the cosine between ˆ (cid:53) f t and ˆ ζ t .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: The last theorem implies that the evolution of the cosine depends heavily on the cosine α t between consecutive gradients.
Revised: The last theorem implies that the evolution of the cosine depends heavily on the cosine α t between consecutive gradients.

Original: Let A =
Revised: Let A =

Original: (1 − α 2 t ) cN − 1 (1 − c N − 1 ) + c N − 1 1 − ( α 2 t
Revised: (1 − α 2 t ) 1 N − 1 (1 − PN − 1 )+ PN − 1 1 − ( α 2 t

Original: +(1 − α 2 t cN − 1 )(1 − cN − 1 )) .
Revised: +(1 − α 2 t ) 1 N − 1 )(1 − PN − 1 ) .

Original: Then, the theorem implies that the drift
Revised: Then, the theorem implies that the drift

Original: E [ X 2 t − X 2 t − 1 | X t − 1 = x t − 1 ] is positive if x t − 1 ≤ A and negative otherwise.
Revised: E [ X 2 t − X 2 t − 1 | X t − 1 = x t − 1 ] is positive if x t − 1 ≤ A and negative otherwise.

Original: Thus, if α t would not change over time, we would expect X t to converge to A . 
Revised: Thus, if α t would not change over time, we would expect X t to converge to A . TODO:

Revised: explain comparison to practise plot:

Revised: The predicted alignement of the gradient estimate with the true gradient of Theorem 2 is plotted in Figure 1b.

Revised: Though the assumptions of Theorem 2 are technically not true, the close ﬁt of the theoretical prediction gives empirical evidence that the analysis captures the general behaviour of the gradient estimation process.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we will empirically evaluate the performance of our gradient estimation scheme when combined with deep neural networks.
Revised: In this section, we will empirically evaluate the performance of our gradient estimation scheme when combined with deep neural networks.

Original: In Section 4.1, we show that it signiﬁcantly improves gradient estimation for digit classiﬁers on MNIST.
Revised: In Section 4.1, we show that it signiﬁcantly improves gradient estimation for digit classiﬁers on MNIST.

Original: In Section 4.2, we suggest how to overcome issues that arise from function evaluation noise.
Revised: In Section 4.2, we suggest how to overcome issues that arise from function evaluation noise.

Original: Finally, in Section 4.3, we evaluate our gradient estimation scheme on RL environments and investigate further issues arising in this setting.
Revised: Finally, in Section 4.3, we evaluate our gradient estimation scheme on RL environments and investigate further issues arising in this setting.

Original: 4. G RADIENT E STIMATION AND P ERFORMANCE ON MNIST
Revised: 4. G RADIENT E STIMATION AND P ERFORMANCE ON MNIST

Original: We observe that our approach signiﬁcantly improves gradient estimation compared to standard ES.
Revised: We observe that our approach signiﬁcantly improves gradient estimation compared to standard ES.

Original: Figure 1a shows that the key requisite of our iterative gradient estimation scheme is satisﬁed during  training on MNIST, that is, that gradients between consecutive parameter update steps are correlated.
Revised: Figure 1a shows that the key requisite of our iterative gradient estimation scheme is satisﬁed during [Figure] training on MNIST, that is, that gradients between consecutive parameter update steps are correlated.

Original: Figure 1b shows that our approach improves gradient estimation compared to ES during the whole training process and strongly improves it in the beginning of training, where consecutive gradients are most correlated, see Figure 1a.
Revised: Figure 1b shows that our approach improves gradient estimation compared to ES during the whole training process and strongly improves it in the beginning of training, where consecutive gradients are most correlated, see Figure 1a.

Original: We observe that our approach strongly outperforms ES in convergence speed and reaches better ﬁnal performance for all hyperparameters we tested, see Figure 2 and
Revised: We observe that our approach strongly outperforms ES in convergence speed and reaches better ﬁnal performance for all hyperparameters we tested, see Figure 2 and

Original: Table 1.
Revised: Table 1.

Original: Implementation details: For these experiments, we used a fully connected neural network with two hidden layers with a tanh non-linearity and 1000 units each, to have a high dimensional model ( ∼ 1 .
Revised: Implementation details: For these experiments, we used a fully connected neural network with two hidden layers with a tanh non-linearity and 1000 units each, to have a high dimensional model ( ∼ 1 .

Original: 8 million parameters) .
Revised: 8 million parameters) .

Original: For standard ES 128 random search directions are evaluated at each step.
Revised: For standard ES 128 random search directions are evaluated at each step.

Original: For our algorithm the previous gradient estimate and 126 random search directions are evaluated.
Revised: For our algorithm the previous gradient estimate and 126 random search directions are evaluated.

Original: We evaluated all directions on the same batch of images in order eliminate function evaluation noise and we resampled after every update step.
Revised: We evaluated all directions on the same batch of images in order eliminate function evaluation noise and we resampled after every update step.

Original: We used small parameter perturbations ( σ = 0 . 001 ).
Revised: We used small parameter perturbations ( σ = 0 . 001 ).

Original: This is possible because no function evaluation noise is present and because the objective function is already differentiable and therefore no smoothing is required.
Revised: This is possible because no function evaluation noise is present and because the objective function is already differentiable and therefore no smoothing is required.

Original: We test both SGD and Adam optimizers with learning rates in the range 10 0 .
Revised: We test both SGD and Adam optimizers with learning rates in the range 10 0 .

Original: , 10 − 3 .
Revised: , 10 − 3 .

Revised: [Table]

Revised: [Figure]

Original: 4. R OBUSTNESS TO F UNCTION E VALUATION N OISE
Revised: 4. R OBUSTNESS TO F UNCTION E VALUATION N OISE

Original: In practice, our iterative gradient estimation scheme may suffer from function evaluation noise because it builds up good gradient estimates over several parameter update steps.
Revised: In practice, our iterative gradient estimation scheme may suffer from function evaluation noise because it builds up good gradient estimates over several parameter update steps.

Original: Suppose that the past update direction is a good descent direction but it performs poorly on the current batch used for evaluation due to randomness in the batch selection or network evaluation process.
Revised: Suppose that the past update direction is a good descent direction but it performs poorly on the current batch used for evaluation due to randomness in the batch selection or network evaluation process.

Original: Then, this direction is weighted lightly when computing the new update direction, see Equation4, and therefore[Figure] the information about this direction will be discarded.
Revised: Then, this direction is weighted lightly when computing the new update direction, see Equation and therefore the information about this direction will be discarded.

Original: We empirically show, that our approach suffers heavily from this issue when artiﬁcially injecting noise in the function evaluation process, see Figure 3b .
Revised: We empirically show, that our approach suffers heavily from this issue when artiﬁcially injecting noise in the function evaluation process, see Figure 3b .

Original: Figure 3c shows, that this issue can be resolved by using the last k update directions for our gradient estimator (see Equation 2).
Revised: Figure 3c shows, that this issue can be resolved by using the last k update directions for our gradient estimator (see Equation 2).

Original: In this case, a good direction is only discarded, when it performs poorly in k consecutive evaluation steps, which is very unlikely.
Revised: In this case, a good direction is only discarded, when it performs poorly in k consecutive evaluation steps, which is very unlikely.

Original: We remark that the magnitude of the parameter updates naturally limits k , because the k -th last update direction is only useful if it is still correlated with the current gradient.
Revised: We remark that the magnitude of the parameter updates naturally limits k , because the k -th last update direction is only useful if it is still correlated with the current gradient.

Original: Concretely, we found that using the last 4 parameters updates was extremely helpful for smaller learning rates, even in the absence of noise (see Figure 3c).
Revised: Concretely, we found that using the last 4 parameters updates was extremely helpful for smaller learning rates, even in the absence of noise (see Figure 3c).

Original: However, it did not offer an advantage for larger learning rates.
Revised: However, it did not offer an advantage for larger learning rates.

Original: 4. R OBOTIC RL ENVIRONMENTS
Revised: 4. R OBOTIC RL ENVIRONMENTS

Original: For the next set of experiments we evaluate our algorithm on three robotics tasks of the Roboschool environment: RoboschoolInvertedPendulum-v1, RoboschoolHalfCheetah-v1 and RoboschoolAnt-v1.
Revised: For the next set of experiments, we evaluate our algorithm on three robotics tasks of the Roboschool environment: RoboschoolInvertedPendulum-v1, RoboschoolHalfCheetah-v1 and RoboschoolAnt-v1.

Original: Our approach outperforms ES in the pendulum task, and offers a small improvement over ES in the other two tasks, see Figure 4.
Revised: Our approach outperforms ES in the pendulum task, and offers a small improvement over ES in the other two tasks, see Figure 4.

Original: The improvement of our approach over standard ES is smaller on RL[Table] [Figure] tasks than on the MNIST task.
Revised: The improvement of our approach over standard ES is smaller on RL tasks than on the MNIST task.

Original: Therefore, we ﬁrst empirically conﬁrmed that past updated direction are also in RL correlated with the gradient.
Revised: Therefore, we ﬁrst empirically conﬁrmed that past updated direction are also in RL correlated with the gradient.

Original: To test this we kept track of the average difference between random perturbations and the direction given by our algorithm, after normalizing the rewards.
Revised: To test this, we kept track of the average difference between random perturbations and the direction given by our algorithm, after normalizing the rewards.

Original: We found that, the direction of our algorithm had an average weight of 1 . 11 versus the 0 .
Revised: We found that, the direction of our algorithm had an average weight of 1 . 11 versus the 0 .

Original: 65 of a random direction.
Revised: 65 of a random direction.

Original: RL robotics tasks bring two additional major challenges compared to the MNIST task.
Revised: RL robotics tasks bring two additional major challenges compared to the MNIST task.

Original: First, exploration is crucial to escape local optima and ﬁnd new solutions and second, the function evaluation noise is huge due to each perturbation being tested only on a single trajectory.
Revised: First, exploration is crucial to escape local optima and ﬁnd new solutions, and second, the function evaluation noise is huge due to each perturbation being tested only on a single trajectory.

Original: Our proposed solution of robustness against function evaluation noise intertwines with the exploration issue.
Revised: Our proposed solution of robustness against function evaluation noise intertwines with the exploration issue.

Original: A rather small step size is necessary in order to use more past directions as surrogate gradients.
Revised: A rather small step size is necessary in order to use more past directions as surrogate gradients.

Original: However, exploration in ES is driven by large perturbation sizes and noisy optimization trajectories.
Revised: However, exploration in ES is driven by large perturbation sizes and noisy optimization trajectories.

Original: We did not observe improvements when combining the approach of using several past directions with standard hyperparameter settings.
Revised: We did not observe improvements when combining the approach of using several past directions with standard hyperparameter settings.

Original: We believe that an exhaustive empirical study can shed light onto the effect of our approach on exploration and may further improve the performance on RL tasks.
Revised: We believe that an exhaustive empirical study can shed light onto the effect of our approach on exploration and may further improve the performance on RL tasks.

Original: However, running extensive experiments for complex RL environment is computationally expensive.
Revised: However, running extensive experiments for complex RL environment is computationally expensive.

Original: Implementation details: We use most of the hyper-parameters from the OpenAI implementation 1 .
Revised: Implementation details: We use most of the hyper-parameters from the OpenAI implementation 1 .

Original: That is, two hidden layers of 256 units each and with a tanh non-linearity.
Revised: That is, two hidden layers of 256 units each and with a tanh non-linearity.

Original: Further, we use a learning rate of 0 .
Revised: Further, we use a learning rate of 0 .

Original: 01 , a perturbation standard deviation of σ = 0 .
Revised: 01 , a perturbation standard deviation of σ = 0 .

Original: 02 and the Adam optimizer, and we also apply ﬁtness shaping (19).
Revised: 02 and the Adam optimizer, and we also apply ﬁtness shaping (19).

Original: For standard ES 128 random perturbations are evaluated at each step.
Revised: For standard ES 128 random perturbations are evaluated at each step.

Original: For our algorithm the previous gradient estimate and 126 random perturbations are evaluated.
Revised: For our algorithm the previous gradient estimate and 126 random perturbations are evaluated.

Original: For the
Revised: For the

Original: Ant and Cheetah environments, we observed with this setup, that agents often get stuck in a local optima where they stay completely still, instead of running forward.
Revised: Ant and Cheetah environments, we observed with this setup, that agents often get stuck in a local optima where they stay completely still, instead of running forward.

Original: As this happens for both, ES and our algorithm, we tweaked the environments in order to ensure that a true solution to the task is learned and not some some degenerate optima, we tweaked the environments in the following way.
Revised: As this happens for both, ES and our algorithm, we tweaked the environments in order to ensure that a true solution to the task is learned and not some some degenerate optima, we tweaked the environments in the following way.

Original: We remove the penalty for using electricity and ﬁnish the episode if the agent does not make any progress in a given amount of time.
Revised: We remove the penalty for using electricity and ﬁnish the episode if the agent does not make any progress in a given amount of time.

Original: In this way, agents consistently escape the local minima.
Revised: In this way, agents consistently escape the local minima.

Original: We use a tanh non-linearity on the output of the network, which increased stability of training, as otherwise the output of the network would become very large without an electricity penalty.
Revised: We use a tanh non-linearity on the output of the network, which increased stability of training, as otherwise the output of the network would become very large without an electricity penalty.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a gradient estimator that optimally incorporates surrogate gradient directions and random search directions, in the sense that it determines the direction with maximal cosine to the true gradient from the subspace of evaluated directions.
Revised: We proposed a gradient estimator that optimally incorporates surrogate gradient directions and random search directions, in the sense that it determines the direction with maximal cosine to the true gradient from the subspace of evaluated directions.

Original: Such a method has many applications as elucidated in (14).
Revised: Such a method has many applications as elucidated in (14).

Original: Importantly, our estimator does not require information about the quality of surrogate directions, which allows us to iteratively use past update directions as surrogate directions for our gradient estimator.
Revised: Importantly, our estimator does not require information about the quality of surrogate directions, which allows us to iteratively use past update directions as surrogate directions for our gradient estimator.

Original: We theoretically quantiﬁed the beneﬁts of the proposed iterative gradient estimation scheme.
Revised: We theoretically quantiﬁed the beneﬁts of the proposed iterative gradient estimation scheme.

Original: Finally, we showed that our approach in combination with deep neural networks considerably improves the gradient estimation capabilities of ES, at no extra computational cost.
Revised: Finally, we showed that our approach in combination with deep neural networks considerably improves the gradient estimation capabilities of ES, at no extra computational cost.

Original: The results on MNIST indicate that the speed of the Evolutionary Strategies themselves, a key part in the current Reinforcement Learning toolbox, is greatly improved.
Revised: The results on MNIST indicate that the speed of the Evolutionary Strategies themselves, a key part in the current Reinforcement Learning toolbox, is greatly improved.

Original: Within Reinforcement Learning an out of the box application of our algorithm yields some improvements.
Revised: Within Reinforcement Learning an out of the box application of our algorithm yields some improvements.

Original: The smaller improvement in RL compared to MNIST is likely due to the interaction of our approach and exploration that is essential in RL.
Revised: The smaller improvement in RL compared to MNIST is likely due to the interaction of our approach and exploration that is essential in RL.

Original: We leave it to future work to explicitly add and study appropriate exploration strategies which might unlock the true potential of our approach in RL.
Revised: We leave it to future work to explicitly add and study appropriate exploration strategies which might unlock the true potential of our approach in RL.

Original: A P ROOF OF T HEOREMS
Revised: A P ROOF OF T HEOREMS

Original: In this Section we prove the theorems from the main paper rigorously.
Revised: In this Section we prove the theorems from the main paper rigorously.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF P ROPOSITION

Original: Note that for this theorem there is no distinction between the directions ζ i and (cid:15) i .
Revised: Note that for this theorem there is no distinction between the directions ζ i and (cid:15) i .

Original: For ease of notation, we denote ζ 1 , . . .
Revised: For ease of notation, we denote ζ 1 , . . .

Original: , ζ k , ζ 1 , . . .
Revised: , ζ k , ζ 1 , . . .

Original: , ζ P by (cid:15) 1 , . . .
Revised: , ζ P by (cid:15) 1 , . . .

Original: , (cid:15) m .
Revised: , (cid:15) m .

Original: The theorem is a simple application of the CauchySchwarz inequality.
Revised: The theorem is a simple application of the CauchySchwarz inequality.

Original: Denote by (cid:53) f (cid:107) (cid:15) i
Revised: Denote by (cid:53) f (cid:107) (cid:15) i

Original: =
Revised: =

Original: (cid:80) mi =1 (cid:104)(cid:53)f, ˆ (cid:15)
Revised: (cid:80) m i =1 (cid:104)(cid:53)

Original:  i (cid:105) ˆ (cid:15) i the projection of (cid:53) f on the subspace spanned by the (cid:15) i s and let (cid:15)
Revised: f, ˆ (cid:15) i (cid:105) ˆ (cid:15) i the projection of (cid:53) f on the subspace spanned by the (cid:15) i s, and let (cid:15)

Original: = (cid:80) mi =

Revised: = (cid:80) mi =

Original: α i ˆ (cid:15) i be a vector in that subspace.
Revised: α i ˆ (cid:15) i be a vector in that subspace.

Original: Then, the Cauchy-Schwarz inequality implies
Revised: Then, the Cauchy-Schwarz inequality implies

Original: [Equation]
Revised: [Equation]

Original: Equality holds if and only if (cid:15) and (cid:53) f (cid:107) (cid:15) i have the same direction, which is equivalent to (cid:15)
Revised: Equality holds if and only if (cid:15) and (cid:53) f (cid:107) (cid:15) i have the same direction, which is equivalent to (cid:15)

Original: = αg our for some α > 0 .
Revised: = αg our for some α > 0 .

Original: In particular, in this case the cosine squared between (cid:53) f and (cid:15) is
Revised: In particular, in this case the cosine squared between (cid:53) f and (cid:15) is

Original: [Equation]
Revised: [Equation]

Revised: A. E XPECTATION OF C OSINE S QUARED OF R ANDOM V ECTORS FROM THE U NIT S PHERE

Revised: We need the following proposition for the proofs of Theorem 1 and 2.

Revised: Proposition 2.

Revised: Let ˆ u be an N -dimensional unit vector, and let ˆ (cid:15) 1 , . . .

Revised: , ˆ (cid:15) P be pairwise orthogonal vectors sampled uniformly from the N -dimensional unit sphere, that is, they are marginally uniformly distributed and conditioned to be pairwise orthogonal.

Revised: Then, the expected cosine squared of ˆ u and (cid:15) =

Revised: (cid:80) Pi =1 (cid:104) u, ˆ (cid:15) i (cid:105) ˆ (cid:15) i is

Revised: [Equation]

Revised: Proof.

Revised: Note that

Revised: [Equation]

Revised: (cid:32)

Revised: [Equation]

Revised: (cid:88)

Revised: i

Revised: [Equation]

Revised: (cid:104)

Revised: [Equation]

Revised: (cid:15)

Revised: [Equation]

Revised: (cid:105)(cid:33)=

Revised: P (cid:88) i = (cid:104)

Revised: ˆ u, ˆ (cid:15) i (cid:105)

Revised: [Equation]

Revised: Denote by S the N -dimensional unit sphere.

Revised: Linearity of expectation implies that

Revised: [Equation]

Revised: By rotational invariance of the unit sphere, we can replace ˆ u by e 1 = (1 , 0 , . .

Revised: . , 0) and obtain

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original:  A. P ROOF OF T HEOREM
Revised: N A. P ROOF OF T HEOREM

Original: In order to prove Theorem 3, use Equation 5 to compute how X 2 t = (cid:104) ˆ (cid:53) f, ˆ ζ t (cid:105) 2 depends on X 2 t − 1 .
Revised: In order to prove Theorem 2, use Equation 6 to compute how X 2 t = (cid:104) ˆ (cid:53) f, ˆ ζ t (cid:105) 2 depends on X 2 t − 1 .

Original: Then, we apply a variable transformation to X t in order to be able to apply the additive and variable drift theorems from (21), which are stated in Section B.
Revised: Then, we apply a variable transformation to X t in order to be able to apply the additive and variable drift theorems from (21), which are stated in Section B.

Original: When taking one sample per time step, ˆ (cid:15) t is a random unit vector orthogonal to ˆ ζ t − 1 .

Original: We can split the normalized gradient ˆ (cid:53) f = (cid:53)
Revised: We can split the normalized gradient ˆ (cid:53) f = (cid:53)

Original: f ⊥  ζ  + (cid:53) f (cid:107)  ζ  into an orthogonal to ˆ ζ t − 1 part and a parallel to ˆ ζ t −part.
Revised: f ⊥ ˆ ζ t − 1 + (cid:53) f (cid:107) ˆ ζ t − 1 into an orthogonal to ˆ ζ t − 1 part and a parallel to ˆ ζ t − 1 part.

Original: It holds (cid:104) ˆ (cid:53) f 
Revised: It holds (cid:107) (cid:53) f ⊥

Original: [Equation] ζ , ˆ (cid:15) t [Equation] = 0 and (cid:107) (cid:53) f⊥ ζ (cid:107) {+ˆ ζ t − 1 , ˆ (cid:15) t (cid:105) = 
+}
Revised: ˆ ζ t − 1 (cid:107) 2 = 1 − (cid:104) ˆ (cid:53) f, ˆ ζ t − 1 (cid:105) 2 and (cid:104) ˆ (cid:53) f (cid:107) {+ˆ ζ t − 1 , ˆ (cid:15) t (cid:105) = 
+}

Original: [Equation]

Original: 1 − (cid:104) ˆ (cid:53) f [Equation] ˆ [Equation]
Revised: since ˆ (cid:15) t is a unit vector orthogonal to ˆ ζ t − 1 .

Original: Recall that ˆ (cid:53)
Revised: Recall that ˆ (cid:53)

Original: f ⊥ ζ [Equation]
Revised: f ⊥ ζ = (cid:53)

Original:  , then (cid:104)ˆ [Equation] =
Revised: f ⊥ ζ (cid:107)(cid:53) f ⊥ ζ (cid:107) , then (cid:104)

Original: [Equation]
Revised: [Equation]

Original: and therefore by Equation
Revised: and therefore by Equation

Original: [Equation]
Revised: [Equation]

Original: Deﬁne the random process Y t = 1 − X 2 t .
Revised: Deﬁne the random process Y t = 1 − X 2 t .

Original: It holds
Revised: It holds

Original: [Equation]
Revised: [Equation]

Original: where we used that ˆ (cid:15) t is a random vector in the N − 1 dimensional space that is orthogonal toˆ ζ t−and (cid:53) f ⊥ ζ is a vector in the same subspace, which implies E [ [Equation] − 
Revised: where we used Proposition 2 in the N − 1 dimensional subspace that is orthogonal to ζ t − 1 .

Original: [Equation]

Original: In order to derive the ﬁrst bound on T , we bound the drift of Y t for Y t ≥ δ .
Revised: In order to derive the ﬁrst bound on T , we bound the drift of Y t for Y t ≥ δ .

Original: [Equation]
Revised: [Equation]

Original: where we used Equation 11 and y t − 1 ≥ δ .
Revised: where we used Equation 12 and y t − 1 ≥ δ .

Original: In order to apply Theorem 4, we deﬁne the auxiliary process Z t =
Revised: In order to apply Theorem 3, we deﬁne the auxiliary process Z t =

Original: Y t − δ .
Revised: Y t − δ .

Original: Then, T is the expected time that Z t hits 0 .
Revised: Then, T is the expected time that Z t hits 0 .

Original: Since E [ Z t | Z t − 1 =
Revised: Since E [ Z t | Z t − 1 =

Original: z t − 1 , z t − 1 ≥ 0] ≤ z t − 1 − δ cN − 1 and Z 0 = 1 − δ , Theorem 4 implies that
Revised: z t − 1 , z t − 1 ≥ 0] ≤ z t − 1 − δ cN − 1 and Z 0 = 1 − δ , Theorem 3 implies that

Original: [Equation]
Revised: [Equation]

Original: In order to apply Theorem 5, to show the second bound on T , we need to rescale Y t such that it takes values in { 0 } ∪ [1 , ∞ ) .
Revised: In order to apply Theorem 4, to show the second bound on T , we need to rescale Y t such that it takes values in { 0 } ∪ [1 , ∞ ) .

Original: Deﬁne the auxiliary process Z t by
Revised: Deﬁne the auxiliary process Z t by

Original: [Equation]
Revised: [Equation]

Original: Then, T is the expected time that Z t hits 0 .
Revised: Then, T is the expected time that Z t hits 0 .

Original: The process Z t satisﬁes
Revised: The process Z t satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where we used Equations 12 and 11.
Revised: where we used Equations 13 and 12.

Original: Since Z 0 = 1 /δ , Theorem 5 implies for h ( z ) =
Revised: Since Z 0 = 1 /δ , Theorem 4 implies for h ( z ) =

Original: z cN − 1 that
Revised: z cN − 1 that

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: In order to prove the theorem, we need to understand how X t depends on the value of X t − 1 .
Revised: In order to prove the theorem, we need to understand how X t depends on the value of X t − 1 .

Original: It holds X 2 t = [-(cid:
-]
Revised: It holds X 2 t = [-(cid:
-][Equation]

Original: ) ˆ (cid:53) f t , ˆ ζ t − 1 (cid:105) 2 + (cid:104) ˆ (cid:53) f t , ˆ (cid:15) t (cid:105) 2 .
Revised: ˆ (cid:53) f t , ˆ ζ t − 1 (cid:105) 2 + (cid:104) ˆ (cid:53) f t , ˆ (cid:15) t (cid:105) 2 .

Original: As in Equation 11, we can write (cid:104) ˆ (cid:53)
Revised: As in Equation 12, we can write (cid:104) ˆ (cid:53)

Original: f t , ˆ (cid:15) t (cid:105) 2 ={+
+}
Revised: f t , ˆ (cid:15) t (cid:105) {+
+}

Original:  (1 − (cid:104) ˆ (cid:53)
Revised: = (1 − (cid:104) ˆ (cid:53)

Original: f t , ˆ ζ [Equation] − 1 (cid:105) 2 ) (cid:104) ˆ (cid:53)
Revised: f t , ˆ ζ t − 1 (cid:105) 2 ) (cid:104) ˆ (cid:53)

Original: f ⊥ ˆ ζ t − 1 , ˆ (cid:15) t (cid:105) 2 , and note that E [ (cid:104) ˆ (cid:53)
Revised: f ⊥ ˆ ζ t − 1 , ˆ (cid:15) t (cid:105) 2 , and note that E [ (cid:104) ˆ (cid:53)

Original: f ⊥ ˆ ζ t − 1 , ˆ (cid:15) t (cid:105) 2 ] =
Revised: f ⊥ ˆ ζ t − 1 , ˆ (cid:15) t (cid:105) 2 ] =

Original: cN − 1 because ˆ (cid:15)is a random direction orthogonal to ˆ ζ t− 1 .
Revised: PN − 1 follows by Proposition 2 and the deﬁnition of ˆ (cid:15) t .

Original: This implies that
Revised: This implies that

Original: [Equation]
Revised: [Equation]

Original: To understand how the X t evolves we need to analyze how (cid:104) ˆ (cid:53) f t , ˆ ζ t − 1 (cid:105) 2 relates to X t − 1 = (cid:104) ˆ (cid:53)
Revised: To understand how the X t evolves we need to analyze how (cid:104) ˆ (cid:53) f t , ˆ ζ t − 1 (cid:105) 2 relates to X t − 1 = (cid:104) ˆ (cid:53)

Original: f t − 1 , ˆ ζ t − 1 (cid:105) 2 .
Revised: f t − 1 , ˆ ζ t − 1 (cid:105) 2 .

Original: To that end, we set α t =
Revised: To that end, we set α t =

Original: (cid:104) ˆ (cid:53)
Revised: (cid:104) ˆ (cid:53)

Original: f t , ˆ (cid:53)
Revised: f t , ˆ (cid:53)

Original: f t − 1 (cid:105) and write ˆ (cid:53) f t = α t ˆ (cid:53) 
Revised: f t − 1 (cid:105) and write ˆ (cid:53) f t = [Equation] (cid:53) f

Original: f t − 1 + (cid:53) f⊥
Revised: [Equation] (cid:53) f

Revised: [Equation]

Original: where (cid:53) f ⊥ is orthogonal toˆ (cid:53) f t  and has norm 1 − α 2 t .
Revised: −where (cid:53) f ⊥(cid:53) f t − 1 is orthogonal to (cid:53) f t − 1 and has norm 1 − α 2 t .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: where we used E [ (cid:104)(cid:53) f ⊥ , ˆ ζ t − 1 (cid:105) ]
Revised: f [Equation] , ˆ [Equation]

Original: =
Revised: It follows that [Equation] [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Revised: where we used E [ (cid:104)(cid:53) f ⊥(cid:53) f t − 1 , ˆ ζ t − 1 (cid:105) ]

Original:  which follows from the assumption of (cid:53) f ⊥ being a random direction orthogonal to (cid:53) f t − 
Revised: = 0 , which follows from the assumption of (cid:53) f ⊥(cid:53) f t − 1 being a random direction orthogonal to (cid:53) f t − 1 , and that E [ (cid:104)(cid:53)

Original: [Equation]
Revised: [Equation]

Original: and that
Revised: ⊥(cid:53)

Original: [Equation]
Revised: [Equation]

Original: which follows from (cid:107) (cid:53) f {+⊥(cid:53) f t − 1 (cid:107) 
+}
Revised: which follows from (cid:107)(cid:53) f {+⊥(cid:53) f t − 1 (cid:107) 
+}

Original: [Equation]
Revised: = 1 − α 2 , (cid:107) ˆ ζ t − 1 ⊥(cid:53)

Original:  − x 2 t and  (cid:53) f ⊥ being a random direction orthogonal to (cid:53) f t − 1 .
Revised: f t − 1 (cid:107) 2 = 1 − x 2 t and Proposition 2 for P =using that (cid:53) f ⊥(cid:53) f t − 1 is a random direction orthogonal to (cid:53) f t − 1 .

Original: Then, plugging Equation (15) into (13), implies the theorem.
Revised: Then, plugging Equation (17) into (14), implies the theorem.

Original: B D RIFT T HEOREMS
Revised: B D RIFT T HEOREMS

Original: For the proof of Theorem 2, we use two drift theorems from (21), which we restate for completeness.
Revised: For the proof of Theorem 1, we use two drift theorems from (21), which we restate for completeness.

Original: Theorem 4 (Additive Drift, Theorem 1 from (21)) .
Revised: Theorem 3 (Additive Drift, Theorem 1 from (21)) .

Original: Let ( X t ) t ∈ N 0 be a Markov chain with state space
Revised: Let ( X t ) t ∈ N 0 be a Markov chain with state space

Original: S ⊂ [0 , ∞ ) and assume X 0 = n .
Revised: S ⊂ [0 , ∞ ) and assume X 0 = n .

Original: Let T be the earliest point in time t ≥ 

Revised: Let T be the earliest point in time t ≥

Original: such that X t = 0 .
Revised: such that X t = 0 .

Original: If there exists c > 0 such that for all x ∈ S , x > 0 and for all t ≥ 0 we have
Revised: If there exists c > 0 such that for all x ∈ S , x > 0 and for all t ≥ 0 we have

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: Theorem 5.
Revised: Theorem 4.

Original: Variable Drift, Theorem 4 from (21)] Let ( X t ) t ∈
Revised: Variable Drift, Theorem 4 from (21)] Let ( X t ) t ∈

Original: N be a Markov chain with state space
Revised: N be a Markov chain with state space

Original: S ⊂ { 0 } ∪ [1 , ∞ ) and with X 0 = n .
Revised: S ⊂ { 0 } ∪ [1 , ∞ ) and with X 0 = n .

Original: Let T be the earliest point in time t ≥ 

Revised: Let T be the earliest point in time t ≥

Original: such that X t = 0 .
Revised: such that X t = 0 .

Original: Suppose furthermore that there is a positive, increasing function h : [1 , ∞ )
Revised: Suppose furthermore that there is a positive, increasing function h : [1 , ∞ )

Original: → R > 0 such that for all x ∈ S , x > 0 we have for all t ≥
Revised: → R > 0 such that for all x ∈ S , x > 0 we have for all t ≥

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Revised: C A DDITIONAL E XPERIMENTS

Revised: This section presents some additional experiments.

Revised: We will incorporate a polished version of this section into the experiments section of the main paper, for the camera ready version of the paper.

Revised: C. C OMPARISON OF OUR APPROACH TO SEPARABLE NES AND GUIDED ES

Revised: We consider the toy task of optimizing a quadratic function f ( x ) = (cid:107) Ax (cid:107) 2 2 as considered in (14).

Revised: Figure 5 compares our approach to a diagonal approximation of CMA-ES, that is, seperable natural

Revised: ES (SNES) from (19), that has the same runtime and memory complexity as our approach.

Revised: While

Revised: SNES works verv well for small diminsional parameter space ( N = 32 , see Figrue 5a ), it is clearly outperformed by our approach for high dimensional paramater spaces ( N = 10000 , see Figure 5b).

Revised: Further, we compare our one step gradient estimator g ⊥ to the guided-ES gradient estimator proposed in (14).

Revised: Again the goal is to optimize a quadratic function f ( x ) = (cid:107) Ax (cid:107) 22 .

Revised: Both estimators receive a surrogate gradient ζ for the optimization.

Revised: The surrogate gradient is created as in (14): A normalized random bias b (drawn once at the beginning of optimization) and a normalized noise direction n (resampled at every iteration) are added to the true gradient.

Revised: that is, ζ ( x ) = (cid:53) f

Revised: ( x )+( b + n ) (cid:107)(cid:53) f

Revised: ( x ) (cid:107)

Revised: Figure 6 shows that our approach outperforms guided-ES no matter how the parameter α controlling the bias variance trade-off of guided-ES is set.

Revised: Implementation details: For Figure 5b, x has dimension N = 10000 and is initalized by sampling x 0 ∼ N (0 , I ) .

Revised: A is a matrix of dimension 100 × 10000 , and its entries are samples from a N (0 , 0 . 001) distribution, where the variance is chosen such that the initial loss is approx 1 .

Revised: For Figure 5a, x has dimension N = 16 and A has dimension 16 × 16 .

Revised: The hyperparamters for our approach and standard

Revised: ES are the learning rate β and perturbation size σ .

Revised: The hyperparamters for SNES are learning rate β , learning rate β σ for the step size adaptation and perturbation size initialization.

Revised: We performed a hyperparameter search for all these hyperparameters and plotted the best performance.