Original: PPO-CMA: P ROXIMAL P OLICY O PTIMIZATION WITH C OVARIANCE M ATRIX A DAPTATION
Revised: PPO-CMA: P ROXIMAL P OLICY O PTIMIZATION WITH C OVARIANCE M ATRIX A DAPTATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Proximal Policy Optimization (PPO) is a highly popular model-free reinforcement learning (RL) approach.
Revised: Proximal Policy Optimization (PPO) is a highly popular model-free reinforcement learning (RL) approach.

Original: However, in continuous state and actions spaces and a Gaussian policy – common in computer animation and robotics – PPO is prone to getting stuck in local optima.
Revised: However, in continuous state and actions spaces and a Gaussian policy – common in computer animation and robotics – PPO is prone to getting stuck in local optima.

Original: In this paper, we observe a tendency of
Revised: In this paper, we observe a tendency of

Original: PPO to prematurely shrink the exploration variance, which naturally leads to slow progress.
Revised: PPO to prematurely shrink the exploration variance, which naturally leads to slow progress.

Original: Motivated by this, we borrow ideas from CMA-ES, a black-box optimization method designed for intelligent adaptive Gaussian exploration, to derive
Revised: Motivated by this, we borrow ideas from CMA-ES, a black-box optimization method designed for intelligent adaptive Gaussian exploration, to derive

Original: PPO-CMA, a novel proximal policy optimization approach that can expand the exploration variance on objective function slopes and shrink the variance when close to the optimum.
Revised: PPO-CMA, a novel proximal policy optimization approach that can expand the exploration variance on objective function slopes and shrink the variance when close to the optimum.

Original: This is implemented by using separate neural networks for policy mean and variance and training the mean and variance in separate passes.
Revised: This is implemented by using separate neural networks for policy mean and variance and training the mean and variance in separate passes.

Original: Our experiments demonstrate a clear improvement over vanilla PPO in many difﬁcult OpenAI Gym MuJoCo tasks.
Revised: Our experiments demonstrate a clear improvement over vanilla PPO in many difﬁcult OpenAI Gym MuJoCo tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: This paper proposes a new solution to the problem of policy optimization with high-dimensional continuous state and action spaces.
Revised: This paper proposes a new solution to the problem of policy optimization with high-dimensional continuous state and action spaces.

Original: This is a problem that has a long history in computer animation, robotics, and machine learning research.
Revised: This is a problem that has a long history in computer animation, robotics, and machine learning research.

Original: More speciﬁcally, our work falls in the domain of simulation-based Monte Carlo approaches; instead of operating with closed-form expressions of the control dynamics, we use a black-box dynamics simulator, sample actions from some distribution, simulate the results, and then adapt the sampling distribution.
Revised: More speciﬁcally, our work falls in the domain of simulation-based Monte Carlo approaches; instead of operating with closed-form expressions of the control dynamics, we use a black-box dynamics simulator, sample actions from some distribution, simulate the results, and then adapt the sampling distribution.

Original: In recent years, such approaches have achieved remarkable success in previously intractable tasks such as real-time locomotion control of (simpliﬁed) biomechanical models of the human body (Wang et al.
Revised: In recent years, such approaches have achieved remarkable success in previously intractable tasks such as real-time locomotion control of (simpliﬁed) biomechanical models of the human body (Wang et al.

Original: ; Geijtenbeek et al.
Revised: ; Geijtenbeek et al.

Original: H¨am¨al¨ainen et al.
Revised: H¨am¨al¨ainen et al.

Original: Liu et al.
Revised: Liu et al.

Original: ; Duan et al.
Revised: ; Duan et al.

Original: ; Rajam¨aki & H¨am¨al¨ainen (2017)).
Revised: ; Rajam¨aki & H¨am¨al¨ainen (2017)).

Original: In 2017, Proximal Policy Optimization (PPO) provided the ﬁrst demonstration of a neural network policy that enables a simulated humanoid not only to run but also to rapidly switch direction and get up after falling (Schulman et al.
Revised: In 2017, Proximal Policy Optimization (PPO) provided the ﬁrst demonstration of a neural network policy that enables a simulated humanoid not only to run but also to rapidly switch direction and get up after falling (Schulman et al.

Original: Previously, such feats had only been achieved with more computationally heavy approaches that used simulation and sampling not only during training but also in run-time (H¨am¨al¨ainen et al.
Revised: Previously, such feats had only been achieved with more computationally heavy approaches that used simulation and sampling not only during training but also in run-time (H¨am¨al¨ainen et al.

Original: (2014; 2015); Rajam¨aki & H¨am¨al¨ainen (2017)).
Revised: (2014; 2015); Rajam¨aki & H¨am¨al¨ainen (2017)).

Original: PPO has been quickly adopted as the default RL algorithm in popular frameworks like Unity Machine Learning Agents 1 and TensorFlow Agents (Hafner et al.
Revised: PPO has been quickly adopted as the default RL algorithm in popular frameworks like Unity Machine Learning Agents Juliani et al. and TensorFlow Agents (Hafner et al.

Original: It has also been extended and applied to even more complex humanoid movement skills such as kung-fu kicks and backﬂips (Peng et al.
Revised: It has also been extended and applied to even more complex humanoid movement skills such as kung-fu kicks and backﬂips (Peng et al.

Original: Outside the continuous control domain, it has demonstrated outstanding performance in complex multi-agent video games 2 .
Revised: Outside the continuous control domain, it has demonstrated outstanding performance in complex multi-agent video games 1 .

Original: However, like many other reinforcement learning methods, PPO can be sensitive to hyperparameter choices and difﬁcult to tune (Henderson et al.
Revised: However, like many other reinforcement learning methods, PPO can be sensitive to hyperparameter choices and difﬁcult to tune (Henderson et al.

Original: In experiments by ourselves and our colleagues, we have also noticed a tendency to get stuck in local optima.
Revised: In experiments by ourselves and our colleagues, we have also noticed a tendency to get stuck in local optima.

Original: In this paper, we make the following contributions:
Revised: In this paper, we make the following contributions:

Original: • We provide visualizations and evidence of how PPO’s exploration variance can shrink prematurely, which leads to slow progress or getting stuck in local optima.
Revised: • We provide visualizations and evidence of how PPO’s exploration variance can shrink prematurely, which leads to slow progress or getting stuck in local optima.

Original: Figure 1 illustrates this in a simple didactic example.
Revised: Figure 1 illustrates this in a simple didactic example.

Original: Subsequently, we discuss how a similar exploration problem is solved in the black-box optimization domain by Covariance Matrix Adaptation Evolution Strategy (CMA-ES).
Revised: Subsequently, we discuss how a similar exploration problem is solved in the black-box optimization domain by Covariance Matrix Adaptation Evolution Strategy (CMA-ES).

Original: CMA-ES dynamically expands the exploration variance on objective function slopes and only shrinks the variance when close to the optimum.
Revised: CMA-ES dynamically expands the exploration variance on objective function slopes and only shrinks the variance when close to the optimum.

Original: • We show how exploration behavior similar to CMA-ES can be achieved in RL with simple changes, resulting in our PPO-CMA algorithm visualized in Figure 1.
Revised: • We show how exploration behavior similar to CMA-ES can be achieved in RL with simple changes, resulting in our PPO-CMA algorithm visualized in Figure 1.

Original: As elaborated in Section 4, a key idea of PPO-CMA is to use separate neural networks for policy mean and variance, and train the mean and variance in separate passes.
Revised: As elaborated in Section 4, a key idea of PPO-CMA is to use separate neural networks for policy mean and variance, and train the mean and variance in separate passes.

Original: Our experiments show a signiﬁcant improvement over PPO in many OpenAI Gym MuJoCo tasks (Brockman et al.
Revised: Our experiments show a signiﬁcant improvement over PPO in many OpenAI Gym MuJoCo tasks (Brockman et al.

Original: In Appendix A, we also investigate solving the variance adaptation problem with simple variance clipping and entropy regularization.
Revised: In Appendix A, we also investigate solving the variance adaptation problem with simple variance clipping and entropy regularization.

Original: The entropy regularization was suggested by Schulman et al.
Revised: The entropy regularization was suggested by Schulman et al.

Original: (2017) but not analyzed in detail.
Revised: (2017) but not analyzed in detail.

Original: Our results suggest that both approaches can help but they are sensitive to tuning parameters.
Revised: Our results suggest that both approaches can help but they are sensitive to tuning parameters.

Original:  P RELIMINARIES 2. R EINFORCEMENT L EARNING
Revised:  P RELIMINARIES 2. R EINFORCEMENT L EARNING

Original: Algorithm 1 Episodic On-policy Reinforcement Learning (high-level summary) 1: for iteration=1,2,... do 2: while iteration simulation budget N not exceeded do 3:
Revised: Algorithm 1 Episodic On-policy Reinforcement Learning (high-level summary) 1: for iteration=1,2,... do 2: while iteration simulation budget N not exceeded do 3:

Original: Reset the simulation to a (random) initial state 4:
Revised: Reset the simulation to a (random) initial state 4:

Original: Run agent on policy π θ for T timesteps or until a terminal state 5: end while 6:
Revised: Run agent on policy π θ for T timesteps or until a terminal state 5: end while 6:

Original: Update policy parameters θ based on the observed experience [ s i , a i , r i , s (cid:48) i ] 7: end for
Revised: Update policy parameters θ based on the observed experience [ s i , a i , r i , s (cid:48) i ] 7: end for

Original: We consider the discounted formulation of the policy optimization problem, following the notation of Schulman et al.
Revised: We consider the discounted formulation of the policy optimization problem, following the notation of Schulman et al.

Original: (2015b).
Revised: (2015b).

Original: At time t , the agent observes a state vector s t and takes an action a t ∼ π θ ( a t | s t ) , where π θ denotes a stochastic policy parameterized by θ , e.g., neural network weights.
Revised: At time t , the agent observes a state vector s t and takes an action a t ∼ π θ ( a t | s t ) , where π θ denotes a stochastic policy parameterized by θ , e.g., neural network weights.

Original: This results in observing a new state s (cid:48) t and receiving a scalar reward r t .
Revised: This results in observing a new state s (cid:48) t and receiving a scalar reward r t .

Original: The goal is to ﬁnd θ that maximizes the expected future-discounted sum of rewards E [ (cid:80) ∞ t =

Revised: The goal is to ﬁnd θ that maximizes the expected future-discounted sum of rewards E [ (cid:80) ∞ t =

Original: γ t r t ] , where γ is a discount factor in the range (0 , 1) .
Revised: γ t r t ] , where γ is a discount factor in the range (0 , 1) .

Original: A lower γ makes the learning prefer instant gratiﬁcation instead of long-term gains.
Revised: A lower γ makes the learning prefer instant gratiﬁcation instead of long-term gains.

Original: The original PPO and the PPO-CMA proposed in this paper collect experience tuples [ s i , a i , r i , s (cid:48) i ] by simulating a number of episodes in each optimization iteration.
Revised: The original PPO and the PPO-CMA proposed in this paper collect experience tuples [ s i , a i , r i , s (cid:48) i ] by simulating a number of episodes in each optimization iteration.

Original: For each episode, an initial state s 0 is sampled from some application-dependent stationary distribution, and the simulation is continued until a terminal (absorbing) state or a predeﬁned maximum episode length T is reached.
Revised: For each episode, an initial state s 0 is sampled from some application-dependent stationary distribution, and the simulation is continued until a terminal (absorbing) state or a predeﬁned maximum episode length T is reached.

Original: After an iteration simulation budget N is exhausted, θ is updated.
Revised: After an iteration simulation budget N is exhausted, θ is updated.

Original: This is summarized in Algorithm 1.
Revised: This is summarized in Algorithm 1.

Original: 2. P OLICY GRADIENT WITH ADVANTAGE ESTIMATION
Revised: 2. P OLICY GRADIENT WITH ADVANTAGE ESTIMATION

Original: Policy gradient methods update policy parameters by estimating the gradient g = ∇ θ E [ (cid:80) ∞ t γ t r t ] .
Revised: Policy gradient methods update policy parameters by estimating the gradient g = ∇ θ E [ (cid:80) ∞ t γ t r t ] .

Original: There are different formulations of the gradient, of which PPO uses the following:
Revised: There are different formulations of the gradient, of which PPO uses the following:

Original: [Equation]
Revised: [Equation]

Original: where A π is the so-called advantage function .
Revised: where A π is the so-called advantage function .

Original: Intuitively, the advantage function is positive if an explored action yields better rewards than expected.
Revised: Intuitively, the advantage function is positive if an explored action yields better rewards than expected.

Original: Updating θ in the direction of g makes negative advantage actions less likely and positive advantage actions more likely (Schulman et al.
Revised: Updating θ in the direction of g makes negative advantage actions less likely and positive advantage actions more likely (Schulman et al.

Original: (2015b)).
Revised: (2015b)).

Original: In practice, using modern compute graph frameworks like TensorFlow (Abadi et al.
Revised: In practice, using modern compute graph frameworks like TensorFlow (Abadi et al.

Original: , one often does not directly operate on the gradients, but instead uses an optimizer like Adam (Kingma & Ba (2014)) to minimize the corresponding loss
Revised: , one often does not directly operate on the gradients, but instead uses an optimizer like Adam (Kingma & Ba (2014)) to minimize the corresponding loss

Original: [Equation]
Revised: [Equation]

Original: where i denotes minibatch sample index and M is minibatch size.
Revised: where i denotes minibatch sample index and M is minibatch size.

Original: In summary, this type of policy gradient RL simply requires a differentiable expression of π θ and a way to measure A π for each explored state-action pair.
Revised: In summary, this type of policy gradient RL simply requires a differentiable expression of π θ and a way to measure A π for each explored state-action pair.

Original: More speciﬁcally, the advantage function is deﬁned as:
Revised: More speciﬁcally, the advantage function is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: s t ) . [Equation] Here, V π is the state value function, i.e., the expected future-discounted sum of rewards for running the agent on-policy starting from state s t .
Revised: Here, V π is the state value function, i.e., the expected future-discounted sum of rewards for running the agent on-policy starting from state s t .

Original: Q π ( s t , a t ) is the state-action value function, i.e., expected sum of rewards for taking action a t in state s t and then following the policy, Q π ( s t , a t ) =
Revised: Q π ( s t , a t ) is the state-action value function, i.e., expected sum of rewards for taking action a t in state s t and then following the policy, Q π ( s t , a t ) =

Original: r ( s t , a t )+ γV π ( s t +1 ) .
Revised: r ( s t , a t )+ γV π ( s t +1 ) .

Original: Thus, the advantage can also be expressed as
Revised: Thus, the advantage can also be expressed as

Original: [Equation]
Revised: [Equation]

Original: In practice, V π is usually approximated by a critic network trained with the observed rewards summed over simulation trajectories.
Revised: In practice, V π is usually approximated by a critic network trained with the observed rewards summed over simulation trajectories.

Original: However, plugging such an approximation directly to Equation 4 tends to be unstable due to approximation bias.
Revised: However, plugging such an approximation directly to Equation 4 tends to be unstable due to approximation bias.

Original: Instead, same as PPO, we use Generalized
Revised: Instead, same as PPO, we use Generalized

Original: Advantage Estimation (GAE) (Schulman et al. (2015b)), which is a simple but effective way to estimate A π such that one can trade variance for bias.
Revised: Advantage Estimation (GAE) (Schulman et al. (2015b)), which is a simple but effective way to estimate A π such that one can trade variance for bias.

Original:  U NDERSTANDING VARIANCE ADAPTATION IN G AUSSIAN POLICY OPTIMIZATION
Revised:  U NDERSTANDING VARIANCE ADAPTATION IN G AUSSIAN POLICY OPTIMIZATION

Original: This paper focuses on the case of continuous control using a Gaussian policy.
Revised: This paper focuses on the case of continuous control using a Gaussian policy.

Original: In other words, the policy network outputs state-dependent mean µ θ ( s ) and covariance C θ ( s ) for sampling the actions.
Revised: In other words, the policy network outputs state-dependent mean µ θ ( s ) and covariance C θ ( s ) for sampling the actions.

Original: The covariance deﬁnes the exploration-exploitation balance.
Revised: The covariance deﬁnes the exploration-exploitation balance.

Original: In practice, one often uses a diagonal covariance matrix parameterized by a vector c θ ( s ) =
Revised: In practice, one often uses a diagonal covariance matrix parameterized by a vector c θ ( s ) =

Original: diag ( C θ ( s )) .
Revised: diag ( C θ ( s )) .

Original: In the most simple case of isotropic unit Gaussian exploration, C
Revised: In the most simple case of isotropic unit Gaussian exploration, C

Original: =
Revised: =

Original: I , the loss function in Equation 2 becomes:
Revised: I , the loss function in Equation 2 becomes:

Original: [Equation]
Revised: [Equation]

Original: Following the original PPO paper, this paper uses diagonal covariance.
Revised: Following the original PPO paper, this paper uses diagonal covariance.

Original: This results in a slightly more complex loss function:
Revised: This results in a slightly more complex loss function:

Original: [Equation]
Revised: [Equation]

Original: where i indexes over a minibatch and j indexes over action variables.
Revised: where i indexes over a minibatch and j indexes over action variables.

Original: 3. T HE INSTABILITY CAUSED BY NEGATIVE ADVANTAGES
Revised: 3. T HE INSTABILITY CAUSED BY NEGATIVE ADVANTAGES

Original: To gain an intuitive understanding of the training dynamics, it is important to note the following:
Revised: To gain an intuitive understanding of the training dynamics, it is important to note the following:

Original: • Equation 5 indicates that the policy is trained to approximate the sampled actions , with each action weighted by A π .
Revised: • Equation 5 indicates that the policy is trained to approximate the sampled actions , with each action weighted by A π .

Original: Equation 5 uses an L 2 loss, while Equation 6 uses a Bayesian loss that allows the network to model aleatoric uncertainty, expressed as the free covariance parameters (Kendall & Gal (2017)).
Revised: Equation 5 uses an L 2 loss, while Equation 6 uses a Bayesian loss that allows the network to model aleatoric uncertainty, expressed as the free covariance parameters (Kendall & Gal (2017)).

Original: • As elaborated below, actions with a negative A π may cause instability , especially when one considers training for several epochs at each iteration using the same data; as pointed out by Schulman et al.
Revised: • As elaborated below, actions with a negative A π may cause instability , especially when one considers training for several epochs at each iteration using the same data; as pointed out by Schulman et al.

Original: (2017), this would be desirable to get the most out of the collected experience.
Revised: (2017), this would be desirable to get the most out of the collected experience.

Original: For actions with positive advantages, training produces a Gaussian approximation where more weight is given to actions with higher advantages.

Original: In other words, the policy Gaussian will not diverge outside the proximity of the sampled actions.

Original: For negative advantages, each  gradient update drives the policy Gaussian further away from the sampled actions .
Revised: For negative advantages, each minibatch gradient step drives the policy Gaussian further away from the sampled actions .

Original: This can easily result in divergence, as shown at the top of Figure 2.
Revised: This can easily result in divergence, as shown at the top of Figure 2.

Original: In contrast, the bottom of Figure 2 shows howthe updates are stable if using only the positive advantage actions.
Revised: In contrast, the bottom of Figure 2 shows how using only the positive advantage actions is stable; the policy Gaussian simply approximates the positive advantage action distribution.

Original: However, in this case the exploration variance shrinks prematurely, leading to poor ﬁnal convergence. 
Revised: However, in this case the exploration variance shrinks prematurely, leading to poor ﬁnal convergence. Note that

Revised: Figure 2 shows the convergence/divergence over policy gradient iterations; Appendix D provides an illustration of gradient steps within an iteration.

Original: Similar to Figure 1, Figure 2 depicts a ”stateless” didactic example, a special case of Algorithm 1 that allows clear visualization of how the action distribution adapts.
Revised: Similar to Figure 1, Figure 2 depicts a ”stateless” didactic example, a special case of Algorithm 1 that allows clear visualization of how the action distribution adapts.

Original: We set γ = 0 , which simpliﬁes the policy optimization objective E [ (cid:80) ∞ t =0 γ t r t ] =
Revised: We set γ = 0 , which simpliﬁes the policy optimization objective E [ (cid:80) ∞ t =0 γ t r t ] =

Original: E [ r 0 ] =
Revised: E [ r 0 ] =

Original: E
Revised: E

Original: [ r ( s , a )] , where s , a denote the ﬁrst state and action of an episode.
Revised: [ r ( s , a )] , where s , a denote the ﬁrst state and action of an episode.

Original: Further, we use a state-agnostic r ( a ) = − a T a .
Revised: Further, we use a state-agnostic r ( a ) = − a T a .

Original: Thus, Q π ( s , a ) = − a T a ,
Revised: Thus, Q π ( s , a ) = − a T a ,

Original: V π ( s ) =
Revised: V π ( s ) =

Original: V π = E
Revised: V π = E

Original: [ r ( a )]
Revised: [ r ( a )]

Original: ≈ 1 N (cid:80) N − 1 i =

Revised: ≈ 1 N (cid:80) N − 1 i =

Original: r ( a i ) , where i is episode index, and we can compute A π directly using Equation 3.
Revised: r ( a i ) , where i is episode index, and we can compute A π directly using Equation 3.

Original: As everything is agnostic of agent state, a simulator is not even needed and one can feed an arbitrary constant input to a policy network.
Revised: As everything is agnostic of agent state, a simulator is not even needed and one can feed an arbitrary constant input to a policy network.

Original: Equivalently, one can simply replace the policy network with optimized variables for the mean and variance of the action distribution.
Revised: Equivalently, one can simply replace the policy network with optimized variables for the mean and variance of the action distribution.

Original: 3. P ROXIMAL P OLICY O PTIMIZATION : V ARIANCE A DAPTATION P ROBLEM
Revised: 3. P ROXIMAL P OLICY O PTIMIZATION : V ARIANCE A DAPTATION P ROBLEM

Original: The basic idea of PPO is that one performs minibatch policy gradient updates for several epochs on the data from each iteration of Algorithm 1, while limiting changes to the policy such that it stays in the proximity of the sampled actions (Schulman et al.
Revised: The basic idea of PPO is that one performs minibatch policy gradient updates for several epochs on the data from each iteration of Algorithm 1, while limiting changes to the policy such that it stays in the proximity of the sampled actions (Schulman et al.

Original: PPO is a simpliﬁcation of Trust
Revised: PPO is a simpliﬁcation of Trust

Original: Region Policy Optimization (TRPO) (Schulman et al.
Revised: Region Policy Optimization (TRPO) (Schulman et al.

Original: (2015a)), which uses a more computationally expensive approach to achieve the same.
Revised: (2015a)), which uses a more computationally expensive approach to achieve the same.

Original: The original PPO paper proposes two variants: 1) using a loss function that penalizes KL-divergence between the old and updated policies, and 2) using the so-called clipped surrogate loss function that limits the likelihood ratio of old and updated policies π θ ( a i | s i ) /π old ( a i | s i ) .
Revised: The original PPO paper proposes two variants: 1) using a loss function that penalizes KL-divergence between the old and updated policies, and 2) using the so-called clipped surrogate loss function that limits the likelihood ratio of old and updated policies π θ ( a i | s i ) /π old ( a i | s i ) .

Original: In extensive testing,
Revised: In extensive testing,

Original: Schulman et al.
Revised: Schulman et al.

Original: (2017) concluded that the clipped surrogate loss with the clipping hyperparameter (cid:15) = 0 .
Revised: (2017) concluded that the clipped surrogate loss with the clipping hyperparameter (cid:15) = 0 .

Original: 2 is the recommended choice.
Revised: 2 is the recommended choice.

Original: This is also the version that we use in this paper in all PPO vs.
Revised: This is also the version that we use in this paper in all PPO vs.

Original: PPO-CMA comparisons.
Revised: PPO-CMA comparisons.

Original: Comparing Figure 1 and Figure 2 shows that PPO is more stable than vanilla policy gradient, but can result in somewhat similar premature shrinkage of exploration variance as in the case of only using the positive advantages for policy gradient.
Revised: Comparing Figure 1 and Figure 2 shows that PPO is more stable than vanilla policy gradient, but can result in somewhat similar premature shrinkage of exploration variance as in the case of only using the positive advantages for policy gradient.

Original: Although Schulman et al.
Revised: Although Schulman et al.

Original: (2017) demonstrated good results in MuJoCo problems with a Gaussian policy, the most impressive Roboschool results did not adapt the variance through gradient updates.
Revised: (2017) demonstrated good results in MuJoCo problems with a Gaussian policy, the most impressive Roboschool results did not adapt the variance through gradient updates.

Original: Instead, the policy network only output the Gaussian mean and a linearly decaying variance with manually tuned decay rate was used.
Revised: Instead, the policy network only output the Gaussian mean and a linearly decaying variance with manually tuned decay rate was used.

Original: 3. C OVARIANCE M ATRIX A DAPTATION E VOLUTION S TRATEGY
Revised: 3. C OVARIANCE M ATRIX A DAPTATION E VOLUTION S TRATEGY

Original: The Evolution Strategies (ES) community has worked on similar variance adaptation and Gaussian exploration problems for decades, culminating in the widely used CMA-ES optimization method and its recent variants (Hansen & Ostermeier (2001);
Revised: The Evolution Strategies (ES) community has worked on similar variance adaptation and Gaussian exploration problems for decades, culminating in the widely used CMA-ES optimization method and its recent variants (Hansen & Ostermeier (2001);

Original: Hansen (2006); Beyer & Sendhoff (2017);
Revised: Hansen (2006); Beyer & Sendhoff (2017);

Original: Loshchilov et al.
Revised: Loshchilov et al.

Original: CMA-ES is a black-box optimization method for ﬁnding a parameter vector x that maximizes some objective or ﬁtness function f ( x ) .
Revised: CMA-ES is a black-box optimization method for ﬁnding a parameter vector x that maximizes some objective or ﬁtness function f ( x ) .

Original: The CMA-ES core iteration is summarized in Algorithm 2.
Revised: The CMA-ES core iteration is summarized in Algorithm 2.

Original: Algorithm
Revised: Algorithm

Original: 1: for iteration=1,2,... do 2:
Revised: 1: for iteration=1,2,... do 2:

Original: Draw samples x i ∼ N ( µ , C ) .
Revised: Draw samples x i ∼ N ( µ , C ) .

Original: Evaluate f ( x i ) 4:
Revised: Evaluate f ( x i ) 4:

Original: Sort the samples based on f ( x i ) and compute weights w i based on the ranks such that best samples have highest weights.
Revised: Sort the samples based on f ( x i ) and compute weights w i based on the ranks such that best samples have highest weights.

Original: Update µ and C using the samples and weights.
Revised: Update µ and C using the samples and weights.

Original: 6: end for
Revised: 6: end for

Original: Using the default CMA-ES parameters, the weights of the worst 50% of samples are set to 0, i.e., those samples are pruned and have no effect.
Revised: Using the default CMA-ES parameters, the weights of the worst 50% of samples are set to 0, i.e., those samples are pruned and have no effect.

Original: The mean µ is updated as a weighted average of the samples, but the covariance update is more involved.
Revised: The mean µ is updated as a weighted average of the samples, but the covariance update is more involved.

Original: Although CMA-ES converges to a single optimum and the sampling distribution is unimodal, the method performs remarkably well on multimodal and/or noisy functions such as Rastrigin if using enough samples per iteration (Hansen & Kern (2004)).
Revised: Although there is no convergence quarantee, CMA-ES performs remarkably well on multimodal and/or noisy functions such as Rastrigin if using enough samples per iteration (Hansen & Kern (2004)).

Original: For full details of the update process, the reader is referred to Hansen’s excellent tutorial (Hansen (2016)).
Revised: For full details of the update process, the reader is referred to Hansen’s excellent tutorial (Hansen (2016)).

Original: Usually, CMA-ES and other ES variants are applied to policy optimization in the form of neuroevolution , i.e., directly optimizing the policy network parameters, x ≡ θ , with f ( x ) evaluated as the sum of rewards over one or more simulation episodes (Wang et al.
Revised: Usually, CMA-ES and other ES variants are applied to policy optimization in the form of neuroevolution , i.e., directly optimizing the policy network parameters, x ≡ θ , with f ( x ) evaluated as the sum of rewards over one or more simulation episodes (Wang et al.

Original: ; Geijtenbeek et al.
Revised: ; Geijtenbeek et al.

Original: Such et al.
Revised: Such et al.

Original: This is both a beneﬁt and a drawback; neuroevolution is simple to implement and requires no critic network, but on the other hand, the sum of rewards may not be very informative in guiding the optimization.
Revised: This is both a beneﬁt and a drawback; neuroevolution is simple to implement and requires no critic network, but on the other hand, the sum of rewards may not be very informative in guiding the optimization.

Original: Especially in long episodes, some explored actions may be good and should be learned, but the sum may still be low.
Revised: Especially in long episodes, some explored actions may be good and should be learned, but the sum may still be low.

Original: In this paper, we are interested in whether ideas from CMA-ES could improve the sampling of actions in RL, using x ≡ a .
Revised: In this paper, we are interested in whether ideas from CMA-ES could improve the sampling of actions in RL, using x ≡ a .

Original: Instead of a single action optimization task, RL is in effect solving multiple action optimization tasks in parallel, one for each possible state.
Revised: Instead of a single action optimization task, RL is in effect solving multiple action optimization tasks in parallel, one for each possible state.

Original: The accumulation of rewards over time further complicates matters.
Revised: The accumulation of rewards over time further complicates matters.

Original: However, we can directly apply CMA-ES to our ”stateless” didactic example with f ( x ) ≡ r ( a ) , illustrated in Figure 3.
Revised: However, we can directly apply CMA-ES to our ”stateless” didactic example with f ( x ) ≡ r ( a ) , illustrated in Figure 3.

Original: Comparing this to Figure 2 reveals two key insights:
Revised: Comparing this to Figure 2 reveals two key insights:

Original: • CMA-ES prunes samples based on sorted ﬁtness values and discards the worst half.
Revised: • CMA-ES prunes samples based on sorted ﬁtness values and discards the worst half.

Original: This is visually and conceptually similar to computing the policy gradient only using positive advantages,
Revised: This is visually and conceptually similar to computing the policy gradient only using positive advantages,

Original: i
Revised: i

Original: e., pruning action samples based on advantage sign.
Revised: e., pruning action samples based on advantage sign.

Original: Thus, the advantage function can be thought analogous to the ﬁtness function, although in policy optimization with a continuous state space, one can’t directly enumerate, sort, and prune the actions and advantages for each state.
Revised: Thus, the advantage function can be thought analogous to the ﬁtness function, although in policy optimization with a continuous state space, one can’t directly enumerate, sort, and prune the actions and advantages for each state.

Original: • Unlike PPO and policy gradient with only positive advantages, CMA-ES avoids the premature variance shrinkage problem.
Revised: • Unlike PPO and policy gradient with only positive advantages, CMA-ES avoids the premature variance shrinkage problem.

Original: Instead, it increases the variance on objective function slopes to speed up progress and shrinks the variance once the optimum has been found.
Revised: Instead, it increases the variance on objective function slopes to speed up progress and shrinks the variance once the optimum has been found.

Original: This can be thought as a form of momentum that acts indirectly through the exploration variance.
Revised: This can be thought as a form of momentum that acts indirectly through the exploration variance.

Original: Considering the above, a good policy optimization approach might be to only use actions with positive advantages, if one could just borrow the variance adaptation techniques of CMA-ES.
Revised: Considering the above, a good policy optimization approach might be to only use actions with positive advantages, if one could just borrow the variance adaptation techniques of CMA-ES.

Original: In the next section, we show that this is indeed possible, resulting in our proposed PPO-CMA algorithm.
Revised: In the next section, we show that this is indeed possible, resulting in our proposed PPO-CMA algorithm.

Original:  PPO-CMA
Revised:  PPO-CMA

Original: Our proposed PPO-CMA algorithm is summarized in Algorithm 3.
Revised: Our proposed PPO-CMA algorithm is summarized in Algorithm 3.

Original: Source code is available at
Revised: Source code is available at

Original: GitHub 3 .
Revised: GitHub 2 .

Original: PPO-CMA is simple to implement, only requiring  minor changes to vanilla PPO, deﬁned on Algorithm 3 lines 8-10:
Revised: PPO-CMA is simple to implement, only requiring the following minor changes to PPO:

Original: • We use the standard policy gradient loss in Equation 6.
Revised: • We use the standard policy gradient loss in Equation 6 and train only on actions with positive advantage estimates.

Original: • We train only on actions with positive advantage estimates, i.e., we implement CMA-ES -style pruning of actions, but based on advantage sign instead of sorted ﬁtness function values. As explained in Section 3.1, this also ensures that the policy stays in  the proximity of the data without an explicit PPO-style KL-divergence penalty or the clipped surrogate loss function.
Revised: This means that 1) we implement CMA-ES - style pruning of actions, but based on advantage sign instead of sorted ﬁtness function values, and 2) similar to CMA-ES, the updated exploration Gaussian mean equals a weighted mean of the actions; in CMA-ES, the weights are based on the ﬁtness sorting, whereas we use the advantages as the weights. The updated Gaussian variance is also affected by the two features below:

Original: • We use separate neural networks for policy mean and variance, trained in separate passes.
Revised: • We use separate neural networks for policy mean and variance, trained in separate passes.

Revised: This implements the CMA-ES two-step update, as elaborated in Section 4.1.

Original:  Wealso maintain a history of training data over H iterations, used for training the variance network.
Revised: • We maintain a history of training data over H iterations, used for training the variance network.

Revised: This approximates the CMA-ES evolution path heuristic, as explained in Section 4.2.

Original: As elaborated below, these result in the  CMA-ES -style variance adaptation behavior shown in Figure 1.
Revised: Together, the features above result in the emergence of the CMA-ES -style variance adaptation behavior shown in Figure 1.

Revised: Section 4.3 also presents an optional technique for utilizing negative advantage actions, resulting in a variant of PPO-CMA we call PPO-CMA-m.

Original: Algorithm
Revised: Algorithm

Original: 1: for iteration=1,2,... do 2: while iteration simulation budget N not exceeded do 3:
Revised: 1: for iteration=1,2,... do 2: while iteration simulation budget N not exceeded do 3:

Original: Reset the simulation to a (random) initial state 4:
Revised: Reset the simulation to a (random) initial state 4:

Original: Run agent on policy π θ for T timesteps or until a terminal state 5: end while 6:
Revised: Run agent on policy π θ for T timesteps or until a terminal state 5: end while 6:

Original: Train critic network for K epochs using the experience from the current iteration 7:
Revised: Train critic network for K epochs using the experience from the current iteration 7:

Original: Estimate advantages A π using GAE (Schulman et al.
Revised: Estimate advantages A π using GAE (Schulman et al.

Original: (2015b))
Revised: (2015b))

Original: Clip negative advantages to zero, A π ← max( A π , 0) 9:
Revised: Clip negative advantages to zero, A π ← max( A π , 0) 9:

Original: Train policy variance for K epochs using experience from past H iterations and Eq.10:
Revised: Train policy variance for K epochs using experience from past H iterations and Eq.10:

Original: Train policy mean for K epochs using the experience from this iteration and Eq.11: end for
Revised: Train policy mean for K epochs using the experience from this iteration and Eq.11: end for

Original: 4. T WO - STEP UPDATING OF MEAN AND COVARIANCE
Revised: 4. T WO - STEP UPDATING OF MEAN AND COVARIANCE

Original: Superﬁcially, the core iteration loop of CMA-ES is similar to other optimization approaches with recursive sampling and distribution ﬁtting such as the Cross-Entropy Method (De Boer et al.
Revised: Superﬁcially, the core iteration loop of CMA-ES is similar to other optimization approaches with recursive sampling and distribution ﬁtting such as the Cross-Entropy Method (De Boer et al.

Original: and Estimation of Multivariate Normal Algorithm (EMNA) (Larra ˜ naga & Lozano (2001)).
Revised: and Estimation of Multivariate Normal Algorithm (EMNA) (Larra ˜ naga & Lozano (2001)).

Original: However, there is a crucial difference:  CMA-ES ﬁrst updates the covariance and only then updates the mean (Hansen (2016)).
Revised: However, there is a crucial difference: in the so-called Rank- µ update, CMA-ES ﬁrst updates the covariance and only then updates the mean (Hansen (2016)).

Original: This has the effect of elongating the exploration distribution along the best search directions instead of shrinking the variance prematurely, as shown in Figure 4.
Revised: This has the effect of elongating the exploration distribution along the best search directions instead of shrinking the variance prematurely, as shown in Figure 4.

Original: This has also been shown to correspond to a natural gradient update of the exploration distribution (Ollivier et al.
Revised: This has also been shown to correspond to a natural gradient update of the exploration distribution (Ollivier et al.

Original: In PPO-CMA, we implement the two-step update by using separate neural networks for the mean and variance.
Revised: In PPO-CMA, we implement the two-step update by using separate neural networks for the mean and variance.

Original: We ﬁrst train the variance network while keeping the mean ﬁxed, and then vice versa, using the Gaussian policy gradient loss in Equation 6.
Revised: We ﬁrst train the variance network while keeping the mean ﬁxed, and then vice versa, using the Gaussian policy gradient loss in Equation 6.

Original:  based on weighted samples, D) CMA-ES update, where covariance is estimated before updating the mean.
Revised: ., estimating mean and covariance based on weighted samples, D) CMA-ES update, where covariance is estimated before updating the mean.

Original: 4. E VOLUTION PATH
Revised: 4. E VOLUTION PATH

Original: CMA-ES also features the so-called evolution path heuristic, where a component α p ( i ) p ( i ) T is added to the covariance, where α is a scalar, the ( i ) superscript denotes iteration index, and p is the evolution path (Hansen (2016)):
Revised: CMA-ES also features the so-called evolution path heuristic, where a component α p ( i ) p ( i ) T is added to the covariance, where α is a scalar, the ( i ) superscript denotes iteration index, and p is the evolution path (Hansen (2016)):

Original: [Equation]
Revised: [Equation]

Original: Although the exact computation of the default β 0 and β 1 multipliers is rather involved, Equationessentially amounts to ﬁrst-order low-pass ﬁltering of the steps taken by the distribution mean between iterations.
Revised: Although the exact computation of the default β 0 and β 1 multipliers is rather involved, Equationessentially amounts to ﬁrst-order low-pass ﬁltering of the steps taken by the distribution mean between iterations.

Original: When CMA-ES progresses along a continuous slope of the ﬁtness landscape,
Revised: When CMA-ES progresses along a continuous slope of the ﬁtness landscape,

Original: ||
Revised: ||

Original: p || is large, and the covariance is elongated and exploration is increased along the progress direction.
Revised: p || is large, and the covariance is elongated and exploration is increased along the progress direction.

Original: Near convergence, when CMA-ES zigzags around the optimum in a random walk, || p
Revised: Near convergence, when CMA-ES zigzags around the optimum in a random walk, || p

Original: ||
Revised: ||

Original: ≈ 

Revised: ≈

Original: and the evolution path heuristic has no effect.
Revised: and the evolution path heuristic has no effect.

Original: In policy optimization, policy mean and variance depend on the agent state and should be adapted differently at different parts of the state space.

Original: For some states, the policy may already be nearly converged while exploration is still needed for other states.

Original: Thus, in principle, one would need yet another neural network to maintain and approximate a state-dependent p ( s ) , which might be unstable.

Original: Fortunately, one can achieve a similar effect bysimply keeping a history of H iterations of data and sampling the variance training minibatches from the history instead of only the latest data.
Revised: PPO-CMA approximates the evolution path heuristic by keeping a history of H iterations of data and sampling the variance training minibatches from the history instead of only the latest data.

Original: Same as the original evolution path heuristic, this elongates the variance for a given  if the mean is moving in a consistent 
Revised: Same as the original evolution path heuristic, this elongates the variance for a given state if the mean is moving in a consistent direction.

Revised: We do not implement the CMA-ES evolution path heuristic directly, because this would need yet another neural network to maintain and approximate a statedependent p ( s ) .

Revised: Similar to exploration mean and variance, p is a CMA-ES algorithm state variable; in policy optimization, such variables become functions of agent state and need to be encoded as neural network weights.

Original: Appendix B provides an empirical analysis of the effect of different H values.
Revised: Appendix B provides an empirical analysis of the effect of different H values.

Original: Values larger than 1 minimize the probability of low-reward outliers, and PPO-CMA does not appear to be highly sensitive to the exact choice.
Revised: Values larger than 1 minimize the probability of low-reward outliers, and PPO-CMA does not appear to be highly sensitive to the exact choice.

Revised: 4. L EVERAGING NEGATIVE ADVANTAGES : PPO-CMA- M

Revised: Disregarding negative-advantage actions may potentially discard valuable information.

Original: E VALUATION
Revised: We observe that assuming linearity of advantage around the current policy mean µ ( s i ) , it is possible to mirror negative-advantage actions about the mean to convert them to positive-advantage actions. More precisely, we set a (cid:48) i = 2 µ ( s i ) − a i , A π ( a (cid:48) i ) = − A π ( a i ) ψ ( a i , s i ) , where ψ ( a i , s i ) is a Gaussian kernel (we use the same shape as the policy) that assigns less weight to actions far from the mean.

Revised: We call this PPO-CMA variant PPO-CMA-m.

Revised: The mirroring drives the policy Gaussian away from worse than average actions, but in a way that does not diverge (see Appendix D for details).

Revised: If the linearity assumption holds, the mirroring effectively doubles the available experience.

Revised: In the CMA-ES literature, a related technique is to use a negative covariance matrix update procedure (Jastrebski & Arnold (2006);

Revised: Hansen & Ros (2010)).

Revised:  E VALUATION

Original: We evaluate PPO-CMA using the 7-task MuJoCo-1M benchmark used by Schulman et al.
Revised: We evaluate PPO-CMA using the 7-task MuJoCo-1M benchmark used by Schulman et al.

Original: (2017) plus the more difﬁcult 3D humanoid locomotion task.
Revised: (2017) plus the more difﬁcult 3D humanoid locomotion task.

Original: We performed the following experiments:
Revised: We performed the following experiments:

Original: • The blue and red curves in Figure 5 visualize the results using PPO-CMA and PPO with exactly same implementation and hyperparameters.
Revised: • The green, blue, and red curves in Figure 5 visualize the results using PPO-CMA, PPOCMA-m, and PPO with exactly same implementation and hyperparameters.

Original: This ensures that the differences are due to algorithm changes instead of hyperparameters.
Revised: This ensures that the differences are due to algorithm changes instead of hyperparameters.

Original: PPO-CMA performs better  in 7 out of 8 tasks.
Revised: PPO-CMA and PPO-CMA-m perform better than PPO in 7 out of 8 tasks.

Original: The hyperparameters are detailed in Appendix B.  • To test whether our results generalize to other hyperparameter settings, we generatedrandom hyperparameter combinations as detailed in Appendix C, and used them to train the 2D walker using both PPO and PPO-CMA.
Revised: The hyperparameters are detailed in Appendix B. PPO-CMA and PPO-CMA-m yield roughly similar results, except for the humanoid task where PPO-CMA-m is clearly superior. • To test whether our results generalize to other hyperparameter settings, we generatedrandom hyperparameter combinations as detailed in Appendix C, and used them to train the 2D walker using both PPO and PPO-CMA.

Original: After 1M simulation steps, PPO-CMA and PPO achieved mean episode reward of 393 and 161, respectively.
Revised: After 1M simulation steps, PPO-CMA and PPO achieved mean episode reward of 393 and 161, respectively.

Original: The standard deviations were 208 and 81.
Revised: The standard deviations were 208 and 81.

Original: A Welch t-test indicates that the difference is statistically signiﬁcant ( p < 0 . 001 ).
Revised: A Welch t-test indicates that the difference is statistically signiﬁcant ( p < 0 . 001 ).

Original: Although the individual training runs have high reward variance, PPO-CMA wins in 47 out of 50 runs.
Revised: Although the individual training runs have high reward variance, PPO-CMA wins in 47 out of 50 runs.

Original: • Reinforcement learning algorithms are notoriously sensitive to implementation details, and even closely related algorithms might have different optimal parameters.
Revised: • Reinforcement learning algorithms are notoriously sensitive to implementation details, and even closely related algorithms might have different optimal parameters.

Original: Thus, to compare our results against a ﬁnetuned PPO implementation, Figure 5 also shows the performance 
Revised: Thus, to compare our results against a ﬁnetuned PPO implementation, Figure 5 also shows the performance of OpenAI’s baseline PPO 3 .

Revised: In this case, PPO-CMA produces better results than PPO inout of 8 tasks.

Original: Overall, PPO-CMA  often progresses somewhat slower  but appears less likely to diverge or stop improving.
Revised: Overall, PPO-CMA and PPO-CMA-m often progress somewhat slower than PPO, but appear less likely to diverge or stop improving.

Original: Appendix A provides further comparisons that also visualize the variance adaptation behavior.
Revised: Appendix A provides further comparisons that also visualize the variance adaptation behavior.

Original: To augment the visual comparison, Appendix C presents statistical signiﬁcance testing results.
Revised: To augment the visual comparison, Appendix C presents statistical signiﬁcance testing results.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: In addition to PPO, our work is closely related to Continuous Actor Critic Learning Automaton (CACLA) (van Hasselt & Wiering (2007)).
Revised: In addition to PPO, our work is closely related to Continuous Actor Critic Learning Automaton (CACLA) (van Hasselt & Wiering (2007)).

Original: Similar to PPO-CMA, CACLA uses the sign of the advantage estimate – in their case the TD-residual – in the updates, shifting policy mean towards actions with positive sign.
Revised: Similar to PPO-CMA, CACLA uses the sign of the advantage estimate – in their case the TD-residual – in the updates, shifting policy mean towards actions with positive sign.

Original: The paper also observes that using actions with negative advantages can have an adverse effect.
Revised: The paper also observes that using actions with negative advantages can have an adverse effect.

Original: In light of our discussion of how only using positive advantage actions guarantees that the policy stays in the proximity of the collected experience, CACLA can be viewed as an early Proximal Policy Optimization approach, which we extend with CMA-ES style variance adaptation.
Revised: In light of our discussion of how only using positive advantage actions guarantees that the policy stays in the proximity of the collected experience, CACLA can be viewed as an early Proximal Policy Optimization approach, which we extend with CMA-ES style variance adaptation.

Original: Although PPO is based on a traditional policy gradient formulation, there is a line of research suggesting that the so-called natural gradient can be more efﬁcient in optimization (Amari (1998);
Revised: Although PPO is based on a traditional policy gradient formulation, there is a line of research suggesting that the so-called natural gradient can be more efﬁcient in optimization (Amari (1998);

Original: Wierstra et al.
Revised: Wierstra et al.

Original: (2008); Ollivier et al.
Revised: (2008); Ollivier et al.

Original: Through the connection between CMA-ES and natural gradient, PPO-CMA is related to various natural gradient RL methods (Kakade (2002); Peters &
Revised: Through the connection between CMA-ES and natural gradient, PPO-CMA is related to various natural gradient RL methods (Kakade (2002); Peters &

Original: Schaal (2008); Wu et al.
Revised: Schaal (2008); Wu et al.

Original: (2017)), although the evolution path heuristic is not motivated from the natural gradient perspective (Ollivier et al.
Revised: (2017)), although the evolution path heuristic is not motivated from the natural gradient perspective (Ollivier et al.

Original: PPO represents on-policy RL methods, i.e., experience is assumed to be collected on-policy and thus must be discarded after the policy is updated.
Revised: PPO represents on-policy RL methods, i.e., experience is assumed to be collected on-policy and thus must be discarded after the policy is updated.

Original: Theoretically, off-policy RL should allow better sample efﬁciency through the reuse of old experience, often implemented using an experience replay buffer, introduced by Lin (1993) and recently brought back to fashion (e.g, Mnih et al.
Revised: Theoretically, off-policy RL should allow better sample efﬁciency through the reuse of old experience, often implemented using an experience replay buffer, introduced by Lin (1993) and recently brought back to fashion (e.g, Mnih et al.

Original: Lillicrap et al.
Revised: Lillicrap et al.

Original: ; Schaul et al.
Revised: ; Schaul et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: PPO-CMA can be considered as a hybrid method, since the policy mean is updated using on-policy experience, but the history or replay buffer for the variance update also includes older off-policy experience.
Revised: PPO-CMA can be considered as a hybrid method, since the policy mean is updated using on-policy experience, but the history or replay buffer for the variance update also includes older off-policy experience.

Original: In addition to neuroevolution (discussed in Section 3.3), CMA-ES has been applied to continuous control in the form of trajectory optimization.
Revised: In addition to neuroevolution (discussed in Section 3.3), CMA-ES has been applied to continuous control in the form of trajectory optimization.

Original: In this case, one searches for a sequence of optimal controls given an initial state, and CMA-ES and other sampling-based approaches (Al Borno et al.
Revised: In this case, one searches for a sequence of optimal controls given an initial state, and CMA-ES and other sampling-based approaches (Al Borno et al.

Original: ; H ¨ am ¨ al ¨ ainen et al.
Revised: ; H¨am¨al¨ainen et al.

Original: Liu et al.
Revised: Liu et al.

Original: complement variants of Differential
Revised: complement variants of Differential

Original: Dynamic Programming, where the optimization utilizes gradient information (Tassa et al.
Revised: Dynamic Programming, where the optimization utilizes gradient information (Tassa et al.

Original: Although trajectory optimization approaches have demonstrated impressive results with complex humanoid characters, they require more computing resources in run-time.
Revised: Although trajectory optimization approaches have demonstrated impressive results with complex humanoid characters, they require more computing resources in run-time.

Original: Finally, it should be noted that PPO-CMA falls in the domain of model-free reinforcement learning approaches.
Revised: Finally, it should be noted that PPO-CMA falls in the domain of model-free reinforcement learning approaches.

Original: In contrast, there are several model-based methods that learn approximate models of the simulation dynamics and use the models for policy optimization, potentially requiring less simulated or real experience.
Revised: In contrast, there are several model-based methods that learn approximate models of the simulation dynamics and use the models for policy optimization, potentially requiring less simulated or real experience.

Original: Both ES and RL approaches can be used for the optimization (Chatzilygeroudis et al.
Revised: Both ES and RL approaches can be used for the optimization (Chatzilygeroudis et al.

Original: Model-based algorithms are an active area of research, with recent work demonstrating excellent results in limited MuJoCo benchmarks (Chua et al.
Revised: Model-based algorithms are an active area of research, with recent work demonstrating excellent results in limited MuJoCo benchmarks (Chua et al.

Original: but model-free approaches still dominate the most complex continuous problems such as humanoid movement.
Revised: but model-free approaches still dominate the most complex continuous problems such as humanoid movement.

Original: For a more in-depth review of continuous control policy optimization methods the reader is referred to Sigaud & Stulp (2018) or the older but mathematically more detailed Deisenroth et al.
Revised: For a more in-depth review of continuous control policy optimization methods the reader is referred to Sigaud & Stulp (2018) or the older but mathematically more detailed Deisenroth et al.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Proximal Policy Optimization (PPO) is a simple, powerful, and widely used model-free reinforcement learning approach.
Revised: Proximal Policy Optimization (PPO) is a simple, powerful, and widely used model-free reinforcement learning approach.

Original: However, we have shown that in continous control with Gaussian policy,
Revised: However, we have shown that in continous control with Gaussian policy,

Original: PPO can adapt the exploration variance in an unreliable manner, which explains how PPO may converge slowly or get stuck in local optima.
Revised: PPO can adapt the exploration variance in an unreliable manner, which explains how PPO may converge slowly or get stuck in local optima.

Original: As a solution to the variance adaptation problem, we have proposed the PPO-CMA algorithm that implements two-step update and evolution path heuristics inspired by the CMA-ES black-box optimization method.
Revised: As a solution to the variance adaptation problem, we have proposed the PPO-CMA algorithm that implements two-step update and evolution path heuristics inspired by the CMA-ES black-box optimization method.

Original: This results in an improved PPO version that is simple to implement but beats the original method in many MuJoCo tasks, is less prone to getting stuck in local optima, and provides a new link between RL and ES approaches to policy optimization.
Revised: This results in an improved PPO version that is simple to implement but beats the original method in many MuJoCo tasks and is less prone to getting stuck in local optima.

Original: Additionally, our simulations in Appendix A show that PPO’s premature convergence can also be prevented with simple clipping of the policy variance or using the entropy loss term proposed by
Revised: Additionally, our simulations in Appendix A show that PPO’s premature convergence can also be prevented with simple clipping of the policy variance or using the entropy loss term proposed by

Original: Schulman et al.
Revised: Schulman et al.

Original: However, the clipping limit or entropy loss weight needs delicate ﬁnetuning and both techniques also result in a more noisy ﬁnal policy.
Revised: However, the clipping limit or entropy loss weight needs delicate ﬁnetuning and both techniques also result in a more noisy ﬁnal policy.

Original: On a more general level, one can draw the following conclusions and algorithm design insights from our work:
Revised: On a more general level, one can draw the following conclusions and algorithm design insights from our work:

Original: • To understand the differences, similarities, and problems of policy optimizationmethods, it can be useful to visualize ”stateless” special cases such as the one in Figure 1.
Revised: • We provide a new link between RL and ES approaches to policy optimization. Typically, ES is used for policy optimization in the form of neuroevolution,

Original: PPO’s problems were not at all clear to us until we created the visualizations, originally meant for teaching.
Revised: i

Original: • It can be possible to adapt a black-box optimization method like CMA-ES  for policy optimization by1) treating the advantage function as the ﬁtness function, 2) approximating  sorting and pruning operations as clipping advantage values below a limit to zero (sorting is not possible because with a continuous state space, one can’t enumerate all the actions sampled for a given state), and 3) considering algorithm state such as exploration mean and variance as functions of agent state instead of plain variables, and encoding them as weights of multiple neural networks to allow independent updates.
Revised: e., directly sampling the neural network weights. In contrast, we demonstrate how CMA-ES can be used to sample actions, but such that the sampling Gaussian is conditional on agent state, implemented using the policy network. Essentially, multiple parallel CMA-ES optimizations of actions are done for different agent states, and the neural networks store and interpolate algorithm state – exploration mean and variance – as a function of agent state. This is enabled by treating the advantage function as the ﬁtness function, and approximating the CMA-ES sorting and pruning operations by clipping advantage values below a limit to zero.

Original: Essentially, the ﬁtnessadvantage analogue allows one to think that multiple parallel CMA-ES optimizations of actions are being done fordifferent agent states, and the neural networks store and interpolate algorithm state as a function of agent state.
Revised: Sorting actions based on advantage or ﬁtness is not possible because with a continuous state space, one can’t enumerate all the actions sampled for a given state.

Original: • Our work highlights how gradient-based RL can have problems if the gradient affects ex - ploration in subsequent iterations, which is the case with Gaussian policies.
Revised: • Our work highlights how gradient-based RL can have problems if the gradient affects exploration in subsequent iterations, which is the case with Gaussian policies.

Original: The fundamental problem is that a gradient that causes an increase in the expected rewards does not quarantee further increases in subsequent iterations.
Revised: The fundamental problem is that a gradient that causes an increase in the expected rewards does not quarantee further increases in subsequent iterations.

Original: Instead, one should adapt exploration such that it provides good results over the whole training process.
Revised: Instead, one should adapt exploration such that it provides good results over the whole training process.

Original: We have shown that one way to achieve this can be through an approximation of CMA-ES variance adaptation.
Revised: We have shown that one way to achieve this can be through an approximation of CMA-ES variance adaptation.

Revised: • To understand the differences, similarities, and problems of policy optimization methods, it can be useful to visualize ”stateless” special cases such as the one in Figure 1.

Revised: PPO’s problems were not at all clear to us until we created the visualizations, originally meant for teaching.

Original: Thinking of the advantage function not as a way to compute accurate gradient but as a tool for pruning the learned actions also begs the question of what other pruning methods might be applicable.
Revised: Thinking of the advantage function not as a way to compute accurate gradient but as a tool for pruning the learned actions also begs the question of what other pruning methods might be applicable.

Original: Presently, we are experimenting with continuous control Monte Carlo Tree Search methods (e g.,
Revised: Presently, we are experimenting with continuous control Monte Carlo Tree Search methods (e.g.,

Original: Rajam¨aki & H¨am¨al¨ainen (2018)) for better exploration and pruning.
Revised: Rajam¨aki & H¨am¨al¨ainen (2018)) for better exploration and pruning.

Original: A F URTHER ANALYSIS OF PPO’ S VARIANCE ADAPTATION A. V ARIANCE ADAPTATION IN H OPPER -V 
Revised: A F URTHER ANALYSIS OF PPO’ S VARIANCE ADAPTATION A. V ARIANCE ADAPTATION IN H OPPER -V

Original: To further inspect the variance adaptation problem we use the Hopper-V2 MuJoCo environment, where the monopedal agent gets rewards for traveling forward while keeping the upper part of the body upright.
Revised: To further inspect the variance adaptation problem we use the Hopper-V2 MuJoCo environment, where the monopedal agent gets rewards for traveling forward while keeping the upper part of the body upright.

Original: There is a local optimum that tends to attract policy optimization: Instead of discovering a stable gait, the agent may greedily lunge forward and fall, either right at episode start or after one or a few steps.
Revised: There is a local optimum that tends to attract policy optimization: Instead of discovering a stable gait, the agent may greedily lunge forward and fall, either right at episode start or after one or a few steps.

Original: Figure 6 shows the training trajectories from 20 independent runs with different random seeds.
Revised: Figure 6 shows the training trajectories from 20 independent runs with different random seeds.

Original: The ﬁgure shows both the growth of rewards and adaptation of variance, the latter plotted as the policy standard deviation averaged over all episodes and explored actions of each iteration.
Revised: The ﬁgure shows both the growth of rewards and adaptation of variance, the latter plotted as the policy standard deviation averaged over all episodes and explored actions of each iteration.

Original: The ﬁgure also includes results of the same using our proposed PPO-CMA algorithm with the same hyperparameters.
Revised: The ﬁgure also includes results of the same using our proposed PPO-CMA algorithm with the same hyperparameters.

Original: Figure 6 uses red color to denote the training runs where the policy failed to escape the local optimum.
Revised: Figure 6 uses red color to denote the training runs where the policy failed to escape the local optimum.

Original: Looking at the reward and variance curves together, one sees that PPO produces worse results than PPO-CMA and PPO’s average standard deviation decreases faster.
Revised: Looking at the reward and variance curves together, one sees that PPO produces worse results than PPO-CMA and PPO’s average standard deviation decreases faster.

Original: Some of PPO’s failure cases also exhibit erratic sharp changes in standard deviation.
Revised: Some of PPO’s failure cases also exhibit erratic sharp changes in standard deviation.

Original: PPO-CMA has no clear low-reward outliers adapts variance robustly and consistently.
Revised: PPO-CMA has no clear low-reward outliers adapts variance robustly and consistently.

Original: Many PPO implementations, including OpenAI’s baseline, use some form of automatic normalization of state observations, as some MuJoCo environments have observation variables with both very large and very small scales.
Revised: Many PPO implementations, including OpenAI’s baseline, use some form of automatic normalization of state observations, as some MuJoCo environments have observation variables with both very large and very small scales.

Original: Figure 7 shows 20 Hopper training curves with the same normalization we use in our evaluation in Section 5.
Revised: Figure 7 shows 20 Hopper training curves with the same normalization we use in our evaluation in Section 5.

Original: PPO-CMA is more robust to the normalization.
Revised: PPO-CMA is more robust to the normalization.

Original: A. C LIPPING AND INITIALIZING THE POLICY NETWORK
Revised: A. C LIPPING AND INITIALIZING THE POLICY NETWORK

Original: Similar to previous work, we use a fully connected policy network with a linear output layer and treat the variance output as log variance v = log( c ) .
Revised: Similar to previous work, we use a fully connected policy network with a linear output layer and treat the variance output as log variance v = log( c ) .

Original: In our initial tests with PPO, we ran into numerical precision errors which could be prevented by soft-clipping the mean as µ clipped = a min + ( a max − a min ) ⊗ σ ( µ ) , where a max and a min are the action space limits.
Revised: In our initial tests with PPO, we ran into numerical precision errors which could be prevented by soft-clipping the mean as µ clipped = a min + ( a max − a min ) ⊗ σ ( µ ) , where a max and a min are the action space limits.

Original: Similarly, we clip the log variance as v clipped = v min + ( v max − v min ) ⊗ σ ( v ) , where v min is a lower limit parameter, and v max = 2 log(  max − a min ) .
Revised: Similarly, we clip the log variance as v clipped = v min + ( v max − v min ) ⊗ σ ( v ) , where v min is a lower limit parameter, and v max = 2 log( a max − a min ) .

Original: To ensure a good initialization, we pretrain the policy in supervised manner with randomly sampled observation vectors and a ﬁxed target output v clipped = 2 log(0 .
Revised: To ensure a good initialization, we pretrain the policy in supervised manner with randomly sampled observation vectors and a ﬁxed target output v clipped = 2 log(0 .

Original: 5( a max − a min )) and µ clipped = 0 .
Revised: 5( a max − a min )) and µ clipped = 0 .

Original: ( a max + a min ) .
Revised: ( a max + a min ) .

Original: The rationale behind this choice is that the initial exploration Gaussian should cover the whole action space but the variance should be lower than the upper clipping limit to prevent zero gradients.
Revised: The rationale behind this choice is that the initial exploration Gaussian should cover the whole action space but the variance should be lower than the upper clipping limit to prevent zero gradients.

Original: Without the pretraining, nothing quarantees sufﬁcient exploration for all observed states.
Revised: Without the pretraining, nothing quarantees sufﬁcient exploration for all observed states.

Original: One might think that premature convergence could be prevented simply by increasing the lower clipping limit.
Revised: One might think that premature convergence could be prevented simply by increasing the lower clipping limit.

Original: However, Figure 8 shows how a fairly high lower limit – considering the valid action range (-1,1) – is needed for the policy’s standard deviation in order to ensure that all training runs escape the local optimum.
Revised: However, Figure 8 shows how a fairly high lower limit – considering the valid action range (-1,1) – is needed for the policy’s standard deviation in order to ensure that all training runs escape the local optimum.

Original: This is not ideal, as it causes considerable motor noise that does not vanish as training progresses.
Revised: This is not ideal, as it causes considerable motor noise that does not vanish as training progresses.

Original: Excessive motor noise is undesireable especially in animation and robotics applications.
Revised: Excessive motor noise is undesireable especially in animation and robotics applications.

Original: Finetuning the limit is also tedious, as to large values rapidly lead to worse results.
Revised: Finetuning the limit is also tedious, as to large values rapidly lead to worse results.

Original: A. E NTROPY LOSS
Revised: A. E NTROPY LOSS

Original: The original PPO paper (Schulman et al.
Revised: The original PPO paper (Schulman et al.

Original: (2017)) discusses adding an entropy loss term to penalize low variance and prevent premature convergence, although their recommended settings for continuous control tasks do not use the entropy loss and the effect of the loss weight was not empirically investigated.
Revised: (2017)) discusses adding an entropy loss term to penalize low variance and prevent premature convergence, although their recommended settings for continuous control tasks do not use the entropy loss and the effect of the loss weight was not empirically investigated.

Original: Figure 8 shows how the entropy loss works similarly to the lower clipping limit, i.e., increasing the entropy loss weight helps to mimimize getting stuck in the local optimum.
Revised: Figure 8 shows how the entropy loss works similarly to the lower clipping limit, i.e., increasing the entropy loss weight helps to mimimize getting stuck in the local optimum.

Original: Too large values cause a rapid decrease in average performance and the entropy loss also results in a more noisy ﬁnal policy.
Revised: Too large values cause a rapid decrease in average performance and the entropy loss also results in a more noisy ﬁnal policy.

Original: To mitigate this, some PPO implementations such as the one in Unity Machine
Revised: To mitigate this, some PPO implementations such as the one in Unity Machine

Original: Learning Agents framework anneal the entropy loss weight to zero during training; however, this adds the cost of ﬁnetuning even more hyperparameters.
Revised: Learning Agents framework anneal the entropy loss weight to zero during training; however, this adds the cost of ﬁnetuning even more hyperparameters.

Original: Ideally, one would like the variance adaptation to be both efﬁcient and automatic.
Revised: Ideally, one would like the variance adaptation to be both efﬁcient and automatic.

Original: If one wants to use PPO with variance clipping or entropy loss instead of PPO-CMA, our recommendation is to try variance clipping, as it results in better results in Figure 8.
Revised: If one wants to use PPO with variance clipping or entropy loss instead of PPO-CMA, our recommendation is to try variance clipping, as it results in better results in Figure 8.

Original: Figure 9 also shows how the entropy loss results in less stable variance curves. 
Revised: Figure 9 also shows how the entropy loss results in less stable variance curves. best-case performance.

Original: B H YPERPARAMETERS AND IMPLEMENTATION DETAILS
Revised: B H YPERPARAMETERS AND IMPLEMENTATION DETAILS

Original: This section describes details of our PPO-CMA and PPO implementations.
Revised: This section describes details of our PPO-CMA and PPO implementations.

Original: OpenAI’s baseline PPO was used in Section 5 without any modiﬁcations.
Revised: OpenAI’s baseline PPO was used in Section 5 without any modiﬁcations.

Original: We use the policy network output clipping described in Section A.2 with a lower standard deviation limit of 0.01.
Revised: We use the policy network output clipping described in Section A.2 with a lower standard deviation limit of 0.01.

Original: Thus, the clipping only ensures numerical precision but has little effect on convergence, as illustrated in Figure 8.
Revised: Thus, the clipping only ensures numerical precision but has little effect on convergence, as illustrated in Figure 8.

Original: We use the clipping because without it, we ran into numerical precision problems in PPO (but not with PPO-CMA) in environments such as the Walker2d-V2.
Revised: We use the clipping because without it, we ran into numerical precision problems in PPO (but not with PPO-CMA) in environments such as the Walker2d-V2.

Original: The clipping is not necessary for PPO-CMA in our experience, but we still use with both algorithms it to ensure a controlled and fair comparison.
Revised: The clipping is not necessary for PPO-CMA in our experience, but we still use with both algorithms it to ensure a controlled and fair comparison.

Original: Following Schulman’s original PPO code, we also use episode time as an extra feature for the critic network to help minimize the value function prediction variance arising from episode termination at the environment time limit.
Revised: Following Schulman’s original PPO code, we also use episode time as an extra feature for the critic network to help minimize the value function prediction variance arising from episode termination at the environment time limit.

Original: Note that as the feature augmentation is not done for the policy, this has no effect on the usability of the training results.
Revised: Note that as the feature augmentation is not done for the policy, this has no effect on the usability of the training results.

Original: Table 1 lists all our hyperparameters.
Revised: Table 1 lists all our hyperparameters.

Original: [Table]
Revised: [Table]

Original: L

Original: We use the same network architecture for all neural networks.
Revised: We use the same network architecture for all neural networks.

Original: Action repeat of 2 means that the policy network is only queried for every other simulation step and the same action is used for two steps.
Revised: Action repeat of 2 means that the policy network is only queried for every other simulation step and the same action is used for two steps.

Original: This speeds up training.
Revised: This speeds up training.

Original: N is speciﬁed in simulation steps.
Revised: N is speciﬁed in simulation steps.

Original: H is used only for PPO-CMA.
Revised: H is used only for PPO-CMA.

Original: Figure 5 is generated assuming early termination to reduce variance, i.e., for each training run, the graphs use the best scoring iteration’s results so far.
Revised: Figure 5 is generated assuming early termination to reduce variance, i.e., for each training run, the graphs use the best scoring iteration’s results so far.

Original: The choice of H = 3 gives slightly better results than H = 7 in environments other than the 3D humanoid.
Revised: The choice of H = 3 gives slightly better results than H = 7 in environments other than the 3D humanoid.

Original: The humanoid also needs a larger N to learn; the default value of 16000 results in very noisy episode rewards that quickly plateau instead of climbing steadily.
Revised: The humanoid also needs a larger N to learn; the default value of 16000 results in very noisy episode rewards that quickly plateau instead of climbing steadily.

Original: This is in line with CMAES, which is said to be quasi-parameter-free; one mainly needs to increase the iteration sampling budget for high-dimensional and difﬁcult optimization problems.
Revised: This is in line with CMAES, which is said to be quasi-parameter-free; one mainly needs to increase the iteration sampling budget for high-dimensional and difﬁcult optimization problems.

Original: Figure 10 shows the effect of different H in the Hopper task.
Revised: Figure 11 shows the effect of different H in the Hopper task.

Original: Values H > 1 remove low-reward outliers, but PPO-CMA does not appear to be highly sensitive to the exact choice.
Revised: Values H > 1 remove low-reward outliers, but PPO-CMA does not appear to be highly sensitive to the exact choice.

Original: Comparing ﬁgures 8 and 10 suggests that the variance clipping limit, entropy loss weight and history length H all behave similarly in that one of them has to be large enough to produce good results.
Revised: Comparing ﬁgures 8 and 11 suggests that the variance clipping limit, entropy loss weight and history length H all behave similarly in that one of them has to be large enough to produce good results.

Original: The crucial difference is that setting H to a value larger than the required one does not cause a rapid decrease in performance; thus, we ﬁnd H easier to adjust.
Revised: The crucial difference is that setting H to a value larger than the required one does not cause a rapid decrease in performance; thus, we ﬁnd H easier to adjust.

Original: We use L1 critic loss as it seems to make both PPO and PPO-CMA less sensitive to the reward scaling.
Revised: We use L1 critic loss as it seems to make both PPO and PPO-CMA less sensitive to the reward scaling.

Original: For better tolerance to varying state observation scales, we use an automatic normalization scheme where observation variable j is scaled by k ( i )j
Revised: For better tolerance to varying state observation scales, we use an automatic normalization scheme where observation variable j is scaled by k ( i )

Original: =
Revised: j

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  , where κ = 0 .
Revised: ρ j + κ ) (cid:17) , where κ = 0 .

Original: 001 and ρ j is the root mean square of the variable over all iterations so far.
Revised: 001 and ρ j is the root mean square of the variable over all iterations so far.

Original: This way, large observations are scaled down but the scaling does not constantly keep adapting as training progresses.
Revised: This way, large observations are scaled down but the scaling does not constantly keep adapting as training progresses.

Original: OpenAI’s baseline PPO normalizes observations based on running mean and standard deviation, but we have found our normalization slightly more stable.
Revised: OpenAI’s baseline PPO normalizes observations based on running mean and standard deviation, but we have found our normalization slightly more stable.

Original: B. O N FINETUNING  PPO-CMA AND PPO
Revised: B. O N FINETUNING AND STABILITY OF PPO-CMA AND PPO

Original: In our experience, the iteration simulation budget is the main parameter to adjust in PPO-CMA; the algorithm can make large updates and a large budget may be needed to reduce noise of the updates in high-dimensional and/or difﬁcult problems.

Original: In addition, it may be beneﬁcial to also experiment with different values for H .

Original: In such ﬁnetuning, it is useful to visualize full learning curve distributions like in Figure 10 instead of only plotting means and standard deviations, in order to see whether there are low-reward outliers where learning gets stuck in local optima.

Original: If there are, increasing H may help. In light of our experiments, it is reasonable to ask why OpenAI’s baseline PPO gives better results than our PPO implementation in most  tasks.
Revised: In light of our experiments, it is reasonable to ask why OpenAI’s baseline PPO gives better results than our PPO implementation in most tasks, and why our own PPO implementation is unstable in the Humanoid and Double Inverted Pendulum tasks.

Original: Our understanding is that this is due to multiple implementation details and hyperparameters that regularizeand slow down the variance updates.
Revised: Our understanding is that this is due to multiple implementation details and hyperparameters that regularize the baseline PPO:

Original: In contrast to our settings, OpenAI’s baseline PPO uses a small  simulation budget of N = 2048 that yields many small updates, heavily regularized by the following in addition to the clipped surrogate loss:
Revised: • A small iteration simulation budget of N = 2048 , considerably less than what we use. The small N yields more iterations with the same total simulation budget; however, each iteration only results in a small update because of the details below that slow down the policy adaptation, in addition to the clipped surrogate loss.

Original: • Training only for K = 10 epochs, which together with the small N amounts to signiﬁcantly less minibatch updates per iteration.
Revised: • Training only for K = 10 epochs, which together with the small N amounts to signiﬁcantly less minibatch updates per iteration.

Original: • Using heavy gradient norm clipping with a clipping threshold of 0.5.
Revised: • Using heavy gradient norm clipping with a clipping threshold of 0.5.

Original: • Using smooth and saturating tanh activation functions in the neural networks, which slow down learning and result in smooth function approximations without sharp discontinuities.
Revised: • Using smooth and saturating tanh activation functions in the neural networks, which slow down learning and result in smooth function approximations without sharp discontinuities.

Original: • Using a smaller neural network with 64 units per layer.
Revised: • Using a smaller neural network with 64 units per layer.

Original: In addition to the above, some PPO implementations such as the one in Unity Machine Learning
Revised: In addition to the above, some PPO implementations such as the one in Unity Machine Learning

Original: Agents framework use a global diagonal covariance independent of state, which adapts  slowly as a compromise over actions of all states.
Revised: Agents framework use a global diagonal covariance independent of state, which adapts more slowly as a compromise over actions of all states.

Original: As a downside, the global covariance is at least theoretically less powerful because the agent may encounter both states where it should already converge and states where it should keep exploring.
Revised: As a downside, the global covariance is at least theoretically less powerful because the agent may encounter both states where it should already converge and states where it should keep exploring.

Revised: In summary, PPO may need other regularizers in addition to the epsilon parameter of the clipped surrogate loss, which can make it difﬁcult to ﬁnetune.

Revised: In contrast, PPO-CMA does not seem to need such regularization.

Revised: On the other hand, PPO-CMA can make large updates and a large N may be needed to reduce noise of the updates in high-dimensional and/or difﬁcult problems.

Revised: This is similar to CMA-ES, where the population size is typically the main parameter to adjust.

Original:  C S TATISTICAL SIGNIFICANCE TESTING DETAILS
Revised: In addition, it may be beneﬁcial to also experiment with different values for H if PPO-CMA does not give good results. In such ﬁnetuning, it is useful to visualize full learning curve distributions like in Figure 11 instead of only plotting means and standard deviations, in order to see whether there are low-reward outliers where learning gets stuck in local optima. If there are, increasing H may help. C S TATISTICAL SIGNIFICANCE TESTING DETAILS

Original: To supplement the visual comparison of Figure
Revised: To supplement the visual comparison of Figure

Original: [Table]
Revised: [Table]

Original: Table 2 compares PPO-CMA with OpenAI’s baseline PPO using two-sided Welch t-tests.
Revised: Table 2 compares PPO-CMA with OpenAI’s baseline PPO using two-sided Welch t-tests.

Original: [Table]
Revised: [Table]

Original: In section
Revised: In section

Original: [Table]
Revised: [Table]

Original: we compare PPO-CMA and PPO on average, over a set of randomized hyperparameter combinations.
Revised: we compare PPO-CMA and PPO on average, over a set of randomized hyperparameter combinations.

Original: The parameters were sampled uniformly in the ranges given in Table 
Revised: The parameters were sampled uniformly in the ranges given in Table 3.

Original: [Table]
Revised: [Table]

Revised: D F URTHER VISUALIZATION OF ALGORITHM DIFFERENCES

Revised: Figure 10 visualizes the effect of minibatch gradient steps taken inside one algorithm iteration in our didactic example.

Revised: This complements Figures 1 and 2, which show the results of multiple iterations.

Revised: The ﬁgure reveals the following:

Revised: • 1st row: Vanilla policy gradient diverges outside the data when taking multiple gradient steps with the same actions and advantages.

Revised: The divergence is not caused by excessive gradient step size; each minibatch only causes a tiny change.

Revised: Recall that the motivation to take multiple steps comes from PPO and TRPO; the goal is to get the most out of the collected experience.

Revised: • 2nd row: Policy gradient becomes stable when only using positive advantage actions.

Revised: The policy converges to approximate the distribution of the positive advantage actions (green).

Revised: • 3rd row: PPO limits the policy change, which prevents divergence.

Revised: On the other hand, it does not make as much progress as could be done based on the data.

Revised: • 4th row: PPO-CMA is stable and elongates the search distribution along the progress direction.

Revised: This makes it more likely that the next iteration will generate samples close to the optimum; it lies within the unit standard deviation ellipse of the updated policy.

Revised: • 5th row: PPO-CMA-m behaves visually similar to PPO-CMA.

Revised: This is in line with our other experimental results; PPO-CMA-m gives similar results as PPO-CMA in easy tasks, where the extra information of the negative advantage actions is not needed.

Revised: The ﬁgure also shows how using the negative-advantage actions through the mirroring is stable; the policy converges similar to when only using positive advantages, but half of the positiveadvantage actions are created through the mirroring.

Original: D L IMITATIONS
Revised: E L IMITATIONS

Original: We only implement an approximation of the full CMA-ES algorithm, because in addition to the mean and covariance of the search distribution, CMA-ES maintains several other auxiliary variables.
Revised: We only implement an approximation of the full CMA-ES algorithm, because in addition to the mean and covariance of the search distribution, CMA-ES maintains several other auxiliary variables.

Original: In policy optimization, all algorithm state and auxiliary variables are functions of agent state and must be encoded as neural network weights, and CMA-ES has some delicately ﬁne-tuned mechanisms which might be unstable with such approximations.
Revised: In policy optimization, all algorithm state and auxiliary variables are functions of agent state and must be encoded as neural network weights, and CMA-ES has some delicately ﬁne-tuned mechanisms which might be unstable with such approximations.

Original: We defer further investigations of this to future work, since even our simple approximation yields beneﬁts over vanillla PPO.
Revised: We defer further investigations of this to future work, since even our simple approximation yields beneﬁts over vanillla PPO.

Original: We have also only focused on Gaussian policies.
Revised: We have also only focused on Gaussian policies, and the special case of diagonal covariance, i.e., the sep-CMA-ES variant (Ros & Hansen (2008)).

Original: Obviously, other exploration noise distributions such as Poisson might also work.
Revised: Obviously, other exploration noise distributions such as Laplace might also work.

Original: However, Gaussian exploration is common in the literature, and the success of CMA-ES shows that it results in state-of-the-art performance in many optimization tasks beyond policy optimization.
Revised: However, Gaussian exploration is common in the literature, and the success of CMA-ES shows that it results in state-of-the-art performance in many optimization tasks beyond policy optimization.

Original: Finally, treating the advantage function as ﬁtness function assumes that the advantage landscape does not change signiﬁcantly when policy is updated.
Revised: Finally, treating the advantage function as ﬁtness function assumes that the advantage landscape does not change signiﬁcantly when policy is updated.

Original: This assumption does not hold in general for γ > 0 .
Revised: This assumption does not hold in general for γ > 0 .

Original: Thus, it is perhaps surprising that PPO-CMA works as well as it does.
Revised: Thus, it is perhaps surprising that PPO-CMA works as well as it does.

Original: In future work, we plan to test whether PPO-CMA is particularly effective in problems that can be modeled using γ = 0 , e.g., ﬁnding an optimal billiards shot that pockets as many balls as possible, or the humanoid climbing moves of Naderi et al.
Revised: In future work, we plan to test whether PPO-CMA is particularly effective in problems that can be modeled using γ = 0 , e.g., ﬁnding an optimal billiards shot that pockets as many balls as possible, or the humanoid climbing moves of Naderi et al.

Original: (2017), with actions parameterized as control splines instead of instantaneous torques.
Revised: (2017), with actions parameterized as control splines instead of instantaneous torques.