Original: G ENETIC P OLICY O PTIMIZATION
Revised: G ENETIC P OLICY O PTIMIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Genetic algorithms have been widely used in many practical optimization problems.
Revised: Genetic algorithms have been widely used in many practical optimization problems.

Original: Inspired by natural selection, operators, including mutation, crossover and selection, provide effective heuristics for search and black-box optimization.
Revised: Inspired by natural selection, operators, including mutation, crossover and selection, provide effective heuristics for search and black-box optimization.

Original: However, they have not been shown useful for deep reinforcement learning, possibly due to the catastrophic consequence of parameter crossovers of neural networks.
Revised: However, they have not been shown useful for deep reinforcement learning, possibly due to the catastrophic consequence of parameter crossovers of neural networks.

Original: Here, we present Genetic Policy Optimization (GPO), a new genetic algorithm for sample-efﬁcient deep policy optimization.
Revised: Here, we present Genetic Policy Optimization (GPO), a new genetic algorithm for sample-efﬁcient deep policy optimization.

Original: GPO uses imitation learning for policy crossover in the state space and applies policy gradient methods for mutation.
Revised: GPO uses imitation learning for policy crossover in the state space and applies policy gradient methods for mutation.

Original: Our experiments on Mujoco tasks show that GPO as a genetic algorithm is able to provide superior performance over the state-of-the-art policy gradient methods and achieves comparable or higher sample efﬁciency.
Revised: Our experiments on MuJoCo tasks show that GPO as a genetic algorithm is able to provide superior performance over the state-of-the-art policy gradient methods and achieves comparable or higher sample efﬁciency.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) has recently demonstrated signiﬁcant progress and achieves state-ofthe-art performance in games (Mnih et al., 2015; Silver et al., 2016), locomotion control (Lillicrap et al., 2015), visual-navigation (Zhu et al., 2017), and robotics (Levine et al., 2016).
Revised: Reinforcement learning (RL) has recently demonstrated signiﬁcant progress and achieves state-ofthe-art performance in games (Mnih et al., 2015; Silver et al., 2016), locomotion control (Lillicrap et al., 2015), visual-navigation (Zhu et al., 2017), and robotics (Levine et al., 2016).

Original: Among these successes, deep neural networks (DNNs) are widely used as powerful functional approximators to enable signal perception, feature extraction and complex decision making.
Revised: Among these successes, deep neural networks (DNNs) are widely used as powerful functional approximators to enable signal perception, feature extraction and complex decision making.

Original: For example, in continuous control tasks, the policy that determines which action to take is often parameterized by a deep neural network that takes the current state observation or sensor measurements as input.
Revised: For example, in continuous control tasks, the policy that determines which action to take is often parameterized by a deep neural network that takes the current state observation or sensor measurements as input.

Original: In order to optimize such policies, various policy gradient methods (Mnih et al., 2016; Schulman et al., 2015; 2017; Heess et al., 2017) have been proposed to estimate gradients approximately from rollout trajectories.
Revised: In order to optimize such policies, various policy gradient methods (Mnih et al., 2016; Schulman et al., 2015; 2017; Heess et al., 2017) have been proposed to estimate gradients approximately from rollout trajectories.

Original: The core idea of these policy gradient methods is to take advantage of the temporal structure in the rollout trajectories to construct a Monte Carlo estimator of the gradient of the expected return.
Revised: The core idea of these policy gradient methods is to take advantage of the temporal structure in the rollout trajectories to construct a Monte Carlo estimator of the gradient of the expected return.

Original: In addition to the popular policy gradient methods, other alternative solutions, such as those for black-box optimization or stochastic optimization, have been recently studied for policy optimization.
Revised: In addition to the popular policy gradient methods, other alternative solutions, such as those for black-box optimization or stochastic optimization, have been recently studied for policy optimization.

Original: Evolution strategies (ES) is a class of stochastic optimization techniques that can search the policy space without relying on the backpropagation of gradients.
Revised: Evolution strategies (ES) is a class of stochastic optimization techniques that can search the policy space without relying on the backpropagation of gradients.

Original: At each iteration, ES samples a candidate population of parameter vectors (“genotypes”) from a probability distribution over the parameter space, evaluates the objective function (“ﬁtness”) on these candidates, and constructs a new probability distribution over the parameter space using the candidates with the high ﬁtness.
Revised: At each iteration, ES samples a candidate population of parameter vectors (“genotypes”) from a probability distribution over the parameter space, evaluates the objective function (“ﬁtness”) on these candidates, and constructs a new probability distribution over the parameter space using the candidates with the high ﬁtness.

Original: This process is repeated iteratively until the objective is maximized.
Revised: This process is repeated iteratively until the objective is maximized.

Original: Covariance matrix adaptation evolution strategy (CMA-ES; Hansen & Ostermeier (2001)) and recent work from Salimans et al.
Revised: Covariance matrix adaptation evolution strategy (CMA-ES; Hansen & Ostermeier (2001)) and recent work from Salimans et al.

Original: (2017) are examples of this procedure.
Revised: (2017) are examples of this procedure.

Original: These ES algorithms have also shown promising results on continuous control tasks and Atari games, but their sample efﬁciency is often not comparable to the advanced policy gradient methods, because ES is black-box and thus does not fully exploit the policy network architectures or the temporal structure of the RL problems.
Revised: These ES algorithms have also shown promising results on continuous control tasks and Atari games, but their sample efﬁciency is often not comparable to the advanced policy gradient methods, because ES is black-box and thus does not fully exploit the policy network architectures or the temporal structure of the RL problems.

Original: Very similar to ES, genetic algorithms (GAs) are a heuristic search technique for search and optimization.
Revised: Very similar to ES, genetic algorithms (GAs) are a heuristic search technique for search and optimization.

Original: Inspired by the process of natural selection, GAs evolve an initial population of genotypes by repeated applications of three genetic operators - mutation, crossover and selection.
Revised: Inspired by the process of natural selection, GAs evolve an initial population of genotypes by repeated application of three genetic operators - mutation, crossover and selection.

Original: The major difference between GA and ES is that the crossover operator in GA is able to provide higher diversity of good candidates in the population.
Revised: One of the main differences between GA and ES is the use of the crossover operator in GA, which is able to provide higher diversity of good candidates in the population.

Original: However, the crossover operator is often performed on the parameter representations of two parents, thus making it not suitable for nonlinear neural networks.
Revised: However, the crossover operator is often performed on the parameter representations of two parents, thus making it unsuitable for nonlinear neural networks.

Original: The straightforward crossover of two neural networks by exchanging their parameters can often destroy the hierarchical relationship of the networks and thus cause a catastrophic drop in per- formance.
Revised: The straightforward crossover of two neural networks by exchanging their parameters can often destroy the hierarchical relationship of the networks and thus cause a catastrophic drop in performance.

Original: NeuroEvolution of Augmenting Topologies (NEAT; Stanley & Miikkulainen (2002b;a)), which evolves neural networks through evolutionary algorithms such as GA, provides a solution to exchange and augment neurons but has found limited success when used as a method of policy search in deep RL for high-dimensional tasks.
Revised: NeuroEvolution of Augmenting Topologies (NEAT; Stanley & Miikkulainen (2002b;a)), which evolves neural networks through evolutionary algorithms such as GA, provides a solution to exchange and augment neurons but has found limited success when used as a method of policy search in deep RL for high-dimensional tasks.

Original: A major challenge to making GAs work for policy optimization is to design a good crossover operator which efﬁciently combines two parent policies represented by neural networks and generates an offspring that takes advantage of both parents.
Revised: A major challenge to making GAs work for policy optimization is to design a good crossover operator which efﬁciently combines two parent policies represented by neural networks and generates an offspring that takes advantage of both parents.

Original: In addition, a good mutation operator is needed as random perturbations are often inefﬁcient for high-dimensional policies.
Revised: In addition, a good mutation operator is needed as random perturbations are often inefﬁcient for high-dimensional policies.

Original: In this paper, we present Genetic Policy Optimization (GPO), a new genetic algorithm for sampleefﬁcient deep policy optimization.
Revised: In this paper, we present Genetic Policy Optimization (GPO), a new genetic algorithm for sampleefﬁcient deep policy optimization.

Original: There are two major technical advances in GPO.
Revised: There are two major technical advances in GPO.

Original: First, instead of using parameter crossover, GPO applies imitation learning for policy crossovers in the state space.
Revised: First, instead of using parameter crossover, GPO applies imitation learning for policy crossovers in the state space.

Original: The state-space crossover effectively combines two parent policies into an offspring or child policy that tries to mimic its best parent in generating similar state visitation distributions.
Revised: The state-space crossover effectively combines two parent policies into an offspring or child policy that tries to mimic its best parent in generating similar state visitation distributions.

Original: Second, GPO applies advanced policy gradient methods for mutation.
Revised: Second, GPO applies advanced policy gradient methods for mutation.

Original: By randomly rolling out trajectories and performing gradient descent updates, this mutation operator is more efﬁcient than random parameter perturbations and also maintains sufﬁcient genetic diversity.
Revised: By randomly rolling out trajectories and performing gradient descent updates, this mutation operator is more efﬁcient than random parameter perturbations and also maintains sufﬁcient genetic diversity.

Original: Our experiments on several continuous control tasks show that GPO as a genetic algorithm is able to provide superior performance over the state-of-the-art policy gradient methods and achieves comparable or higher sample efﬁciency.
Revised: Our experiments on several continuous control tasks show that GPO as a genetic algorithm is able to provide superior performance over the state-of-the-art policy gradient methods and achieves comparable or higher sample efﬁciency.

Original:  B ACKGROUND AND R ELATED W ORK 2. R EINFORCEMENT L EARNING
Revised:  B ACKGROUND AND R ELATED W ORK 2. R EINFORCEMENT L EARNING

Original: In the standard RL setting, an agent interacts with an environment E modeled as a Markov Decision
Revised: In the standard RL setting, an agent interacts with an environment E modeled as a Markov Decision

Original: Process (MDP).
Revised: Process (MDP).

Original: At each discrete time step t , the agent observes a state s t and choose an action a t ∈
Revised: At each discrete time step t , the agent observes a state s t and choose an action a t ∈

Original: A using a policy π ( a t | s t ) , which is a mapping from states to a distribution over possible actions.
Revised: A using a policy π ( a t | s t ) , which is a mapping from states to a distribution over possible actions.

Original: Here we consider high-dimensional, continuous state and action spaces.
Revised: Here we consider high-dimensional, continuous state and action spaces.

Original: After performing the action a t , the agent collects a scalar reward r ( s t , a t ) ∈ R at each time step.
Revised: After performing the action a t , the agent collects a scalar reward r ( s t , a t ) ∈ R at each time step.

Original: The goal in reinforcement learning is to learn a policy which maximizes the expected sum of (discounted) rewards starting from the initial state.
Revised: The goal in reinforcement learning is to learn a policy which maximizes the expected sum of (discounted) rewards starting from the initial state.

Original: Formally, the objective is
Revised: Formally, the objective is

Original: [Equation]
Revised: [Equation]

Original: where the states s t are sampled from the environment E using an unknown system dynamics model p ( s t +1 | s t , a t ) and an initial state distribution p ( s 0 ) , the actions a t are sampled from the policy π ( a t | s t ) and γ ∈ (0 , 1] is the discount factor.
Revised: where the states s t are sampled from the environment E using an unknown system dynamics model p ( s t +1 | s t , a t ) and an initial state distribution p ( s 0 ) , the actions a t are sampled from the policy π ( a t | s t ) and γ ∈ (0 , 1] is the discount factor.

Original: 2. P OLICY G RADIENT M ETHODS
Revised: 2. P OLICY G RADIENT M ETHODS

Original: Policy-based RL methods search for an optimum policy directly in the policy space.
Revised: Policy-based RL methods search for an optimum policy directly in the policy space.

Original: One popular approach is to parameterize the policy π ( a t
Revised: One popular approach is to parameterize the policy π ( a t

Original: | s t ; θ ) with θ , express the objective J ( π ( a t
Revised: | s t ; θ ) with θ , express the objective J ( π ( a t

Original: | s t ; θ )) as a function of θ and perform gradient descent methods to optimize it.
Revised: | s t ; θ )) as a function of θ and perform gradient descent methods to optimize it.

Original: The REINFORCE algorithm (Williams, 1992) calculates an unbiased estimation of the gradient ∇ θ J ( θ ) using the likelihood ratio trick.
Revised: The REINFORCE algorithm (Williams, 1992) calculates an unbiased estimation of the gradient ∇ θ J ( θ ) using the likelihood ratio trick.

Original: Speciﬁcally, REINFORCE updates the policy parameters in the direction of the following approximation to policy gradient
Revised: Speciﬁcally, REINFORCE updates the policy parameters in the direction of the following approximation to policy gradient

Original: [Equation]
Revised: [Equation]

Original: based on a single rollout trajectory, where R t = (cid:80) ∞ i =

Revised: based on a single rollout trajectory, where R t = (cid:80) ∞ i =

Original: γ i r ( s t
Revised: γ i r ( s t

Original: + i , a t + i ) is the discounted sum of rewards from time step t .
Revised: + i , a t + i ) is the discounted sum of rewards from time step t .

Original: The advantage actor-critic (A2C) algorithm (Sutton & Barto; Mnih et al., 2016) uses the state value function (or critic) to reduce the variance in the above gradient estimation.
Revised: The advantage actor-critic (A2C) algorithm (Sutton & Barto; Mnih et al., 2016) uses the state value function (or critic) to reduce the variance in the above gradient estimation.

Original: The contribution to the gradient at time step t is ∇ θ log π ( a t | s t ; θ )( R t − V π
Revised: The contribution to the gradient at time step t is ∇ θ log π ( a t | s t ; θ )( R t − V π

Original: ( s t )) .
Revised: ( s t )) .

Original: R t − V π ( s t ) is an estimate of the advantage function A π ( s t , a t ) =
Revised: R t − V π ( s t ) is an estimate of the advantage function A π ( s t , a t ) =

Original: Q π ( s t , a t ) − V π
Revised: Q π ( s t , a t ) − V π

Original: ( s t ) .
Revised: ( s t ) .

Original: In practice, multiple rollouts are performed to get the policy gradient, and V π ( s t ) is learned using a function approximator.
Revised: In practice, multiple rollouts are performed to get the policy gradient, and V π ( s t ) is learned using a function approximator.

Original: High variance in policy gradient estimates can sometimes lead to large, destructive updates to the policy parameters.
Revised: High variance in policy gradient estimates can sometimes lead to large, destructive updates to the policy parameters.

Original: Trust-region methods such as TRPO (Schulman et al., 2015) avoid this by restricting the amount by which an update is allowed to change the policy.
Revised: Trust-region methods such as TRPO (Schulman et al., 2015) avoid this by restricting the amount by which an update is allowed to change the policy.

Original: TRPO is a second order algorithm that solves an approximation to a constrained optimization problem using conjugate gradient.
Revised: TRPO is a second order algorithm that solves an approximation to a constrained optimization problem using conjugate gradient.

Original: Proximal policy optimization (PPO) algorithm (Schulman et al., 2017) is an approximation to TRPO that relies only on ﬁrst order gradients.
Revised: Proximal policy optimization (PPO) algorithm (Schulman et al., 2017) is an approximation to TRPO that relies only on ﬁrst order gradients.

Original: The PPO objective penalizes the Kullback-Leibler (KL) divergence change between the policy before the update ( π θ old ) and the policy at the current step ( π θ ).
Revised: The PPO objective penalizes the Kullback-Leibler (KL) divergence change between the policy before the update ( π θ old ) and the policy at the current step ( π θ ).

Original: The penalty weight β is adaptive and adjusted based on observed change in KL divergence after multiple policy update steps have been performed using the same batch of data.
Revised: The penalty weight β is adaptive and adjusted based on observed change in KL divergence after multiple policy update steps have been performed using the same batch of data.

Original: [Equation]
Revised: [Equation]

Original: where ˆ E t [ ... ] indicates the empirical average over a ﬁnite batch of samples, and ˆ A t is the advantage estimation.
Revised: where ˆ E t [ ... ] indicates the empirical average over a ﬁnite batch of samples, and ˆ A t is the advantage estimation.

Original: Schulman et al.
Revised: Schulman et al.

Original: (2017) propose another objective based on clipping of the likelihood ratio, but we use the adaptive-KL objective due to its better empirical performance (Heess et al., 2017; Hafner et al., 2017).
Revised: (2017) propose another objective based on clipping of the likelihood ratio, but we use the adaptive-KL objective due to its better empirical performance (Heess et al., 2017; Hafner et al., 2017).

Original: 2. E VOLUTIONARY A LGORITHMS
Revised: 2. E VOLUTIONARY A LGORITHMS

Original: There is growing interest in using evolutionary algorithms as a policy search procedure in RL.
Revised: There is growing interest in using evolutionary algorithms as a policy search procedure in RL.

Original: We provide a brief summary; a detailed survey is provided by Whiteson (2012).
Revised: We provide a brief summary; a detailed survey is provided by Whiteson (2012).

Revised: Neuroevolution is the process of using evolutionary algorithms to generate neural network weights and topology.

Revised: Among early applications of neuroevolution algorithms to control tasks are SANE (Moriarty & Mikkulainen, 1996) and ESP (Gomez & Miikkulainen, 1999).

Revised: NEAT has also been successfully applied for policy optimization (Stanley & Miikkulainen, 2002a).

Revised: NEAT provides a rich representation for genotypes, and tracks the historical origin of every gene to allow for principled crossover between networks of disparate topologies.

Original: Recently, Salimans et al. 
Revised: In Gomez et al. (2008), the authors introduce an algorithm based on cooperative co-evolution (CoSyNE) and compare it favorably against Q-learning and policy-gradient based RL algorithms. They do crossover between ﬁxed topology networks using a multi-point strategy at the granularity of network layers (weight segments).

Revised: HyperNEAT (D’Ambrosio & Stanley, 2007), which extends on NEAT and uses CPPN-based indirect encoding, has been used to learn to play

Revised: Atari games from raw game pixels (Hausknecht et al., 2014).

Revised: Recently, Salimans et al.

Original: (2017) proposed a version of Evolution Strategies (ES) for black-box policy optimization.
Revised: (2017) proposed a version of Evolution Strategies (ES) for black-box policy optimization.

Original: At each iteration k , the algorithm samples candidate parameter vectors (policies) using a ﬁxed covariance Gaussian N (0 , σ 2 I ) perturbation on the mean vector m ( k ) .
Revised: At each iteration k , the algorithm samples candidate parameter vectors (policies) using a ﬁxed covariance Gaussian N (0 , σ 2 I ) perturbation on the mean vector m ( k ) .

Original: The mean vector is then updated in the direction of the weighted average of the perturbations, where weight is proportional to the ﬁtness of the candidate.
Revised: The mean vector is then updated in the direction of the weighted average of the perturbations, where weight is proportional to the ﬁtness of the candidate.

Original: CMA-NeuroES (Heidrich-Meisner & Igel, 2009) uses CMA-ES  to learn neural network policies forepisodic reinforcement learning.
Revised: CMA-ES has been used to learn neural network policies for reinforcement learning (CMA-NeuroES, Heidrich-Meisner & Igel (2009); Igel (2003)).

Original: CMA-ES samples candidate parameter vectors using a Gaussian N (0 , C ( k ) ) perturbation on the mean vector m ( k ) .
Revised: CMAES samples candidate parameter vectors using a Gaussian N (0 , C ( k ) ) perturbation on the mean vector m ( k ) .

Original: The covariance matrix and the mean vector for the next iteration are then calculated using the candidates with high ﬁtness.
Revised: The covariance matrix and the mean vector for the next iteration are then calculated using the candidates with high ﬁtness.

Original: Cross-Entropy methods use similar ideas and have been found to work reasonably well in simple environments (Szita & L ¨ orincz, 2006).
Revised: Cross-Entropy methods use similar ideas and have been found to work reasonably well in simple environments (Szita & L ¨ orincz, 2006).

Original: Among genetic algorithm approaches to policy optimization, NEAT and its extension, HyperNEAT have enjoyed some success (Stanley & Miikkulainen, 2002a). NEAT can evolve connection weights as well as the network topology.
Revised: In this work, we consider policy networks of ﬁxed topology.

Original: The crossover betweenﬁxed-topology parentsis done by copying the weights of each DNN edge randomly from one of the parents.
Revised: Existing neuroevolution algorithms perform crossover between parents by copying segments—single weight or layer(s)—of DNN parameters from either of the parents.

Original: The weights are mutated using random perturbations. This mode of mutation is also used by Salimans et al. A more principled take on mutating weights with perturbations can be found in (Hansen & Ostermeier, 2001; Sehnke et al., 2010).
Revised: Also, mutation is generally done by random perturbations of the weights, although more rigorous approaches have been proposed (Hansen & Ostermeier, 2001; Sehnke et al., 2010; Lehman et al., 2017).

Original: In this work, we use policy gradient algorithms for efﬁcient mutation of high-dimensional policies, and also depart from prior work in implementing the crossover operator.
Revised: In this work, we use policy gradient algorithms for efﬁcient mutation of high-dimensional policies, and also depart from prior work in implementing the crossover operator.

Original:  G ENETIC P OLICY O PTIMIZATION 3. O VERALL A LGORITHM
Revised:  G ENETIC P OLICY O PTIMIZATION 3. O VERALL A LGORITHM

Original: Our procedure for policy optimization proceeds by evolving the policies (genotypes) through a series of selection, crossover and mutation operators (Algorithm 1).
Revised: Our procedure for policy optimization proceeds by evolving the policies (genotypes) through a series of selection, crossover and mutation operators (Algorithm 1).

Original: We start with an ensemble of policies initialized with random parameters.
Revised: We start with an ensemble of policies initialized with random parameters.

Original: In line 3, we mutate each of the policies separately by performing a few iterations of updates on the policy parameters.
Revised: In line 3, we mutate each of the policies separately by performing a few iterations of updates on the policy parameters.

Original: Any standard policy gradient method, such as PPO or A2C, can be used for mutation.
Revised: Any standard policy gradient method, such as PPO or A2C, can be used for mutation.

Original: In line 4, we create a set of parents using a selection procedure guided by a ﬁtness function.
Revised: In line 4, we create a set of parents using 1: population ← π 1 , . . . , π m (cid:46)

Revised: Initial policies with random parameters 2: repeat 3: population ← MUTATE ( population ) 4: parents set ← SELECT ( population , F ITNESS -F N ) 5: children ← empty set 6: for tuple ( π x , π y ) ∈ parents set do 7: π c ← CROSSOVER ( π x , π y ) 8: add π c to children 9: end for 10: population ← children 11: until k steps of genetic optimization a selection procedure guided by a ﬁtness function.

Original: Each element of this set is a policy-pair ( π x , π y ) that is used in the reproduction (crossover) step to produce a new child policy ( π c ) .
Revised: Each element of this set is a policy-pair ( π x , π y ) that is used in the reproduction (crossover) step to produce a new child policy ( π c ) .

Original: This is done in line 7 by mixing the policies of the parents.
Revised: This is done in line 7 by mixing the policies of the parents.

Original: In line 10, we obtain the population for the next generation by collecting all the newly created children.
Revised: In line 10, we obtain the population for the next generation by collecting all the newly created children.

Original: The algorithm terminates after k rounds of optimization.
Revised: The algorithm terminates after k rounds of optimization.

Original: 3. GPO C ROSSOVER AND M UTATION
Revised: 3. GPO C ROSSOVER AND M UTATION

Original: We consider policies that are parameterized using deep neural networks of ﬁxed architectures.
Revised: We consider policies that are parameterized using deep neural networks of ﬁxed architectures.

Original: If the policy is Gaussian, as is common for many robotics and locomotion tasks (Duan et al., 2016), then 1: population ← π 1 , . . . , π m (cid:46)
Revised: If the policy is Gaussian, as is common for many robotics and locomotion tasks (Duan et al., 2016), then the network outputs the mean and the standard-deviation of each action in the action-space.

Original: Initial policies with random parameters 2: repeat 3: population ← MUTATE ( population ) 4: parents set ← SELECT ( population , F ITNESS -F N ) 5: children ← empty set 6: for tuple ( π x , π y ) ∈ parents set do 7: π c ← CROSSOVER ( π x , π y ) 8: add π c to children 9: end for 10: population ← children 11: until k steps of genetic optimization the network outputs the mean and the standard-deviation of each action in the action-space.

Original: Combining two DNN policies such that the ﬁnal child policy possibly absorbs the best traits of both the parents is non-trivial.
Revised: Combining two DNN policies such that the ﬁnal child policy possibly absorbs the best traits of both the parents is non-trivial.

Original: Figure 1 illustrates different crossover strategies.
Revised: Figure 1 illustrates different crossover strategies.

Original: The ﬁgure includes neural network policies along with the state-visitation distribution plots (in a 2D space) corresponding to some high return rollouts using that policy.
Revised: The ﬁgure includes neural network policies along with the state-visitation distribution plots (in a 2D space) corresponding to some high return rollouts using that policy.

Original: The two parent networks are shown in the top half of the ﬁgure.
Revised: The two parent networks are shown in the top half of the ﬁgure.

Original: The state-visitation distributions are made non-overlapping to indicate that the parents policies have good state-to-action mapping for disparate local regions of the state-space.
Revised: The state-visitation distributions are made non-overlapping to indicate that the parents policies have good state-to-action mapping for disparate local regions of the state-space.

Original: A na¨ıve approach is to do crossover in the parameter space (bottom-right in ﬁgure).
Revised: A na¨ıve approach is to do crossover in the parameter space (bottom-right in ﬁgure).

Original: In this approach, a DNN child policy is created by copying over certain edge weights from either of the parents.
Revised: In this approach, a DNN child policy is created by copying over certain edge weights from either of the parents.

Original: The crossover could be at the granularity of multiple DNN layers, a single layer of edges or even a single edge (e.g. NEAT(Stanley & Miikkulainen, 2002b)).
Revised: The crossover could be at the granularity of multiple DNN layers, a single layer of edges or even a single edge (e.g. NEAT(Stanley & Miikkulainen, 2002b)).

Original: However, this type of crossover is expected to yield a low-performance composition due to the complex non-linear interactions between policyparameters and the expected policy return.
Revised: However, this type of crossover is expected to yield a low-performance composition due to the complex non-linear interactions between policyparameters and the expected policy return.

Original: For the same reason, the state-visitation distribution of the child doesn’t hold any semblance to that of either of the parents.
Revised: For the same reason, the state-visitation distribution of the child doesn’t hold any semblance to that of either of the parents.

Original: The bottom-left part of the ﬁgure shows the outcome of an ideal crossover in state-space.
Revised: The bottom-left part of the ﬁgure shows the outcome of an ideal crossover in state-space.

Original: The state-visitation distribution of the child includes regions from both the parents, leading to better performance (in expectation) than either of them.
Revised: The state-visitation distribution of the child includes regions from both the parents, leading to better performance (in expectation) than either of them.

Original: In this work, we propose a new crossover operator that utilizes imitation learning to combine the best traits from both parents and generate a high-performance child or offspring policy.
Revised: In this work, we propose a new crossover operator that utilizes imitation learning to combine the best traits from both parents and generate a high-performance child or offspring policy.

Original: So this crossover is not done directly in the parameter space but in the behavior or the state visitation space.
Revised: So this crossover is not done directly in the parameter space but in the behavior or the state visitation space.

Original: We quantify the effect of these two types of crossovers in Section 4 by mixing several DNN pairs and measuring the policy performance in a simulated environment.
Revised: We quantify the effect of these two types of crossovers in Section 4 by mixing several DNN pairs and measuring the policy performance in a simulated environment.

Revised: Imitation learning can broadly be categorized into behavioral cloning, where the agent is trained to match the action of the expert in a given state using supervised learning, and inverse reinforcement learning, where the agent infers a cost function for the environment using expert demonstrations and then learns an optimal policy for that cost function.

Revised: We use behavioral cloning in this paper, and all our references to imitation learning should be taken to mean that.

Original: Our second contribution is in utilizing policy gradient algorithms for mutation of neural network weights in lieu of the Gaussian perturbations used in prior work on evolutionary algorithms for policy search.
Revised: Our second contribution is in utilizing policy gradient algorithms for mutation of neural network weights in lieu of the Gaussian perturbations used in prior work on evolutionary algorithms for policy search.

Original: Because of the randomness in rollout samples, the policy-gradient mutation operator also maintains sufﬁcient genetic diversity in the population.
Revised: Because of the randomness in rollout samples, the policy-gradient mutation operator also maintains sufﬁcient genetic diversity in the population.

Original: This helps our overall genetic algorithm achieve similar or higher sample efﬁciency compared to the state-of-the-art policy gradient methods.
Revised: This helps our overall genetic algorithm achieve similar or higher sample efﬁciency compared to the state-of-the-art policy gradient methods.

Original: 3. G ENETIC O PERATORS
Revised: 3. G ENETIC O PERATORS

Original: This section details the three genetic operators.
Revised: This section details the three genetic operators.

Original: We use different subscripts for different policies.
Revised: We use different subscripts for different policies.

Original: The corresponding parameters of the neural network are sub-scripted with the same letter (e.g. θ x for π x ).
Revised: The corresponding parameters of the neural network are sub-scripted with the same letter (e.g. θ x for π x ).

Original: We also use π x and π θ x interchangeably.
Revised: We also use π x and π θ x interchangeably.

Original: m (cid:83) i =1 π
Revised: m (cid:83) i =1 π

Original: i represents an ensemble of m policies { π 1 , . .
Revised: i represents an ensemble of m policies { π 1 , . .

Original: π m } .
Revised: π m } .

Original: 3.3. CROSSOVER ( π x , π y )
Revised: 3.3. CROSSOVER ( π x , π y )

Original: This operator mixes two input policies π x and π y in state-space and produces a new child policy π c .
Revised: This operator mixes two input policies π x and π y in state-space and produces a new child policy π c .

Original: The three policies have identical network architecture.
Revised: The three policies have identical network architecture.

Original: The child policy is learned using a twostep procedure.
Revised: The child policy is learned using a two-step procedure.

Revised: A schematic of the methodology is shown in Figure 2.

Original: Firstly, we train a two-level policy π H ( a | s )
Revised: Firstly, we train a two-level policy π H ( a | s )

Original: =
Revised: =

Original: π S ( h = x | s ) π x ( a | s ) + π S ( h =
Revised: π S ( parent = x | s ) π x ( a | s ) + π S ( parent =

Original: y | s ) π y ( a | s ) which, given an observation, ﬁrst chooses between π x and π y , and then outputs the action  of the chosen policy.
Revised: y | s ) π y ( a | s ) which, given an observation, ﬁrst chooses between π x and π y , and then outputs the action distribution of the chosen parent.

Original: π S is a binary policy  trained with the maximum likelihood objective (cross-entropy loss) on therecent rollout trajectories from both parents.
Revised: π S is a binary policy which is trained using trajectories from the parents involved in the crossover. In our implementation, we reuse the trajectories from the parents’ previous mutation phase rather than generating new samples. The training objective for π S is weighted maximum likelihood, where normalized trajectory returns are used as weights. Formally, given two parents π x and π y , the log loss is given by:

Revised: [Equation]

Revised: where p x := π S ( parent = x | s ) , w s is the weight assigned to the trajectory which contained the state s , and D is the set of parent trajectories.

Original: This hierarchical reinforcement learning step acts a medium of knowledge transfer from the parents to the child.
Revised: This hierarchical reinforcement learning step acts a medium of knowledge transfer from the parents to the child.

Original: We use only high-reward trajectories from π x and π y as data samples for training π S to avoid transfer of negative behavior.
Revised: We use only high-reward trajectories from π x and π y as data samples for training π S to avoid transfer of negative behavior.

Original: It is possible to further reﬁne π S by running a few iterations of any policy-gradient algorithm, but we ﬁnd that the maximum likelihood approach works well in practice and can also avoid extra rollout samples.
Revised: It is possible to further reﬁne π S by running a few iterations of any policy-gradient algorithm, but we ﬁnd that the maximum likelihood approach works well in practice and can also avoid extra rollout samples.

Original: Next, to distill the information from π H into a policy with the same architecture as the parents, we use imitation learning to train a child policy π c .
Revised: Next, to distill the information from π H into a policy with the same architecture as the parents, we use imitation learning to train a child policy π c .

Original: We use trajectories from π H (expert) as supervised data and train π c to predict the expert action under the state distribution induced by the expert.
Revised: We use trajectories from π H (expert) as supervised data and train π c to predict the expert action under the state distribution induced by the expert.

Original: The surrogate loss for imitation learning is:
Revised: The surrogate loss for imitation learning is:

Original: [Equation]
Revised: [Equation]

Original: where d ∗ is the state-visitation distribution induced by π H .
Revised: where d ∗ is the state-visitation distribution induced by π H .

Original: To avoid compounding errors due to state distribution mismatch between the expert and the student, we adopt the Dataset Aggregation (DAgger) algorithm (Ross et al., 2011).
Revised: To avoid compounding errors due to state distribution mismatch between the expert and the student, we adopt the Dataset Aggregation (DAgger) algorithm (Ross et al., 2011).

Original: Our training dataset D is initialized with trajectories from the expert.
Revised: Our training dataset D is initialized with trajectories from the expert.

Original: After iteration i of training, we sample some trajectories from the current student ( π ( i ) c ), label the actions in these trajectories using the expert and form a dataset D i .
Revised: After iteration i of training, we sample some trajectories from the current student ( π ( i ) c ), label the actions in these trajectories using the expert and form a dataset D i .

Original: Training for iteration i + 1 then uses {D∪D 1 . . .
Revised: Training for iteration i + 1 then uses {D ∪ D 1 . . .

Original: ∪D i } to minimize the  loss.
Revised: ∪ D i } to minimize the KL-divergence loss.

Original: This helps to achieve a policy that performs well under its own induced state distribution.
Revised: This helps to achieve a policy that performs well under its own induced state distribution.

Original: The direction of KL-divergence in Equation 1 encourages high entropy in π c , and empirically, we found this to be marginally better than the reverse direction.
Revised: The direction of KL-divergence in Equation 2 encourages high entropy in π c , and empirically, we found this to be marginally better than the reverse direction.

Original: For policies with Gaussian actions, the KL has a closed form and therefore the surrogate loss is easily optimized using a ﬁrst order method.
Revised: For policies with Gaussian actions, the KL has a closed form and therefore the surrogate loss is easily optimized using a ﬁrst order method.

Original: In experiments, we found that this crossover operator is very efﬁcient in terms of sample complexity, which only requires  a small size of rollout samples. 
Revised: In experiments, we found that this crossover operator is very efﬁcient in terms of sample complexity, as it requires only a small size of rollout samples. More implementation details can be found in Appendix 6.3.

Original: 3.3. MUTATE( m (cid:83)
Revised: 3.3. MUTATE

Original: [Equation]
Revised: [Equation]

Original: This operator modiﬁes (in parallel) each policy of the input policy ensemble by running some iterations of a policy gradient algorithm.
Revised: This operator modiﬁes (in parallel) each policy of the input policy ensemble by running some iterations of a policy gradient algorithm.

Original: The policies have different initial parameters and are updated with high-variance gradients estimated using rollout trajectories.
Revised: The policies have different initial parameters and are updated with high-variance gradients estimated using rollout trajectories.

Original: This leads to sufﬁcient genetic diversity and good exploration of the state-space, especially in the initial rounds of GPO.
Revised: This leads to sufﬁcient genetic diversity and good exploration of the state-space, especially in the initial rounds of GPO.

Original: For two popular policy gradient algorithms—PPO and A2C—the gradients for policy π i are calculated as
Revised: For two popular policy gradient algorithms—PPO and A2C—the gradients for policy π i are calculated as

Original: [Equation]
Revised: [Equation]

Original: where ˆ E i,t [ ... ] indicates the empirical average over a ﬁnite batch of samples from π i , and ˆ A t is the advantage.
Revised: where ˆ E i,t [ ... ] indicates the empirical average over a ﬁnite batch of samples from π i , and ˆ A t is the advantage.

Original: We use an MLP to model the critic baseline V π ( s t ) for advantage estimation.
Revised: We use an MLP to model the critic baseline V π ( s t ) for advantage estimation.

Original: PPO does multiple updates on the policy π θ i using the same batch of data collected using π θ ( old ) i , whereas A2C does only a single update.
Revised: PPO does multiple updates on the policy π θ i using the same batch of data collected using π θ ( old ) i , whereas A2C does only a single update.

Original: During mutation, a policy π i can also use data samples from other similar policies in the ensemble for off-policy learning.
Revised: During mutation, a policy π i can also use data samples from other similar policies in the ensemble for off-policy learning.

Original: A larger data-batch (generally) leads to a better estimate of the gradient and stabilizes learning in policy gradient methods.
Revised: A larger data-batch (generally) leads to a better estimate of the gradient and stabilizes learning in policy gradient methods.

Original: When using data-sharing, the gradients for π i are
Revised: When using data-sharing, the gradients for π i are

Original: [Equation]
Revised: [Equation]

Original: where S i ≡ { j | KL [ π i , π j ] < (cid:15) before the start of current round of mutation } contains similar policies to π i (including π i ).3.3. SELECT
Revised: where S i ≡ { j | KL [ π i , π j ] < (cid:15) before the start of current round of mutation } contains similar policies to π i (including π i ).

Original: [Equation]
Revised: [Equation]

Original: Given a set of m policies and a ﬁtness function, this operator returns a set of policy-couples for use in the crossover step.
Revised: Given a set of m policies and a ﬁtness function, this operator returns a set of policy-couples for use in the crossover step.

Original: From all possible (cid:0) m 2 (cid:1) couples, the ones with maximum ﬁtness are selected.
Revised: From all possible (cid:0) m 2 (cid:1) couples, the ones with maximum ﬁtness are selected.

Original: The ﬁtness function f ( π x , π y ) can be deﬁned according two criteria, as below
Revised: The ﬁtness function f ( π x , π y ) can be deﬁned according two criteria, as below.

Original: • Performance ﬁtness as sum of expected returns of both policies, i.e. f ( π x , π y ) def =
Revised: • Performance ﬁtness as sum of expected returns of both policies, i e. f ( π x , π y ) def =

Original: E τ ∼ π x
Revised: E τ ∼ π x

Original: [ R ( τ )]
Revised: [ R ( τ )]

Original: + E τ ∼ π y
Revised: + E τ ∼ π y

Original: [ R ( τ )]
Revised: [ R ( τ )]

Original: • Diversity ﬁtness as KL-divergence between policies,
Revised: • Diversity ﬁtness as KL-divergence between policies,

Original: i
Revised: i

Original: e. f ( π x , π y ) def =
Revised: e. f ( π x , π y ) def =

Original: KL [ π x , π y ]
Revised: KL [ π x , π y ]

Original: While the ﬁrst variant favors couples with high cumulative performance, the second variant explicitly encourages crossover between diverse (high KL divergence) parents.
Revised: While the ﬁrst variant favors couples with high cumulative performance, the second variant explicitly encourages crossover between diverse (high KL divergence) parents.

Original: A linear combination provides a trade-off of these two measures of ﬁtness that can vary during the genetic optimization process.
Revised: A linear combination provides a trade-off of these two measures of ﬁtness that can vary during the genetic optimization process.

Original: In the early rounds, a relatively higher weight could be provided to KL-driven ﬁtness to encourage exploration of the state-space.
Revised: In the early rounds, a relatively higher weight could be provided to KL-driven ﬁtness to encourage exploration of the state-space.

Original: The weight could be annealed with rounds of Algorithmfor encouraging high-performance policies.
Revised: The weight could be annealed with rounds of Algorithmfor encouraging high-performance policies.

Revised: For our experiments, we use a simple variant where we put all the weight on the performance ﬁtness for all rounds, and rely on the randomness in the starting seed for different policies in the ensemble for diversity in the initial rounds.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct experiments to measure the efﬁcacy and robustness of the proposed GPO algorithm on a set of continuous control benchmarks.
Revised: In this section, we conduct experiments to measure the efﬁcacy and robustness of the proposed GPO algorithm on a set of continuous control benchmarks.

Original: We begin by describing the experimental setup and our policy representation.
Revised: We begin by describing the experimental setup and our policy representation.

Original: We then analyze the effect of our crossover operator.
Revised: We then analyze the effect of our crossover operator.

Original: This is followed by learning curves for the simulated environments and comparison with baselines.
Revised: This is followed by learning curves for the simulated environments, comparison with baselines and ablations.

Original: We concludethe section with  discussion on the quality of policies learned by GPO and scalability issues.
Revised: We conclude with a discussion on the quality of policies learned by GPO and scalability issues.

Original: 4. S ETUP
Revised: 4. S ETUP

Original: All our experiments are done using the OpenAI rllab framework (Duan et al., 2016).
Revised: All our experiments are done using the OpenAI rllab framework (Duan et al., 2016).

Original: We benchmark 9 continuous-control locomotion tasks based on the MuJoCo physics simulator 1 .
Revised: We benchmark 9 continuous -control locomotion tasks based on the MuJoCo physics simulator 1 .

Original: All our control policies are Gaussian, with the mean parameterized by a neural network of two hidden layers ( hidden units each), and linear units for the ﬁnal output layer.
Revised: All our control policies are Gaussian, with the mean parameterized by a neural network of two hidden layers ( hidden units each), and linear units for the ﬁnal output layer.

Original: The diagonal co-variance matrix is learnt as a parameter, independent of the input observation, similar to (Schulman et al., 2015; 2017).
Revised: The diagonal co-variance matrix is learnt as a parameter, independent of the input observation, similar to (Schulman et al., 2015; 2017).

Original: The binary policy ( π S ) used for crossover has two hidden layers (32 hidden units each), followed by a softmax.
Revised: The binary policy ( π S ) used for crossover has two hidden layers (32 hidden units each), followed by a softmax.

Original: The value-function baseline used for advantage estimation also has two hidden layers (32 hidden units each).
Revised: The value-function baseline used for advantage estimation also has two hidden layers (32 hidden units each).

Original: All neural networks use tanh as the non-linearity at the hidden units.
Revised: All neural networks use tanh as the non-linearity at the hidden units.

Original: We show results with PPO and A2C as policy gradient algorithms for mutation.
Revised: We show results with PPO and A2C as policy gradient algorithms for mutation.

Original: PPO performs 10 steps of full-batch gradient descent on the policy parameters using the same collected batch of simulation data, while A2C does a single descent step.
Revised: PPO performs 10 steps of full-batch gradient descent on the policy parameters using the same collected batch of simulation data, while A2C does a single descent step.

Original: Other hyperparameters are in the Appendix.
Revised: Other hyperparameters are in Appendix 6.3.

Original: 4. C ROSSOVER P ERFORMANCE
Revised: 4. C ROSSOVER P ERFORMANCE

Original: To measure the efﬁcacy of our crossover operator, we run GPO on the HalfCheetah environment, and plot the performance of all the policies involved in 8 different crossovers that occur in the ﬁrst round of Algorithm 1.
Revised: To measure the efﬁcacy of our crossover operator, we run GPO on the HalfCheetah environment, and plot the performance of all the policies involved in 8 different crossovers that occur in the ﬁrst round of Algorithm 1.

Original: Figure 2a shows the average episode reward for the parent policies and their corresponding child.
Revised: Figure 3a shows the average episode reward for the parent policies and their corresponding child.

Original: All bars are normalized to the ﬁrst parent in each crossover.
Revised: All bars are normalized to the ﬁrst parent in each crossover.

Original: The left subplot depicts state-space crossover.
Revised: The left subplot depicts state-space crossover.

Original: We observe that in many cases, the child either maintains or improves on the better parent.
Revised: We observe that in many cases, the child either maintains or improves on the better parent.

Original: This is in contrast to the right subplot where parameter-space crossover breaks the information structure contained in either of the parents to create a child with very low performance.
Revised: This is in contrast to the right subplot where parameter-space crossover breaks the information structure contained in either of the parents to create a child with very low performance.

Original: To visualize the state-space crossover better, in Figure 2b we plot the state-visitation 
Revised: To visualize the state-space crossover better, in Figure 3b we plot the state-visitation distribution for high reward rollouts from all policies involved in one of the crossovers. All states

Original: [Table]
Revised: [Table]

Original: distribution for high reward rollouts from all policies involved in one of the crossovers.
Revised: InvertedDoublePendulum

Original: All states are projected from a 20 dimensional space (for HalfCheetah) into a 2D space by t-SNE (Maaten & Hinton, 2008).
Revised: [Table] are projected from a 20 dimensional space (for HalfCheetah) into a 2D space by t-SNE (Maaten & Hinton, 2008).

Original: Notwithstanding artifacts due to dimensionality reduction, we observe that high reward rollouts from the child policy obtained with state-space crossover visit regions frequented by both the parents, unlike the parameter-space crossover (rightmost subplot) where the policy mostly meanders in regions for which neither of the parents have strong supervision.
Revised: Notwithstanding artifacts due to dimensionality reduction, we observe that high reward rollouts from the child policy obtained with state-space crossover visit regions frequented by both the parents, unlike the parameter-space crossover (rightmost subplot) where the policy mostly meanders in regions for which neither of the parents have strong supervision.

Original: 4. C OMPARISON WITH P OLICY G RADIENT M ETHODS
Revised: 4. C OMPARISON WITH P OLICY G RADIENT M ETHODS

Original: In this subsection [Table] we compare the performance of policies trained using GPO with those trained with standard policy gradient algorithms.
Revised: In this subsection, we compare the performance of policies trained using GPO with those trained with standard policy gradient algorithms.

Original: GPO is run for 12 rounds (Algorithm 1) with a population size of 8, and simulates 8 million timesteps in total for each environment (1 million steps per candidate policy).
Revised: GPO is run for 12 rounds (Algorithm 1) with a population size of 8, and simulates 8 million timesteps in total for each environment (1 million steps per candidate policy).

Original: We compare with two baselines which use the same amount of data.
Revised: We compare with two baselines which use the same amount of data.

Original: The ﬁrst baseline algorithm, Single , trains 8 independent policies with policy gradient using 1 million timesteps each, and selects the policy with the maximum performance at the end of training.
Revised: The ﬁrst baseline algorithm, Single , trains 8 independent policies with policy gradient using 1 million timesteps each, and selects the policy with the maximum performance at the end of training.

Original: Unlike
Revised: Unlike

Original: GPO, these policies do not participate in state-space crossover or interact in any way.
Revised: GPO, these policies do not participate in state-space crossover or interact in any way.

Original: The second baseline algorithm, Joint , trains a single policy with policy gradient using 8 million timesteps.
Revised: The second baseline algorithm, Joint , trains a single policy with policy gradient using 8 million timesteps.

Original: [Table]

Original: Both Joint and Single do the same number of gradient update steps on the policy parameters, but each gradient step in Joint uses 8 times the batch-size.
Revised: Both Joint and Single do the same number of gradient update steps on the policy parameters, but each gradient step in Joint uses 8 times the batch-size.

Original: For all methods, we replicate 8 runs with different seeds.
Revised: For all methods, we replicate 8 runs with different seeds and show the mean and standard error.

Original: Figure 3 plots the moving average of per-episode reward when training with PPO as the policy gradient method for all algorithms.
Revised: Figure 4 plots the moving average of per-episode reward when training with PPO as the policy gradient method for all algorithms.

Revised: The x-axis is the total timesteps of simulation, including the data required for DAgger imitation learning in the crossover step.

Original: We observe that GPO achieves better performance than Single is almost all environments.
Revised: We observe that GPO achieves better performance than Single is almost all environments.

Original: Joint is a more challenging baseline since each gradient step uses a larger batch-size, possibly leading to well-informed, low-variance gradient estimates.
Revised: Joint is a more challenging baseline since each gradient step uses a larger batch-size, possibly leading to well-informed, low-variance gradient estimates.

Original: Nonetheless, GPO reaches a much better score for environments such as Walker2D and HalfCheetah, and also their more difﬁcult (hilly) versions.
Revised: Nonetheless, GPO reaches a much better score for environments such as Walker2D and HalfCheetah, and also their more difﬁcult (hilly) versions.

Original: We think this is due to better exploration and exploitation by the nature of the genetic algorithm.
Revised: We believe this is due to better exploration and exploitation by the nature of the genetic algorithm.

Original: The performance at the end of training is shown in Table 1.
Revised: The performance at the end of training is shown in Table 1.

Original: Results with A2C as the policy gradient method are in Figure 4 and Table 2.
Revised: Results with A2C as the policy gradient method are in Appendix 6.1. With

Original: Here, GPO beats the baselines in all but one environments.
Revised: A2C, GPO beats the baselines in all but one environments.

Original: In summary, these results indicate that, with the new crossover and mutation operators, genetic algorithms could be an alternative policy optimization approach that competes with the state-of-the-arts policy gradient methods. 
Revised: In summary, these results indicate that, with the new crossover and mutation operators, genetic algorithms could be an alternative policy optimization approach that competes with the state-of-the-arts policy gradient methods. 4. A BLATIONS Our policy optimization procedure uses crossover, select and mutate operators on an ensemble of policies over multiple rounds.

Revised: In this section, we perform ablation studies to measure the impact on performance when only certain operator(s) are applied.

Revised: Figure 5 shows the results and uses the following symbols:

Revised: • Crossover (C) - Presence of the symbol indicates state-space crossover using imitation learning; otherwise a simple crossover is done by copying the parameters of the stronger parent to the offspring policy.

Original: 4. R OBUSTNESS AND S CALABILITY
Revised: • Select (S) - Presence of the symbol denotes use of a ﬁtness function (herein performanceﬁtness, Section 3.3.3) to select policy-pairs to breed; otherwise random selection is used. • Data-sharing during Mutate (M) - Mutation in GPO is done using policy-gradient, and policies in the ensemble share batch samples with other similar policies (Section 3.3.2). We use this symbol when sharing is enabled; otherwise omit it. In Figure we refer to setting where none of the components { C, S, M } are used as Base and apply components over it.

Revised: We also show Single which trains an ensemble of policies that do not interact in any manner.

Revised: We note that each of the components applied in isolation provide some improvement over Single , with data-sharing (M) having the highest beneﬁt.

Revised: Also, combining two components generally leads to better performance than using either of the constituents alone.

Revised: Finally using all components results in GPO and it gives the best performance.

Revised: The normalized numbers are mentioned in the ﬁgure caption.

Revised: 4. R OBUSTNESS AND S CALABILITY

Original: The SELECTION operator selects high-performing individuals for crossover in every round of Algorithm 1.
Revised: The SELECTION operator selects high-performing individuals for crossover in every round of Algorithm 1.

Original: Natural selection weeds out poorly-performing policies during the optimization process.
Revised: Natural selection weeds out poorly-performing policies during the optimization process.

Original: In Figure5, we measure the average episode reward for each of the policies in the ensemble at the ﬁnal round of GPO.
Revised: In Figure we measure the average episode reward for each of the policies in the ensemble at the ﬁnal round of GPO.

Original: We compare this with the ﬁnal performance of the 8 policies trained using the
Revised: We compare this with the ﬁnal performance of the 8 policies trained using the

Original: Single baseline.
Revised: Single baseline.

Original: We conclude that the GPO policies are more robust.
Revised: We conclude that the GPO policies are more robust.

Original: In Figure 6, we experiment with varying the population size for GPO.
Revised: In Figure 7, we experiment with varying the population size for GPO.

Original: All the policies in this experiment use the same batch-size for the gradient steps and do the same number of gradient steps.
Revised: All the policies in this experiment use the same batch-size for the gradient steps and do the same number of gradient steps.

Original: Performance improves by increasing the population size suggesting that GPO is a scalable optimization procedure.
Revised: Performance improves by increasing the population size suggesting that GPO is a scalable optimization procedure.

Original: Moreover, the MUTATE and CROSSOVER genetic operators lend themselves perfectly to multiprocessor parallelism.
Revised: More- over, the MUTATE and CROSSOVER genetic operators lend themselves perfectly to multiprocessor parallelism.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented Genetic Policy Optimization (GPO), a new approach to deep policy optimization which combines ideas from evolutionary algorithms and reinforcement learning.
Revised: We presented Genetic Policy Optimization (GPO), a new approach to deep policy optimization which combines ideas from evolutionary algorithms and reinforcement learning.

Original: First, GPO does efﬁcient policy crossover in state space using imitation learning.
Revised: First, GPO does efﬁcient policy crossover in state space using imitation learning.

Original: Our experiments show the beneﬁts of crossover in state-space over parameter-space for deep neural network policies.
Revised: Our experiments show the beneﬁts of crossover in state-space over parameter-space for deep neural network policies.

Original: Second, GPO mutates the policy weights by using advanced policy gradient algorithms instead of random perturbations.
Revised: Second, GPO mutates the policy weights by using advanced policy gradient algorithms instead of random perturbations.

Original: We conjecture that the noisy gradient estimates used by policy gradient methods offer sufﬁcient genetic diversity, while providing a strong learning signal.
Revised: We conjecture that the noisy gradient estimates used by policy gradient methods offer sufﬁcient genetic diversity, while providing a strong learning signal.

Original: Our experiments on several
Revised: Our experiments on several

Original: MuJoCo locomotion tasks show that GPO has superior performance over the state-of-the-art policy gradient methods and achieves comparable or higher sample efﬁciency.
Revised: MuJoCo locomotion tasks show that GPO has superior performance over the state-of-the-art policy gradient methods and achieves comparable or higher sample efﬁciency.

Original: Future advances in policy gradient methods and imitation learning will also likely improve the performance of GPO for challenging RL tasks.
Revised: Future advances in policy gradient methods and imitation learning will also likely improve the performance of GPO for challenging RL tasks

Original:  A PPENDIX 6. H YPERPARAMETERS
Revised:  A PPENDIX 6. P ERFORMANCE WITH A2C

Original: [Table]
Revised: [Table]

Revised: 6. R LLAB VS . G YM ENVIRONMENTS

Revised: We use the OpenAI rllab framework, including the MuJoCo environments provided therein, for all our experiments.

Revised: There are subtle differences in the environments included in rllab and OpenAI

Revised: Gym repositories 2 in terms of the coefﬁcients used for different reward components and aliveness bonuses.

Revised: In Figure 9, we compare GPO and Joint baseline using PPO on three Gym environments, for two different time-horizon values (512, 1024).

Revised: 6. C ROSSOVER I MPLEMENTATION D ETAILS

Revised: The crossover stage is divided into two phases - a) training the binary policy ( π S ) and 2) imitation learning.

Revised: For training π S , the dataset consists of trajectories from the parents involved in the crossover.

Revised: We combine the trajectories from the parents’ previous mutation phase, and ﬁlter based on trajectory rewards.

Revised: For our experiments, we select top 60% trajectories from the pool, although we did not ﬁnd the ﬁnal GPO performance to be very sensitive to this hyperparameter.

Revised: We doepochs of supervised training with Adam (mini-batch=64, learning rate=5e-4) on the loss deﬁned in

Revised: Equation 1.

Revised: After training π S , we obtain expert trajectories (5k transitions) from π H .

Revised: Imitation learning is done in a loop which is run for 10 iterations.

Revised: In each iteration i , we form the dataset D i using existing expert trajectories, plus new trajectories (500 transitions) sampled from the student (child) policy π ( i ) c with the actions labelled by the expert.

Revised: Therefore, the ratio of student to expert transitions in D i is linearly increased from 0 to 1 over the 10 iterations.

Revised: We then update the student by minimizing the

Revised: KL divergence objective over D i .

Revised: We do 10 epochs of training with Adam (mini-batch=64, learning rate=5e-4) in each iteration.

Revised: 6.3. O THER H YPERPARAMETERS

Revised: - Horizon (T) =- Discount ( γ )

Revised: = 0. - PPO epochs =- PPO/A2C batch-size (GPO, Single) =- PPO/

Revised: A2C batch-size (Joint) =

Revised: 6. M ISCELLANEOUS 6.4. W ALL - CLOCK TIME

Revised: Table 3 shows the wall-clock time (in minutes), averaged over 3 runs, for GPO and the Joint baseline.

Revised: All runs use the same number of simulation timesteps, and are done on an Intel Xeon machinewith 12 cores.

Revised: For GPO, Mutate takes the major chunk of total time.

Revised: This is partially due to the fact that data-sharing between ensemble policies leads to communication overheads.

Revised: Having said that, our current implementation based on Python Multiprocessing module and ﬁle-based sharing in Unix leaves much on the table in terms of improving the efﬁciency for Mutate, for example by using MPI.

Revised: Joint trains 1 policy with 8 × the number of samples as each policy in the GPO ensemble.

Revised: However, sample-collection exploits 8-way multi-core parallelism by simulating multiple independent environments in separate processes.

Revised: [Table]

Revised: 6.4. S CALABILITY

Revised: , we ran Walker2D with different population-size values, and compared performance.

Revised: All policies used the same batch-size and number of gradient steps, making the total number of simulation timesteps grow with the population-size.

Revised: In Figure 10, we show results for the same environment but reduce the batch-size for each gradient step in proportion to the increase in populationsize.

Revised: Therefore, all experiments here use equal simulation timesteps.

Revised: We observe that the samplecomplexity for a population of 32 is quite competitive with our default GPO value of 8.