Original: ERL-R E 2 : E FFICIENT E VOLUTIONARY R EINFORCEMENT L EARNING WITH S HARED S TATE R EPRESENTATION AND I NDIVIDUAL P OLICY R EPRESENTATION
Revised: ERL-R E 2 : E FFICIENT E VOLUTIONARY R EINFORCEMENT L EARNING WITH S HARED S TATE R EPRESENTATION AND I NDIVIDUAL P OLICY R EPRESENTATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep Reinforcement Learning (Deep RL) and Evolutionary Algorithm (EA) are two major paradigms of policy optimization with distinct learning principles, i.e., gradient-based v.s. gradient-free.
Revised: Deep Reinforcement Learning (Deep RL) and Evolutionary Algorithms (EA) are two major paradigms of policy optimization with distinct learning principles, i.e., gradient-based v.s. gradient-free.

Original: An appealing research direction is integrating
Revised: An appealing research direction is integrating

Original: Deep RL and EA to devise new methods by fusing their complementary advantages.
Revised: Deep RL and EA to devise new methods by fusing their complementary advantages.

Original: However, existing works on combining Deep RL and EA have two common drawbacks: 1) the RL agent and EA agents learn their policies individually, neglecting efficient sharing of useful common knowledge; 2) parameter-level policy optimization guarantees no semantic level of behavior evolution for the EA side.
Revised: However, existing works on combining Deep RL and EA have two common drawbacks: 1) the RL agent and EA agents learn their policies individually, neglecting efficient sharing of useful common knowledge; 2) parameter-level policy optimization guarantees no semantic level of behavior evolution for the EA side.

Original: In this paper, we propose E volutionary R einforcement
Revised: In this paper, we propose E volutionary R einforcement

Original: L earning with Two-scale
Revised: L earning with Two-scale

Original: State Re presentation and Policy Re presentation (ERL-Re 2 ), a novel solution to the aforementioned two drawbacks.
Revised: State Re presentation and Policy Re presentation (ERL-Re 2 ), a novel solution to the aforementioned two drawbacks.

Original: The key idea of ERL-Re 2 is two-scale representation: all EA and RL policies share the same nonlinear state representation while maintaining individual linear policy representations.
Revised: The key idea of ERL-Re 2 is two-scale representation: all EA and RL policies share the same nonlinear state representation while maintaining individual linear policy representations.

Original: The state representation conveys expressive common features of the environment learned by all the agents collectively; the linear policy representation provides a favorable space for efficient policy optimization, where novel behavior-level crossover and mutation operations can be performed.
Revised: The state representation conveys expressive common features of the environment learned by all the agents collectively; the linear policy representation provides a favorable space for efficient policy optimization, where novel behavior-level crossover and mutation operations can be performed.

Original: Moreover, the linear policy representation allows convenient generalization of policy fitness with the help of Policy-extended Value Function
Revised: Moreover, the linear policy representation allows convenient generalization of policy fitness with the help of Policy-extended Value Function

Original: Approximator (PeVFA), further improving the sample efficiency of fitness estimation.
Revised: Approximator (PeVFA), further improving the sample efficiency of fitness estimation.

Original: The experiments on a range of continuous control tasks show that ERL-Reconsistently outperforms strong baselines and achieves significant improvement over both its Deep RL and EA components.
Revised: The experiments on a range of continuous control tasks show that ERL-Reconsistently outperforms strong baselines and achieves significant improvement over both its Deep RL and EA components.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) has achieved many successes in robot control (Johannink et al., 2019), game AI (Vinyals et al., 2019), supply chain Ni et al.
Revised: Reinforcement learning (RL) has achieved many successes in robot control (Johannink et al., 2019), game AI (Vinyals et al., 2019), supply chain Ni et al.

Original: (2021) and etc.
Revised: (2021) and etc.

Original: With function approximation like deep neural networks, the policy can be learned efficiently by trial-and-error with reliable gradient updates.
Revised: With function approximation like deep neural networks, the policy can be learned efficiently by trial-and-error with reliable gradient updates.

Original: However, RL is widely known to be unstable, poor in exploration, and struggling when the gradient signals are noisy and less informative.
Revised: However, RL is widely known to be unstable, poor in exploration, and struggling when the gradient signals are noisy and less informative.

Original: By contrast, Evolutionary Algorithm (EA) (Bäck & Schwefel, 1993) is a class of black-box optimization methods, which is demonstrated to be competitive with RL (Such et al., 2017).
Revised: By contrast, Evolutionary Algorithms (EA) (Bäck & Schwefel, 1993) are a class of black-box optimization methods, which is demonstrated to be competitive with RL (Such et al., 2017).

Original: EA models natural evolution processes by maintaining a population of individuals and searches for favorable solutions by iteration.
Revised: EA model natural evolution processes by maintaining a population of individuals and searches for favorable solutions by iteration.

Original: In each iteration, individuals with high fitness are selected to produce offspring by inheritance and variation, while those with low fitness are eliminated.
Revised: In each iteration, individuals with high fitness are selected to produce offspring by inheritance and variation, while those with low fitness are eliminated.

Original: Different from RL, EA is gradient-free and offers several strengths: strong exploration ability, robustness, and stable convergence (Sigaud, 2022).
Revised: Different from RL, EA are gradient-free and offers several strengths: strong exploration ability, robustness, and stable convergence (Sigaud, 2022).

Original: Despite the advantages, one major bottleneck of EA is the low sample efficiency due to the iterative evaluation of the population.
Revised: Despite the advantages, one major bottleneck of EA is the low sample efficiency due to the iterative evaluation of the population.

Original: This issue becomes more stringent when the policy space is large (Sigaud, 2022).
Revised: This issue becomes more stringent when the policy space is large (Sigaud, 2022).

Original: Since EA and RL have distinct and complementary advantages, a natural idea is to combine these two heterogeneous policy optimization approaches and devise better policy optimization algorithms.
Revised: Since EA and RL have distinct and complementary advantages, a natural idea is to combine these two heterogeneous policy optimization approaches and devise better policy optimization algorithms.

Original: Many efforts in recent years have been made in this direction (Khadka & Tumer, 2018;
Revised: Many efforts in recent years have been made in this direction (Khadka & Tumer, 2018;

Original: Gangwani & Peng, 2018; Khadka et al., 2019; Bodnar et al., 2020; Wang et al., 2022; Pourchot & Sigaud, 2019).
Revised: Gangwani & Peng, 2018; Khadka et al., 2019; Bodnar et al., 2020; Wang et al., 2022; Pourchot & Sigaud, 2019).

Original: One representative work is ERL (Khadka & Tumer, 2018) which combines Genetic Algorithm (GA)
Revised: One representative work is ERL (Khadka & Tumer, 2018) which combines Genetic Algorithm (GA)

Original: (Mitchell, 1998) and DDPG (Lillicrap et al., 2016).
Revised: (Mitchell, 1998) and DDPG (Lillicrap et al., 2016).

Original: ERL maintains an evolution population and a RL agent meanwhile.
Revised: ERL maintains an evolution population and a RL agent meanwhile.

Original: The population and the RL agent interact with each other in a coherent framework: the RL agent learns by DDPG with diverse off-policy experiences collected by the population; while the population includes a copy of the RL agent periodically among which genetic evolution operates.
Revised: The population and the RL agent interact with each other in a coherent framework: the RL agent learns by DDPG with diverse off-policy experiences collected by the population; while the population includes a copy of the RL agent periodically among which genetic evolution operates.

Original: In this way, EA and RL cooperate during policy optimization.
Revised: In this way, EA and RL cooperate during policy optimization.

Original: Subsequently, many variants and improvements of ERL are proposed, e.g., to incorporate Cross-Entropy Method (CEM) (Pourchot & Sigaud, 2019) rather than GA (Pourchot & Sigaud, 2019), to devise gradientbased genetic operators (Gangwani & Peng, 2018), to use multiple parallel RL agents (Khadka et al., 2019) and etc.
Revised: Subsequently, many variants and improvements of ERL are proposed, e.g., to incorporate Cross-Entropy Method (CEM) (Pourchot & Sigaud, 2019) rather than GA (Pourchot & Sigaud, 2019), to devise gradientbased genetic operators (Gangwani & Peng, 2018), to use multiple parallel RL agents (Khadka et al., 2019) and etc.

Original: However, we observe that most existing methods seldom break the performance ceiling of either their EA or RL components (e.g., Swimmer and Humanoid on MuJoCo are dominated by EA and RL respectively).
Revised: However, we observe that most existing methods seldom break the performance ceiling of either their EA or RL components (e.g., Swimmer and Humanoid on MuJoCo are dominated by EA and RL respectively).

Original: This indicates that the strengths of EA and RL are not sufficiently blended.
Revised: This indicates that the strengths of EA and RL are not sufficiently blended.

Original: We attribute this to two major drawbacks.
Revised: We attribute this to two major drawbacks.

Original: First, each agent of EA and RL learns its policy individually.
Revised: First, each agent of EA and RL learns its policy individually.

Original: The state representation learned by individuals can inevitably be redundant yet specialized (Dabney et al., 2021), thus slowing down the learning and limiting the convergence performance.
Revised: The state representation learned by individuals can inevitably be redundant yet specialized (Dabney et al., 2021), thus slowing down the learning and limiting the convergence performance.

Original: Second, typical evolutionary variation occurs at the level of parameter (e.g., network weights).
Revised: Second, typical evolutionary variation occurs at the level of parameter (e.g., network weights).

Original: It guarantees no semantic level of evolution and may induce policy crash (Bodnar et al., 2020).
Revised: It guarantees no semantic level of evolution and may induce policy crash (Bodnar et al., 2020).

Original: In the literature of linear approximation RL (Sutton & Barto, 1998) and state representation learning (Chung et al., 2019; Dabney et al., 2021; Kumar et al., 2021), a policy is usually understood as the composition of nonlinear state features and linear policy weights.
Revised: In the literature of linear approximation RL (Sutton & Barto, 1998) and state representation learning (Chung et al., 2019; Dabney et al., 2021; Kumar et al., 2021), a policy is usually understood as the composition of nonlinear state features and linear policy weights.

Original: Taking this inspiration, we propose a new approach named E volutionary R einforcement
Revised: Taking this inspiration, we propose a new approach named E volutionary R einforcement

Original: L earning with Two-scale State
Revised: L earning with Two-scale State

Original: Re presentation and Policy Re presentation (ERL-Re 2 ) to address the aforementioned two drawbacks.
Revised: Re presentation and Policy Re presentation (ERL-Re 2 ) to address the aforementioned two drawbacks.

Original: ERL-Re 2 is devised based on a novel concept, i.e., two-scale representation: all EA and RL agents maintained in ERL-Re 2 are composed of a shared nonlinear state representation and an individual linear policy representation.
Revised: ERL-Re 2 is devised based on a novel concept, i.e., two-scale representation: all EA and RL agents maintained in ERL-Re 2 are composed of a shared nonlinear state representation and an individual linear policy representation.

Original: The shared state representation takes the responsibility of learning general and expressive features of the environment, which is not specific to any single policy, e.g., the common decision-related knowledge.
Revised: The shared state representation takes the responsibility of learning general and expressive features of the environment, which is not specific to any single policy, e.g., the common decision-related knowledge.

Original: In particular, it is learned by following a unifying update direction derived from value function maximization regarding all EA and RL agents collectively.
Revised: In particular, it is learned by following a unifying update direction derived from value function maximization regarding all EA and RL agents collectively.

Original: Thanks to the expressivity of the shared state representation, the individual policy representation can have a simple linear form.
Revised: Thanks to the expressivity of the shared state representation, the individual policy representation can have a simple linear form.

Original: It leads to a fundamental distinction of ERL-Re 2 : evolution and reinforcement occur in the linear policy representation space rather than in a nonlinear parameter (e.g., policy network) space as the convention.
Revised: It leads to a fundamental distinction of ERL-Re 2 : evolution and reinforcement occur in the linear policy representation space rather than in a nonlinear parameter (e.g., policy network) space as the convention.

Original: Thus, policy optimization can be more efficient with ERL-Re 2 .
Revised: Thus, policy optimization can be more efficient with ERL-Re 2 .

Original: In addition, we propose novel behavior-level crossover and mutation that allow to imposing variations on designated dimensions of action while incurring no interference on the others.
Revised: In addition, we propose novel behavior-level crossover and mutation that allow to imposing variations on designated dimensions of action while incurring no interference on the others.

Original: Compared to parameter-level operators, our behavior-level operators have clear genetic semantics of behavior, thus are more effective and stable.
Revised: Compared to parameter-level operators, our behavior-level operators have clear genetic semantics of behavior, thus are more effective and stable.

Original: Moreover, we further reduce the sample cost of EA by introducing a new surrogate of fitness, based on the convenient incorporation of Policy-extended
Revised: Moreover, we further reduce the sample cost of EA by introducing a new surrogate of fitness, based on the convenient incorporation of Policy-extended

Original: Value Function Approximator (PeVFA) favored by the linear policy representations.
Revised: Value Function Approximator (PeVFA) favored by the linear policy representations.

Original: Without loss of generality, we use GA and TD3 (and DDPG) for the concrete choices of EA and RL algorithms.
Revised: Without loss of generality, we use GA and TD3 (and DDPG) for the concrete choices of EA and RL algorithms.

Original: Finally, we evaluate ERL-Re 2 on MuJoCo continuous control tasks with strong ERL baselines and typical RL algorithms, along with a comprehensive study on ablation, hyperparameter analysis, etc.
Revised: Finally, we evaluate ERL-Re 2 on MuJoCo continuous control tasks with strong ERL baselines and typical RL algorithms, along with a comprehensive study on ablation, hyperparameter analysis, etc.

Original: We summarize our major contributions below: 1) We propose a novel approach ERL-Re 2 to integrate
Revised: We summarize our major contributions below: 1) We propose a novel approach ERL-Re 2 to integrate

Original: EA and RL based on the concept of two-scale representation; 2) We devise behavior-level crossover and mutation which have clear genetic semantics; 3) We empirically show that ERL-Re 2 consistently outperforms related methods and improves both its EA and RL components significantly on MuJoCo.
Revised: EA and RL based on the concept of two-scale representation; 2) We devise behavior-level crossover and mutation which have clear genetic semantics; 3) We empirically show that ERL-Re 2 consistently outperforms related methods and improves both its EA and RL components significantly on MuJoCo.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Reinforcement Learning Consider a Markov decision process (MDP), defined by a tuple ⟨S , A , P , R , γ, T ⟩ .
Revised: Reinforcement Learning Consider a Markov decision process (MDP), defined by a tuple ⟨S , A , P , R , γ, T ⟩ .

Original: At each step t , the agent uses a policy π to select an action a t ∼ π ( s t )
Revised: At each step t , the agent uses a policy π to select an action a t ∼ π ( s t )

Original: ∈
Revised: ∈

Original: A according to the state s t ∈ S and the environment transits to the next state s t +

Revised: A according to the state s t ∈ S and the environment transits to the next state s t +

Original: according to transition function P ( s t , a t ) and the agent receives a reward r t =
Revised: according to transition function P ( s t , a t ) and the agent receives a reward r t =

Original: R ( s t , a t ) .
Revised: R ( s t , a t ) .

Original: The return is defined as the discounted cumulative reward, denoted by R t = (cid:80)
Revised: The return is defined as the discounted cumulative reward, denoted by R t = (cid:80)

Original: Ti = t γ i − t r i where γ ∈ [0 , 1) is the discount factor and
Revised: Ti = t γ i − t r i where γ ∈ [0 , 1) is the discount factor and

Original: T is the maximum episode horizon.
Revised: T is the maximum episode horizon.

Original: The goal of RL is to learn an optimal policy π ∗ that maximizes the expected return.
Revised: The goal of RL is to learn an optimal policy π ∗ that maximizes the expected return.

Original: DDPG (Lillicrap et al., 2016) is a representative off-policy Actor-Critic algorithm, consisting of a deterministic policy π ω (i.e., the actor) and a state-action value function approximation
Revised: DDPG (Lillicrap et al., 2016) is a representative off-policy Actor-Critic algorithm, consisting of a deterministic policy π ω (i.e., the actor) and a state-action value function approximation

Original: Q ψ (i.e., the critic), with the parameters ω and ψ respectively.
Revised: Q ψ (i.e., the critic), with the parameters ω and ψ respectively.

Original: The critic is optimized with the Temporal Difference (TD) (Sutton & Barto, 1998) loss and the actor is updated by maximizing the estimated
Revised: The critic is optimized with the Temporal Difference (TD) (Sutton & Barto, 1998) loss and the actor is updated by maximizing the estimated

Original: Q value.
Revised: Q value.

Original: The loss functions are defined as: L ( ψ ) =
Revised: The loss functions are defined as: L ( ψ ) =

Original: E D [( r + γQ ψ ′ ( s ′ , π ω ′ ( s ′ ))
Revised: E D [( r + γQ ψ ′ ( s ′ , π ω ′ ( s ′ ))

Original: − Q ψ ( s, a )) 2 ] and
Revised: − Q ψ ( s, a )) 2 ] and

Original: L ( ω ) =
Revised: L ( ω ) =

Original: − E D [ Q ψ ( s, π ω ( s ))] , where the experiences ( s, a, r, s ′ ) are sampled from the replay buffer
Revised: − E D [ Q ψ ( s, π ω ( s ))] , where the experiences ( s, a, r, s ′ ) are sampled from the replay buffer

Original: D , ψ ′ and ω ′ are the parameters of the target networks.
Revised: D , ψ ′ and ω ′ are the parameters of the target networks.

Original: TD3 (Fujimoto et al., 2018) improves DDPG by addressing overestimation issue mainly by clipped double- Q learning.
Revised: TD3 (Fujimoto et al., 2018) improves DDPG by addressing overestimation issue mainly by clipped double- Q learning.

Original: Conventional value functions are defined on a specific policy.
Revised: Conventional value functions are defined on a specific policy.

Original: Recently, a new extension called
Revised: Recently, a new extension called

Original: Policy-extended Value Function Approximator (PeVFA) (Tang et al., 2020) is proposed to preserve the values of multiple policies.
Revised: Policy-extended Value Function Approximator (PeVFA) (Tang et al., 2022) is proposed to preserve the values of multiple policies.

Original: Concretely, given some representation χ π of policy π , a PeVFA parameterized by θ takes as input χ π additionally, i.e., Q θ ( s, a, χ π ) .
Revised: Concretely, given some representation χ π of policy π , a PeVFA parameterized by θ takes as input χ π additionally, i.e., Q θ ( s, a, χ π ) .

Original: Through the explicit policy representation χ π , one appealing characteristic of PeVFA is the value generalization among policies (or policy space).
Revised: Through the explicit policy representation χ π , one appealing characteristic of PeVFA is the value generalization among policies (or policy space).

Original: PeVFA is originally proposed to leverage the local value generalization along the policy improvement path to improve online RL (we refer the reader to their original paper).
Revised: PeVFA is originally proposed to leverage the local value generalization along the policy improvement path to improve online RL (we refer the reader to their original paper).

Original: In our work, we adopt PeVFA for the value estimation of the EA population, which naturally fits the ability of PeVFA well.
Revised: In our work, we adopt PeVFA for the value estimation of the EA population, which naturally fits the ability of PeVFA well.

Original: Additionally, different from the on-policy learning of PeVFA adopted in (Tang et al., 2020), we propose a new off-policy learning algorithm of PeVFA which is described later.
Revised: Additionally, different from the on-policy learning of PeVFA adopted in (Tang et al., 2022), we propose a new off-policy learning algorithm of PeVFA which is described later.

Original: Evolutionary Algorithm Evolutionary Algorithm (EA) (Bäck & Schwefel, 1993) is a class of black-box optimization methods.
Revised: Evolutionary Algorithm Evolutionary Algorithms (EA) (Bäck & Schwefel, 1993) are a class of black-box optimization methods.

Original: EA maintains a population of policies P
Revised: EA maintains a population of policies P

Original: = { π 1 , π 2 , ..., π n } in which policy evolution is iteratively performed.
Revised: = { π 1 , π 2 , ..., π n } in which policy evolution is iteratively performed.

Original: In each iteration, all agents interact with the environment for e episodes to obtain Monte Carlo (MC) estimates of policy fitness { f ( π 1 ) , f ( π 2 ) , ..., f ( π n ) } where f ( π ) = 1 e (cid:80) ei =1 [ (cid:80) Tt =

Revised: In each iteration, all agents interact with the environment for e episodes to obtain Monte Carlo (MC) estimates of policy fitness { f ( π 1 ) , f ( π 2 ) , ..., f ( π n ) } where f ( π ) = 1 e (cid:80) ei =1 [ (cid:80) Tt =

Original: r t | π ] .
Revised: r t | π ] .

Original: The policy with higher fitness is more likely to be selected as parents to produce the next generation in many ways such as Genetic Algorithm (GA)
Revised: The policy with higher fitness is more likely to be selected as parents to produce the next generation in many ways such as Genetic Algorithm (GA)

Original: (Mitchell, 1998) and
Revised: (Mitchell, 1998) and

Original: Cross-Entropy Method (CEM) (Pourchot & Sigaud, 2019).
Revised: Cross-Entropy Method (CEM) (Pourchot & Sigaud, 2019).

Original: With GA, offspring are generated by applying genetic operators: the parents π i and π j are selected randomly to produce offspring π
Revised: With GA, offspring are generated by applying genetic operators: the parents π i and π j are selected randomly to produce offspring π

Original: ′ i and π ′ j
Revised: ′ i and π ′ j

Original: by performing the crossover operator, i.e., π ′ i , π ′ j
Revised: by performing the crossover operator, i.e., π ′ i , π ′ j

Original: =
Revised: =

Original: Crossover ( π i , π j ) or the mutation operator π ′ i =
Revised: Crossover ( π i , π j ) or the mutation operator π ′ i =

Original: Mutation ( π i ) .
Revised: Mutation ( π i ) .

Original: In most prior methods, the crossover and mutation operate at the parameter level .
Revised: In most prior methods, the crossover and mutation operate at the parameter level .

Original: Typically, k -point crossover randomly exchange segment-wise (network) parameters of parents while Gaussian mutation adds Gaussian noises to the parameters.
Revised: Typically, k -point crossover randomly exchange segment-wise (network) parameters of parents while Gaussian mutation adds Gaussian noises to the parameters.

Original: Thanks to the diversity brought by abundant candidates and consistent variation, EA has strong exploration ability and convergence.
Revised: Thanks to the diversity brought by abundant candidates and consistent variation, EA has strong exploration ability and convergence.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Recently, an emergent research field is integrating the advantages of EA and RL from different angles to devise new methods (Sigaud, 2022), for example, combining EA and RL for efficient policy optimization (Khadka & Tumer, 2018; Bodnar et al., 2020), using EA to approximate the greedy action selection in continuous action space (Kalashnikov et al., 2018; Simmons-Edler et al., 2019;
Revised: Recently, an emergent research field is integrating the advantages of EA and RL from different angles to devise new methods (Sigaud, 2022), for example, combining EA and RL for efficient policy optimization (Khadka & Tumer, 2018; Bodnar et al., 2020), using EA to approximate the greedy action selection in continuous action space (Kalashnikov et al., 2018; Simmons-Edler et al., 2019;

Original: Shi & Singh, 2021; Shao et al., 2021; Ma et al., 2022), population-based hyperparameter tuning of
Revised: Shi & Singh, 2021; Shao et al., 2021; Ma et al., 2022), population-based hyperparameter tuning of

Original: RL (Jaderberg et al., 2017; Pretorius & Pillay, 2021), and genetic programming for interpretable RL policies (Hein et al., 2018; Hein, 2019).
Revised: RL (Jaderberg et al., 2017; Pretorius & Pillay, 2021), and genetic programming for interpretable RL policies (Hein et al., 2018; Hein, 2019).

Original: In this work, we focus on combining EA and RL for efficient policy optimization.
Revised: In this work, we focus on combining EA and RL for efficient policy optimization.

Original: ERL (Khadka & Tumer, 2018) first proposes a hybrid framework where a DDPG agent is trained alongside a genetic population.
Revised: ERL (Khadka & Tumer, 2018) first proposes a hybrid framework where a DDPG agent is trained alongside a genetic population.

Original: The RL agent benefits from the diverse experiences collected by the EA population, while the population periodically includes a copy of the RL agent.
Revised: The RL agent benefits from the diverse experiences collected by the EA population, while the population periodically includes a copy of the RL agent.

Original: In parallel, CEM-RL (Pourchot & Sigaud, 2019) integrates CEM and TD3.
Revised: In parallel, CEM-RL (Pourchot & Sigaud, 2019) integrates CEM and TD3.

Original: In particular, the critic function of TD3 is used to provide update gradients for half of the individuals in the CEM population.
Revised: In particular, the critic function of TD3 is used to provide update gradients for half of the individuals in the CEM population.

Original: Later, ERL serves as a popular framework upon which many improvements are made.
Revised: Later, ERL serves as a popular framework upon which many improvements are made.

Original: CERL
Revised: CERL

Original: (Khadka et al., 2019) extends the single RL agent to multiple ones with different hyperparameter settings to make better use of the RL side.
Revised: (Khadka et al., 2019) extends the single RL agent to multiple ones with different hyperparameter settings to make better use of the RL side.

Original: GPO (Gangwani & Peng, 2018) devises gradient-based crossover and mutation by policy distillation and policy gradient algorithms, respectively.
Revised: GPO (Gangwani & Peng, 2018) devises gradient-based crossover and mutation by policy distillation and policy gradient algorithms, respectively.

Original: Further, PDERL (Bodnar et al., 2020) devises the Q -filtered distillation crossover and Proximal mutation to alleviate the policy crash caused by conventional genetic operators at the parameter level.
Revised: Further, PDERL (Bodnar et al., 2020) devises the Q -filtered distillation crossover and Proximal mutation to alleviate the policy crash caused by conventional genetic operators at the parameter level.

Original: All these works adopt no sharing among agents and each agent of EA and RL learns its own state representation, which is inefficient and specialized.
Revised: All these works adopt no sharing among agents and each agent of EA and RL learns its own state representation, which is inefficient and specialized.

Original: By contrast, our approach ERL-Re 2 makes use of a expressive state representation function which is shared and learned by all agents.
Revised: By contrast, our approach ERL-Re 2 makes use of a expressive state representation function which is shared and learned by all agents.

Original: Another common point of these works is, evolution variations are imposed at the parameter level (i.e., policy network).
Revised: Another common point of these works is, evolution variations are imposed at the parameter level (i.e., policy network).

Original: Despite the existence of GPO and PDERL, the semantics of genetic operators on policy behavior cannot be guaranteed due to the nonlinearity nature of policy parameters.
Revised: Despite the existence of GPO and PDERL, the semantics of genetic operators on policy behavior cannot be guaranteed due to the nonlinearity nature of policy parameters.

Original: In ERL-Re 2 , we propose behaviorlevel crossover and mutation operators that have clear semantics with linear policy representations.
Revised: In ERL-Re 2 , we propose behaviorlevel crossover and mutation operators that have clear semantics with linear policy representations.

Original:  R EPRESENTATION - BASED E VOLUTIONARY R EINFORCEMENT L EARNING
Revised:  R EPRESENTATION - BASED E VOLUTIONARY R EINFORCEMENT L EARNING

Original: In this section, we introduce the overview of ERL-Re 2 to gain the holistic understanding of the key concept.
Revised: In this section, we introduce the overview of ERL-Re 2 to gain the holistic understanding of the key concept.

Original: In addition, we introduce how ERL-Re 2 can be realized by a general form of algorithm framework.
Revised: In addition, we introduce how ERL-Re 2 can be realized by a general form of algorithm framework.

Original: We defer the concrete implementation details in Sec. 5.
Revised: We defer the concrete implementation details in Sec. 5.

Original: 4. T HE C ONCEPT OF T WO - SCALE S TATE R EPRESENTATION AND P OLICY R EPRESENTATION
Revised: 4. T HE C ONCEPT OF T WO - SCALE S TATE R EPRESENTATION AND P OLICY R EPRESENTATION

Original: EA side
Revised: EA side

Original: RL side
Revised: RL side

Original: Evolution-based optimization
Revised: Evolution-based optimization

Original: Gradient-based optimization
Revised: Gradient-based optimization

Original: Evaluation
Revised: Evaluation

Original: Experiences
Revised: Experiences

Original: Inject RL policy
Revised: Inject RL policy

Original: EA side
Revised: EA side

Original: RL side
Revised: RL side

Original: Gradient-based optimization
Revised: Gradient-based optimization

Original: Evaluation
Revised: Evaluation

Original: Experiences
Revised: Experiences

Original: Inject RL policy
Revised: Inject RL policy

Original: Convey common useful knowledge by the shared state representation
Revised: Convey common useful knowledge by the shared state representation

Original: ERL Framework
Revised: ERL Framework

Original: ERL- Re 𝟐 Framework 𝑾 𝟏 𝑾𝑾 𝒏 𝑾 𝒓𝒍 𝝅 𝒓𝒍 𝝅𝝅𝝅 𝒏 𝒁 𝝓
Revised: ERL- Re 𝟐 Framework 𝑾 𝟏 𝑾𝑾 𝒏 𝑾 𝒓𝒍 𝝅 𝒓𝒍 𝝅𝝅𝝅 𝒏 𝒁 𝝓

Original: Evolution-based optimization
Revised: Evolution-based optimization

Original: The previous algorithms that integrate EA and RL for policy optimization primarily follow the ERL interaction architecture shown on the left of Fig.
Revised: The previous algorithms that integrate EA and RL for policy optimization primarily follow the ERL interaction architecture shown on the left of Fig.

Original: 1, where the population’s policies offer a variety of experiences for
Revised: 1, where the population’s policies offer a variety of experiences for

Original: RL training and the RL side injects its policy into the population to participate in the iterative evolution.
Revised: RL training and the RL side injects its policy into the population to participate in the iterative evolution.

Original: However, two significant issues exist: 1) each agent maintains an independent nonlinear policy and searches in parameter space, which is inefficient because each agent has to independently and repeatedly learn common and useful knowledge.
Revised: However, two significant issues exist: 1) each agent maintains an independent nonlinear policy and searches in parameter space, which is inefficient because each agent has to independently and repeatedly learn common and useful knowledge.

Original: 2) Parameter-level perturbations might result in catastrophic failures, making parameter-level evolution exceedingly unstable.
Revised: 2) Parameter-level perturbations might result in catastrophic failures, making parameter-level evolution exceedingly unstable.

Original: To address the aforementioned problems, we propose Two-scale Representation-based policy construction, based on which we maintain and optimize the EA population and RL agent.
Revised: To address the aforementioned problems, we propose Two-scale Representation-based policy construction, based on which we maintain and optimize the EA population and RL agent.

Original: The policy construction is illustrated on the right of Fig.
Revised: The policy construction is illustrated on the right of Fig.

Original: Specifically, the policies for EA and RL agents are all composed of a shared nonlinear state representation z t =
Revised: Specifically, the policies for EA and RL agents are all composed of a shared nonlinear state representation z t =

Original: Z ϕ ( s t ) ∈ R d (given a state s t ) and an individual linear policy representation W ∈ R ( d +1) ×|A| .
Revised: Z ϕ ( s t ) ∈ R d (given a state s t ) and an individual linear policy representation W ∈ R ( d +1) ×|A| .

Original: We refer to the different representation scopes (shared/individual + state/policy representation) of the policy construction as the two scales.
Revised: We refer to the different representation scopes (shared/individual + state/policy representation) of the policy construction as the two scales.

Original: The agent i makes decisions by combining the shared state representation and the policy representation:
Revised: The agent i makes decisions by combining the shared state representation and the policy representation:

Original: [Equation]
Revised: [Equation]

Original: where W [ m (: n )] denotes the slice of matrix W that consists of row m (to n ) and act ( · ) denotes some activation function (e.g., tanh ) 1 .
Revised: where W [ m (: n )] denotes the slice of matrix W that consists of row m (to n ) and act ( · ) denotes some activation function (e.g., tanh ) 1 .

Original: In turn, we also denote the EA population by P = { W 1 , W 2 , ..., W n } and the RL agent by W rl .
Revised: In turn, we also denote the EA population by P = { W 1 , W 2 , ..., W n } and the RL agent by W rl .

Original: Intuitively, we expect the shared state representation Z ϕ to be useful to all possible policies encountered during the learning process.
Revised: Intuitively, we expect the shared state representation Z ϕ to be useful to all possible policies encountered during the learning process.

Original: It ought to contain the general decision-related features of the environment, e.g., common knowledge, while not specific to any single policy.
Revised: It ought to contain the general decision-related features of the environment, e.g., common knowledge, while not specific to any single policy.

Original: By sharing the state representation Z ϕ , it does not require each agent to learn how to represent the state independently.
Revised: By sharing the state representation Z ϕ , it does not require each agent to learn how to represent the state independently.

Original: Thus, higher efficiency and more expressive state representation can be fulfilled through learning in a collective manner with the EA population and
Revised: Thus, higher efficiency and more expressive state representation can be fulfilled through learning in a collective manner with the EA population and

Original: RL agent.
Revised: RL agent.

Original: Since Z ϕ is responsible for expressivity, each individual policy representation can have a straightforward linear form that is easy to optimize and evaluate (with PeVFA).
Revised: Since Z ϕ is responsible for expressivity, each individual policy representation can have a straightforward linear form that is easy to optimize and evaluate (with PeVFA).

Original: 4. T HE A LGORITHM F RAMEWORK OF ERL-R E
Revised: 4. T HE A LGORITHM F RAMEWORK OF ERL-R E

Original: Due to the representation-based policy construction, the state representation function Z ϕ determines a policy space denoted by Π( ϕ ) , where we optimize the individual representations of the EA and RL agents.
Revised: Due to the representation-based policy construction, the state representation function Z ϕ determines a policy space denoted by Π( ϕ ) , where we optimize the individual representations of the EA and RL agents.

Original: The optimization flow of ERL-Re 2 is shown in Fig.
Revised: The optimization flow of ERL-Re 2 is shown in Fig.

Original: The top and bottom panels depict the learning dynamics of Z ϕ and the agents, respectively.
Revised: The top and bottom panels depict the learning dynamics of Z ϕ and the agents, respectively.

Original: In each iteration t , the agents in the EA population P and the RL agent W rl evolve or reinforce their representations in the policy space Π( ϕ t ) provided by Z ϕ t (Sec. 5.2).
Revised: In each iteration t , the agents in the EA population P and the RL agent W rl evolve or reinforce their representations in the policy space Π( ϕ t ) provided by Z ϕ t (Sec. 5.2).

Original: After the optimization at the scale of individual representation, the shared state representation is optimized (i.e., ϕ t → ϕ t +1 ) towards a unifying direction, derived from value function maximization regarding all the EA and RL agents (Sec. 5.1).
Revised: After the optimization at the scale of individual representation, the shared state representation is optimized (i.e., ϕ t → ϕ t +1 ) towards a unifying direction, derived from value function maximization regarding all the EA and RL agents (Sec. 5.1).

Original: By this means, the shared state representation is optimized in the direction of a superior policy space for successive policy optimization.
Revised: By this means, the shared state representation is optimized in the direction of a superior policy space for successive policy optimization.

Original: In an iterative manner, the shared state representation and the individual policy representations play distinct roles and complement each other in optimizing the EA and RL agents.
Revised: In an iterative manner, the shared state representation and the individual policy representations play distinct roles and complement each other in optimizing the EA and RL agents.

Original: In principle, ERL-Re 2 is a general framework that can be implemented with different EA and RL algorithms.
Revised: In principle, ERL-Re 2 is a general framework that can be implemented with different EA and RL algorithms.

Original: For the side of EA, we mainly consider Genetic Algorithm (GA)
Revised: For the side of EA, we mainly consider Genetic Algorithm (GA)

Original: (Mitchell, 1998)
Revised: (Mitchell, 1998)

Original: [Figure]
Revised: [Figure]

Original: as a representative choice while another popular choice CEM (Pourchot & Sigaud, 2019) is also discussed in Appendix F. Our linear policy representation can realized the genetic operations with clear semantics.
Revised: as a representative choice while another popular choice CEM (Pourchot & Sigaud, 2019) is also discussed in Appendix F. Our linear policy representation can realized the genetic operations with clear semantics.

Original: For the side of RL, we use DDPG (Lillicrap et al., 2016) and TD3 (Fujimoto et al., 2018).
Revised: For the side of RL, we use DDPG (Lillicrap et al., 2016) and TD3 (Fujimoto et al., 2018).

Original: For efficient knowledge sharing and policy optimization, we learn a shared state representation by all the agents, then we evolve and reinforce the policies in the policy representation space rather than in the original policy parameter space.
Revised: For efficient knowledge sharing and policy optimization, we learn a shared state representation by all the agents, then we evolve and reinforce the policies in the policy representation space rather than in the original policy parameter space.

Original: A general pseudo-code of ERL-Re 2 is shown in Algorithm 1.
Revised: A general pseudo-code of ERL-Re 2 is shown in Algorithm 1.

Original: In each iteration, the algorithm proceeds across three phases (denoted by blue).
Revised: In each iteration, the algorithm proceeds across three phases (denoted by blue).

Original: First, each agent of EA and RL interacts with the environment and collects the experiences.
Revised: First, each agent of EA and RL interacts with the environment and collects the experiences.

Original: Specifically, the agents in the EA population P have the probability 1 − p to rollout partially and estimate the surrogate fitness with higher sample efficiency (Line 5-6), in contrast to the conventional way (Line 7-8).
Revised: Specifically, the agents in the EA population P have the probability 1 − p to rollout partially and estimate the surrogate fitness with higher sample efficiency (Line 5-6), in contrast to the conventional way (Line 7-8).

Original: Next, evolution and reinforcement update /optimize
Revised: Next, evolution and reinforcement update /optimize

Original: their policy representations in the linear policy space (Line 13-14).
Revised: their policy representations in the linear policy space (Line 13-14).

Original: The agents are optimized by EA and RL, where RL agent learns with additional off-policy experiences collected by the agents in P (Line 14) and periodically injects its policy to P (Line 15).
Revised: The agents are optimized by EA and RL, where RL agent learns with additional off-policy experiences collected by the agents in P (Line 14) and periodically injects its policy to P (Line 15).

Original: Finally, the shared state representation is updated to provide superior policy space for the following iteration (Line 17).
Revised: Finally, the shared state representation is updated to provide superior policy space for the following iteration (Line 17).

Original:  E VOLUTION AND R EINFORCEMENT IN R EPRESENTATION S PACE
Revised:  E VOLUTION AND R EINFORCEMENT IN R EPRESENTATION S PACE

Original: In this section, we present the algorithm details of how to optimize the two-scale representation.
Revised: In this section, we present the algorithm details of how to optimize the two-scale representation.

Original: In addition, we introduce a novel surrogate of policy fitness to further improve the sample efficiency.
Revised: In addition, we introduce a new surrogate of policy fitness to further improve the sample efficiency.

Original: 5. O PTIMIZING THE S HARED S TATE R EPRESENTATION FOR A S UPERIOR P OLICY S PACE
Revised: 5. O PTIMIZING THE S HARED S TATE R EPRESENTATION FOR A S UPERIOR P OLICY S PACE

Original: The shared state representation Z ϕ designates the policy space Π ϕ , in which EA and RL are conducted.
Revised: The shared state representation Z ϕ designates the policy space Π ϕ , in which EA and RL are conducted.

Original: As discussed in the previous section, the shared state representation takes the responsibility of learning useful features of the environment from the overall policy learning.
Revised: As discussed in the previous section, the shared state representation takes the responsibility of learning useful features of the environment from the overall policy learning.

Original: We argue that this is superior to the prior way, i.e., each agent learns its individual state representation which can be less efficient and limited in expressivity.
Revised: We argue that this is superior to the prior way, i.e., each agent learns its individual state representation which can be less efficient and limited in expressivity.

Original: In this paper, we propose to learn the shared state representation by the principle of value function maximization regarding all EA and RL agents .
Revised: In this paper, we propose to learn the shared state representation by the principle of value function maximization regarding all EA and RL agents .

Original: To be specific, we first detail the value function approximation for both EA and RL agents.
Revised: To be specific, we first detail the value function approximation for both EA and RL agents.

Original: For the EA agents, we learn a
Revised: For the EA agents, we learn a

Original: PeVFA Q θ ( s, a, W i ) based on the linear policy representation W i in EA population P ; for the RL agent, we learn a critic Q ψ ( s, a ) .
Revised: PeVFA Q θ ( s, a, W i ) based on the linear policy representation W i in EA population P ; for the RL agent, we learn a critic Q ψ ( s, a ) .

Original: In principle, RL can use PeVFA as its critic.
Revised: In principle, RL can use PeVFA as its critic.

Original: We experimentally show that both approaches can achieve similar performance in Appendix.
Revised: We experimentally show that both approaches can achieve similar performance in Appendix.

Original: D.4.
Revised: D.4.

Original: The loss functions of Q θ and Q ψ are formulated below:
Revised: The loss functions of Q θ and Q ψ are formulated below:

Original: [Equation]
Revised: [Equation]

Original: (1) where D is the experience buffer collected by both the EA and RL agents, θ ′ , ψ ′ denote the target networks of the PeVFA and the RL critic, π ′ rl denote the target actor with policy representation W ′ rl , and recall π i ( s )
Revised: (1) where D is the experience buffer collected by both the EA and RL agents, θ ′ , ψ ′ denote the target networks of the PeVFA and the RL critic, π ′ rl denote the target actor with policy representation W ′ rl , and recall π i ( s )

Original: = act ( Z ϕ ( s ) T W i, [1: d ] + W i, [ d +1] ) ∈ R |A| .
Revised: = act ( Z ϕ ( s ) T W i, [1: d ] + W i, [ d +1] ) ∈ R |A| .

Original: Note that we make Q θ and Q ψ take the raw state s as input rather than Z ϕ ( s ) .
Revised: Note that we make Q θ and Q ψ take the raw state s as input rather than Z ϕ ( s ) .

Original: Since sharing state representation between actor and critic may induce interference and degenerated performance as designated by recent studies (Cobbe et al., 2021; Raileanu & Fergus, 2021).
Revised: Since sharing state representation between actor and critic may induce interference and degenerated performance as designated by recent studies (Cobbe et al., 2021; Raileanu & Fergus, 2021).

Original: Another thing to notice is, to our knowledge, we are the first to train PeVFA in an off-policy fashion (Eq. 1).
Revised: Another thing to notice is, to our knowledge, we are the first to train PeVFA in an off-policy fashion (Eq. 1).

Original: We discuss more on this in Appendix F.
Revised: We discuss more on this in Appendix F.

Original: For each agent of EA and RL, an individual update direction of the shared state representation Z ϕ is now ready to obtain by ∇ ϕ Q θ ( s, π i ( s ) , W i ) for any W i ∈ P or ∇ ϕ Q ψ ( s, π rl ( s )) through π i and π rl respectively.
Revised: For each agent of EA and RL, an individual update direction of the shared state representation Z ϕ is now ready to obtain by ∇ ϕ Q θ ( s, π i ( s ) , W i ) for any W i ∈ P or ∇ ϕ Q ψ ( s, π rl ( s )) through π i and π rl respectively.

Original: This is the value function maximization principle where we adjust Z ϕ to induce superior policy (space) for the corresponding agent.
Revised: This is the value function maximization principle where we adjust Z ϕ to induce superior policy (space) for the corresponding agent.

Original: To be expressive , Z ϕ should not take either individual update direction solely; instead, the natural way is to take a unifying update direction regarding all the agents (i.e., the currently targeted policies).
Revised: To be expressive , Z ϕ should not take either individual update direction solely; instead, the natural way is to take a unifying update direction regarding all the agents (i.e., the currently targeted policies).

Original: Finally, the loss function of Z ϕ is defined:
Revised: Finally, the loss function of Z ϕ is defined:

Original: [Equation]
Revised: [Equation]

Original: where K is the size of the sampled subset of EA agents that engage in updating Z ϕ .
Revised: where K is the size of the sampled subset of EA agents that engage in updating Z ϕ .

Original: By minimizing
Revised: By minimizing

Original: Eq. 2, the shared state representation Z ϕ is optimized towards a superior policy space Π ϕ pertaining to all the EA and RL agents iteratively, as the learning dynamics depicted in Fig.
Revised: Eq. 2, the shared state representation Z ϕ is optimized towards a superior policy space Π ϕ pertaining to all the EA and RL agents iteratively, as the learning dynamics depicted in Fig.

Original: Note that the value maximization is not the only possible choice.
Revised: Note that the value maximization is not the only possible choice.

Original: For a step further, we also investigate on the incorporation of self-supervised state representation learning in Appendix D.
Revised: For a step further, we also investigate on the incorporation of self-supervised state representation learning in Appendix D.

Original: 5. O PTIMIZING THE P OLICY R EPRESENTATION BY E VOLUTION AND R EINFORCEMENT
Revised: 5. O PTIMIZING THE P OLICY R EPRESENTATION BY E VOLUTION AND R EINFORCEMENT

Original: Given the shared state representation Z ϕ , all the agents of EA and RL optimize their individual policy representation in the policy space Π( ϕ ) .
Revised: Given the shared state representation Z ϕ , all the agents of EA and RL optimize their individual policy representation in the policy space Π( ϕ ) .

Original: The fundamental distinction here is, that the evolution and reinforcement occur in the linear policy representation space rather than in a nonlinear policy network parameter space as the convention.
Revised: The fundamental distinction here is, that the evolution and reinforcement occur in the linear policy representation space rather than in a nonlinear policy network parameter space as the convention.

Original: Thus, policy optimization can be efficiently conducted.
Revised: Thus, policy optimization can be efficiently conducted.

Original: Moreover, the linear policy representation has a special characteristic: it allows performing genetic operations at the behavior level.
Revised: Moreover, the linear policy representation has a special characteristic: it allows performing genetic operations at the behavior level.

Original: We detail the policy representation optimization of EA and RL below.
Revised: We detail the policy representation optimization of EA and RL below.

Original: The evolution of the EA population P mainly consists of: 1) interaction and selection, 2) genetic evolution.
Revised: The evolution of the EA population P mainly consists of: 1) interaction and selection, 2) genetic evolution.

Original: For the process of interaction and selection, most prior methods rollout each agent in P for one or several episodes and calculate the MC fitness.
Revised: For the process of interaction and selection, most prior methods rollout each agent in P for one or several episodes and calculate the MC fitness.

Original: The incurred sample cost is regarded as one major bottleneck of EA especially when the population is large.
Revised: The incurred sample cost is regarded as one major bottleneck of EA especially when the population is large.

Original: To this end, we propose a novel surrogate fitnesswhich is estimated based on the PeVFA Q θ (Eq. 1) and partial rollout.
Revised: To this end, we propose a new surrogate fitness based on the PeVFA Q θ (Eq. 1) and partial rollout.

Original: At the beginning of each iteration, we have a probability p to rollout the EA population for H steps.
Revised: At the beginning of each iteration, we have a probability p to rollout the EA population for H steps.

Original: For each agent W i , the surrogate fitness is estimated by H -step TD:
Revised: For each agent W i , the surrogate fitness is estimated by H -step bootstrapping (Sutton & Barto, 1998):

Original: [Equation]
Revised: [Equation]

Original: Thanks to the PeVFA Q θ , the surrogate fitness can be conveniently estimated and reduce the sample cost effectively.
Revised: Thanks to the PeVFA Q θ , the surrogate fitness can be conveniently estimated and reduce the sample cost effectively.

Original: Moreover, ˆ f ( W i ) is free of the off-policy bias in the surrogate proposed in the concurrent work (Wang et al., 2022), we experimentally prove that ˆ f ( W i ) is more efficient.
Revised: Moreover, ˆ f ( W i ) is free of the off-policy bias in the surrogate proposed in the concurrent work (Wang et al., 2022), we experimentally prove that ˆ f ( W i ) is more efficient.

Original: According to the fitness estimated by MC or our surrogate, parents with high fitness are more likely to be selected as parents.
Revised: According to the fitness estimated by MC or our surrogate, parents with high fitness are more likely to be selected as parents.

Original: The details of the selection process follow PDERL (Bodnar et al., 2020).
Revised: The details of the selection process follow PDERL (Bodnar et al., 2020).

Original: For the process of genetic evolution, two genetic operators are involved: crossover and mutation.
Revised: For the process of genetic evolution, two genetic operators are involved: crossover and mutation.

Original: Typical k -point crossover and Gaussian mutation that directly operate at the parameter level can easily lead to large fluctuations in the decision and even cause a crash to policy (Gangwani & Peng, 2018; Bodnar et al., 2020).
Revised: Typical k -point crossover and Gaussian mutation that directly operate at the parameter level can easily lead to large fluctuations in the decision and even cause a crash to policy (Gangwani & Peng, 2018; Bodnar et al., 2020).

Original: The reason behind this is that the policy parameter is highly nonlinear to behavior, thus the natural semantics could be hardly guaranteed by such parameter-level operators.
Revised: The reason behind this is that the policy parameter is highly nonlinear to behavior, thus the natural semantics could be hardly guaranteed by such parameter-level operators.

Original: In contrast, recall the linear policy representation W in a matrix form.
Revised: In contrast, recall the linear policy representation W in a matrix form.

Original: Each row W [ i ] determines the i -th dimension of action and thus the change to W [ i ] does not affect the behavior of other dimensions.
Revised: Each row W [ i ] determines the i -th dimension of action and thus the change to W [ i ] does not affect the behavior of other dimensions.

Original: Based on the granularity of clearer semantics, we propose novel behavior-level crossover and mutation, as the illustration in Fig.3.
Revised: Based on the granularity of clearer semantics, we propose novel behavior-level crossover and mutation, as the illustration in Fig.3.

Original: For the behavior-level crossover ( b -Crossover ), the offspring are produced by inheriting the behaviors of the parents at the corresponding randomly selected dimensions.
Revised: For the behavior-level crossover ( b -Crossover ), the offspring are produced by inheriting the behaviors of the parents at the corresponding randomly selected dimensions.

Original: For the behavior-level mutation ( b -Mutation ), each dimension of behavior has the probability α to be perturbed.
Revised: For the behavior-level mutation ( b -Mutation ), each dimension of behavior has the probability α to be perturbed.

Original: Formally, we formulate the two operations below:
Revised: Formally, we formulate the two operations below:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: The offspring and parents are indexed by c 1 , c 2 , m 1 , m 2 and p 1 , p 2 respectively.
Revised: The offspring and parents are indexed by c 1 , c 2 , m 1 , m 2 and p 1 , p 2 respectively.

Original: where d 1 , d 2 , ˆ d 1 are the randomly sampled subsets of all dimension indices, P 1 is the randomly generated perturbation matrix, and we use A ⊗ d B to denote A being replaced by B at the corresponding rows in d .
Revised: where d 1 , d 2 , ˆ d 1 are the randomly sampled subsets of all dimension indices, P 1 is the randomly generated perturbation matrix, and we use A ⊗ d B to denote A being replaced by B at the corresponding rows in d .

Original: For the dimension selection and perturbation generation, we follow the same approach used in (Bodnar et al., 2020)
Revised: For the dimension selection and perturbation generation, we follow the same approach used in (Bodnar et al., 2020)

Original: (detailed in Appendix C) and we analyze the hyperparameter choices specific to our approach in Sec.
Revised: (detailed in Appendix C) and we analyze the hyperparameter choices specific to our approach in Sec.

Original: By imposing variations on specific behavior dimensions while incurring no interference on the others, our proposed operators (i.e., b -Crossover and b -Mutation ) can be more effective and stable, as well as more informative in the sense of genetic semantics.
Revised: By imposing variations on specific behavior dimensions while incurring no interference on the others, our proposed operators (i.e., b -Crossover and b -Mutation ) can be more effective and stable, as well as more informative in the sense of genetic semantics.

Original: As to the learning of the RL agent W rl , it resembles the conventional circumstance except done with respect to the linear policy representation.
Revised: As to the learning of the RL agent W rl , it resembles the conventional circumstance except done with respect to the linear policy representation.

Original: Taking DDPG (Lillicrap et al., 2016) for a typical example, the loss function of W rl is defined below, based on the RL critic Q ψ (learned by Eq. 1):
Revised: Taking DDPG (Lillicrap et al., 2016) for a typical example, the loss function of W rl is defined below, based on the RL critic Q ψ (learned by Eq. 1):

Original: [Equation]
Revised: [Equation]

Original: The RL agent learns from the off-policy experience in the buffer D collected also by the EA population.
Revised: The RL agent learns from the off-policy experience in the buffer D collected also by the EA population.

Original: Meanwhile, the EA population incorporates the RL policy representation W rl at the end of each iteration.
Revised: Meanwhile, the EA population incorporates the RL policy representation W rl at the end of each iteration.

Original: By such an interaction, EA and RL complement each other consistently and the respective advantages of gradient-based and gradient-free policy optimization are merged effectively.
Revised: By such an interaction, EA and RL complement each other consistently and the respective advantages of gradient-based and gradient-free policy optimization are merged effectively.

Original:  E XPERIMENTS 6. E XPERIMENTAL S ETUPS
Revised:  E XPERIMENTS 6. E XPERIMENTAL S ETUPS

Original: We evaluate ERL-Re 2 on six MuJoCo (Todorov et al., 2012) continuous control tasks as commonly used in the literature: HalfCheetach , Swimmer , Hopper , Ant , Walker , Humanoid (all in version 2).
Revised: We evaluate ERL-Re 2 on six MuJoCo (Todorov et al., 2012) continuous control tasks as commonly used in the literature: HalfCheetach , Swimmer , Hopper , Ant , Walker , Humanoid (all in version 2).

Original: For a comprehensive evaluation, we implement two instances of ERL-Re 2 based on DDPG (Lillicrap et al., 2016) and TD3 (Fujimoto et al., 2018), respectively.
Revised: For a comprehensive evaluation, we implement two instances of ERL-Re 2 based on DDPG (Lillicrap et al., 2016) and TD3 (Fujimoto et al., 2018), respectively.

Original: We compare ERL-Re 2 with the following baselines: 1) basic baselines , i.e., PPO (Schulman et al., 2017), SAC (Haarnoja et al., 2018), DDPG,
Revised: We compare ERL-Re 2 with the following baselines: 1) basic baselines , i.e., PPO (Schulman et al., 2017), SAC (Haarnoja et al., 2018), DDPG,

Original: TD3 and EA (Mitchell, 1998); 2) ERL-related baselines , including ERL (Khadka & Tumer, 2018),
Revised: TD3 and EA (Mitchell, 1998); 2) ERL-related baselines , including ERL (Khadka & Tumer, 2018),

Original: CERL (Khadka et al., 2019), PDERL (Bodnar et al., 2020), CEM-RL (Pourchot & Sigaud, 2019).
Revised: CERL (Khadka et al., 2019), PDERL (Bodnar et al., 2020), CEM-RL (Pourchot & Sigaud, 2019).

Original: The ERL-related baselines are originally built on different RL algorithms (either DDPG or TD3), thus we modify them to obtain both the two versions for each of them.
Revised: The ERL-related baselines are originally built on different RL algorithms (either DDPG or TD3), thus we modify them to obtain both the two versions for each of them.

Original: We use the official implementation and stable-baseline3 for the mentioned baselines in our experiments.
Revised: We use the official implementation and stable-baseline3 for the mentioned baselines in our experiments.

Original: For a fair comparison, we
Revised: For a fair comparison, we

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: compare the methods built on the same RL algorithm (i.e., DDPG and TD3) and fine-tune them in each task to provide the best performance.
Revised: compare the methods built on the same RL algorithm (i.e., DDPG and TD3) and fine-tune them in each task to provide the best performance.

Original: All statistics are obtained from 5 independent runs
Revised: All statistics are obtained from 5 independent runs

Original: [Figure]
Revised: [Figure]

Original: This is consistent with the setting in ERL and PDERL.
Revised: This is consistent with the setting in ERL and PDERL.

Original: We report the average with 95% confidence regions.
Revised: We report the average with 95% confidence regions.

Original: For the population size n , we consider the common choices and use the best one in [5 , 10] for each concerned method.
Revised: For the population size n , we consider the common choices and use the best one in { 5 , 10 } for each concerned method.

Original: We implement our method ERL-Re 2 based on the codebase of PDERL and the common hyperparameters remain the same.
Revised: We implement our method ERL-Re 2 based on the codebase of PDERL and the common hyperparameters remain the same.

Original: For the hyperparameters specific to ERL-Re(both DDPG and TD3 version), we set α to 1.0 and select H from [50 , 200] , K from [1 , 3] , p from [0 . 3 , 0 . 5 , 0 . 7 , 0 . 8] for different tasks.
Revised: For the hyperparameters specific to ERL-Re(both DDPG and TD3 version), we set α to 1.0 and select H from { 50 , 200 } , K from { 1 , 3 } , p from { 0 . 3 , 0 . 5 , 0 . 7 , 0 . 8 } for different tasks.

Original: All implementation details are provided in Appendix E.
Revised: All implementation details are provided in Appendix E.

Original: 6. P ERFORMANCE E VALUATION
Revised: 6. P ERFORMANCE E VALUATION

Original: We evaluate ERL-Re 2 and baselines in TD3 and DDPG versions separately.
Revised: We evaluate ERL-Re 2 and baselines in TD3 and DDPG versions separately.

Original: The results in Fig.
Revised: The results in Fig.

Original: and Fig.5 show that both ERL-Re 2 (TD3) and ERL-Re 2 (DDPG) significantly outperform other methods in most tasks.
Revised: and Fig.5 show that both ERL-Re 2 (TD3) and ERL-Re 2 (DDPG) significantly outperform other methods in most tasks.

Original: It is worth noting that, to our knowledge, ERL-Re 2 is the first algorithm that outperforms EA in Swimmer .
Revised: It is worth noting that, to our knowledge, ERL-Re 2 is the first algorithm that outperforms EA in Swimmer .

Original: We can see that both ERL-Re 2 (TD3) and ERL-Re 2 (DDPG) achieve a 5x improvement in convergence rate compared to EA and obtain higher and more stable performance.
Revised: We can see that both ERL-Re 2 (TD3) and ERL-Re 2 (DDPG) achieve a 5x improvement in convergence rate compared to EA and obtain higher and more stable performance.

Original: Moreover, in some more difficult environments such Humanoid which is dominated by
Revised: Moreover, in some more difficult environments such Humanoid which is dominated by

Original: RL algorithms, ERL-Re 2 also achieves significant improvements.
Revised: RL algorithms, ERL-Re 2 also achieves significant improvements.

Original: Overall, ERL-Re 2 is an effective and general framework that significantly improves both EA and RL in all the six tasks while other methods fails.
Revised: Overall, ERL-Re 2 is an effective and general framework that significantly improves both EA and RL in all the six tasks while other methods fails.

Original: Beyond MuJoCo, we also demonstrate the performance improvement achieved by
Revised: Beyond MuJoCo, we also demonstrate the performance improvement achieved by

Original: ERL-Re 2 in several visual control tasks of DMC (Tassa et al., 2018) in Appendix D.5.
Revised: ERL-Re 2 in several visual control tasks of DMC (Tassa et al., 2018) in Appendix D.5.

Original: 6. S UPERIORITY OF C OMPONENTS & P ARAMETER A NALYSIS
Revised: 6. S UPERIORITY OF C OMPONENTS & P ARAMETER A NALYSIS

Original: We conduct experiments to compare our proposed genetic operators and surrogate fitness with other related methods.
Revised: We conduct experiments to compare our proposed genetic operators and surrogate fitness with other related methods.

Original: In Fig.6a, we compare the behavior-level operators with other alternatives.
Revised: In Fig.6a, we compare the behavior-level operators with other alternatives.

Revised: The results show that the behavior-level operators are more effective than other operators.

Original: The
Revised: Next, we

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: results show that the behavior-level operators are more effective than other operators.

Original: Next, we examine whether the surrogate fitness estimate ˆ f ( W i ) (i.e., H -step TD (PeVFA)) is better than using the RL critic to evaluate fitness based on the latest state from replay buffer (Wang et al., 2022) (i.e., buffer (Critic)).
Revised: examine whether the surrogate fitness estimate ˆ f ( W i ) (i.e., H -step bootstrapping (PeVFA)) is better than using the RL critic to evaluate fitness based on the latest state from replay buffer (Wang et al., 2022) (i.e., buffer (Critic)).

Original: The results in Fig.6b show that using PeVFA is more effective than using the RL critic.
Revised: The results in Fig.6b show that using PeVFA is more effective than using the RL critic.

Original: This is because the RL critic cannot elude the off-policy bias while PeVFA is free of it thanks to the generalized approximation.
Revised: This is because the RL critic cannot elude the off-policy bias while PeVFA is free of it thanks to the generalized approximation.

Original: Second, we provide ablation study on how to update Z ϕ and analyze the effect of different K .
Revised: Second, we provide ablation study on how to update Z ϕ and analyze the effect of different K .

Original: We evaluate ERL-Re 2 (TD3) with different K ∈ [1 , 3] and
Revised: We evaluate ERL-Re 2 (TD3) with different K ∈ [1 , 3] and

Original: ERL-Re 2 (TD3) with only PeVFA or critic to optimize the shared state representation.
Revised: ERL-Re 2 (TD3) with only PeVFA or critic to optimize the shared state representation.

Original: The results in
Revised: The results in

Original: Fig.6c demonstrate the conclusion of two aspects: 1) K affects the results and appropriate tuning are beneficial; 2) both the RL critic and PeVFA play important roles.
Revised: Fig.6c demonstrate the conclusion of two aspects: 1) K affects the results and appropriate tuning are beneficial; 2) both the RL critic and PeVFA play important roles.

Original: Only using the RL critic or
Revised: Only using the RL critic or

Original: PeVFA leads to inferior shared state representation, crippling the overall performance.
Revised: PeVFA leads to inferior shared state representation, crippling the overall performance.

Original: Third, we perform ablation experiments on ˆ f ( W i ) .
Revised: Third, we perform ablation experiments on ˆ f ( W i ) .

Original: The results in Fig.6d show that using ˆ f ( W i ) can deliver performance gains and achieve further improvement to ERL-Re 2 especially in Swimmer .
Revised: The results in Fig.6d show that using ˆ f ( W i ) can deliver performance gains and achieve further improvement to ERL-Re 2 especially in Swimmer .

Original: For hyperparameter analysis, the results in Fig.7a show that larger α is more effective and stable, which is mainly since that larger α can lead to stronger exploration.
Revised: For hyperparameter analysis, the results in Fig.7a show that larger α is more effective and stable, which is mainly since that larger α can lead to stronger exploration.

Original: Thus we use α = 1 . 0 on all tasks.
Revised: Thus we use α = 1 . 0 on all tasks.

Original: Finally, the results in Fig.
Revised: Finally, the results in Fig.

Original: 7b demonstrate that p has an impact on performance and appropriate tuning for different tasks is necessary.
Revised: 7b demonstrate that p has an impact on performance and appropriate tuning for different tasks is necessary.

Original: It is worth noting that setting p to 1.0 also gives competitive performances, i.e., do not use the surrogate fitness ˆ f ( W i )
Revised: It is worth noting that setting p to 1.0 also gives competitive performances, i.e., do not use the surrogate fitness ˆ f ( W i )

Original: (see blue in Fig.
Revised: (see blue in Fig.

Original: 6d).
Revised: 6d).

Original: Others Due to space limitations, more experiments on the hyperparameter analysis of H , β , different combinations of genetic operators, incorporating self-supervised state representation learning, the interplay between the EA population and the RL agent and etc. are placed in Appendix D.
Revised: Others Due to space limitations, more experiments on the hyperparameter analysis of H , β , different combinations of genetic operators, incorporating self-supervised state representation learning, the interplay between the EA population and the RL agent and etc. are placed in Appendix D.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose a novel approach called ERL-Re 2 to fuse the distinct advantages of EA and RL for efficient policy optimization.
Revised: We propose a novel approach called ERL-Re 2 to fuse the distinct advantages of EA and RL for efficient policy optimization.

Original: In ERL-Re 2 , all agents are composed of the shared state representation and an individual policy representation.
Revised: In ERL-Re 2 , all agents are composed of the shared state representation and an individual policy representation.

Original: The shared state representation is endowed with expressive and useful features of the environment by value function maximization regarding all the agents.
Revised: The shared state representation is endowed with expressive and useful features of the environment by value function maximization regarding all the agents.

Original: For the optimization of individual policy representation, we propose behavior-level genetic operators and a new surrogate of policy fitness to improve effectiveness and efficiency.
Revised: For the optimization of individual policy representation, we propose behavior-level genetic operators and a new surrogate of policy fitness to improve effectiveness and efficiency.

Original: In our experiments, we demonstrate the significant superiority of ERL-Re 2 compared with various baselines on MuJoCo.
Revised: In our experiments, we demonstrate the significant superiority of ERL-Re 2 compared with various baselines on MuJoCo.

Original: A L IMITATIONS & F UTURE W ORK
Revised: A L IMITATIONS & F UTURE W ORK

Original: For limitations and future work, firstly our work is empirical proof of the effectiveness of the ERLRe 2 idea
Revised: For limitations and future work, firstly our work is empirical proof of the effectiveness of the ERLRe 2 idea

Original: and we provide no theory on optimality, convergence, and complexity.
Revised: and we provide no theory on optimality, convergence, and complexity.

Original: Secondly, one critical point is the expressivity of the shared state representation, which is not optimally addressed by this work.
Revised: Secondly, one critical point is the expressivity of the shared state representation, which is not optimally addressed by this work.

Original: Self-supervised state representation learning and nonlinear policy representation with behavior semantics can be potential directions.
Revised: Self-supervised state representation learning and nonlinear policy representation with behavior semantics can be potential directions.

Original: Thirdly, diversity is one key aspect of EA while in ERL-Re 2 we currently do not consider an explicit mechanism or objective to optimize the diversity of EA population (and also RL policy).
Revised: Thirdly, diversity is one key aspect of EA while in ERL-Re 2 we currently do not consider an explicit mechanism or objective to optimize the diversity of EA population (and also RL policy).

Original: One important future work is to integrate
Revised: One important future work is to integrate

Original: Quality-Diversity principle (Parker-Holder et al., 2020; Fontaine & Nikolaidis, 2021;
Revised: Quality-Diversity principle (Parker-Holder et al., 2020; Fontaine & Nikolaidis, 2021;

Original: Wang et al., 2021) for further development of ERL-Re 2 .
Revised: Wang et al., 2021) for further development of ERL-Re 2 .

Original: Fourthly, most current ERL methods including our
Revised: Fourthly, most current ERL methods including our

Original: ERL-Re 2 are model-free.
Revised: ERL-Re 2 are model-free.

Original: Obviously, the model-free paradigm limits the advance in sample efficiency.
Revised: Obviously, the model-free paradigm limits the advance in sample efficiency.

Original: One promising direction can be ERL with a world model, as a pioneer work (Ha & Schmidhuber, 2018) has demonstrated the feasibility with CMA-ES.
Revised: One promising direction can be ERL with a world model, as a pioneer work (Ha & Schmidhuber, 2018) has demonstrated the feasibility with CMA-ES.

Original: For the final one, we follow the convention in the literature and evaluate ERL-Re 2 on MuJoCo in the main text and additional tasks DMC (Tassa et al., 2018) with image inputs in Appendix D.5; while performance in other environments, e.g.,
Revised: For the final one, we follow the convention in the literature and evaluate ERL-Re 2 on MuJoCo in the main text and additional tasks DMC (Tassa et al., 2018) with image inputs in Appendix D.5; while performance in other environments, e.g.,

Original: Atari (Mnih et al., 2013), is not thoroughly studied.
Revised: Atari (Mnih et al., 2013), is not thoroughly studied.

Original: B C OMPARISON BETWEEN ERL AND ERL-R E
Revised: B C OMPARISON BETWEEN ERL AND ERL-R E

Original: In this section, we compare ERL (Khadka & Tumer, 2018) and ERL-Re 2 in detail.
Revised: In this section, we compare ERL (Khadka & Tumer, 2018) and ERL-Re 2 in detail.

Original: The ERL framework is currently one popular framework to integrate EA and RL for policy optimization.
Revised: The ERL framework is currently one popular framework to integrate EA and RL for policy optimization.

Original: We illustrate the difference between the two frameworks intuitively in Fig.8.
Revised: We illustrate the difference between the two frameworks intuitively in Fig.8.

Original: On the foundation of the interaction schema between EA and RL in ERL, ERL-Re 2 makes all policies composed of the shared state representation and individual policy representations.
Revised: On the foundation of the interaction schema between EA and RL in ERL, ERL-Re 2 makes all policies composed of the shared state representation and individual policy representations.

Original: Based on the shared state representation, useful common knowledge can be delivered efficiently among agents.
Revised: Based on the shared state representation, useful common knowledge can be delivered efficiently among agents.

Original: Meanwhile, the linear policy representations allow for imposing the novel behavior-level crossover and mutation operations, which are more stable and effective.
Revised: Meanwhile, the linear policy representations allow for imposing the novel behavior-level crossover and mutation operations, which are more stable and effective.

Original: We detail how to update the shared state representation by the principle of value function maximization .
Revised: We detail how to update the shared state representation by the principle of value function maximization .

Original: Here we illustrate the process in Fig.
Revised: Here we illustrate the process in Fig.

Original: 9 to help the readers understand it more intuitively.
Revised: 9 to help the readers understand it more intuitively.

Original: After interacting with the environment for t steps, we optimize the shared state representation t times.
Revised: After interacting with the environment for t steps, we optimize the shared state representation t times.

Original: In each update, we maximize Q ( s, a ) of the RL agent on the one hand.
Revised: In each update, we maximize Q ( s, a ) of the RL agent on the one hand.

Original: On the other hand, we sample
Revised: On the other hand, we sample

Original: K policies { W 1 , ...
Revised: K policies { W 1 , ...

Original: W K } from the population to maximize Q ( s, a, W i ) .
Revised: W K } from the population to maximize Q ( s, a, W i ) .

Original: Finally, the shared state representation is optimized toward a unifying direction according to Eq.
Revised: Finally, the shared state representation is optimized toward a unifying direction according to Eq.

Original: [Figure]
Revised: [Figure]

Original: C D ETAILS OF G ENETIC A LGORITHM IN ERL-R E
Revised: C D ETAILS OF G ENETIC A LGORITHM IN ERL-R E

Original: In this section, we provide the genetic algorithm flow and the corresponding pseudo-code in Algorithm 2.
Revised: In this section, we provide the genetic algorithm flow and the corresponding pseudo-code in Algorithm 2.

Original: Since ERL-Re 2 is built on the official code of PDERL (Bodnar et al., 2020), thus all hyperparameters and genetic algorithm flow remain the same.
Revised: Since ERL-Re 2 is built on the official code of PDERL (Bodnar et al., 2020), thus all hyperparameters and genetic algorithm flow remain the same.

Original: [Figure]
Revised: [Figure]

Original: 𝒘
Revised: 𝒘

Original: [Figure]
Revised: [Figure]

Original: PeVFA
Revised: PeVFA

Original: Critic 𝒂 𝒕 𝒔 𝒕 𝒂 𝒕 𝒔
Revised: Critic 𝒂 𝒕 𝒔 𝒕 𝒂 𝒕 𝒔

Original: [Figure]
Revised: [Figure]

Original: RL
Revised: RL

Original: EA
Revised: EA

Original: [Figure]
Revised: [Figure]

Original: 𝒘 𝒊 𝑄(𝑠 𝑡 ,𝑎 𝑡 ) ℚ(s t ,a t ,w i )
Revised: 𝒘 𝒊 𝑄(𝑠 𝑡 ,𝑎 𝑡 ) ℚ(s t ,a t ,w i )

Original: [Figure]
Revised: [Figure]

Original: Specifically, we first get the fitness of all policies in the population and divide all policies into three groups: elites, winners, and discarders.
Revised: Specifically, we first get the fitness of all policies in the population and divide all policies into three groups: elites, winners, and discarders.

Original: We choose the best e policies as the elites E
Revised: We choose the best e policies as the elites E

Original: (In PDERL, e is different across tasks.
Revised: (In PDERL, e is different across tasks.

Original: We choose 1 for all tasks in ERL-Re 2 ).
Revised: We choose 1 for all tasks in ERL-Re 2 ).

Original: Then we randomly select 3 policies from the non-elites policies and choose the best one as the winner.
Revised: Then we randomly select 3 policies from the non-elites policies and choose the best one as the winner.

Original: Then we repeat the process for n − e times and get the set of these policies Win without duplicates.
Revised: Then we repeat the process for n − e times and get the set of these policies Win without duplicates.

Original: The policies that are not selected as the elites and winners are denoted as discarders Dis .
Revised: The policies that are not selected as the elites and winners are denoted as discarders Dis .

Original: After sorting the categories, we perform the crossover operator.
Revised: After sorting the categories, we perform the crossover operator.

Original: Specifically, we obtain the parents W p 1 and W p 2 by cloning the randomly sampled policies from E and Win respectively.
Revised: Specifically, we obtain the parents W p 1 and W p 2 by cloning the randomly sampled policies from E and Win respectively.

Original: For each action dimension a i , the weights and biases corresponding to the a i -th action of W p 1 will be replaced by those of W p 2 with probability 50% .
Revised: For each action dimension a i , the weights and biases corresponding to the a i -th action of W p 1 will be replaced by those of W p 2 with probability 50% .

Original: Conversely, the weights and biases of W p 2 will be replaced by those of W p 1 .
Revised: Conversely, the weights and biases of W p 2 will be replaced by those of W p 1 .

Original: After the exchange, we get two offspring and replace the policies in Dis with these offspring.
Revised: After the exchange, we get two offspring and replace the policies in Dis with these offspring.

Original: We repeat the above crossover process until all the policies in Dis are replaced.
Revised: We repeat the above crossover process until all the policies in Dis are replaced.

Original: Then we perform mutation on all policies except the elites with a probability 90% .
Revised: Then we perform mutation on all policies except the elites with a probability 90% .

Original: For the selected agent, we perturb the weights corresponding to each action with probability α .
Revised: For the selected agent, we perturb the weights corresponding to each action with probability α .

Original: For each selected action, the same kind of perturbation is added to a certain percentage (i.e., β ) parameters.
Revised: For each selected action, the same kind of perturbation is added to a certain percentage (i.e., β ) parameters.

Original: Specifically, we perform a small Gaussian perturbation, a large
Revised: Specifically, we perform a small Gaussian perturbation, a large

Original: Gaussian perturbation, or reset these parameters with probability 90% , 5% , and 5% , respectively.
Revised: Gaussian perturbation, or reset these parameters with probability 90% , 5% , and 5% , respectively.

Original: By completing the above operation, one iteration of population evolution is completed.
Revised: By completing the above operation, one iteration of population evolution is completed.

Original: Except for α and β we introduced, we use the same settings of the hyperparameters as in PDERL.
Revised: Except for α and β we introduced, we use the same settings of the hyperparameters as in PDERL.

Original: D C OMPLETE E XPERIMENTAL D ETAILS AND F ULL R ESULTS
Revised: D C OMPLETE E XPERIMENTAL D ETAILS AND F ULL R ESULTS

Original: This session provides more experiments to help understand ERL-Re 2 more comprehensively.
Revised: This session provides more experiments to help understand ERL-Re 2 more comprehensively.

Original: We report the average with 95% confidence region based on 5 different seeds.
Revised: We report the average with 95% confidence region based on 5 different seeds.

Original: A navigation summary of additional experiments is provided as follows:
Revised: A navigation summary of additional experiments is provided as follows:

Original: Exp 1 Ablation study on behavior-level crossover and mutation operators.
Revised: Exp 1 Ablation study on behavior-level crossover and mutation operators.

Original: Exp 2 Evaluation of different choices of H used in the calculation of our surrogate fitness in Eq.
Revised: Exp 2 Evaluation of different choices of H used in the calculation of our surrogate fitness in Eq.

Original: Exp 3 Analysis of the repeated episodes used in the calculation of our surrogate fitness in Eq.
Revised: Exp 3 Analysis of the repeated episodes used in the calculation of our surrogate fitness in Eq.

Original: Exp 4 Evaluation of different choices of the mutation ratio β in Algorithm 2.
Revised: Exp 4 Evaluation of different choices of the mutation ratio β in Algorithm 2.

Original: Exp 5 Evaluation of different choices of population size.
Revised: Exp 5 Evaluation of different choices of population size.

Original: Exp 6 Experiments on using the idea of ERL-Re 2 to improve EA, i.e., Improve EA with the shared state representation and individual policy representations.
Revised: Exp 6 Experiments on using the idea of ERL-Re 2 to improve EA, i.e., Improve EA with the shared state representation and individual policy representations.

Original: Exp 7 A comparative experiment on participation rates of the elites and discarders.
Revised: Exp 7 A comparative experiment on participation rates of the elites and discarders.

Original: Exp 8 Experiments on the change of the average/maximum/minimum return of the population before and after optimizing the shared state representation.
Revised: Exp 8 Experiments on the change of the average/maximum/minimum return of the population before and after optimizing the shared state representation.

Original: Exp 9 Further exploration on unsupervised learning to improve shared state representation.
Revised: Exp 9 Further exploration on unsupervised learning to improve shared state representation.

Original: Exp 10 Further exploration on the Value Improvement Path (VIP) (Dabney et al., 2021) to improve shared state representation..
Revised: Exp 10 Further exploration on the Value Improvement Path (VIP) (Dabney et al., 2021) to improve shared state representation..

Original: Exp 11 Evaluation of different architectures, i.e., maintaining a PeVFA and an RL Critic for EA and
Revised: Exp 11 Evaluation of different architectures, i.e., maintaining a PeVFA and an RL Critic for EA and

Original: RL separately or
Revised: RL separately or

Original: only a PeVFA for both EA and RL.
Revised: only a PeVFA for both EA and RL.

Original: Exp 12 Time-consuming analysis of ERL-Re 2 (TD3) and other related algorithms.
Revised: Exp 12 Time-consuming analysis of ERL-Re 2 (TD3) and other related algorithms.

Original: Exp 13 Analysis of the number of parameters of ERL-Re 2 (TD3) and other related algorithms.
Revised: Exp 13 Analysis of the number of parameters of ERL-Re 2 (TD3) and other related algorithms.

Original: Exp 14 Experiments on the influence of other factors i.e., layer normalization, discount reward, and actor structure.
Revised: Exp 14 Experiments on the influence of other factors i.e., layer normalization, discount reward, and actor structure.

Original: Exp 15 More experiments on Deepmind Control Suite (Tassa et al., 2018) with pixel-level inputs.
Revised: Exp 15 More experiments on Deepmind Control Suite (Tassa et al., 2018) with pixel-level inputs.

Revised: Exp 16 More experiments on Starcraft II (Samvelyan et al., 2019).

Original: {+Exp 
+}
Revised: {+Exp 
+}

Revised: The experiments on MuJoCo with longer timesteps.

Original: D. A DDITIONAL A BLATION & H YPERPARAMETER A NALYSIS
Revised: D. A DDITIONAL A BLATION & H YPERPARAMETER A NALYSIS

Original: In this subsection, we provide experiments on ablation and hyperparameter analysis in Exp 1 , Exp 2 ,
Revised: In this subsection, we provide experiments on ablation and hyperparameter analysis in Exp 1 , Exp 2 ,

Original: Exp 3 , Exp 4 , Exp 5 , Exp 6 .
Revised: Exp 3 , Exp 4 , Exp 5 , Exp 6 .

Original: Exp 1 :
Revised: Exp 1 :

Original: To demonstrate the respective roles of operators, we perform ablation experiments on the behavior-level crossover and mutation operators.
Revised: To demonstrate the respective roles of operators, we perform ablation experiments on the behavior-level crossover and mutation operators.

Original: ERL-Re 2 (TD3) w/o crossover(mutation) means using parameter-level crossover(mutation) in ERL to replace the behavior-level crossover(mutation).
Revised: ERL-Re 2 (TD3) w/o crossover(mutation) means using parameter-level crossover(mutation) in ERL to replace the behavior-level crossover(mutation).

Original: The results in Fig.
Revised: The results in Fig.

Original: 10 show that both behavior-level mutation and crossover operators can deliver performance gains and the best performance can be obtained by using a combination of both.
Revised: 10 show that both behavior-level mutation and crossover operators can deliver performance gains and the best performance can be obtained by using a combination of both.

Original: Exp 2 : To analyze the effect of hyperparameter H on performance, we evaluate ERL-Re 2 (TD3) with different H on Swimmer and Walker .
Revised: Exp 2 : To analyze the effect of hyperparameter H on performance, we evaluate ERL-Re 2 (TD3) with different H on Swimmer and Walker .

Original: The results in Fig.
Revised: The results in Fig.

Original: 11 show that ERL-Re 2 is not sensitive to H , but appropriate adjustments can deliver some performance gains.
Revised: 11 show that ERL-Re 2 is not sensitive to H , but appropriate adjustments can deliver some performance gains.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Exp 3 : In the surrogate function ˆ f ( W i ) in Eq. 3, we do not consider the repeated episodes.
Revised: Exp 3 : In the surrogate function ˆ f ( W i ) in Eq. 3, we do not consider the repeated episodes.

Original: Here we provide experiments to perform the analysis.
Revised: Here we provide experiments to perform the analysis.

Original: The results in Fig.12 show that one episode is often enough to achieve a good performance.
Revised: The results in Fig.12 show that one episode is often enough to achieve a good performance.

Original: To reduce hyperparameters and simplify the tuning process, we set the episode to 1 in all tasks.
Revised: To reduce hyperparameters and simplify the tuning process, we set the episode to 1 in all tasks.

Original: Exp 4 :
Revised: Exp 4 :

Original: To analyze the effect of hyperparameter β on performance, we evaluate ERL-Re 2 (TD3) with different β on Swimmer and Walker .
Revised: To analyze the effect of hyperparameter β on performance, we evaluate ERL-Re 2 (TD3) with different β on Swimmer and Walker .

Original: The results in Fig.
Revised: The results in Fig.

Original: 13 show ERL-Re 2 is a little sensitive to β .
Revised: 13 show ERL-Re 2 is not very sensitive to β .

Original: The β determines the degree of mutation for an action.
Revised: The β determines the degree of mutation for an action.

Original: A large β means strong exploration.
Revised: A large β means strong exploration.

Original: According to our experience, in some stable environments such as Swimmer and HalfCheetach , a larger β is generally chosen, while in some unstable environments such as Walker and Hopper , a smaller β is chosen.
Revised: According to our experience, in some stable environments such as Swimmer and HalfCheetach , a larger β is generally chosen, while in some unstable environments such as Walker and Hopper , a smaller β is chosen.

Original: Exp 5 :
Revised: Exp 5 :

Original: Although the population size remains consistent 5 across all tasks in our experiments, we give some additional experiments to explore the effect of population size on performance.
Revised: Although the population size remains consistent 5 across all tasks in our experiments, we give some additional experiments to explore the effect of population size on performance.

Original: The results in Fig.
Revised: The results in Fig.

Original: 14 show that 5 is more effective than 10 in ERL-Re 2 (TD3).
Revised: 14 show that 5 is more effective than 10 in ERL-Re 2 (TD3).

Original: The main reason behind this may come from the fact that when the population is too large, the number of steps that the RL agent interacts with the environment is drastically reduced, which may lead to a decrease in RL performance.
Revised: The main reason behind this may come from the fact that when the population is too large, the number of steps that the RL agent interacts with the environment is drastically reduced, which may lead to a decrease in RL performance.

Original: This problem can be solved by considering a more efficient mechanism for evaluating the individuals to reduce the total interaction of EA properly to improve the number of steps of RL interactions.
Revised: This problem can be solved by considering a more efficient mechanism for evaluating the individuals to reduce the total interaction of EA properly to improve the number of steps of RL interactions.

Original: For a fair comparison, we also evaluate other algorithms based on different population size 5 and 10 and report the best performance.
Revised: For a fair comparison, we also evaluate other algorithms based on different population size 5 and 10 and report the best performance.

Original: Exp 6 : We provide an ablation study on ERL-Re 2 without the RL side, which can verify whether the idea of ERL-Re 2 can improve EA.
Revised: Exp 6 : We provide an ablation study on ERL-Re 2 without the RL side, which can verify whether the idea of ERL-Re 2 can improve EA.

Original: Specifically, we construct the policies with the share state representation and individual policy representations in EA and maintain a PeVFA to share the useful common knowledge in the population.
Revised: Specifically, we construct the policies with the share state representation and individual policy representations in EA and maintain a PeVFA to share the useful common knowledge in the population.

Original: The results in Fig.
Revised: The results in Fig.

Original: 15 show that the idea of ERL-Re 2 can significantly improve EA, which demonstrates the validity of the idea of ERL-Re 2 .
Revised: 15 show that the idea of ERL-Re 2 can significantly improve EA, which demonstrates the validity of the idea of ERL-Re 2 .

Original: [Figure]
Revised: [Figure]

Original:  20 means 
Revised: 10 × 20 means that 10 episodes

Original: [Figure]
Revised: [Figure]

Original:  In the results, 1 × H is effective in the tasks.
Revised: e., In the results, 1 × H is effective in the tasks.

Original: To reduce hyperparameters and simplify the tuning process, we use 1 episode in all tasks
Revised: To reduce hyperparameters and simplify the tuning process, we use 1 episode in all tasks

Original: [Figure]
Revised: [Figure]

Original: D. F URTHER U NDERSTANDING ERL-R E
Revised: D. F URTHER U NDERSTANDING ERL-R E

Original: In this subsection
Revised: In this subsection

Original: [Figure]
Revised: [Figure]

Original: we provide experiments to help readers better understand the advantages of
Revised: we provide experiments to help readers better understand the advantages of

Original: ERL-Re 2 in Exp 7 and Exp 8 .
Revised: ERL-Re 2 in Exp 7 and Exp 8 .

Original: Exp 7 : To further investigate the reason why ERL-Re 2 is more efficient than ERL and PDERL, we provide a comparative experiment on elites and discarded rates of RL agents.
Revised: Exp 7 : To further investigate the reason why ERL-Re 2 is more efficient than ERL and PDERL, we provide a comparative experiment on elites and discarded rates of RL agents.

Original: The results in
Revised: The results in

Original: Fig.
Revised: Fig.

Original: 16 show that RL agent in ERL-Re 2 (TD3) is selected as elites with a larger probability and as discarders with a smaller probability than RL agent in ERL(TD3) and PDERL(TD3), which indicates that both EA and RL in ERL-Re 2 (TD3) have more competitive effects on population evolution than in ERL(TD3) and PDERL(TD3).
Revised: 16 show that RL agent in ERL-Re 2 (TD3) is selected as elites with a larger probability and as discarders with a smaller probability than RL agent in ERL(TD3) and PDERL(TD3), which indicates that both EA and RL in ERL-Re 2 (TD3) have more competitive effects on population evolution than in ERL(TD3) and PDERL(TD3).

Original: Exp 8 : To further study the effect of the state representation on the population, we record the added undiscounted return after optimizing the shared state representation.
Revised: Exp 8 : To further study the effect of the state representation on the population, we record the added undiscounted return after optimizing the shared state representation.

Original: Specifically, we record the change in average return, maximum return, and minimum return of the population before and after the optimization of the shared state representation.
Revised: Specifically, we record the change in average return, maximum return, and minimum return of the population before and after the optimization of the shared state representation.

Original: The results in Fig.
Revised: The results in Fig.

Original: 17 show the improvement of shared state representations can improve the average reward of the population, where the major improvement comes from poor policies.
Revised: 17 show the improvement of shared state representations can improve the average reward of the population, where the major improvement comes from poor policies.

Original: This demonstrates that the optimization of shared state representations can construct a better policy space for all policies, thus facilitating the evolution of the population.
Revised: This demonstrates that the optimization of shared state representations can construct a better policy space for all policies, thus facilitating the evolution of the population.

Original: D. A DVANCED S TATE R EPRESENTATION
Revised: D. A DVANCED S TATE R EPRESENTATION

Original: In this subsection, we further explore how to learn more effective shared state representation in Exp 9 and Exp 10 .
Revised: In this subsection, we further explore how to learn more effective shared state representation in Exp 9 and Exp 10 .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Exp 9 : We conduct additional exploration to enable further improvements of shared state representations by unsupervised learning.
Revised: Exp 9 : We conduct additional exploration to enable further improvements of shared state representations by unsupervised learning.

Original: To make the representation have dynamic information about the environment, we add the mutual information (MI) loss to the first layer of the shared state representation network.
Revised: To make the representation have dynamic information about the environment, we add the mutual information (MI) loss to the first layer of the shared state representation network.

Original: Specifically, When given current state s and the next state s ′ , we want to make the representation based on s and s ′ have a high correlation with the corresponding action a , which can be donated as I ( s, s ′ ; a ) .
Revised: Specifically, When given current state s and the next state s ′ , we want to make the representation based on s and s ′ have a high correlation with the corresponding action a , which can be donated as I ( s, s ′ ; a ) .

Original: We use the MINE (Belghazi et al., 2018) to estimate MI.
Revised: We use the MINE (Belghazi et al., 2018) to estimate MI.

Original: MINE is a neural network to estimate the MI of any two random variables.
Revised: MINE is a neural network to estimate the MI of any two random variables.

Original: To make the shared state representation contain the information, we obtain one variable by combining the representations of s and s ′ from the first layer of the shared state representation network and use the representations and a as the inputs to
Revised: To make the shared state representation contain the information, we obtain one variable by combining the representations of s and s ′ from the first layer of the shared state representation network and use the representations and a as the inputs to

Original: MINE.
Revised: MINE.

Original: We evaluate ERL-Re 2 (TD3) with the MI loss and ERL-Re 2 on Ant and Walker .
Revised: We evaluate ERL-Re 2 (TD3) with the MI loss and ERL-Re 2 on Ant and Walker .

Original: The results in Fig.18 show that the MI loss can bring some performance improvement in Ant and fail in Walker .
Revised: The results in Fig.18 show that the MI loss can bring some performance improvement in Ant and fail in Walker .

Original: We find that the MI loss has both good and bad effects on the final performance.
Revised: We find that the MI loss has both good and bad effects on the final performance.

Original: A more stable and efficient way to improve the performance is left as a subsequent work.
Revised: A more stable and efficient way to improve the performance is left as a subsequent work.

Original: Exp 10 : Inspired by the idea of VIP (Dabney et al., 2021), we conduct experiments to optimize the shared state representation with the previous policy representations.
Revised: Exp 10 : Inspired by the idea of VIP (Dabney et al., 2021), we conduct experiments to optimize the shared state representation with the previous policy representations.

Original: Specifically, we optimize the shared state representation with policy representations not only current but also the previous generation.
Revised: Specifically, we optimize the shared state representation with policy representations not only current but also the previous generation.

Original: The results in Fig.
Revised: The results in Fig.

Original: 19 show that this approach does not guarantee performance gains.
Revised: 19 show that this approach does not guarantee performance gains.

Original: This is just an initial attempt to see how the shared state representation can be further improved by self-supervised learning, which deserves subsequent study.
Revised: This is just an initial attempt to see how the shared state representation can be further improved by self-supervised learning, which deserves subsequent study.

Original: We consider this as future work.
Revised: We consider this as future work.

Original: D. D ISCUSSION O F O THER A SPECTS
Revised: D. D ISCUSSION O F O THER A SPECTS

Original: In this subsection, we further discuss some other aspects of ERL-Re 2 such as the architectures selection, time consumption, the number of parameters and other factors in Exp 11 , Exp 12 , Exp 13 ,
Revised: In this subsection, we further discuss some other aspects of ERL-Re 2 such as the architectures selection, time consumption, the number of parameters and other factors in Exp 11 , Exp 12 , Exp 13 ,

Original: Exp 14 .
Revised: Exp 14 .

Original: Exp 11 :
Revised: Exp 11 :

Original: In ERL-Re 2 , we introduce a PeVFA for EA while maintaining the RL critic for RL.
Revised: In ERL-Re 2 , we introduce a PeVFA for EA while maintaining the RL critic for RL.

Original: In principle, RL can also be improved by the PeVFA.
Revised: In principle, RL can also be improved by the PeVFA.

Original: leading to only one PeVFA for both the EA and
Revised: leading to only one PeVFA for both the EA and

Original: RL agents.
Revised: RL agents.

Original: Both architectures can achieve useful common knowledge sharing.
Revised: Both architectures can achieve useful common knowledge sharing.

Original: To further verify the effectiveness, we conduct comparison experiments between these two architectures in the TD
Revised: To further verify the effectiveness, we conduct comparison experiments between these two architectures in the TD

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: version.
Revised: version.

Original: The results in Fig.
Revised: The results in Fig.

Original: 20 show that similar performance can be obtained with both architectures.
Revised: 20 show that similar performance can be obtained with both architectures.

Original: But to preserve the flexibility and corresponding characteristics of RL methods, we retain the original critic.
Revised: But to preserve the flexibility and corresponding characteristics of RL methods, we retain the original critic.

Original: From the perspective of RL, our architecture does not make changes to the original structure, e.g., maintaining the architecture of the value function, and this separately maintained approach is more convenient to analyze and combine.
Revised: From the perspective of RL, our architecture does not make changes to the original structure, e.g., maintaining the architecture of the value function, and this separately maintained approach is more convenient to analyze and combine.

Original: Therefore, we choose this architecture as the final solution.
Revised: Therefore, we choose this architecture as the final solution.

Original: Exp 12 : We evaluate the time consumption of different algorithms on Walker .
Revised: Exp 12 : We evaluate the time consumption of different algorithms on Walker .

Original: The experiment is carried out on NVIDIA GTX 

Revised: The experiment is carried out on NVIDIA GTX

Original: Ti GPU with Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz.
Revised: Ti GPU with Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz.

Original: We evaluate the time consumption of each algorithm individually with no additional programs running on the device.
Revised: We evaluate the time consumption of each algorithm individually with no additional programs running on the device.

Original: As shown in Table 1, ERL-Re 2 incurs some additional time consumption compared to other methods.
Revised: As shown in Table 1, ERL-Re 2 incurs some additional time consumption compared to other methods.

Original: The main time consumption comes from the training of PeVFA and the shared state representation.
Revised: The main time consumption comes from the training of PeVFA and the shared state representation.

Original: 73% of the total time is spent on training.
Revised: 73% of the total time is spent on training.

Original: If the researchers are sensitive to time overhead, a distributed approach can be used to reduce the time overhead.
Revised: If the researchers are sensitive to time overhead, a distributed approach can be used to reduce the time overhead.

Original: Exp 13 : We provide the number of parameters (i.e., weights and biases) on HalfCheetach in Table 2.
Revised: Exp 13 : We provide the number of parameters (i.e., weights and biases) on HalfCheetach in Table 2.

Original: The results show that ERL-Re 2 (TD3) is competitive with other methods in terms of the number of parameters.
Revised: The results show that ERL-Re 2 (TD3) is competitive with other methods in terms of the number of parameters.

Original: The number of parameters of ERL-Re 2 (TD3) is mainly derived from PeVFA, i.e., 47 .
Revised: The number of parameters of ERL-Re 2 (TD3) is mainly derived from PeVFA, i.e., 47 .

Original: In addition, ERL-Re 2 has better scalability capability than other ERL-related methods as the population size increases.
Revised: In addition, ERL-Re 2 has better scalability capability than other ERL-related methods as the population size increases.

Original: Since ERL-Re 2 only introduces a linear policy instead of an independent nonlinear policy network for each agent.
Revised: Since ERL-Re 2 only introduces a linear policy instead of an independent nonlinear policy network for each agent.

Original: For example, on HalfCheetach , when the population size
Revised: For example, on HalfCheetach , when the population size

Original: [Figure]
Revised: [Figure]

Original: increase to 100, ERL-Re 2 only need additional 22 . 4% parameters (i.e., 171570), while other methods such as CERL(TD3) and CEM-TD3 needs additional 1047378600 parameters, which is not feasible.
Revised: increase to 100, ERL-Re 2 only need additional 22 . 4% parameters (i.e., 171570), while other methods such as CERL(TD3) and CEM-TD3 needs additional 1047378600 parameters, which is not feasible.

Original: Exp 14 : We provide more experiments to eliminate the influence of some other factors.
Revised: Exp 14 : We provide more experiments to eliminate the influence of some other factors.

Original: In the official codebase of ERL and PDERL, layer normalization is used for both actor and critic.
Revised: In the official codebase of ERL and PDERL, layer normalization is used for both actor and critic.

Original: Our code is built on the codebase of PDERL, thus ERL-Re 2 also uses layer normalization.
Revised: Our code is built on the codebase of PDERL, thus ERL-Re 2 also uses layer normalization.

Original: But in the official codebase of TD3, layer normalization is not employed.
Revised: But in the official codebase of TD3, layer normalization is not employed.

Original: To exclude the effect of layer normalization, we provide experiments of TD3 with layer normalization.
Revised: To exclude the effect of layer normalization, we provide experiments of TD3 with layer normalization.

Original: The results in Fig.21 show that TD3 with layer normalization has a similar performance to TD3, which demonstrates that the strengths of
Revised: The results in Fig.21 show that TD3 with layer normalization has a similar performance to TD3, which demonstrates that the strengths of

Original: ERL-Re 2 do not come from the layer normalization.
Revised: ERL-Re 2 do not come from the layer normalization.

Original: Second, we want to exclude the influence of discount rewards on performance.
Revised: Second, we want to exclude the influence of discount rewards on performance.

Original: Thus we provide experiments by comparing ERL-Re 2 (TD3) using MC return of one episode as the surrogate of the fitness and ERL-Re 2 (TD3) using discount MC return of one episode as a surrogate of the fitness.
Revised: Thus we provide experiments by comparing ERL-Re 2 (TD3) using MC return of one episode as the surrogate of the fitness and ERL-Re 2 (TD3) using discount MC return of one episode as a surrogate of the fitness.

Original: The results in Fig.
Revised: The results in Fig.

Original: 22 show that only using discounted MC return as the surrogate of the fitness is not effective as using undiscounted MC return, which indicates that the advantages of ERL-Re 2 do not come from the discounted form in the surrogate fitness ˆ f
Revised: 22 show that only using discounted MC return as the surrogate of the fitness is not effective as using undiscounted MC return, which indicates that the advantages of ERL-Re 2 do not come from the discounted form in the surrogate fitness ˆ f

Original: ( W i ) .
Revised: ( W i ) .

Original: Third, the network structure of the actor of ERL and PDERL is not consistent with the structure of the TD3 paper.
Revised: Third, the network structure of the actor of ERL and PDERL is not consistent with the structure of the TD3 paper.

Original: Thus we provide a comprehensive comparison, one maintains the structure of the original paper and one is consistent with the structure in the TD3 paper.
Revised: Thus we provide a comprehensive comparison, one maintains the structure of the original paper and one is consistent with the structure in the TD3 paper.

Original: The results in Fig.and Fig.
Revised: The results in Fig.and Fig.

Original: 24 show that the algorithms with the structure of the TD3 paper improve the performance in some environments and degrade in others.
Revised: 24 show that the algorithms with the structure of the TD3 paper improve the performance in some environments and degrade in others.

Original: We can find that a large network structure (i.e., the structure of TD3 paper) leads to large fluctuations in ERL such as Swimmer , where EA plays a more critical role than RL.
Revised: We can find that a large network structure (i.e., the structure of TD3 paper) leads to large fluctuations in ERL such as Swimmer , where EA plays a more critical role than RL.

Original: The main reason behind this is that parameter-level crossover and mutation in large networks are more unstable than in small networks.
Revised: The main reason behind this is that parameter-level crossover and mutation in large networks are more unstable than in small networks.

Original: In summary, ERL-Re 2 (TD3) is still significantly better than the best performance in both structures.
Revised: In summary, ERL-Re 2 (TD3) is still significantly better than the best performance in both structures.

Original: [Table]
Revised: [Table]

Original: For ERL and PDERL, we use | to separate
Revised: For ERL and PDERL, we use | to separate

Original: Algorithm
Revised: Algorithm

Original: [Table]
Revised: [Table]

Original: number  456302 | 456302 |
Revised: number  456302 | 456302 |

Original: [Figure]
Revised: [Figure]

Original: We design two versions.
Revised: We design two versions.

Original: The version 1 means firstly optimizing PeVFA with the current and previous generation’ policy representations, then optimizing the state representations with these policy representations.
Revised: The version 1 means firstly optimizing PeVFA with the current and previous generation’ policy representations, then optimizing the state representations with these policy representations.

Original: The version 2 means directly optimizing the state representations with the current and previous generation’ policy representations
Revised: The version 2 means directly optimizing the state representations with the current and previous generation’ policy representations

Original: [Figure]
Revised: [Figure]

Original: is using the PeVFA to update RL without introducing the RL Critic.
Revised: is using the PeVFA to update RL without introducing the RL Critic.

Original: PeVFA for EA and Critic for RL is the architecture chosen in this paper.
Revised: PeVFA for EA and Critic for RL is the architecture chosen in this paper.

Original: The results show that both architectures are feasible and can achieve competitive performances.
Revised: The results show that both architectures are feasible and can achieve competitive performances.

Original: D. E XPERIMENTS WITH VISUAL STATE INPUTS
Revised: D. E XPERIMENTS WITH VISUAL STATE INPUTS

Original: We evaluate whether ERL-Re 2 could bring improvement in Deepmind Control Suite (Tassa et al., 2018) with visual state inputs in Exp 15 .
Revised: We evaluate whether ERL-Re 2 could bring improvement in Deepmind Control Suite (Tassa et al., 2018) with visual state inputs in Exp 15 .

Original: Exp 15 : We evaluate ERL-Re 2 on four tasks.
Revised: Exp 15 : We evaluate ERL-Re 2 on four tasks.

Original: In these tasks, we take pixel-level images as inputs.
Revised: In these tasks, we take pixel-level images as inputs.

Original: Specifically, we implement ERL-Re 2 based on RAD (Laskin et al., 2020) (which can be simply viewed as SAC + Image Augmentation for image inputs).
Revised: Specifically, we implement ERL-Re 2 based on RAD (Laskin et al., 2020) (which can be simply viewed as SAC + Image Augmentation for image inputs).

Original: We follow the conventional setting and provide the performance comparison in 100k environment steps.
Revised: We follow the conventional setting and provide the performance comparison in 100k environment steps.

Original: We use 4 representative DMC tasks, covering different robot morphology.
Revised: We use 4 representative DMC tasks, covering different robot morphology.

Original: The results shown in Table 3 are means and stds over 5 seeds.
Revised: The results shown in Table 3 are means and stds over 5 seeds.

Original: [Table]
Revised: [Table]

Original: We can observe that ERL-Re 2 also leads to significant improvements in visual control, which demonstrates the effectiveness of ERL-Re 2 .
Revised: We can observe that ERL-Re 2 also leads to significant improvements in visual control, which demonstrates the effectiveness of ERL-Re 2 .

Original: This also indicates that the techniques of addressing visual states (e.g., image augmentation) are orthogonal to our method, which is worthwhile for further development in the future. 
Revised: This also indicates that the techniques of addressing visual states (e.g., image augmentation) are orthogonal to our method, which is worthwhile for further development in the future. [Figure] [Figure]

Original:  E M ETHOD I MPLEMENTATION D ETAILS
Revised: c o un t e d R e t u r n [Figure] D. E XPERIMENTS IN H IGH - DIMENSIONAL C OMPLEX T ASKS

Revised: For complex high-dimensional tasks, we choose the multi-agent the StarCraft II micromanagement (SMAC) benchmark

Revised: (Samvelyan et al., 2019) as the testbed for its rich environments and high complexity of control.

Revised: The SMAC benchmark requires learning policies in a large action space.

Revised: Agents can move in four cardinal directions, stop, take noop (do nothing), or select an enemy to attack at each timestep.

Revised: Therefore, if there are n e enemies in the map, the action space for each allied unit contains n e + 6 discrete actions.

Revised: To solve these high complexity tasks, we need to integrate

Revised: ERL-Re 2 with Multi-Agent Reinforcement Learning (MARL) algorithm.

Revised: Fortunately, ERL-Re 2 can be combined with the MARL algorithms easily.

Revised: Specifically, we need to maintain a population of teams (i.e., Multiple policies that control multiple agents for collaboration are treated as a team).

Revised: Each policy in the team needs to follow the two-scale representation-based policy construction.

Revised: Thus knowledge can be efficiently conveyed through shared representations and reinforcement and evolution occur in the linear policy space.

Revised: For the base MARL algorithm, we choose FACMAC, an advanced Multi-Agent Reinforcement

Revised: Learning algorithm published in NeurIPS 2021 (Peng et al., 2021).

Revised: Then we combine ERL-Rewith FACMAC and evaluate it on four tasks.

Revised: Our implementation is based on the official code of

Revised: FACMAC 2 and the hyperparameters are kept consistent.

Revised: To simplify hyperparameter introduced by

Revised: ERL-Re 2 , we set p to 1.0, K to 1 on all tasks and select β from { 0 . 2 , 0 . 05 , 0 . 01 } .

Revised: The results in

Revised: Figure 25 show that ERL-Re 2 can significantly improve FACMAC in terms of convergence speed and final performance.

Original: All experimentsare carried out on [-NVIDIA GTX 
-]
Revised: It is worth noting that the joint action space of so m any b aneling , MMM 2 and 2 c v s 6 4 zg is greater than 100 dimensions, which demonstrates that ERL-Re 2 can bring improvements to the original algorithm in high-dimensional complex control tasks. Through the above experiments, we further demonstrated the efficiency and generalization of ERL-Re 2 . D. E XPERIMENTS ON M U J O C O WITH L ONGER T IMESTEPS In Figure 4, we provide the experiments on [-NVIDIA GTX 
-]MuJoCo with 1 million timesteps, which is mainly consistent with the training timestep in SAC (Haarnoja et al., 2018) and TD3 (Fujimoto et al., 2018).

Original: Ti GPU with Intel(R) Xeon(R) CPU E5v4 @ 2.40GHz.
Revised: To demonstrate the competitiveness of ERL-Re 2 in longer timesteps, we provide the experiments in the TD3 version with 3 million timesteps.

Original: [Figure]
Revised: Since Swimmer and Hopper have already achieved convergence performance, we mainly provide the experiments on the remaining four tasks. The results may differ slightly from those in the main text, as they are obtained by running on different servers.

Original: [Figure]
Revised: The results in Figure 26 show ERL-Re 2 still significantly outperforms other algorithms with 3 million timesteps, which further verifies the efficiency of ERL-Re 2 .

Original: c o un t e d R e t u r n
Revised: E M ETHOD I MPLEMENTATION D ETAILS

Original: Humanoid{+All experiments are carried out on NVIDIA GTX 
 Ti GPU with Intel(R) Xeon(R) CPU E5v4 @ 2.40GHz.+}
Revised: {+All experiments are carried out on NVIDIA GTX 
 Ti GPU with Intel(R) Xeon(R) CPU E5v4 @ 2.40GHz.+}

Original: E. I MPLEMENTATION OF B ASELINES
Revised: E. I MPLEMENTATION OF B ASELINES

Original: For all baseline algorithms, we use the official implementation.
Revised: For all baseline algorithms, we use the official implementation.

Original: In our paper, there are two main implementations, one combines TD3 and the other combines DDPG.
Revised: In our paper, there are two main implementations, one combines TD3 and the other combines DDPG.

Original: Many of these official implementations contain two implementations such as CEM-RL 2 and CERL 3 .
Revised: Many of these official implementations contain two implementations such as CEM-RL 3 and CERL 4 .

Original: For methods that do not have two implementations such as ERL 4 and PDERL 5 (both based on DDPG), we modify the DDPG to
Revised: For methods that do not have two implementations such as ERL 5 and PDERL 6 (both based on DDPG), we modify the DDPG to

Original: TD3 which is a very simple and straightforward extension.
Revised: TD3 which is a very simple and straightforward extension.

Original: For the basic algorithm TD3 6 , we use the official implementation.
Revised: For the basic algorithm TD3 7 , we use the official implementation.

Original: For EA, we use the official implementation in PDERL.
Revised: For EA, we use the official implementation in PDERL.

Original: For SAC and PPO, we use the implementation from stable-baseline3 7 .
Revised: For SAC and PPO, we use the implementation from stable-baseline3 8 .

Original: We fine-tuned ERL-related baselines, mainly including population size, episodes of agents interacting with the environment in the population, elite rate, injection frequency, etc.
Revised: We fine-tuned ERL-related baselines, mainly including population size, episodes of agents interacting with the environment in the population, elite rate, injection frequency, etc.

Original: E. N ETWORK A RCHITECTURE
Revised: E. N ETWORK A RCHITECTURE

Original: This section details the architecture of the networks.
Revised: This section details the architecture of the networks.

Original: Our code is built on the official codebase of
Revised: Our code is built on the official codebase of

Original: PDERL.
Revised: PDERL.

Original: Most of the structure remains the same.
Revised: Most of the structure remains the same.

Original: For structures specific to our framework, the shared [Figure] still significantly better than the best performance of the two structures.
Revised: For structures specific to our framework, the shared state representation network is constructed by two fully connected layers with 400 and 300 units.

Original: [Figure]

Original: with 400 and 300 units.

Original: The policy representation is constructed by one fully connected layer with action_dim units.
Revised: The policy representation is constructed by one fully connected layer with action_dim units.

Original: PeVFA takes state, action and policy representation as inputs and maintains double Q networks which are similar to TD3.
Revised: PeVFA takes state, action and policy representation as inputs and maintains double Q networks which are similar to TD3.

Original: The policy representation can be regarded as a combination of a matrix with shape [300 , action_dim ] (i.e., weights) and a vector with shape [ action_dim ] (i.e., biases) which can be concatenated as a matrix with shape [300 + 1 , action_dim ] .
Revised: The policy representation can be regarded as a combination of a matrix with shape [300 , action_dim ] (i.e., weights) and a vector with shape [ action_dim ] (i.e., biases) which can be concatenated as a matrix with shape [300 + 1 , action_dim ] .

Original: We first encode each vector with shape [300 + 1] of the policy representations with 3 fully connected layers with units 64 and leaky_relu activation function.
Revised: We first encode each vector with shape [300 + 1] of the policy representations with 3 fully connected layers with units 64 and leaky_relu activation function.

Original: Thus we can get an embedding list with shape [64 , action_dim ] and get the final policy embedding with shape [64] by taking the mean value of the embedding list in the action dimension.
Revised: Thus we can get an embedding list with shape [64 , action_dim ] and get the final policy embedding with shape [64] by taking the mean value of the embedding list in the action dimension.

Original: With the policy embedding, we concatenate the policy embedding, states, and actions as the input to an MLP with 2 fully connected layers with units 400 and 300 and get the predicted value by PeVFA.
Revised: With the policy embedding, we concatenate the policy embedding, states, and actions as the input to an MLP with 2 fully connected layers with units 400 and 300 and get the predicted value by PeVFA.

Original: The activation functions in PeVFA all use leaky_relu .
Revised: The activation functions in PeVFA all use leaky_relu .

Original: We list structures in Table 4 and 5.
Revised: We list structures in Table 4 and 5.

Original: For other structures, we take the settings directly from the codebase of PDERL.
Revised: For other structures, we take the settings directly [Figure] still significantly better than the best performance of the two structures.

Original: The network structure is the same in ERL-Re 2 (TD3) and ERL-Re(DDPG).[Table]
Revised: [Figure] network structure is the same in ERL-Re 2 (TD3) and ERL-Re(DDPG).

Original: [Table]
Revised: [Table]

Original: E. H YPERPARAMETERS
Revised: E. H YPERPARAMETERS

Original: This section details the hyperparameters across different tasks.
Revised: This section details the hyperparameters across different tasks.

Original: Some hyperparameters are kept consistent across all tasks.
Revised: Some hyperparameters are kept consistent across all tasks.

Original: Population size is 5 for both ERL-Re 2 (TD3) and ERL-Re 2 (DDPG).
Revised: Population size is 5 for both ERL-Re 2 (TD3) and ERL-Re 2 (DDPG).

Original: α for mutation operators is 1.0 for both ERL-Re 2 (TD3) and ERL-Re 2 (DDPG). 
Revised: α for mutation operators is 1.0 for both ERL-Re 2 (TD3) and ERL-Re 2 (DDPG). The number of [Figure] [Figure]

Revised: ( state-action_dim + 64 , 400) (301 , 64)

Revised: leaky_relu leaky_relu (400 , 300) (64 , 64)

Original: The number of episodes of interaction with the environment for surrogate fitness is 1.
Revised: leaky_relu leaky_relu (300 , 1) (64 , 64) episodes of interaction with the environment for surrogate fitness is 1.

Original: γ is 0.99 except 0.999 for
Revised: γ is 0.99 except 0.999 for

Original: Swimmer (all baselines follow this setting).
Revised: Swimmer (all baselines follow this setting).

Original: Since ERL-Re 2 is built on PDERL, other settings remain the same.
Revised: Since ERL-Re 2 is built on PDERL, other settings remain the same.

Original: In addition, we list other hyperparameters specific to ERL-Re 2 which varied across tasks in
Revised: In addition, we list other hyperparameters specific to ERL-Re 2 which varied across tasks in

Original: Table 6 and Table 7.
Revised: Table 6 and Table 7.

Original: In addition to this, we find that ERL-Re 2 is very stable, and if you don’t want to tune too many hyperparameters , competitive performance can be obtained when keeping K and β consistent with the settings in the table and do not use our surrogate fitness (i.e., p =1.0).
Revised: In addition to this, we find that ERL-Re 2 is very stable, and if you don’t want to tune too many hyperparameters , competitive performance can be obtained when keeping K and β consistent with the settings in the table and do not use our surrogate fitness (i [Table] e., p =1.0).

Original: We prove this in Fig.6d.
Revised: We prove this in Fig.6d.

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: Env

Original:  [Table] 
Revised: [Figure] [Table] [Table] [Figure]

Revised: Walker

Revised: [Table]

Revised: [Table]

Original: Humanoid 0.
Revised: Swimmer [Table] [Figure] Hopper

Original: [Table]
Revised: [Table]

Revised: Ant 0.

Revised: Humanoid 0.

Original: F A DDITIONAL D ISCUSSION F. T HE A LTERNATIVE OF R EPLACING GA BY CEM
Revised: F A DDITIONAL D ISCUSSION F. T HE A LTERNATIVE OF R EPLACING GA BY CEM

Original: In this paper, we use GA as the basic evolutionary algorithm to combine with different RL algorithms.
Revised: In this paper, we use GA as the basic evolutionary algorithm to combine with different RL algorithms.

Original: ERL-Re 2 can utilize different evolutionary algorithms to exploit the advantages of these algorithms.
Revised: ERL-Re 2 can utilize different evolutionary algorithms to exploit the advantages of these algorithms.

Original: Here we discuss how to use CEM (Pourchot & Sigaud, 2019) to replace GA in ERL-Re 2 .
Revised: Here we discuss how to use CEM (Pourchot & Sigaud, 2019) to replace GA in ERL-Re 2 .

Original: The pseudo-code is shown in Algorithm 3.
Revised: The pseudo-code is shown in Algorithm 3.

Original: To achieve this, we need to modify two aspects: 1) population generation.
Revised: To achieve this, we need to modify two aspects: 1) population generation.

Original: 2) population evolution.
Revised: 2) population evolution.

Original: For the first aspect, we need to use CEM to generate the population instead of pre-defining the entire population by constructing a mean policy W µ to sample the population based on a predefined covariance matrix Σ = σ init
Revised: For the first aspect, we need to use CEM to generate the population instead of pre-defining the entire population by constructing a mean policy W µ to sample the population based on a predefined covariance matrix Σ = σ init

Original: I .
Revised: I .

Original: For the second aspect, we need to select the top T policies to update W µ and Σ =
Revised: For the second aspect, we need to select the top T policies to update W µ and Σ =

Original: σ init I instead of the crossover and mutation operators.
Revised: σ init I instead of the crossover and mutation operators.

Original: F. T RAINING P E VFA IN A N O FF -P OLICY F ASHION
Revised: F. T RAINING P E VFA IN A N O FF -P OLICY F ASHION

Original: To our knowledge, PeVFA (Tang et al., 2020) and related variants are trained in an on-policy fashion in prior works.
Revised: To our knowledge PeVFA (Tang et al., 2022) and related variants are trained in an on-policy fashion in prior works.

Original: In the original paper of PeVFA, a newly implemented PPO algorithm, called[Table] [Table] [Table]
Revised: In the original paper of PeVFA, a newly implemented PPO algorithm, called

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: PPO-PeVFA is studied.
Revised: PPO-PeVFA is studied.

Original: In PPO-PeVFA, the algorithm saves states, returns and the corresponding policies in a replay buffer; then a PeVFA V θ ( s, χ π ) is trained by the Monte-Carlo method, based on the experiences collected in the replay buffer with some policy representation χ π .
Revised: In PPO-PeVFA, the algorithm saves states, returns and the corresponding policies in a replay buffer; then a PeVFA V θ ( s, χ π ) is trained by the Monte-Carlo method, based on the experiences collected in the replay buffer with some policy representation χ π .

Original: Note that although historical experiences are replayed, each policy uses its own collected experiences to train V θ , thus being on-policy.
Revised: Note that although historical experiences are replayed, each policy uses its own collected experiences to train V θ , thus being on-policy.

Original: Training the PeVFA V θ in off-policy manner needs to introduce additional mechanisms such as importance sampling method, which are not implemented in their work.
Revised: Training the PeVFA V θ in off-policy manner needs to introduce additional mechanisms such as importance sampling method, which are not implemented in their work.

Original: Similarly, this case can be also be found in (Faccio et al., 2021).
Revised: Similarly, this case can be also be found in (Faccio et al., 2021).

Original: In this work, we train PeVFA in an off-policy manner for the first time.
Revised: In this work, we train PeVFA in an off-policy manner for the first time.

Original: We achieve this in a more efficient way by training a PeVFA Q θ ( s, a, W ) with 1-step TD learning and the linear policy representation W .
Revised: We achieve this in a more efficient way by training a PeVFA Q θ ( s, a, W ) with 1-step TD learning and the linear policy representation W .

Original: That is, the value function of each policy can be learned (with its representation
Revised: That is, the value function of each policy can be learned (with its representation

Original: W i and Q θ ) from any off-policy experience ( s, a, s ′ , r ) according to Eq.
Revised: W i and Q θ ) from any off-policy experience ( s, a, s ′ , r ) according to Eq.

Original: We emphasize that this is appealing, and we believe that this can realize effective training of PeVFA.
Revised: We emphasize that this is appealing, and we believe that this can realize effective training of PeVFA.

Original: In turn, the ability of
Revised: In turn, the ability of

Original: PeVFA in conveying the value generalization among policies can be better rendered.
Revised: PeVFA in conveying the value generalization among policies can be better rendered.

Original: We will further investigate the potential of this point in the future.
Revised: We will further investigate the potential of this point in the future.

Original: We use PeVFA to provide value estimates for each agent (in the EA population).
Revised: We use PeVFA to provide value estimates for each agent (in the EA population).

Original: One may note that the same purpose can be achieved by maintaining a conventional Q network for each agent individually.
Revised: One may note that the same purpose can be achieved by maintaining a conventional Q network for each agent individually.

Original: However, this means suffers from the issue of scalability, resulting in significant time and resource overhead.
Revised: However, this means suffers from the issue of scalability, resulting in significant time and resource overhead.

Original: In addition, this approach does not use policy representations and thus cannot take advantage of value function generalization to improve value function learning.
Revised: In addition, this approach does not use policy representations and thus cannot take advantage of value function generalization to improve value function learning.