Original: R EWARDLESS O PEN -E NDED L EARNING (ROEL)
Revised: R EWARDLESS O PEN -E NDED L EARNING (ROEL)

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Open-ended learning algorithms aim to automatically generate challenges and solutions to an unending sequence of learning opportunities.
Revised: Open-ended learning algorithms aim to automatically generate challenges and solutions to an unending sequence of learning opportunities.

Original: In Reinforcement
Revised: In Reinforcement

Original: Learning (RL) recent approaches to open-ended learning, such as Paired OpenEnded Trailblazer (POET), focus on collecting a diverse set of solutions based on the novelty of an agents pre-deﬁned reward function.
Revised: Learning (RL) recent approaches to open-ended learning, such as Paired OpenEnded Trailblazer (POET), focus on collecting a diverse set of solutions based on the novelty of an agents pre-deﬁned reward function.

Original: In many practical RL tasks deﬁning an effective reward function a priori is often hard and can hinder an agents ability to explore many behaviors that could ultimately be more performant.
Revised: In many practical RL tasks deﬁning an effective reward function a priori is often hard and can hinder an agents ability to explore many behaviors that could ultimately be more performant.

Original: In this work we combine open-ended learning with unsupervised reinforcement learning to train agents to learn a diverse set of complex skills.
Revised: In this work we combine open-ended learning with unsupervised reinforcement learning to train agents to learn a diverse set of complex skills.

Original: We propose a procedure to combine skill-discovery via mutual information, using the POET algorithm as an open-ended framework to teach agents increasingly complex groups of diverse skills.
Revised: We propose a procedure to combine skill-discovery via mutual information, using the POET algorithm as an open-ended framework to teach agents increasingly complex groups of diverse skills.

Original: Experimentally we demonstrate this approach yields agents capable of demonstrating identiﬁable skills over a range of environments, that can be extracted and utilized to solve a variety of tasks.
Revised: Experimentally we demonstrate this approach yields agents capable of demonstrating identiﬁable skills over a range of environments, that can be extracted and utilized to solve a variety of tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Development of machine learning over the past several decades has principally relied upon researchers solving a series of challenges proposed by experts within the community (e.g. MNIST
Revised: Development of machine learning over the past several decades has principally relied upon researchers solving a series of challenges proposed by experts within the community (e.g. MNIST

Original: Lecun et al.
Revised: Lecun et al.

Original: (1998), ATARI Bellemare et al.
Revised: (1998), ATARI Bellemare et al.

Original: and robotics Yu et al.
Revised: and robotics Yu et al.

Original: Learning algorithms typically solve these challenges through a huge number of observations, and whilst they have exceeded human-level performance on a variety of these tasks (Silver et al., 2017), (Berner et al., 2019), they often require signiﬁcantly more examples before coming close to matching human performance (Toromanoff et al., 2019).
Revised: Learning algorithms typically solve these challenges through a huge number of observations, and whilst they have exceeded human-level performance on a variety of these tasks (Silver et al., 2017), (Berner et al., 2019), they often require signiﬁcantly more examples before coming close to matching human performance (Toromanoff et al., 2019).

Original: Humans in contrast leverage many diverse experiences to solve complex tasks (Wang et al., 2018) often without a well deﬁned objective or reward.
Revised: Humans in contrast leverage many diverse experiences to solve complex tasks (Wang et al., 2018) often without a well deﬁned objective or reward.

Original: Should we attempt to teach reinforcement learning (RL) agents in a similar fashion?
Revised: Should we attempt to teach reinforcement learning (RL) agents in a similar fashion?

Original: We use this question to motivate the design a procedure that aims to learn a diverse set of increasingly complex novel behaviors without supervision (labels) or incentives (external rewards).
Revised: We use this question to motivate the design a procedure that aims to learn a diverse set of increasingly complex novel behaviors without supervision (labels) or incentives (external rewards).

Original: Open-ended algorithms aim to automatically generate challenges to be solved (Forestier et al., 2020), (Schmidhuber, 2012), (Standish, 2002), (Secretan et al., 2011) that can serve as stepping stones to increasingly complex problems.
Revised: Open-ended algorithms aim to automatically generate challenges to be solved (Forestier et al., 2020), (Schmidhuber, 2012), (Standish, 2002), (Secretan et al., 2011) that can serve as stepping stones to increasingly complex problems.

Original: These algorithms are useful in designing a general unsupervised learning procedure because they do not rely upon manually selected challenges; instead they search for challenges that form stepping stones to solve larger ones, collecting a diverse set of experiences along the way.
Revised: These algorithms are useful in designing a general unsupervised learning procedure because they do not rely upon manually selected challenges; instead they search for challenges that form stepping stones to solve larger ones, collecting a diverse set of experiences along the way.

Original: Designing algorithms to achieve these open-endedness can be challenging.
Revised: Designing algorithms to achieve these open-endedness can be challenging.

Original: Often requiring a careful balance between diversity, optimization (Mouret & Clune, 2015), (Pugh et al., 2016), and a domain complex enough for sufﬁcient experiences/challenges to be generated in order to learn a suitable curricula.
Revised: Often requiring a careful balance between diversity, optimization (Mouret & Clune, 2015), (Pugh et al., 2016), and a domain complex enough for sufﬁcient experiences/challenges to be generated in order to learn a suitable curricula.

Original: The Paired Open-Ended Trailblazer (POET) algorithm
Revised: The Paired Open-Ended Trailblazer (POET) algorithm

Original: (Wang et al., 2019), (Wang et al., 2020) is a recent example of an attempt to solve these challenges in the context of supervised RL.
Revised: (Wang et al., 2019), (Wang et al., 2020) is a recent example of an attempt to solve these challenges in the context of supervised RL.

Original: POET uses the Minimal Criterion Coevolution (MCC) algorithm (Brant & Stanley, 2017) to co-evolve increasingly complex environments and agents and then directly optimizes an RL objective either with evolution strategies (ES) (Salimans et al., 2017) or proximal policy optimization (PPO)
Revised: POET uses the Minimal Criterion Coevolution (MCC) algorithm (Brant & Stanley, 2017) to co-evolve increasingly complex environments and agents and then directly optimizes an RL objective either with evolution strategies (ES) (Salimans et al., 2017) or proximal policy optimization (PPO)

Original: (Schulman et al., 2017).
Revised: (Schulman et al., 2017).

Original: Within POET both the MCC for suitable learning outcomes and agent optimization rely upon the extrinsic reward within the environment.
Revised: Within POET both the MCC for suitable learning outcomes and agent optimization rely upon the extrinsic reward within the environment.

Original: This approach works well for the reward-dense bipedal walker control problem (Brockman et al., 2016) but can often become challenging, even in conceivably simple RL problems, to generate desired outcomes (Vecerik et al., 2018).
Revised: This approach works well for the reward-dense bipedal walker control problem (Brockman et al., 2016) but can often become challenging, even in conceivably simple RL problems, to generate desired outcomes (Vecerik et al., 2018).

Original: This problem becomes more challenging when the reward is sparse, especially with a long-horizon, where the agent effectively has no reward until termination.
Revised: This problem becomes more challenging when the reward is sparse, especially with a long-horizon, where the agent effectively has no reward until termination.

Original: Reward-shaping can be used (Ng et al., 1999) to help improve the density of reward and aide exploration.
Revised: Reward-shaping can be used (Ng et al., 1999) to help improve the density of reward and aide exploration.

Original: However, reward-shaping does not encourage open-ended learning, because it rewards the agent for following a hand-designed objective driven path, which limits open-endedness and often leads to poor performance (Lehman &
Revised: However, reward-shaping does not encourage open-ended learning, because it rewards the agent for following a hand-designed objective driven path, which limits open-endedness and often leads to poor performance (Lehman &

Original: Stanley, 2011b).
Revised: Stanley, 2011b).

Original: An alternate approach is found in the ﬁeld of unsupervised RL where we don’t use an extrinsic reward function at all.
Revised: An alternate approach is found in the ﬁeld of unsupervised RL where we don’t use an extrinsic reward function at all.

Original: Instead the objective is to learn a group of behaviors and hope to observe the agent solving task through one of our explorations, an approach that shares some characteristics with the problem of novelty search (NS) in open-ended learning (Lehman & Stanley, 2011a).
Revised: Instead the objective is to learn a group of behaviors and hope to observe the agent solving task through one of our explorations, an approach that shares some characteristics with the problem of novelty search (NS) in open-ended learning (Lehman & Stanley, 2011a).

Original: The acquisition of a diverse set of useful skills without extrinsic reward is in itself no-easy task.
Revised: The acquisition of a diverse set of useful skills without extrinsic reward is in itself no-easy task.

Original: We consider a skill as a latent-variable conditioned policy that alters the state of the environment consistently (Gregor et al., 2016), (Eysenbach et al., 2018).
Revised: We consider a skill as a latent-variable conditioned policy that alters the state of the environment consistently (Gregor et al., 2016), (Eysenbach et al., 2018).

Original: The utility of a skill is typically based on its diversity to another, for example whilst one skill might be standing still, walking, dancing and running are all vastly different and a good algorithm will aim to learn the entire set of skills.
Revised: The utility of a skill is typically based on its diversity to another, for example whilst one skill might be standing still, walking, dancing and running are all vastly different and a good algorithm will aim to learn the entire set of skills.

Original: Skills are then often selected for a task speciﬁc problem either by imitation learning or by directly selecting an encoding on the latent-variable conditioned policy.
Revised: Skills are then often selected for a task speciﬁc problem either by imitation learning or by directly selecting an encoding on the latent-variable conditioned policy.

Original: The learned behaviors can also be used as a pre-conditioned policy for regular RL with rewards similar to pre-training on ImageNet (Deng et al., 2009)
Revised: The learned behaviors can also be used as a pre-conditioned policy for regular RL with rewards similar to pre-training on ImageNet (Deng et al., 2009)

Original: In this paper we propose Rewardless Open-Ended Learning (ROEL) an unsupervised deep RL algorithm based upon the POET framework.
Revised: In this paper we propose Rewardless Open-Ended Learning (ROEL) an unsupervised deep RL algorithm based upon the POET framework.

Original: The key-idea of this work is to use constant environment mutation to generate a diverse set of skills capable of solving increasingly complex and novel problems.
Revised: The key-idea of this work is to use constant environment mutation to generate a diverse set of skills capable of solving increasingly complex and novel problems.

Original: We hypothesize this combination of open-ended learning with unsupervised RL will generate sets of behaviors that are more diverse and general than unsupervised RL agents simply trained on a static environment.
Revised: We hypothesize this combination of open-ended learning with unsupervised RL will generate sets of behaviors that are more diverse and general than unsupervised RL agents simply trained on a static environment.

Original: Behaviors can then be selected for a speciﬁc task by directly selecting from a latent encoding, or by comparing the state-space transition to a demonstration.
Revised: Behaviors can then be selected for a speciﬁc task by directly selecting from a latent encoding, or by comparing the state-space transition to a demonstration.

Original: Then, if a skill can only be demoed on a single environment, our environment generalist approach ensures this type of behavior will generalize across many environments.
Revised: Then, if a skill can only be demoed on a single environment, our environment generalist approach ensures this type of behavior will generalize across many environments.

Original: Our paper makes the following contributions:
Revised: Our paper makes the following contributions:

Original: • We present a new algorithm, ROEL, that combines unsupervised RL with open-ended learning to generate diverse and general behaviors (Section 3).
Revised: • We present a new algorithm, ROEL, that combines unsupervised RL with open-ended learning to generate diverse and general behaviors (Section 3).

Original: An approach that has not been attempted before with agent-environment co-evolution (Section 2).
Revised: An approach that has not been attempted before with agent-environment co-evolution (Section 2).

Original: • We train an unsupervised learning algorithm on the bipedal-walker environment.
Revised: • We train an unsupervised learning algorithm on the bipedal-walker environment.

Original: An environment with a difﬁcult task objective and a non-stationary environment transition.
Revised: An environment with a difﬁcult task objective and a non-stationary environment transition.

Original: • We prove empirically, that our approach is capable of learning more general and diverse groups of policies than single environments, especially when deployed to environments with dynamics that differ from the training set (Section 5).
Revised: • We prove empirically, that our approach is capable of learning more general and diverse groups of policies than single environments, especially when deployed to environments with dynamics that differ from the training set (Section 5).

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Central to recent open-ended learning solutions is quality diversity (QD), based on NS, (Lehman & Stanley, 2011a), (Lehman & Stanley, 2011b)
Revised: Central to recent open-ended learning solutions is quality diversity (QD), based on NS, (Lehman & Stanley, 2011a), (Lehman & Stanley, 2011b)

Original: QD algorithms aim to return a diverse set of highquality solutions by carefully balancing diversity and optimization (Pugh et al., 2016), (Mouret &
Revised: QD algorithms aim to return a diverse set of highquality solutions by carefully balancing diversity and optimization (Pugh et al., 2016), (Mouret &

Original: Clune, 2015).
Revised: Clune, 2015).

Original: In traditionally hard RL exploration problems Ecoffet et al.
Revised: In traditionally hard RL exploration problems Ecoffet et al.

Original: (2021) produced signiﬁcant performance improvements by maintaining an archive of stepping stones from state to state.
Revised: (2021) produced signiﬁcant performance improvements by maintaining an archive of stepping stones from state to state.

Original: QD itself however does not necessarily provide open-ended learning, most approaches for openended learning rely upon constantly generating new challenges for the agent to overcome.
Revised: QD itself however does not necessarily provide open-ended learning, most approaches for openended learning rely upon constantly generating new challenges for the agent to overcome.

Original: This can be through co-evolution (de Jong & Pollack, 2004), with an adversarial agent (Florensa et al., 2018a) or simply the agent itself (Team et al., 2021).
Revised: This can be through co-evolution (de Jong & Pollack, 2004), with an adversarial agent (Florensa et al., 2018a) or simply the agent itself (Team et al., 2021).

Original: In a similar vein research into self-generated curricula aims to automatically generate a sequence of objectives that efﬁciently allow the agent to learn complex behaviors.
Revised: In a similar vein research into self-generated curricula aims to automatically generate a sequence of objectives that efﬁciently allow the agent to learn complex behaviors.

Original: These approaches include: reverse goal-oriented curriculum generation (Florensa et al., 2018b), teacher-student curriculum learning (Matiisen et al., 2017), and procedural content generation (Justesen et al., 2018), (Hafner, 2021).
Revised: These approaches include: reverse goal-oriented curriculum generation (Florensa et al., 2018b), teacher-student curriculum learning (Matiisen et al., 2017), and procedural content generation (Justesen et al., 2018), (Hafner, 2021).

Original: Another approach, world-agent based co-evolution , aims to constantly evolve the environment with the agent to generate pressure for agents to gather new skills.
Revised: Another approach, world-agent based co-evolution , aims to constantly evolve the environment with the agent to generate pressure for agents to gather new skills.

Original: To facilitate open-ended learning we make use of a similar framework to POET (Wang et al., 2019)
Revised: To facilitate open-ended learning we make use of a similar framework to POET (Wang et al., 2019)

Original: (Wang et al., 2020) where environment-agent pairs are evolved based on their QD using insights from the minimal criterion coevolution algorithm (MCC) (Brant & Stanley, 2017).
Revised: (Wang et al., 2020) where environment-agent pairs are evolved based on their QD using insights from the minimal criterion coevolution algorithm (MCC) (Brant & Stanley, 2017).

Original: The MCC performance ranking of agents in
Revised: The MCC performance ranking of agents in

Original: POET is based upon the agents total average episodic return from the extrinsic reward.
Revised: POET is based upon the agents total average episodic return from the extrinsic reward.

Original: Our approach extends this problem to the unsupervised domain, allowing for the discovery of diverse and useful sets of behaviors.
Revised: Our approach extends this problem to the unsupervised domain, allowing for the discovery of diverse and useful sets of behaviors.

Original: PAIRED (Dennis et al., 2021)
Revised: PAIRED (Dennis et al., 2021)

Original: co-evolves 2 agents, a protagonist and antagonist where the protagonist aims to solve tasks generated by the antagonist.
Revised: co-evolves 2 agents, a protagonist and antagonist where the protagonist aims to solve tasks generated by the antagonist.

Original: Regret is then deﬁned as the average score of the protagonist and the best score of the antagonist over multiple trials.
Revised: Regret is then deﬁned as the average score of the protagonist and the best score of the antagonist over multiple trials.

Original: A separate adversary agent is then also trained, with the objective to minimax regret between the 2 agents by automatic environment generation.
Revised: A separate adversary agent is then also trained, with the objective to minimax regret between the 2 agents by automatic environment generation.

Original: In contrast POET effectively maintains a population of minimax agents.
Revised: In contrast POET effectively maintains a population of minimax agents.

Original: We found the POET framework to be a suitable procedure to design a rewardless open ended learning algorithm.
Revised: We found the POET framework to be a suitable procedure to design a rewardless open ended learning algorithm.

Original: The results in PAIRED are inconclusive as to whether POETs population QD approach outperforms PAIREDs adversarial approach, we leave this investigation to future work.
Revised: The results in PAIRED are inconclusive as to whether POETs population QD approach outperforms PAIREDs adversarial approach, we leave this investigation to future work.

Original: Our methodology aims to extend open-ended learning to the unsupervised RL setting, in an effort to generate increasingly sophisticated predictable and diverse behaviors.
Revised: Our methodology aims to extend open-ended learning to the unsupervised RL setting, in an effort to generate increasingly sophisticated predictable and diverse behaviors.

Original: Signiﬁcant previous work in ﬁxed environment unsupervised RL has focused on intrinsic motivation (Oudeyer, 2007), (Oudeyer et al., 2007), (Chentanez et al., 2005), with examples including: empowerment (Klyubin et al., 2005), (Mohamed & Rezende, 2015a), state count based exploration (Bellemare et al., 2016), curiosity driven exploration (Pathak et al., 2017), surprise maximization (Achiam & Sastry, 2017) and minimization (Berseth et al., 2021).
Revised: Signiﬁcant previous work in ﬁxed environment unsupervised RL has focused on intrinsic motivation (Oudeyer, 2007), (Oudeyer et al., 2007), (Chentanez et al., 2005), with examples including: empowerment (Klyubin et al., 2005), (Mohamed & Rezende, 2015a), state count based exploration (Bellemare et al., 2016), curiosity driven exploration (Pathak et al., 2017), surprise maximization (Achiam & Sastry, 2017) and minimization (Berseth et al., 2021).

Original: Compared to intrinsic motivation skill discovery aims to explore a group of skills that can then be exploited by imitation learning (Schaal, 1997), inverse reinforcement learning (Russell, 1998), or ﬁne-tuning.
Revised: Compared to intrinsic motivation skill discovery aims to explore a group of skills that can then be exploited by imitation learning (Schaal, 1997), inverse reinforcement learning (Russell, 1998), or ﬁne-tuning.

Original: Skill discovery has generally made use of relations between RL and information theory (Ziebart et al., 2008), (Florensa et al., 2017), (Daniel et al., 2012), (Eysenbach et al., 2018), (Wang et al., 2018), (Sharma et al., 2020).
Revised: Skill discovery has generally made use of relations between RL and information theory (Ziebart et al., 2008), (Florensa et al., 2017), (Daniel et al., 2012), (Eysenbach et al., 2018), (Wang et al., 2018), (Sharma et al., 2020).

Original: DIAYN (Eysenbach et al., 2018) explicitly aimed to exploit HRL by designing a reward function to maximize stateentropy, so that skills are differentiated by visiting different state-space regions.
Revised: DIAYN (Eysenbach et al., 2018) explicitly aimed to exploit HRL by designing a reward function to maximize stateentropy, so that skills are differentiated by visiting different state-space regions.

Original: DADS (Sharma et al., 2020) subsequently aimed to discover skills that are both diverse and predictable .
Revised: DADS (Sharma et al., 2020) subsequently aimed to discover skills that are both diverse and predictable .

Original: Our work then develops the methodology in DADS to train agents over a range of increasingly complex environments, creating unending environmental pressure for agents to evolve ever more complex and diverse skills.
Revised: Our work then develops the methodology in DADS to train agents over a range of increasingly complex environments, creating unending environmental pressure for agents to evolve ever more complex and diverse skills.

Original: Gupta et al.
Revised: Gupta et al.

Original: (2020) provides another interesting approach to the problem of generalized unsupervised reinforcement learning by combining meta-learning (Finn et al., 2017) with DIAYN (Eysenbach et al., 2018) to automatically generate meta-training tasks.
Revised: (2020) provides another interesting approach to the problem of generalized unsupervised reinforcement learning by combining meta-learning (Finn et al., 2017) with DIAYN (Eysenbach et al., 2018) to automatically generate meta-training tasks.

Original: This is a useful approach for the ﬁxed environment setting, but does not address the challenge of open-ended learning, only meta-training on a static state-transition distribution.
Revised: This is a useful approach for the ﬁxed environment setting, but does not address the challenge of open-ended learning, only meta-training on a static state-transition distribution.

Original:  R EWARDLESS O PEN -E NDED L EARNING (ROEL)
Revised:  R EWARDLESS O PEN -E NDED L EARNING (ROEL)

Original: Building on previous work in unsupervised RL and mutual-information based exploration (Eysenbach et al., 2018), (Sharma et al., 2020), ROEL uses the POET framework (Wang et al., 2019) to endlessly generate agents capable of exhibiting increasingly complex, diverse and predictable behaviors.
Revised: Building on previous work in unsupervised RL and mutual-information based exploration (Eysenbach et al., 2018), (Sharma et al., 2020), ROEL uses the POET framework (Wang et al., 2019) to endlessly generate agents capable of exhibiting increasingly complex, diverse and predictable behaviors.

Original: The policies learnt by ROEL can then be exploited to maximize a task speciﬁc objective using some form of supervision.
Revised: The policies learnt by ROEL can then be exploited to maximize a task speciﬁc objective using some form of supervision.

Original: Importantly our algorithms use of non-stationary environmental representations allows for the discovery of new emergent behaviors based on previous simpler behaviors.
Revised: Importantly our algorithms use of non-stationary environmental representations allows for the discovery of new emergent behaviors based on previous simpler behaviors.

Original: 3. POET FRAMEWORK
Revised: 3. POET FRAMEWORK

Original: We use POET as a framework to endlessly discover agents with increasingly complex novel behaviors.
Revised: We use POET as a framework to endlessly discover agents with increasingly complex novel behaviors.

Original: POET accomplishes this through growing and optimizing a paired population of agents and unique environments (ea-pairs), contained within an environment-action list (ea-list).
Revised: POET accomplishes this through growing and optimizing a paired population of agents and unique environments (ea-pairs), contained within an environment-action list (ea-list).

Original: To control the evolution of the ea-list each environment is represented by an environmental encoding (EE), which maps environment attributes to an environment representation, and can easily be encoded and mutated.
Revised: To control the evolution of the ea-list each environment is represented by an environmental encoding (EE), which maps environment attributes to an environment representation, and can easily be encoded and mutated.

Original: The Environmental Characterization (EC) is then used to describe the qualities of that environment, in ROEL we use the domain-general Performance of All Transferred Agents (PATA-EC) from (Wang et al., 2020).
Revised: The Environmental Characterization (EC) is then used to describe the qualities of that environment, in ROEL we use the domain-general Performance of All Transferred Agents (PATA-EC) from (Wang et al., 2020).

Original: The procedure to generate and search for behaviors within POETs main loop can then be summarized in the following steps:
Revised: The procedure to generate and search for behaviors within POETs main loop can then be summarized in the following steps:

Original: Mutate : Every M steps, provided an ea-pair exhibits suitable performance, it is mutated to a generate new child EEs.
Revised: Mutate : Every M steps, provided an ea-pair exhibits suitable performance, it is mutated to a generate new child EEs.

Original: The children are then ﬁltered via a minimal criterion (MC) (Brant & Stanley, 2017) to ensure the environment is not too easy or hard.
Revised: The children are then ﬁltered via a minimal criterion (MC) (Brant & Stanley, 2017) to ensure the environment is not too easy or hard.

Original: EEs that satisfy the MC are then ranked based on the performance of the best agents and the most novel EEs added to the active population.
Revised: EEs that satisfy the MC are then ranked based on the performance of the best agents and the most novel EEs added to the active population.

Original: When the active population reaches a predeﬁned size the least novel pairs are dropped from the active environment and added to a set of archived environments.
Revised: When the active population reaches a predeﬁned size the least novel pairs are dropped from the active environment and added to a set of archived environments.

Original: Novelty of newly-generated environments is based on how different the ranking of agents trained on the new environment changes compared to the ranking of agents on old environments, based on the idea that useful challenges should make novel distinctions amongst agents in a system (de Jong & Pollack, 2004).
Revised: Novelty of newly-generated environments is based on how different the ranking of agents trained on the new environment changes compared to the ranking of agents on old environments, based on the idea that useful challenges should make novel distinctions amongst agents in a system (de Jong & Pollack, 2004).

Original: Optimize : Every step POET optimizes every agent to maximize its reward on the given environment.
Revised: Optimize : Every step POET optimizes every agent to maximize its reward on the given environment.

Original: In the original POET architecture this is the extrinsic reward from a Markov decision process, optimized with the Evolution Strategies (ES) algorithm (Salimans et al., 2017).
Revised: In the original POET architecture this is the extrinsic reward from a Markov decision process, optimized with the Evolution Strategies (ES) algorithm (Salimans et al., 2017).

Original: In ROEL we optimize the mutual-information based reward as an RL optimization problem using Soft-Actor Critic (Haarnoja et al., 2018a).
Revised: In ROEL we optimize the mutual-information based reward as an RL optimization problem using Soft-Actor Critic (Haarnoja et al., 2018a).

Original: Transfer : Every N steps of POET’s outer loop, the performance of all active agents is compared and ranked.
Revised: Transfer : Every N steps of POET’s outer loop, the performance of all active agents is compared and ranked.

Original: If an agent on a different environment outperforms the current agent then the agent is replaced with the more performant agent.
Revised: If an agent on a different environment outperforms the current agent then the agent is replaced with the more performant agent.

Original: This constant transfer proved critical in POETs original implementation (Wang et al., 2019) for discovering stepping stones to more performant behaviors and to drive useful open-ended learning.
Revised: This constant transfer proved critical in POETs original implementation (Wang et al., 2019) for discovering stepping stones to more performant behaviors and to drive useful open-ended learning.

Original: 3. R EWARDING ROEL
Revised: 3. R EWARDING ROEL

Original: ROEL aims to create agents that exhibit a diverse set of predictable and useful behaviors across a range of problems.
Revised: ROEL aims to create agents that exhibit a diverse set of predictable and useful behaviors across a range of problems.

Original: POET uses the extrinsic return generated from the agents interaction with the environment to generate an objective to minimize, as a supervised learning problem.
Revised: POET uses the extrinsic return generated from the agents interaction with the environment to generate an objective to minimize, as a supervised learning problem.

Original: The return of each ea-pair in POET is key to calculate each EEs PATA-EC and decide how to evolve the ea-pair.
Revised: The return of each ea-pair in POET is key to calculate each EEs PATA-EC and decide how to evolve the ea-pair.

Original: Subsequently in ROEL we use mutual information I to construct a reward function that aims to drive open-ended learning and motivate exploration, similar to Houthooft et al.
Revised: Subsequently in ROEL we use mutual information I to construct a reward function that aims to drive open-ended learning and motivate exploration, similar to Houthooft et al.

Original: and Mohamed &
Revised: and Mohamed &

Original: Rezende (2015b).
Revised: Rezende (2015b).

Original: [Equation]
Revised: [Equation]

Original: Following the deﬁnition of mutual information, shown in equation 1.
Revised: Following the deﬁnition of mutual information, shown in equation 1.

Original: Maximizing I with respect to
Revised: Maximizing I with respect to

Original: Y is equivalent to maximizing the entropy H ( X ) whilst minimizing conditional entropy H ( X | Y ) , where X and Y are functions of state and actions respectively.
Revised: Y is equivalent to maximizing the entropy H ( X ) whilst minimizing conditional entropy H ( X | Y ) , where X and Y are functions of state and actions respectively.

Original: This objective therefore aims to ﬁnd actions to maximize state entropy, and is therefore exploratory as an RL policy.
Revised: This objective therefore aims to ﬁnd actions to maximize state entropy, and is therefore exploratory as an RL policy.

Original: This property was recently used by Achiam et al.
Revised: This property was recently used by Achiam et al.

Original: (2018), Eysenbach et al.
Revised: (2018), Eysenbach et al.

Original: (2018), Gregor et al.
Revised: (2018), Gregor et al.

Original: (2016) and Sharma et al.
Revised: (2016) and Sharma et al.

Original: (2020) to learn a diverse and distinct set of skills in a policy.
Revised: (2020) to learn a diverse and distinct set of skills in a policy.

Original: In ROEL we make use of the reward function speciﬁed by DADS (Sharma et al., 2020) to learn a policy conditioned on a latent variable z , sampled from the transition dynamics prior p ( z ) ∼ Z .
Revised: In ROEL we make use of the reward function speciﬁed by DADS (Sharma et al., 2020) to learn a policy conditioned on a latent variable z , sampled from the transition dynamics prior p ( z ) ∼ Z .

Original: DADS differs from previous conditional entropy based approaches through the insight to leverage work in model based reinforcement learning (MBRL), where an objective is to learn a model to estimate the dynamics of the system
Revised: DADS differs from previous conditional entropy based approaches through the insight to leverage work in model based reinforcement learning (MBRL), where an objective is to learn a model to estimate the dynamics of the system

Original: ˆ p ( s (cid:48) | s, a ) .
Revised: ˆ p ( s (cid:48) | s, a ) .

Original: Therefore DADS learns 2 functions:
Revised: Therefore DADS learns 2 functions:

Original: • A skill-conditioned policy π ( a | s ; z ) where skills are drawn from the larger skill set such that z ∈ Z .
Revised: • A skill-conditioned policy π ( a | s ; z ) where skills are drawn from the larger skill set such that z ∈ Z .

Original: • Skill-dynamics :
Revised: • Skill-dynamics :

Original: A skill-conditioned transition function q φ ( s (cid:48) | s ; z ) , that predicts the state transition given a skill z .
Revised: A skill-conditioned transition function q φ ( s (cid:48) | s ; z ) , that predicts the state transition given a skill z .

Original: [Equation]
Revised: [Equation]

Original: DADS then uses the mutual-information between the next state with respect to a skill z , given the current state, to quantify how informative a skill z is of the transition s (cid:48)
Revised: DADS then uses the mutual-information between the next state with respect to a skill z , given the current state, to quantify how informative a skill z is of the transition s (cid:48)

Original: → s , this is shown in equation 2.
Revised: → s , this is shown in equation 2.

Original: Using a suitable approximation and the deﬁnition of conditional mutual information a reward function can be deﬁned, as shown in equation 3 (full derivation found in Sharma et al.
Revised: Using a suitable approximation and the deﬁnition of conditional mutual information a reward function can be deﬁned, as shown in equation 3 (full derivation found in Sharma et al.

Original: [Equation]
Revised: [Equation]

Original: The reward r z in equation 3 encourages agents to exhibit skill diversity by sampling L skills from the prior z i ∼ p
Revised: The reward r z in equation 3 encourages agents to exhibit skill diversity by sampling L skills from the prior z i ∼ p

Original: ( z ) , to be used with the skill dynamics estimate q φ .
Revised: ( z ) , to be used with the skill dynamics estimate q φ .

Original: We then use the return generated over several episodes to generate a statistic for the average return, that can be used to mutate and transfer agents within ROEL under the POET framework.
Revised: We then use the return generated over several episodes to generate a statistic for the average return, that can be used to mutate and transfer agents within ROEL under the POET framework.

Original: This approach introduces non-stationarity into the skill transition and offers the following distinct advantages over simply training on a static environment.
Revised: This approach introduces non-stationarity into the skill transition and offers the following distinct advantages over simply training on a static environment.

Original: ROEL learns an open-ended diverse set of skills
Revised: ROEL learns an open-ended diverse set of skills

Original: The combination of mutation and transfer learning was the key to POET learning increasingly complex skills in a supervised setting.
Revised: The combination of mutation and transfer learning was the key to POET learning increasingly complex skills in a supervised setting.

Original: For ROEL this procedure of gradually improving novelty via increased environmental complexity allows for simple, foundational, skills to be uncovered early in training.
Revised: For ROEL this procedure of gradually improving novelty via increased environmental complexity allows for simple, foundational, skills to be uncovered early in training.

Original: These are then used by ROEL as stepping stones to more complex skills when the challenge of more complex environments are presented (which in turn provide the opportunity to learn more complex behaviors).
Revised: These are then used by ROEL as stepping stones to more complex skills when the challenge of more complex environments are presented (which in turn provide the opportunity to learn more complex behaviors).

Original: Generally this has the effect of improving the diversity of the set of skills uncovered as there is simply a larger distribution of possible state trajectories in more complex environments.
Revised: Generally this has the effect of improving the diversity of the set of skills uncovered as there is simply a larger distribution of possible state trajectories in more complex environments.

Original: But, designing building blocks to transfer the set of skills to the agent is key to efﬁciently discovering a large set of skills.
Revised: But, designing building blocks to transfer the set of skills to the agent is key to efﬁciently discovering a large set of skills.

Original: ROELs outer loop searches for environments that allow access to greater intrinsic reward, balancing novelty over performance via an MCC algorithm.
Revised: ROELs outer loop searches for environments that allow access to greater intrinsic reward, balancing novelty over performance via an MCC algorithm.

Original: This is implemented with a PATA-EC (Performance of All Transferred Agents Environmental Characterization) ranking, described in Wang et al.
Revised: This is implemented with a PATA-EC (Performance of All Transferred Agents Environmental Characterization) ranking, described in Wang et al.

Original: (2020), where each agent is ranked on an environment based on its performance against all other environments.
Revised: (2020), where each agent is ranked on an environment based on its performance against all other environments.

Original: EEs that change the ordering of other agents PATA-EC are favoured as this implies that the EE provides a new set of challenges to the set of active agents.
Revised: EEs that change the ordering of other agents PATA-EC are favoured as this implies that the EE provides a new set of challenges to the set of active agents.

Original: To overcome this new challenge the agent will need to optimize different behaviors improving the diversity of skills learnt in the long run.
Revised: To overcome this new challenge the agent will need to optimize different behaviors improving the diversity of skills learnt in the long run.

Original: ROEL is able to generalize over a large group of environments
Revised: ROEL is able to generalize over a large group of environments

Original: The search for novel environments in ROELs outer loop can also serve to improve the generalization of ROEL at test-time.
Revised: The search for novel environments in ROELs outer loop can also serve to improve the generalization of ROEL at test-time.

Original: By evolving EEs to generate novel behaviors ROEL is able to sample a large distribution of possible environments, but unlike domain randomization this search is directed to ﬁnd environments that explicitly cause a change in behavior.
Revised: By evolving EEs to generate novel behaviors ROEL is able to sample a large distribution of possible environments, but unlike domain randomization this search is directed to ﬁnd environments that explicitly cause a change in behavior.

Original: This allows for ROEL to replicate the skills it learns across a much broader range of environments than if we were to simply train on a single-static environment.
Revised: This allows for ROEL to replicate the skills it learns across a much broader range of environments than if we were to simply train on a single-static environment.

Original: We explore this characteristic further in section 5.2 by comparing the performance of ROEL to DADS when tested on a hard EE unobserved at train-time.
Revised: We explore this characteristic further in section 5.2 by comparing the performance of ROEL to DADS when tested on a hard EE unobserved at train-time.

Original:  E XPERIMENT S ETUP
Revised:  E XPERIMENT S ETUP

Original: Using an implementation similar to DADS (Sharma et al., 2020), we use Soft Actor-Critic (SAC) (Haarnoja et al., 2019), an off-policy actor-critic deep RL algorithm, speciﬁcally EC-SAC, to train our policy π ( a | s, z ) .
Revised: Using an implementation similar to DADS (Sharma et al., 2020), we use Soft Actor-Critic (SAC) (Haarnoja et al., 2019), an off-policy actor-critic deep RL algorithm, speciﬁcally EC-SAC, to train our policy π ( a | s, z ) .

Original: SAC (Haarnoja et al., 2018b) is based on the maximum entropy framework and uses entropy regularization to tradeoff robustness for exploration, as policies are encouraged to discover action-sequences with similar trajectories under a given skill.
Revised: SAC (Haarnoja et al., 2018b) is based on the maximum entropy framework and uses entropy regularization to tradeoff robustness for exploration, as policies are encouraged to discover action-sequences with similar trajectories under a given skill.

Original: To test ROEL we use the bipedal control environment (Brockman et al., 2016) with the custom EE extension described in Wang et al.
Revised: To test ROEL we use the bipedal control environment (Brockman et al., 2016) with the custom EE extension described in Wang et al.

Original: (2019), allowing us to mutate the environment by varying a set of hand-picked obstacle hyper-parameters, such as stump shape, surface-roughness and gap width.
Revised: (2019), allowing us to mutate the environment by varying a set of hand-picked obstacle hyper-parameters, such as stump shape, surface-roughness and gap width.

Original: This encoding offers sufﬁcient opportunities for ROEL to discover diverse behaviors and is described in more detail in Appendix B.
Revised: This encoding offers sufﬁcient opportunities for ROEL to discover diverse behaviors and is described in more detail in Appendix B.

Original: In future work we would like to investigate how much improving the
Revised: In future work we would like to investigate how much improving the

Original: EEs expressiveness effects ROELs performance by using CPPNs (Compositional Pattern Producing
Revised: EEs expressiveness effects ROELs performance by using CPPNs (Compositional Pattern Producing

Original: Networks) (Stanley, 2007), similar to Wang et al.
Revised: Networks) (Stanley, 2007), similar to Wang et al.

Original: For the experiments in section 5 we trained ROEL over 24 iterations (5400 episodic iterations) using a total of 224 processing cores distributed over 8 nodes (each with dual socket CPUs) for ≈ 12 hours.
Revised: For the experiments in section 5 we trained ROEL over 24 iterations (5400 episodic iterations) using a total of 224 processing cores distributed over 8 nodes (each with dual socket CPUs) for ≈ 12 hours.

Original: ROEL is highly parallel allowing for each ea-pair to be run asynchronously during the training step.
Revised: ROEL is highly parallel allowing for each ea-pair to be run asynchronously during the training step.

Original: Our baseline DADS environment is trained using the same parameters as the ROEL inner-loop, using the original bipedal normal environment.
Revised: Our baseline DADS environment is trained using the same parameters as the ROEL inner-loop, using the original bipedal normal environment.

Original: We ﬁnd that ROEL takes slightly longer to train than DADS ( ≈ 9 hours) as there is no need to calculate the PATA-EC and minimal criterion at the mutation and transfer steps.
Revised: We ﬁnd that ROEL takes slightly longer to train than DADS ( ≈ 9 hours) as there is no need to calculate the PATA-EC and minimal criterion at the mutation and transfer steps.

Original: We also train a domain randomized (DR) DADS baseline on the parameters contained within the ROEL EC to test the capabilities of ROELs auto-curricula to generalize to unseen environments.
Revised: We also train a domain randomized (DR) DADS baseline on the parameters contained within the ROEL EC to test the capabilities of ROELs auto-curricula to generalize to unseen environments.

Original: In our current implementation these steps are implemented synchronously and consequently cause a signiﬁcant slow-down.

Original: We provide further details of our speciﬁc training-evaluation architecture and hyperparameters in Appendix A.
Revised: We provide further details of our speciﬁc training-evaluation architecture and hyperparameters in Appendix A.

Original:  R ESULTS
Revised:  R ESULTS

Original: In this section, we present our results from using ROEL with the bipedal walker environment.
Revised: In this section, we present our results from using ROEL with the bipedal walker environment.

Original: Initially we qualitatively evaluate the types of skills learnt by ROEL and compare ROELs in and outof-sample performance when used on a ﬁxed skill z .
Revised: Initially we qualitatively evaluate the types of skills learnt by ROEL and compare ROELs in and outof-sample performance when used on a ﬁxed skill z .

Original: We then quantitatively evaluate ROEL using
Revised: We then quantitatively evaluate ROEL using

Original: DADS as a baseline, aiming to assess our initial hypothesis that ROELs use of open-endedness allows it to learn skills that are more diverse and general than unsupervised RL, trained on a single environment.
Revised: DADS as a baseline, aiming to assess our initial hypothesis that ROELs use of open-endedness allows it to learn skills that are more diverse and general than unsupervised RL, trained on a single environment.

Original: 5. Q UALITATIVE ANALYSIS
Revised: 5. Q UALITATIVE ANALYSIS

Original: Figure 1 shows 7 of the 20 skills learnt by ROEL hand selected from a single run.
Revised: Figure 1 shows 7 of the 20 skills learnt by ROEL hand selected from a single run.

Original: Some of the skills are clearly motivated by the environments ROEL is exposed to during training.
Revised: Some of the skills are clearly motivated by the environments ROEL is exposed to during training.

Original: For example the shufﬂing forward skill allows the agent to move over pits generated in the training environment and keeps the LIDAR sensor close to the ground, helping to improve sensor accuracy.
Revised: For example the shufﬂing forward skill allows the agent to move over pits generated in the training environment and keeps the LIDAR sensor close to the ground, helping to improve sensor accuracy.

Original: The leap skill likely improves the intrinsic return when hills are present in the environment.
Revised: The leap skill likely improves the intrinsic return when hills are present in the environment.

Original: The one-leg and perching skills may not immediately appear useful, but could prove key stepping stones to learn more complex modes of locomotion.
Revised: The one-leg and perching skills may not immediately appear useful, but could prove key stepping stones to learn more complex modes of locomotion.

Original: For example the one-leg prior has learnt how to use the ungrounded leg to maintain balance for a short period.
Revised: For example the one-leg prior has learnt how to use the ungrounded leg to maintain balance for a short period.

Original: Figure 2 demonstrates how ROELs behavior differs from DADS in out-of-sample environments, using an EE containing gaps and roughness different from that seen by any agent at during training.
Revised: Figure 2 demonstrates how ROELs behavior differs from DADS in out-of-sample environments, using an EE containing gaps and roughness different from that seen by any agent at during training.

Original: When the ROEL agent in ﬁgure 2 crosses the pit it rapidly places a leg on the other side and pushes itself over the hole, it is then able to move its leg up to pull itself over the hill.
Revised: When the ROEL agent in ﬁgure 2 crosses the pit it rapidly places a leg on the other side and pushes itself over the hole, it is then able to move its leg up to pull itself over the hill.

Original: The DADS agent on the other hand stops upon reaching the pit and explores the hole with its legs ground sensor, the agent then slowly shufﬂes over the hole, it never gets over the hill in the time horizon.
Revised: The DADS agent on the other hand stops upon reaching the pit and explores the hole with its legs ground sensor, the agent then slowly shufﬂes over the hole, it never gets over the hill in the time horizon.

Original: We ﬁnd this is a good qualitative example of how ROEL is able to leverage experience gained through mutation and goal-switching to respond to out-of-sample challenges.
Revised: We ﬁnd this is a good qualitative example of how ROEL is able to leverage experience gained through mutation and goal-switching to respond to out-of-sample challenges.

Original: 5. Q UANTITATIVE ANALYSIS
Revised: 5. Q UANTITATIVE ANALYSIS

Original: To assess the ability of ROEL to perform out-of-sample we use the EE seen in ﬁgure 2 and the original environmental encoding to generate samples under each skill prior.
Revised: To assess the ability of ROEL to perform out-of-sample we use the EE seen in ﬁgure 2 and the original environmental encoding to generate samples under each skill prior.

Original: It is difﬁcult to directly measure the diversity of each agents prior based on samples from the environment.
Revised: It is difﬁcult to directly measure the diversity of each agents prior based on samples from the environment.

Original: Observing diversity from the agents interaction with the environment alone requires us to characterize each with a suitable statistic.
Revised: Observing diversity from the agents interaction with the environment alone requires us to characterize each with a suitable statistic.

Original: If the environment itself features stationary agentsenvironment dynamics, the variance of the trajectories can be used to measure behavioral diversity as the behavior is time-invariant.
Revised: If the environment itself features stationary agentsenvironment dynamics, the variance of the trajectories can be used to measure behavioral diversity as the behavior is time-invariant.

Original: This is the case in Mujoco (Todorov et al., 2012) where the environment is a ﬂat-plane: DADS therefore compares novelty via the agents state-transition.
Revised: This is the case in Mujoco (Todorov et al., 2012) where the environment is a ﬂat-plane: DADS therefore compares novelty via the agents state-transition.

Original: With bipedal, we cannot make this assumption, because the terrain is non-uniform and the agent-environment dynamics change with time.
Revised: With bipedal, we cannot make this assumption, because the terrain is non-uniform and the agent-environment dynamics change with time.

Original: For ROEL, this shift in dynamics is also made intentionally via our constant evolution of the EE.
Revised: For ROEL, this shift in dynamics is also made intentionally via our constant evolution of the EE.

Original: An ideal statistic for ROEL would characterize each skills policy dynamics, independent of the environment based on observations alone.
Revised: An ideal statistic for ROEL would characterize each skills policy dynamics, independent of the environment based on observations alone.

Original: We ﬁnd that the state-space bayesian forecasting literature
Revised: We ﬁnd that the state-space bayesian forecasting literature

Original: West & Harrison (1997) does offer some interesting possible solutions to this problem but this is still challenging.
Revised: West & Harrison (1997) does offer some interesting possible solutions to this problem but this is still challenging.

Original: We would need to design a transfer function capable of capturing the skills fundamental dynamics but we ﬁnd this to be outside the scope of this paper and leave this future work.
Revised: We would need to design a transfer function capable of capturing the skills fundamental dynamics but we ﬁnd this to be outside the scope of this paper and leave this future work.

Original: Alternatively, we use the return (cumulative reward) from each episode to characterize each prior.
Revised: Alternatively, we use the return (cumulative reward) from each episode to characterize each prior.

Original: This is based on the intuition that if the reward function itself describes an example of a novel behavior, the difference in return of priors can be used as a heuristic to measure diversity.
Revised: This is based on the intuition that if the reward function itself describes an example of a novel behavior, the difference in return of priors can be used as a heuristic to measure diversity.

Original: In bipedal, the reward function is based on the ability of the agent to reach the end of the episode.
Revised: In bipedal, the reward function is based on the ability of the agent to reach the end of the episode.

Original: Figure 3 shows the ﬁtted distribution of returns from each prior, both for ROEL and DADS, on the in- and out-of-sample environments.
Revised: Figure 3 shows the ﬁtted distribution of returns from each prior, both for ROEL and DADS, on the in- and out-of-sample environments.

Original: To ﬁt the distribution we sample 100 episodes from each prior and calculate the empirical mean and variance of each to ﬁt a normal distribution.
Revised: To ﬁt the distribution we sample 100 episodes from each prior and calculate the empirical mean and variance of each to ﬁt a normal distribution.

Original: In-sample, the prior of each sampled mean is more dispersed than in DADS.
Revised: In-sample, the prior of each sampled mean is more dispersed than in DADS.

Original: This suggests greater skill diversity, helping to explain the behavior we qualitatively observe in ﬁgure 1.
Revised: This suggests greater skill diversity, helping to explain the behavior we qualitatively observe in ﬁgure 1.

Original: Out-of-sample, the return of each prior in ROEL is generally unchanged, whilst DADS features a large change in dispersion.
Revised: Out-of-sample, the return of each prior in ROEL is generally unchanged, whilst DADS features a large change in dispersion.

Original: This indicates that the skills learnt by ROEL generalize well to this new out-of-sample challenge and highlights the importance of providing a diverse set of environments during training if we want to learn a diverse set of skills.
Revised: This indicates that the skills learnt by ROEL generalize well to this new out-of-sample challenge and highlights the importance of providing a diverse set of environments during training if we want to learn a diverse set of skills.

Original: Another useful indicator of the capability of our agent to generalize to new environments is the error between the expected observation trajectory from the skill-dynamics network and the actual observation trajectory.
Revised: Another useful indicator of the capability of our agent to generalize to new environments is the error between the expected observation trajectory from the skill-dynamics network and the actual observation trajectory.

Original: If the skill-dynamics network is capable of accurately predicting the future observations, then the value of intrinsic motivation used to train agents will promote skills that are more diverse and predictable.
Revised: If the skill-dynamics network is capable of accurately predicting the future observations, then the value of intrinsic motivation used to train agents will promote skills that are more diverse and predictable.

Original: The implementation of probabilistic ensembles with trajectory sampling (PETS) in Chua et al.
Revised: The implementation of probabilistic ensembles with trajectory sampling (PETS) in Chua et al.

Original: (2018) suggests, that accurate state estimation is key for successful model based reinforcement learning, and adds a degree of conﬁdence in our approach.
Revised: (2018) suggests, that accurate state estimation is key for successful model based reinforcement learning, and adds a degree of conﬁdence in our approach.

Original: The accuracy of our skill-dynamic network is therefore a useful heuristic for the performance of ROEL on downstream tasks.
Revised: The accuracy of our skill-dynamic network is therefore a useful heuristic for the performance of ROEL on downstream tasks.

Original: Figure 4 plots the error in the prediction from the skill-dynamics network for both in- and out-ofsample EEs on the core hull dynamics.
Revised: Figure 4 plots the error in the prediction from the skill-dynamics network for both in- and out-ofsample EEs on the core hull dynamics.

Original: For our baseline, we instead used DADS trained with domain randomization (DR) across the same EE as ROEL.
Revised: For our baseline, we instead used DADS trained with domain randomization (DR) across the same EE as ROEL.

Original: This provides a more accurate comparison of how our auto-curricula based approach in ROEL compares to a naive DR approach (when training without DR the error of DADS out-of-sample diverges to 100 times that of ROEL due to over-ﬁtting).
Revised: This provides a more accurate comparison of how our auto-curricula based approach in ROEL compares to a naive DR approach (when training without DR the error of DADS out-of-sample diverges to 100 times that of ROEL due to over-ﬁtting).

Original: In-sample, we ﬁnd DADS outperforms ROEL consistently with a smaller mean and variance on all 3 parameters.
Revised: In-sample, we ﬁnd DADS outperforms ROEL consistently with a smaller mean and variance on all 3 parameters.

Original: Out-of-sample however, DADS often fails to predict the skill dynamics, rapidly diverging to an error 3 times that of ROEL in some cases whilst ROEL is generally unchanged.
Revised: Out-of-sample however, DADS often fails to predict the skill dynamics, rapidly diverging to an error 3 times that of ROEL in some cases whilst ROEL is generally unchanged.

Original: An exception to this is in the y-velocity , where the mean of ROEL and DADS-DR is generally the same, but DADS-DR has less variance.
Revised: An exception to this is in the y-velocity , where the mean of ROEL and DADS-DR is generally the same, but DADS-DR has less variance.

Original: The x-velocity state offers some of the most interesting results, showing DADS-DR rapidly diverging in both mean and variance, whilst the error in ROEL is minimal.
Revised: The x-velocity state offers some of the most interesting results, showing DADS-DR rapidly diverging in both mean and variance, whilst the error in ROEL is minimal.

Original: This difference can be explained by the differences in the DADS-DR and ROEL training procedures.
Revised: This difference can be explained by the differences in the DADS-DR and ROEL training procedures.

Original: In ROEL, the auto-curricula encourages longer episodes so that the behavior dynamics are longer.
Revised: In ROEL, the auto-curricula encourages longer episodes so that the behavior dynamics are longer.

Original: However, in DADS-DR, the environment selection is random and this means that the hardest obstacles often appear within the training environment so that the agent ﬁnds it difﬁcult to traverse and this causes the episode to terminate earlier.
Revised: However, in DADS-DR, the environment selection is random and this means that the hardest obstacles often appear within the training environment so that the agent ﬁnds it difﬁcult to traverse and this causes the episode to terminate earlier.

Original: This helps to explain why the x-velocity prediction is better, because ROEL is able to observe a more practical set of trajectories to learn generally diverse skills.
Revised: This helps to explain why the x-velocity prediction is better, be- cause ROEL is able to observe a more practical set of trajectories to learn generally diverse skills.

Original: Whilst the y-velocity prediction is relatively accurate for both ROEL and DADS-DR the reduced variance in ROEL is likely due to the environment observing a larger range of failures caused by the environment, which are more likely in the off-sample harder bipedal environment.
Revised: Whilst the y-velocity prediction is relatively accurate for both ROEL and DADS-DR the reduced variance in ROEL is likely due to the environment observing a larger range of failures caused by the environment, which are more likely in the off-sample harder bipedal environment.

Original: The fact that ROELs prediction error is unchanged in the out-of-sample environment is evidence that the skill-dynamics accurately generalize to unknown environments.
Revised: The fact that ROELs prediction error is unchanged in the out-of-sample environment is evidence that the skill-dynamics accurately generalize to unknown environments.

Original: Consequently, this suggests that the intrinsic reward received by ROEL during training captures the general environment dynamics.
Revised: Consequently, this suggests that the intrinsic reward received by ROEL during training captures the general environment dynamics.

Original: ROEL is therefore able to learn skills that generalize over a much larger distribution of possible environments.
Revised: ROEL is therefore able to learn skills that generalize over a much larger distribution of possible environments.

Original: These skills are then pushed to be more diverse, as the outer loop of ROEL encourages the development of environments that lead to long term maximization of intrinsic reward leading to a set of novel, predictable and useful skills.
Revised: These skills are then pushed to be more diverse, as the outer loop of ROEL encourages the development of environments that lead to long term maximization of intrinsic reward leading to a set of novel, predictable and useful skills.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we described how combining previous work in conditional entropy based unsupervised
Revised: In this work we described how combining previous work in conditional entropy based unsupervised

Original: RL with POETs open-ended framework lead to the development of ROEL.
Revised: RL with POETs open-ended framework lead to the development of ROEL.

Original: An algorithm that aims to automatically generate increasingly complex and useful skills by generating its own curricula without the need for supervision.
Revised: An algorithm that aims to automatically generate increasingly complex and useful skills by generating its own curricula without the need for supervision.

Original: We demonstrated that this approach produces agents with skills that are more robust, when used in out-of-sample environments, than simply training on a single static environment.
Revised: We demonstrated that this approach produces agents with skills that are more robust, when used in out-of-sample environments, than simply training on a single static environment.

Original: The most compelling potential of this work lies in the open-ended nature of our methodology.
Revised: The most compelling potential of this work lies in the open-ended nature of our methodology.

Original: Our skills are automatically generated by solving challenges presented by co-evolution, but as we do not need to specify an extrinsic reward function, provided a suitably expressive EC, ROEL should learn to generate skills indeﬁnitely that constantly increase in complexity as the agent is presented with increasingly difﬁcult challenges.
Revised: Our skills are automatically generated by solving challenges presented by co-evolution, but as we do not need to specify an extrinsic reward function, provided a suitably expressive EC, ROEL should learn to generate skills indeﬁnitely that constantly increase in complexity as the agent is presented with increasingly difﬁcult challenges.

Original: In future work, we aim to further investigate the open-endedness of ROEL, by training the agent for many more iterations and aiming to discover how diverse the learnt skills may become.
Revised: In future work, we aim to further investigate the open-endedness of ROEL, by training the agent for many more iterations and aiming to discover how diverse the learnt skills may become.

Original: Further, we plan to investigate how ROEL can be combined with CPPNs to generate a more expressive EC (Stanley, 2007), and how this could be directed to guide novel skill generation beyond the MCC criteria.
Revised: Further, we plan to investigate how ROEL can be combined with CPPNs to generate a more expressive EC (Stanley, 2007), and how this could be directed to guide novel skill generation beyond the MCC criteria.

Original: Following this, we plan to investigate how ROELs learnt skills can be used in concert to exhibit more complex behaviors at test time, perhaps by simultaneously learning a meta-controller (Gupta et al., 2020) that searches for novel skill combinations.
Revised: Following this, we plan to investigate how ROELs learnt skills can be used in concert to exhibit more complex behaviors at test time, perhaps by simultaneously learning a meta-controller (Gupta et al., 2020) that searches for novel skill combinations.

Original: S TATEMENT OF R EPRODUCIBILITY
Revised: S TATEMENT OF R EPRODUCIBILITY

Original: To ensure reproducibility of the algorithm and results described within our paper we have included parameters of the setup used to train and evaluate ROEL in Appendix A, along with a general description of our implementation in section 4.
Revised: To ensure reproducibility of the algorithm and results described within our paper we have included parameters of the setup used to train and evaluate ROEL in Appendix A, along with a general description of our implementation in section 4.

Original: After opening the discussion forums for all submitted papers we shall make a directed comment to the reviewers and area chairs with an anonymous repository containing:
Revised: After opening the discussion forums for all submitted papers we shall make a directed comment to the reviewers and area chairs with an anonymous repository containing:

Original: • Source Code of our model, including methods to train, test and evaluate ROEL.
Revised: • Source Code of our model, including methods to train, test and evaluate ROEL.

Original: • Videos of demonstrations of the behavior of our trained agents.
Revised: • Videos of demonstrations of the behavior of our trained agents.

Original: • Models and Datasets used to evaluate the agents and generate ﬁgures in the report.
Revised: • Models and Datasets used to evaluate the agents and generate ﬁgures in the report.

Original: A I MPLEMENTATION D ETAILS
Revised: A I MPLEMENTATION D ETAILS

Original: In this appendix we provide details of our training setup for our DADS baseline and ROEL implementation.
Revised: In this appendix we provide details of our training setup for our DADS baseline and ROEL implementation.

Original: A. B ASELINE
Revised: A. B ASELINE

Original: We use similar hyperparameters to the DADS implementation for our baseline.
Revised: We use similar hyperparameters to the DADS implementation for our baseline.

Original: All skills are parameterized via a one-hot vector sampled from the uniform prior p ( z )
Revised: All skills are parameterized via a one-hot vector sampled from the uniform prior p ( z )

Original: = 1 D , where D = 20 is the number of skills, we only resample z at the beginning of an episode.
Revised: = 1 D , where D = 20 is the number of skills, we only resample z at the beginning of an episode.

Original: We use the EC-SAC implementation (Haarnoja et al., 2018a) to optimize the policy π ( a | s ; z ) , based on the implementation found in TF-Agents Guadarrama et al.
Revised: We use the EC-SAC implementation (Haarnoja et al., 2018a) to optimize the policy π ( a | s ; z ) , based on the implementation found in TF-Agents Guadarrama et al.

Original: The SAC actor and critic networks both use 2 hidden layers with dimensions (512 , 512) .
Revised: The SAC actor and critic networks both use 2 hidden layers with dimensions (512 , 512) .

Original: The critic network is updated using Adam (Kingma & Ba, 2015) with a ﬁxed learning rate 3 × 10 − 4 , trained over a ﬁxed initial entropy coefﬁcient α = 0 .
Revised: The critic network is updated using Adam (Kingma & Ba, 2015) with a ﬁxed learning rate 3 × 10 − 4 , trained over a ﬁxed initial entropy coefﬁcient α = 0 .

Original: 1 , updated via a soft update target function with co-efﬁcient 0 .
Revised: 1 , updated via a soft update target function with co-efﬁcient 0 .

Original: Similar to DADS the policy is stochastic and parameterized on a normal distribution N ( µ ( s, z ) , (cid:80) ( s, z )) , where Σ is a diagonal covariance matrix that transforms the output to be in the range [ − 1 , +1] via a tanh function.
Revised: Similar to DADS the policy is stochastic and parameterized on a normal distribution N ( µ ( s, z ) , (cid:80) ( s, z )) , where Σ is a diagonal covariance matrix that transforms the output to be in the range [ − 1 , +1] via a tanh function.

Original: The skill-dynamics network uses the same dimensions (512 , 512) as the actor and critic networks.
Revised: The skill-dynamics network uses the same dimensions (512 , 512) as the actor and critic networks.

Original: Our implementation uses a replay buffer to collect samples of reward and trajectory, with batch size of 256.
Revised: Our implementation uses a replay buffer to collect samples of reward and trajectory, with batch size of 256.

Original: The policy and critic networks are then updated via gradient descent every 500 steps forsteps of Adam.
Revised: The policy and critic networks are then updated via gradient descent every 500 steps forsteps of Adam.

Original: A. ROEL
Revised: A. ROEL

Original: Our ROEL implementation is built on-top of the DADS baseline and use the same architecture and hyper-parameters to calculate ROELs intrinsic reward.
Revised: Our ROEL implementation is built on-top of the DADS baseline and use the same architecture and hyper-parameters to calculate ROELs intrinsic reward.

Original: To train ROEL we maintain a list of 6 active ea-pairs ranked via their PATA-EC (Wang et al., 2020).
Revised: To train ROEL we maintain a list of 6 active ea-pairs ranked via their PATA-EC (Wang et al., 2020).

Original: Each active ea-pair is eligible to reproduce if it’s average intrinsic reward is > -0.1, the child will then be considered for addition if its average intrinsic reward over 5 episodes on any active agent satisﬁes an MCC (Brant & Stanley, 2017) in the range [0 . 05 , 100] .
Revised: Each active ea-pair is eligible to reproduce if it’s average intrinsic reward is > -0.1, the child will then be considered for addition if its average intrinsic reward over 5 episodes on any active agent satisﬁes an MCC (Brant & Stanley, 2017) in the range [0 . 05 , 100] .

Original: We found that a min MCC of 0 .
Revised: We found that a min MCC of 0 .

Original: 05 provided frequent updates but prevented
Revised: 05 provided frequent updates but prevented

Original: [Table] [Table]
Revised: [Table] [Table]

Original: the worst environments, we did not investigate the effect of reducing the max criterion.
Revised: the worst environments, we did not investigate the effect of reducing the max criterion.

Original: The same parameters are used to re-rank based on PATA-EC and conduct transfer learning across the ea-list every 8 steps of ROEL.
Revised: The same parameters are used to re-rank based on PATA-EC and conduct transfer learning across the ea-list every 8 steps of ROEL.

Original: We train each environment in the active ea-pairs list for 40 steps every ROEL iteration, each parent can only produce 3 offspring each reproduction step, which itself occurs every 3 steps of ROELs outer loop.
Revised: We train each environment in the active ea-pairs list for 40 steps every ROEL iteration, each parent can only produce 3 offspring each reproduction step, which itself occurs every 3 steps of ROELs outer loop.

Original: Therefore, during the ﬁrst 3 loops of ROEL we only have 3 children and for the rest of training the entire capacity of the ea-pairs list is full.
Revised: Therefore, during the ﬁrst 3 loops of ROEL we only have 3 children and for the rest of training the entire capacity of the ea-pairs list is full.

Original: This produces a total number of 5400 training steps on all ea-pairs using 24 ROEL iterations: (( 3 × 3 × 40) + (3 × 21 × 40) ).
Revised: This produces a total number of 5400 training steps on all ea-pairs using 24 ROEL iterations: (( 3 × 3 × 40) + (3 × 21 × 40) ).

Original: The baseline is trained for 5600 steps, the closest value to 5400 when check-pointing every 400 iterations.
Revised: The baseline is trained for 5600 steps, the closest value to 5400 when check-pointing every 400 iterations.

Original: B B IPEDAL E NVIRONMENT B. W ALKER
Revised: B B IPEDAL E NVIRONMENT B. W ALKER

Original: The bipedal walker environment used is an openAI gym environment (Brockman et al., 2016) with a continuous observation type Box(24) and continuos action space, type Box(4).
Revised: The bipedal walker environment used is an openAI gym environment (Brockman et al., 2016) with a continuous observation type Box(24) and continuos action space, type Box(4).

Original: Table 1 shows theobservations received by the agent at each state that we represent with s in our deﬁnition of ROEL.
Revised: Table 1 shows theobservations received by the agent at each state that we represent with s in our deﬁnition of ROEL.

Original: Similarly, table 2 shows the 4 actions available to the walker.
Revised: Similarly, table 2 shows the 4 actions available to the walker.

Original: Figure 5 shows the states and actions labelled on the bipedal walker.
Revised: Figure 5 shows the states and actions labelled on the bipedal walker.

Original: B. E XTRINSIC R EWARD
Revised: B. E XTRINSIC R EWARD

Original: Algorithm 1 outlines the procedure used to obtain the extrinsic reward r from each step of the bipedal walker environment.
Revised: Algorithm 1 outlines the procedure used to obtain the extrinsic reward r from each step of the bipedal walker environment.

Original: This is based principally on the distance from the start point in the horizontal x direction and 4 motor action states, encoded as a tuple, a = (cid:104) a hip 1 , a hip 2 , a knee 1 , a knee 2 (cid:105) .
Revised: This is based principally on the distance from the start point in the horizontal x direction and 4 motor action states, encoded as a tuple, a = (cid:104) a hip 1 , a hip 2 , a knee 1 , a knee 2 (cid:105) .

Original: A reward shaping term ( ∆ ) is also included to encourage the agent to move forward and ensure the head is kept straight.
Revised: A reward shaping term ( ∆ ) is also included to encourage the agent to move forward and ensure the head is kept straight.

Original: The episode terminates if it reaches the ﬁnish position ( x finish ) or falls over and collides with the ground.
Revised: The episode terminates if it reaches the ﬁnish position ( x finish ) or falls over and collides with the ground.

Original: All terms are normalized by a scale variable to ensure reward behavior is generated correctly regardless of the rendering viewport size.
Revised: All terms are normalized by a scale variable to ensure reward behavior is generated correctly regardless of the rendering viewport size.

Original: Algorithm 1: Bipedal Walker Reward reward function takes an input of hull x position from the state-transition, actions a , and reward shaping term ∆ previous
Revised: Algorithm 1: Bipedal Walker Reward reward function takes an input of hull x position from the state-transition, actions a , and reward shaping term ∆ previous

Original: Function reward( x , a , ∆ previous ) is ∆ current ← 130 x scale − 5 | x | ; (cid:46)
Revised: Function reward( x , a , ∆ previous ) is ∆ current ← 130 x scale − 5 | x | ; (cid:46)

Original: Calculate reward shaping r ← ∆ previous − ∆ current ; for a i in a do r ← r − (0 .
Revised: Calculate reward shaping r ← ∆ previous − ∆ current ; for a i in a do r ← r − (0 .

Original: 00035 τ motor × clip ( | a | , 0 , 1)) ; (cid:46) Slow motor rate endif fallen =
Revised: 00035 τ motor × clip ( | a | , 0 , 1)) ; (cid:46) Slow motor rate endif fallen =

Original: =
Revised: =

Original: True then r ← − 100 ; (cid:46) Punish model if fallen over end ∆ previous ← ∆ current ; (cid:46) Update reward shaping term return r , ∆ previous end
Revised: True then r ← − 100 ; (cid:46) Punish model if fallen over end ∆ previous ← ∆ current ; (cid:46) Update reward shaping term return r , ∆ previous end

Original: B. E NVIRONMENTAL E NCODING (EE)
Revised: B. E NVIRONMENTAL E NCODING (EE)

Original: Our EE is inherited from the original POET bipedal implementation (Wang et al., 2019) and includes the following attributes:
Revised: Our EE is inherited from the original POET bipedal implementation (Wang et al., 2019) and includes the following attributes:

Original: • Ground roughness , controls how much the terrain moves up and down.
Revised: • Ground roughness , controls how much the terrain moves up and down.

Original: ROEL sees a max of 1.0 at train time and our out-of-sample evaluation uses 5.
Revised: ROEL sees a max of 1.0 at train time and our out-of-sample evaluation uses 5.

Original: • Pit , controls how large the pit gaps are in the environment, we use a large range at train time but the out-of-sample evaluation [0.5, 0.5] small gap is not seen during train time.
Revised: • Pit , controls how large the pit gaps are in the environment, we use a large range at train time but the out-of-sample evaluation [0.5, 0.5] small gap is not seen during train time.

Original: • Stump , controls how large stumps are made in the environment and are used in several train environments and are not used during the out-of-sample environment.
Revised: • Stump , controls how large stumps are made in the environment and are used in several train environments and are not used during the out-of-sample environment.

Original: • Stair , adds stairs for the agent to traverse and are not used during training or out-of-sample.
Revised: • Stair , adds stairs for the agent to traverse and are not used during training or out-of-sample.

Original: The default EC used to start ROEL and train the DADS baseline sets all the EE variables to zero.
Revised: The default EC used to start ROEL and train the DADS baseline sets all the EE variables to zero.