Original: Open-Ended Reinforcement Learning with Neural Reward Functions
Revised: Open-Ended Reinforcement Learning with Neural Reward Functions

Original: Abstract
Revised: Abstract

Original: Inspired by the success of unsupervised learning in Computer Vision and Natural
Revised: Inspired by the success of unsupervised learning in Computer Vision and Natural

Original: Language Processing, the reinforcement learning community has recently started to focus more on unsupervised discovery of skills.
Revised: Language Processing, the reinforcement learning community has recently started to focus more on unsupervised discovery of skills.

Original: Most current approaches, like
Revised: Most current approaches, like

Original: DIAYN or DADS, optimize some form of mutual information objective.
Revised: DIAYN or DADS, optimize some form of mutual information objective.

Original: We propose a different approach that uses reward functions encoded by neural networks.
Revised: We propose a different approach that uses reward functions encoded by neural networks.

Original: These are trained iteratively to reward more complex behavior.
Revised: These are trained iteratively to reward more complex behavior.

Original: In high-dimensional robotic environments our approach learns a wide range of interesting skills including front-flips for H ALF -C HEETAH and one-legged running for H UMANOID .
Revised: In high-dimensional robotic environments our approach learns a wide range of interesting skills including front-flips for H ALF -C HEETAH and one-legged running for H UMANOID .

Original: It is the first skill discovery algorithm that can learn such skills without relying on any form of feature engineering.
Revised: It is the first skill discovery algorithm that can learn such skills without relying on any form of feature engineering.

Original: In the pixel-based Montezuma’s Revenge environment our method also works with minimal changes and it learns complex skills that involve interacting with items and visiting diverse locations.
Revised: In the pixel-based Montezuma’s Revenge environment our method also works with minimal changes and it learns complex skills that involve interacting with items and visiting diverse locations.

Original: The implementation of our approach can be found here.
Revised: The implementation of our approach can be found here.

Original:  Introduction
Revised:  Introduction

Original: Deep reinforcement learning (RL) has proven to be very successful in many challenging tasks (Mnih et al., 2015;
Revised: Deep reinforcement learning (RL) has proven to be very successful in many challenging tasks (Mnih et al., 2015;

Original: Silver et al., 2017b; Berner et al., 2019).
Revised: Silver et al., 2017b; Berner et al., 2019).

Original: These were considered intractable just a few years ago.
Revised: These were considered intractable just a few years ago.

Original: However, current methods require enormous amounts of compute to achieve great performance in individual tasks.
Revised: However, current methods require enormous amounts of compute to achieve great performance in individual tasks.

Original: Most of the time, previous models can not be utilised when new tasks are considered, even if the environment does not change.
Revised: Most of the time, previous models can not be utilised when new tasks are considered, even if the environment does not change.

Original: This was also the case in Computer
Revised: This was also the case in Computer

Original: Vision and Natural Language Processing.
Revised: Vision and Natural Language Processing.

Original: However, recently unsupervised learning has been shown to be very effective in both fields.
Revised: However, recently unsupervised learning has been shown to be very effective in both fields.

Original: By using task agnostic pre-training schemes (Tenney et al., 2019;
Revised: By using task agnostic pre-training schemes (Tenney et al., 2019;

Original: Brown et al., 2020; Caron et al., 2021), unsupervised models can solve most tasks with minimal or even no fine-tuning.
Revised: Brown et al., 2020; Caron et al., 2021), unsupervised models can solve most tasks with minimal or even no fine-tuning.

Original: Unsupervised reinforcement learning aims to bring similar successes to the reinforcement learning community.
Revised: Unsupervised reinforcement learning aims to bring similar successes to the reinforcement learning community.

Original: Agents which learn a wide range of unsupervised skills may be able to leverage those to solve new tasks faster and with minimal fine-tuning.
Revised: Agents which learn a wide range of unsupervised skills may be able to leverage those to solve new tasks faster and with minimal fine-tuning.

Original: Here, we propose a new method for open-ended, unsupervised skill discovery.
Revised: Here, we propose a new method for open-ended, unsupervised skill discovery.

Original: We devise an iterative process which creates pairs of neural reward functions and policies.
Revised: We devise an iterative process which creates pairs of neural reward functions and policies.

Original: The policy optimizes the corresponding reward function.
Revised: The policy optimizes the corresponding reward function.

Original: Each of them corresponds to a skill that the agent has learned.
Revised: Each of them corresponds to a skill that the agent has learned.

Original: The neural reward function is a neural network that maps the current observation to a scalar reward.
Revised: The neural reward function is a neural network that maps the current observation to a scalar reward.

Original: In each iteration, the neural reward function is modified to differ from the previous one.
Revised: In each iteration, the neural reward function is modified to differ from the previous one.

Original: This results in increasing the complexity of the encoded task.
Revised: This results in increasing the complexity of the encoded task.

Original: Then, a new skill is trained to optimize this reward function.
Revised: Then, a new skill is trained to optimize this reward function.

Original: We devise several techniques to transfer the knowledge from previously learned skills.
Revised: We devise several techniques to transfer the knowledge from previously learned skills.

Original: These mechanisms enable learning the most complex reward functions that our method creates.
Revised: These mechanisms enable learning the most complex reward functions that our method creates.

Original: In fact, we show that some of the functions are impossible to learn from scratch.
Revised: In fact, we show that some of the functions are impossible to learn from scratch.

Original: We empirically test our framework in a diverse set of environments.
Revised: We empirically test our framework in a diverse set of environments.

Original: First, we apply it to a simple 2d navigation task.
Revised: First, we apply it to a simple 2d navigation task.

Original: This lets us perform experiments quickly and gain a solid empirical understanding of the different components of our method.
Revised: This lets us perform experiments quickly and gain a solid empirical understanding of the different components of our method.

Original: Then, we apply it to three robotic environments where we have continuous high-dimensional observations and actions.
Revised: Then, we apply it to three robotic environments where we have continuous high-dimensional observations and actions.

Original: In contrast to previous methods, our approach can deal with high-dimensional input directly without feature engineering.
Revised: In contrast to previous methods, our approach can deal with high-dimensional input directly without feature engineering.

Original: Finally, we apply it on the challenging Montezuma’s Revenge Atari game which has visually rich pixel based observations and discrete actions.
Revised: Finally, we apply it on the challenging Montezuma’s Revenge Atari game which has visually rich pixel based observations and discrete actions.

Original: Despite their differences, our method manages to learn useful and interesting skills in all of them.
Revised: Despite their differences, our method manages to learn useful and interesting skills in all of them.

Original: This shows that neural reward functions are equipped to encode meaningful tasks in diverse environments.
Revised: This shows that neural reward functions are equipped to encode meaningful tasks in diverse environments.

Original: We highlight the main contributions of this paper:
Revised: We highlight the main contributions of this paper:

Original: • Propose the first skill-discovery algorithm to work in high-dimensional environments without prior expert knowledge.
Revised: • Propose the first skill-discovery algorithm to work in high-dimensional environments without prior expert knowledge.

Original: • Solve a 2d maze that cannot be solved by using random exploration.
Revised: • Solve a 2d maze that cannot be solved by using random exploration.

Original: • Acquire complex skills including performing front- and back-flip in the H ALF -C HEETAH , running in all directions in A NT and H UMANOID , standing and jumping on one leg in H UMANOID .
Revised: • Acquire complex skills including performing front- and back-flip in the H ALF -C HEETAH , running in all directions in A NT and H UMANOID , standing and jumping on one leg in H UMANOID .

Original: • Learn to run as fast in H UMANOID as a supervised agent trained for tens of millions of steps.
Revised: • Learn to run as fast in H UMANOID as a supervised agent trained for tens of millions of steps.

Original: • Achieve a higher Particle-Based Mutual Information Metric (Gu et al., 2021) than approaches that explicitly optimize this metric.
Revised: • Achieve a higher Particle-Based Mutual Information Metric (Gu et al., 2021) than approaches that explicitly optimize this metric.

Original: • Collect the first key and reach several rooms in Montezuma’s Revenge.
Revised: • Collect the first key and reach several rooms in Montezuma’s Revenge.

Original:  Related Work
Revised:  Related Work

Original: Our work fits best into the unsupervised skill discovery literature (Mohamed & Rezende, 2015;
Revised: Our work fits best into the unsupervised skill discovery literature (Mohamed & Rezende, 2015;

Original: Gregor et al., 2016; Florensa et al., 2017a; Achiam et al., 2018;
Revised: Gregor et al., 2016; Florensa et al., 2017a; Achiam et al., 2018;

Original: Eysenbach et al., 2018; Sharma et al., 2019; Choi et al., 2021).
Revised: Eysenbach et al., 2018; Sharma et al., 2019; Choi et al., 2021).

Original: Compared to DIAYN (Eysenbach et al., 2018) and similar approaches, we do not fix the number of skills to be learned at the beginning of training.
Revised: Compared to DIAYN (Eysenbach et al., 2018) and similar approaches, we do not fix the number of skills to be learned at the beginning of training.

Original: This allows us to learn new skills in an open-ended fashion.
Revised: This allows us to learn new skills in an open-ended fashion.

Original: On top of that, maximizing mutual information can lead to degenerate behaviors in high-dimensional environments.
Revised: On top of that, maximizing mutual information can lead to degenerate behaviors in high-dimensional environments.

Original: By manipulating a small subset of the dimensions, a lot of information can be encoded, without exploring the rest of the state space.
Revised: By manipulating a small subset of the dimensions, a lot of information can be encoded, without exploring the rest of the state space.

Original: Due to this issue, previous methods only consider a hand-picked subset of the dimensions to perform well.
Revised: Due to this issue, previous methods only consider a hand-picked subset of the dimensions to perform well.

Original: Open-ended learning (Srivastava et al., 2012; Wang et al., 2019, 2020; Campero et al., 2020; Dennis et al., 2020; Ecoffet et al., 2021; Stooke et al., 2021) is closely related to unsupervised skill discovery.
Revised: Open-ended learning (Srivastava et al., 2012; Wang et al., 2019, 2020; Campero et al., 2020; Dennis et al., 2020; Ecoffet et al., 2021; Stooke et al., 2021) is closely related to unsupervised skill discovery.

Original: However, most approaches require either a parameterizable environment (Wang et al., 2019, 2020), some fixed encoding of tasks (Stooke et al., 2021) or self-competition (Silver et al., 2017a; Baker et al., 2019).
Revised: However, most approaches require either a parameterizable environment (Wang et al., 2019, 2020), some fixed encoding of tasks (Stooke et al., 2021) or self-competition (Silver et al., 2017a; Baker et al., 2019).

Original: This limits the applicability to environments that are engineered with these restrictions in mind.
Revised: This limits the applicability to environments that are engineered with these restrictions in mind.

Original: In contrast, neural networks are universal function approximators and thus, our approach can encode any possible task in any possible environment, as long as the input is chosen appropriately.
Revised: In contrast, neural networks are universal function approximators and thus, our approach can encode any possible task in any possible environment, as long as the input is chosen appropriately.

Original: Similar to our method, the Go-Explore algorithm (Ecoffet et al., 2021) also explores around the frontier of the known.
Revised: Similar to our method, the Go-Explore algorithm (Ecoffet et al., 2021) also explores around the frontier of the known.

Original: It uses a handcrafted feature map to group similar states.
Revised: It uses a handcrafted feature map to group similar states.

Original: Instead, we use neural reward functions to group them as skills.
Revised: Instead, we use neural reward functions to group them as skills.

Original: This does not use any expert knowledge.
Revised: This does not use any expert knowledge.

Original: Another related line of research to our approach is intrinsic motivation (Stadie et al., 2015; Bellemare et al., 2016; Pathak et al., 2017; Burda et al., 2018b,a; Pathak et al., 2019; Raileanu & Rockt¨aschel, 2020).
Revised: Another related line of research to our approach is intrinsic motivation (Stadie et al., 2015; Bellemare et al., 2016; Pathak et al., 2017; Burda et al., 2018b,a; Pathak et al., 2019; Raileanu & Rockt¨aschel, 2020).

Original: These approaches have managed great success in hard-exploration Atari games.
Revised: These approaches have managed great success in hard-exploration Atari games.

Original: However, these approaches do not learn multiple skills by default.
Revised: However, these approaches do not learn multiple skills by default.

Original: Additionally, their goal is to reward all novel states, which leads to general policies that visit many states.
Revised: Additionally, their goal is to reward all novel states, which leads to general policies that visit many states.

Original: In contrast, skill discovery algorithms try to reward narrow regions of the state space to achieve controllability of meaningful dimensions.
Revised: In contrast, skill discovery algorithms try to reward narrow regions of the state space to achieve controllability of meaningful dimensions.

Original: Encoding reward functions as neural networks has also been considered in the literature.
Revised: Encoding reward functions as neural networks has also been considered in the literature.

Original: Compared to our work, they are trained with supervised signals (Abbeel & Ng, 2004; Fu et al., 2017; Singh et al., 2019; Li et al., 2021).
Revised: Compared to our work, they are trained with supervised signals (Abbeel & Ng, 2004; Fu et al., 2017; Singh et al., 2019; Li et al., 2021).

Original: Other approaches train auxiliary rewards with meta-learning (Zheng et al., 2018; Du et al., 2019; Veeriah et al., 2019) to enhance the learning of the original reward function.
Revised: Other approaches train auxiliary rewards with meta-learning (Zheng et al., 2018; Du et al., 2019; Veeriah et al., 2019) to enhance the learning of the original reward function.

Original: Another approach to train multiple behaviors is goal-conditioned learning (Kaelbling, 1993;
Revised: Another approach to train multiple behaviors is goal-conditioned learning (Kaelbling, 1993;

Original: Schaul et al., 2015; Andrychowicz et al., 2017; Rauber et al., 2017; Nair et al., 2018; Veeriah et al., 2018;
Revised: Schaul et al., 2015; Andrychowicz et al., 2017; Rauber et al., 2017; Nair et al., 2018; Veeriah et al., 2018;

Original: Warde-Farley et al., 2018; Pong et al., 2019; Choi et al., 2021).
Revised: Warde-Farley et al., 2018; Pong et al., 2019; Choi et al., 2021).

Original: In automated curriculum learning (Bengio et al., 2009; Florensa et al., 2017b; Forestier et al., 2017; Graves et al., 2017; Sukhbaatar et al., 2017; Florensa et al., 2018; Matiisen et al., 2019; Narvekar et al., 2020;
Revised: In automated curriculum learning (Bengio et al., 2009; Florensa et al., 2017b; Forestier et al., 2017; Graves et al., 2017; Sukhbaatar et al., 2017; Florensa et al., 2018; Matiisen et al., 2019; Narvekar et al., 2020;

Original: Portelas et al., 2020a,b;
Revised: Portelas et al., 2020a,b;

Original: Zhang et al., 2020), a sequence of goals is created such that each of them is not too hard nor too easy for the current agent.
Revised: Zhang et al., 2020), a sequence of goals is created such that each of them is not too hard nor too easy for the current agent.

Original: These approaches mostly rely on low-dimensional goal embeddings.
Revised: These approaches mostly rely on low-dimensional goal embeddings.

Original: When dealing with high-dimensional observations, they must use dimensionality reduction techniques.
Revised: When dealing with high-dimensional observations, they must use dimensionality reduction techniques.

Original: These techniques can introduce instabilities or destroy relevant information from the input.
Revised: These techniques can introduce instabilities or destroy relevant information from the input.

Original: Our approach, on the other hand, can deal with high-dimensional inputs directly.
Revised: Our approach, on the other hand, can deal with high-dimensional inputs directly.

Original: On top of that, goals encode a narrow region of the state space, while each of our reward functions can be rewarding in a large region.
Revised: On top of that, goals encode a narrow region of the state space, while each of our reward functions can be rewarding in a large region.

Original: This speeds-up the exploration in ‘easy’ regions of the state space.
Revised: This speeds-up the exploration in ‘easy’ regions of the state space.

Original:  Method
Revised:  Method

Original: We introduce a method that performs open-ended, unsupervised skill discovery.
Revised: We introduce a method that performs open-ended, unsupervised skill discovery.

Original: It iteratively creates pairs of neural reward functions R ψ and policies π θ trained to maximize the corresponding R ψ .
Revised: It iteratively creates pairs of neural reward functions R ψ and policies π θ trained to maximize the corresponding R ψ .

Original: Our proposed method alternates between increasing the complexity of the reward function R ψ and leveraging the previously learned skills to learn a policy π θ that can solve the new R ψ .
Revised: Our proposed method alternates between increasing the complexity of the reward function R ψ and leveraging the previously learned skills to learn a policy π θ that can solve the new R ψ .

Original: This yields a general learning procedure that learns complex skills in a diverse set of environments.
Revised: This yields a general learning procedure that learns complex skills in a diverse set of environments.

Original: See Figurefor a high level overview.
Revised: See Figurefor a high level overview.

Original: 3.1 Increasing the complexity of R ψ
Revised: 3.1 Increasing the complexity of R ψ

Original: The reward function in a fully observable
Revised: The reward function in a fully observable

Original: Markov Decision Process (MDP) is by definition a function of the current observation, the next one and the action that was performed.
Revised: Markov Decision Process (MDP) is by definition a function of the current observation, the next one and the action that was performed.

Original: However, in many cases this can be reduced to a function of just the current observation.
Revised: However, in many cases this can be reduced to a function of just the current observation.

Original: Because of this, we opt for these simpler reward functions as the basis of our R ψ .
Revised: Because of this, we opt for these simpler reward functions as the basis of our R ψ .

Original: In our method, the reward function
Revised: In our method, the reward function

Original: R ψ is a neural network which takes observations o t as input and outputs a single scalar value, the instantaneous reward r ψt .
Revised: R ψ is a neural network which takes observations o t as input and outputs a single scalar value, the instantaneous reward r ψt .

Original: Assume that we have a policy π θ that can reach states in the MDP which are rewarding under R ψ .
Revised: Assume that we have a policy π θ that can reach states in the MDP which are rewarding under R ψ .

Original: To increase the complexity of the reward function, we do the following:
Revised: To increase the complexity of the reward function, we do the following:

Original: • Decrease the reward of states which are visited by π θ , as we already have a policy that reaches these states.
Revised: • Decrease the reward of states which are visited by π θ , as we already have a policy that reaches these states.

Original: We create a data set O neg of such states.
Revised: We create a data set O neg of such states.

Original: We refer to these states as negative samples .
Revised: We refer to these states as negative samples .

Original: • Increase the reward of states that can almost be reached by the current policy.
Revised: • Increase the reward of states that can almost be reached by the current policy.

Original: This allows us to leverage π θ to learn the new reward function.
Revised: This allows us to leverage π θ to learn the new reward function.

Original: We create a data set O pos of such states.
Revised: We create a data set O pos of such states.

Original: We refer to these states as positive samples .
Revised: We refer to these states as positive samples .

Original: To generate the negative samples, we run π θ for a given number of steps (ideally until it reaches rewarding states) and store the visited states.
Revised: To generate the negative samples, we run π θ for a given number of steps (ideally until it reaches rewarding states) and store the visited states.

Original: To then generate positive samples, we change to performing random actions 2 for a fixed number of steps.
Revised: To then generate positive samples, we change to performing random actions 2 for a fixed number of steps.

Original: To ensure that the new reward function is different from all previous ones, we also keep track of all the negative samples that we have collected for all skills in a data set
Revised: To ensure that the new reward function is different from all previous ones, we also keep track of all the negative samples that we have collected for all skills in a data set

Original: O neg all .
Revised: O neg all .

Original: Finally, we set target values a and − a for the positive and negative samples respectively, and train the reward network using standard supervised learning on the following loss:
Revised: Finally, we set target values a and − a for the positive and negative samples respectively, and train the reward network using standard supervised learning on the following loss:

Original: [Equation]
Revised: [Equation]

Original: This loss ensures that positive samples that have never been seen before have positive reward in the next R ψ , while all other samples that have been seen before decrease their reward.
Revised: This loss ensures that positive samples that have never been seen before have positive reward in the next R ψ , while all other samples that have been seen before decrease their reward.

Original: In the reinforcement learning phase we clip rewards to the [0 , a ] range.
Revised: In the reinforcement learning phase we clip rewards to the [0 , a ] range.

Original: This ensures that the agent seeks positive samples, rather than less negative ones.
Revised: This ensures that the agent seeks positive samples, rather than less negative ones.

Original: 3.2 Forward transfer for π θ
Revised: 3.2 Forward transfer for π θ

Original: Given the procedure presented in Section 3.1, we create increasingly complex reward functions.
Revised: Given the procedure presented in Section 3.1, we create increasingly complex reward functions.

Original: While this is great for open-ended learning, it eventually leads to skills that are too complex and cannot be learned from scratch.
Revised: While this is great for open-ended learning, it eventually leads to skills that are too complex and cannot be learned from scratch.

Original: In order to learn these skills, we must leverage previous knowledge about the environment.
Revised: In order to learn these skills, we must leverage previous knowledge about the environment.

Original: In this section we present several forward transfer mechanisms that are necessary for the most complex skills.
Revised: In this section we present several forward transfer mechanisms that are necessary for the most complex skills.

Original: In principle, our method can be combined with any standard reinforcement learning (RL) technique.
Revised: In principle, our method can be combined with any standard reinforcement learning (RL) technique.

Original: But here, we focus on actor-critic methods like Advantage Actor-Critic (Mnih et al., 2016) or Proximal
Revised: But here, we focus on actor-critic methods like Advantage Actor-Critic (Mnih et al., 2016) or Proximal

Original: Policy Optimization (Schulman et al., 2017) for learning π θ for several reasons.
Revised: Policy Optimization (Schulman et al., 2017) for learning π θ for several reasons.

Original: These methods have a value network that is separate from the policy which allows us to use different transfer mechanisms for the value and policy network.
Revised: These methods have a value network that is separate from the policy which allows us to use different transfer mechanisms for the value and policy network.

Original: Also, these approaches work for both continuous and discrete action spaces.
Revised: Also, these approaches work for both continuous and discrete action spaces.

Original: This allows us to use the same technique for the robotic environments and for the 2d navigation tasks.
Revised: This allows us to use the same technique for the robotic environments and for the 2d navigation tasks.

Original: Finally the learned policies are stochastic which increases the diversity of negative samples and speeds up the skill discovery process.
Revised: Finally the learned policies are stochastic which increases the diversity of negative samples and speeds up the skill discovery process.

Original: See Section 4.1.2.
Revised: See Section 4.1.2.

Original: We present our three forward transfer mechanisms below.
Revised: We present our three forward transfer mechanisms below.

Original: All of them exploit the similarity between consecutive reward functions to ensure that even very complex reward functions can be solved by the
Revised: All of them exploit the similarity between consecutive reward functions to ensure that even very complex reward functions can be solved by the

Original: RL agent in a reasonable number of environment interactions.
Revised: RL agent in a reasonable number of environment interactions.

Original: • Value Reuse: Initialize the value network to the final value network of the previous agent.
Revised: • Value Reuse: Initialize the value network to the final value network of the previous agent.

Original: While two consecutive reward functions are different, both still reward close-by regions of the state space.
Revised: While two consecutive reward functions are different, both still reward close-by regions of the state space.

Original: Thus, by keeping the previous value function, the policy network will be nudged towards that region of the state space from the very first gradient updates.
Revised: Thus, by keeping the previous value function, the policy network will be nudged towards that region of the state space from the very first gradient updates.

Original: • Policy Feature Reuse: Initialize the policy network to the final policy of the previous agent but setting the weights of the final layer to 0 .
Revised: • Policy Feature Reuse: Initialize the policy network to the final policy of the previous agent but setting the weights of the final layer to 0 .

Original: This keeps the previously learned features, but outputs a uniform policy over all actions (or mean 0 and a fixed standard deviation in the continuous case) which allows for proper learning and exploration 3 .
Revised: This keeps the previously learned features, but outputs a uniform policy over all actions (or mean 0 and a fixed standard deviation in the continuous case) which allows for proper learning and exploration 3 .

Original: • Guiding Policy: Act with the previous policy for a random number of steps at the beginning of each episode.
Revised: • Guiding Policy: Act with the previous policy for a random number of steps at the beginning of each episode.

Original: This heavily simplifies the exploration problem.
Revised: This heavily simplifies the exploration problem.

Original: The agent will start exploring from states that are much closer to the rewards defined by R ψ .
Revised: The agent will start exploring from states that are much closer to the rewards defined by R ψ .

Original: This is because the previous policy could already solve the
Revised: This is because the previous policy could already solve the

Original: previous reward function.
Revised: previous reward function.

Original: In contrast to the other two mechanisms, this one does not rely on initialization.
Revised: In contrast to the other two mechanisms, this one does not rely on initialization.

Original: Thus, this is the most effective in sparse reward functions that need many parameter updates to be learned.
Revised: Thus, this is the most effective in sparse reward functions that need many parameter updates to be learned.

Original: In Section 4.1.1 we individually evaluate these three techniques and show that their combination is necessary in complex environments.
Revised: In Section 4.1.1 we individually evaluate these three techniques and show that their combination is necessary in complex environments.

Original: Putting everything together we get an algorithm which learns reward functions that encode increasingly complex behaviors and learns RL agents that solve those reward functions.
Revised: Putting everything together we get an algorithm which learns reward functions that encode increasingly complex behaviors and learns RL agents that solve those reward functions.

Original: Figure 1 illustrates the main steps of our training loop and see Algorithm 1 in the appendix for more detail.
Revised: Figure 1 illustrates the main steps of our training loop and see Algorithm 1 in the appendix for more detail.

Original:  Experiments
Revised:  Experiments

Original: We now proceed to experimentally test our method.
Revised: We now proceed to experimentally test our method.

Original: First, in Section 4.1, we thoroughly test all different components of our model in a 2d navigation task.
Revised: First, in Section 4.1, we thoroughly test all different components of our model in a 2d navigation task.

Original: This task allows us to verify the function of each component and also to explicitly visualize what each reward function is encoding.
Revised: This task allows us to verify the function of each component and also to explicitly visualize what each reward function is encoding.

Original: Then, in Section 4.2, we move to BRAX robotic environments (Freeman et al., 2021).
Revised: Then, in Section 4.2, we move to BRAX robotic environments (Freeman et al., 2021).

Original: These have a lot of flexibility and thus allow the agent to learn very complex tasks.
Revised: These have a lot of flexibility and thus allow the agent to learn very complex tasks.

Original: In these tasks, we evaluate the complexity of our skills by measuring their zero-shot transfer ability to the environment rewards.
Revised: In these tasks, we evaluate the complexity of our skills by measuring their zero-shot transfer ability to the environment rewards.

Original: In the H UMANOID environment, our unsupervised skills outperform supervised agents trained for tens of millions of time steps.
Revised: In the H UMANOID environment, our unsupervised skills outperform supervised agents trained for tens of millions of time steps.

Original: We also compute the one dimensional particle-based mutual information metric that has been proposed in the literature before (Gu et al., 2021) and show that our method outperforms previous approaches, even when other approaches only consider handcrafted feature dimensions in their objective.
Revised: We also compute the one dimensional particle-based mutual information metric that has been proposed in the literature before (Gu et al., 2021) and show that our method outperforms previous approaches, even when other approaches only consider handcrafted feature dimensions in their objective.

Original: Finally, in Section 4.3, we apply our method to Montezuma’s Revenge.
Revised: Finally, in Section 4.3, we apply our method to Montezuma’s Revenge.

Original: We show that the learned reward functions keep getting increasingly complex and we are mostly limited by the amount of compute that it takes to learn each new reward function.
Revised: We show that the learned reward functions keep getting increasingly complex and we are mostly limited by the amount of compute that it takes to learn each new reward function.

Original: 4.1 2D Navigation Task
Revised: 4.1 2D Navigation Task

Original: The task consists of a 32 by 32 maze with several walls and a ‘danger zone’.
Revised: The task consists of a 32 by 32 maze with several walls and a ‘danger zone’.

Original: observation is given as a 32 x
Revised: observation is given as a 32 x

Original: image with all values set to 0 , except a 1 in the current position of the agent.
Revised: image with all values set to 0 , except a 1 in the current position of the agent.

Original: The agent starts in the top corner and can perform 5 actions, either move in one of the cardinal directions or stay in the current position.
Revised: The agent starts in the top corner and can perform 5 actions, either move in one of the cardinal directions or stay in the current position.

Original: If the agent moves into a wall it stays at its current position instead.
Revised: If the agent moves into a wall it stays at its current position instead.

Original: If the agent is in the ‘danger zone’ and moves up, moves down or stays, the episode is terminated and the agent is moved back to the starting position.
Revised: If the agent is in the ‘danger zone’ and moves up, moves down or stays, the episode is terminated and the agent is moved back to the starting position.

Original: Figure 2 shows the layout of the maze.
Revised: Figure 2 shows the layout of the maze.

Original: The ‘danger zone’ ensures that random exploration will not work to reach many parts of the environment and lets us easily test both the increasing complexity of the reward functions and the importance of forward transfer.
Revised: The ‘danger zone’ ensures that random exploration will not work to reach many parts of the environment and lets us easily test both the increasing complexity of the reward functions and the importance of forward transfer.

Original: We computed the expected number of steps to reach the bottom right corner with a random walk using Dynamic Programming.
Revised: We computed the expected number of steps to reach the bottom right corner with a random walk using Dynamic Programming.

Original: In expectation, 7 · 10episodes are needed to do so.
Revised: In expectation, 7 · 10episodes are needed to do so.

Original: This shows that this maze is difficult to navigate.
Revised: This shows that this maze is difficult to navigate.

Original: To train our agent we use the Advantage Actor Critic (A2C) (Mnih et al., 2016) algorithm.
Revised: To train our agent we use the Advantage Actor Critic (A2C) (Mnih et al., 2016) algorithm.

Original: To learn the rewards we use the full algorithm presented in Section 3.
Revised: To learn the rewards we use the full algorithm presented in Section 3.

Original: We use the same architecture for the reward, policy and value networks, but do not share any parameters.
Revised: We use the same architecture for the reward, policy and value networks, but do not share any parameters.

Original: The architecture is a ReLU network with 2 convolutional layers followed by 2 fully connected layers.
Revised: The architecture is a ReLU network with 2 convolutional layers followed by 2 fully connected layers.

Original: For the exact hyper-parameters see Table 3 in Appendix B. Figure 2 plots the most visited locations for each skill of one run of our algorithm.
Revised: For the exact hyper-parameters see Table 3 in Appendix B. Figure 2 plots the most visited locations for each skill of one run of our algorithm.

Original: The first few skills visit points near the origin, later skills start moving to harder to reach parts of the state space.
Revised: The first few skills visit points near the origin, later skills start moving to harder to reach parts of the state space.

Original: After roughly 40 iterations they reach the bottom right part.
Revised: After roughly 40 iterations they reach the bottom right part.

Original: As stated before, this would take unreasonably long when using only random exploration.
Revised: As stated before, this would take unreasonably long when using only random exploration.

Original: Inspired by the BRAX library (Freeman et al., 2021), we implemented both the environment and an A2C agent inside a single JAX (Bradbury et al., 2018) compiled function.
Revised: Inspired by the BRAX library (Freeman et al., 2021), we implemented both the environment and an A2C agent inside a single JAX (Bradbury et al., 2018) compiled function.

Original: By doing this, the computation graph of the environment and agent are optimized jointly and both run on the GPU.
Revised: By doing this, the computation graph of the environment and agent are optimized jointly and both run on the GPU.

Original: This eliminates the need to send data between the CPU and GPU which is one of the main bottlenecks in RL.
Revised: This eliminates the need to send data between the CPU and GPU which is one of the main bottlenecks in RL.

Original: Using just one NVIDIA RTX 3090 GPU, the training process runs at over one million frames per second which enables training of agents in just a few seconds.
Revised: Using just one NVIDIA RTX 3090 GPU, the training process runs at over one million frames per second which enables training of agents in just a few seconds.

Original: This allows us to experiment quickly and at a lower economic and environmental cost.
Revised: This allows us to experiment quickly and at a lower economic and environmental cost.

Original: We believe this code is useful for the RL community on its own and provide it in the supplementary material.
Revised: We believe this code is useful for the RL community on its own and provide it in the supplementary material.

Original: 4.1.1 Forward transfer of skills
Revised: 4.1.1 Forward transfer of skills

Original: [Figure]
Revised: [Figure]

Original: As pointed out in Section 3.2, our reward functions become too complex to be learned from scratch with random exploration in a reasonable number of steps.
Revised: As pointed out in Section 3.2, our reward functions become too complex to be learned from scratch with random exploration in a reasonable number of steps.

Original: When this happens, our agent must rely on transferring knowledge from previous generations.
Revised: When this happens, our agent must rely on transferring knowledge from previous generations.

Original: Our navigation task is specifically designed to test this transfer ability, as random exploration would never reach the bottom right corner ( 7 · 10episodes in expectation).
Revised: Our navigation task is specifically designed to test this transfer ability, as random exploration would never reach the bottom right corner ( 7 · 10episodes in expectation).

Original: We experimentally evaluate the three forward transfer mechanisms proposed in Section reuse , Policy feature reuse and Guiding policy .
Revised: We experimentally evaluate the three forward transfer mechanisms proposed in Section reuse , Policy feature reuse and Guiding policy .

Original: The reward functions from Figure 2 serve as tasks, sorted according to creation order.
Revised: The reward functions from Figure 2 serve as tasks, sorted according to creation order.

Original: We train ablations of the three mechanisms sequentially on these tasks.
Revised: We train ablations of the three mechanisms sequentially on these tasks.

Original: This allows us to ignore the skill discovery process and only measure the forward transfer of skills.
Revised: This allows us to ignore the skill discovery process and only measure the forward transfer of skills.

Original: We repeat each experiment three times.
Revised: We repeat each experiment three times.

Original: The agent with all mechanisms always manages to solve 4 all reward functions.
Revised: The agent with all mechanisms always manages to solve 4 all reward functions.

Original: On the other hand, the Policy, Value and Guiding ablations fail to learn after solving 29 .
Revised: On the other hand, the Policy, Value and Guiding ablations fail to learn after solving 29 .

Original: 3 ± 9 . 5 , 18 .
Revised: 3 ± 9 . 5 , 18 .

Original: 7 ± 11 .
Revised: 7 ± 11 .

Original: 6 and 19 ± 8 .
Revised: 6 and 19 ± 8 .

Original: 5 reward functions, respectively.
Revised: 5 reward functions, respectively.

Original: 4.1.2 Speeding-up deep exploration
Revised: 4.1.2 Speeding-up deep exploration

Original: [Figure]
Revised: [Figure]

Original: One key parameter when training actor critic methods is entropy regularization.
Revised: One key parameter when training actor critic methods is entropy regularization.

Original: In our method, policies with a lot of entropy generate a diverse set of negative samples.
Revised: In our method, policies with a lot of entropy generate a diverse set of negative samples.

Original: Diverse negative samples lead to reward functions that evolve more in each generation.
Revised: Diverse negative samples lead to reward functions that evolve more in each generation.

Original: This is especially beneficial in environments where many steps are necessary to reach certain states, like in this 2D navigation task or Montezuma’s Revenge.
Revised: This is especially beneficial in environments where many steps are necessary to reach certain states, like in this 2D navigation task or Montezuma’s Revenge.

Original: We empirically verify this claim by training a set of agents with varying levels of entropy regularization.
Revised: We empirically verify this claim by training a set of agents with varying levels of entropy regularization.

Original: Figure 3 shows the coverage of the state space after 60 generations as a function of the entropy regularization.
Revised: Figure 3 shows the coverage of the state space after 60 generations as a function of the entropy regularization.

Original: We observed that higher entropy leads to a faster coverage of the state space up to a certain threshold.
Revised: We observed that higher entropy leads to a faster coverage of the state space up to a certain threshold.

Original: However, too much entropy leads to policies that do not learn to reach the rewarding states when these are far away from the origin.
Revised: However, too much entropy leads to policies that do not learn to reach the rewarding states when these are far away from the origin.

Original: Observe that this problem arises independent of the training procedure as entropy regularization changes which policy is considered optimal.
Revised: Observe that this problem arises independent of the training procedure as entropy regularization changes which policy is considered optimal.

Original: Hence, we want to have policies that gather diverse samples to change the reward function but can also be deterministic in parts of the trajectory that already have no reward.
Revised: Hence, we want to have policies that gather diverse samples to change the reward function but can also be deterministic in parts of the trajectory that already have no reward.

Original: To achieve this, we use adaptive entropy regularization.
Revised: To achieve this, we use adaptive entropy regularization.

Original: That is, we apply a small entropy regularization term everywhere.
Revised: That is, we apply a small entropy regularization term everywhere.

Original: We increase the entropy regularization in states where the reward function is positive, i.e. states in which we want to decrease the reward function.
Revised: We increase the entropy regularization in states where the reward function is positive, i.e. states in which we want to decrease the reward function.

Original: See Figure 4 for a visualization of the impact of the different entropy regularization strategies.
Revised: See Figure 4 for a visualization of the impact of the different entropy regularization strategies.

Original: With this, the method manages to consistently visit all states in the maze.
Revised: With this, the method manages to consistently visit all states in the maze.

Original: This technique was necessary to create the reward functions from Figure 2.
Revised: This technique was necessary to create the reward functions from Figure 2.

Original: 4.2 Robotic Environments
Revised: 4.2 Robotic Environments

Original: Quantitatively measuring unsupervised reinforcement learning progress remains an open problem and active area of research (Gu et al., 2021).
Revised: Quantitatively measuring unsupervised reinforcement learning progress remains an open problem and active area of research (Gu et al., 2021).

Original: Because of this, it is still important to visualize and qualitatively study the learned skills.
Revised: Because of this, it is still important to visualize and qualitatively study the learned skills.

Original: We do this in Section 4.2.1.
Revised: We do this in Section 4.2.1.

Original: In the next two Sections, 4.2.2 and 4.2.3, we quantitatively measure the performance of our algorithm on downstream task performance and with the so-called particle-based mutual information metric.
Revised: In the next two Sections, 4.2.2 and 4.2.3, we quantitatively measure the performance of our algorithm on downstream task performance and with the so-called particle-based mutual information metric.

Original: In contrast to the previous experiments, the action space is continuous and the input modality is now a vector of features, like relative position, angle and speed of the different joints.
Revised: In contrast to the previous experiments, the action space is continuous and the input modality is now a vector of features, like relative position, angle and speed of the different joints.

Original: We do not use adaptive entropy here, as we need deterministic policies to maximize the downstream task performance and the mutual information metric.
Revised: We do not use adaptive entropy here, as we need deterministic policies to maximize the downstream task performance and the mutual information metric.

Original: Given that the environments do not have far away regions to reach, we did not implement the guiding algorithm here.
Revised: Given that the environments do not have far away regions to reach, we did not implement the guiding algorithm here.

Original: We do ablations for the other two transfer mechanisms.
Revised: We do ablations for the other two transfer mechanisms.

Original: Note that the agent does not see the x - y position.
Revised: Note that the agent does not see the x - y position.

Original: Details about hyper-parameters and architecture can be found in Appendix C.
Revised: Details about hyper-parameters and architecture can be found in Appendix C.

Original: In addition to skill discovery methods, we also compare our methods to RND (Burda et al., 2018b) and ‘Disagreement’ (Pathak et al., 2019), two of the most prominent works in intrinsic motivation.
Revised: In addition to skill discovery methods, we also compare our methods to RND (Burda et al., 2018b) and ‘Disagreement’ (Pathak et al., 2019), two of the most prominent works in intrinsic motivation.

Original: By default, these methods only create a single final policy.
Revised: By default, these methods only create a single final policy.

Original: We extract multiple skills by taking checkpoints of the policy regularly during training 5 .
Revised: We extract multiple skills by taking checkpoints of the policy regularly during training 5 .

Original: To our knowledge, this has not been done before and we believe this showcases how closely related intrinsic motivation and unsupervised skill discovery are.
Revised: To our knowledge, this has not been done before and we believe this showcases how closely related intrinsic motivation and unsupervised skill discovery are.

Original: We tuned the hyper-parameters of both methods extensively to maximize the Mutual
Revised: We tuned the hyper-parameters of both methods extensively to maximize the Mutual

Original: Information metric (c.f. Section 4.2.3).
Revised: Information metric (c.f. Section 4.2.3).

Original: 4.2.1 Qualitative Analysis
Revised: 4.2.1 Qualitative Analysis

Original: In Figure 5 we illustrate a selection of particularly interesting skills.
Revised: In Figure 5 we illustrate a selection of particularly interesting skills.

Original: Videos of these and additional skills can be found here.
Revised: Videos of these and additional skills can be found here.

Original: In Figure 6 we see how the velocity of learned skills in A NT and H UMANOID evolve over training.
Revised: In Figure 6 we see how the velocity of learned skills in A NT and H UMANOID evolve over training.

Original: See Appendix E for the other runs.
Revised: See Appendix E for the other runs.

Original: One can see that consecutive skills slightly change the direction and speed of moving.
Revised: One can see that consecutive skills slightly change the direction and speed of moving.

Original: However, it is important to realize that the observations contain tens to hundreds of dimensions.
Revised: However, it is important to realize that the observations contain tens to hundreds of dimensions.

Original: Thus, the skills can encode much more complex behaviors than speed and direction of movement.
Revised: Thus, the skills can encode much more complex behaviors than speed and direction of movement.

Original: For example, Figureshows that the skill involves keeping one leg in the air on top of moving in the right direction.
Revised: For example, Figureshows that the skill involves keeping one leg in the air on top of moving in the right direction.

Original: We also show scatter plots for RND and ‘Disagreement’ in Appendix E.
Revised: We also show scatter plots for RND and ‘Disagreement’ in Appendix E.

Original: While the two methods also work well in the H UMANOID environment (c.f.
Revised: While the two methods also work well in the H UMANOID environment (c.f.

Original: Figures 14 and 15), they do not find any useful behavior in the A NT environment (c.f. Figures 11 and 12).
Revised: Figures 14 and 15), they do not find any useful behavior in the A NT environment (c.f. Figures 11 and 12).

Original: These approaches reward all novel states, they lead to policies that cover large regions of the state space, as can be seen in A NT .
Revised: These approaches reward all novel states, they lead to policies that cover large regions of the state space, as can be seen in A NT .

Original: We believe that the inherent instability of H UMANOID forces the policies to be more specific.
Revised: We believe that the inherent instability of H UMANOID forces the policies to be more specific.

Original: 4.2.2 Zero-Shot Transfer
Revised: 4.2.2 Zero-Shot Transfer

Original: After discovering 50 unsupervised skills, we identify the skill that aligns best with the reward given by the environment.
Revised: After discovering 50 unsupervised skills, we identify the skill that aligns best with the reward given by the environment.

Original: We report this zero-shotperformance in Table 1.
Revised: We report this zero-shotperformance in Table 1.

Original: As a baseline, we train an agent from scratch and measure how long it takes it to reach an equivalent performance.
Revised: As a baseline, we train an agent from scratch and measure how long it takes it to reach an equivalent performance.

Original: We use the Optuna hyper-parameter optimizer (Akiba et al., 2019) to find hyper-parameters which maximize the speed of learning beforehand.
Revised: We use the Optuna hyper-parameter optimizer (Akiba et al., 2019) to find hyper-parameters which maximize the speed of learning beforehand.

Original: The exact procedure can be found in Appendix D.
Revised: The exact procedure can be found in Appendix D.

Original: The simplest environment, H ALFC HEETAH , benefits the least from unsupervised learning, while the hardest one, H UMANOID , benefits the most.
Revised: The simplest environment, H ALFC HEETAH , benefits the least from unsupervised learning, while the hardest one, H UMANOID , benefits the most.

Original: We report the zero-shot performance of RND, ‘Disagreement’ and the ablations in Table 7 in the Appendix.
Revised: We report the zero-shot performance of RND, ‘Disagreement’ and the ablations in Table 7 in the Appendix.

Original: As already seen in Section 4.2.1, the intrinsic motivation methods do not find any interesting behavior in A NT .
Revised: As already seen in Section 4.2.1, the intrinsic motivation methods do not find any interesting behavior in A NT .

Original: Conversely, they perform well on the H UMANOID environment, the latter one even slightly outperforming our method 7 .
Revised: Conversely, they perform well on the H UMANOID environment, the latter one even slightly outperforming our method 7 .

Original: 4.2.3 Particle-based Mutual Information metric
Revised: 4.2.3 Particle-based Mutual Information metric

Original: [Table]
Revised: [Table]

Original: Measuring how well the agent can control relevant state dimensions is another way to track progress in Unsupervised reinforcement learning.
Revised: Measuring how well the agent can control relevant state dimensions is another way to track progress in Unsupervised reinforcement learning.

Original: This is measured using the mutual information between state dimensions and skills.
Revised: This is measured using the mutual information between state dimensions and skills.

Original: While high-dimensional estimation of mutual information is an active area of research, sampling can be an effective form of estimation in the 1dimensional case, c.f.
Revised: While high-dimensional estimation of mutual information is an active area of research, sampling can be an effective form of estimation in the 1dimensional case, c.f.

Original: Algorithm 1 in (Gu et al
Revised: Algorithm 1 in (Gu et al

Original: We report the particle-based mutual information for the x -velocity in Table 2, using the same bucketing strategy 8 as in (Gu et al., 2021).
Revised: We report the particle-based mutual information for the x -velocity in Table 2, using the same bucketing strategy 8 as in (Gu et al., 2021).

Original: Our method heavily outperforms DIAYN and DADS, when both methods look at the complete observation space.
Revised: Our method heavily outperforms DIAYN and DADS, when both methods look at the complete observation space.

Original: DIAYN and DADS achieve diversity by learning a set of skills that can be correctly labeled by a neural network.
Revised: DIAYN and DADS achieve diversity by learning a set of skills that can be correctly labeled by a neural network.

Original: In high-dimensional spaces this is easy to do by relying on a small subset of all state dimensions.
Revised: In high-dimensional spaces this is easy to do by relying on a small subset of all state dimensions.

Original: This leads to non-diverse behaviors across most state dimensions.
Revised: This leads to non-diverse behaviors across most state dimensions.

Original: Even when expert knowledge about relevant dimensions is supplied to other methods, i.e. only taking the x - and y -velocity into account, our method still fares well.
Revised: Even when expert knowledge about relevant dimensions is supplied to other methods, i.e. only taking the x - and y -velocity into account, our method still fares well.

Original: Particularly, in the most complex environment, H UMANOID , our method performs best.
Revised: Particularly, in the most complex environment, H UMANOID , our method performs best.

Original: We believe that the iterative increase in complexity leads to better coverage of hard to reach regions of the state like high speed or running backwards.
Revised: We believe that the iterative increase in complexity leads to better coverage of hard to reach regions of the state like high speed or running backwards.

Original: With all this, our approach achieves greater controllability of the x -velocity than
Revised: With all this, our approach achieves greater controllability of the x -velocity than

Original: DIAYN and DADS without any kind of feature engineering.
Revised: DIAYN and DADS without any kind of feature engineering.

Original: [Table]
Revised: [Table]

Original: Surprisingly, our novel 9 baselines,
Revised: Surprisingly, our novel 9 baselines,

Original: RND and ‘Disagreement’, also achieve better performance than
Revised: RND and ‘Disagreement’, also achieve better performance than

Original: DIAYN and DADS in the H UMANOID environment, but not in
Revised: DIAYN and DADS in the H UMANOID environment, but not in

Original: A NT .
Revised: A NT .

Original: As already discussed in
Revised: As already discussed in

Original: Section 4.2.1, we believe that the larger instability of H UMANOID compared to A NT is the reason for this difference in performance.
Revised: Section 4.2.1, we believe that the larger instability of H UMANOID compared to A NT is the reason for this difference in performance.

Original: 4.3 Montezuma’s Revenge
Revised: 4.3 Montezuma’s Revenge

Original: To show the generality of our approach we evaluate it on the notoriously hard Montezuma’s Revenge
Revised: To show the generality of our approach we evaluate it on the notoriously hard Montezuma’s Revenge

Original: Atari game.
Revised: Atari game.

Original: In this game, the agent controls a character in a complex 2d world with several rooms.
Revised: In this game, the agent controls a character in a complex 2d world with several rooms.

Original: Appendix G shows the initial room and various items with which the agent may interact.
Revised: Appendix G shows the initial room and various items with which the agent may interact.

Original: Same as in (Mnih et al., 2015), the observation is a stack of the last 4 frames.
Revised: Same as in (Mnih et al., 2015), the observation is a stack of the last 4 frames.

Original: This gives the agent information about speed and direction of movement.
Revised: This gives the agent information about speed and direction of movement.

Original: This is done for all networks, that is, the value, policy and neural re-
Revised: This is done for all networks, that is, the value, policy and neural re-

Original: ward networks.
Revised: ward networks.

Original: We use the same simple CNN architecture as in (Mnih et al., 2015) for all three networks.
Revised: We use the same simple CNN architecture as in (Mnih et al., 2015) for all three networks.

Original: For exact learning details see Appendix H.
Revised: For exact learning details see Appendix H.

Original: Our algorithm uses finite episode lengths because once it reaches a rewarding state, the agent can stay there forever.
Revised: Our algorithm uses finite episode lengths because once it reaches a rewarding state, the agent can stay there forever.

Original: Because of this, we reset the environment every 500 steps.
Revised: Because of this, we reset the environment every 500 steps.

Original: One of the main difficulties when dealing with Montezuma’s Revenge is that it cannot be simulated as fast as the other studied environments.
Revised: One of the main difficulties when dealing with Montezuma’s Revenge is that it cannot be simulated as fast as the other studied environments.

Original: On top of that, an agent can learn hundreds of different skills without ever leaving the first room.
Revised: On top of that, an agent can learn hundreds of different skills without ever leaving the first room.

Original: Finally, skills learned by our agent evolve from simple to very complex and extended in time.
Revised: Finally, skills learned by our agent evolve from simple to very complex and extended in time.

Original: In the beginning, the agent just needs to stay close to the initial position.
Revised: In the beginning, the agent just needs to stay close to the initial position.

Original: By the end of training, the agent learns to collect a key, open the door, avoid several enemies and visit four different rooms.
Revised: By the end of training, the agent learns to collect a key, open the door, avoid several enemies and visit four different rooms.

Original: In the most complex skills, it takes the agent several hundred steps to reach a rewarding state.
Revised: In the most complex skills, it takes the agent several hundred steps to reach a rewarding state.

Original: This means that experiments can take several days to visit a different room.
Revised: This means that experiments can take several days to visit a different room.

Original: [Figure]
Revised: [Figure]

Original: We found out that the first thing the agent learns is to get the life counter to 0 .
Revised: We found out that the first thing the agent learns is to get the life counter to 0 .

Original: Then it starts exploring the room, with 0 lives already.
Revised: Then it starts exploring the room, with 0 lives already.

Original: This is because losing a life happens very easily during random exploration and because once the agent reaches 0 lives, all future positive samples will have 0 lives.
Revised: This is because losing a life happens very easily during random exploration and because once the agent reaches 0 lives, all future positive samples will have 0 lives.

Original: This has two side effects.
Revised: This has two side effects.

Original: On the one hand, the initial 100 − 200 steps of an episode are spent losing lives.
Revised: On the one hand, the initial 100 − 200 steps of an episode are spent losing lives.

Original: On the other hand, exploration is harder, as the agent will reach a terminal state very easily.
Revised: On the other hand, exploration is harder, as the agent will reach a terminal state very easily.

Original: Because of this, we cut out the part of the image that shows the remaining lives, see Appendix G.
Revised: Because of this, we cut out the part of the image that shows the remaining lives, see Appendix G.

Original: Figure 7(a) visualizes different skills.
Revised: Figure 7(a) visualizes different skills.

Original: The first observations which get the maximum possible reward are overlaid.
Revised: The first observations which get the maximum possible reward are overlaid.

Original: All the shown skills go to the bottom and kill the skull first.
Revised: All the shown skills go to the bottom and kill the skull first.

Original: Then, they move to a specific location.
Revised: Then, they move to a specific location.

Original: It can be seen that later skills explore states further and further away from the initial state.
Revised: It can be seen that later skills explore states further and further away from the initial state.

Original: Figure 7(b) shows that the skill receives no reward for many steps.
Revised: Figure 7(b) shows that the skill receives no reward for many steps.

Original: Only after the agent has killed the skull being on the ladder becomes rewarding 10 .
Revised: Only after the agent has killed the skull being on the ladder becomes rewarding 10 .

Original: This shows that both the reward network and the agent have learned a lot of the concepts that are necessary to tackle Montezuma’s Revenge.
Revised: This shows that both the reward network and the agent have learned a lot of the concepts that are necessary to tackle Montezuma’s Revenge.

Original: This includes controlling the agent in the 2d environment, killing enemies and interacting with different objects.
Revised: This includes controlling the agent in the 2d environment, killing enemies and interacting with different objects.

Original: All of this without ever accessing the original reward function.
Revised: All of this without ever accessing the original reward function.

Original: Eventually, after learning a few hundred skills, the neural reward function pushes the rewarding states all the way to the fourth room.
Revised: Eventually, after learning a few hundred skills, the neural reward function pushes the rewarding states all the way to the fourth room.

Original: This video shows a run of this skill.
Revised: This video shows a run of this skill.

Original:  Limitations
Revised:  Limitations

Original: Our approach has three main limitations: the randomized exploration strategy, the re-learning necessary for each new skill and the depth-first nature of the search of skills.
Revised: Our approach has three main limitations: the randomized exploration strategy, the re-learning necessary for each new skill and the depth-first nature of the search of skills.

Original: We have shown that the random exploration allows for discovering diverse skills.
Revised: We have shown that the random exploration allows for discovering diverse skills.

Original: However, it can lead to slow progress in complex regions of the state space.
Revised: However, it can lead to slow progress in complex regions of the state space.

Original: Using more sophisticated exploration policies could lead to faster evolution of the skills.
Revised: Using more sophisticated exploration policies could lead to faster evolution of the skills.

Original: A potential avenue for improving this is using previously discovered skills as exploration policies.
Revised: A potential avenue for improving this is using previously discovered skills as exploration policies.

Original: As stated in Section 3.2, we use multiple techniques to make the learning of the next policy more efficient.
Revised: As stated in Section 3.2, we use multiple techniques to make the learning of the next policy more efficient.

Original: Still, the vast majority of our compute is used for learning these policies.
Revised: Still, the vast majority of our compute is used for learning these policies.

Original: Using more advanced transfer learning or even meta-learning algorithms, could make the policy learning more efficient.
Revised: Using more advanced transfer learning or even meta-learning algorithms, could make the policy learning more efficient.

Original: Another possibility would be to relabel previously seen trajectories with ideas similar to (Rauber et al., 2017).
Revised: Another possibility would be to relabel previously seen trajectories with ideas similar to (Rauber et al., 2017).

Original: Finally, our approach only uses the current skill to create the next one.
Revised: Finally, our approach only uses the current skill to create the next one.

Original: On a high level, it uses a single search point at each time.
Revised: On a high level, it uses a single search point at each time.

Original: Then, the iterative increase in complexity of the skills leads to a
Revised: Then, the iterative increase in complexity of the skills leads to a

Original: DFS-like exploration.
Revised: DFS-like exploration.

Original: This may not always be optimal.
Revised: This may not always be optimal.

Original: For example, in Montezuma’s Revenge it causes the agent to explore only one of the two possible exits of the first room.
Revised: For example, in Montezuma’s Revenge it causes the agent to explore only one of the two possible exits of the first room.

Original: On top of that, our algorithm can sometimes get stuck in places were backtracking is complicated or impossible, see
Revised: On top of that, our algorithm can sometimes get stuck in places were backtracking is complicated or impossible, see

Original: Figure 9 in the Appendix.
Revised: Figure 9 in the Appendix.

Original: This also happens in Montezuma’s Revenge when the life-counter is not removed.
Revised: This also happens in Montezuma’s Revenge when the life-counter is not removed.

Original: Figuring out smart back-tracking strategies or keeping multiple search points could address this problem and make our algorithm even more efficient.
Revised: Figuring out smart back-tracking strategies or keeping multiple search points could address this problem and make our algorithm even more efficient.

Original:  Conclusion
Revised:  Conclusion

Original: We have presented an unsupervised reinforcement learning algorithm that uses reward functions encoded by neural networks.
Revised: We have presented an unsupervised reinforcement learning algorithm that uses reward functions encoded by neural networks.

Original: Our algorithm alternates between increasing the complexity of the reward function and transferring previous knowledge to learn a new skill that finds rewarding states.
Revised: Our algorithm alternates between increasing the complexity of the reward function and transferring previous knowledge to learn a new skill that finds rewarding states.

Original: This allows it to learn an unbounded number of skills.
Revised: This allows it to learn an unbounded number of skills.

Original: We have thoroughly tested the different components of our model in a 2d navigation task.
Revised: We have thoroughly tested the different components of our model in a 2d navigation task.

Original: This has allowed us to better understand our method in practice.
Revised: This has allowed us to better understand our method in practice.

Original: We have shown that our method works both with high dimensional feature inputs, in robotic environments, and pixel inputs, in
Revised: We have shown that our method works both with high dimensional feature inputs, in robotic environments, and pixel inputs, in

Original: Montezuma’s Revenge.
Revised: Montezuma’s Revenge.

Original: Our algorithm learned a diverse set of skills in both settings.
Revised: Our algorithm learned a diverse set of skills in both settings.

Original: In H UMANOID and Montezuma’s Revenge, skills found by our method achieve a zero-shot performance that takes millions of steps to learn in the classical reinforcement learning setup.
Revised: In H UMANOID and Montezuma’s Revenge, skills found by our method achieve a zero-shot performance that takes millions of steps to learn in the classical reinforcement learning setup.

Original: We believe our algorithm is one step in a direction that may one day allow reinforcement learning agents to fully understand an environment without making use of any predefined reward function.
Revised: We believe our algorithm is one step in a direction that may one day allow reinforcement learning agents to fully understand an environment without making use of any predefined reward function.

Original: Just like in Computer Vision and Natural Language Processing, this will lead to agents that need very few labels from the task at hand to be able to solve it and will drastically expand the applicability of reinforcement learning.
Revised: Just like in Computer Vision and Natural Language Processing, this will lead to agents that need very few labels from the task at hand to be able to solve it and will drastically expand the applicability of reinforcement learning.

Original: Acknowledgements
Revised: Acknowledgements

Original: We would like to thank Xun Zuo for helping with the implementation of the maze environment, on top of many great discussions.
Revised: We would like to thank Xun Zuo for helping with the implementation of the maze environment, on top of many great discussions.

Original: We would also like to thank Frederik Benzing and Yassir Akram for many fruitful discussions.
Revised: We would also like to thank Frederik Benzing and Yassir Akram for many fruitful discussions.

Original: Finally, we also thank the anonymous reviewers for the thorough and insightful reviews and discussions.
Revised: Finally, we also thank the anonymous reviewers for the thorough and insightful reviews and discussions.

Original: Robert Meier and Asier Mujika were supported by grant no.
Revised: Robert Meier and Asier Mujika were supported by grant no.

Original: CRSII5 173721 of the Swiss National Science Foundation
Revised: CRSII5 173721 of the Swiss National Science Foundation

Original: B 2d Navigation: Experimental details

Original: The episode length in the 2d navigation task is 250 steps.

Original: The guiding phase lasts for 2 / 3 of the total steps of the 1-steps A2C agent learning.

Original: We sample the guiding length uniformly at random in theto 200 range.

Original: For the negative samples we follow the learnt policy for 200 steps.

Original: For the positive samples, we take random actions for 50 steps after following the policy for 200 steps.

Original: [Table]

Original: C Robotic environments: Architecture and hyper-parameters

Original: We use the PPO (Schulman et al., 2017) implementation from (Freeman et al., 2021) with modifications such that it allows our training method to work.

Original: In Table 4 and Table 5 we list the hyper-parameters we used.

Original: Note that we use a smaller learning rate for the bodies of the policy and value network.

Original: We found that this was beneficial for transferring more knowledge from the previous skill.

Original: The environment specific parameters were found using Optuna (Akiba et al., 2019).

Original: For each environment we ran a search to optimize final performance on the environment rewards (running in positive x -direction).

Original: We used 200 runs and trained each of them for the same number of steps as a

Original: [Table]

Original: [Table]

Original: skill in our method.

Original: This yielded hyper-parameters which are able to learn tasks in the corresponding environment.

Original: We did not take the downstream performance metrics (zero-shot performance and particle-based information) into account.

Original: Doing so would have exceeded our compute budget.

Original: In

Original: Table 6 the architecture and hyper-parameters used for the neural reward functions are listed.

Original: For the supervised reward training we use the Adam optimizer (Kingma & Ba, 2014).

Original: All code can be found in the supplementary material

Original: [Table]

Original: For DADS (Sharma et al., 2019) we used the code provided by the authors.

Original: For RND (Burda et al., 2018b) and ‘Disagreement’ (Pathak et al., 2019)

Original: we tuned the method specific hyper-parameters optimizing for the MI-metric of the x -velocity.

Original: In particular, for each environment and method we used 100 runs of 200 M environment steps per run.

Original: D Robotic environments: Zero-Shot Transfer

Original: We use the Optuna hyperparameter tuning library (Akiba et al., 2019) to create Table 1.

Original: We ran multiple optimization procedures.

Original: For each one, we fix a number of environment steps, then optimize the final performance on the environment reward over 200 runs.

Original: We iteratively increased the number of timesteps until the highest score of the 200 runs beats our averaged zero-shot performance.

Original: We take the hyperparemters of that run and train 10 agents until they outperform our averaged zero-shot performance.

Original: The average number of training steps needed for this is reported in Table 1.

Original: In the

Original: H UMANOID environment three of the runs did not reach the score in 100 M steps, at which point we stopped training.

Original: We nonetheless took 100 M into the average.

Original: The code for the hyperparameter search and the results of our conducted studies can be found in the supplementary material.

Original: E Robotic environments: Additional plots and Tables

Original: In Figures 10, 11 respectively 12 we show more scatter plots for the A NT environment of our method,

Original: RND (Burda et al., 2018b) respectively ‘Disagreement’ (Pathak et al., 2019).

Original: In Figures 13,respectively

Original: we show more scatter plots for the H UMANOID environment of our method, RND respectively ‘Disagreement’.

Original: In Table 7 we report the zero-shot results for the ablations and the intrinsic curiosity baselines.

Original: In

Original: Table 8 we report the MI-metric results for the ablations.

Original: [Table]

Original: EWARD

Original: [Table]

Original: HYPERPAREMETERS

Original: [Table]

Original: BRAX

Original: [Table]

Original: NEGATIVE

Original: [Table]

Original: [Table]

Original: [Table]

Original: F Montezuma’s Revenge: Adapting episode length

Original: In order to save computation

Original: [Table]

Original: we adapt the number of training steps according to several criteria.

Original: This lets us save a lot of compute on the skills which are easy to learn.

Original: We use the following measures:

Original: • We train for 1 M steps with guiding from the previous policy.

Original: The number of guiding steps is sampled uniformly at random between 0 and 450 , each time the environment is reset.

Original: Then we train for another 0 .

Original: 65 M steps without any guiding.

Original: • If the average reward goes down after removing the guiding or is too low at any point, we restart the guiding phase for 0 .

Original: 65 M steps.

Original: • If the agent is reaching a terminal state in more than 10% of the episodes, we continue training.

Original: On top of this, we ignore positive samples that receive almost no reward.

Original: All these tricks enable us to considerably reduce the training time.

Original: However, they are not a core change in our algorithm as they could all be replaced by just training all generations for a longer fixed number of steps, just as before.

Original: G Montezuma’s Revenge: Environment details

Original: Figure 8 shows the initial state of Montezuma’s Revenge and the cropped version of it that is used for our agent.

Original: [Figure]

Original: H Montezuma’s Revenge: Experimental details

Original: In Montezuma’s Revenge

Original: [Figure]

Original: we use the PPO implementation from the coax 11 library, with 1-step temporal differences.

Original: If the average score per 500 steps is below 5 , we go back to the guiding phase.

Original: Due to the large memory requirements to store all negative samples

Original: [Figure]

Original: after the 15 -th epoch we rewrite old negative samples to add the new ones.

Original: Table 9 summarizes our hyper-parameters.

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: 11 https://github.com/coax-dev/coax

Original: I Additional 2d navigation experiments

Original: In Figure 9 we show results of the 2d navigation task with three alternative environments.

Original: These environments illustrate the behavior of our algorithm in environments that require backtracking.

Original: In particular, Figure 9(b) shows a failure mode of our approach.

Original: A trap prevents the agent from backtracking and thus stops the progress of our algorithm once the search reaches that region.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]