Original: Open-Ended Reinforcement Learning with Neural Reward Functions
Revised: Open-Ended Reinforcement Learning with Neural Reward Functions

Original: Abstract
Revised: Abstract

Original: Inspired by the success of unsupervised learning in Computer Vision and Natural
Revised: Inspired by the success of unsupervised learning in Computer Vision and Natural

Original: Language Processing, the reinforcement learning community has recently startedto focus more on unsupervised discovery of skills.
Revised: Language Processing, the reinforcement learning community has recently startedto focus more on unsupervised discovery of skills.

Original: Most current approaches, like
Revised: Most current approaches, like

Original: DIAYN or DADS, optimize some form of mutual information objective.
Revised: DIAYN or DADS, optimize some form of mutual information objective.

Original: We propose a different approach that uses reward functions encoded by neural networks.
Revised: We propose a different approach that uses reward functions encoded by neural networks.

Original: These are trained iteratively to reward more complex behavior.
Revised: These are trained iteratively to reward more complex behavior.

Original: In high-dimensionalrobotic environments our approach learns a wide range of interesting skills including front-flips for H ALF -C HEETAH and one-legged running for H UMANOID .
Revised: In high-dimensionalrobotic environments our approach learns a wide range of interesting skills including front-flips for H ALF -C HEETAH and one-legged running for H UMANOID .

Original: It isthe first skill discovery algorithm that can learn such skills without relying on anyform of feature engineering.
Revised: It isthe first skill discovery algorithm that can learn such skills without relying on anyform of feature engineering.

Original: In the pixel-based Montezuma’s Revenge environmentour method also works with minimal changes and it learns complex skills thatinvolve interacting with items and visiting diverse locations.
Revised: In the pixel-based Montezuma’s Revenge environmentour method also works with minimal changes and it learns complex skills thatinvolve interacting with items and visiting diverse locations.

Original:  Introduction
Revised:  Introduction

Original: Deep reinforcement learning (RL) has proven to be very successful in many challenging tasks (Mnihet al., 2015;
Revised: Deep reinforcement learning (RL) has proven to be very successful in many challenging tasks (Mnihet al., 2015;

Original: Silver et al., 2017b; Berner et al., 2019).
Revised: Silver et al., 2017b; Berner et al., 2019).

Original: These were considered intractable just afew years ago.
Revised: These were considered intractable just afew years ago.

Original: However, current methods require enormous amounts of compute to achieve greatperformance in individual tasks.
Revised: However, current methods require enormous amounts of compute to achieve greatperformance in individual tasks.

Original: Most of the time, previous models can not be utilised when newtasks are considered, even if the environment does not change.
Revised: Most of the time, previous models can not be utilised when newtasks are considered, even if the environment does not change.

Original: This was also the case in Computer
Revised: This was also the case in Computer

Original: Vision and Natural Language Processing.
Revised: Vision and Natural Language Processing.

Original: However, recently unsupervised learning has been shownto be very effective in both fields.
Revised: However, recently unsupervised learning has been shownto be very effective in both fields.

Original: By using task agnostic pre-training schemes (Caron et al., 2021;
Revised: By using task agnostic pre-training schemes (Caron et al., 2021;

Original: Tenney et al., 2019; Brown et al., 2020), unsupervised models can solve most tasks with minimalor even no fine-tuning.
Revised: Tenney et al., 2019; Brown et al., 2020), unsupervised models can solve most tasks with minimalor even no fine-tuning.

Original: Unsupervised reinforcement learning aims to bring similar successes to thereinforcement learning community.
Revised: Unsupervised reinforcement learning aims to bring similar successes to thereinforcement learning community.

Original: Agents which learn a wide range of unsupervised skills may beable to leverage those to solve new tasks faster and with minimal fine-tuning.
Revised: Agents which learn a wide range of unsupervised skills may beable to leverage those to solve new tasks faster and with minimal fine-tuning.

Original: Here, we propose a new method for open-ended, unsupervised skill discovery.
Revised: Here, we propose a new method for open-ended, unsupervised skill discovery.

Original: We devise an iterativeprocess which creates pairs of neural reward functions and policies.
Revised: We devise an iterativeprocess which creates pairs of neural reward functions and policies.

Original: The policy optimizes thecorresponding reward function.
Revised: The policy optimizes thecorresponding reward function.

Original: Each of them corresponds to a skill that the agent has learned.
Revised: Each of them corresponds to a skill that the agent has learned.

Original: Theneural reward function is a neural network that maps the current observation to a scalar reward.
Revised: Theneural reward function is a neural network that maps the current observation to a scalar reward.

Original: Ineach iteration, the neural reward function is modified to differ from the previous one.
Revised: Ineach iteration, the neural reward function is modified to differ from the previous one.

Original: This resultsin increasing the complexity of the encoded task.
Revised: This resultsin increasing the complexity of the encoded task.

Original: Then, a new skill is learned that optimizes thisreward function.
Revised: Then, a new skill is learned that optimizes thisreward function.

Original: We devise several techniques to transfer the knowledge from previously learnedskills.
Revised: We devise several techniques to transfer the knowledge from previously learnedskills.

Original: These mechanisms enable learning the most complex reward functions that our method creates.
Revised: These mechanisms enable learning the most complex reward functions that our method creates.

Original: In fact, we show that some of the functions are impossible to learn from scratch.
Revised: In fact, we show that some of the functions are impossible to learn from scratch.

Original: We empirically test our framework in a diverse set of environments.
Revised: We empirically test our framework in a diverse set of environments.

Original: First, we apply it to a simple 2dnavigation task.
Revised: First, we apply it to a simple 2dnavigation task.

Original: This lets us perform experiments quickly and gain a solid empirical understanding ofthe different components of our method.
Revised: This lets us perform experiments quickly and gain a solid empirical understanding ofthe different components of our method.

Original: Then, we apply it to three robotic environments where we have continuous high-dimensional observations and actions.
Revised: Then, we apply it to three robotic environments where we have continuous high-dimensional observations and actions.

Original: In contrast to previous methods, ourapproach can deal with high-dimensional input directly without feature engineering.
Revised: In contrast to previous methods, ourapproach can deal with high-dimensional input directly without feature engineering.

Original: Finally, weapply it on the challenging Montezuma’s Revenge Atari game which has visually rich pixel basedobservations and discrete actions.
Revised: Finally, weapply it on the challenging Montezuma’s Revenge Atari game which has visually rich pixel basedobservations and discrete actions.

Original: Despite their differences, our method manages to learn useful andinteresting skills in all of them.
Revised: Despite their differences, our method manages to learn useful andinteresting skills in all of them.

Original: This shows that neural reward functions are equipped to encodemeaningful tasks in diverse environments.
Revised: This shows that neural reward functions are equipped to encodemeaningful tasks in diverse environments.

Original: We highlight the most important findings of this paper:
Revised: We highlight the most important findings of this paper:

Original: • Propose the first skill-discovery algorithm to work in high-dimensional environments without
Revised: • Propose the first skill-discovery algorithm to work in high-dimensional environments without

Original: prior expert knowledge.
Revised: prior expert knowledge.

Original: • Solve a 2d maze that cannot be solved by using random exploration.
Revised: • Solve a 2d maze that cannot be solved by using random exploration.

Original: • Acquire complex skills including performing front- and back-flip in the H ALF -C HEETAH ,running in all directions in A NT and H UMANOID , standing and jumping on one leg inH UMANOID .•
Revised: • Acquire complex skills including performing front- and back-flip in the H ALF -C HEETAH ,running in all directions in A NT and H UMANOID , standing and jumping on one leg inH UMANOID .•

Original: Learn to run as fast in H UMANOID as a supervised agent trained for tens of millions of steps.
Revised: Learn to run as fast in H UMANOID as a supervised agent trained for tens of millions of steps.

Original: • Achieve a higher Particle-Based Mutual Information Metric (Gu et al., 2021) than approachesthat explicitly optimize this metric.• Collect the first key and reach several rooms in Montezuma’s Revenge.
Revised: • Achieve a higher Particle-Based Mutual Information Metric (Gu et al., 2021) than approachesthat explicitly optimize this metric.• Collect the first key and reach several rooms in Montezuma’s Revenge.

Original:  Related Work
Revised:  Related Work

Original: Our work fits best into the unsupervised skill discovery literature (Mohamed & Rezende, 2015;
Revised: Our work fits best into the unsupervised skill discovery literature (Mohamed & Rezende, 2015;

Original: Gregor et al., 2016; Florensa et al., 2017a; Achiam et al., 2018;
Revised: Gregor et al., 2016; Florensa et al., 2017a; Achiam et al., 2018;

Original: Eysenbach et al., 2018; Sharma et al.
Revised: Eysenbach et al., 2018; Sharma et al.

Original: ,2019; Choi et al., 2021).
Revised: ,2019; Choi et al., 2021).

Original: Compared to DIAYN (Eysenbach et al., 2018) and similar approaches, wedo not set the number of skills to be learned at the beginning of training.
Revised: Compared to DIAYN (Eysenbach et al., 2018) and similar approaches, wedo not set the number of skills to be learned at the beginning of training.

Original: This allows us to learnnew skills in an open-ended fashion.
Revised: This allows us to learnnew skills in an open-ended fashion.

Original: On top of that, maximizing mutual information can lead todegenerate behaviours in high-dimensional environments.
Revised: On top of that, maximizing mutual information can lead todegenerate behaviours in high-dimensional environments.

Original: By manipulating a small subset of thedimensions, a lot of information can be encoded, without exploring the rest of the state space.
Revised: By manipulating a small subset of thedimensions, a lot of information can be encoded, without exploring the rest of the state space.

Original: Due tothis issue, previous methods only consider a hand-picked subset of the dimensions to perform well.
Revised: Due tothis issue, previous methods only consider a hand-picked subset of the dimensions to perform well.

Original: Open-ended learning (Wang et al., 2019; Campero et al., 2020; Dennis et al., 2020; Wang et al., 2020;
Revised: Open-ended learning (Wang et al., 2019; Campero et al., 2020; Dennis et al., 2020; Wang et al., 2020;

Original: Ecoffet et al., 2021; Stooke et al., 2021) is closely related to unsupervised skill discovery.
Revised: Ecoffet et al., 2021; Stooke et al., 2021) is closely related to unsupervised skill discovery.

Original: However,most approaches require either a parameterizable environment (Wang et al., 2019, 2020), some fixedencoding of tasks (Stooke et al., 2021) or self-competition
Revised: However,most approaches require either a parameterizable environment (Wang et al., 2019, 2020), some fixedencoding of tasks (Stooke et al., 2021) or self-competition

Original: (Silver et al., 2017a; Baker et al., 2019).
Revised: (Silver et al., 2017a; Baker et al., 2019).

Original: This limits the applicability to environments that are engineered with these restrictions in mind.
Revised: This limits the applicability to environments that are engineered with these restrictions in mind.

Original: Incontrast, neural networks are universal function approximators and thus, our approach can encodeany possible task in any possible environment, as long as the input is chosen appropriately.
Revised: Incontrast, neural networks are universal function approximators and thus, our approach can encodeany possible task in any possible environment, as long as the input is chosen appropriately.

Original: Similarto our method, the Go-Explore algorithm (Ecoffet et al., 2021) also explores around the frontier ofthe known.
Revised: Similarto our method, the Go-Explore algorithm (Ecoffet et al., 2021) also explores around the frontier ofthe known.

Original: It uses a handcrafted feature map to group similar states.
Revised: It uses a handcrafted feature map to group similar states.

Original: Instead, we use neural rewardfunctions to group them as skills.
Revised: Instead, we use neural rewardfunctions to group them as skills.

Original: This does not use any expert knowledge.
Revised: This does not use any expert knowledge.

Original: Another related line of research to our approach is intrinsic motivation (Stadie et al., 2015; Bellemareet al., 2016; Pathak et al., 2017; Burda et al., 2018b,a; Raileanu & Rockt¨aschel, 2020).
Revised: Another related line of research to our approach is intrinsic motivation (Stadie et al., 2015; Bellemareet al., 2016; Pathak et al., 2017; Burda et al., 2018b,a; Raileanu & Rockt¨aschel, 2020).

Original: Theseapproaches have managed great success in hard-exploration Atari games.
Revised: Theseapproaches have managed great success in hard-exploration Atari games.

Original: However, these approachesdo not learn discrete skills that can be composed or fine-tuned for fast learning of new tasks.
Revised: However, these approachesdo not learn discrete skills that can be composed or fine-tuned for fast learning of new tasks.

Original: This haslimited their applicability to robotic environments where exploration is not usually the limiting factor.
Revised: This haslimited their applicability to robotic environments where exploration is not usually the limiting factor.

Original: There has been previous work using neural networks to output reward to RL agents.
Revised: There has been previous work using neural networks to output reward to RL agents.

Original: Compared toour work, they use supervision to train the reward function.
Revised: Compared toour work, they use supervision to train the reward function.

Original: Many of them train a reward functiontrained with states/trajectories labeled by supervisors in some fashion (Abbeel & Ng, 2004; Fu et al.
Revised: Many of them train a reward functiontrained with states/trajectories labeled by supervisors in some fashion (Abbeel & Ng, 2004; Fu et al.

Original: ,2017; Singh et al., 2019;
Revised: ,2017; Singh et al., 2019;

Original: Li et al., 2021).
Revised: Li et al., 2021).

Original: Other approaches train auxiliary rewards with meta-learning(Zheng et al., 2018; Du et al., 2019; Veeriah et al., 2019) to enhance the learning of the originalreward function.
Revised: Other approaches train auxiliary rewards with meta-learning(Zheng et al., 2018; Du et al., 2019; Veeriah et al., 2019) to enhance the learning of the originalreward function.

Original: Another approach to train multiple behaviours is goal-conditioned learning (Kaelbling, 1993; Schaulet al., 2015; Andrychowicz et al., 2017; Rauber et al., 2017; Nair et al., 2018;
Revised: Another approach to train multiple behaviours is goal-conditioned learning (Kaelbling, 1993; Schaulet al., 2015; Andrychowicz et al., 2017; Rauber et al., 2017; Nair et al., 2018;

Original: Veeriah et al., 2018;
Revised: Veeriah et al., 2018;

Original: Warde-Farley et al., 2018; Pong et al., 2019; Choi et al., 2021).
Revised: Warde-Farley et al., 2018; Pong et al., 2019; Choi et al., 2021).

Original: In automated curriculum learning(Bengio et al., 2009; Florensa et al., 2017b; Forestier et al., 2017; Graves et al., 2017; Sukhbaataret al., 2017; Florensa et al., 2018; Matiisen et al., 2019; Narvekar et al., 2020; Portelas et al., 2020a,b;
Revised: In automated curriculum learning(Bengio et al., 2009; Florensa et al., 2017b; Forestier et al., 2017; Graves et al., 2017; Sukhbaataret al., 2017; Florensa et al., 2018; Matiisen et al., 2019; Narvekar et al., 2020; Portelas et al., 2020a,b;

Original: Zhang et al., 2020), a sequence of goals is created such that each of them is not too hard nor too easy for the current agent.
Revised: Zhang et al., 2020), a sequence of goals is created such that each of them is not too hard nor too easy for the current agent.

Original: These approaches mostly rely on low-dimensional goal embeddings.
Revised: These approaches mostly rely on low-dimensional goal embeddings.

Original: Whendealing with high-dimensional observations, they must use dimensionality reduction techniques.
Revised: Whendealing with high-dimensional observations, they must use dimensionality reduction techniques.

Original: These techniques can introduce instabilities or destroy relevant information from the input.
Revised: These techniques can introduce instabilities or destroy relevant information from the input.

Original: Ourapproach, on the other hand, can deal with high-dimensional inputs directly.
Revised: Ourapproach, on the other hand, can deal with high-dimensional inputs directly.

Original: On top of that, goalsencode a narrow region of the state space, while each of our reward functions can be rewarding in alarge region.
Revised: On top of that, goalsencode a narrow region of the state space, while each of our reward functions can be rewarding in alarge region.

Original: This speeds-up the exploration in ’easy’ regions of the state space.
Revised: This speeds-up the exploration in ’easy’ regions of the state space.

Original:  Method
Revised:  Method

Original: [Figure]
Revised: [Figure]

Original: We introduce a method that performs open-ended, unsupervised skill discovery.
Revised: We introduce a method that performs open-ended, unsupervised skill discovery.

Original: It iteratively createspairs of neural reward functions R ψ and policies π θ trained to maximize the corresponding R ψ .
Revised: It iteratively createspairs of neural reward functions R ψ and policies π θ trained to maximize the corresponding R ψ .

Original: Our proposed method alternates between increasing the complexity of the reward function R ψ andleveraging the previously learned skills to learn a policy π θ that can solve the new R ψ .
Revised: Our proposed method alternates between increasing the complexity of the reward function R ψ andleveraging the previously learned skills to learn a policy π θ that can solve the new R ψ .

Original: This yields ageneral learning procedure that learns complex skills in a diverse set of environments.
Revised: This yields ageneral learning procedure that learns complex skills in a diverse set of environments.

Original: See Figure 1for a high level overview.
Revised: See Figure 1for a high level overview.

Original: 3.1 Increasing the complexity of R ψ
Revised: 3.1 Increasing the complexity of R ψ

Original: [Figure]
Revised: [Figure]

Original: The reward function in a fully observable
Revised: The reward function in a fully observable

Original: Markov Decision Process (MDP) is by definitiona function of the current observation, the nextone and the action that was performed.
Revised: Markov Decision Process (MDP) is by definitiona function of the current observation, the nextone and the action that was performed.

Original: However,in many cases this can be reduced to a function of just the current observation.
Revised: However,in many cases this can be reduced to a function of just the current observation.

Original: Because of this, weopt for these simpler reward functions as the basis of our R ψ .
Revised: Because of this, weopt for these simpler reward functions as the basis of our R ψ .

Original: In our method, the reward function
Revised: In our method, the reward function

Original: R ψ is a neural network which takes observations o t as input and outputs a single scalar value, theinstantaneous reward r ψt .
Revised: R ψ is a neural network which takes observations o t as input and outputs a single scalar value, theinstantaneous reward r ψt .

Original: Assume that we have a policy π θ that can reach states in the MDP which are rewarding under R ψ .
Revised: Assume that we have a policy π θ that can reach states in the MDP which are rewarding under R ψ .

Original: Toincrease the complexity of the reward function, we do the following:
Revised: Toincrease the complexity of the reward function, we do the following:

Original: • Decrease the reward of states which are visited by π θ , as we already have a policy thatreaches these states.
Revised: • Decrease the reward of states which are visited by π θ , as we already have a policy thatreaches these states.

Original: We create a data set O neg of such states.
Revised: We create a data set O neg of such states.

Original: We refer to these states asnegative samples .•
Revised: We refer to these states asnegative samples .•

Original: Increase the reward of states that can almost be reached by the current policy.
Revised: Increase the reward of states that can almost be reached by the current policy.

Original: This
Revised: This

Original: allowsus to leverage π θ to learn the new reward function.
Revised: allowsus to leverage π θ to learn the new reward function.

Original: We create a data set O pos of such states.
Revised: We create a data set O pos of such states.

Original: We refer to these states as positive samples .
Revised: We refer to these states as positive samples .

Original: To generate the negative samples, we run π θ for a given number of steps (ideally until it reachesrewarding states) and store the visited states.
Revised: To generate the negative samples, we run π θ for a given number of steps (ideally until it reachesrewarding states) and store the visited states.

Original: To then generate positive samples, we change toperforming random actions 1 for a fixed number of steps.
Revised: To then generate positive samples, we change toperforming random actions 1 for a fixed number of steps.

Original: To ensure that the new reward function isdifferent from all previous ones, we also keep track of all the negative samples that we have collectedfor all skills in a data set
Revised: To ensure that the new reward function isdifferent from all previous ones, we also keep track of all the negative samples that we have collectedfor all skills in a data set

Original: O neg all .
Revised: O neg all .

Original: Finally, we set target values a and − a for the positive and negative samples respectively, and train thereward network using standard supervised learning on the following loss:
Revised: Finally, we set target values a and − a for the positive and negative samples respectively, and train thereward network using standard supervised learning on the following loss:

Original: [Equation]
Revised: [Equation]

Original: This loss ensures that positive samples that have never been seen before have positive reward inthe next R ψ , while all other samples that have been seen before decrease their reward.
Revised: This loss ensures that positive samples that have never been seen before have positive reward inthe next R ψ , while all other samples that have been seen before decrease their reward.

Original: In thereinforcement learning phase we clip rewards to the [0 , a ] range.
Revised: In thereinforcement learning phase we clip rewards to the [0 , a ] range.

Original: This ensures that the agent seekspositive samples, rather than less negative ones.
Revised: This ensures that the agent seekspositive samples, rather than less negative ones.

Original: 3.2 Forward transfer for π θ
Revised: 3.2 Forward transfer for π θ

Original: Given the procedure presented in Section 3.1, we create increasingly complex reward functions.
Revised: Given the procedure presented in Section 3.1, we create increasingly complex reward functions.

Original: While this is great for open-ended learning, it eventually leads to skills that are too complex and cannot be learned from scratch.
Revised: While this is great for open-ended learning, it eventually leads to skills that are too complex and cannot be learned from scratch.

Original: In order to learn these skills, we must leverage previous knowledgeabout the environment.
Revised: In order to learn these skills, we must leverage previous knowledgeabout the environment.

Original: In this section we present several forward transfer mechanisms that arenecessary for the most complex skills.
Revised: In this section we present several forward transfer mechanisms that arenecessary for the most complex skills.

Original: In principle, our method can be combined with any standard reinforcement learning (RL) technique.
Revised: In principle, our method can be combined with any standard reinforcement learning (RL) technique.

Original: But here, we focus on actor-critic methods like Advantage Actor-Critic (Mnih et al., 2016) or Proximal
Revised: But here, we focus on actor-critic methods like Advantage Actor-Critic (Mnih et al., 2016) or Proximal

Original: Policy Optimization (Schulman et al., 2017) for learning π θ for several reasons.
Revised: Policy Optimization (Schulman et al., 2017) for learning π θ for several reasons.

Original: These methods havea value network that is separate from the policy which allows us to use different transfer mechanismsfor the value and policy network.
Revised: These methods havea value network that is separate from the policy which allows us to use different transfer mechanismsfor the value and policy network.

Original: Also, these approaches work for both continuous and discreteaction spaces.
Revised: Also, these approaches work for both continuous and discreteaction spaces.

Original: This allows us to use the same technique for the robotic environments and for the 2dnavigation tasks.
Revised: This allows us to use the same technique for the robotic environments and for the 2dnavigation tasks.

Original: Finally the learned policies are stochastic which increases the diversity of negativesamples and speeds up the skill discovery process.
Revised: Finally the learned policies are stochastic which increases the diversity of negativesamples and speeds up the skill discovery process.

Original: See Section 4.1.2.
Revised: See Section 4.1.2.

Original: We present our three forward transfer mechanisms below.
Revised: We present our three forward transfer mechanisms below.

Original: All of them exploit the similarity betweenconsecutive reward functions to ensure that even very complex reward functions can be solved by the
Revised: All of them exploit the similarity betweenconsecutive reward functions to ensure that even very complex reward functions can be solved by the

Original: RL agent in a reasonable number of environment interactions.
Revised: RL agent in a reasonable number of environment interactions.

Original: • Value Reuse: Initialize the value network to the final value network of the previous agent.
Revised: • Value Reuse: Initialize the value network to the final value network of the previous agent.

Original: While two consecutive reward functions are different, both still reward close-by regions of
Revised: While two consecutive reward functions are different, both still reward close-by regions of

Original: the state space.
Revised: the state space.

Original: Thus, by keeping the previous value function, the policy network will be
Revised: Thus, by keeping the previous value function, the policy network will be

Original: nudged towards that region of the state space from the very first gradient updates.
Revised: nudged towards that region of the state space from the very first gradient updates.

Original: • Policy Feature Reuse: Initialize the policy network to the final policy of the previous agent
Revised: • Policy Feature Reuse: Initialize the policy network to the final policy of the previous agent

Original: but setting the weights of the final layer to 0 .
Revised: but setting the weights of the final layer to 0 .

Original: This keeps the previously learned features, but
Revised: This keeps the previously learned features, but

Original: outputs a uniform policy over all actions (or mean 0 and a fixed standard deviation in the
Revised: outputs a uniform policy over all actions (or mean 0 and a fixed standard deviation in the

Original: continuous case) which allows for proper learning and exploration 2 .
Revised: continuous case) which allows for proper learning and exploration 2 .

Original: • Guiding Policy: Act with the previous policy for a random number of steps at the beginning
Revised: • Guiding Policy: Act with the previous policy for a random number of steps at the beginning

Original: of each episode.
Revised: of each episode.

Original: This heavily simplifies the exploration problem.
Revised: This heavily simplifies the exploration problem.

Original: The agent will start
Revised: The agent will start

Original: exploring from states that are much closer to the rewards defined by R ψ .
Revised: exploring from states that are much closer to the rewards defined by R ψ .

Original: This is because the
Revised: This is because the

Original: previous policy could already solve the previous reward function.
Revised: previous policy could already solve the previous reward function.

Original: In contrast to the other
Revised: In contrast to the other

Original: two mechanisms, this one does not rely on initialization.
Revised: two mechanisms, this one does not rely on initialization.

Original: Thus, this is the most effective insparse reward functions that need many parameter updates to be learned.
Revised: Thus, this is the most effective insparse reward functions that need many parameter updates to be learned.

Original: In Section 4.1.1 we individually evaluate these three techniques and show that their combination isnecessary in complex environments.
Revised: In Section 4.1.1 we individually evaluate these three techniques and show that their combination isnecessary in complex environments.

Original: Putting everything together we get an algorithm which learns reward functions that encode increasingly complex behaviours and learns RL agents that solve those reward functions.
Revised: Putting everything together we get an algorithm which learns reward functions that encode increasingly complex behaviours and learns RL agents that solve those reward functions.

Original: Figure 1 illustratesthe main steps of our training loop and see Algorithm 1 in the appendix for more detail.
Revised: Figure 1 illustratesthe main steps of our training loop and see Algorithm 1 in the appendix for more detail.

Original:  Experiments
Revised:  Experiments

Original: We now proceed to experimentally test our method.
Revised: We now proceed to experimentally test our method.

Original: First, in Section 4.1, we thoroughly test alldifferent components of our model in a 2d navigation task.
Revised: First, in Section 4.1, we thoroughly test alldifferent components of our model in a 2d navigation task.

Original: This task allows us to verify the functionof each component and also to explicitly visualize what each reward function is encoding.
Revised: This task allows us to verify the functionof each component and also to explicitly visualize what each reward function is encoding.

Original: Then, in Section 4.2, we move to BRAX robotic environments (Freeman et al., 2021).
Revised: Then, in Section 4.2, we move to BRAX robotic environments (Freeman et al., 2021).

Original: These have themost flexibility and thus allow the agent to learn very complex tasks.
Revised: These have themost flexibility and thus allow the agent to learn very complex tasks.

Original: In these tasks, we evaluate thecomplexity of our skills by measuring their zero-shot transfer ability to the environment rewards.
Revised: In these tasks, we evaluate thecomplexity of our skills by measuring their zero-shot transfer ability to the environment rewards.

Original: Inthe H UMANOID environment, our unsupervised skills outperform supervised agents trained for tensof millions of time steps.
Revised: Inthe H UMANOID environment, our unsupervised skills outperform supervised agents trained for tensof millions of time steps.

Original: We also compute the one dimensional particle-based mutual informationmetric that has been proposed in the literature before (Gu et al., 2021) and show that our methodoutperforms previous approaches, even when other approaches only consider handcrafted featuredimensions in their objective.
Revised: We also compute the one dimensional particle-based mutual informationmetric that has been proposed in the literature before (Gu et al., 2021) and show that our methodoutperforms previous approaches, even when other approaches only consider handcrafted featuredimensions in their objective.

Original: Finally, in Section 4.3, we apply our method to Montezuma’s Revenge.
Revised: Finally, in Section 4.3, we apply our method to Montezuma’s Revenge.

Original: We show that the learnedreward functions keep getting increasingly complex and we are mostly limited by the amount ofcompute that it takes to learn each new reward function.
Revised: We show that the learnedreward functions keep getting increasingly complex and we are mostly limited by the amount ofcompute that it takes to learn each new reward function.

Original: 4.1 2D Navigation Task
Revised: 4.1 2D Navigation Task

Original: The task consists of a 32 by 32 maze with several walls anda ”danger zone”.
Revised: The task consists of a 32 by 32 maze with several walls anda ”danger zone”.

Original: The observation is given as a 32 x 32 x 1image with all values set to 0 , except a 1 in the currentposition of the agent.
Revised: The observation is given as a 32 x 32 x 1image with all values set to 0 , except a 1 in the currentposition of the agent.

Original: The agent starts in the top left cornerand can perform 5 actions, either move in one of the cardinal directions or stay in the current position.
Revised: The agent starts in the top left cornerand can perform 5 actions, either move in one of the cardinal directions or stay in the current position.

Original: If the agentmoves into a wall it stays at its current position instead.
Revised: If the agentmoves into a wall it stays at its current position instead.

Original: If the agent is in the ”danger zone” and moves up, movesdown or stays, the episode is terminated and the agent is moved back to the starting position.
Revised: If the agent is in the ”danger zone” and moves up, movesdown or stays, the episode is terminated and the agent is moved back to the starting position.

Original: Figure 2 showsthe layout of the maze.
Revised: Figure 2 showsthe layout of the maze.

Original: The ”danger zone” ensures thatrandom exploration will not work to reach many parts ofthe environment and lets us easily test both the increasingcomplexity of the reward functions and the importance offorward transfer.
Revised: The ”danger zone” ensures thatrandom exploration will not work to reach many parts ofthe environment and lets us easily test both the increasingcomplexity of the reward functions and the importance offorward transfer.

Original: We computed the expected number ofsteps to reach the bottom right corner with a random walkusing Dynamic Programming.
Revised: We computed the expected number ofsteps to reach the bottom right corner with a random walkusing Dynamic Programming.

Original: In expectation, 7 · 10 27episodes are needed to do so.
Revised: In expectation, 7 · 10 27episodes are needed to do so.

Original: This shows that this mazeis difficult to navigate.
Revised: This shows that this mazeis difficult to navigate.

Original: To train our agent we use the Advantage Actor Critic(A2C) (Mnih et al., 2016) algorithm.
Revised: To train our agent we use the Advantage Actor Critic(A2C) (Mnih et al., 2016) algorithm.

Original: To learn the rewardswe use the full algorithm presented in Section 3.
Revised: To learn the rewardswe use the full algorithm presented in Section 3.

Original: We usethe same architecture for the reward, policy and value networks, but do not share any parameters.
Revised: We usethe same architecture for the reward, policy and value networks, but do not share any parameters.

Original: The architecture isa ReLU network with 2 convolutional layers followed by2 fully connected layers.
Revised: The architecture isa ReLU network with 2 convolutional layers followed by2 fully connected layers.

Original: For the exact hyper-parameterssee Table 3 in Appendix B. Figure 2 plots the most visitedlocations for each skill of one run of our algorithm.
Revised: For the exact hyper-parameterssee Table 3 in Appendix B. Figure 2 plots the most visitedlocations for each skill of one run of our algorithm.

Original: Thefirst few skills visit points near the origin, later skills startmoving to harder to reach parts of the state space.
Revised: Thefirst few skills visit points near the origin, later skills startmoving to harder to reach parts of the state space.

Original: Afterroughly 40 iterations they reach the bottom right part.
Revised: Afterroughly 40 iterations they reach the bottom right part.

Original: Asstated before, this would take unreasonably long whenusing only random exploration.
Revised: Asstated before, this would take unreasonably long whenusing only random exploration.

Original: Inspired by the BRAX library (Freeman et al., 2021), weimplemented both the environment and an A2C agentinside a single JAX
Revised: Inspired by the BRAX library (Freeman et al., 2021), weimplemented both the environment and an A2C agentinside a single JAX

Original: (Bradbury et al., 2018) compiled function.
Revised: (Bradbury et al., 2018) compiled function.

Original: By doing this, the computation graph of the environment and agent are optimized jointly andboth run on the GPU.
Revised: By doing this, the computation graph of the environment and agent are optimized jointly andboth run on the GPU.

Original: This eliminates the need to send data between the CPU and GPU which is oneof the main bottlenecks in RL.
Revised: This eliminates the need to send data between the CPU and GPU which is oneof the main bottlenecks in RL.

Original: Using just one NVIDIA RTX 3090 GPU, the training process runsat over one million frames per second which enables training of agents in just a few seconds.
Revised: Using just one NVIDIA RTX 3090 GPU, the training process runsat over one million frames per second which enables training of agents in just a few seconds.

Original: Thisallows us to experiment quickly and at a lower economic and environmental cost.
Revised: Thisallows us to experiment quickly and at a lower economic and environmental cost.

Original: We believe thiscode is useful for the RL community on its own and provide it in the supplementary material.
Revised: We believe thiscode is useful for the RL community on its own and provide it in the supplementary material.

Original: 4.1.1 Forward transfer of skills
Revised: 4.1.1 Forward transfer of skills

Original: [Figure]
Revised: [Figure]

Original: As pointed out in Section 3.2, our reward functions become too complex to be learned from scratchwith random exploration in a reasonable number of steps.
Revised: As pointed out in Section 3.2, our reward functions become too complex to be learned from scratchwith random exploration in a reasonable number of steps.

Original: When this happens, our agent must rely ontransferring knowledge from previous generations.
Revised: When this happens, our agent must rely ontransferring knowledge from previous generations.

Original: Our navigation task is specifically designed totest this transfer ability, as random exploration would never reach the bottom right corner ( 7 · 10 27episodes in expectation).
Revised: Our navigation task is specifically designed totest this transfer ability, as random exploration would never reach the bottom right corner ( 7 · 10 27episodes in expectation).

Original: We experimentally evaluate the three forward transfer mechanisms proposed in Section 3.2: Valuereuse , Policy feature reuse and Guiding policy .
Revised: We experimentally evaluate the three forward transfer mechanisms proposed in Section 3.2: Valuereuse , Policy feature reuse and Guiding policy .

Original: The reward functions from Figure 2 serve as tasks,sorted according to creation order.
Revised: The reward functions from Figure 2 serve as tasks,sorted according to creation order.

Original: We train ablations of the three mechanisms sequentially on thesetasks.
Revised: We train ablations of the three mechanisms sequentially on thesetasks.

Original: This allows us to ignore the skill discovery process and only measure the forward transfer ofskills.
Revised: This allows us to ignore the skill discovery process and only measure the forward transfer ofskills.

Original: We repeat each experiment three times.
Revised: We repeat each experiment three times.

Original: The agent with all mechanisms always manages to solve 3 all reward functions.
Revised: The agent with all mechanisms always manages to solve 3 all reward functions.

Original: On the other hand, the Policy, Value and Guiding ablations fail to learnafter solving 29 .
Revised: On the other hand, the Policy, Value and Guiding ablations fail to learnafter solving 29 .

Original: 3 ± 9 . 5 , 18 .
Revised: 3 ± 9 . 5 , 18 .

Original: 7 ± 11 .
Revised: 7 ± 11 .

Original: 6 and 19 ± 8 .
Revised: 6 and 19 ± 8 .

Original: 5 reward functions, respectively.4.1.2 Speeding-up deep exploration
Revised: 5 reward functions, respectively.4.1.2 Speeding-up deep exploration

Original: [Figure]
Revised: [Figure]

Original: parameter when training actor criticmethods is entropy regularization.
Revised: parameter when training actor criticmethods is entropy regularization.

Original: In ourmethod, policies with a lot of entropy generatea diverse set of negative samples.
Revised: In ourmethod, policies with a lot of entropy generatea diverse set of negative samples.

Original: Diverse negative samples lead to reward functions that evolvemore in each generation.
Revised: Diverse negative samples lead to reward functions that evolvemore in each generation.

Original: This is especially beneficial in environments where many steps arenecessary to reach certain states, like in this 2Dnavigation task or Montezuma’s Revenge.
Revised: This is especially beneficial in environments where many steps arenecessary to reach certain states, like in this 2Dnavigation task or Montezuma’s Revenge.

Original: Weempirically verify this claim by training a set ofagents with of after as of the entropy regularization higherentropy leads to a faster coverage of the
Revised: Weempirically verify this claim by training a set ofagents with of after as of the entropy regularization higherentropy leads to a faster coverage of the

Original: space up to a certain threshold.
Revised: space up to a certain threshold.

Original: However much entropy leads to poor policies that do not learn to reach the rewarding states
Revised: However much entropy leads to poor policies that do not learn to reach the rewarding states

Original: are enough the origin.
Revised: are enough the origin.

Original: changeswhat the optimal policy is.
Revised: changeswhat the optimal policy is.

Original: Hence, we want to have policies that gather diverse samples to change the reward function but canalso be deterministic in parts of the trajectory that already have no reward.
Revised: Hence, we want to have policies that gather diverse samples to change the reward function but canalso be deterministic in parts of the trajectory that already have no reward.

Original: To achieve this, we useadaptive entropy regularization.
Revised: To achieve this, we useadaptive entropy regularization.

Original: That is, we apply a small entropy regularization term everywhere.
Revised: That is, we apply a small entropy regularization term everywhere.

Original: We increase the entropy regularization in states where the reward function is positive, i.e. states inwhich we want to decrease the reward function.
Revised: We increase the entropy regularization in states where the reward function is positive, i.e. states inwhich we want to decrease the reward function.

Original: See Figure 4 for a visualization of the impact of thedifferent entropy regularization strategies.
Revised: See Figure 4 for a visualization of the impact of thedifferent entropy regularization strategies.

Original: With this, the method manages to consistently visit allstates in the maze.
Revised: With this, the method manages to consistently visit allstates in the maze.

Original: This technique was necessary to create the reward functions from Figure 2.
Revised: This technique was necessary to create the reward functions from Figure 2.

Original: 4.2 Robotic Environments
Revised: 4.2 Robotic Environments

Original: Quantitatively measuring Unsupervised reinforcement learning progressremains an open problem and activearea of research (Gu et al., 2021).
Revised: Quantitatively measuring Unsupervised reinforcement learning progressremains an open problem and activearea of research (Gu et al., 2021).

Original: Because of this, it is still important tovisualize and qualitatively study thelearned skills.
Revised: Because of this, it is still important tovisualize and qualitatively study thelearned skills.

Original: We do this in Section 4.2.1.
Revised: We do this in Section 4.2.1.

Original: In the next two Sections,4.2.2 and 4.2.3, we quantitatively measure the performance of our algorithmon downstream task performance andwith the so-called particle-based mutual information metric, respectively.
Revised: In the next two Sections,4.2.2 and 4.2.3, we quantitatively measure the performance of our algorithmon downstream task performance andwith the so-called particle-based mutual information metric, respectively.

Original: In contrast to the previous experiments, the action space is continuousand the input modality is now a vector of features, like relative position,angle and speed of the different joints.
Revised: In contrast to the previous experiments, the action space is continuousand the input modality is now a vector of features, like relative position,angle and speed of the different joints.

Original: We do not use adaptive entropy here,as we need deterministic policies to maximize the downstream task performance and the mutual information metric.
Revised: We do not use adaptive entropy here,as we need deterministic policies to maximize the downstream task performance and the mutual information metric.

Original: Given that theenvironments do not have far away regions to reach, we did not implement the guiding algorithmhere.
Revised: Given that theenvironments do not have far away regions to reach, we did not implement the guiding algorithmhere.

Original: Note that the agent does not see the x - y position.
Revised: Note that the agent does not see the x - y position.

Original: Hyper-paremeters and architecture can befound in Appendix C.
Revised: Hyper-paremeters and architecture can befound in Appendix C.

Original: 4.2.1 Qualitative Analysis
Revised: 4.2.1 Qualitative Analysis

Original: In Figure 5 we illustrate a selection of particularly interesting skills.
Revised: In Figure 5 we illustrate a selection of particularly interesting skills.

Original: Videos of these andadditional skills can be found in the Supplementary Material.
Revised: Videos of these andadditional skills can be found in the Supplementary Material.

Original: In Figure 6 we see how the velocity of learned skills in A NT and H UMANOIDevolve over training.
Revised: In Figure 6 we see how the velocity of learned skills in A NT and H UMANOIDevolve over training.

Original: See Appendix E for theother runs.
Revised: See Appendix E for theother runs.

Original: One can see that consecutive skillsslightly change the direction and speed of moving.
Revised: One can see that consecutive skillsslightly change the direction and speed of moving.

Original: However, it is important to realize that theobservations contain tens to hundreds of dimensions.
Revised: However, it is important to realize that theobservations contain tens to hundreds of dimensions.

Original: Thus, the skills can encode much morecomplex behaviors than speed and direction ofmovement.
Revised: Thus, the skills can encode much morecomplex behaviors than speed and direction ofmovement.

Original: For example, Figure 5 shows thatthe skill involves keeping one leg in the air on top of moving in the right direction
Revised: For example, Figure 5 shows thatthe skill involves keeping one leg in the air on top of moving in the right direction

Original: 4.2.2 Zero-Shot Transfer
Revised: 4.2.2 Zero-Shot Transfer

Original: [Table]
Revised: [Table]

Original: After discovering 50 unsupervised skills, weidentify the skill that aligns best with the reward given by the environment.
Revised: After discovering 50 unsupervised skills, weidentify the skill that aligns best with the reward given by the environment.

Original: We report thiszero-shot 4 performance in Table 1.
Revised: We report thiszero-shot 4 performance in Table 1.

Original: As a baseline, we train an agent from scratch and measurehow long it takes it to reach an equivalent performance.
Revised: As a baseline, we train an agent from scratch and measurehow long it takes it to reach an equivalent performance.

Original: We use the Optuna hyper-parameteroptimizer (Akiba et al., 2019) to find hyper-parameters which maximize the speed of learningbeforehand.
Revised: We use the Optuna hyper-parameteroptimizer (Akiba et al., 2019) to find hyper-parameters which maximize the speed of learningbeforehand.

Original: The exact procedure can be found in Appendix D.
Revised: The exact procedure can be found in Appendix D.

Original: The simplest environment, H ALF -C HEETAH , benefits the least from unsupervised learning, whilethe hardest one, H UMANOID , benefits the most.
Revised: The simplest environment, H ALF -C HEETAH , benefits the least from unsupervised learning, whilethe hardest one, H UMANOID , benefits the most.

Original: This is in line with what is observed in Computer
Revised: This is in line with what is observed in Computer

Original: Vision and Natural Language Processing, of more complex tasks benefiting the most from a longerpre-training phase.
Revised: Vision and Natural Language Processing, of more complex tasks benefiting the most from a longerpre-training phase.

Original: 4.2.3 Particle-based Mutual Information metric
Revised: 4.2.3 Particle-based Mutual Information metric

Original: Measuring how well the agent can control relevant state dimensions is another way to track progressin Unsupervised reinforcement learning.
Revised: Measuring how well the agent can control relevant state dimensions is another way to track progressin Unsupervised reinforcement learning.

Original: This is measured using the mutual information betweenstate dimensions and skills.
Revised: This is measured using the mutual information betweenstate dimensions and skills.

Original: While high-dimensional estimation of mutual information an active areaof research, sampling can be an effective form of estimation in the 1-dimensional case, c.f.
Revised: While high-dimensional estimation of mutual information an active areaof research, sampling can be an effective form of estimation in the 1-dimensional case, c.f.

Original: Algorithm

Revised: Algorithm

Original: in (Gu et al., 2021).
Revised: in (Gu et al., 2021).

Original: We report the particle-based mutual information for the x -velocity in Table 2,using the same bucketing strategy 5 as in (Gu et al., 2021).
Revised: We report the particle-based mutual information for the x -velocity in Table 2,using the same bucketing strategy 5 as in (Gu et al., 2021).

Original: Our method heavily outperforms DIAYN, when both methods look at the complete observation space.
Revised: Our method heavily outperforms DIAYN, when both methods look at the complete observation space.

Original: DIAYN achieves diversity by learning a set of skills that can be correctly labeled by a neural network.
Revised: DIAYN achieves diversity by learning a set of skills that can be correctly labeled by a neural network.

Original: In high-dimensional spaces this is easy to do by relying on a small subset of all state dimensions.
Revised: In high-dimensional spaces this is easy to do by relying on a small subset of all state dimensions.

Original: This leads to non-diverse behaviours across most state dimensions.
Revised: This leads to non-diverse behaviours across most state dimensions.

Original: Even when expert knowledgeabout relevant dimensions is supplied to other methods, i.e. only taking the x - and y -velocity intoaccount, our method still fares well.
Revised: Even when expert knowledgeabout relevant dimensions is supplied to other methods, i.e. only taking the x - and y -velocity intoaccount, our method still fares well.

Original: Particularly, in the most complex environment, H UMANOID , our method performs best.
Revised: Particularly, in the most complex environment, H UMANOID , our method performs best.

Original: We believe that the iterative increase in complexity leads to better coverage ofhard to reach regions of the state like high speed or running backwards.
Revised: We believe that the iterative increase in complexity leads to better coverage ofhard to reach regions of the state like high speed or running backwards.

Original: With all this, our approachachieves greater controllability of the x -velocity without any kind of feature engineering.
Revised: With all this, our approachachieves greater controllability of the x -velocity without any kind of feature engineering.

Original: 4.3 Montezuma’s Revenge
Revised: 4.3 Montezuma’s Revenge

Original: [Table]
Revised: [Table]

Original: To show the generality of our approachwe evaluate it on the notoriously hard
Revised: To show the generality of our approachwe evaluate it on the notoriously hard

Original: Montezuma’s Revenge Atari game.
Revised: Montezuma’s Revenge Atari game.

Original: Inthis game, the agent controls a character in a complex 2d world with severalrooms.
Revised: Inthis game, the agent controls a character in a complex 2d world with severalrooms.

Original: Appendix G shows the initialroom and various items with which theagent may interact.
Revised: Appendix G shows the initialroom and various items with which theagent may interact.

Original: Same as in (Mnihet al., 2015), the observation is a stackof the last 4 frames.
Revised: Same as in (Mnihet al., 2015), the observation is a stackof the last 4 frames.

Original: This gives the agentinformation about speed and direction ofmovement.
Revised: This gives the agentinformation about speed and direction ofmovement.

Original: This is done for all networks,that is, the value, policy and neural reward networks.
Revised: This is done for all networks,that is, the value, policy and neural reward networks.

Original: We use the same simple CNN architecture as in (Mnih et al.
Revised: We use the same simple CNN architecture as in (Mnih et al.

Original: for all three networks.
Revised: for all three networks.

Original: For exactlearning details see Appendix H.
Revised: For exactlearning details see Appendix H.

Original: Our algorithm uses finite episode lengthsbecause once it reaches a rewarding state,the agent can stay there forever.
Revised: Our algorithm uses finite episode lengthsbecause once it reaches a rewarding state,the agent can stay there forever.

Original: Because of this, we reset the environment every 500 steps.
Revised: Because of this, we reset the environment every 500 steps.

Original: One of the main difficulties when dealing with Montezuma’s Revenge is that it cannot be simulatedas fast as the other studied environments.
Revised: One of the main difficulties when dealing with Montezuma’s Revenge is that it cannot be simulatedas fast as the other studied environments.

Original: On top of that, an agent can learn hundreds of differentskills without ever leaving the first room.
Revised: On top of that, an agent can learn hundreds of differentskills without ever leaving the first room.

Original: Finally, skills learned by our agent evolve from simple tovery complex and extended in time.
Revised: Finally, skills learned by our agent evolve from simple tovery complex and extended in time.

Original: In the beginning, the agent just needs to stay close to the initialposition.
Revised: In the beginning, the agent just needs to stay close to the initialposition.

Original: By the end of training, the agent learns to collect a key, open the door, avoid several enemiesand visit four different rooms.
Revised: By the end of training, the agent learns to collect a key, open the door, avoid several enemiesand visit four different rooms.

Original: In the most complex skills, it takes the agent several hundred steps toreach a rewarding state.
Revised: In the most complex skills, it takes the agent several hundred steps toreach a rewarding state.

Original: This means that experiments can take several days to visit a different room.
Revised: This means that experiments can take several days to visit a different room.

Original: [Figure]
Revised: [Figure]

Original: In order to save computation, we adapt the number of training steps to ensure the agent haslearned each skill.
Revised: In order to save computation, we adapt the number of training steps to ensure the agent haslearned each skill.

Original: We found out that the first thing the agent learnsis to get the life counter to 0 .
Revised: We found out that the first thing the agent learnsis to get the life counter to 0 .

Original: Then it startsexploring the room, with 0 lives already.
Revised: Then it startsexploring the room, with 0 lives already.

Original: This isbecause losing a life happens very easily duringrandom exploration and because once the agentreaches 0 lives, all future positive samples willhave 0 lives.
Revised: This isbecause losing a life happens very easily duringrandom exploration and because once the agentreaches 0 lives, all future positive samples willhave 0 lives.

Original: This has two side effects.
Revised: This has two side effects.

Original: Onthe one hand, the initial 100 − 200 steps of anepisode are spent losing lives.
Revised: Onthe one hand, the initial 100 − 200 steps of anepisode are spent losing lives.

Original: On the otherhand, exploration is harder, as the agent willreach a terminal state very easily.
Revised: On the otherhand, exploration is harder, as the agent willreach a terminal state very easily.

Original: Because ofthis, we cut out the part of the image that showsthe remaining lives, see Appendix G.
Revised: Because ofthis, we cut out the part of the image that showsthe remaining lives, see Appendix G.

Original: Figure 7(a) visualizes different skills.
Revised: Figure 7(a) visualizes different skills.

Original: The first observations which get the maximum possible rewardare overlaid.
Revised: The first observations which get the maximum possible rewardare overlaid.

Original: All the shown skills go to the bottom and kill the skull first.
Revised: All the shown skills go to the bottom and kill the skull first.

Original: Then, they move to aspecific location.
Revised: Then, they move to aspecific location.

Original: It can be seen that later skills explore states further and further away from the initialstate.
Revised: It can be seen that later skills explore states further and further away from the initialstate.

Original: Figure 7(b) shows that the skill receives no reward for many steps.
Revised: Figure 7(b) shows that the skill receives no reward for many steps.

Original: Only after the agent haskilled the skull being on the ladder becomes rewarding 6 .
Revised: Only after the agent haskilled the skull being on the ladder becomes rewarding 6 .

Original: This shows that both the reward network and the agent have learned a lot of the concepts that are necessary to tackle Montezuma’s Revenge.
Revised: This shows that both the reward network and the agent have learned a lot of the concepts that are necessary to tackle Montezuma’s Revenge.

Original: Thisincludes controlling the agent in the 2d environment, killing enemies and interacting with differentobjects.
Revised: Thisincludes controlling the agent in the 2d environment, killing enemies and interacting with differentobjects.

Original: All of this without ever accessing the original reward function.
Revised: All of this without ever accessing the original reward function.

Original: Eventually, after learning afew hundred skills, the neural reward function pushes the rewarding states all the way to the fourthroom.
Revised: Eventually, after learning afew hundred skills, the neural reward function pushes the rewarding states all the way to the fourthroom.

Original: In the montezuma exploring other rooms.gif file in the Supplementary Material we show arun of this skill.
Revised: In the montezuma exploring other rooms.gif file in the Supplementary Material we show arun of this skill.

Original:  Conclusion
Revised:  Conclusion

Original: We have presented an unsupervised reinforcement learning algorithm that uses reward functionsencoded by neural networks.
Revised: We have presented an unsupervised reinforcement learning algorithm that uses reward functionsencoded by neural networks.

Original: Our algorithm alternates between increasing the complexity of thereward function and transferring previous knowledge to learn a new skill that finds rewarding states.
Revised: Our algorithm alternates between increasing the complexity of thereward function and transferring previous knowledge to learn a new skill that finds rewarding states.

Original: This allows it to learn an unbounded number of skills that is mostly limited by the available computepower.
Revised: This allows it to learn an unbounded number of skills that is mostly limited by the available computepower.

Original: We have thoroughly tested the different components of our model in a 2d navigation task.
Revised: We have thoroughly tested the different components of our model in a 2d navigation task.

Original: Thishas allowed us to better understand our method in practice.
Revised: Thishas allowed us to better understand our method in practice.

Original: We have shown that our methodworks both with high dimensional feature inputs, in robotic environments, and pixel inputs, in
Revised: We have shown that our methodworks both with high dimensional feature inputs, in robotic environments, and pixel inputs, in

Original: Montezuma’s Revenge.
Revised: Montezuma’s Revenge.

Original: Our algorithm learned a diverse set of skills in both settings.
Revised: Our algorithm learned a diverse set of skills in both settings.

Original: In H UMANOIDand Montezuma’s Revenge, skills found by our method achieve a zero-shot performance that takesmillions of steps to learn in the classical reinforcement learning setup.
Revised: In H UMANOIDand Montezuma’s Revenge, skills found by our method achieve a zero-shot performance that takesmillions of steps to learn in the classical reinforcement learning setup.

Original: Our approach is general and opens up many possibilities for future research.
Revised: Our approach is general and opens up many possibilities for future research.

Original: The forward transfermechanism can be replaced by a more complex meta-learning or off-policy relabelling techniques.
Revised: The forward transfermechanism can be replaced by a more complex meta-learning or off-policy relabelling techniques.

Original: Collecting data for reward function training can benefit from smarter exploration strategies.
Revised: Collecting data for reward function training can benefit from smarter exploration strategies.

Original: Ourcurrent method keeps a single search point at a time which results in a narrow and deep search of theskill space.
Revised: Ourcurrent method keeps a single search point at a time which results in a narrow and deep search of theskill space.

Original: Applying similar ideas as (Wang et al., 2020) and keeping a population of skills to searchfrom may lead to faster and better exploration.
Revised: Applying similar ideas as (Wang et al., 2020) and keeping a population of skills to searchfrom may lead to faster and better exploration.

Original: We will explore these ideas in future work.
Revised: We will explore these ideas in future work.

Original: We believe our algorithm is one step in a direction that may one day allow reinforcement learningagents to fully understand an environment without making use of any predefined reward function.
Revised: We believe our algorithm is one step in a direction that may one day allow reinforcement learningagents to fully understand an environment without making use of any predefined reward function.

Original: Just like in Computer Vision and Natural Language Processing, this will lead to agents that need veryfew labels from the task at hand to be able to solve it and will drastically expand the applicability ofreinforcement learning.
Revised: Just like in Computer Vision and Natural Language Processing, this will lead to agents that need veryfew labels from the task at hand to be able to solve it and will drastically expand the applicability ofreinforcement learning.

Original: B 2d Navigation: Experimental details
Revised: B 2d Navigation: Experimental details

Original: The episode length in the 2d navigation task is 250 steps.
Revised: The episode length in the 2d navigation task is 250 steps.

Original: The guiding phase lasts for 2 / 3 of the totalsteps of the 1-steps A2C agent learning.
Revised: The guiding phase lasts for 2 / 3 of the totalsteps of the 1-steps A2C agent learning.

Original: We sample the guiding length uniformly at random in the 0to 200 range.
Revised: We sample the guiding length uniformly at random in the 0to 200 range.

Original: For the negative samples we follow the learnt policy for 200 steps.
Revised: For the negative samples we follow the learnt policy for 200 steps.

Original: For the positivesamples, we take random actions for 50 steps after following the policy for 200 steps.
Revised: For the positivesamples, we take random actions for 50 steps after following the policy for 200 steps.

Original: [Table]
Revised: [Table]

Original: C Robotic environments: Architecture and hyperparameters
Revised: C Robotic environments: Architecture and hyperparameters

Original: We use the PPO (Schulman et al., 2017) implementation from (Freeman et al., 2021) with modifications such that it allows our training method to work.
Revised: We use the PPO (Schulman et al., 2017) implementation from (Freeman et al., 2021) with modifications such that it allows our training method to work.

Original: In Table 4 and Table 5 we list thehyperparemeters we used.
Revised: In Table 4 and Table 5 we list thehyperparemeters we used.

Original: Note that we use a smaller learning rate for the bodies of the policy andvalue network.
Revised: Note that we use a smaller learning rate for the bodies of the policy andvalue network.

Original: We found that this was beneficial for transferring more knowledge from the previousskill.
Revised: We found that this was beneficial for transferring more knowledge from the previousskill.

Original: The environment specific parameters were found using Optuna (Akiba et al., 2019).
Revised: The environment specific parameters were found using Optuna (Akiba et al., 2019).

Original: For eachenvironment we ran a search to optimize final performance on the environment rewards (running inpositive x -direction).
Revised: For eachenvironment we ran a search to optimize final performance on the environment rewards (running inpositive x -direction).

Original: We used 200 runs and trained each of them for the same number of steps as askill in our method.
Revised: We used 200 runs and trained each of them for the same number of steps as askill in our method.

Original: This yielded hyperparameters which are able to learn tasks in the correspondingenvironment.
Revised: This yielded hyperparameters which are able to learn tasks in the correspondingenvironment.

Original: We did not take the downstream performance metrics (zero-shot perforamce and
Revised: We did not take the downstream performance metrics (zero-shot perforamce and

Original: [Table] [Table]
Revised: [Table] [Table]

Original: particle-based information) into account.
Revised: particle-based information) into account.

Original: Doing so would have exceeded our compute budget.
Revised: Doing so would have exceeded our compute budget.

Original: In
Revised: In

Original: Table 6 the architecure and hyperparameters used for the neural reward functions are listed.
Revised: Table 6 the architecure and hyperparameters used for the neural reward functions are listed.

Original: For thesupervised reward training we use the Adam optimizer (Kingma & Ba, 2014).
Revised: For thesupervised reward training we use the Adam optimizer (Kingma & Ba, 2014).

Original: All code can be foundin the supplementary material.
Revised: All code can be foundin the supplementary material.

Original: D Robotic environments: Zero-Shot Transfer
Revised: D Robotic environments: Zero-Shot Transfer

Original: We use the Optuna hyperparameter tuning library (Akiba et al., 2019) to create Table 1.
Revised: We use the Optuna hyperparameter tuning library (Akiba et al., 2019) to create Table 1.

Original: We ranmultiple optimization procedures.
Revised: We ranmultiple optimization procedures.

Original: For each one, we fix a number of environment steps, then optimizethe final performance on the environment reward over 200 runs.
Revised: For each one, we fix a number of environment steps, then optimizethe final performance on the environment reward over 200 runs.

Original: We iteratively increased the numberof timesteps until the highest score of the 200 runs beats our averaged zero-shot performance.
Revised: We iteratively increased the numberof timesteps until the highest score of the 200 runs beats our averaged zero-shot performance.

Original: Wetake the hyperparemters of that run and train 10 agents until they outperform our averaged zero-shotperformance.
Revised: Wetake the hyperparemters of that run and train 10 agents until they outperform our averaged zero-shotperformance.

Original: The average number of training steps needed for this is reported in Table 1.
Revised: The average number of training steps needed for this is reported in Table 1.

Original: In the
Revised: In the

Original: H UMANOID environment three of the runs did not reach the score in 100 M steps, at which pointwe stopped training.
Revised: H UMANOID environment three of the runs did not reach the score in 100 M steps, at which pointwe stopped training.

Original: We nonetheless took 100 M into the average.
Revised: We nonetheless took 100 M into the average.

Original: The code for the hyperparametersearch and the results of our conducted studies can be found in the supplementary material.
Revised: The code for the hyperparametersearch and the results of our conducted studies can be found in the supplementary material.

Original: E Robotic environments: Additional x − y velocity plots
Revised: E Robotic environments: Additional x − y velocity plots

Original: In Figure 8 and 9 one scatter plot as in Figure 6 for every run is shown.
Revised: In Figure 8 and 9 one scatter plot as in Figure 6 for every run is shown.

Original: F Montezuma’s Revenge: Adapting episode length
Revised: F Montezuma’s Revenge: Adapting episode length

Original: In order to save computation, we adapt the number of training steps according to several criteria.
Revised: In order to save computation, we adapt the number of training steps according to several criteria.

Original: Thislets us save a lot of compute on the skills which are easy to learn.
Revised: Thislets us save a lot of compute on the skills which are easy to learn.

Original: We use the following measures:
Revised: We use the following measures:

Original: • We train for 1 M steps with guiding from the previous policy.
Revised: • We train for 1 M steps with guiding from the previous policy.

Original: The number of guiding steps
Revised: The number of guiding steps

Original: is sampled uniformly at random between 0 and 450 , each time the environment is reset.
Revised: is sampled uniformly at random between 0 and 450 , each time the environment is reset.

Original: Then we train for another 0 .
Revised: Then we train for another 0 .

Original: 65 M steps without any guiding.•
Revised: 65 M steps without any guiding.•

Original: If the average reward goes down after removing the guiding or is too low at any point, we
Revised: If the average reward goes down after removing the guiding or is too low at any point, we

Original: restart the guiding phase for 0 .
Revised: restart the guiding phase for 0 .

Original: 65 M steps.
Revised: 65 M steps.

Original: [Table]
Revised: [Table]

Original: • If the agent is reaching a terminal state in more than 10% of the episodes, we continuetraining.
Revised: • If the agent is reaching a terminal state in more than 10% of the episodes, we continuetraining.

Original: [Table]
Revised: [Table]

Original: On top of this, we ignore positive samples that receive almost no reward.
Revised: On top of this, we ignore positive samples that receive almost no reward.

Original: All these tricks enable us toconsiderably reduce the training time.
Revised: All these tricks enable us toconsiderably reduce the training time.

Original: However, they are not a core change in our algorithm as theycould all be replaced by just training all generations for a longer fixed number of steps, just as before.
Revised: However, they are not a core change in our algorithm as theycould all be replaced by just training all generations for a longer fixed number of steps, just as before.

Original: G Montezuma’s Revenge: Environment details
Revised: G Montezuma’s Revenge: Environment details

Original: Figure 10 shows the initial state of Montezuma’s Revenge and the cropped version of it that is usedfor our agent.
Revised: Figure 10 shows the initial state of Montezuma’s Revenge and the cropped version of it that is usedfor our agent.

Original: H Montezuma’s Revenge: Experimental details
Revised: H Montezuma’s Revenge: Experimental details

Original: In Montezuma’s Revenge, we use the PPO implementation from the coax 7 library, with 1-steptemporal differences.
Revised: In Montezuma’s Revenge, we use the PPO implementation from the coax 7 library, with 1-steptemporal differences.

Original: If the average score per 500 steps is below 5 , we go back to the guiding phase.
Revised: If the average score per 500 steps is below 5 , we go back to the guiding phase.

Original: Due to the large memory requirements to store all negative samples, after the 15 -th epoch we rewriteold negative samples to add the new ones.
Revised: Due to the large memory requirements to store all negative samples, after the 15 -th epoch we rewriteold negative samples to add the new ones.

Original: Table 7 summarizes our hyper-parameters.
Revised: Table 7 summarizes our hyper-parameters.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: OSITIVE /

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: I Additional 2d navigation experiments

Revised: In Figure 11 we show results of the 2d navigation task with three alternative environments.

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]