Original: The enemy of my enemy is my friend: Learning to Generate Goals via Curriculum Self Play
Revised: I T T AKES F OUR TO T ANGO : M ULTIAGENT S ELFPLAY FOR A UTOMATIC C URRICULUM G ENERATION Anonymous authors Paper under double-blind review

Original: Abstract
Revised: A BSTRACT

Original: We are interested in training general-purpose reinforcement learning agents that cansolve a wide variety of goals.
Revised: We are interested in training general-purpose reinforcement learning agents that can solve a wide variety of goals.

Original: Training such agents without excessive interactionsrequires automatic generation of a goal curriculum .
Revised: Training such agents efﬁciently requires automatic generation of a goal curriculum .

Original: This is challenging as itrequires (a) exploring goals of increasing difﬁculty while ensuring that the agent(b) is exposed to a diverse set of goals in a sample efﬁcient manner and (c) doesnot catastrophically forget previously solved goals.
Revised: This is challenging as it requires (a) exploring goals of increasing difﬁculty, while ensuring that the agent (b) is exposed to a diverse set of goals in a sample efﬁcient manner and (c) does not catastrophically forget previously solved goals.

Original: We propose Curriculum Self Play (CuSP), an automated goal generation framework that seeks to satisfy thesedesiderata by virtue of a multi-player game with 4 agents.
Revised: We propose Curriculum Self Play (CuSP), an automated goal generation framework that seeks to satisfy these desiderata by virtue of a multi-player game with 4 agents.

Original: Our method builds onthe two-player method introduced in Asymmetric Self-Play and carefully balancescooperation and competition between two off-policy student learners and tworegret-maximizing teachers that generate goals which are challenging yet feasible.
Revised: The game carefully balances cooperation and competition between two off-policy student learners and two regret-maximizing teachers who generate goals that are challenging yet feasible.

Original: The teachers also account for goal coverage and the non-stationary nature of thestudents, allowing us to automatically induce a curriculum that balances progressiveexploration with anti-catastrophic exploitation.
Revised: The teachers also account for goal coverage and the non-stationary nature of the students, allowing us to automatically induce a curriculum that balances progressive exploration with anti-catastrophic exploitation.

Original: Our experiments demonstrate thatour method succeeds at generating effective curricula of goals for a range ofcomplex robotics tasks and signiﬁcantly outperforms ASP and other competingmethods at zero-shot test-time generalization to novel out-of-distribution goals.
Revised: Our experiments demonstrate that our method succeeds at generating effective curricula of goals for a range of complex robotics tasks and signiﬁcantly outperforms asymmetric self-play and other competing methods at zero-shot test-time generalization to novel out-of-distribution goals.

Original:  Introduction
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) has seen great success in a range of applications, from game-playing[37, 33] to real world robotics tasks [27].
Revised: Reinforcement learning (RL) has seen great success in a range of applications, from game-playing (Silver et al., 2016; Schrittwieser et al., 2020) to real world robotics tasks (OpenAI et al., 2019).

Original: However, these accomplishments are often in the realm ofmastering a single speciﬁc task.
Revised: However, these accomplishments are often in the realm of mastering a single speciﬁc task.

Original: For many applications, we further desire agents who are capable ofgeneralizing to a variety of tasks.
Revised: For many applications, we further desire agents who are capable of generalizing to a variety of tasks.

Original: This can achieved via goal-conditioned RL where an RL agentis trained to solve many goals by specifying a sequence of goals, i e. a goal curriculum.
Revised: This can achieved via goal-conditioned RL, where an RL agent is trained to solve many goals by specifying a sequence of goals, i.e. a goal curriculum.

Original: Manuallydesigning such curricula is tedious and moreover, does not scale due to being environment speciﬁcand heuristic-driven.
Revised: Manually designing such curricula is tedious and moreover, does not scale due to being environment speciﬁc and heuristic-driven.

Original: Even when it is easy to engineer a curriculum, we would prefer curricula thatcan dynamically adapt to the learner’s current capabilities.
Revised: Even when it is easy to engineer a curriculum, we would prefer curricula that can dynamically adapt to the learner’s current capabilities.

Original: For example, effective human tutors usean understanding of their pupil’s current knowledge and learning trajectory to create a curriculum.
Revised: For example, effective human tutors use an understanding of their pupil’s current knowledge and learning trajectory to create a curriculum.

Original: Our work builds on two broad desiderata for curriculum generation [4] that enable sample-efﬁcientlearning of goal-conditioned policies: progressive exploration and anti-catastrophic exploitation .
Revised: Our work builds on two broad desiderata for curriculum generation (Bengio et al., 2009) that enable sample-efﬁcient learning of goal-conditioned policies: progressive exploration and anti-catastrophic exploitation .

Original: Progressive exploration refers to curricula where the agent gradually learns to solve tasks of increasingdifﬁculty and diversity.
Revised: Progressive exploration refers to curricula where the agent gradually learns to solve tasks of increasing difﬁculty and diversity.

Original: Anti-catastrophic exploitation ensures that the agent periodically resolvespreviously attempted goals that could become harder to solve during the course of training 
Revised: Anti-catastrophic exploitation ensures that the agent periodically resolves previously attempted goals that could become harder to solve during the course of training a.k.a.

Original: a.k.a.catastrophic forgetting [9, 17].
Revised: catastrophic forgetting (French, 1999; Kirkpatrick et al., 2017).

Original: Many prior works have attempted to incorporate aspects of these criteria for automatic curriculum generation.
Revised: Many prior works have attempted to incorporate aspects of these criteria for automatic curriculum generation.

Original: One approach is to parameterize a goal generator and pit it against a discriminator that assesses goal difﬁculty [8] or feasibility [32] for the current agent.
Revised: One approach is to parameterize a goal generator and pit it against a discriminator that assesses goal difﬁculty (Florensa et al., 2018) or feasibility (Racaniere et al., 2020) for the current agent.

Original: Racaniere et al.
Revised: Racaniere et al.

Original: [32] also address[Figure] desired qualities of validity and diversity by crafting speciﬁc losses for these characteristics.
Revised: (2020) also address desired qualities of validity and diversity by crafting speciﬁc losses for these characteristics.

Original: However,these approaches rely on learning an accurate discriminator for assessing goal difﬁculty, which can bechallenging.
Revised: However, these approaches rely on learning an accurate discriminator for assessing goal difﬁculty, which can be challenging.

Original: Rather than employing discriminators in the standard adversarial minimax formulation, Dennis et al.
Revised: Rather than employing discriminators in the standard adversarial minimax formulation, Dennis et al.

Original: [6] proposes an alternate minimax objective based on the agent’s regret.
Revised: (2020) proposes an alternate minimax objective based on the agent’s regret.

Original: The ‘expert’used for computing the regret is another goal-conditioned policy whose utility is maximized by anenvironment generator.
Revised: The ‘expert’ used for computing the regret is another goal-conditioned policy whose utility is maximized by an environment generator.

Original: An alternative method for automatic curriculum generation that has seen great success is self-play,where an agent plays against other versions of itself [37, 3, 2] (Fig. 1a).
Revised: An alternative method for automatic curriculum generation that has seen great success is self-play, where an agent plays against other versions of itself (Silver et al., 2016; Bansal et al., 2018; Baker [Figure] et al., 2019) (Fig. 1a).

Original: Unfortunately, this methoddoes not apply directly to complex tasks where we do not have the structure of a two player, zero-sumgame.
Revised: Unfortunately, this method does not apply directly to complex tasks where we do not have the structure of a two player, zero-sum game.

Original: A promising variation is Asymmetric Self-Play (ASP) [40], which involves a minimax gamebetween two similar agents: a goal demonstrator, Alice, and the desired goal-conditioned agent, Bob(Fig. 1b).
Revised: A promising variation is Asymmetric Self-Play (ASP) (Sukhbaatar et al., 2018b), which involves a minimax game between two similar agents: a goal demonstrator, Alice, and the desired goal-conditioned agent, Bob (Fig. 1b).

Original: Bob is asked to solve or reverse tasks demonstrated by Alice in the same environment.
Revised: Bob is asked to solve or reverse tasks demonstrated by Alice in the same environment.

Original: Tomotivate Alice to propose successively challenging goals, Alice is rewarded whenever Bob fails.
Revised: To motivate Alice to propose successively challenging goals, Alice is rewarded whenever Bob fails.

Original: Thismethod inherently captures an easy-to-hard curriculum, since any goal solved by Alice is achievableby Bob in principle.
Revised: This method inherently captures an easy-to-hard curriculum, since any goal solved by Alice is achievable by Bob in principle.

Original: However, ASP tends to be sample inefﬁcient as it is bottlenecked by Alice’sability to remember earlier goals and explore a diverse goal space.
Revised: However, ASP tends to be sample inefﬁcient as it is bottlenecked by Alice’s ability to remember earlier goals and explore a diverse goal space.

Original: Our work aims to bring together the strengths of goal generative methods and ASP while addressingthe two desiderata.
Revised: Our work aims to bring together the strengths of goal generative methods and ASP while addressing the two desiderata.

Original: Our work focuses on the multi-goal setting with the curriculum generatorcontrolling goals [31].
Revised: Our work focuses on the multi-goal setting with the curriculum generator controlling goals ? .

Original: We propose Curriculum Self Play (CuSP), a symmetric multi-agent setup forautomatic curricula generation.
Revised: We propose Curriculum Self Play (CuSP), a symmetric multi-agent setup for automatic curricula generation.

Original: CuSP optimizes for two off-policy goal-conditioned students, Aliceand Bob, and two regret-maximizing goal generators, G A and G B (Fig. 1c).
Revised: CuSP optimizes for two off-policy goal-conditioned students, Alice and Bob, and two regret-maximizing goal generators, G A and G B (Fig. 1c).

Original: Drawing inspiration fromthe minimax regret objective in PAIRED [6], we designate each goal generator as being ‘friendly’ toone of the students ( G A for Alice, G B for Bob), and optimize each goal generator using the differencein utility between the preferred student and the other student (i.e., G A is rewarded R A − R B , where R A , R B are Alice’s and Bob’s returns respectively when challenged with the same goal).
Revised: Drawing inspiration from the minimax regret objective in PAIRED (Dennis et al., 2020), we designate each goal generator as being ‘friendly’ to one of the students ( G A for Alice, G B for Bob), and optimize each goal generator using the difference in utility between the preferred student and the other student (i.e., G A is rewarded R A − R B , where R A , R B are Alice’s and Bob’s returns respectively when challenged with the same goal).

Original: Thisencourages proposing goals that are neither too easy nor too hard.
Revised: This encourages proposing goals that are neither too easy nor too hard.

Original: With our symmetrized setup, Aliceand Bob each have a ‘friendly’ goal generator that is inclined to propose more feasible goals withoutbeing too simple, and an ‘unfriendly’ goal generator that is inclined to propose more challenginggoals.
Revised: With our symmetrized setup, Alice and Bob each have a ‘friendly’ goal generator that is inclined to propose more feasible goals without being too simple, and an ‘unfriendly’ goal generator that is inclined to propose more challenging goals.

Original: As the tasks increase in difﬁculty overall, the former’s cooperation helps in scaffolding learningwhile the latter’s competition helps for pushing the boundaries of the agent’s capabilities.
Revised: As the tasks increase in difﬁculty overall, the former’s cooperation helps in scaffolding learning while the latter’s competition helps for pushing the boundaries of the agent’s capabilities.

Original: We further address goal diversity and sample efﬁciency by leveraging the off policy entropyregularized optimization of Soft Actor Critic (SAC) [13] for training the goal generators.
Revised: We further address goal diversity and sample efﬁciency by leveraging the off policy entropyregularized optimization of Soft Actor Critic (SAC) (Haarnoja et al., 2018) for training the goal generators.

Original: Thegoal generators’ replay buffers track the regrets for all proposed goals.
Revised: The goal generators’ replay buffers track the regrets for all proposed goals.

Original: To account for non-stationarylearners, we propose an update rule for the regrets in the goal generators’ replay buffers based onthe critic networks of Alice and Bob.
Revised: To account for non-stationary learners, we propose an update rule for the regrets in the goal generators’ replay buffers based on the critic networks of Alice and Bob.

Original: In doing so, we constantly keep track of the learners’ currentabilities and can propose increasingly challenging goals.
Revised: In doing so, we constantly keep track of the learners’ current abilities and can propose increasingly challenging goals.

Original: At the same time, using the regret objectivealso places the current abilities in context.
Revised: At the same time, using the regret objective also places the current abilities in context.

Original: For example, if Bob has forgotten how to achieve aparticular goal, the regret R A − R B increases and G A is more inclined to re-propose that goal.
Revised: For example, if Bob has forgotten how to achieve a particular goal, the regret R A − R B increases and G A is more inclined to re-propose that goal.

Original: Thusthe combination of using a replay buffer and the regret objective aids in addressing the twin goals ofprogressive exploration and anti-catastrophic exploitation.
Revised: Thus the combination of using a replay buffer and the regret objective aids in addressing the twin goals of progressive exploration and anti-catastrophic exploitation.

Original: Compared to prior methods for goal generation, we demonstrate that CuSP is able to satisfy thedesiderata for good curricula.
Revised: Compared to prior methods for goal generation, we demonstrate that CuSP is able to satisfy the desiderata for good curricula.

Original: We show that this translates to improved sample efﬁciency and generalization to novel out-of-distribution goals across a range of robotics tasks, encompassingnavigation, manipulation, and locomotion.
Revised: We show that this translates to improved sample efﬁciency and generalization to novel out-of-distribution goals across a range of robotics tasks, encompassing navigation, manipulation, and locomotion.

Original: We highlight some emergent skills that arise via CuSPwhen generalizing to particularly challenging goals, and ﬁnd that our method is able to handle caseswhen portions of the goal space are infeasible.
Revised: We highlight some emergent skills that arise via CuSP when generalizing to particularly challenging goals, and ﬁnd that our method is able to handle cases when portions of the goal space are infeasible.

Original:  Background
Revised:  B ACKGROUND

Original: Problem Setup.
Revised: Problem Setup.

Original: We model the environment and the goal-conditioned learner using a Markovdecision process, M =
Revised: We model the environment and the goal-conditioned learner using a Markov decision process, M =

Original: (cid:104)S , A , T , R , G , γ (cid:105) , where S is the state space, A is the action space,
Revised: (cid:104)S , A , T , R , G , γ (cid:105) , where S is the state space, A is the action space,

Original: T : S ×
Revised: T : S ×

Original: A × S → R is the environment transition function, R : G × S
Revised: A × S → R is the environment transition function, R : G × S

Original: → R is the goal-conditionedreward function, G ⊆ S is the goal space, and γ is a discount factor.
Revised: → R is the goal-conditioned reward function, G ⊆ S is the goal space, and γ is a discount factor.

Original: We consider both dense andsparse reward functions: r ( s | g ) = − d ( s, g ) , where d is a distance metric (e.g. L 2 norm) betweenthe state s and the goal g for the former, or r ( s | g )
Revised: We consider both dense and sparse reward functions: r ( s | g ) = − d ( s, g ) , where d is a distance metric (e.g. L 2 norm) between the state s and the goal g for the former, or r ( s | g )

Original: = 1 { d ( s, g ) <
Revised: = 1 { d ( s, g ) <

Original: (cid:15) } for the latter.
Revised: (cid:15) } for the latter.

Original: Our objective isto learn a policy π ( a t | s t , g ) that maximizes the success rate of reaching all goals g ∈ G .
Revised: Our objective is to learn a policy π ( a t | s t , g ) that maximizes the success rate of reaching all goals g ∈ G .

Original: To assessability to generalize to out-of-distribution goals, we split the goal space into G id and G ood where thegoal generators can only propose goals g ∈ G id , and we evaluate on sampled g ∈ G ood .
Revised: To assess ability to generalize to out-of-distribution goals, we split the goal space into G id and G ood where the goal generators can only propose goals g ∈ G id , and we evaluate on sampled g ∈ G ood .

Original: Self Play (SP).
Revised: Self Play (SP).

Original: SP is an approach for automatic curriculum generation in competitive, zero-sumgames.
Revised: SP is an approach for automatic curriculum generation in competitive, zero-sum games.

Original: In this paradigm, an agent plays against past versions of itself in order to automaticallygenerate a learning curriculum (Figure 1a).
Revised: In this paradigm, an agent plays against past versions of itself in order to automatically generate a learning curriculum (Figure 1a).

Original: That is, we consider Alice and Bob as two versions ofthe same underlying agent, and each agent’s objective is the negative of the others’, so R A = − R B .
Revised: That is, we consider Alice and Bob as two versions of the same underlying agent, and each agent’s objective is the negative of the others’, so R A = − R B .

Original: However, this approach is only applicable for symmetric zero-sum games, and does not lend itself tolearning curricula for goal-conditioned agents in other setups.
Revised: However, this approach is only applicable for symmetric zero-sum games, and does not lend itself to learning curricula for goal-conditioned agents in other setups.

Original: Asymmetric Self Play (ASP).
Revised: Asymmetric Self Play (ASP).

Original: ASP [40] extends self play to non-zero sum games by proposingan asymmetric parametrization of two competing agents.
Revised: ASP (Sukhbaatar et al., 2018b) extends self play to non-zero sum games by proposing an asymmetric parametrization of two competing agents.

Original: During each episode, Alice’s objectiveis to propose and demonstrate a task for Bob to complete (Figure 1b).
Revised: During each episode, Alice’s objective is to propose and demonstrate a task for Bob to complete (Figure 1b).

Original: In the goal-conditionedparadigm, we take Alice’s ﬁnal state s AT as the proposed goal g .
Revised: In the goalconditioned paradigm, we take Alice’s ﬁnal state s AT as the proposed goal g .

Original: Alice and Bob now have differentobjectives: Bob’s return, R B = (cid:80) t γ t r ( s Bt | g ) , is simply based on reaching the proposed goal,while Alice’s reward is based on Bob’s inability to reach the goal, R A
Revised: Alice and Bob now have different objectives: Bob’s return, R B = (cid:80) t γ t r ( s Bt | g ) , is simply based on reaching the proposed goal, while Alice’s reward is based on Bob’s inability to reach the goal, R A

Original: = 1 ( d ( s BT , g ) ≥ (cid:15) ) .
Revised: = 1 ( d ( s BT , g ) ≥ (cid:15) ) .

Original: Whilethis approach lends itself well to automatic curriculum generation in more complex tasks such asrobotic manipulation [28], it has poor sample efﬁciency.
Revised: While this approach lends itself well to automatic curriculum generation in more complex tasks such as robotic manipulation (OpenAI et al., 2021), it has poor sample efﬁciency.

Original: Since we rely on a learning agent, Alice, topropose and solve goals, the generated goals are limited by Alice’s own ability to explore and learnchallenging yet feasible goals.
Revised: Since we rely on a learning agent, Alice, to propose and solve goals, the generated goals are limited by Alice’s own ability to explore and learn challenging yet feasible goals.

Original: This can be particularly problematic towards the beginning of training,where we can waste many episodes proposing simple and redundant goals for Bob.
Revised: This can be particularly problematic towards the beginning of training, where we can waste many episodes proposing simple and redundant goals for Bob.

Original:  Automatic Curriculum Generation via CuSP
Revised:  A UTOMATIC C URRICULUM G ENERATION VIA C U SP

Original: Our curriculum generation method consists of four main players: two goal-conditioned learningpeers, Alice π A and Bob π B , and two goal generators, G A and G B .
Revised: Our curriculum generation method consists of four main players: two goal-conditioned learning peers, Alice π A and Bob π B , and two goal generators, G A and G B .

Original: π A ( a | s, g ) and π B ( a | s, g ) areboth optimized to maximize the discounted sum of goal-conditioned rewards R ( g ) = (cid:80) t γ t r t ( g ) ,across proposed goals g ∈ G id .
Revised: π A ( a | s, g ) and π B ( a | s, g ) are both optimized to maximize the discounted sum of goal-conditioned rewards R ( g ) = (cid:80) t γ t r t ( g ) , across proposed goals g ∈ G id .

Original: Aside from any initial randomness in initializing π A and π B , the twopolicies are parameterized with the same architecture.
Revised: Aside from any initial randomness in initializing π A and π B , the two policies are parameterized with the same architecture.

Original: G A ( g | s 0 , z ) and G B ( g | s 0 , z ) are parametrizedas policies that output a goal g ∈ G id given an initial observation of the environment s 0 and a latentnoise variable z ∼ N (0 , 1) .
Revised: G A ( g | s 0 , z ) and G B ( g | s 0 , z ) are parametrized as policies that output a goal g ∈ G id given an initial observation of the environment s 0 and a latent noise variable z ∼ N (0 , 1) .

Original: The inclusion of such a noise variable draws from work in generativemodels (e.g. GANs, VAEs) where such a z is the input to the generator, and is also used in priorwork in goal generative models [8].
Revised: The inclusion of such a noise variable draws from work in generative models (e.g. GANs, VAEs) where such a z is the input to the generator, and is also used in prior work in goal generative models Florensa et al.

Original: The goal generators are optimized to maximize the regret of theircorresponding agent:
Revised: The goal generators are optimized to maximize the regret of their corresponding agent:

Original: R G A = R A ( g ) − R B
Revised: R G A = R A ( g ) − R B

Original: ( g ) and R GB =
Revised: ( g ) and R G

Revised: [Equation]

Original:  − R GA
Revised: = − R G

Original:  = 
Revised: [Equation] = R [Equation]

Original: R B ( g ) − R A ( g ) .
Revised: ( g ) − [Equation] A ( g ) .

Original: We refer to our overall curriculum generation approach as Curriculum Self Play (CuSP).
Revised: We refer to our overall curriculum generation approach as Curriculum Self Play (CuSP).

Original: We presentan overview of CuSP in Algorithm 1 and illustrate it in Figure 1c.
Revised: We present an overview of CuSP in Algorithm 1 and illustrate it in Figure 1c.

Original: The algorithm proceeds in rounds.
Revised: The algorithm proceeds in rounds.

Original: In every round, both goal generators propose a goal, g A from G A and g B from G B .
Revised: In every round, both goal generators propose a goal, g A from G A and g B from G B .

Original: To keep thesymmetrization fair and hopefully scaffold learning initially, we ﬁrst rollout the ‘easier’ goal for eachrespective agent (ie.
Revised: To keep the symmetrization fair and hopefully scaffold learning initially, we ﬁrst rollout the ‘easier’ goal for each respective agent (ie.

Original: g A for Alice, g B for Bob) and compute the corresponding environment returnsand update the learners.
Revised: g A for Alice, g B for Bob) and compute the corresponding environment returns and update the learners.

Original: Next, we rollout the ‘harder’ goal for each respective agent (ie.
Revised: Next, we rollout the ‘harder’ goal for each respective agent (ie.

Original: g B for Alice,g A for Bob) and update the learners again.
Revised: g B for Alice, g A for Bob) and update the learners again.

Original: At the end of the round, we update the goal generatorswith the respective regrets.
Revised: At the end of the round, we update the goal generators with the respective regrets.

Original: Symmetrization.
Revised: Symmetrization.

Original: The regret objective R A − R B helps G A propose goals that are challenging for
Revised: The regret objective R A − R B helps G A propose goals that are challenging for

Original: Bob while still being feasible for Alice.
Revised: Bob while still being feasible for Alice.

Original: However, this can be detrimental when Alice and Bob’s 
Revised: However, this can be detrimental when Alice and Bob’s performance diverge too quickly.

Revised: To address this, we propose symmetrizing the multi-agent system by introducing a corresponding goal generator G B whose regret objective is R B − R A , essentially pitting it in a zero-sum game against G A .

Revised: Our insight is that having these two goal generators produces two types of goals for each agent: a) a feasible yet not too simple goal from the ‘friendly’

Original: Initialize goal-conditioned learners π A , π B , goal generators G A , G B ; while not converged do // generate goals g a ∼ G A , g b ∼ G B ; // evaluate easier goals first
Revised: Initialize goal-conditioned learners π A , π B , goal generators G A , G B ; while not converged do // generate goals g a ∼ G A , g b ∼ G B ; // evaluate easier goals first

Original: R AEasy ← rollout ( π A , g A ) ; R BEasy ← rollout ( π B , g B ) ; // update policies π A ← train_ learner ( π A , R AEasy ) ; π B ← train_ learner ( π B , R BEasy ) ; // evaluate harder goals second
Revised: R AEasy ← rollout ( π A , g A ) ; R BEasy ← rollout ( π B , g B ) ; // update policies π A ← train learner ( π A , R AEasy ) ; π B ← train learner ( π B , R BEasy ) ; // evaluate harder goals second

Original: R AHard ← rollout ( π A , g B ) ; R BHard ← rollout ( π B , g A ) ; // update policies π A ← train_ learner ( π A , R AHard ) ; π B ← train_ learner ( π B , R BHard ) ; // compute regrets and update generators
Revised: R AHard ← rollout ( π A , g B ) ; R BHard ← rollout ( π B , g A ) ; // update policies π A ← train learner ( π A , R AHard ) ; π B ← train learner ( π B , R BHard ) ; // compute regrets and update generators

Original: G A ← train_ generator ( G A , { ( R AEasy − R BHard ) , g A } , { ( R AHard − R BEasy ) , g B } ) ;
Revised: G A ← train generator ( G A , { ( R AEasy − R BHard ) , g A } , { ( R AHard − R BEasy ) , g B } ) ;

Original: G B ← train_ generator ( G B , { ( R BEasy − R AHard ) , g B } , { ( R BHard − R AEasy ) , g A } ) ; endperformance diverge too quickly. To address this, we propose symmetrizing the multi-agent system byintroducing a corresponding goal generator G B whose regret objective is R B − R A , essentially pittingit in a zero-sum game against G A . Our insight is that having these two goal generators produces twotypes of goals for each agent: a) a feasible yet not too simple goal from the ‘friendly’ goal generator( G A for Alice, G B for Bob), which can help scaffold learning quickly, and b) a challenging yet nottoo difﬁcult goal from the ‘unfriendly’ goal generator ( G A for Bob, G B for Alice), which can helppush the boundaries of the agent’s abilities.
Revised: G B ← train generator ( G B , { ( R BEasy − R AHard ) , g B } , { ( R BHard − R AEasy ) , g A } ) ; end goal generator ( G A for Alice, G B for Bob), which can help scaffold learning quickly, and b) a challenging yet not too difﬁcult goal from the ‘unfriendly’ goal generator ( G A for Bob, G B for Alice), which can help push the boundaries of the agent’s abilities.

Original: Both of these properties help encourage the generation ofgoals that are feasible yet progressively challenging.
Revised: Both of these properties help encourage the generation of goals that are feasible yet progressively challenging.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: Goal Generator Design.
Revised: Goal Generator Design.

Original: We motivate our goal generator design with our proposed desiderata forcurriculum generation: progressive exploration balanced with anti-catastrophic exploitation.
Revised: We motivate our goal generator design with our proposed desiderata for curriculum generation: progressive exploration balanced with anti-catastrophic exploitation.

Original: That is,we want to propose goals that are progressively more challenging but also prioritize revisiting goals that may have been forgotten by the learners.
Revised: That is, we want to propose goals that are progressively more challenging, but also prioritize revisiting goals that may have been forgotten by the learners.

Original: Although the goal generator is not an RL agent in thetraditional sense – it only ‘acts’ in a single timestep by proposing a goal and its ‘observation’ onlychanges based on a random noise variable – optimizing such an explore-exploit tradeoff effectivelynaturally inspires the use of an RL objective.
Revised: Although the goal generator is not an RL agent in the traditional sense – it only ‘acts’ in a single timestep by proposing a goal and its ‘observation’ only changes based on a random noise variable – optimizing such an explore-exploit tradeoff effectively naturally inspires the use of an RL objective.

Original: RL objectives have been used for one-step optimizationin other contexts as well (e.g., one-step Q-learning [45]).
Revised: RL objectives have been used for one-step optimization in other contexts as well (e.g., one-step Q-learning (Watkins & Dayan, 1992)).

Original: Thus, to train the goal generators we use an RL learning objective and parameterization inspired by asingle-time step variant of Soft-Actor Critic (SAC) [13].
Revised: Thus, to train the goal generators we use an RL learning objective and parameterization inspired by a single-time step variant of Soft-Actor Critic (SAC) (Haarnoja et al., 2018).

Original: SAC is an off policy algorithm that optimizesfor a trade-off between expected reward and policy entropy H ( π ) with a coefﬁcient α .
Revised: SAC is an off policy algorithm that optimizes for a trade-off between expected reward and policy entropy H ( π ) with a coefﬁcient α .

Original: In our setup,our goal generator only acts once to propose a goal per episode, so each ‘trajectory’ is only a singletime step consisting of the initial state s (for which we concatenate the initial environment observations 0 with a latent noise variable z ), the proposed goal action g , and we get the corresponding “reward"for the goal generator policy by computing the regret between π A and π B on g .
Revised: In our setup, our goal generator only acts once to propose a goal per episode, so each ‘trajectory’ is only a single time step consisting of the initial state s (for which we concatenate the initial environment observation s 0 with a latent noise variable z ), the proposed goal action g , and we get the corresponding “reward” for the goal generator policy by computing the regret between π A and π B on g .

Original: Thus the objectivefor each goal generator policy G is given by
Revised: Thus the objective for each goal generator policy G is given by

Original: [Equation]
Revised: [Equation]

Original:  where the regret term addresses anti-catastrophic exploitation by prioritizing goals where regret hasincreased, and the entropy term addresses progressive goal space exploration.
Revised:   where the regret term addresses anti-catastrophic exploitation by prioritizing goals where regret has increased, and the entropy term addresses progressive goal space exploration.

Original: We equip the SAC goal generators with a replay buffer to store all past goals and their regrets.
Revised: We equip the SAC goal generators with a replay buffer to store all past goals and their regrets.

Original: Thishelps address forgetting and improve the sample efﬁciency of goal generation compared to priormethods which optimize the generator as another agent using on-policy algorithms such as PPO [34].
Revised: This helps address forgetting and improve the sample efﬁciency of goal generation compared to prior methods which optimize the generator as another agent using on-policy algorithms such as PPO (Schulman et al., 2017).

Original: We further take advantage of SAC’s entropy regularization to progressively increase the diversity ofthe goals, with consideration of Alice and Bob’s capabilities using the stored regrets.
Revised: We further take advantage of SAC’s entropy regularization to progressively increase the diversity of the goals, with consideration of Alice and Bob’s capabilities using the stored regrets.

Original: To empiricallymotivate the use of SAC, we compare SAC against PPO and Adam [16] in optimizing a syntheticlandscape.
Revised: To empirically motivate the use of SAC, we compare SAC against PPO and Adam (Kingma & Ba, 2015) in optimizing a synthetic landscape.

Original: One challenge of the regret objective is that at the beginning of training, the landscape willmostly be ﬂat when π A and π B are similarly unskilled.
Revised: One challenge of the regret objective is that at the beginning of training, the landscape will mostly be ﬂat when π A and π B are similarly unskilled.

Original: In Fig.
Revised: In Fig.

Original: 2, we ﬁnd that only SAC is proﬁcientat optimizing a primarily ﬂat landscape.
Revised: 2, we ﬁnd that only SAC is proﬁcient at optimizing a primarily ﬂat landscape.

Original: Dynamic Regret Updates for Non-stationary Agents.
Revised: Dynamic Regret Updates for Non-stationary Agents.

Original: However, using an off policy RL algorithmfor optimizing such a goal generator requires further modiﬁcations.
Revised: However, using an off policy RL algorithm for optimizing such a goal generator requires further modiﬁcations.

Original: For more typical agents, thereceived rewards are a stationary property of the environment.
Revised: For more typical agents, the received rewards are a stationary property of the environment.

Original: In our case this is no longer true – Alice and Bob are continuously learning and part of the multi-agent environment, so the regretsfor the same goal will also change over time.
Revised: In our case this is no longer true – Alice and Bob are continuously learning and part of the multi-agent environment, so the regrets for the same goal will also change over time.

Original: To address this issue, we would like to update thenon-stationary regret values in the goal generator’s replay buffer.
Revised: To address this issue, we would like to update the non-stationary regret values in the goal generator’s replay buffer.

Original: However, it is expensive for Aliceand Bob to replay all the goals in the environment.
Revised: However, it is expensive for Alice and Bob to replay all the goals in the environment.

Original: As a sample efﬁcient trick, we leverage theobservation that the critics of Alice and Bob can provide a current estimate of their performance.
Revised: As a sample efﬁcient trick, we leverage the observation that the critics of Alice and Bob can provide a current estimate of their performance.

Original: Hence, after every round, we update the regrets of previous goals in the replay buffer using thedifference between the corresponding value estimates obtained via the critic networks of Alice and
Revised: Hence, after every round, we update the regrets of previous goals in the replay buffer using the difference between the corresponding value estimates obtained via the critic networks of Alice and

Original: Bob.
Revised: Bob.

Original: This allows us to keep an updated estimate of goal feasibility based on Alice and Bob’s currentskills.
Revised: This allows us to keep an updated estimate of goal feasibility based on Alice and Bob’s current skills.

Original: To empirically validate that updating the replay buffer regrets helps in a non-stationary landscape, wecarry out investigations of a SAC goal generator both with and without regret updates in Fig.
Revised: To empirically validate that updating the replay buffer regrets helps in a non-stationary landscape, we carry out investigations of a SAC goal generator both with and without regret updates in Fig.

Original: Ashypothesized, updating the replay buffer allows the generator to ﬁnd the moving optimum.
Revised: As hypothesized, updating the replay buffer allows the generator to ﬁnd the moving optimum.

Original: The combination of the replay buffer and dynamic regret updates also strengthens our method byautomatically prioritizing goals that have been forgotten.
Revised: The combination of the replay buffer and dynamic regret updates also strengthens our method by automatically prioritizing goals that have been forgotten.

Original: With the non-stationary regret updates, thegoal generators always have an updated estimate of each learner’s abilities.
Revised: With the non-stationary regret updates, the goal generators always have an updated estimate of each learner’s abilities.

Original: Thus if Bob performsworse on a past goal, the regret increases for G A and it is more likely to repropose it, addressingcatastrophic forgetting for Bob.
Revised: Thus if Bob performs worse on a past goal, the regret increases for G A and it is more likely to repropose it, addressing catastrophic forgetting for Bob.

Original: Likewise, symmetrization helps to address catastrophic forgettingfor Alice in the same manner.
Revised: Likewise, symmetrization helps to address catastrophic forgetting for Alice in the same manner.

Original: Without the corresponding G B , if Alice performs worse at a goal, theregret R A − R B decreases and G A is less likely to repropose it unless Bob’s performance decreaseseven further.
Revised: Without the corresponding G B , if Alice performs worse at a goal, the regret R A − R B decreases and G A is less likely to repropose it unless Bob’s performance decreases even further.

Original: This can potentially lead to neglecting to propose these challenging goals for Bob.
Revised: This can potentially lead to neglecting to propose these challenging goals for Bob.

Original: Summary.
Revised: Summary.

Original: Inspired by Racaniere et al.
Revised: Inspired by Racaniere et al.

Original: [32], we specify desired qualities of a goal generator:
Revised: (2020), we specify desired qualities of a goal generator:

Original: • Progressive Feasibility – the goals should be incrementally more difﬁcult, accounting for the agent’s current skill level and the fact that the agent is a non-stationary learner.
Revised: • Progressive Feasibility – the goals should be incrementally more difﬁcult, accounting for the agent’s current skill level and the fact that the agent is a non-stationary learner.

Original: [Table]

Original: • Progressive Diversity – the goals should encourage increasing exploration of goals.
Revised: • Progressive Diversity – the goals should encourage increasing exploration of goals.

Original: [Table]

Original: • Anti-Forgetting – if previous goals are forgotten, they should be reproposed.
Revised: • Anti-Forgetting – if previous goals are forgotten, they should be reproposed.

Original: [Table] We summarize the contributions of each component of our method to desired qualities of feasibility,diversity, and anti-forgetting in Table 1.
Revised: We summarize the contributions of each component of our method to desired qualities of feasibility, diversity, and anti-forgetting in Table 1.

Original: We visualize generated goal distributions in Appendix A.
Revised: We visualize generated goal distributions in Appendix A.

Revised: [Table]

Original:  Experiments
Revised:  E XPERIMENTS

Original: [Figure]
Revised: [Figure]

Original: Environments.
Revised: Environments.

Original: We test our method across a suite of continuous control tasks adapted from the
Revised: We test our method across a suite of continuous control tasks adapted from the

Original: Deepmind Control Suite [41] and OpenAI Gym [29], using the MuJoCo simulator [42].
Revised: Deepmind Control Suite (Tassa et al., 2020) and OpenAI Gym (Plappert et al., 2018), using the MuJoCo simulator (Todorov et al., 2012).

Original: We analyzeour method across a variety of task types: navigation , locomotion , and manipulation .
Revised: We analyze our method across a variety of task types: navigation , locomotion , and manipulation .

Original: Point Mass Obstacle is a navigation task with some walls.
Revised: Point Mass Obstacle is a navigation task with some walls.

Original: Walker is a locomotion task where success is achievedwhen the torso reaches the goal.
Revised: Walker is a locomotion task where success is achieved when the torso reaches the goal.

Original: Reach and Toss are manipulation tasks where success is achievedwhen moving a ball that starts in the gripper to the goal.
Revised: Reach and Toss are manipulation tasks where success is achieved when moving a ball that starts in the gripper to the goal.

Original: The latter environment provides an interestingchallenge as gravity plays a large role – the robot must learn to aim precisely.
Revised: The latter environment provides an interesting challenge as gravity plays a large role – the robot must learn to aim precisely.

Original: Pick and Reach [26]is a manipulation task where success is measured by picking up a block and moving it to a targetposition.
Revised: Pick and Reach (OpenAI, 2020) is a manipulation task where success is measured by picking up a block and moving it to a target position.

Original: Environment speciﬁcs can be found in Appendix B.2.
Revised: Environment speciﬁcs can be found in Appendix B.2.

Original: Training Setup.
Revised: Training Setup.

Original: For CuSP, we train the goal generators using the SAC implementation from Yaratsand Kostrikov [47] with their default architecture and hyperparameters.
Revised: For CuSP, we train the goal generators using the SAC implementation from Yarats & Kostrikov (2020) with their default architecture and hyperparameters.

Original: We compare against a uniformrandom generator baseline and two other baselines that use minimax formulations for curriculumgeneration: the GoalGAN generator-discriminator approach [8] and ASP+BC [28].
Revised: We compare against a uniform random generator baseline and two other baselines that use minimax formulations for curriculum generation: the GoalGAN generator-discriminator approach (Florensa et al., 2018) and ASP+BC (OpenAI et al., 2021).

Original: For all methodswe train the learner policies with the same architecture and SAC to isolate differences to the goalgeneration method.
Revised: For all methods we train the learner policies with the same architecture and SAC to isolate differences to the goal generation method.

Original: Code and further training details can be found in the Supplementary Materials.
Revised: Code and further training details can be found in the Supplementary Materials.

Original: We aim to answer the following questions with our experiments:
Revised: We aim to answer the following questions with our experiments:

Original: • Does the CuSP curricula induce agents that can generalize to novel goals from G ood ?
Revised: • Does the CuSP curricula induce agents that can generalize to novel goals from G ood ?

Original: • Does the CuSP curricula induce interesting emergent skills for solving harder goals?
Revised: • Does the CuSP curricula induce interesting emergent skills for solving harder goals?

Original: • Does the CuSP curricula handle cases where portions of the goal space are poorly speciﬁed (e
Revised: • Does the CuSP curricula handle cases where portions of the goal space are poorly speciﬁed (e

Original: g. with impossible dimensions)?
Revised: g. with impossible dimensions)?

Original: [Figure]

Original: Behind Obstacles
Revised: 4. T EST -T IME O UT -O F -D ISTRIBUTION G ENERALIZATION

Original: S u cce ss R a t e [Figure] 4.1 Test-Time Out-Of-Distribution Generalization

Original: To assess whether our curriculum improves generalizability, we evaluate each method on its abilityto reach randomly sampled out-of-distribution goals g ∼ G ood .
Revised: To assess whether our curriculum improves generalizability, we evaluate each method on its ability to reach randomly sampled out-of-distribution goals g ∼ G ood .

Original: In Fig.
Revised: In Fig.

Original: 5 we see that acrossenvironments, generally CuSP matches or outperforms all the baselines for task success.
Revised: 5 we see that across environments, generally CuSP matches or outperforms all the baselines for task success.

Original: ASP+BC’ssample inefﬁciency due to Alice’s limitations leads to poor performance, while GoalGAN does wellin environments where the goal space is more symmetrical (e.g. with navigation or the reaching tasks,where most points in the goal space are similarly difﬁcult).
Revised: ASP+BC’s sample inefﬁciency due to Alice’s limitations leads to poor performance, while GoalGAN does well in environments where the goal space is more symmetrical (e.g. with navigation or the reaching tasks, [Figure] Behind Obstacles S u cce ss R a t e

Original:  That said, we note GoalGAN strugglesmore in environments where portions of the goal space are more challenging (e.g. walking to the rightor aiming and tossing upwards).
Revised: [Figure] where most points in the goal space are similarly difﬁcult). That said, we note GoalGAN struggles more in environments where portions of the goal space are more challenging (e.g. walking to the right or aiming and tossing upwards).

Original: We hypothesize this is due to the challenges of accurately learningthe feasibility of different goals in an asymmetric goal space.
Revised: We hypothesize this is due to the challenges of accurately learning the feasibility of different goals in an asymmetric goal space.

Original: Lastly, we ﬁnd that the uniform randombaseline is particularly well suited for this OOD random evaluation.
Revised: Lastly, we ﬁnd that the uniform random baseline is particularly well suited for this OOD random evaluation.

Original: The baseline is trained on goodcoverage of the in-distribution goal space, which is beneﬁcial for generalization to random goalsin the navigation and manipulation tasks, but is less helpful for the more 
Revised: The baseline is trained on good coverage of the in-distribution goal space, which is beneﬁcial for generalization to random goals in the navigation and manipulation tasks, but is less helpful for the more asymmetric goal spaces like Walker and Toss. For ablations on the components of our method, see Appendix B.5.1 and for ablations on ASP, see Appendix B.5.2.

Original: 4.2 Task-Speciﬁc Skills
Revised: 4. T ASK -S PECIFIC S KILLS

Original: Next, we examine if CuSP can induce emergent skills in learners for tackling harder, environmentspeciﬁc goals.
Revised: Next, we examine if CuSP can induce emergent skills in learners for tackling harder, environment speciﬁc goals.

Original: Below, we enlist 3 representative examples in the more asymmetric environments[Figure] We test on both in distribution and out of distribution randomly sampled goals in the 3D goal space, where the third dimension is not feasible in the environment.
Revised: Below, we enlist 3 representative examples in the more asymmetric environments where certain goals are clearly more challenging to solve.

Original: Generally we ﬁnd CuSP is least affected while the baselines generally performs worse with the higher dimensional space.

Original: where certain goals are clearly more challenging to solve.

Original: As seen in Fig.
Revised: As seen in Fig.

Original: 6, our method is able toimprove upon success rates on these goals.
Revised: 6, our method is able to improve upon success rates on these goals.

Original: • Point Mass Obstacle – Behind Obstacle s: requires learning to move around the walls to the edge of the training space.
Revised: • Point Mass Obstacle – Behind Obstacle s: requires learning to move around the walls to the edge of the training space.

Original: Here, we learn to succeed much faster than the other methods.
Revised: Here, we learn to succeed much faster than the other methods.

Original: • Walker – Far Right : requires learning to move forward quickly to reach a goal far out of the training space.
Revised: • Walker – Far Right : requires learning to move forward quickly to reach a goal far out of the training space.

Original: Here, we learn to succeed at the task where all other baselines fail, suggesting that our method produces a useful curriculum for learning challenging locomotion tasks while the others cannot, or require many more samples to do so. 
Revised: Here, we learn to succeed at the task where all other baselines fail, suggesting that our method produces a useful curriculum for learning challenging locomotion tasks while the others cannot, or require many more samples to do so. [Figure]

Original: • Toss – Upwards : requires learning to toss the ball accurately upwards outside of the training space.
Revised: • Toss – Upwards : requires learning to toss the ball accurately upwards outside of the training space.

Original: Here, we learn to succeed much faster than the other methods.
Revised: Here, we learn to succeed much faster than the other methods.

Original: 4.3 Misspeciﬁed Goal Dimensions
Revised: 4. M ISSPECIFIED G OAL D IMENSIONS

Original: Lastly, we investigate if the goal generator methods (CuSP, GoalGAN, and Uniform) can handlethe case where parts of the speciﬁed goal space is misspeciﬁed.
Revised: Lastly, we investigate if the goal generator methods (CuSP, GoalGAN, and Uniform) can handle the case where parts of the speciﬁed goal space is misspeciﬁed.

Original: To do this, we append an extradimension to the goal space, bounded between [ − 1 , 1] , and evaluate performance on G id and G oodwith randomly sampling the third dimension during evaluation.
Revised: To do this, we append an extra dimension to the goal space, bounded between [ − 1 , 1] , and evaluate performance on G id and G ood with randomly sampling the third dimension during evaluation.

Original: We use Reach as a case study, andas seen in Fig.
Revised: We use Reach as a case study, and as seen in Fig.

Original: 7, agents trained using CuSP’s goal curricula are less affected by the increased goalspace while the baselines plateau at a lower success rate.
Revised: 7, agents trained using CuSP’s goal curricula are less affected by the increased goal space while the baselines plateau at a lower success rate.

Original: In particular, the Uniform baseline is mostadversely affected by the higher dimensionality.
Revised: In particular, the Uniform baseline is most adversely affected by the higher dimensionality.

Original: This suggests that while random sampling providesgood coverage and is amenable to more simple goal spaces, as seen above, the induced curriculadegrades in quality with increasingly complex goal spaces.
Revised: This suggests that while random sampling provides good coverage and is amenable to more simple goal spaces, as seen above, the induced curricula degrades in quality with increasingly complex goal spaces.

Original: Note that we exclude ASP from theseexperiments as it does not use a pre-speciﬁed goal space and only sets achieved states as goals.
Revised: Note that we exclude ASP from these experiments as it does not use a pre-speciﬁed goal space and only sets achieved states as goals.

Original:  Related Work
Revised:  R ELATED W ORK

Original: We summarize four main thrusts of work within automatic curriculum generation.
Revised: We summarize four main thrusts of work within automatic curriculum generation.

Original: Modeling Goal Distributions.
Revised: Modeling Goal Distributions.

Original: Many previous works have tackled the problem of generating “appropriately” difﬁcult goals for an autocurriculum.
Revised: Many previous works have tackled the problem of generating “appropriately” difﬁcult goals for an autocurriculum.

Original: In discrete environments, Campero et al.
Revised: In discrete environments, Campero et al.

Original: [5]propose an adversarial goal-generator that is rewarded when the learning agent achieves the goalwith a thresholded level of effort.
Revised: (2021) propose an adversarial goal-generator that is rewarded when the learning agent achieves the goal with a thresholded level of effort.

Original: However, they rely on a heuristic time limit hyperparameter todetermine ‘suitable effort’ (i.e. goal feasibility) and hyperparameters for the teacher reward, whereasour approach directly uses regrets from multiagent interactions to automatically gauge feasibility andprovide a reward signal to the teacher.
Revised: However, they rely on a heuristic time limit hyperparameter to determine ‘suitable effort’ (i.e. goal feasibility) and hyperparameters for the teacher reward, whereas our approach directly uses regrets from multiagent interactions to automatically gauge feasibility and provide a reward signal to the teacher.

Original: In continuous domains, Zhang et al.
Revised: In continuous domains, Zhang et al.

Original: [48] samples goals thatmaximize the uncertainty of the learner’s Q-function, and Nair et al.
Revised: samples goals that maximize the uncertainty of the learner’s Q-function, and Nair et al.

Original: [24] learn a latent representationto sample ‘imagined’ goals from. GoalGAN [8] trains a GAN to generate goals of intermediatedifﬁculty for the agent, using a discriminator to predict whether the generated goals would be appropriate by thresholding success rates.
Revised: (2018) learn a latent representation to sample ‘imagined’ goals from. GoalGAN (Florensa et al., 2018) trains a GAN to generate goals of intermediate difﬁculty for the agent, using a discriminator to predict whether the generated goals would be appropriate by thresholding success rates.

Original: These methods use a heuristic measure of challenginggoals, while we leverage the multi-agent setting to automatically ﬁnd interesting goals.
Revised: These methods use a heuristic measure of challenging goals, while we leverage the multi-agent setting to automatically ﬁnd interesting goals.

Original: Similarly to GoalGAN, the Setter-Solver paradigm [32] trains a goal setter whose objectives are a balance of goalvalidity, feasibility, and coverage, as well as a discriminative judge for predicting goal feasibility.
Revised: Similarly to GoalGAN, the Setter-Solver paradigm (Racaniere et al., 2020) trains a goal setter whose objectives are a balance of goal validity, feasibility, and coverage, as well as a discriminative judge for predicting goal feasibility.

Original: Our approach differs in that we do not need to explicitly weigh the contributions of each loss, and we donot need an accurate discriminator for estimating feasibility.
Revised: Our approach differs in that we do not need to explicitly weigh the contributions of each loss, and we do not need an accurate discriminator for estimating feasibility.

Original: Multi-Agent Curriculum Learning.
Revised: Multi-Agent Curriculum Learning.

Original: The idea that interactions between multiple agents can lead toemergent curricula has been proposed in prior works [19, 2] and can be connected with how humanslearn through social interactions.
Revised: The idea that interactions between multiple agents can lead to emergent curricula has been proposed in prior works (Leibo et al., 2019; Baker et al., 2019) and can be connected with how humans learn through social interactions.

Original: One parametrization of such interactions is through designatinganother agent(s) as a teacher who aims to select an appropriate curriculum for the student.
Revised: One parametrization of such interactions is through designating another agent(s) as a teacher who aims to select an appropriate curriculum for the student.

Original: Fortask-based curricula, Matiisen et al.
Revised: For task-based curricula, Matiisen et al.

Original: [23] train a teacher policy forselecting sub-tasks for the student,and Graves et al. [11], Portelas et al.
Revised: (2019) train a teacher policy for

Original: [30] treat task selection as a bandit problem for the teacher. Rather than having an explicit teacher, Asymmetric Self-Play (ASP) [40] automatically generatesa curriculum for exploration by training a similarly parametrized agent, Alice, to demonstrate theeasiest possible task that the learning agent, Bob, can not yet complete.
Revised: Rather than having an explicit teacher, Asymmetric Self-Play (ASP) (Sukhbaatar et al., 2018b) automatically generates a curriculum for exploration by training a similarly parametrized agent, Alice, to demonstrate the easiest possible task that the learning agent, Bob, can not yet complete.

Original: OpenAI et al.
Revised: OpenAI et al.

Original: [28] extend ASP to robotic manipulation tasks by incorporating an imitation learning loss for Bob to imitate Alice on challenging goals.
Revised: (2021) extend ASP to robotic manipulation tasks by incorporating an imitation learning loss for Bob to imitate Alice on challenging goals.

Original: As Alice is no longer setting the goals, we do not make use of behaviourcloning in CuSP.
Revised: As Alice is no longer setting the goals, we do not make use of behaviour cloning in CuSP.

Original: While they demonstrate zero-shot generalization to a diverse set of target tasks,relying on Alice to set the goals can also be incredibly sample inefﬁcient, as we saw in our evaluations.
Revised: While they demonstrate zero-shot generalization to a diverse set of target tasks, relying on Alice to set the goals can also be incredibly sample inefﬁcient, as we saw in our evaluations.

Original: Sodhani and Pahuja [38] propose a memory-augmented version of ASP by providing the learnersan external memory to speed up exploration, which is analogous to our method training with offpolicy learners.
Revised: Sodhani & Pahuja (2018) propose a memory-augmented version of ASP by providing the learners an external memory to speed up exploration, which is analogous to our method training with off-policy learners.

Original: Sukhbaatar et al.
Revised: Sukhbaatar et al.

Original: [39] extends ASP to a hierarchical learning framework by learningsubgoal representations.
Revised: (2018a) extends ASP to a hierarchical learning framework by learning subgoal representations.

Original: Competitive Experience Replay [21] generates an exploratory competitivecurriculum through penalizing Alice if visiting states Bob has visited, but rewards Bob for visitingstates found by Alice.
Revised: Competitive Experience Replay (Liu et al., 2018) generates an exploratory competitive curriculum through penalizing Alice if visiting states Bob has visited, but rewards Bob for visiting states found by Alice.

Original: Environment Design.
Revised: Environment Design.

Original: Another method for inducing curricula is through generating a sequence ofenvironments to learn in.
Revised: Another method for inducing curricula is through generating a sequence of environments to learn in.

Original: Procedural level generation is a common approach for developing games,and can lead to increased generalizability [15].
Revised: Procedural level generation is a common approach for developing games, and can lead to increased generalizability (Justesen et al., 2018).

Original: POET [44] uses a population of adversaries to generateincreasingly challenging environments for a population of learning agents.
Revised: POET (Wang et al., 2019) uses a population of adversaries to generate increasingly challenging environments for a population of learning agents.

Original: PAIRED [6] proposesoptimizing an adversarial environment designer through minimax regret between two learning agents,inspiring the objective we use for our goal generators.
Revised: PAIRED (Dennis et al., 2020) proposes optimizing an adversarial environment designer through minimax regret between two learning agents, inspiring the objective we use for our goal generators.

Original: However, this approach does not account forcatastrophic forgetting and can suffer from sample inefﬁciencies from using an on-policy method andin cases where the regret landscape is ﬂat.
Revised: However, this approach does not account for catastrophic forgetting and can suffer from sample inefﬁciencies from using an on-policy method and in cases where the regret landscape is ﬂat.

Original: That said, environment design approaches can be used in acomplementary way with our approach, extending it beyond goal generation.
Revised: That said, environment design approaches can be used in a complementary way with our approach, extending it beyond goal generation.

Original: Skill Discovery.
Revised: Skill Discovery.

Original: A closely related method for curriculum generation is through unsupervised skilldiscovery, where an agent explores an environment to automatically discover skills that can becomposed to achieve a variety of tasks.
Revised: A closely related method for curriculum generation is through unsupervised skill discovery, where an agent explores an environment to automatically discover skills that can be composed to achieve a variety of tasks.

Original: Prior methods have used information theoretic objectivesto encourage skill acquisition that covers a diverse set of behaviours [7] or generate a multi-taskdistribution [12].
Revised: Prior methods have used information theoretic objectives to encourage skill acquisition that covers a diverse set of behaviours (Eysenbach et al., 2018) or generate a multi-task distribution (Gupta et al., 2020).

Original: For robotics tasks, [14] propose a method for learning a latent skill space that can becomposed and is transferable between tasks, DADS [36] proposes a combination of model-based andmodel-free RL that optimizes for discovering predictable skills that can be composed by a planner,and Play-LMP [22] proposes using self-supervision across play data for skills discovery.
Revised: For robotics tasks, (Hausman et al., 2018) propose a method for learning a latent skill space that can be composed and is transferable between tasks, DADS (Sharma et al., 2019) proposes a combination of model-based and model-free RL that optimizes for discovering predictable skills that can be composed by a planner, and Play-LMP (Lynch et al., 2020) proposes using self-supervision across play data for skills discovery.

Original: Anothermethod for curriculum generation is through using hindsight [1, 20].
Revised: Another method for curriculum generation is through using hindsight (Andrychowicz et al., 2017; Li et al., 2020).

Original: These works are complementaryto ours as our method aims to induce a curriculum through sparse goal generation, which can beaugmented with unsupervised skill discovery.
Revised: These works are complementary to ours as our method aims to induce a curriculum through sparse goal generation, which can be augmented with unsupervised skill discovery.

Original:  Conclusion
Revised:  C ONCLUSION

Original: We propose a new method, CuSP, for automatic curriculum generation based on symmetric extensionof self play approaches for non zero-sum games.
Revised: We propose a new method, CuSP, for automatic curriculum generation based on symmetric extension of self play approaches for non zero-sum games.

Original: Our proposed method aims to address weaknesses ofprior approaches by being designed to speciﬁcally consider a balance of progressive exploration andanti-catastrophic exploitation.
Revised: Our proposed method aims to address weaknesses of prior approaches by being designed to speciﬁcally consider a balance of progressive exploration and anti-catastrophic exploitation.

Original: To this end, we make use of two off-policy students and two off-policyregret-maximizing goal generators, each of which is more ‘friendly’ to one of the students.
Revised: To this end, we make use of two off-policy students and two off-policy regret-maximizing goal generators, each of which is more ‘friendly’ to one of the students.

Original: As aresult, we generate goals that are a) challenging yet feasible, b) diverse, and c) are anti-forgetting.
Revised: As a result, we generate goals that are a) challenging yet feasible, b) diverse, and c) are anti-forgetting.

Original: We qualitatively highlight these desired qualities in goal generation compared to prior methods,demonstrate quantitative gains at generalizing to out-of-distribution unseen goals, and ﬁnd someemergent task-speciﬁc skills.
Revised: We qualitatively highlight these desired qualities in goal generation compared to prior methods, demonstrate quantitative gains at generalizing to out-of-distribution unseen goals, and ﬁnd some emergent task-speciﬁc skills.

Original: Limitations and Future Work.
Revised: Limitations and Future Work.

Original: While our method shows promise for addressing key componentsof generating useful curriculum, there is still signiﬁcant room for improvement.
Revised: While our method shows promise for addressing key components of generating useful curriculum, there is still signiﬁcant room for improvement.

Original: One limitation of ourcurrent approach is that it depends on specifying a goal space for the generator to act in, which was not necessary in the original ASP method.
Revised: One limitation of our current approach is that it depends on specifying a goal space for the generator to act in, which was not necessary in the original ASP method.

Original: That said, specifying a goal space also allows us to imbuethe system with human priors for what goals should or should not be valid, whereas ASP may needpost-hoc goal ﬁltering if Alice ends up in an undesirable state.
Revised: That said, specifying a goal space also allows us to imbue the system with human priors for what goals should or should not be valid, whereas ASP may need post-hoc goal ﬁltering if Alice ends up in an undesirable state.

Original: To further tackle automatic curriculum generation, future work should explore scenarios where goalspaces are not fully pre-speciﬁed, as well as extending to high dimensional goal spaces (e.g. images).
Revised: To further tackle automatic curriculum generation, future work should explore scenarios where goal spaces are not fully pre-speciﬁed, as well as extending to high dimensional goal spaces (e.g. images).

Original: Furthermore, another interesting avenue would be extrapolating CuSP to a larger number of studentand teacher agents, which can potentially provide beneﬁts through better goal space coverage andgiving a more accurate estimate of feasibility Checklist
Revised: Furthermore, another interesting avenue would be extrapolating CuSP to a larger number of student and teacher agents, which can potentially provide beneﬁts through better goal space coverage and giving a more accurate estimate of feasibility.

Original: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] or[N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:

Original: • Did you include the license to the code and datasets? [Yes] See Section ?? . • Did you include the license to the code and datasets? [No] The code and the data are
Revised: E THICS S TATEMENT

Original: proprietary. • Did you include the license to the code and datasets? [N/A] Please do not modify the questions and only use the provided macros for your answers.
Revised: Automatic curriculum generation has potential as a method for learning complex agent behaviours in an unsupervised way.

Original: Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below. For all authors...
Revised: Requiring less direct supervision (eg.

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work?
Revised: through hand-tuned, shaped reward functions) can lead to more robust AI systems for real-world applications.

Original: [Yes] See Section 6 for a discussion onlimitations. (c) Did you discuss any potentialnegative societal impacts of your work? [N/A] No direct
Revised: At the same time, it is important to consider potential impacts unsupervised reinforcement learning, as it may lead to unintended behaviours.

Original: ties to applications.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]3.

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes] See supplemental material for codebase.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Section 4 for architecture and training details.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [Yes] See Section 4 evaluation plots with error bars, and we

Original: run all our experiments across three seeds. (d) Did you include the total amount of compute and  the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Supplementary Materials for
Revised: By incorporating human engineering and priors through goal space speciﬁcation in the goal generative methods, like CuSP, we can help restrict the set of goal spaces to more desirable (e.g. safe) behaviours.

Original: compute details. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] The SAC implemen-
Revised: While our empirical studies have been done in simulated systems, careful and thorough analysis of learned behaviours should be done before deploying CuSP on real world systems.

Original: tation we build off of is Yarats and Kostrikov [47], the PPO implementation we build

Original: off if os [18] and we additionally use the simulated environments of Tassa et al.

Original: and Plappert et al. [29] , cited in Section 4. (b) Did you mention the license of the assets? [N/A]
Revised: R EPRODUCIBILITY S TATEMENT

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]

Original: We include our codebase in the supplementary material. [Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
Revised: See the anonymized codebase in the supplementary materials for reproducing the experiments.

Original: using/curating? [N/A] [Table] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: For experimental environment details, see Appendices B.1, B.2.

Original: [Table] information or offensive content? [N/A] [Table]
Revised: For hyperparameters, see Appendix

Original: If you used crowdsourcing or conducted research with human subjects...

Original: [Table] (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable? [N/A] [Table]
Revised: B.3.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review [Table] Board (IRB) approvals, if applicable? [N/A]
Revised: For ablations of CuSP and ASP, see Appendices B.5.1 and B.5.2 respectively.

Original: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A] [Table] A Desired Goal Generation Qualities
Revised: A D ESIRED G OAL G ENERATION Q UALITIES

Original: We begin by qualitatively examining the key properties of the curriculum of goals generated inPoint
Revised: We begin by qualitatively examining the key properties of the curriculum of goals generated in

Original:  Mass Obstacle.
Revised: Point Mass Obstacle.

Original: We see that qualitatively our method (Fig. 8a) proposes goals with more progressiveexploration than either ASP (Fig. 8d)
Revised: We see that qualitatively our method (Fig. 8a) proposes goals with more progressive exploration than either ASP (Fig. 8d)

Original: (which seems to get stuck on the walls, limited by Alice’sability to explore) or GoalGAN (Fig. 8c) (which seems to get stuck more along the borders of thespace, only prioritizing difﬁcult goals).
Revised: (which seems to get stuck on the walls, limited by Alice’s ability to explore) or GoalGAN (Fig. 8c) (which seems to get stuck more along the borders of the space, only prioritizing difﬁcult goals).

Original: The uniform random generator (Fig. 8b) has the oppositeresult, having great coverage but without any consideration for feasibility or difﬁculty.
Revised: The uniform random generator (Fig. 8b) has the opposite result, having great coverage but without any consideration for feasibility or difﬁculty.

Original: Our methodmanages to balance the exploration with progressive feasibility, where we see CuSP proposing goalsmore towards the corners at ﬁrst, then progressively moving towards the center and covering the spacemore thoroughly, with slightly more density in the top right where the maze is.
Revised: Our method manages to balance the exploration with progressive feasibility, where we see CuSP proposing goals more towards the corners at ﬁrst, then progressively moving towards the center and covering the space more thoroughly, with slightly more density in the top right where the maze is

Original: We hypothesize thereis more concentration in the corners initially when the goal generators are untrained and proposinggoals towards the edges of the goal constraints.
Revised: We hypothesize there is more concentration in the corners initially when the goal generators are untrained and proposing goals towards the edges of the goal constraints

Original: [Figure]
Revised: [Figure]

Revised: Obstacle methods.

Revised: We see more progressive diversity

Revised: CuSP, whereas GoalGAN mostly stays on the outer border and ASP+BC gets stuck on the environment walls, limited by Alice’s abilities.

Original: B Experiments
Revised: B E XPERIMENTS

Original: Compute.
Revised: Compute.

Original: To train each method in each environment, we use 1 NVIDIA V100 GPU per seed inan internal cluster.
Revised: To train each method in each environment, we use 1 NVIDIA V100 GPU per seed in an internal cluster.

Original: This amounts to an approximate total of 400 GPUs for the ﬁnal evaluations andhyperparameter sweeps.
Revised: This amounts to an approximate total of 400 GPUs for the ﬁnal evaluations and hyperparameter sweeps.

Original: We additionally did extensive prototyping on the Point Mass environmentbefore scaling to the more complex environments, but do not have a reasonable estimate for howmuch compute was expended in that phase of the project.
Revised: We additionally did extensive prototyping on the Point Mass environment before scaling to the more complex environments, but do not have a reasonable estimate for how much compute was expended in that phase of the project.

Original: B.1 Toy Experiments
Revised: B. T OY E XPERIMENTS

Original: For the toy evaluations in Figures 2 and 3, we evaluate on the landscaperegret ( x, y ) 
Revised: For the toy evaluations in Figures 2 and 3, we evaluate on the landscape regret ( x, y ) =

Original: = (cid:26) − (( x + . 1) 

Revised: (cid:26) − (( x + . 1)

Original: + ( y − . 1) 2 )
Revised: + ( y − . 1) 2 )

Original: if ( x + . 1) 2 + ( y − . 1) 2 < .− .otherwise
Revised: if ( x + . 1) 2 + ( y − . 1) 2 < .− .otherwise

Original: For Adam, we use the default conﬁguration from PyTorch (lr=0.001, β s = (0.9, 0.999), (cid:15) =1e-8, noweight decay).
Revised: For Adam, we use the default conﬁguration from PyTorch (lr=0.001, β s = (0.9, 0.999), (cid:15) =1e-8, no weight decay).

Original: For PPO, we use the default conﬁguration from [18].
Revised: For PPO, we use the default conﬁguration from (Kostrikov, 2018).

Original: We found our trends to holdrobustly for a fairly broad choice of lr’s and both with/without GAE [35].
Revised: We found our trends to hold robustly for a fairly broad choice of lr’s and both with/without GAE (Schulman et al., 2018).

Original: In the toy experiments, wecarry out 5000 steps of goal proposal with one gradient update for every method at each step.
Revised: In the toy experiments, we carry out 5000 steps of goal proposal with one gradient update for every method at each step.

Original: Forthe non-stationary experiment in Figure 3, we diagonally perturb the center of the regret landscapefrom top left quadrant (initialized at (-0.1, 0.1)) to the bottom right quadrant at a constant rate of 2e-4every round.
Revised: For the non-stationary experiment in Figure 3, we diagonally perturb the center of the regret landscape from top left quadrant (initialized at (-0.1, 0.1)) to the bottom right quadrant at a constant rate of 2e-4 every round.

Original: Hence, at the end of 2500 rounds, the center of the landscape is at the origin and at theend of 5000 rounds, the center of the landscape is at (0.1, -0.1).
Revised: Hence, at the end of 2500 rounds, the center of the landscape is at the origin and at the end of 5000 rounds, the center of the landscape is at (0.1, -0.1).

Original: B.2 Environment Speciﬁcs
Revised: B. E NVIRONMENT S PECIFICS

Original: Table 2 describes the reward conﬁgurations we used in our environments.
Revised: Table 2 describes the reward conﬁgurations we used in our environments.

Original: For the manipulationenvironments, we also incorporated HER [1].
Revised: For the manipulation environments, we also incorporated HER (Andrychowicz et al., 2017).

Original: (cid:15) is the tolerance for success (i.e. a goal g is successfullyachieved at state s if the normed distance L 2 ( s − g ) < (cid:15) ).
Revised: (cid:15) is the tolerance for success (i.e. a goal g is successfully achieved at state s if the normed distance L 2 ( s − g ) < (cid:15) ).

Original: We terminate the episode early if successful.
Revised: We terminate the episode early if successful.

Revised: [Table]

Revised: Use HER (Andrychowicz et al., 2017) (cid:15)

Original: [Table]
Revised: [Table]

Original: We modify the default Point Mass environment by adding in walls to the top right corner and initializethe agent in between the walls 10% of the time to help with exploration.
Revised: We modify the default Point Mass environment by adding in walls to the top right corner and initialize the agent in between the walls 10% of the time to help with exploration.

Original: We modify the manipulatorenvironment by always initializing with the ball in the gripper and with the gripper in an uprightposition.
Revised: We modify the manipulator environment by always initializing with the ball in the gripper and with the gripper in an upright position.

Original: We use an episode length of 1000 steps for Point Mass Obstacle and Walker and an episodelength of 100 steps for Reach, Toss, and Pick.
Revised: We use an episode length of 1000 steps for Point Mass Obstacle and Walker and an episode length of 100 steps for Reach, Toss, and Pick.

Original: Speciﬁc deﬁnitions of in-distribution and out-of-distribution goal spaces for each environment are in Table
Revised: Speciﬁc deﬁnitions of in-distribution and out-of-distribution goal spaces for each environment are in Table

Original: [Table]
Revised: [Table]

Original: B.3 Hyperparameters
Revised: B. H YPERPARAMETERS

Original: We train the learners and goal generators using the default SAC conﬁguration and implementationfrom [47], as listed in Table 4.
Revised: We train the learners and goal generators using the default SAC conﬁguration and implementation from (Yarats & Kostrikov, 2020), as listed in Table 4.

Revised: To scale the output of the goal generators to the correct action space

Original: [Table]
Revised: [Table]

Original: To scale the output of the goal generators to the correct action space, we take the Tanh scaled outputand rescale each dimension to the goal space.
Revised: we take the Tanh scaled output and rescale each dimension to the goal space.

Original: We use Appendix C of [13] to modify the log loss for SAC accordingly.
Revised: We use Appendix C of (Haarnoja et al., 2018) to modify the log loss for SAC accordingly.

Original: To train the goal generator, we update after each round (i.e. per goal proposed) with100 gradient updates.
Revised: To train the goal generator, we update after each round (i [Table] e. per goal proposed) with 100 gradient updates.

Original: As the learners are updated once per time-step in the environment (100-1000updates depending on the environment), we increased the update frequency  for the goal-generator tokeep pace with the learner updates since the goal generator operates on a single time step per episode. 
Revised: As the learners are updated once per time-step in the environment (100-1000 updates depending on the environment), we increased the update frequency [Table] for the goal-generator to keep pace with the learner updates since the goal generator operates on a single time step per episode. [Table]

Original: B.4 Baselines
Revised: B

Revised: [Table]

Original: We reimplement each baseline by comparing against any available public code and matching eachpaper’s recommended training hyperparameters as closely as possible within our compute limitations.
Revised: We reimplement each baseline by comparing against any available public code and matching each paper’s recommended training hyperparameters as closely as possible within our compute limitations.

Original: Off vs. On Policy Learners In ASP+BC[28], Alice and Bob are trained using PPO.
Revised: Off vs. On Policy Learners In ASP+BC(OpenAI et al., 2021), Alice and Bob are trained using PPO.

Original: We alsoimplemented this as a baseline but found that the agents were unable to reach high success at thesame rate as our SAC-trained agents.
Revised: We also implemented this as a baseline but found that the agents were unable to reach high success at the same rate as our SAC-trained agents.

Original: For a comparison that’s more favorable to the baselines, wereport the success rates from training with the same agent conﬁguration (ie.
Revised: For a comparison that’s more favorable to the baselines, we report the success rates from training with the same agent conﬁguration (ie.

Original: Bob trained with SAC)in the main paper to isolate learning differences to the goal generation method.
Revised: Bob trained with SAC) in the main paper to isolate learning differences to the goal generation method.

Original: B.5 Ablations
Revised: B

Revised: [Table]

Original: SAC Motivation Without symmetrization, the regret-based goal generation objective is the same asthe proposed objective in PAIRED [6].
Revised: SAC Motivation Without symmetrization , the regret-based goal generation objective is the same as the proposed objective in PAIRED Dennis et al.

Original: However, our contribution extends beyond the symmetrizedobjective as our proposed method reframes the goal generation process into a entropy-regularizedagent with a memory buffer – which motivated designing our method around SAC.
Revised: However, our contribution extends beyond the symmetrized objective as our proposed method reframes the goal generation process into a entropy-regularized agent with a memory buffer – which motivated designing our method around SAC.

Original: PAIRED uses PPO to optimize their environment generators, which we found was not sample efﬁcient enough forour tasks (i.e. success rates were too low), as our toy experiment in Figure 2 suggested.
Revised: PAIRED uses PPO to optimize their environment generators, which we found was not sample efﬁcient enough for our tasks (i.e. success rates were too low), as our toy experiment in Figuresuggested.

Original: Here we highlight a case study with the Toss task into why the entropy regularization and replaybuffer components are important.
Revised: Here we highlight a case study with the Toss task into why the entropy regularization and replay buffer components are important.

Original: Building up from just the regret objective as in PAIRED, we havein Figure 9:
Revised: Building up from just the regret objective as in PAIRED, we have in Figure 9:

Original: β
Revised: β

Original: = 1 , α = 0 , no replay buffer – Regret objective only for a single goal generator, as inPAIRED.
Revised: = 1 , α = 0 , no replay buffer – Regret objective only for a single goal generator, as in PAIRED.

Original: β
Revised: β

Original: = 1 , α = 0 , no replay buffer, symmetrized – Regret objective with two symmetrized goalgenerators.
Revised: = 1 , α = 0 , no replay buffer, symmetrized – Regret objective with two symmetrized goal generators.

Original: Symmetrization seems to improve performance slightly, but is not sufﬁcient alone.
Revised: Symmetrization seems to improve performance slightly, but is not sufﬁcient alone.

Original: β
Revised: β

Original: = 1 , α = 0 – Regret objective with a replay buffer for the goal generator.
Revised: = 1 , α = 0 – Regret objective with a replay buffer for the goal generator.

Original: Also improves performance slightly, although not as much as symmetrization.
Revised: Also improves performance slightly, although not as much as symmetrization.

Original: β=
Revised: β

Original:  1 – Regret objective with a replay buffer and entropy regularization.
Revised: = 1 – Regret objective with a replay buffer and entropy regularization.

Original: This is particularlyhelpful for the random OOD evaluation as the greater goal diversity improves multi-goalperformance on the multi-goal evaluation task, as expected.
Revised: This is particularly helpful for the random OOD evaluation as the greater goal diversity improves multi-goal performance on the multi-goal evaluation task, as expected.

Original: β = 1 symmetrized – Regret objective with a replay buffer and entropy regularization, and symmetrized goal generators.
Revised: β = 1 symmetrized – Regret objective with a replay buffer and entropy regularization, and symmetrized goal generators.

Original: T = 300 , β = .
Revised: T = 300 , β = .

Original: 1 , symmetrized (Full CuSP) – Regret objective with a dynamically updated replay buffer and entropy regularization, and symmetrized goal generators.
Revised: 1 , symmetrized (Full CuSP) – Regret objective with a dynamically updated replay buffer and entropy regularization, and symmetrized goal generators.

Original: Looking at a sample of the generated goals in Figure 10, we ﬁnd a stark qualitative improvement ingoal diversity between the regret-only PAIRED condition and CuSP.
Revised: Looking at a sample of the generated goals in Figure 10, we ﬁnd a stark qualitative improvement in goal diversity between the regret-only PAIRED condition and CuSP.

Original: In particular, in the baseline thegenerated goals converge only on the corners.
Revised: In particular, in the baseline the generated goals converge only on the corners.

Original: Regret update and Symmetrization Ablations As we use the learners’ critics for estimating thecurrent regret for a particular goal, we need to be careful about how we use the critic data since itcan be inaccurate, especially towards the beginning of training.
Revised: Regret update and Symmetrization Ablations As we use the learners’ critics for estimating the current regret for a particular goal, we need to be careful about how we use the critic data since it can be inaccurate, especially towards the beginning of training.

Original: We introduce two hyperparameters:one for specifying at which episode we begin regret updates T , and a weighting parameter β for howmuch to update the regrets by.
Revised: We introduce two hyperparameters: one for specifying at which episode we begin regret updates T , and a weighting parameter β for how much to update the regrets by.

Original: We use the update rule
Revised: We use the update rule

Original: [Equation]
Revised: [Equation]

Original: We sweep across T = 50 , 100 , 200 and β = .
Revised: We sweep across T = 50 , 100 , 200 and β = .

Original: 9 to select the best performing hyperparametersbased on average performance on the training goal set, G id .
Revised: 9 to select the best performing hyperparameters based on average performance on the training goal set, G id .

Original: In Figure 11, we ablate over thedifferent components of our method: CuSP+SAC with no entropy regularization ( β = 1 , α =0 ), CuSP+SAC only ( β = 1 ), CuSP+SAC+Symmetrization ( β = 1 ), and CuSP+SAC+Regret Updates+Symmetrization to investigate the effects of each component.
Revised: In Figure 11, we ablate over the different components of our method: CuSP+SAC with no entropy regularization ( β = 1 , α = 0 ), CuSP+SAC only ( β = 1 ), CuSP+SAC+Symmetrization ( β = 1 ), and CuSP+SAC+Regret Updates+Symmetrization to investigate the effects of each component.

Original: In most of the tasks, asidefrom Reach on random out of distribution goals, symmetrization improves performance.
Revised: In most of the tasks, aside from Reach on random out of distribution goals, symmetrization improves performance.

Original: In particular,we see larger gains on tasks where β = 1 performance is lower to begin with, such as Walker.
Revised: In particular, we see larger gains on tasks where β = 1 performance is lower to begin with, such as Walker.

Original: Furthergains from stale regret updates vary from being very small on tasks where β = 1 performance isalready high (eg. for Point Mass Obstacles or Toss on random OOD goals), to larger improvementson the harder skill tasks (eg.
Revised: Further gains from stale regret updates vary from being very small on tasks where β = 1 performance is already high (eg. for Point Mass Obstacles or Toss on random OOD goals), to larger improvements on the harder skill tasks (eg.

Original: Behind Obstacles or Walker Far Right goals).
Revised: Behind Obstacles or Walker Far Right goals).

Original: Interestingly, we also notethat while entropy regularization is crucial for most of the environments, setting α = 0 seems to behelpful for Walker.[Figure] S
Revised: Interestingly, we also note that while entropy regularization is crucial for most of the environments, setting α = 0 seems to be helpful for Walker.

Original: [Figure]
Revised: [Figure]

Original: cce ss R a t e
Revised: cce ss R a t e

Original: [Figure]
Revised: [Figure]

Original:  cce ss R a t e
Revised: S u cce ss R a t e

Original: [Figure]
Revised: [Figure]

Original: Multiagent Motivation.
Revised: Multiagent Motivation.

Original: Here we investigate whether it is sufﬁcient to formulate the CuSP gamewith only a single learner.
Revised: Here we investigate whether it is sufﬁcient to formulate the CuSP game with only a single learner.

Original: Rather than using regret deﬁned as the difference in returns of two agents,we look at the difference in the returns of a single agent across two rollouts to identify if stochasticityin the policy alone is sufﬁcient for providing the goal generator signal.
Revised: Rather than using regret deﬁned as the difference in returns of two agents, we look at the difference in the returns of a single agent across two rollouts to identify if stochasticity in the policy alone is sufﬁcient for providing the goal generator signal.

Original: As a case study we look atthe Toss task in Figure 12, where we ﬁnd the success rate increases much more slowly with a singleagent.
Revised: As a case study we look at the Toss task in Figure 12, where we ﬁnd the success rate increases much more slowly with a single agent.

Original: We hypothesize that a challenge with only using a single agent fro deﬁning regret is that theregret landscape is much more ﬂat – i.e. the regret is more likely to be very small and not providemuch signal to the goal generators, as the stochasticity of a single agent policy is comparatively lesserthan two separate agents.
Revised: We hypothesize that a challenge with only using a single agent fro deﬁning regret is that the regret landscape is much more ﬂat – i.e. the regret is more likely to be very small and not provide much signal to the goal generators, as the stochasticity of a single agent policy is comparatively lesser than two separate agents.

Original: As a result, the goals are less likely to be a useful curriculum for the agent.
Revised: As a result, the goals are less likely to be a useful curriculum for the agent.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  u cce ss R a t e
Revised: S u cce ss R a t e

Original: [Figure]
Revised: [Figure]

Original: S u cce ss R a t e
Revised: S u cce ss R a t e

Original: [Figure]
Revised: [Figure]

Original: In the original implementation, ASP+BC uses PPO to optimize both Alice and Bob.
Revised: In the original implementation, ASP+BC uses PPO to optimize both Alice and Bob.

Original: We ﬁnd this verysuboptimal for the environment setups in this work.
Revised: We ﬁnd this very suboptimal for the environment setups in this work.

Original: For empirical validation, here we report resultsusing 3 variations of ASP+BC:
Revised: For empirical validation, here we report results using 3 variations of ASP+BC:

Original: • ASP+BC (Sparse): Alice is rewarded +1 at the end of an episode of Bob does not succeed,or receives 0 reward otherwise.
Revised: • ASP+BC (Sparse): Alice is rewarded +1 at the end of an episode of Bob does not succeed, or receives 0 reward otherwise.

Original: Both Alice and Bob are optimized using PPO.
Revised: Both Alice and Bob are optimized using PPO.

Original: • ASP+BC (Dense): Alice is rewarded negative Bob’s reward at each transition.
Revised: • ASP+BC (Dense): Alice is rewarded negative Bob’s reward at each transition.

Original: Both Aliceand Bob are optimized using PPO.• ASP+BC (Dense, SAC): Alice is rewarded negative Bob’s reward at each transition.
Revised: Both Alice and Bob are optimized using PPO. • ASP+BC (Dense, SAC): Alice is rewarded negative Bob’s reward at each transition.

Original: Aliceis optimized with PPO while Bob is optimized with SAC.
Revised: Alice is optimized with PPO while Bob is optimized with SAC.

Original: For all cases, we use the behavioural cloning (BC) mechanism proposed in Section 3.2 of [28], where Bob is updated with the same clipped behavioural cloning loss on trajectories where it does notsuccessfully achieve Alice’s proposed goal.
Revised: For all cases, we use the behavioural cloning (BC) mechanism proposed in Section 3.2 of (OpenAI et al., 2021), where Bob is updated with the same clipped behavioural cloning loss on trajectories where it does not successfully achieve Alice’s proposed goal.

Original: The results are shown in Figure 14.
Revised: The results are shown in Figure 14.

Original: We observe that with a sparse reward, Alice would tend toget stuck with proposing similar goals in a particular location.
Revised: We observe that with a sparse reward, Alice would tend to get stuck with proposing similar goals in a particular location.

Original: While Bob learns to succeed at thatsingle goal, the lack of diversity causes Bob to perform poorly at generalization to novel goals in
Revised: While Bob learns to succeed at that single goal, the lack of diversity causes Bob to perform poorly at generalization to novel goals in

Original: G ood , as seen in the evaluations.
Revised: G ood , as seen in the evaluations.

Original: To overcome this, we incorporated a dense reward implementationand swapped to optimizing for Bob’s policy using SAC.
Revised: To overcome this, we incorporated a dense reward implementation and swapped to optimizing for Bob’s policy using SAC.

Original: Our results presented in Section 4 use thisstronger baseline for comparison against CuSP.
Revised: Our results presented in Section 4 use this stronger baseline for comparison against CuSP.

Original: C Theoretical Analysis
Revised: C T HEORETICAL A NALYSIS

Original: Our overall framework contains 4 players: two goal-conditioned agents, Alice π A and Bob π B , andtheir corresponding goal generators, G A and G B .
Revised: Our overall framework contains 4 players: two goal-conditioned agents, Alice π A and Bob π B , and their corresponding goal generators, G A and G B .

Original: Further, let Π and G denote the space of policiesand goal generative models respectively.
Revised: Further, let Π and G denote the space of policies and goal generative models respectively.

Original: At any given round, let g A ∼ G A and g B ∼ G B denote thecorresponding goals sampled from G A and G B respectively.
Revised: At any given round, let g A ∼ G A and g B ∼ G B denote the corresponding goals sampled from G A and G B respectively.

Original: Accordingly, we can deﬁne two regretsfor the sampled goals:
Revised: Accordingly, we can deﬁne two regrets for the sampled goals:

Original: [Equation]
Revised: [Equation]

Original: where R ( g, π A ) and R ( g, π B ) denote the empirical discounted sum of rewards for Alice and Bobrespectively on a goal g .
Revised: where R ( g, π A ) and R ( g, π B ) denote the empirical discounted sum of rewards for Alice and Bob respectively on a goal g .

Original: Now, we deﬁne the objective function f below:
Revised: Now, we deﬁne the objective function f below:

Original: [Equation]
Revised: [Equation]

Original: The overall objective function for CuSP can be written as:
Revised: The overall objective function for CuSP can be written as:

Original: [Equation]
Revised: [Equation]

Original: To analyze the above objective, we deﬁne coordinating goal generator-solver teams ( G, π ) as a pairof goal generator G ∈ G and goal conditioned agent π ∈ Π .
Revised: To analyze the above objective, we deﬁne coordinating goal generator-solver teams ( G, π ) as a pair of goal generator G ∈ G and goal conditioned agent π ∈ Π .

Original: Accordingly, we can interpret CuSPas a 2 player zero-sum game between two coordinating goal generator-solver teams ( G A , π A ) and( G B , π B ) deﬁned via goal-conditioned agents π A ∈ Π (Alice) and π B ∈ Π (Bob) along with theircorresponding friendly goal generators G A ∈ G and G B ∈ G respectively.
Revised: Accordingly, we can interpret CuSP as a 2 player zero-sum game between two coordinating goal generator-solver teams ( G A , π A ) and ( G B , π B ) deﬁned via goal-conditioned agents π A ∈ Π (Alice) and π B ∈ Π (Bob) along with their corresponding friendly goal generators G A ∈ G and G B ∈ G respectively.

Original: Deﬁnition C.1.
Revised: Deﬁnition C.1.

Original: (CuSP game)
Revised: (CuSP game)

Original: The CuSP game is a 2 player zero-sum game between two coordinatinggoal generator-solver teams ( G A , π A ) and ( G B , π B ) deﬁned via:
Revised: The CuSP game is a 2 player zero-sum game between two coordinating goal generator-solver teams ( G A , π A ) and ( G B , π B ) deﬁned via:

Original: • Alice π A ∈ Π and its friendly goal generator G A ∈ G and • Bob π B ∈ Π and its friendly goal generator G B ∈ G .
Revised: • Alice π A ∈ Π and its friendly goal generator G A ∈ G and • Bob π B ∈ Π and its friendly goal generator G B ∈ G .

Original: The game is zero-sum with payoff function given as F ( G A , G B , π A , π B ) for ( G A , π A ) , and as− F ( G A , G B , π A , π B ) for ( G B , π B ) .
Revised: The game is zero-sum with payoff function given as F ( G A , G B , π A , π B ) for ( G A , π A ) , and as − F ( G A , G B , π A , π B ) for ( G B , π B ) .

Original: Next, we state conditions under which a Nash Equilibrium exists for coordinating goal generatorsolver agents in CuSP.
Revised: Next, we state conditions under which a Nash Equilibrium exists for coordinating goal generatorsolver agents in CuSP.

Original: The conditions depend highly on whether the game strategies (given by theproduct of goal generator space and policy space) are ﬁnite or continuous.
Revised: The conditions depend highly on whether the game strategies (given by the product of goal generator space and policy space) are ﬁnite or continuous.

Original: Proposition C.2.
Revised: Proposition C.2.

Original: (Finite Games) Let ( G A , π A ) ∈ G× Π and ( G B ,
Revised: (Finite Games) Let ( G A , π A ) ∈ G× Π and ( G B ,

Original: π B ) ∈
Revised: π B ) ∈

Original: G× Π be two coordinatinggoal generator-solver agents for CuSP deﬁned over ﬁnite goal space G and policy space Π .
Revised: G× Π be two coordinating goal generator-solver agents for CuSP deﬁned over ﬁnite goal space G and policy space Π .

Original: Then,there exists a mixed strategy Nash Equilibrium for the CuSP game.
Revised: Then, there exists a mixed strategy Nash Equilibrium for the CuSP game.

Original: Proof.
Revised: Proof.

Original: Since G and Π are ﬁnite, we know that the product space G × Π is also ﬁnite.
Revised: Since G and Π are ﬁnite, we know that the product space G × Π is also ﬁnite.

Original: The result thenfollows directly from Nash’s Theorem [25].
Revised: The result then follows directly from Nash’s Theorem (Nash, 1951).

Original: Proposition C.3.
Revised: Proposition C.3.

Original: (Continuous Games) Let ( G A ,
Revised: (Continuous Games) Let ( G A ,

Original: π A ) ∈ G × Π
Revised: π A ) ∈ G × Π

Original: and
Revised: and

Original: ( G B , π B ) ∈
Revised: ( G B , π B ) ∈

Original: G × Π be twocoordinating goal generator-solver agents for CuSP deﬁned over continuous goal space G and policyspace Π .
Revised: G × Π be two coordinating goal generator-solver agents for CuSP deﬁned over continuous goal space G and policy space Π .

Original: Further, let G and Π be nonempty compact metric spaces and the payoff function F (Eq. 4)be continuous.
Revised: Further, let G and Π be nonempty compact metric spaces and the payoff function F (Eq. 4) be continuous.

Original: Then, there exists a mixed strategy Nash Equilibrium for the CuSP game.
Revised: Then, there exists a mixed strategy Nash Equilibrium for the CuSP game.

Original: Proof.
Revised: Proof.

Original: Since G and Π are compact, we know by Tychonoff’s theorem [43, 46] that the product space G × Π is also compact.
Revised: Since G and Π are compact, we know by Tychonoff’s theorem (Tychonoff, 1930; Willard, 2012) that the product space G × Π is also compact.

Original: Combined with the fact that F is assumed to be continuous, we can apply Glicksberg’s Theorem [10] to ﬁnish the proof.
Revised: Combined with the fact that F is assumed to be continuous, we can apply Glicksberg’s Theorem (Glicksberg, 1952) to ﬁnish the proof.

Original: Since we have established conditions for the existence of Nash equilibria for the CuSP game, weknow from the minimax theorem that we can recover the Nash equilibirum solution by optimizing
Revised: Since we have established conditions for the existence of Nash equilibria for the CuSP game, we know from the minimax theorem that we can recover the Nash equilibirum solution by optimizing

Original: Eq. 4.
Revised: Eq. 4.

Original: Note 1:
Revised: Note 1:

Original: In practice, our algorithms for optimizing Eq.
Revised: In practice, our algorithms for optimizing Eq.

Original: 4 might not satisfy the necessary assumptionsfor the theoretical results.
Revised: 4 might not satisfy the necessary assumptions for the theoretical results.

Original: For example, we optimize the parameters of generators and policiesspeciﬁed as neural networks as opposed to optimizing in the space of functions directly.
Revised: For example, we optimize the parameters of generators and policies speciﬁed as neural networks as opposed to optimizing in the space of functions directly.

Original: Further, weconsider a single-sample Monte Carlo estimate for the CuSP objective in Eq. 4 and optimize thegoal generators and policy networks sequentially for a ﬁxed number of gradient updates in everyround using RL algorithms such as SAC.
Revised: Further, we consider a single-sample Monte Carlo estimate for the CuSP objective in Eq. 4 and optimize the goal generators and policy networks sequentially for a ﬁxed number of gradient updates in every round using RL algorithms such as SAC.

Original: While these protocols are standard practice even in relatedworks, deriving theoretical guarantees in such scenarios is extremely challenging and an active areaof theoretical research with many open questions.
Revised: While these protocols are standard practice even in related works, deriving theoretical guarantees in such scenarios is extremely challenging and an active area of theoretical research with many open questions.

Original: Note 2: If π A = π B , then r A ( g ) =
Revised: Note 2: If π A = π B , then r A ( g ) =

Original: r B ( g ) for all g , so all corresponding regrets will also be zeroand thus give no signal to the goal generators.
Revised: r B ( g ) for all g , so all corresponding regrets will also be zero and thus give no signal to the goal generators.

Original: The key point to note here is that in practice, the goalgenerators have an exploration component that can avoid getting stuck, e.g., this can be achieved via(cid:15) -greedy or via SAC which has an exploration bonus (as done in CuSP).
Revised: The key point to note here is that in practice, the goal generators have an exploration component that can avoid getting stuck, e.g., this can be achieved via (cid:15) -greedy or via SAC which has an exploration bonus (as done in CuSP).

Original: Concretely, let π A
Revised: Concretely, let π A

Original: = π B = π rand be randomly initialized agents.
Revised: = π B = π rand be randomly initialized agents.

Original: We claim that at initialization,( G A , π rand ) , ( G B , π rand ) will not be in a Nash Equilibrium except in the case where a randompolicy is already able to achieve all possible goals.
Revised: We claim that at initialization, ( G A , π rand ) , ( G B , π rand ) will not be in a Nash Equilibrium except in the case where a random policy is already able to achieve all possible goals.

Original: To see why, suppose for the sake of contradictionthat the agent teams ( G A , π rand ) , ( G B , π rand ) are at a Nash Equilibrium.
Revised: To see why, suppose for the sake of contradiction that the agent teams ( G A , π rand ) , ( G B , π rand ) are at a Nash Equilibrium.

Original: That is, the teams havethe joint best responses to each other.
Revised: That is, the teams have the joint best responses to each other.

Original: Let G be the set of all possible goals and G (cid:48) ⊆
Revised: Let G be the set of all possible goals and G (cid:48) ⊆

Original: G be the set ofgoals that π rand can solve.
Revised: G be the set of goals that π rand can solve.

Original: If G (cid:48)
Revised: If G (cid:48)

Original: =
Revised: =

Original: G , then π A , π B are already maximally competent and are therefore done learning.
Revised: G , then π A , π B are already maximally competent and are therefore done learning.

Original: Note thatthis is also what we hope to converge to eventually during training, and the two policies matching cancorrespond to a Nash Equilibrium in such a case.
Revised: Note that this is also what we hope to converge to eventually during training, and the two policies matching can correspond to a Nash Equilibrium in such a case.

Original: At this point R ( g A , π A , π B ) =
Revised: At this point R ( g A , π A , π B ) =

Original: R ( g B , π B , π A ) = {+
+}
Revised: R ( g B , π B , π A ) = {+
+}

Original: 0for all g A , g B so neither G A , G B can increase their payoffs, and since π A , π B are parameterizedidentically, if either policy is able to increase its payoff on some g we can identically update the otherpolicy such that π A
Revised: for all g A , g B so neither G A , G B can increase their payoffs, and since π A , π B are parameterized identically, if either policy is able to increase its payoff on some g we can identically update the other policy such that π A

Original: = π B until neither policy is able to improve its payoff further either.
Revised: = π B until neither policy is able to improve its payoff further either.

Original: If G (cid:48) ⊂ G , then for either team we can pick any single goal g ∗ ∈
Revised: If G (cid:48) ⊂ G , then for either team we can pick any single goal g ∗ ∈

Original: G \ G (cid:48) and learn a policy π ∗capable of achieving { g ∗ }∪ G (cid:48) using policy iteration (e.g., with greedy policy improvement assumingno forgetting of past goals).
Revised: G \ G (cid:48) and learn a policy π ∗ capable of achieving { g ∗ } ∪ G (cid:48) using policy iteration (e.g., with greedy policy improvement assuming no forgetting of past goals).

Original: Such a g ∗ will eventually be proposed as long as the goal generatorshave an exploratory component.
Revised: Such a g ∗ will eventually be proposed as long as the goal generators have an exploratory component.

Original: This means there is a strategy π ∗ that acquires equal utility toπ rand on G (cid:48) and higher utility on g ∗ .
Revised: This means there is a strategy π ∗ that acquires equal utility to π rand on G (cid:48) and higher utility on g ∗ .

Original: Thus there exists a strictly better strategy for either team( G A , π rand ) , ( G B , π rand ) by switching from π rand to π ∗ , which contradicts that the players arealready at a Nash Equilibrium to begin with.
Revised: Thus there exists a strictly better strategy for either team ( G A , π rand ) , ( G B , π rand ) by switching from π rand to π ∗ , which contradicts that the players are already at a Nash Equilibrium to begin with.