Original: I T T AKES F OUR TO T ANGO : M ULTIAGENT S ELFPLAY FOR A UTOMATIC C URRICULUM G ENERATION
Revised: I T T AKES F OUR TO T ANGO : M ULTIAGENT S ELFPLAY FOR A UTOMATIC C URRICULUM G ENERATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We are interested in training general-purpose reinforcement learning agents that can solve a wide variety of goals.
Revised: We are interested in training general-purpose reinforcement learning agents that can solve a wide variety of goals.

Original: Training such agents efﬁciently requires automatic generation of a goal curriculum .
Revised: Training such agents efﬁciently requires automatic generation of a goal curriculum .

Original: This is challenging as it requires (a) exploring goals of increasing difﬁculty, while ensuring that the agent (b) is exposed to a diverse set of goals in a sample efﬁcient manner and (c) does not catastrophically forget previously solved goals.
Revised: This is challenging as it requires (a) exploring goals of increasing difﬁculty, while ensuring that the agent (b) is exposed to a diverse set of goals in a sample efﬁcient manner and (c) does not catastrophically forget previously solved goals.

Original: We propose Curriculum Self Play (CuSP), an automated goal generation framework that seeks to satisfy these desiderata by virtue of a multi-player game with 4 agents.
Revised: We propose Curriculum Self Play (CuSP), an automated goal generation framework that seeks to satisfy these desiderata by virtue of a multi-player game with 4 agents.

Original: The game  carefully balances cooperation and competition between two off-policy student learners and two regret-maximizing teachers who generate goals that are challenging yet feasible.
Revised: We extend the asymmetric curricula learning in PAIRED (Dennis et al., 2020) to a symmetrized game that carefully balances cooperation and competition between two off-policy student learners and two regret-maximizing teachers.

Original: The teachers also account for goal coverage and  the non-stationary nature of the students, allowing us to automatically induce a curriculum that balances progressive exploration with anti-catastrophic exploitation.
Revised: CuSP additionally introduces entropic goal coverage and accounts for the non-stationary nature of the students, allowing us to automatically induce a curriculum that balances progressive exploration with anticatastrophic exploitation.

Original: Our experiments demonstrate that our method succeeds at generating  effective curricula of goals for a range of complex robotics tasks and signiﬁcantly outperforms asymmetric self-play and othercompeting methods at zero-shot test-time generalization to novel out-of-distribution goals.
Revised: We demonstrate that our method succeeds at generating an effective curricula of goals for a range of control tasks, outperforming other methods at zero-shot test-time generalization to novel out-of-distribution goals.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) has seen great success in a range of applications, from game-playing (Silver et al., 2016; Schrittwieser et al., 2020) to real world robotics tasks (OpenAI et al., 2019).
Revised: Reinforcement learning (RL) has seen great success in a range of applications, from game-playing (Silver et al., 2016; Schrittwieser et al., 2020) to real world robotics tasks (OpenAI et al., 2019).

Original: However, these accomplishments are often in the realm of mastering a single speciﬁc task.
Revised: However, these accomplishments are often in the realm of mastering a single speciﬁc task.

Original: For many applications, we further desire agents who are capable of generalizing to a variety of tasks.
Revised: For many applications, we further desire agents who are capable of generalizing to a variety of tasks.

Original: This can achieved via goal-conditioned RL, where an RL agent is trained to solve many goals by specifying a sequence of goals, i.e. a goal curriculum.
Revised: This can achieved via goal-conditioned RL, where an RL agent is trained to solve many goals by specifying a sequence of goals, i.e. a goal curriculum.

Original: Manually designing such curricula is tedious and moreover, does not scale due to being environment speciﬁc and heuristic-driven.
Revised: Manually designing such curricula is tedious and moreover, does not scale due to being environment speciﬁc and heuristic-driven.

Original: Even when it is easy to engineer a curriculum, we would prefer curricula that can dynamically adapt to the learner’s current capabilities.
Revised: Even when it is easy to engineer a curriculum, we would prefer curricula that can dynamically adapt to the learner’s current capabilities.

Original: For example, effective human tutors use an understanding of their pupil’s current knowledge and learning trajectory to create a curriculum.
Revised: For example, effective human tutors use an understanding of their pupil’s current knowledge and learning trajectory to create a curriculum.

Original: Our work builds on two broad desiderata for curriculum generation (Bengio et al., 2009) that enable sample-efﬁcient learning of goal-conditioned policies: progressive exploration and anti-catastrophic exploitation .
Revised: Our work builds on two broad desiderata for curriculum generation (Bengio et al., 2009) that enable sample-efﬁcient learning of goal-conditioned policies: progressive exploration and anti-catastrophic exploitation .

Original: Progressive exploration refers to curricula where the agent gradually learns to solve tasks of increasing difﬁculty and diversity.
Revised: Progressive exploration refers to curricula where the agent gradually learns to solve tasks of increasing difﬁculty and diversity.

Original: Anti-catastrophic exploitation ensures that the agent periodically resolves previously attempted goals that could become harder to solve during the course of training a.k.a.
Revised: Anti-catastrophic exploitation ensures that the agent periodically resolves previously attempted goals that could become harder to solve during the course of training a.k.a.

Original: catastrophic forgetting (French, 1999; Kirkpatrick et al., 2017).
Revised: catastrophic forgetting (French, 1999; Kirkpatrick et al., 2017).

Original: Many prior works have attempted to incorporate aspects of these criteria for automatic curriculum generation.
Revised: Many prior works have attempted to incorporate aspects of these criteria for automatic curriculum generation.

Original: One approach is to parameterize a goal generator and pit it against a discriminator that assesses goal difﬁculty (Florensa et al., 2018) or feasibility (Racaniere et al., 2020) for the current agent.
Revised: One approach is to parameterize a goal generator and pit it against a discriminator that assesses goal difﬁculty (Florensa et al., 2018) or feasibility (Racaniere et al., 2020) for the current agent.

Original: Racaniere et al.
Revised: Racaniere et al.

Original: (2020) also address desired qualities of validity and diversity by crafting speciﬁc losses for these characteristics.
Revised: (2020) also address desired qualities of validity and diversity by crafting speciﬁc losses for these characteristics.

Original: However, these approaches rely on learning an accurate discriminator for assessing goal difﬁculty, which can be challenging.
Revised: However, these approaches rely on learning an accurate discriminator for assessing goal difﬁculty, which can be challenging.

Original: Rather than employing discriminators in the standard adversarial minimax formulation, Dennis et al.
Revised: Rather than employing discriminators in the standard adversarial minimax formulation, Dennis et al.

Original: (2020) proposes an alternate minimax objective based on the agent’s regret.
Revised: (2020) proposes an alternate minimax objective based on the agent’s regret.

Original: The ‘expert’ used for computing the regret is another goal-conditioned policy whose utility is maximized by an environment generator.
Revised: The ‘expert’ used for computing the regret is another goal-conditioned policy whose utility is maximized by an environment generator.

Original:  An alternative method for automatic curriculum generation that has seen great success is self-play, where an agent plays against other versions of itself (Silver et al., 2016; Bansal et al., 2018;Baker [Figure]
Revised: [Figure] An alternative method for automatic curriculum generation that has seen great success is self-play, where an agent plays against other versions of itself (Silver et al., 2016; Bansal et al., 2018;

Original:  et al., 2019) (Fig. 1a).
Revised: Baker et al., 2019) (Fig. 1a).

Original: Unfortunately, this method does not apply directly to complex tasks where we do not have the structure of a two player, zero-sum game.
Revised: Unfortunately, this method does not apply directly to complex tasks where we do not have the structure of a two player, zero-sum game.

Original: A promising variation is Asymmetric
Revised: A promising variation is Asymmetric

Original: Self-Play (ASP) (Sukhbaatar et al., 2018b), which involves a minimax game between two similar agents: a goal demonstrator, Alice, and the desired goal-conditioned agent, Bob (Fig. 1b).
Revised: Self-Play (ASP) (Sukhbaatar et al., 2018b), which involves a minimax game between two similar agents: a goal demonstrator, Alice, and the desired goal-conditioned agent, Bob (Fig. 1b).

Original: Bob is asked to solve or reverse tasks demonstrated by Alice in the same environment.
Revised: Bob is asked to solve or reverse tasks demonstrated by Alice in the same environment.

Original: To motivate Alice to propose successively challenging goals, Alice is rewarded whenever Bob fails.
Revised: To motivate Alice to propose successively challenging goals, Alice is rewarded whenever Bob fails.

Original: This method inherently captures an easy-to-hard curriculum, since any goal solved by Alice is achievable by Bob in principle.
Revised: This method inherently captures an easy-to-hard curriculum, since any goal solved by Alice is achievable by Bob in principle.

Original: However, ASP tends to be sample inefﬁcient as it is bottlenecked by Alice’s ability to remember earlier goals and explore a diverse goal space.
Revised: However, ASP tends to be sample inefﬁcient as it is bottlenecked by Alice’s ability to remember earlier goals and explore a diverse goal space.

Original: Our work aims to bring together the strengths of goal generative methods and ASP while addressing the two desiderata.
Revised: Our work aims to bring together the strengths of goal generative methods and ASP while addressing the two desiderata.

Original: Our work focuses on the multi-goal setting with the curriculum generator controlling goals Portelas et al.
Revised: Our work focuses on the multi-goal setting with the curriculum generator controlling goals Portelas et al.

Original: We propose Curriculum Self Play (CuSP), a symmetric multiagent setup for automatic curricula generation.
Revised: We propose Curriculum Self Play (CuSP), a symmetric multiagent setup for automatic curricula generation.

Original: CuSP optimizes for two off-policy goal-conditioned students, Alice and Bob, and two regret-maximizing goal generators, G A and G B (Fig. 1c).
Revised: CuSP optimizes for two off-policy goal-conditioned students, Alice and Bob, and two regret-maximizing goal generators, G A and G B (Fig. 1c).

Original: Drawing inspiration from the minimax regret objective in PAIRED (Dennis et al., 2020), we designate each goal generator as being ‘friendly’ to one of the students ( G A for Alice, G B for Bob), and optimize each goal generator using the difference in utility between the preferred student and the other student (i.e., G A is rewarded R A − R B , where R A , R B are Alice’s and Bob’s returns respectively when challenged with the same goal).
Revised: Drawing inspiration from the minimax regret objective in PAIRED (Dennis et al., 2020), we designate each goal generator as being ‘friendly’ to one of the students ( G A for Alice, G B for Bob), and optimize each goal generator using the difference in utility between the preferred student and the other student (i.e., G A is rewarded R A − R B , where R A , R B are Alice’s and Bob’s returns respectively when challenged with the same goal).

Original: This encourages proposing goals that are neither too easy nor too hard.
Revised: This encourages proposing goals that are neither too easy nor too hard.

Original: With our symmetrized setup, Alice and Bob each have a ‘friendly’ goal generator that is inclined to propose more feasible goals without being too simple, and an ‘unfriendly’ goal generator that is inclined to propose more challenging goals.
Revised: With our symmetrized setup, Alice and Bob each have a ‘friendly’ goal generator that is inclined to propose more feasible goals without being too simple, and an ‘unfriendly’ goal generator that is inclined to propose more challenging goals.

Original: As the tasks increase in difﬁculty overall, the former’s cooperation helps in scaffolding learning while the latter’s competition helps for pushing the boundaries of the agent’s capabilities.
Revised: As the tasks increase in difﬁculty overall, the former’s cooperation helps in scaffolding learning while the latter’s competition helps push the boundaries of the agent’s capabilities.

Original: We further address goal diversity and sample efﬁciency by leveraging the off policy entropyregularized optimization of Soft Actor Critic (SAC) (Haarnoja et al., 2018) for training the goal generators.
Revised: We further address goal diversity and sample efﬁciency by leveraging the off policy entropyregularized optimization of Soft Actor Critic (SAC) (Haarnoja et al., 2018) for training the goal generators.

Original: The goal generators’ replay buffers track the regrets for all proposed goals.
Revised: The goal generators’ replay buffers track the regrets for all proposed goals.

Original: To account for non-stationary learners, we propose an update rule for the regrets in the goal generators’ replay buffers based on the critic networks of Alice and Bob.
Revised: To account for non-stationary learners, we propose an update rule for the regrets in the goal generators’ replay buffers based on the critic networks of Alice and Bob.

Original: In doing so, we constantly keep track of the learners’ current abilities and can propose increasingly challenging goals.
Revised: In doing so, we constantly keep track of the learners’ current abilities and can propose increasingly challenging goals.

Original: At the same time, using the regret objective also places the current abilities in context.
Revised: At the same time, using the regret objective also places the current abilities in context.

Original: For example, if Bob has forgotten how to achieve a particular goal, the regret R A − R B increases and G A is more inclined to re-propose that goal.
Revised: For example, if Bob has forgotten how to achieve a particular goal, the regret R A − R B increases and G A is more inclined to re-propose that goal.

Original: Thus the combination of using a replay buffer and the regret objective aids in addressing the twin goals of progressive exploration and anti-catastrophic exploitation.
Revised: Thus the combination of using a replay buffer and the regret objective aids in addressing the twin goals of progressive exploration and anti-catastrophic exploitation.

Original: Compared to prior methods for goal generation, we demonstrate that CuSP is able to satisfy the desiderata for good curricula.
Revised: Compared to prior methods for goal generation, we demonstrate that CuSP is able to satisfy the desiderata for good curricula.

Original: We show that this translates to improved sample efﬁciency and generalization to novel out-of-distribution goals across a range of robotics tasks, encompassing navigation, manipulation, and locomotion.
Revised: We show that this translates to improved sample efﬁciency and generalization to novel out-of-distribution goals across a range of robotics tasks, encompassing navigation, manipulation, and locomotion.

Original: We highlight some emergent skills that arise via CuSP when generalizing to particularly challenging goals, and ﬁnd that our method is able to handle cases when portions of the goal space are infeasible.
Revised: We highlight some emergent skills that arise via CuSP when generalizing to particularly challenging goals, and ﬁnd that our method is able to handle cases when portions of the goal space are infeasible.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Problem Setup.
Revised: Problem Setup.

Original: We model the environment and the goal-conditioned learner using a Markov decision process, M =
Revised: We model the environment and the goal-conditioned learner using a Markov decision process, M =

Original: (cid:104)S , A , T , R , G , γ (cid:105) , where S is the state space, A is the action space,
Revised: (cid:104)S , A , T , R , G , γ (cid:105) , where S is the state space, A is the action space,

Original: T : S ×
Revised: T : S ×

Original: A × S → R is the environment transition function, R : G × S
Revised: A × S → R is the environment transition function, R : G × S

Original: → R is the goal-conditioned reward function, G ⊆ S is the goal space, and γ is a discount factor.
Revised: → R is the goal-conditioned reward function, G ⊆ S is the goal space, and γ is a discount factor.

Original: We consider both dense and sparse reward functions: r ( s | g ) = − d ( s, g ) , where d is a distance metric (e.g. L 2 norm) between the state s and the goal g for the former, or r ( s | g )
Revised: We consider both dense and sparse reward functions: r ( s | g ) = − d ( s, g ) , where d is a distance metric (e.g. L 2 norm) between the state s and the goal g for the former, or r ( s | g )

Original: = 1 { d ( s, g ) <
Revised: = 1 { d ( s, g ) <

Original: (cid:15) } for the latter.
Revised: (cid:15) } for the latter.

Original: Our objective is to learn a policy π ( a t | s t , g ) that maximizes the success rate of reaching all goals g ∈ G .
Revised: Our objective is to learn a policy π ( a t | s t , g ) that maximizes the success rate of reaching all goals g ∈ G .

Original: To assess ability to generalize to out-of-distribution goals, we split the goal space into G id and G ood where the goal generators can only propose goals g ∈ G id , and we evaluate on sampled g ∈ G ood .
Revised: To assess ability to generalize to out-of-distribution goals, we split the goal space into G id and G ood where the goal generators can only propose goals g ∈ G id , and we evaluate on sampled g ∈ G ood .

Original: Self Play (SP).
Revised: Self Play (SP).

Original: SP is an approach for automatic curriculum generation in competitive, zero-sum games.
Revised: SP is an approach for automatic curriculum generation in competitive, zero-sum games.

Original: In this paradigm, an agent plays against past versions of itself in order to automatically generate a learning curriculum (Figure 1a).
Revised: In this paradigm, an agent plays against past versions of itself in order to automatically generate a learning curriculum (Figure 1a).

Original: That is, we consider Alice and Bob as two versions of the same underlying agent, and each agent’s objective is the negative of the others’, so R A = − R B .
Revised: That is, we consider Alice and Bob as two versions of the same underlying agent, and each agent’s objective is the negative of the others’, so R A = − R B .

Original: However, this approach is only applicable for symmetric zero-sum games, and does not lend itself to learning curricula for goal-conditioned agents in other setups.
Revised: However, this approach is only applicable for symmetric zero-sum games, and does not lend itself to learning curricula for goal-conditioned agents in other setups.

Original: Asymmetric Self Play (ASP).
Revised: Asymmetric Self Play (ASP).

Original: ASP (Sukhbaatar et al., 2018b) extends self play to non-zero sum games by proposing an asymmetric parametrization of two competing agents.
Revised: ASP (Sukhbaatar et al., 2018b) extends self play to non-zero sum games by proposing an asymmetric parametrization of two competing agents.

Original: During each episode,
Revised: During each episode,

Original: Alice’s objective is to propose and demonstrate a task for Bob to complete (Figure 1b).
Revised: Alice’s objective is to propose and demonstrate a task for Bob to complete (Figure 1b).

Original: In the goalconditioned paradigm, we take Alice’s ﬁnal state s AT as the proposed goal g .
Revised: In the goalconditioned paradigm, we take Alice’s ﬁnal state s AT as the proposed goal g .

Original: Alice and Bob now have different objectives: Bob’s return, R B = (cid:80) t γ t r ( s Bt | g ) , is simply based on reaching the proposed goal, while Alice’s reward is based on Bob’s inability to reach the goal, R A
Revised: Alice and Bob now have different objectives: Bob’s return, R B = (cid:80) t γ t r ( s Bt | g ) , is simply based on reaching the proposed goal, while Alice’s reward is based on Bob’s inability to reach the goal, R A

Original: = 1 ( d ( s BT , g ) ≥ (cid:15) ) .
Revised: = 1 ( d ( s BT , g ) ≥ (cid:15) ) .

Original: While this approach lends itself well to more complex tasks such as robotic manipulation (OpenAI et al., 2021), it has poor sample efﬁciency.
Revised: While this approach lends itself well to more complex tasks such as robotic manipulation (OpenAI et al., 2021), it has poor sample efﬁciency.

Original: Since we rely on a learning agent, Alice, to propose and solve goals, the generated goals are limited by Alice’s own ability to explore and learn challenging yet feasible goals.
Revised: Since we rely on a learning agent, Alice, to propose and solve goals, the generated goals are limited by Alice’s own ability to explore and learn challenging yet feasible goals.

Original: This can be particularly problematic towards the start of training, where we can waste many episodes proposing simple and redundant goals for Bob.
Revised: This can be particularly problematic towards the start of training, where we can waste many episodes proposing simple and redundant goals for Bob.

Original:  A UTOMATIC C URRICULUM G ENERATION VIA C U SP
Revised:  A UTOMATIC C URRICULUM G ENERATION VIA C U SP

Original: Our curriculum generation method consists of four main players: two goal-conditioned learning peers, Alice π A and Bob π B , and two goal generators, G A and G B .
Revised: Our curriculum generation method consists of four main players: two goal-conditioned learning peers, Alice π A and Bob π B , and two goal generators, G A and G B .

Original: π A ( a | s, g ) and π B ( a | s, g ) are both optimized to maximize the discounted sum of goal-conditioned rewards R ( g )
Revised: π A ( a | s, g ) and π B ( a | s, g ) are both optimized to maximize the discounted sum of goal-conditioned rewards R ( g )

Original: =
Revised: =

Original: (cid:80) t γ t r t ( g ) , across proposed goals g ∈ G id .
Revised: (cid:80) t γ t r t ( g ) , across proposed goals g ∈ G id .

Original: Aside from any initial randomness in initializing π A and π B , the two policies are parameterized with the same architecture.
Revised: Aside from any initial randomness in initializing π A and π B , the two policies are parameterized with the same architecture.

Original: G A ( g | s 0 , z ) and G B ( g | s 0 , z ) are parametrized as policies that output a goal g ∈ G id given an initial observation of the environment sand a latent noise variable z ∼ N (0 , 1) .
Revised: G A ( g | s 0 , z ) and G B ( g | s 0 , z ) are parametrized as policies that output a goal g ∈ G id given an initial observation of the environment sand a latent noise variable z ∼ N (0 , 1) .

Original: The inclusion of such a noise variable draws from generative models (e.g. GANs, VAEs) where such a z is the input to the generator, and is also found in prior goal generative methods (Florensa et al., 2018).
Revised: The inclusion of such a noise variable draws from generative models (e.g. GANs, VAEs) where such a z is the input to the generator, and is also found in prior goal generative methods (Florensa et al., 2018).

Original: The goal generators are optimized to maximize the regret of their corresponding agent:
Revised: The goal generators are optimized to maximize the regret of their corresponding agent:

Original: R G A = R A ( g ) − R B
Revised: R G A = R A ( g ) − R B

Original: ( g ) and R G B =
Revised: ( g ) and R G B =

Original: − R G A
Revised: − R G A

Original: =
Revised: =

Original: R B ( g ) − R A ( g ) .
Revised: R B ( g ) − R A ( g ) .

Original: We refer to our overall curriculum generation approach as Curriculum Self Play (CuSP).
Revised: We refer to our overall curriculum generation approach as Curriculum Self Play (CuSP).

Original: We present an overview of CuSP in Algorithm 1 and illustrate it in Figure 1c.
Revised: We present an overview of CuSP in Algorithm 1 and illustrate it in Figure 1c.

Original: The algorithm proceeds in rounds.
Revised: The algorithm proceeds in rounds.

Original: In every round, both goal generators propose a goal, g A from G A and g B from G B .
Revised: In every round, both goal generators propose a goal, g A from G A and g B from G B .

Original: To keep the symmetrization fair and hopefully scaffold learning initially, we ﬁrst rollout the ‘easier’ goal for each respective agent (ie.
Revised: To keep the symmetrization fair and hopefully scaffold learning initially, we ﬁrst rollout the ‘easier’ goal for each respective agent (ie.

Original: g A for Alice, g B for Bob) and compute the corresponding environment returns and update the learners.
Revised: g A for Alice, g B for Bob) and compute the corresponding environment returns and update the learners.

Original: Next, we rollout the ‘harder’ goal for each respective agent (ie.
Revised: Next, we rollout the ‘harder’ goal for each respective agent (ie.

Original: g B for Alice, g A for Bob) and update again.
Revised: g B for Alice, g A for Bob) and update again.

Original: At the end of the round, we update the goal generators.
Revised: At the end of the round, we update the goal generators.

Original: Symmetrization.
Revised: Symmetrization.

Original: The regret objective R A − R B helps G A propose goals that are challenging for
Revised: The regret objective R A − R B helps G A propose goals that are challenging for

Original: Bob while still being feasible for Alice.
Revised: Bob while still being feasible for Alice.

Original: However, this can be detrimental when Alice and Bob’s performance diverge too quickly.
Revised: However, this can be detrimental when Alice and Bob’s performance diverge too quickly.

Original: To address this, we propose symmetrizing the multi-agent system by introducing a corresponding goal generator G B whose regret objective is R B − R A , essentially pitting it in a zero-sum game against G A .
Revised: To address this, we propose symmetrizing the multi-agent system by introducing a corresponding goal generator G B whose regret objective is R B − R A , essentially pitting it in a zero-sum game against G A .

Original: Having these two goal generators produces two types of goals for each agent: a) a feasible yet not too simple goal from the ‘friendly’ goal generator ( G A for Alice, G B for Bob), which can help scaffold learning quickly, and b) a challenging yet not too difﬁcult goal from the ‘unfriendly’ goal generator ( G A for Bob, G B for Alice), which can help push
Revised: Having these two goal generators produces two types of goals for each agent: a) a feasible yet not too simple goal from the ‘friendly’ goal generator ( G A for Alice, G B for Bob), which can help scaffold learning quickly, and b) a challenging yet not too difﬁcult goal from the ‘unfriendly’ goal generator ( G A for Bob, G B for Alice), which can help push

Original: Initialize goal-conditioned learners π A , π B , goal generators G A , G B ; while not converged do // generate goals g a ∼ G A , g b ∼ G B ; // evaluate easier goals first
Revised: Initialize goal-conditioned learners π A , π B , goal generators G A , G B ; while not converged do // generate goals g a ∼ G A , g b ∼ G B ; // evaluate easier goals first

Original: R AEasy ← rollout ( π A , g A ) ; R BEasy ← rollout ( π B , g B ) ; // update policies π A ← train learner ( π A , R AEasy ) ; π B ← train learner ( π B , R BEasy ) ; // evaluate harder goals second
Revised: R AEasy ← rollout ( π A , g A ) ; R BEasy ← rollout ( π B , g B ) ; // update policies π A ← train learner ( π A , R AEasy ) ; π B ← train learner ( π B , R BEasy ) ; // evaluate harder goals second

Original: R AHard ← rollout ( π A , g B ) ; R BHard ← rollout ( π B , g A ) ; // update policies π A ← train learner ( π A , R AHard ) ; π B ← train learner ( π B , R BHard ) ; // compute regrets and update generators
Revised: R AHard ← rollout ( π A , g B ) ; R BHard ← rollout ( π B , g A ) ; // update policies π A ← train learner ( π A , R AHard ) ; π B ← train learner ( π B , R BHard ) ; // compute regrets and update generators

Original: G A ← train generator ( G A , { ( R AEasy − R BHard ) , g A } , { ( R AHard − R BEasy ) , g B } ) ;
Revised: G A ← train generator ( G A , { ( R AEasy − R BHard ) , g A } , { ( R AHard − R BEasy ) , g B } ) ;

Original: G B ← train generator ( G B , { ( R BEasy − R AHard ) , g B } , { ( R BHard − R AEasy ) , g A } ) ; end
Revised: G B ← train generator ( G B , { ( R BEasy − R AHard ) , g B } , { ( R BHard − R AEasy ) , g A } ) ; end

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: the boundaries of the agent’s abilities.
Revised: the boundaries of the agent’s abilities.

Original: Both of these properties help encourage the generation of goals that are feasible yet progressively challenging.
Revised: Both of these properties help encourage the generation of goals that are feasible yet progressively challenging.

Original: Goal Generator Design.
Revised: Goal Generator Design.

Original: We motivate our goal generator design with our proposed desiderata: progressive exploration with anti-catastrophic exploitation.
Revised: We motivate our goal generator design with our proposed desiderata: progressive exploration with anti-catastrophic exploitation.

Original: Although the goal generator is not an
Revised: Although the goal generator is not an

Original: RL agent in the traditional sense – it only ‘acts’ in a single timestep by proposing a goal and its ‘observation’ only changes based on a random noise variable – optimizing such an explore-exploit tradeoff effectively naturally inspires the use of an RL objective, and RL objectives have previously been used for one-step optimization (e.g., one-step Q-learning (Watkins & Dayan, 1992)).
Revised: RL agent in the traditional sense – it only ‘acts’ in a single timestep by proposing a goal and its ‘observation’ only changes based on a random noise variable – optimizing such an explore-exploit tradeoff effectively naturally inspires the use of an RL objective, and RL objectives have previously been used for one-step optimization (e.g., one-step Q-learning (Watkins & Dayan, 1992)).

Original: Thus, to train the goal generators we use an RL learning objective and parameterization inspired by a single-time step variant of Soft-Actor Critic (SAC) (Haarnoja et al., 2018).
Revised: Thus, to train the goal generators we use an RL learning objective and parameterization inspired by a single-time step variant of Soft-Actor Critic (SAC) (Haarnoja et al., 2018).

Original: SAC is an off policy algorithm that optimizes for a trade-off between expected reward and policy entropy H ( π ) with a coefﬁcient α .
Revised: SAC is an off policy algorithm that optimizes for a trade-off between expected reward and policy entropy H ( π ) with a coefﬁcient α .

Original: In our setup, our goal generator only acts once to propose a goal per episode, so each ‘trajectory’ is only a single time step consisting of the initial state s (for which we concatenate the initial environment observation s 0 with a latent noise variable z ), the proposed goal action g , and we get the corresponding “reward” for the goal generator policy by computing the regret between π A and π B on g .
Revised: In our setup, our goal generator only acts once to propose a goal per episode, so each ‘trajectory’ is only a single time step consisting of the initial state s (for which we concatenate the initial environment observation s 0 with a latent noise variable z ), the proposed goal action g , and we get the corresponding “reward” for the goal generator policy by computing the regret between π A and π B on g .

Original: Thus the objective for each goal generator policy G is given by
Revised: Thus the objective for each goal generator policy G is given by

Original: [Equation]
Revised: [Equation]

Original:  where the regret term addresses anti-catastrophic exploitation by prioritizing goals where regret has increased, and the entropy term addresses progressive goal space exploration.
Revised:   where the regret term addresses anti-catastrophic exploitation by prioritizing goals where regret has increased, and the entropy term addresses progressive goal space exploration.

Original: We equip the SAC goal generators with a replay buffer to store all past goals and their regrets.
Revised: We equip the SAC goal generators with a replay buffer to store all past goals and their regrets.

Original: This helps address forgetting and improve the sample efﬁciency of goal generation compared to prior methods which optimize the generator as another agent using on-policy algorithms such as PPO (Schulman et al., 2017).
Revised: This helps address forgetting and improve the sample efﬁciency of goal generation compared to prior methods which optimize the generator as another agent using on-policy algorithms such as PPO (Schulman et al., 2017).

Original: We further take advantage of SAC’s entropy regularization to progressively increase the diversity of the goals, with consideration of Alice and Bob’s capabilities using the stored regrets.
Revised: We further take advantage of SAC’s entropy regularization to progressively increase the diversity of the goals, with consideration of Alice and Bob’s capabilities using the stored regrets.

Original: To empirically motivate the use of SAC, we compare SAC against PPO and Adam (Kingma & Ba, 2015) in optimizing a synthetic landscape.
Revised: To empirically motivate the use of SAC, we compare SAC against PPO and Adam (Kingma & Ba, 2015) in optimizing a synthetic landscape.

Original: One challenge of the regret objective is that at the beginning of training, the landscape will mostly be ﬂat when π A and π B are similarly unskilled.
Revised: One challenge of the regret objective is that at the beginning of training, the landscape will mostly be ﬂat when π A and π B are similarly unskilled.

Original: In Fig.
Revised: In Fig.

Original: 2, we ﬁnd that only SAC is proﬁcient at optimizing a primarily ﬂat landscape.
Revised: 2, we ﬁnd that only SAC is proﬁcient at optimizing a primarily ﬂat landscape.

Original: Dynamic Regret Updates for Non-stationary Agents.
Revised: Dynamic Regret Updates for Non-stationary Agents.

Original: However, using an off policy RL algorithm for optimizing such a goal generator requires further modiﬁcations.
Revised: However, using an off policy RL algorithm for optimizing such a goal generator requires further modiﬁcations.

Original: For more typical agents, the received rewards are a stationary property of the environment.
Revised: For more typical agents, the received rewards are a stationary property of the environment.

Original: In our case this is no longer true – Alice and Bob are continuously learning and part of the multi-agent environment, so the regrets for the same goal will also change over time.
Revised: In our case this is no longer true – Alice and Bob are continuously learning and part of the multi-agent environment, so the regrets for the same goal will also change over time.

Original: To address this issue, we would like to update the non-stationary regret values in the goal generator’s replay buffer.
Revised: To address this issue, we would like to update the non-stationary regret values in the goal generator’s replay buffer.

Original: However, it is expensive for Alice and Bob to replay all the goals in the environment.
Revised: However, it is expensive for Alice and Bob to replay all the goals in the environment.

Original: As a sample efﬁcient trick, we leverage the observation that the critics of Alice and Bob can provide a current estimate of their performance.
Revised: As a sample efﬁcient trick, we leverage the observation that the critics of Alice and Bob can provide a current estimate of their performance.

Original: Hence, after every round, we update the regrets of previous goals in the replay buffer using the difference between the corresponding value estimates from the critic networks of Alice and Bob.
Revised: Hence, after every round, we update the regrets of previous goals in the replay buffer using the difference between the corresponding value estimates from the critic networks of Alice and Bob.

Original: This allows us to keep an updated estimate of feasibility based on Alice and Bob’s current skills.
Revised: This allows us to keep an updated estimate of feasibility based on Alice and Bob’s current skills.

Original: To empirically validate that updating the replay buffer regrets helps in a non-stationary landscape, we carry out investigations of a SAC goal generator both with and without regret updates in Fig.
Revised: To empirically validate that updating the replay buffer regrets helps in a non-stationary landscape, we carry out investigations of a SAC goal generator both with and without regret updates in Fig.

Original: As hypothesized, updating the replay buffer allows the generator to ﬁnd the moving optimum.
Revised: As hypothesized, updating the replay buffer allows the generator to ﬁnd the moving optimum.

Original: The combination of the replay buffer and dynamic regret updates also strengthens our method by automatically prioritizing goals that have been forgotten.
Revised: The combination of the replay buffer and dynamic regret updates also strengthens our method by automatically prioritizing goals that have been forgotten.

Original: With the non-stationary regret updates, the goal generators always have an updated estimate of each learner’s abilities.
Revised: With the non-stationary regret updates, the goal generators always have an updated estimate of each learner’s abilities.

Original: Thus if Bob performs worse on a past goal, the regret increases for G A and it is more likely to repropose it, addressing catastrophic forgetting for Bob.
Revised: Thus if Bob performs worse on a past goal, the regret increases for G A and it is more likely to repropose it, addressing catastrophic forgetting for Bob.

Original: Likewise, symmetrization helps to address catastrophic forgetting for Alice in the same manner.
Revised: Likewise, symmetrization helps to address catastrophic forgetting for Alice.

Original: Without the corresponding G B , if Alice performs worse at a goal, the regret R A − R B decreases and G A is less likely to repropose it unless Bob’s performance decreases even further.
Revised: Without the corresponding G B , if Alice performs worse at a goal, the regret R A − R B decreases and G A is less likely to repropose it unless Bob’s performance decreases even further.

Original: This can potentially lead to neglecting to propose these challenging goals for Bob.
Revised: This can potentially lead to neglecting to propose these challenging goals for Bob.

Original: Summary.
Revised: Summary.

Original: Inspired by Racaniere et al.
Revised: Inspired by Racaniere et al.

Original: (2020), we specify desired qualities of a goal generator:
Revised: (2020), we specify desired qualities of a goal generator:

Original: • Progressive Feasibility – the goals should be incrementally more difﬁcult, accounting for the agent’s current skill level and the fact that the agent is a non-stationary learner.
Revised: • Progressive Feasibility – the goals should be incrementally more difﬁcult, accounting for the agent’s current skill level and the fact that the agent is a non-stationary learner.

Original: • Progressive Diversity – the goals should encourage increasing exploration of goals.
Revised: • Progressive Diversity – the goals should encourage increasing exploration of goals.

Original: • Anti-Forgetting – if previous goals are forgotten, they should be reproposed.
Revised: • Anti-Forgetting – if previous goals are forgotten, they should be reproposed.

Original: We summarize the contributions of each component of our method to desired qualities of feasibility, diversity, and anti-forgetting in Table 1.
Revised: We summarize the contributions of each component of our method to desired qualities of feasibility, diversity, and anti-forgetting in Table 1.

Original: We visualize generated goal distributions in Appendix A and ablate over each component of Table 1 in Appendix B.5.1 to illustrate their importance.
Revised: We visualize generated goal distributions in Appendix A and ablate over each component of Table 1 in Appendix B.5.1 to illustrate their importance.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Environments.
Revised: Environments.

Original: We test our method across a suite of continuous control tasks adapted from the
Revised: We test our method across a suite of continuous control tasks adapted from the

Original: Deepmind Control Suite (Tassa et al., 2020) and OpenAI Gym (Plappert et al., 2018), using the[Figure] [Table] [Figure]
Revised: Deepmind Control Suite (Tassa et al., 2020) and OpenAI Gym (Plappert et al., 2018), using the

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: MuJoCo simulator (Todorov et al., 2012).
Revised: MuJoCo simulator (Todorov et al., 2012).

Original: We analyze our method across a variety of task types: navigation , locomotion , and manipulation .
Revised: We analyze our method across a variety of task types: navigation , locomotion , and manipulation .

Original: Point Mass Obstacle is a navigation task with some walls.
Revised: Point Mass Obstacle is a navigation task with some walls.

Original: Walker is a locomotion task with a goal torso position.
Revised: Walker is a locomotion task with a goal torso position.

Original: Reach and Toss are manipulation tasks with a goal ball position.
Revised: Reach and Toss are manipulation tasks with a goal ball position.

Original: Toss provides an interesting challenge as gravity plays a large role – the robot must learn to aim precisely.
Revised: Toss provides an interesting challenge as gravity plays a large role – the robot must learn to aim precisely.

Original: Pick and Reach (OpenAI, 2020) is a manipulation task where success is measured by picking up a block and moving it to a target position.
Revised: Pick and Reach (OpenAI, 2020) is a manipulation task where success is measured by picking up a block and moving it to a target position.

Original: Environments and goal spaces are visualized in Figure 4, with details in Appendix B.2.
Revised: Environments and goal spaces are visualized in Figure 4, with details in Appendix B.2.

Original: Training Setup.
Revised: Training Setup.

Original: For CuSP, we train the goal generators using the SAC implementation from Yarats & Kostrikov (2020) with their default architecture and hyperparameters.
Revised: For CuSP, we train the goal generators using the SAC implementation from Yarats & Kostrikov (2020) with their default architecture and hyperparameters.

Original: Since symmetrization leads to similar performance from Alice and Bob in expectation, we report the results from Bob only for consistency.
Revised: Since symmetrization leads to similar performance from Alice and Bob in expectation, we report the results from Bob only for consistency.

Original: We compare against a uniform random generator baseline and two other baselines that use minimax formulations for curriculum generation: the GoalGANgenerator-discriminator approach (Florensa et al., 2018) and ASP+BC (OpenAI et al., 2021).
Revised: We compare against a domain randomization Tobin et al. (2017) baseline and two other baselines that use minimax formulations for curriculum generation: the GoalGAN approach (Florensa et al., 2018) and ASP+BC (OpenAI et al., 2021).

Original: For all methods we train the learner policies with the same architecture and SAC to isolate differences to the goal generation method.
Revised: We train all learner policies with the same architecture and SAC to isolate differences to the goal generation method.

Original: Code and further training details can be found in the Supplementary Materials.

Original: We aim to answer the following questions with our experiments:
Revised: We aim to answer the following questions with our experiments:

Original: • Does the CuSP curricula induce agents that can generalize to novel goals from G ood ?
Revised: • Does the CuSP curricula induce agents that can generalize to novel goals from G ood ?

Original: • Does the CuSP curricula induce interesting emergent skills for solving harder goals?
Revised: • Does the CuSP curricula induce interesting emergent skills for solving harder goals?

Original: • Does the CuSP curricula handle cases where portions of the goal space are poorly speciﬁed (e
Revised: • Does the CuSP curricula handle cases where portions of the goal space are poorly speciﬁed (e

Original: g. with impossible dimensions)?
Revised: g. with impossible dimensions)?

Original: 4. T EST -T IME O UT -O F -D ISTRIBUTION G ENERALIZATION
Revised: 4. T EST -T IME O UT -O F -D ISTRIBUTION G ENERALIZATION

Original: To assess whether our curriculum improves generalizability, we evaluate each method on its ability to reach randomly sampled out-of-distribution goals g ∼ G ood .
Revised: To assess whether our curriculum improves generalizability, we evaluate each method on its ability to reach randomly sampled out-of-distribution goals g ∼ G ood .

Original: In Fig.
Revised: In Fig.

Original: 5 we see that across environments, generally CuSP matches or outperforms all the baselines for task success.
Revised: 5 we see that across environments, generally CuSP matches or outperforms all the baselines for task success.

Original: ASP+BC’s sample inefﬁciency due to Alice’s limitations leads to poor performance, while GoalGAN does well in environments where the goal space is more symmetrical (e.g. with navigation or the reaching tasks, where most points in the goal space are similarly difﬁcult).
Revised: ASP+BC’s sample inefﬁciency due to Alice’s limitations leads to poor performance, while GoalGAN does well in environments where the goal space is more symmetrical (e.g. with navigation or the reaching tasks, where most points in the goal space are similarly difﬁcult).

Original: That said, we note GoalGAN struggles more in environments where portions of the goal space are more challenging (e.g. walking to the right or aiming and tossing upwards).
Revised: That said, we note GoalGAN struggles more in environments where portions of the goal space are more challenging (e.g. walking to the right or aiming and tossing upwards).

Original: We hypothesize this is due to the challenges of accurately learning the feasibility of different goals in an asymmetric goal space.
Revised: We hypothesize this is due to the challenges of accurately learning the feasibility of different goals in an asymmetric goal space.

Original: Lastly, we ﬁnd that the uni-
Revised: Lastly, we ﬁnd [Figure] ss

Original: [Figure]
Revised: [Figure]

Revised: Behind Obstacles

Original: [Figure]
Revised: [Figure]

Original: S u cce ss R a t e
Revised: cce ss R a t e

Original: [Figure]
Revised: [Figure]

Original: form random baseline is particularly well suited for this OOD random evaluation.
Revised: that the domain randomization baseline is particularly well suited for this OOD random evaluation.

Original: The baseline is trained on good coverage of the in-distribution goal space, which is beneﬁcial for generalization to random goals in the navigation and manipulation tasks, but is less helpful for the more asymmetric goal spaces like Walker and Toss.
Revised: The baseline is trained on good coverage of the in-distribution goal space, which is beneﬁcial for generalization to random goals in the navigation and manipulation tasks, but is less helpful for the more asymmetric goal spaces like Walker and Toss.

Original: For ablations on ASP, see Appendix B.5.2.
Revised: For ablations on ASP, see Appendix B.5.2.

Original: We also ablate over each contribution of our method as detailed in Table 1 in Appendix B.5.1.
Revised: We also ablate over each contribution of our method as detailed in Table 1 in Appendix B.5.1.

Original: Speciﬁcally, we ablate the regret replay buffer, the dynamic regret updates, entropy regularization, and symmetrization.
Revised: Specifically, we ablate the regret replay buffer, the dynamic regret updates, entropy regularization, and symmetrization.

Original: We ﬁnd that only using the regret objective from PAIRED (Dennis et al., 2020) is insufﬁcient, even with naively applying symmetrization, and that both entropy regularization and an updated replay buffer are crucial for high evaluation task success.
Revised: We ﬁnd that only using the regret objective from PAIRED (Dennis et al., 2020) is insufﬁcient, even with naively applying symmetrization, and that both entropy regularization and an updated replay buffer are crucial for high evaluation task success.

Original: 4. T ASK -S PECIFIC S KILLS
Revised: 4. T ASK -S PECIFIC S KILLS

Original: Next, we examine if CuSP can induce emergent skills in learners for tackling harder, environment speciﬁc goals.
Revised: Next, we examine if CuSP can induce emergent skills in learners for tackling harder, environment speciﬁc goals.

Original: Below, we enlist 3 representative examples in the more asymmetric environments where certain goals are clearly more challenging to solve.
Revised: Below, we enlist 3 representative examples in the more asymmetric environments where certain goals are clearly more challenging to solve.

Original: As seen in Fig.
Revised: As seen in Fig.

Original: 6, our method is able to improve upon success rates on these goals
Revised: our method is able to improve upon success rates on these goals.

Original: • Point Mass Obstacle – Behind Obstacle s: requires learning to move around the walls to the edge of the training space.
Revised: • Point Mass Obstacle – Behind Obstacle s: requires learning to move around the walls to the edge of the training space.

Original: Here, we learn to succeed much faster than the other methods.
Revised: Here, we learn to succeed much faster than the other methods.

Original: • Walker – Far Right : requires learning to move forward quickly to reach a goal far out of the training space.
Revised: • Walker – Far Right : requires learning to move forward quickly to reach a goal far out of the training space.

Original: Here, we learn to succeed at the task where all other baselines fail, suggesting that our method produces a useful curriculum for learning challenging locomotion tasks while the others cannot, or require many more samples to do so.
Revised: Here, we learn to succeed at the task where all other baselines fail, suggesting that our method produces a useful curriculum for learning challenging locomotion tasks while the others cannot, or require many more samples to do so.

Original: • Toss – Upwards : requires learning to toss the ball accurately upwards outside of the training space.
Revised: • Toss – Upwards : requires learning to toss the ball accurately upwards outside of the training space.

Original: Here, we learn to succeed much faster than the other methods.
Revised: Here, we learn to succeed much faster than the other methods.

Original: [Figure]
Revised: [Figure]

Original: 4. M ISSPECIFIED G OAL D IMENSIONS
Revised: 4. M ISSPECIFIED G OAL D IMENSIONS

Original: Lastly, we investigate if the goal generator methods (CuSP, GoalGAN, and Uniform) can handle the case where parts of the speciﬁed goal space is misspeciﬁed.
Revised: Lastly, we investigate if the goal generator methods (CuSP, GoalGAN, and DR) can handle the case where parts of the speciﬁed goal space is misspeciﬁed.

Original: To do this, we append an extra dimension to the goal space, bounded between [ − 1 , 1] , and evaluate performance on G id and G ood with randomly sampling the third dimension during evaluation.
Revised: To do this, we append an extra dimension to the goal space, bounded between [ − 1 , 1] , and evaluate performance on G id and G ood with randomly sampling the third dimension during evaluation.

Original: We use Reach as a case study, and as seen in Fig.
Revised: We use Reach as a case study, and as seen in Fig.

Original: 7, agents trained using CuSP’s goal curricula are less affected by the increased goal space while the baselines plateau at a lower success rate.
Revised: 7, agents trained using CuSP’s goal curricula are less affected by the increased goal space while the baselines plateau at a lower success rate.

Original: In particular, the Uniform baseline is most adversely affected by the higher dimensionality.
Revised: In particular, the DR baseline is most adversely affected by the higher dimensionality.

Original: This suggests that while random sampling provides good coverage and is amenable to more simple goal spaces, as seen above, the induced curricula degrades in quality with increasingly complex goal spaces.
Revised: This suggests that while random sampling provides good coverage and is amenable to more simple goal spaces, as seen above, the induced curricula degrades in quality with increasingly complex goal spaces.

Original: Note that we exclude ASP from these experiments as it does not use a pre-speciﬁed goal space and only sets achieved states as goals.
Revised: Note that we exclude ASP from these experiments as it does not use a pre-speciﬁed goal space and only sets achieved states as goals.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We summarize four main thrusts of work within automatic curriculum generation.
Revised: We summarize four main thrusts of work within automatic curriculum generation.

Original: Modeling Goal Distributions.
Revised: Modeling Goal Distributions.

Original: Many previous works have tackled the problem of generating “appropriately” difﬁcult goals for an autocurriculum.
Revised: Many previous works have tackled the problem of generating “appropriately” difﬁcult goals for an autocurriculum.

Original: In discrete environments, Campero et al.
Revised: In discrete environments, Campero et al.

Original: (2021) propose an adversarial goal-generator that is rewarded when the learning agent achieves the goal with a thresholded level of effort.
Revised: (2021) propose an adversarial goal-generator that is rewarded when the learning agent achieves the goal with a thresholded level of effort.

Original: However, they rely on a heuristic time limit hyperparameter to determine ‘suitable effort’ (i.e. goal feasibility) and hyperparameters for the teacher reward, whereas our approach directly uses regrets from multiagent interactions to automatically gauge feasibility and provide a reward signal to the teacher.
Revised: However, they rely on a heuristic time limit hyperparameter to determine ‘suitable effort’ (i.e. goal feasibility) and hyperparameters for the teacher reward, whereas our approach directly uses regrets from multiagent interactions to automatically gauge feasibility and provide a reward signal to the teacher.

Original: In continuous domains, Zhang et al.
Revised: In continuous domains, Zhang et al.

Original: samples goals that maximize the uncertainty of the learner’s Q-function, and Nair et al.
Revised: samples goals that maximize the uncertainty of the learner’s Q-function, and Nair et al.

Original: (2018) learn a latent representation to sample ‘imagined’ goals from.
Revised: (2018) learn a latent representation to sample ‘imagined’ goals from.

Original: GoalGAN (Florensa et al., 2018) trains a GAN to generate goals of intermediate difﬁculty for the agent, using a discriminator to predict whether the generated goals would be appropriate by thresholding success rates.
Revised: GoalGAN (Florensa et al., 2018) trains a GAN to generate goals of intermediate difﬁculty for the agent, using a discriminator to predict whether the generated goals would be appropriate by thresholding success rates.

Original: These methods use a heuristic measure of challenging goals, while we leverage the multi-agent setting to automatically ﬁnd interesting goals.
Revised: These methods use a heuristic measure of challenging goals, while we leverage the multi-agent setting to automatically ﬁnd interesting goals.

Original: Similarly to GoalGAN, the Setter-Solver paradigm (Racaniere et al., 2020) trains a goal setter whose objectives are a balance of goal validity, feasibility, and coverage, as well as a discriminative judge for predicting goal feasibility.
Revised: Similarly to GoalGAN, the Setter-Solver paradigm (Racaniere et al., 2020) trains a goal setter whose objectives are a balance of goal validity, feasibility, and coverage, as well as a discriminator for predicting goal feasibility.

Original: Our approach differs in that we do not need to explicitly weigh the contributions of each loss, and we do not need an accurate discriminator for estimating feasibility.
Revised: In contrast, our approach does not need to explicitly weigh the contributions of each loss or incorporate discriminators.

Original: Multi-Agent Curriculum Learning.
Revised: Multi-Agent Curriculum Learning.

Original: The idea that interactions between multiple agents can lead to emergent curricula has been proposed in prior works (Leibo et al., 2019; Baker et al., 2019) and can be connected with how humans learn through social interactions.
Revised: The idea that interactions between multiple agents can lead to emergent curricula has been proposed in prior works (Leibo et al., 2019; Baker et al., 2019) and relates to the theory of mind (Rabinowitz et al., 2018; Grover et al., 2018a;b) postulating human learning via social interactions.

Original: One parametrization of such interactions is through designating another agent(s) as a teacher who aims to select an appropriate curriculum for the student.
Revised: One parametrization of such interactions is through designating another agent(s) as a teacher who aims to select an appropriate curriculum for the student.

Original: For task-based curricula, Matiisen et al.
Revised: For taskbased curricula, Matiisen et al.

Original: (2019) train a teacher policy for selecting sub-tasks for the student, and Graves et al.
Revised: (2019) train a teacher policy for selecting sub-tasks for the student, and Graves et al.

Original: ; Portelas et al.
Revised: ; Portelas et al.

Original: (2019) treat task selection as a bandit problem for the teacher.
Revised: (2019) treat task selection as a bandit problem for the teacher.

Original: Rather than having an explicit teacher, Asymmetric Self-Play (ASP) (Sukhbaatar et al., 2018b) automatically generates a curriculum for exploration by training a similarly parametrized agent, Alice, to demonstrate the easiest possible task that the learning agent, Bob, can not yet complete.
Revised: Rather than having an explicit teacher, Asymmetric Self-Play (ASP) (Sukhbaatar et al., 2018b) automatically generates a curriculum for exploration by training a similarly parametrized agent, Alice, to demonstrate tasks that the learning agent, Bob, struggles to complete.

Original: OpenAI et al.
Revised: OpenAI et al.

Original: (2021) extend ASP to robotic manipulation tasks by incorporating an imitation learning loss for Bob to imitate Alice on challenging goals.
Revised: (2021) extend ASP to robotic manipulation tasks by incorporating an imitation learning loss for Bob to imitate Alice on challenging goals.

Original: As Alice is no longer setting the goals, we do not make use of behaviour cloning in CuSP.
Revised: As Alice is no longer setting the goals, we do not make use of behaviour cloning in CuSP.

Original: While they demonstrate zero-shot generalization to a diverse set of target tasks, relying on Alice to set the goals can also be incredibly sample inefﬁcient, as we saw in our evaluations.
Revised: While they demonstrate zero-shot generalization to a diverse set of target tasks, relying on Alice to set the goals can also be incredibly sample inefﬁcient, as we saw in our evaluations.

Original: Sodhani & Pahuja (2018) propose a memory-augmented version of ASP by providing the learners an external memory to speed up exploration, which is analogous to our method training with off-policy learners.
Revised: Sodhani & Pahuja (2018) propose a memory-augmented version of ASP by providing the learners an external memory to speed up exploration, which is analogous to our method training with off-policy learners.

Original: Sukhbaatar et al.
Revised: Sukhbaatar et al.

Original: (2018a) extends ASP to a hierarchical learning framework by learning subgoal representations.
Revised: (2018a) extends ASP to a hierarchical learning framework by learning subgoal representations.

Original: Competitive Experience Replay (Liu et al., 2018) generates an exploratory competitive curriculum through penalizing Alice if visiting states Bob has visited, but rewards Bob for visiting states found by Alice.
Revised: Competitive Experience Replay (Liu et al., 2018) generates an exploratory competitive curriculum through penalizing Alice if visiting states Bob has visited, but rewards Bob for visiting states found by Alice.

Original: Environment Design.
Revised: Environment Design.

Original: Another method for inducing curricula is through generating a sequence of environments to learn in.
Revised: Another method for inducing curricula is through generating a sequence of environments to learn in.

Original: Procedural level generation is a common approach for developing games, and can lead to increased generalizability (Justesen et al., 2018).
Revised: Procedural level generation is a common approach for developing games, and can lead to increased generalizability (Justesen et al., 2018).

Original: POET (Wang et al., 2019) uses a population of adversaries to generate increasingly challenging environments for a population of learning agents.
Revised: POET (Wang et al., 2019) uses a population of adversaries to generate increasingly challenging environments for a population of learning agents.

Original: PAIRED (Dennis et al., 2020) proposes optimizing an adversarial environment designer through minimax regret between two learning agents, inspiring the objective we use for our goal generators.
Revised: PAIRED (Dennis et al., 2020) proposes optimizing an adversarial environment designer through minimax regret between two learning agents, inspiring the objective we use for our goal generators.

Original: However, this approach does not account for catastrophic forgetting and can suffer from sample inefﬁciencies from using an on-policy method and in cases where the regret landscape is ﬂat.
Revised: However, this approach does not account for catastrophic forgetting and can suffer from sample inefﬁciencies from using an on-policy method and in cases where the regret landscape is ﬂat.

Original: That said, environment design approaches can be used in a complementary way with our approach, extending it beyond goal generation.
Revised: That said, environment design approaches can be used in a complementary way with our approach, extending it beyond goal generation.

Original: Skill Discovery.
Revised: Skill Discovery.

Original: A closely related method for curriculum generation is through unsupervised skill discovery, where an agent explores an environment to automatically discover skills that can be composed to achieve a variety of tasks.
Revised: A closely related method for curriculum generation is through unsupervised skill discovery, where an agent explores an environment to automatically discover skills that can be composed to achieve a variety of tasks.

Original: Prior methods have used information theoretic objectives to encourage skill acquisition that covers a diverse set of behaviours (Eysenbach et al., 2018) or generate a multi-task distribution (Gupta et al., 2020).
Revised: Prior methods have used information theoretic objectives to encourage skill acquisition that covers a diverse set of behaviours (Eysenbach et al., 2018) or generate a multi-task distribution (Gupta et al., 2020).

Original: For robotics tasks, (Hausman et al., 2018) propose a method for learning a latent skill space that can be composed and is transferable between tasks,
Revised: For robotics tasks, (Hausman et al., 2018) propose a method for learning a latent skill space that can be composed and is transferable between tasks,

Original: DADS (Sharma et al., 2019) proposes a combination of model-based and model-free RL that optimizes for discovering predictable skills that can be composed by a planner, and Play-LMP (Lynch et al., 2020) proposes using self-supervision across play data for skills discovery.
Revised: DADS (Sharma et al., 2019) proposes a combination of model-based and model-free RL that optimizes for discovering predictable skills that can be composed by a planner, and Play-LMP (Lynch et al., 2020) proposes using self-supervision across play data for skills discovery.

Original: Another method for curriculum generation is through using hindsight (Andrychowicz et al., 2017; Li et al., 2020).
Revised: Another method for curriculum generation is through using hindsight (Andrychowicz et al., 2017; Li et al., 2020).

Original: These works are complementary to ours as our method aims to induce a curriculum through sparse goal generation, which can be augmented with unsupervised skill discovery.
Revised: These works are complementary to ours as our method aims to induce a curriculum through sparse goal generation, which can be augmented with unsupervised skill discovery.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose a new method, CuSP, for automatic curriculum generation based on symmetric extension of self play approaches for non zero-sum games.
Revised: We propose a new method, CuSP, for automatic curriculum generation based on symmetric extension of self play approaches for non zero-sum games.

Original: Our proposed method aims to address weaknesses of prior approaches by being designed to speciﬁcally consider a balance of progressive exploration and anti-catastrophic exploitation.
Revised: Our proposed method aims to address weaknesses of prior approaches by being designed to speciﬁcally consider a balance of progressive exploration and anti-catastrophic exploitation.

Original: To this end, we make use of two off-policy students and two off-policy regret-maximizing goal generators, each of which is more ‘friendly’ to one of the students.
Revised: To this end, we make use of two off-policy students and two off-policy regret-maximizing goal generators, each of which is more ‘friendly’ to one of the students.

Original: As a result, we generate goals that are a) challenging yet feasible, b) diverse, and c) are anti-forgetting.
Revised: As a result, we generate goals that are a) challenging yet feasible, b) diverse, and c) revisited to mitigate anti-forgetting.

Original: We qualitatively highlight these desired qualities in goal generation compared to prior methods, demonstrate quantitative gains at generalizing to out-of-distribution unseen goals, and ﬁnd some emergent task-speciﬁc skills.
Revised: We qualitatively highlight these desired qualities in goal generation compared to prior methods, demonstrate quantitative gains at generalizing to out-of-distribution unseen goals, and ﬁnd some emergent task-speciﬁc skills.

Original: Limitations and Future Work.
Revised: Limitations and Future Work.

Original: While our method shows promise for addressing key components of generating useful curriculum, there is still signiﬁcant room for improvement.
Revised: While our method shows promise for addressing key components of generating useful curriculum, there is still signiﬁcant room for improvement.

Original: One limitation of our current approach is that it depends on specifying a goal space for the generator to act in, which was not necessary in the original ASP method.
Revised: One limitation of our current approach is that it depends on specifying a goal space for the generator to act in, which was not necessary in the original ASP method.

Original: That said, specifying a goal space also allows us to imbue the system with human priors for what goals should or should not be valid, whereas ASP may need post-hoc goal ﬁltering if Alice ends up in an undesirable state.
Revised: That said, specifying a goal space also allows us to imbue the system with human priors for what goals should or should not be valid, whereas ASP may need post-hoc goal ﬁltering if Alice ends up in an undesirable state.

Original: To further tackle automatic curriculum generation, future work should explore scenarios where goal spaces are not fully pre-speciﬁed, as well as extending to high dimensional goal spaces (e.g. images).
Revised: To further tackle automatic curriculum generation, future work should explore scenarios where goal spaces are not fully pre-speciﬁed, as well as extending to high dimensional goal spaces (e.g. images).

Original: Furthermore, another interesting avenue would be extrapolating CuSP to a larger number of student and teacher agents, which can potentially provide beneﬁts through better goal space coverage and giving a more accurate estimate of feasibility. 
Revised: Furthermore, another interesting avenue would be extrapolating CuSP to a larger number of student and teacher agents, which can potentially provide beneﬁts through better goal space coverage and giving a more accurate estimate of feasibility. A CKNOWLEDGEMENTS

Revised: Thanks to Eugene Vinitsky for helpful discussions, and Hao Liu and Olivia Watkins for feedback on drafts.

Revised: This work was supported by the Center for Human-Compatible Artiﬁcial Intelligence and a

Revised: FAIR-BAIR collaboration between UC Berkeley and Meta.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Automatic curriculum generation has potential as a method for learning complex agent behaviours in an unsupervised way.
Revised: Automatic curriculum generation has potential as a method for learning complex agent behaviours in an unsupervised way.

Original: Requiring less direct supervision (eg.
Revised: Requiring less direct supervision (eg.

Original: through hand-tuned, shaped reward functions) can lead to more robust AI systems for real-world applications.
Revised: through hand-tuned, shaped reward functions) can lead to more robust AI systems for real-world applications.

Original: At the same time, it is important to consider potential impacts unsupervised reinforcement learning, as it may lead to unintended behaviours.
Revised: At the same time, it is important to consider potential impacts unsupervised reinforcement learning, as it may lead to unintended behaviours.

Original: By incorporating human engineering and priors through goal space speciﬁcation in the goal generative methods, like CuSP, we can help restrict the set of goal spaces to more desirable (e g. safe) behaviours.
Revised: By incorporating human engineering and priors through goal space speciﬁcation in the goal generative methods, like CuSP, we can help restrict the set of goal spaces to more desirable (e.g. safe) behaviours.

Original: While our empirical studies have been done in simulated systems, careful and thorough analysis of learned behaviours should be done before deploying CuSP on real world systems.
Revised: While our empirical studies have been done in simulated systems, careful and thorough analysis of learned behaviours should be done before deploying CuSP on real world systems.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: See the anonymized codebase in the supplementary materials for reproducing the experiments.
Revised: Code is available at https://github.com/yuqingd/cusp .

Original: For experimental environment details, see Appendices B.1, B.2.
Revised: For experimental environment details, see Appendices B.1, B.2.

Original: For hyperparameters, see Appendix B.3.
Revised: For hyperparameters, see Appendix B.3.

Original: For ablations of CuSP and ASP, see Appendices B.5.1 and B.5.2 respectively.
Revised: For ablations of CuSP and ASP, see Appendices B.5.1 and B.5.2 respectively.

Original: A D ESIRED G OAL G ENERATION Q UALITIES
Revised: A D ESIRED G OAL G ENERATION Q UALITIES

Original: We begin by qualitatively examining the key properties of the curriculum of goals generated in 
Revised: We begin by qualitatively examining the key properties of the curriculum of goals generated in Point

Original: Point Mass Obstacle.
Revised: Mass Obstacle.

Original: We see that qualitatively our method (Fig. 8a) proposes goals with more progressive exploration than either ASP (Fig. 8d)
Revised: We see that qualitatively our method (Fig. 8a) proposes goals with more progressive exploration than either ASP (Fig. 8d)

Original: (which seems to get stuck on the walls, limited by Alice’s ability to explore) or GoalGAN (Fig. 8c) (which seems to get stuck more along the borders of the space, only prioritizing difﬁcult goals).
Revised: (which seems to get stuck on the walls, limited by Alice’s ability to explore) or GoalGAN (Fig. 8c) (which seems to get stuck more along the borders of the space, only prioritizing difﬁcult goals).

Original: The uniform random generator (Fig. 8b) has the opposite result, having great coverage but without any consideration for feasibility or difﬁculty.
Revised: The DR generator (Fig. 8b) has the opposite result, having great coverage but without any consideration for feasibility or difﬁculty.

Original: Our method manages to balance the exploration with progressive feasibility, where we see CuSP proposing goals more towards the corners at ﬁrst, then progressively moving towards the center and covering the space more thoroughly, with slightly more density in the top right where the maze is.
Revised: Our method manages to balance the exploration with progressive feasibility, where we see CuSP proposing goals more towards the corners at ﬁrst, then progressively moving towards the center and covering the space more thoroughly, with slightly more density in the top right where the maze is.

Original: We hypothesize there is more concentration in the corners initially when the goal generators are untrained and proposing goals towards the edges of the goal constraints
Revised: We hypothesize there is more concentration in the corners initially when the goal generators are untrained and proposing goals towards the edges of the goal constraints.

Original: [Figure]
Revised: [Figure]

Original: stuck on the environment walls, limited by Alice’s abilities.

Original: B E XPERIMENTS
Revised: B E XPERIMENTS

Original: Compute.
Revised: Compute.

Original: To train each method in each environment, we use 1 NVIDIA V100 GPU per seed in an internal cluster.
Revised: To train each method in each environment, we use 1 NVIDIA V100 GPU per seed in an internal cluster.

Original: This amounts to an approximate total of 400 GPUs for the ﬁnal evaluations and hyperparameter sweeps.
Revised: This amounts to an approximate total of 400 GPUs for the ﬁnal evaluations and hyperparameter sweeps.

Original: We additionally did extensive prototyping on the Point Mass environment before scaling to the more complex environments, but do not have a reasonable estimate for how much compute was expended in that phase of the project.
Revised: We additionally did extensive prototyping on the Point Mass environment before scaling to the more complex environments, but do not have a reasonable estimate for how much compute was expended in that phase of the project.

Original: B. T OY E XPERIMENTS
Revised: B. T OY E XPERIMENTS

Original: For the toy evaluations in Figures 2 and 3, we evaluate on the landscape regret ( x, y ) =
Revised: For the toy evaluations in Figures 2 and 3, we evaluate on the landscape regret ( x, y ) =

Original: (cid:26) − (( x + . 1) 

Revised: (cid:26) − (( x + . 1)

Original: + ( y − . 1) 2 )
Revised: + ( y − . 1) 2 )

Original: if ( x + . 1) 2 + ( y − . 1) 2 < .− .otherwise
Revised: if ( x + . 1) 2 + ( y − . 1) 2 < .− .otherwise

Original: For Adam, we use the default conﬁguration from PyTorch (lr=0.001, β s = (0.9, 0.999), (cid:15) =1e-8, no weight decay).
Revised: For Adam, we use the default conﬁguration from PyTorch (lr=0.001, β s = (0.9, 0.999), (cid:15) =1e-8, no weight decay).

Original: For PPO, we use the default conﬁguration from (Kostrikov, 2018).
Revised: For PPO, we use the default conﬁguration from (Kostrikov, 2018).

Original: We found our trends to hold robustly for a fairly broad choice of lr’s and both with/without GAE (Schulman et al., 2018).
Revised: We found our trends to hold robustly for a fairly broad choice of lr’s and both with/without GAE (Schulman et al., 2018).

Original: In the toy experiments, we carry out 5000 steps of goal proposal with one gradient update for every method at each step.
Revised: In the toy experiments, we carry out 5000 steps of goal proposal with one gradient update for every method at each step.

Original: For the non-stationary experiment in Figure 3, we diagonally perturb the center of the regret landscape from top left quadrant (initialized at (-0.1, 0.1)) to the bottom right quadrant at a constant rate of 2e-4 every round.
Revised: For the non-stationary experiment in Figure 3, we diagonally perturb the center of the regret landscape from top left quadrant (initialized at (-0.1, 0.1)) to the bottom right quadrant at a constant rate of 2e-4 every round.

Original: Hence, at the end of 2500 rounds, the center of the landscape is at the origin and at the end of 5000 rounds, the center of the landscape is at (0.1, -0.1).
Revised: Hence, at the end of 2500 rounds, the center of the landscape is at the origin and at the end of 5000 rounds, the center of the landscape is at (0.1, -0.1).

Original: B. E NVIRONMENT S PECIFICS
Revised: B. E NVIRONMENT S PECIFICS

Original: Table 2 describes the reward conﬁgurations we used in our environments.
Revised: Table 2 describes the reward conﬁgurations we used in our environments.

Original: For the manipulation environments, we also incorporated HER (Andrychowicz et al., 2017).
Revised: For the manipulation environments, we also incorporated HER (Andrychowicz et al., 2017).

Original: (cid:15) is the tolerance for success (i.e. a goal g is successfully achieved at state s if the normed distance L 2 ( s − g ) < (cid:15) ).
Revised: (cid:15) is the tolerance for success (i.e. a goal g is successfully achieved at state s if the normed distance L 2 ( s − g ) < (cid:15) ).

Original: We terminate the episode early if successful.
Revised: We terminate the episode early if successful.

Original: [Table]
Revised: [Table]

Original: [Table]

Original:  al.,[Table] ) (cid:15) [Table]
Revised: (Andrychowicz et al.,

Original: if

Original: [Table]
Revised: [Table]

Original: We modify the default Point Mass environment by adding in walls to the top right corner and initialize the agent in between the walls 10% of the time to help with exploration.
Revised: We modify the default Point Mass environment by adding in walls to the top right corner and initialize the agent in between the walls 10% of the time to help with exploration.

Original: We modify the manipulator environment by always initializing with the ball in the gripper and with the gripper in an upright position.
Revised: We modify the manipulator environment by always initializing with the ball in the gripper and with the gripper in an upright position.

Original: We use an episode length of 1000 steps for Point Mass Obstacle and Walker and an episode length of 100 steps for Reach, Toss, and Pick.
Revised: We use an episode length of 1000 steps for Point Mass Obstacle and Walker and an episode length of 100 steps for Reach, Toss, and Pick.

Original: Speciﬁc deﬁnitions of in-distribution and out-of-distribution goal spaces for each environment are in Table
Revised: Speciﬁc deﬁnitions of in-distribution and out-of-distribution goal spaces for each environment are in Table

Original: [Table]
Revised: [Table]

Original: B. H YPERPARAMETERS
Revised: B. H YPERPARAMETERS

Original: We train the learners and goal generators using the default SAC conﬁguration and implementation from (Yarats & Kostrikov, 2020), as listed in Table 4, with separate networks for the actor and critic. 
Revised: We train the learners and goal generators using the default SAC conﬁguration and implementation from (Yarats & Kostrikov, 2020), as listed in Table 4, with separate networks for the actor and critic. [Table]

Original: To scale the output of the goal generators to the correct action space, we take the Tanh scaled output and rescale each dimension to the goal space.
Revised: To scale the output of the goal generators to the correct action space, we take the Tanh scaled output and rescale each dimension to the goal space.

Original: We use Appendix C of (Haarnoja et al., 2018) to modify the log loss for SAC accordingly.
Revised: We use Appendix C of (Haarnoja et al., 2018) to modify the log loss for SAC accordingly.

Original: To train the goal generator, we update after each round (i.e. per goal proposed) with 100 gradient updates.
Revised: To train the goal generator, we update after each round (i.e. per goal proposed) with 100 gradient updates.

Original: As the learners are updated once per time-step in the environment (100-1000 updates depending on the environment), we increased the update frequency for the goal-generator to keep pace with the learner updates since the goal generator operates on a single time step per episode.[Table] [Table]
Revised: As the learners are updated once per time-step in the environment (100-1000 updates depending on the environment), we increased the update frequency for the goal-generator to keep pace with the learner updates since the goal generator operates on a single time step per episode.

Original:  B ASELINES
Revised: B. B ASELINES

Original: We reimplement each baseline by comparing against any available public code and matching each paper’s recommended training hyperparameters as closely as possible within our compute limitations.
Revised: We reimplement each baseline by comparing against any available public code and matching each paper’s recommended training hyperparameters as closely as possible within our compute limitations.

Original: Off vs. On Policy Learners In ASP+BC(OpenAI et al., 2021), Alice and Bob are trained using
Revised: Off vs. On Policy Learners In ASP+BC(OpenAI et al., 2021), Alice and Bob are trained using

Original: PPO.
Revised: PPO.

Original: We also implemented this as a baseline but found that the agents were unable to reach high success at the same rate as our SAC-trained agents.
Revised: We also implemented this as a baseline but found that the agents were unable to reach high success at the same rate as our SAC-trained agents.

Original: For a comparison that’s more favorable to the baselines, we report the success rates from training with the same agent conﬁguration (ie.
Revised: For a comparison that’s more favorable to the baselines, we report the success rates from training with the same agent conﬁguration (ie.

Original: Bob trained with SAC) in the main paper to isolate learning differences to the goal generation method.
Revised: Bob trained with SAC) in the main paper to isolate learning differences to the goal generation method.

Original: B. A BLATIONS B.5. C U SP A BLATIONS
Revised: B. A BLATIONS B.5. C U SP A BLATIONS

Original: SAC Motivation Without symmetrization
Revised: SAC Motivation Without symmetrization

Original: , the regret-based goal generation objective is the same as the proposed objective in PAIRED Dennis et al.
Revised: , the regret-based goal generation objective is the same as the proposed objective in PAIRED Dennis et al.

Original: However, our contribution extends beyond the symmetrized objective as our proposed method reframes the goal generation process into a entropy-regularized agent with a memory buffer – which motivated designing our method around
Revised: However, our contribution extends beyond the symmetrized objective as our proposed method reframes the goal generation process into a entropy-regularized agent with a memory buffer – which motivated designing our method around

Original: SAC.
Revised: SAC.

Original: PAIRED uses PPO to optimize their environment generators, which we found was not sample efﬁcient enough for our tasks (i.e. success rates were too low), as our toy experiment in Figuresuggested.
Revised: PAIRED uses PPO to optimize their environment generators, which we found was not sample efﬁcient enough for our tasks (i.e. success rates were too low), as our toy experiment in Figuresuggested.

Original: Here we highlight a case study with the Toss task into why the entropy regularization and replay buffer components are important.
Revised: Here we highlight a case study with the Toss task into why the entropy regularization and replay buffer components are important.

Original: Building up from just the regret objective as in PAIRED, we have in Figure 9:
Revised: Building up from just the regret objective as in PAIRED, we have in Figure 9:

Original: β
Revised: β

Original: = 1 , α = 0 , no replay buffer – Regret objective only for a single goal generator, as in PAIRED.
Revised: = 1 , α = 0 , no replay buffer – Regret objective only for a single goal generator, as in PAIRED.

Original: β
Revised: β

Original: = 1 , α = 0 , no replay buffer, symmetrized – Regret objective with two symmetrized goal generators.
Revised: = 1 , α = 0 , no replay buffer, symmetrized – Regret objective with two symmetrized goal generators.

Original: Symmetrization seems to improve performance slightly, but is not sufﬁcient alone.
Revised: Symmetrization seems to improve performance slightly, but is not sufﬁcient alone.

Original: β
Revised: β

Original: = 1 , α = 0 – Regret objective with a replay buffer for the goal generator.
Revised: = 1 , α = 0 – Regret objective with a replay buffer for the goal generator.

Original: Also improves performance slightly, although not as much as symmetrization.
Revised: Also improves performance slightly, although not as much as symmetrization.

Original: β
Revised: β

Original: = 1 – Regret objective with a replay buffer and entropy regularization.
Revised: = 1 – Regret objective with a replay buffer and entropy regularization.

Original: This is particularly helpful for the random OOD evaluation as the greater goal diversity improves multi-goal performance on the multi-goal evaluation task, as expected.
Revised: This is particularly helpful for the random OOD evaluation as the greater goal diversity improves multi-goal performance on the multi-goal evaluation task, as expected.

Original: β = 1 symmetrized – Regret objective with a replay buffer and entropy regularization, and symmetrized goal generators.
Revised: β = 1 symmetrized – Regret objective with a replay buffer and entropy regularization, and symmetrized goal generators.

Original: T = 300 , β = .
Revised: T = 300 , β = .

Original: 1 , symmetrized (Full CuSP) – Regret objective with a dynamically updated replay buffer and entropy regularization, and symmetrized goal generators.
Revised: 1 , symmetrized (Full CuSP) – Regret objective with a dynamically updated replay buffer and entropy regularization, and symmetrized goal generators.

Original: Looking at a sample of the generated goals in Figure 10, we ﬁnd a stark qualitative improvement in goal diversity between the regret-only PAIRED condition and CuSP.
Revised: Looking at a sample of the generated goals in Figure 10, we ﬁnd a stark qualitative improvement in goal diversity between the regret-only PAIRED condition and CuSP.

Original: In particular, in the baseline the generated goals converge only on the corners.
Revised: In particular, in the baseline the generated goals converge only on the corners.

Original: Regret update and Symmetrization Ablations As we use the learners’ critics for estimating the current regret for a particular goal, we need to be careful about how we use the critic data since it can be inaccurate, especially towards the beginning of training.
Revised: Regret update and Symmetrization Ablations As we use the learners’ critics for estimating the current regret for a particular goal, we need to be careful about how we use the critic data since it can be inaccurate, especially towards the beginning of training.

Original: We introduce two hyperparameters: one for specifying at which episode we begin regret updates T , and a weighting parameter β for how much to update the regrets by.
Revised: We introduce two hyperparameters: one for specifying at which episode we begin regret updates T , and a weighting parameter β for how much to update the regrets by.

Original: We use the update rule
Revised: We use the update rule

Original: [Equation]
Revised: [Equation]

Original: We sweep across T = 50 , 100 , 200 and β = .
Revised: We sweep across T = 50 , 100 , 200 and β = .

Original: 9 to select the best performing hyperparameters based on average performance on the training goal set, G id .
Revised: 9 to select the best performing hyperparameters based on average performance on the training goal set, G id .

Original: In Figure 11, we ablate over the different components of our method: CuSP+SAC with no entropy regularization ( β = 1 , α = 0 ),
Revised: In Figure 11, we ablate over the different components of our method: CuSP+SAC with no entropy regularization ( β = 1 , α = 0 ),

Original: CuSP+SAC only ( β = 1 ), CuSP+SAC+Symmetrization ( β = 1 ), and CuSP+SAC+Regret Updates+Symmetrization to investigate the effects of each component.
Revised: CuSP+SAC only ( β = 1 ), CuSP+SAC+Symmetrization ( β = 1 ), and CuSP+SAC+Regret Updates+Symmetrization to investigate the effects of each component.

Original: In most of the tasks, aside from
Revised: In most of the tasks, aside from

Original: Reach on random out of distribution goals, symmetrization improves performance.
Revised: Reach on random out of distribution goals, symmetrization improves performance.

Original: In particular, we see larger gains on tasks where β = 1 performance is lower to begin with, such as Walker.
Revised: In particular, we see larger gains on tasks where β = 1 performance is lower to begin with, such as Walker.

Original: Further gains from stale regret updates vary from being very small on tasks where β = 1 performance is already high (eg. for Point Mass Obstacles or Toss on random OOD goals), to larger improvements on the harder skill tasks (eg.
Revised: Further gains from stale regret updates vary from being very small on tasks where β = 1 performance is already high (eg. for Point Mass Obstacles or Toss on random OOD goals), to larger improvements on the harder skill tasks (eg.

Original: Behind Obstacles or Walker Far Right goals).
Revised: Behind Obstacles or Walker Far Right goals).

Original: Interestingly, we also note that while entropy regularization is crucial for most of the environments, setting α = 0 seems to be helpful for Walker.
Revised: Interestingly, we also note that while entropy regularization is crucial for most of the environments, setting α = 0 seems to be helpful for Walker.

Original: [Figure]
Revised: [Figure]

Original: cce ss R a t e
Revised: cce ss R a t e

Original: [Figure]
Revised: [Figure]

Original: S u cce ss R a t e
Revised: S u cce ss R a t e

Original: [Figure]
Revised: [Figure]

Original: Multiagent Motivation.
Revised: Multiagent Motivation.

Original: Here we investigate whether it is sufﬁcient to formulate the CuSP game with only a single learner.
Revised: Here we investigate whether it is sufﬁcient to formulate the CuSP game with only a single learner.

Original: Rather than using regret deﬁned as the difference in returns of two agents, we look at the difference in the returns of a single agent across two rollouts to identify if stochasticity in the policy alone is sufﬁcient for providing the goal generator signal.
Revised: Rather than using regret deﬁned as the difference in returns of two agents, we look at the difference in the returns of a single agent across two rollouts to identify if stochasticity in the policy alone is sufﬁcient for providing the goal generator signal.

Original: As a case study we look at the Toss task in Figure 12, where we ﬁnd the success rate increases much more slowly with a single agent.
Revised: As a case study we look at the Toss task in Figure 12, where we ﬁnd the success rate increases much more slowly with a single agent.

Original: We hypothesize that a challenge with only using a single agent fro deﬁning regret is that the regret landscape is much more ﬂat – i.e. the regret is more likely to be very small and not provide much signal to the goal generators, as the stochasticity of a single agent policy is comparatively lesser than two separate agents.
Revised: We hypothesize that a challenge with only using a single agent fro deﬁning regret is that the regret landscape is much more ﬂat – i.e. the regret is more likely to be very small and not provide much signal to the goal generators, as the stochasticity of a single agent policy is comparatively lesser than two separate agents.

Original: As a result, the goals are less likely to be a useful curriculum for the agent.
Revised: As a result, the goals are less likely to be a useful curriculum for the agent.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: ASP

Revised: [Figure]

Revised: [Figure]

Original: S u cce ss R a t e
Revised: S u cce ss R a t e

Original: [Figure]
Revised: [Figure]

Original: S u cce ss R a t e
Revised: S u cce ss R a t e

Original: [Figure]
Revised: [Figure]

Original: In the original implementation, ASP+BC uses PPO to optimize both Alice and Bob.
Revised: In the original implementation, ASP+BC uses PPO to optimize both Alice and Bob.

Original: We ﬁnd this very suboptimal for the environment setups in this work.
Revised: We ﬁnd this very suboptimal for the environment setups in this work.

Original: For empirical validation, here we report results using 3 variations of ASP+BC:
Revised: For empirical validation, here we report results using 3 variations of ASP+BC:

Original: • ASP+BC (Sparse): Alice is rewarded +1 at the end of an episode of Bob does not succeed, or receives 0 reward otherwise.
Revised: • ASP+BC (Sparse): Alice is rewarded +1 at the end of an episode of Bob does not succeed, or receives 0 reward otherwise.

Original: Both Alice and Bob are optimized using PPO.
Revised: Both Alice and Bob are optimized using PPO.

Original: • ASP+BC (Dense): Alice is rewarded negative Bob’s reward at each transition.
Revised: • ASP+BC (Dense): Alice is rewarded negative Bob’s reward at each transition.

Original: Both Alice and Bob are optimized using PPO.
Revised: Both Alice and Bob are optimized using PPO.

Original: • ASP+BC (Dense, SAC): Alice is rewarded negative Bob’s reward at each transition.
Revised: • ASP+BC (Dense, SAC): Alice is rewarded negative Bob’s reward at each transition.

Original: Alice is optimized with PPO while Bob is optimized with SAC.
Revised: Alice is optimized with PPO while Bob is optimized with SAC.

Original: For all cases, we use the behavioural cloning (BC) mechanism proposed in Section 3.2 of (OpenAI et al., 2021), where Bob is updated with the same clipped behavioural cloning loss on trajectories where it does not successfully achieve Alice’s proposed goal.
Revised: For all cases, we use the behavioural cloning (BC) mechanism proposed in Section 3.2 of (OpenAI et al., 2021), where Bob is updated with the same clipped behavioural cloning loss on trajectories where it does not successfully achieve Alice’s proposed goal.

Original: The results are shown in Figure 14.
Revised: The results are shown in Figure 14.

Original: We observe that with a sparse reward, Alice would tend to get stuck with proposing similar goals in a particular location.
Revised: We observe that with a sparse reward, Alice would tend to get stuck with proposing similar goals in a particular location.

Original: While Bob learns to succeed at that single goal, the lack of diversity causes Bob to perform poorly at generalization to novel goals in
Revised: While Bob learns to succeed at that single goal, the lack of diversity causes Bob to perform poorly at generalization to novel goals in

Original: G ood , as seen in the evaluations.
Revised: G ood , as seen in the evaluations.

Original: To overcome this, we incorporated a dense reward implementation and swapped to optimizing for Bob’s policy using SAC.
Revised: To overcome this, we incorporated a dense reward implementation and swapped to optimizing for Bob’s policy using SAC.

Original: Our results presented in Section 4 use this stronger baseline for comparison against CuSP.
Revised: Our results presented in Section 4 use this stronger baseline for comparison against CuSP.

Original: C T HEORETICAL A NALYSIS
Revised: C T HEORETICAL A NALYSIS

Original: Our overall framework contains 4 players: two goal-conditioned agents, Alice π A and Bob π B , and their corresponding goal generators, G A and G B .
Revised: Our overall framework contains 4 players: two goal-conditioned agents, Alice π A and Bob π B , and their corresponding goal generators, G A and G B .

Original: Further, let Π and G denote the space of policies and goal generative models respectively.
Revised: Further, let Π and G denote the space of policies and goal generative models respectively.

Original: At any given round, let g A ∼ G A and g B ∼ G B denote the corresponding goals sampled from G A and G B respectively.
Revised: At any given round, let g A ∼ G A and g B ∼ G B denote the corresponding goals sampled from G A and G B respectively.

Original: Accordingly, we can deﬁne two regrets for the sampled goals:
Revised: Accordingly, we can deﬁne two regrets for the sampled goals:

Original: [Equation]
Revised: [Equation]

Original: where R ( g, π A ) and R ( g, π B ) denote the empirical discounted sum of rewards for Alice and Bob respectively on a goal g .
Revised: where R ( g, π A ) and R ( g, π B ) denote the empirical discounted sum of rewards for Alice and Bob respectively on a goal g .

Original: Now, we deﬁne the objective function f below:
Revised: Now, we deﬁne the objective function f below:

Original: [Equation]
Revised: [Equation]

Original: The overall objective function for CuSP can be written as:
Revised: The overall objective function for CuSP can be written as:

Original: [Equation]
Revised: [Equation]

Original: To analyze the above objective, we deﬁne coordinating goal generator-solver teams ( G, π ) as a pair of goal generator G ∈ G and goal conditioned agent π ∈ Π .
Revised: To analyze the above objective, we deﬁne coordinating goal generator-solver teams ( G, π ) as a pair of goal generator G ∈ G and goal conditioned agent π ∈ Π .

Original: Accordingly, we can interpret CuSP as a 2 player zero-sum game between two coordinating goal generator-solver teams ( G A , π A ) and ( G B , π B ) deﬁned via goal-conditioned agents π A ∈ Π (Alice) and π B ∈ Π (Bob) along with their corresponding friendly goal generators G A ∈ G and G B ∈ G respectively.
Revised: Accordingly, we can interpret CuSP as a 2 player zero-sum game between two coordinating goal generator-solver teams ( G A , π A ) and ( G B , π B ) deﬁned via goal-conditioned agents π A ∈ Π (Alice) and π B ∈ Π (Bob) along with their corresponding friendly goal generators G A ∈ G and G B ∈ G respectively.

Original: Deﬁnition C.1.
Revised: Deﬁnition C.1.

Original: (CuSP game)
Revised: (CuSP game)

Original: The CuSP game is a 2 player zero-sum game between two coordinating goal generator-solver teams ( G A , π A ) and ( G B , π B ) deﬁned via:
Revised: The CuSP game is a 2 player zero-sum game between two coordinating goal generator-solver teams ( G A , π A ) and ( G B , π B ) deﬁned via:

Original: • Alice π A ∈ Π and its friendly goal generator G A ∈ G and • Bob π B ∈ Π and its friendly goal generator G B ∈ G .
Revised: • Alice π A ∈ Π and its friendly goal generator G A ∈ G and • Bob π B ∈ Π and its friendly goal generator G B ∈ G .

Original: The game is zero-sum with payoff function given as F ( G A , G B , π A , π B ) for ( G A , π A ) , and as − F ( G A , G B , π A , π B ) for ( G B , π B ) .
Revised: The game is zero-sum with payoff function given as F ( G A , G B , π A , π B ) for ( G A , π A ) , and as − F ( G A , G B , π A , π B ) for ( G B , π B ) .

Original: Next, we state conditions under which a Nash Equilibrium exists for coordinating goal generatorsolver agents in CuSP.
Revised: Next, we state conditions under which a Nash Equilibrium exists for coordinating goal generatorsolver agents in CuSP.

Original: The conditions depend highly on whether the game strategies (given by the product of goal generator space and policy space) are ﬁnite or continuous.
Revised: The conditions depend highly on whether the game strategies (given by the product of goal generator space and policy space) are ﬁnite or continuous.

Original: Proposition C.2.
Revised: Proposition C.2.

Original: (Finite Games) Let ( G A , π A ) ∈ G× Π and ( G B ,
Revised: (Finite Games) Let ( G A , π A ) ∈ G× Π and ( G B ,

Original: π B ) ∈
Revised: π B ) ∈

Original: G× Π be two coordinating goal generator-solver agents for CuSP deﬁned over ﬁnite goal space G and policy space Π .
Revised: G× Π be two coordinating goal generator-solver agents for CuSP deﬁned over ﬁnite goal space G and policy space Π .

Original: Then, there exists a mixed strategy Nash Equilibrium for the CuSP game.
Revised: Then, there exists a mixed strategy Nash Equilibrium for the CuSP game.

Original: Proof.
Revised: Proof.

Original: Since G and Π are ﬁnite, we know that the product space G × Π is also ﬁnite.
Revised: Since G and Π are ﬁnite, we know that the product space G × Π is also ﬁnite.

Original: The result then follows directly from Nash’s Theorem (Nash, 1951).
Revised: The result then follows directly from Nash’s Theorem (Nash, 1951).

Original: Proposition C.3.
Revised: Proposition C.3.

Original: (Continuous Games) Let ( G A ,
Revised: (Continuous Games) Let ( G A ,

Original: π A ) ∈ G × Π
Revised: π A ) ∈ G × Π

Original: and
Revised: and

Original: ( G B , π B ) ∈
Revised: ( G B , π B ) ∈

Original: G × Π be two coordinating goal generator-solver agents for CuSP deﬁned over continuous goal space G and policy space Π .
Revised: G × Π be two coordinating goal generator-solver agents for CuSP deﬁned over continuous goal space G and policy space Π .

Original: Further, let G and Π be nonempty compact metric spaces and the payoff function F (Eq. 4) be continuous.
Revised: Further, let G and Π be nonempty compact metric spaces and the payoff function F (Eq. 4) be continuous.

Original: Then, there exists a mixed strategy Nash Equilibrium for the CuSP game.
Revised: Then, there exists a mixed strategy Nash Equilibrium for the CuSP game.

Original: Proof.
Revised: Proof.

Original: Since G and Π are compact, we know by Tychonoff’s theorem (Tychonoff, 1930; Willard, 2012) that the product space G × Π is also compact.
Revised: Since G and Π are compact, we know by Tychonoff’s theorem (Tychonoff, 1930; Willard, 2012) that the product space G × Π is also compact.

Original: Combined with the fact that F is assumed to be continuous, we can apply Glicksberg’s Theorem (Glicksberg, 1952) to ﬁnish the proof.
Revised: Combined with the fact that F is assumed to be continuous, we can apply Glicksberg’s Theorem (Glicksberg, 1952) to ﬁnish the proof.

Original: Since we have established conditions for the existence of Nash equilibria for the CuSP game, we know from the minimax theorem that we can recover the Nash equilibirum solution by optimizing
Revised: Since we have established conditions for the existence of Nash equilibria for the CuSP game, we know from the minimax theorem that we can recover the Nash equilibirum solution by optimizing

Original: Eq. 4.
Revised: Eq. 4.

Original: Note 1:
Revised: Note 1:

Original: In practice, our algorithms for optimizing Eq.
Revised: In practice, our algorithms for optimizing Eq.

Original: 4 might not satisfy the necessary assumptions for the theoretical results.
Revised: 4 might not satisfy the necessary assumptions for the theoretical results.

Original: For example, we optimize the parameters of generators and policies speciﬁed as neural networks as opposed to optimizing in the space of functions directly.
Revised: For example, we optimize the parameters of generators and policies speciﬁed as neural networks as opposed to optimizing in the space of functions directly.

Original: Further, we consider a single-sample Monte Carlo estimate for the CuSP objective in Eq.
Revised: Further, we consider a single-sample Monte Carlo estimate for the CuSP objective in Eq.

Original: 4 and optimize the goal generators and policy networks sequentially for a ﬁxed number of gradient updates in every round using RL algorithms such as SAC.
Revised: 4 and optimize the goal generators and policy networks sequentially for a ﬁxed number of gradient updates in every round using RL algorithms such as SAC.

Original: While these protocols are standard practice even in related works, deriving theoretical guarantees in such scenarios is extremely challenging and an active area of theoretical research with many open questions.
Revised: While these protocols are standard practice even in related works, deriving theoretical guarantees in such scenarios is extremely challenging and an active area of theoretical research with many open questions.

Original: Note 2: If π A = π B , then r A ( g ) =
Revised: Note 2: If π A = π B , then r A ( g ) =

Original: r B ( g ) for all g , so all corresponding regrets will also be zero and thus give no signal to the goal generators.
Revised: r B ( g ) for all g , so all corresponding regrets will also be zero and thus give no signal to the goal generators.

Original: The key point to note here is that in practice, the goal generators have an exploration component that can avoid getting stuck, e.g., this can be achieved via (cid:15) -greedy or via SAC which has an exploration bonus (as done in CuSP).
Revised: The key point to note here is that in practice, the goal generators have an exploration component that can avoid getting stuck, e.g., this can be achieved via (cid:15) -greedy or via SAC which has an exploration bonus (as done in CuSP).

Original: Concretely, let π A
Revised: Concretely, let π A

Original: = π B = π rand be randomly initialized agents.
Revised: = π B = π rand be randomly initialized agents.

Original: We claim that at initialization, ( G A , π rand ) , ( G B , π rand ) will not be in a Nash Equilibrium except in the case where a random policy is already able to achieve all possible goals.
Revised: We claim that at initialization, ( G A , π rand ) , ( G B , π rand ) will not be in a Nash Equilibrium except in the case where a random policy is already able to achieve all possible goals.

Original: To see why, suppose for the sake of contradiction that the agent teams ( G A , π rand ) , ( G B , π rand ) are at a Nash Equilibrium.
Revised: To see why, suppose for the sake of contradiction that the agent teams ( G A , π rand ) , ( G B , π rand ) are at a Nash Equilibrium.

Original: That is, the teams have the joint best responses to each other.
Revised: That is, the teams have the joint best responses to each other.

Original: Let G be the set of all possible goals and G (cid:48) ⊆
Revised: Let G be the set of all possible goals and G (cid:48) ⊆

Original: G be the set of goals that π rand can solve.
Revised: G be the set of goals that π rand can solve.

Original: If G (cid:48)
Revised: If G (cid:48)

Original: =
Revised: =

Original: G , then π A , π B are already maximally competent and are therefore done learning.
Revised: G , then π A , π B are already maximally competent and are therefore done learning.

Original: Note that this is also what we hope to converge to eventually during training, and the two policies matching can correspond to a Nash Equilibrium in such a case.
Revised: Note that this is also what we hope to converge to eventually during training, and the two policies matching can correspond to a Nash Equilibrium in such a case.

Original: At this point R ( g A , π A , π B ) =
Revised: At this point R ( g A , π A , π B ) =

Original: R ( g B , π B , π A ) = 

Revised: R ( g B , π B , π A ) =

Original: for all g A , g B so neither G A , G B can increase their payoffs, and since π A , π B are parameterized identically, if either policy is able to increase its payoff on some g we can identically update the other policy such that π A
Revised: for all g A , g B so neither G A , G B can increase their payoffs, and since π A , π B are parameterized identically, if either policy is able to increase its payoff on some g we can identically update the other policy such that π A

Original: = π B until neither policy is able to improve its payoff further either.
Revised: = π B until neither policy is able to improve its payoff further either.

Original: If G (cid:48) ⊂ G , then for either team we can pick any single goal g ∗ ∈
Revised: If G (cid:48) ⊂ G , then for either team we can pick any single goal g ∗ ∈

Original: G \ G (cid:48) and learn a policy π ∗ capable of achieving { g ∗ } ∪ G (cid:48) using policy iteration (e.g., with greedy policy improvement assuming no forgetting of past goals).
Revised: G \ G (cid:48) and learn a policy π ∗ capable of achieving { g ∗ } ∪ G (cid:48) using policy iteration (e.g., with greedy policy improvement assuming no forgetting of past goals).

Original: Such a g ∗ will eventually be proposed as long as the goal generators have an exploratory component.
Revised: Such a g ∗ will eventually be proposed as long as the goal generators have an exploratory component.

Original: This means there is a strategy π ∗ that acquires equal utility to π rand on G (cid:48) and higher utility on g ∗ .
Revised: This means there is a strategy π ∗ that acquires equal utility to π rand on G (cid:48) and higher utility on g ∗ .

Original: Thus there exists a strictly better strategy for either team ( G A , π rand ) , ( G B , π rand ) by switching from π rand to π ∗ , which contradicts that the players are already at a Nash Equilibrium to begin with.
Revised: Thus there exists a strictly better strategy for either team ( G A , π rand ) , ( G B , π rand ) by switching from π rand to π ∗ , which contradicts that the players are already at a Nash Equilibrium to begin with.