Original: T OWARDS S KILLED P OPULATION C URRICULUM FOR M ULTI -A GENT R EINFORCEMENT L EARNING
Revised: T OWARDS S KILLED P OPULATION C URRICULUM FOR M ULTI -A GENT R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent advances in multi-agent reinforcement learning (MARL) allow agents to coordinate their behaviors in complex environments.
Revised: Recent advances in multi-agent reinforcement learning (MARL) allow agents to coordinate their behaviors in complex environments.

Original: However, common MARL algorithms still suffer from scalability and sparse reward issues.
Revised: However, common MARL algorithms still suffer from scalability and sparse reward issues.

Original: One promising approach to resolve them is automatic curriculum learning (ACL), where a student (curriculum learner) train on tasks of increasing difficulty controlled by a teacher (curriculum generator).
Revised: One promising approach to resolve them is automatic curriculum learning (ACL), where a student (curriculum learner) train on tasks of increasing difficulty controlled by a teacher (curriculum generator).

Original: Unfortunately, in spite of its success, ACL’s applicability is restricted due to: (1) lack of a general student framework to deal with the varying number of agents across tasks and the sparse reward problem, and (2) the nonstationarity in the teacher’s task due to the ever-changing student strategies.
Revised: Unfortunately, in spite of its success, ACL’s applicability is restricted due to: (1) lack of a general student framework to deal with the varying number of agents across tasks and the sparse reward problem, and (2) the nonstationarity in the teacher’s task due to the ever-changing student strategies.

Original: As a remedy for ACL, we introduce a novel automatic curriculum learning framework,
Revised: As a remedy for ACL, we introduce a novel automatic curriculum learning framework,

Original: Skilled Population Curriculum (SPC), adapting curriculum learning to multi-agent coordination.
Revised: Skilled Population Curriculum (SPC), adapting curriculum learning to multi-agent coordination.

Original: To be specific, we endow the student with population-invariant communication and a hierarchical skill set.
Revised: To be specific, we endow the student with population-invariant communication and a hierarchical skill set.

Original: Thus, the student can learn cooperation and behavior skills from distinct tasks with a varying number of agents.
Revised: Thus, the student can learn cooperation and behavior skills from distinct tasks with a varying number of agents.

Original: In addition, we model the teacher as a contextual bandit conditioned by student policies.
Revised: In addition, we model the teacher as a contextual bandit conditioned by student policies.

Original: As a result, a team of agents can change its size while retaining previously acquired skills.
Revised: As a result, a team of agents can change its size while retaining previously acquired skills.

Original: We also analyze the inherent non-stationarity of this multi-agent automatic curriculum teaching problem, and provide a corresponding regret bound.
Revised: We also analyze the inherent non-stationarity of this multi-agent automatic curriculum teaching problem, and provide a corresponding regret bound.

Original: Empirical results show that our method improves scalability, sample efficiency, and generalization in multiple MARL environments.
Revised: Empirical results show that our method improves scalability, sample efficiency, and generalization in multiple MARL environments.

Original: The source code and the video can be found at https://sites.google.com/view/marl-spc/ .
Revised: The source code and the video can be found at https://sites.google.com/view/marl-spc/ .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multi-agent reinforcement learning (MARL) has long been a go-to tool in complex robotic and strategic domains (RoboCup, 2019; OpenAI, 2019).
Revised: Multi-agent reinforcement learning (MARL) has long been a go-to tool in complex robotic and strategic domains (RoboCup, 2019; OpenAI, 2019).

Original: However, learning effective policies with sparse reward from scratch for large-scale multi-agent systems remains challenging.
Revised: However, learning effective policies with sparse reward from scratch for large-scale multi-agent systems remains challenging.

Original: One of the challenges is that the joint observation-action space grows exponentially with varying numbers of agents.
Revised: One of the challenges is that the joint observation-action space grows exponentially with varying numbers of agents.

Original: Meanwhile, the sparse reward signal requires a large number of training trajectories.
Revised: Meanwhile, the sparse reward signal requires a large number of training trajectories.

Original: Hence, applying existing MARL algorithms directly to complex environments with a large number of agents is not effective.
Revised: Hence, applying existing MARL algorithms directly to complex environments with a large number of agents is not effective.

Original: In fact, they may produce agents that do not collaborate with each other even when it is of significant benefit (Zhang et al., 2021; Yang & Wang, 2020).
Revised: In fact, they may produce agents that do not collaborate with each other even when it is of significant benefit (Zhang et al., 2021; Yang & Wang, 2020).

Original: There are several lines of work related to the large-scale MARL problem with sparse reward, including: reward shaping (Hu et al., 2020), curriculum learning (Chen et al., 2021), and learning from demonstrations (Huang et al., 2021).
Revised: There are several lines of work related to the large-scale MARL problem with sparse reward, including: reward shaping (Hu et al., 2020), curriculum learning (Chen et al., 2021), and learning from demonstrations (Huang et al., 2021).

Original: Among these approaches, the curriculum learning paradigm, in which the difficulty of experienced tasks and the population of training agents progressively grow, shows particular promise.
Revised: Among these approaches, the curriculum learning paradigm, in which the difficulty of experienced tasks and the population of training agents progressively grow, shows particular promise.

Original: In automatic curriculum learning (ACL), a teacher (curriculum generator) learns to adjust the complexity and sequencing of tasks faced by a student (curriculum learner).
Revised: In automatic curriculum learning (ACL), a teacher (curriculum generator) learns to adjust the complexity and sequencing of tasks faced by a student (curriculum learner).

Original: Several works have even proposed multi-agent ACL algorithms, based on approximate or heuristic approaches to teaching, such as DyMA-CL (Wang et al., 2020c), EPC (Long et al., 2020), and VACL (Chen et al., 2021).
Revised: Several works have even proposed multi-agent ACL algorithms, based on approximate or heuristic approaches to teaching, such as DyMA-CL (Wang et al., 2020c), EPC (Long et al., 2020), and VACL (Chen et al., 2021).

Original: However, DyMA-CL and EPC rely on a framework of an off-policy student with replay buffer, and ignore the forgetting problem that arises when the agent population size grows.
Revised: However, DyMA-CL and EPC rely on a framework of an off-policy student with replay buffer, and ignore the forgetting problem that arises when the agent population size grows.

Original: ACL relies on the strong assumption that the value of the learned policy does not change when agents switch to a different task.
Revised: ACL relies on the strong assumption that the value of the learned policy does not change when agents switch to a different task.

Original: Moreover, the teacher in these approaches is still facing an unmitigated non-stationarity problem due to the ever-changing student strategies.
Revised: Moreover, the teacher in these approaches is still facing an unmitigated non-stationarity problem due to the ever-changing student strategies.

Original: In addition, if we somewhat expand the ACL paradigm and presume that the teacher may have another purpose for the sequence of tasks performed by the student, another class of larger-scale MARL solutions should be mentioned.
Revised: In addition, if we somewhat expand the ACL paradigm and presume that the teacher may have another purpose for the sequence of tasks performed by the student, another class of larger-scale MARL solutions should be mentioned.

Original: Namely, hierarchical MARL, which learns temporal abstraction with more dense rewards, including: skill discovery (Yang et al., 2019), option as response (Vezhnevets et al., 2019), role-based MARL (Wang et al., 2020b), and two levels of abstraction (Pang et al., 2019).
Revised: Namely, hierarchical MARL, which learns temporal abstraction with more dense rewards, including: skill discovery (Yang et al., 2019), option as response (Vezhnevets et al., 2019), role-based MARL (Wang et al., 2020b), and two levels of abstraction (Pang et al., 2019).

Original: Alas, hierarchical MARL mostly focuses on one specific task with a fixed number of agents and does not consider the transfer ability of learned complementary skills.
Revised: Alas, hierarchical MARL mostly focuses on one specific task with a fixed number of agents and does not consider the transfer ability of learned complementary skills.

Original: In this paper, we informally give our answer to a question:
Revised: In this paper, we informally give our answer to a question:

Original: Can an elaborate combination of ACL and hierarchical principles learn complex cooperation with sparse reward in MARL ?
Revised: Can an elaborate combination of ACL and hierarchical principles learn complex cooperation with sparse reward in MARL ?

Original: Specifically, we present a novel automatic curriculum learning algorithm, Skilled Population Curriculum (SPC), which learns cooperative behaviors from scratch.
Revised: Specifically, we present a novel automatic curriculum learning algorithm, Skilled Population Curriculum (SPC), which learns cooperative behaviors from scratch.

Original: The core idea of SPC is to encourage the student to learn skills from tasks with different numbers of agents.
Revised: The core idea of SPC is to encourage the student to learn skills from tasks with different numbers of agents.

Original: Motivation from the real world is team sports, where players often train their skills by gradually increasing the difficulty of tasks and the number of coordinating players.
Revised: Motivation from the real world is team sports, where players often train their skills by gradually increasing the difficulty of tasks and the number of coordinating players.

Original: In particular, we implement SPC with three key components.
Revised: In particular, we implement SPC with three key components with the teacher-student framework.

Revised: First, to solve the final complex cooperative tasks, we model the teacher as a contextual bandit, where we utilize an RNN-based (Hochreiter & Schmidhuber, 1997) imitation model to represent student policies and use this to generate the bandit’s context.

Original: First, to handle the varying number of agents across  tasks, motivated by the transformer (Vaswani et al., 2017), which can process sentences of varying lengths, we implement population-invariant communication by treating each agent’s message as a word.
Revised: Second, to handle the varying number of agents across these tasks, motivated by the transformer (Vaswani et al., 2017), which can process sentences of varying lengths, we implement population-invariant communication by treating each agent’s message as a word.

Original: Thus, a self-attention communication channel is used to support an arbitrary number of agents sharing their messages.
Revised: Thus, a self-attention communication channel is used to support an arbitrary number of agents sharing their messages.

Original: Second, to learn transferable skills in the sparse reward setting, we utilize the skill framework in the student.
Revised: Third, to learn transferable skills in the sparse reward setting, we utilize the skill framework in the student.

Original: Agents communicate on the high level about a set of shared low-level policies.
Revised: Agents communicate on the high level about a set of shared low-level policies.

Original: Third, to address the non-stationarity arising from ever-changing student strategies, we model the teacher as a contextual bandit, where we utilize an

Original: RNN-based (Hochreiter & Schmidhuber, 1997) imitation model to represent student policies and use this to generate the bandit’s context.

Original: Empirical results show that our method achieves state-of-the-art performance in several tasks in the multi-particle environment (MPE) (Lowe et al., 2017) and the challenging 5vs5 competition in Google Research Football (GRF)
Revised: Empirical results show that our method achieves state-of-the-art performance in several tasks in the multi-particle environment (MPE) (Lowe et al., 2017) and the challenging 5vs5 competition in Google Research Football (GRF)

Original: (Kurach et al., 2019).
Revised: (Kurach et al., 2019).

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Dec-POMDP.
Revised: Dec-POMDP.

Original: A cooperative MARL problem can be formulated as a decentralized partially observable Markov decision process (Dec-POMDP)
Revised: A cooperative MARL problem can be formulated as a decentralized partially observable Markov decision process (Dec-POMDP)

Original: (Bernstein et al., 2002), which is described as a tuple ⟨ n, S , A , P, R, O , Ω , γ ⟩ , where n represents the number of agents.
Revised: (Bernstein et al., 2002), which is described as a tuple ⟨ n, S , A , P, R, O , Ω , γ ⟩ , where n represents the number of agents.

Original: S represents the space of global states.
Revised: S represents the space of global states.

Original: A =
Revised: A =

Original: { A i } i =1 , ··· ,n denotes the space of actions of all agents.
Revised: { A i } i =1 , ··· ,n denotes the space of actions of all agents.

Original: O =
Revised: O =

Original: { O i } i =1 , ··· ,n denotes the space of observations of all agents.
Revised: { O i } i =1 , ··· ,n denotes the space of observations of all agents.

Original: P : S ×
Revised: P : S ×

Original: A → S denotes the state transition probability function.
Revised: A → S denotes the state transition probability function.

Original: All agents share the same reward as a function of the states and actions of the agents
Revised: All agents share the same reward as a function of the states and actions of the agents

Original: R : S ×
Revised: R : S ×

Original: A → R .
Revised: A → R .

Original: Each agent i receives a private observation o i ∈
Revised: Each agent i receives a private observation o i ∈

Original: O i according to the observation function Ω ( s, i ) :
Revised: O i according to the observation function Ω ( s, i ) :

Original: S → O i .
Revised: S → O i .

Original: γ ∈ [0 , 1] denotes the discount factor.
Revised: γ ∈ [0 , 1] denotes the discount factor.

Original: Multi-armed Bandit.
Revised: Multi-armed Bandit.

Original: Multi-armed bandits (MABs) are a simple but very powerful framework that repeatedly makes decisions under uncertainty.
Revised: Multi-armed bandits (MABs) are a simple but very powerful framework that repeatedly makes decisions under uncertainty.

Original: In an MAB, a learner performs a sequence of actions.
Revised: In an MAB, a learner performs a sequence of actions.

Original: After every action, the learner immediately observes the reward corresponding to its action.
Revised: After every action, the learner immediately observes the reward corresponding to its action.

Original: Given a set of K actions and a time horizon T , the objective is to maximize its total reward over T rounds.
Revised: Given a set of K actions and a time horizon T , the objective is to maximize its total reward over T rounds.

Original: The regret is used to measure the gap between the cumulative reward of an MAB algorithm and the best-arm benchmark.
Revised: The regret is used to measure the gap between the cumulative reward of an MAB algorithm and the best-arm benchmark.

Original: An representative work is the Exp3 algorithm (Auer et al., 2002), which is proposed to increase the probability of pulling good arms and achieves a regret of O ( (cid:112) KT log( K )) under a time-varying reward distribution.
Revised: An representative work is the Exp3 algorithm (Auer et al., 2002), which is proposed to increase the probability of pulling good arms and achieves a regret of O ( (cid:112) KT log( K )) under a time-varying reward distribution.

Original:  S KILLED P OPULATION C URRICULUM
Revised:  S KILLED P OPULATION C URRICULUM

Original: In this section, we first provide a formal definition of the curriculum-enhanced Dec-POMDP framework, which formulates the MARL with curriculum problem under the Dec-POMDP framework.
Revised: In this section, we first provide a formal definition of the curriculum-enhanced Dec-POMDP framework, which formulates the MARL with curriculum problem under the Dec-POMDP framework.

Original: It provides insights to the non-differentiable and non-stationary nature of multi-agent automatic curriculum learning.
Revised: It provides insights to the non-differentiable and non-stationary nature of multi-agent automatic curriculum learning.

Original: For tackling those challenges, we then present our multi-agent automatic curriculum learning algorithm named Skilled Population Curriculum (SPC) as shown in Fig.
Revised: For tackling those challenges, we then present our multi-agent automatic curriculum learning algorithm named Skilled Population Curriculum (SPC) as shown in Fig.

Original: In the following subsections, we first formulate the curriculum learning framework in 3.1, then a propose contextual multi-armed bandit algorithm as the teacher to deal with the non-stationarity in 3.2, and last introduce the student with a skill and population-invariant communication framework to tackle the varying number of agents and the sparse reward problem in 3.3.
Revised: In the following subsections, we first formulate the curriculum learning framework in 3.1, then a propose contextual multi-armed bandit algorithm as the teacher to deal with the non-stationarity in 3.2, and last introduce the student with a skill and population-invariant communication framework to tackle the varying number of agents and the sparse reward problem in 3.3.

Original: [Figure]
Revised: [Figure]

Original: skill
Revised: skill

Original: [Figure]
Revised: [Figure]

Original: Tasks
Revised: Tasks

Original: Prob.
Revised: Prob.

Original: Contexts
Revised: Contexts

Original: Teacher
Revised: Teacher

Original: [Figure]
Revised: [Figure]

Original: Population
Revised: Population

Original: [Figure]
Revised: [Figure]

Original: invariant
Revised: invariant

Original: [Figure]
Revised: [Figure]

Original: 3. P ROBLEM F ORMULATION
Revised: 3. P ROBLEM F ORMULATION

Original: We assume environments from multi-agent automatic curriculum learning problems are equipped with parameterized task spaces and thus can be modeled as curriculum-enhanced Dec-POMDPs.
Revised: We assume environments from multi-agent automatic curriculum learning problems are equipped with parameterized task spaces and thus can be modeled as curriculum-enhanced Dec-POMDPs.

Original: Definition 3.1 (Curriculum-enhanced Dec-POMDP) .
Revised: Definition 3.1 (Curriculum-enhanced Dec-POMDP) .

Original: A curriculum-enhanced Dec-POMDP is defined by a tuple ⟨ Φ , M⟩ , where Φ and M represent a task space and a Dec-POMDP, respectively.
Revised: A curriculum-enhanced Dec-POMDP is defined by a tuple ⟨ Φ , M⟩ , where Φ and M represent a task space and a Dec-POMDP, respectively.

Original: Given the task ϕ , the Dec-POMDP M ( ϕ ) is presented as (cid:8) n ϕ , S ϕ , A ϕ , P ϕ , r ϕ , O ϕ , Ω ϕ , γ ϕ (cid:9) .
Revised: Given the task ϕ , the Dec-POMDP M ( ϕ ) is presented as (cid:8) n ϕ , S ϕ , A ϕ , P ϕ , r ϕ , O ϕ , Ω ϕ , γ ϕ (cid:9) .

Original: The superscript ϕ denotes that the Dec-POMDP elements are determined by the task ϕ .
Revised: The superscript ϕ denotes that the Dec-POMDP elements are determined by the task ϕ .

Original: Note that task ϕ can be a few parameters of the environment or task IDs in a finite task space.
Revised: Note that task ϕ can be a few parameters of the environment or task IDs in a finite task space.

Original: In a curriculum-enhanced
Revised: In a curriculum-enhanced

Original: Dec-POMDP, the objective is to improve the student’s performance on the target tasks through the sequence of training tasks given by the teacher. .
Revised: Dec-POMDP, the objective is to improve the student’s performance on the target tasks through the sequence of training tasks given by the teacher. .

Original: Let τ denote a trajectory whose unconditional distribution Pr π,ϕµ ( τ ) under a policy π and a task ϕ
Revised: Let τ denote a trajectory whose unconditional distribution Pr π,ϕµ ( τ ) under a policy π and a task ϕ

Original: with initial state distribution µ ( s 0 ) is Pr π,ϕµ ( τ ) =
Revised: with initial state distribution µ ( s 0 ) is Pr π,ϕµ ( τ ) =

Original: µ ( s 0 ) (cid:80) ∞
Revised: µ ( s 0 ) (cid:80) ∞

Original: t =

Revised: t =

Original: π ( a t | s t ) P ϕ ( s t +1 | s t , a t ) .
Revised: π ( a t | s t ) P ϕ ( s t +1 | s t , a t ) .

Original: We use p ( ϕ ) to represent the distribution of target tasks and q ( ϕ ) to represent the distribution of training tasks at each task sampling step.
Revised: We use p ( ϕ ) to represent the distribution of target tasks and q ( ϕ ) to represent the distribution of training tasks at each task sampling step.

Original: Considering the joint agents’ policies π θ ( a | s ) and q ψ ( ϕ ) parameterized by θ and ψ , respectively.
Revised: We consider the joint agents’ policies π θ ( a | s ) and q ψ ( ϕ ) parameterized by θ and ψ , respectively.

Original: The overall objective to maximize in a curriculum-enhanced Dec-POMDP is:
Revised: The overall objective to maximize in a curriculum-enhanced Dec-POMDP is:

Original: [Equation]
Revised: [Equation]

Original: where R ϕ ( τ ) = (cid:80) t γ t r ϕ ( s t , a t ; s 0 ) and V ( ϕ, π θ ) represent the value function of π θ in Dec-POMDP
Revised: where R ϕ ( τ ) = (cid:80) t γ t r ϕ ( s t , a t ; s 0 ) and V ( ϕ, π θ ) represent the value function of π θ in Dec-POMDP

Original: M ( ϕ ) .
Revised: M ( ϕ ) .

Original: However, when optimizing q ψ ( ϕ ) , we cannot get the partial derivative ∇ ψ J ( θ, ψ ) =
Revised: However, when optimizing q ψ ( ϕ ) , we cannot get the partial derivative ∇ ψ J ( θ, ψ ) =

Original: ∇ ψ (cid:80) τ 1 q ψ ( ϕ ) R ϕ
Revised: ∇ ψ (cid:80) τ 1 q ψ ( ϕ ) R ϕ

Original: ( τ ) Pr π,ϕµ ( τ ) 1 since the reward function and the transition probability function w.r.t number of agents are non-parametric, non-differentiable, and discontinuous in most MARL scenarios.
Revised: ( τ ) Pr π,ϕµ ( τ ) 1 since the reward function and the transition probability function w.r.t number of agents are non-parametric, non-differentiable, and discontinuous in most MARL scenarios.

Original: Thus, we use the non-differentiable method, i.e., multi-armed bandit algorithms to optimize q ψ ( ϕ ) , and optimize the overall objective by learning the distribution of training tasks (the teacher) and an RL algorithm (the student) in alternating periods.
Revised: Thus, we use the non-differentiable method, i.e., multi-armed bandit algorithms to optimize q ψ ( ϕ ) , and optimize the overall objective by learning the distribution of training tasks (the teacher) and an RL algorithm (the student) in alternating periods.

Original: However, there are three key challenges in solving this problem: (1) There is a lack of a general student framework to deal with the varying number of agents across tasks and the sparse reward problem.
Revised: However, there are three key challenges in solving this problem: (1) There is a lack of a general student framework to deal with the varying number of agents across tasks and the sparse reward problem.

Original: The teacher is facing a non-stationarity problem due to the ever-changing student’s strategies.
Revised: The teacher is facing a non-stationarity problem due to the ever-changing student’s strategies.

Original: The forgetting and relearning problem.
Revised: The forgetting and relearning problem.

Original: Some tasks can be the prerequisites of other tasks and sometasks can be inter-independent and parallel.
Revised: Some tasks can be the prerequisites of other tasks, while some can be inter-independent and parallel.

Original: 3. T EACHER  AS A N ON -S TATIONARY C ONTEXTUAL B ANDIT
Revised: 3. T EACHER MODELED AS A N ON -S TATIONARY C ONTEXTUAL B ANDIT

Original: As mentioned in the previous subsection, the teacher is facing a non-stationarity problem due to the ever-changing student’s strategies, since the student learns across different tasks in process.
Revised: As mentioned in the previous subsection, the teacher is facing a non-stationarity problem due to the ever-changing student’s strategies, since the student learns across different tasks in process.

Original: That is, in different stages of student learning, the teacher will observe different student’s performances when giving the same task to the student, thus leading to a time-varying reward distribution of the teacher.
Revised: That is, in different stages of student learning, the teacher will observe different student’s performances when giving the same task to the student, thus leading to a time-varying reward distribution of the teacher.

Original: In addition, there exists the forgetting and relearning problem of the student, where the student forgets the learned policy.
Revised: In addition, there exists the forgetting and relearning problem of the student, where the student forgets the learned policy.

Original: To avoid this problem, the teacher should offer some trained tasks to the student.
Revised: To avoid this problem, the teacher should offer some trained tasks to the student.

Original: It can be seen as the exploitation and exploration problem of the teacher.
Revised: It can be seen as the exploitation and exploration problem of the teacher.

Original: The teacher is encouraged to give the training tasks that benefit the student’s performance on the target tasks; however, there is still a need for sufficient exploration on various training tasks.
Revised: The teacher is encouraged to give the training tasks that benefit the student’s performance on the target tasks; however, there is still a need for sufficient exploration on various training tasks that may not directly facilitate the student’s learning.

Original: Fortunately, we notice that the non-stationarity stems from the student, which can be mitigated with a contextual bandit which embeds the student policy into the context.
Revised: Fortunately, we notice that the non-stationarity stems from the student, which can be mitigated with a contextual bandit which embeds the student policy into the context.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1 Left, the teacher takes the student’s policy representation as the context and chooses a task from the distribution of training tasks.
Revised: 1 Left, the teacher takes the student’s policy representation as the context and chooses a task from the distribution of training tasks.

Original: Specifically, we extend the Exp3 algorithm (Auer et al., 2002) with context by two-step online clustering (see Alg.1(Zhang et al., 1996)).
Revised: Specifically, we extend the Exp3 algorithm (Auer et al., 2002) with context by two-step online clustering (see Alg.1(Zhang et al., 1996)).

Original: The context x is the student’s policy representation, the teacher’s action is a certain task ϕ , and the teacher’s reward is the return of the student in the target tasks.
Revised: The context x is the student’s policy representation, the teacher’s action is a certain task ϕ , and the teacher’s reward is the return of the student in the target tasks.

Revised: The teacher’s algorithm is listed below.

Original: In steps 1-4, the teacher samples a task for the student’s training.
Revised: In steps 1-4, the teacher samples a task for the student’s training.

Original: In steps 6-7, the teacher would update the parameters based on the evaluation reward of the student.
Revised: In steps 6-7, the teacher would update the parameters based on the evaluation reward of the student.

Revised: So, we need to learn a good representation for the student’s policy as the context.

Original: Input: Context x , the number of Clusters N c , N c instances of Exp3 with task distribution w ( ϕ k , c ) for k = 1 , . .
Revised: Input: Context x , the number of Clusters N c , N c instances of Exp3 with task distribution w ( ϕ k , c ) for k = 1 , . .

Original: , K and for c = 1 , . .
Revised: , K and for c = 1 , . .

Original: . , N c , learning rate α , a buffer maintaining the historical contexts
Revised: . , N c , learning rate α , a buffer maintaining the historical contexts

Original: Output: M ( ϕ ) = (cid:8) n ϕ , S ϕ , A ϕ , P ϕ , r ϕ , O ϕ , Ω ϕ , γ ϕ (cid:9) , the teacher bandit parameters
Revised: Output: M ( ϕ ) = (cid:8) n ϕ , S ϕ , A ϕ , P ϕ , r ϕ , O ϕ , Ω ϕ , γ ϕ (cid:9) , the teacher bandit parameters

Original: Sampling
Revised: Sampling

Original: Get the the context x , and save it to the buffer 2.
Revised: Get the the context x , and save it to the buffer 2.

Original: Run the online cluster algorithm and get the index of the cluster center c ( x ) 3.
Revised: Run the online cluster algorithm and get the index of the cluster center c ( x ) 3.

Original: Let the active Exp3 instance be the instance with index c ( x ) 4.
Revised: Let the active Exp3 instance be the instance with index c ( x ) 4.

Original: Set the probability p ( ϕ k , c ( x )) =
Revised: Set the probability p ( ϕ k , c ( x )) =

Original: (1 − α ) w ( ϕ k ,c ( x ))
Revised: (1 − α ) w ( ϕ k ,c ( x ))

Original: (cid:80) Kj =1 w ( ϕ k ,c ( x )) + αK for each task ϕ
Revised: (cid:80) Kj =1 w ( ϕ k ,c ( x )) + αK for each task ϕ

Original: k 5.
Revised: k 5.

Original: Sample a new task according to the distribution of p ϕ k ,c
Revised: Sample a new task according to the distribution of p ϕ k ,c

Original: Training 6.
Revised: Training 6.

Original: Get the return (discounted cumulative rewards) from student testing r 7.
Revised: Get the return (discounted cumulative rewards) from student testing r 7.

Original: Update the active Exp3 instance by setting w ( ϕ k , c ( x ))
Revised: Update the active Exp3 instance by setting w ( ϕ k , c ( x ))

Original: =
Revised: =

Original: w ( ϕ k , c ( x ))
Revised: w ( ϕ k , c ( x ))

Original: e αr/K
Revised: e αr/K

Original: C ONTEXT R EPRESENTATION
Revised: C ONTEXT R EPRESENTATION

Original: We learn the representation of the student policy as a context.

Original: A straightforward representation is to directly use the student parameters θ as the context.
Revised: A straightforward representation is to directly use the student parameters θ as the context.

Original: However, the number of parameters is large and ever-changing if we change the student’s architecture.
Revised: However, the number of parameters is large and ever-changing if we change the student’s architecture.

Original: Thus, we turn to an alternative method.
Revised: Thus, we turn to an alternative method.

Original: A principle to learn a good representation of a policy is predictive representation , that is, the representation should be accurate to predict policy actions given states.
Revised: A principle to learn a good representation of a policy is predictive representation , that is, the representation should be accurate to predict policy actions given states.

Original: According to the principle, we utilize an imitation function through supervised learning.
Revised: According to the principle, we utilize an imitation function through supervised learning.

Original: Supervised learning does not require direct access to reward signals, making it an attractive task for reward-agnostic representation learning.
Revised: Supervised learning does not require direct access to reward signals, making it an attractive approach for reward-agnostic representation learning.

Original: Intuitively, the imitation function attempts to mimic low-level policy based on historical behaviors.
Revised: Intuitively, the imitation function attempts to mimic low-level policy based on historical behaviors.

Original: In practice, we use an RNN-based imitation function f
Revised: In practice, we use an RNN-based imitation function f

Original: im : S ×
Revised: im : S ×

Original: A → [0 , 1] .
Revised: A → [0 , 1] .

Original: Since recurrent neural networks are theoretically Turing complete (Hy¨otyniemi, 1996), its internal states can be used as the representation of the student’s policy.
Revised: Since recurrent neural networks are theoretically Turing complete (Hy¨otyniemi, 1996), its internal states can be used as the representation of the student’s policy.

Original: Regarding the training of this imitation function, we use the negative cross entropy objective E [log f im ( s, a )] .
Revised: Regarding the training of this imitation function, we use the negative cross entropy objective E [log f im ( s, a )] .

Original: 3.2. R EGRET A NALYSIS
Revised: 3.2. R EGRET A NALYSIS

Original: In this subsection, we show the regret bound of the proposed teacher algorithm E
Revised: In this subsection, we show the regret bound of the proposed teacher algorithm E

Original: [ R ( T )] =
Revised: [ R ( T )] =

Original: O (cid:0) T 2 / 3 ( LK log T ) 1 / 3 (cid:1) , where T is the number of total rounds, L is the Lipschitz constant, and K is the number of arms (the number of the teacher’s actions).
Revised: O (cid:0) T 2 / 3 ( LK log T ) 1 / 3 (cid:1) , where T is the number of total rounds, L is the Lipschitz constant, and K is the number of arms (the number of the teacher’s actions).

Revised: The regret analysis is used to justify the usage of the bandit algorithm in the non-stationary setting.

Original: Since the teacher’s reward is the return of the student in the target tasks, the regret bound shows the optimality of the proposed method.
Revised: Since the teacher’s reward is the return of the student in the target tasks, the regret bound shows the optimality of the proposed method.

Original: First, we introduce the Lipschitz assumption about the generalization ability of the task space.
Revised: First, we introduce the Lipschitz assumption about the generalization ability of the task space.

Original: Assumption 3.2 (Lipschitz continuity w.r.t the context) .
Revised: Assumption 3.2 (Lipschitz continuity w.r.t the context) .

Original: Without loss of generality, the contexts are mapped into the [0 , 1] interval, so that the expected rewards for the teacher are Lipschitz with respect to the context.[Equation] for
Revised: Without loss of generality, the contexts are mapped into the [0 , 1] interval, so that the expected rewards for the teacher are Lipschitz with respect to the context.

Original: [Equation]
Revised: [Equation]

Original: where L is the Lipschitz constant, and X is the context space.
Revised: where L is the Lipschitz constant, and X is the context space.

Original: This assumption suggests that, for any policy that is trained on a set of tasks, the rate of performance change is not faster than the rate of policy change.
Revised: This assumption suggests that, for any policy that is trained on a set of tasks, the rate of performance change is not faster than the rate of policy change.

Original: It is a realistic assumption since we cannot expect the student to achieve a dramatic improvement on a given task when the student is represented by a new context via a few training steps.
Revised: It is a realistic assumption since we cannot expect the student to achieve a dramatic improvement on a given task when the student is represented by a new context via a few training steps.

Original: Then, we borrow a contextual bandit algorithm for a small number of contexts (Auer et al., 2002)
Revised: Then, we borrow a contextual bandit algorithm for a small number of contexts (Auer et al., 2002)

Original: (see Appendix Alg. 2) and the lemma 3.3, as a stepping stone for the proof of Theorem 3.4.
Revised: (see Appendix Alg. 2) and the lemma 3.3, as a stepping stone for the proof of Theorem 3.4.

Original: Lemma 3.3.
Revised: Lemma 3.3.

Original: Alg. 2 has regret E [ R ( T )]
Revised: Alg. 2 has regret E [ R ( T )]

Original: = O ( (cid:112) TK |X| log K ) .
Revised: = O ( (cid:112) TK |X| log K ) .

Original: Lemma 3.3 introduces a square root dependence on |X| if running a separate copy of Exp3 for each context (Auer et al., 2002).
Revised: Lemma 3.3 introduces a square root dependence on |X| if running a separate copy of Exp3 for each context (Auer et al., 2002).

Original: It motivates us to handle large context space by discretization.
Revised: It motivates us to handle large context space by discretization.

Original: Theorem 3.4.
Revised: Theorem 3.4.

Original: Consider the Lipschitz contextual bandit problem with contexts in [0 , 1] .
Revised: Consider the Lipschitz contextual bandit problem with contexts in [0 , 1] .

Original: The Alg.yields regret E [ R ( T )]
Revised: The Alg.yields regret E [ R ( T )]

Original: = O (cid:0) T 2 / 3 ( LK ln T ) 1 / 3 (cid:1) .
Revised: = O (cid:0) T 2 / 3 ( LK ln T ) 1 / 3 (cid:1) .

Original: Proof.
Revised: Proof.

Original: See Appendix B for the proof.
Revised: See Appendix B for the proof.

Original: In practice, the contextual space is high-dimensional instead of in [0 , 1] , andin the proof a uniform mesh is used to discretize the context space.
Revised: In practice, the contextual space is high-dimensional instead of in [0 , 1] , and a uniform mesh is used to discretize the context space in the proof.

Original: Since we cannot have such a uniform mesh, without loss of generality, we utilize the Balanced Iterative Reducing and Clustering using Hierarchies (BIRCH) online clustering algorithm
Revised: Since we cannot have such a uniform mesh, without loss of generality, we utilize the Balanced Iterative Reducing and Clustering using Hierarchies (BIRCH) online clustering algorithm

Original: (Zhang et al., 1996) to generate and discretize the context space.
Revised: (Zhang et al., 1996) to generate and discretize the context space.

Original: BIRCH summarizes large datasets into a smaller tree with clustered leaf nodes.
Revised: BIRCH summarizes large datasets into a smaller tree with clustered leaf nodes.

Original: The clustering is efficient and easy to update for newly added data.
Revised: The clustering is efficient and easy to update for newly added data.

Original: Thus, it is suitable for clustering the RNN-based policy representation.
Revised: Thus, it is suitable for clustering the RNN-based policy representation.

Original: At the end of the training, the cluster can be seen as an approximation of the uniform mesh.
Revised: At the end of the training, the cluster can be seen as an approximation of the uniform mesh.

Original: 3. S TUDENT WITH P OPULATION -I NVARIANT S KILLS
Revised: 3. S TUDENT WITH P OPULATION -I NVARIANT S KILLS

Original: Here, we propose a population-invariant communication framework where agents can communicate via a self-attention channel.
Revised: Here, we propose a population-invariant communication framework where agents can communicate via a self-attention channel.

Original: Moreover, to deal with the sparse reward problem, we introduce a hierarchical framework so that agents can learn the skills that are transferable across different tasks.
Revised: Moreover, to deal with the sparse reward problem, we introduce a hierarchical framework so that agents can learn the skills that are transferable across different tasks.

Original: The student module is MARL algorithm-agnostic.
Revised: The student module is MARL algorithm-agnostic.

Original: It is orthogonal to any state-of-the-art MARL algorithms.
Revised: It is orthogonal to any state-of-the-art MARL algorithms.

Original: Though there are few research (Iqbal et al., 2021;
Revised: Though there are few researches (Iqbal et al., 2021;

Original: Hu et al., 2021) try to address the varying number of agents, they heavily rely on the prior knowledge of the environments.
Revised: Hu et al., 2021) trying to address the varying number of agents, they heavily rely on the prior knowledge of the environments.

Original: Population-Invariant Teamwork Communication.
Revised: Population-Invariant Teamwork Communication.

Original: Instead of learning independent policies for agents in the student, we introduce communication to enable the population-invariant property and learn tactics among agents.
Revised: Instead of learning independent policies for agents in the student, we introduce communication to enable the population-invariant property and learn tactics among agents.

Original: Motivated by the fact that the transformer (Vaswani et al., 2017) in natural language processing can handle varying lengths of sentences, we use the self-attention mechanism in our communication.
Revised: Motivated by the fact that the transformer (Vaswani et al., 2017) in natural language processing can handle varying lengths of sentences, we use the self-attention mechanism in our communication.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1 Right, each agent j receives an observation o j .
Revised: 1 Right, each agent j receives an observation o j .

Original: In each round of communication, each agent j sends a message vector m j = f
Revised: In each round of communication, each agent j sends a message vector m j = f

Original: ( o j ) to a self-attention channel, where f is an observation encoder function.
Revised: ( o j ) to a self-attention channel, where f is an observation encoder function.

Original: The channel aggregates all messages and sends the new message vector ˜ m j through the self-attention mechanism.
Revised: The channel aggregates all messages and sends the new message vector ˜ m j through the self-attention mechanism.

Original: Concretely, given the input of the channel M =
Revised: Concretely, given the input of the channel M =

Original: [ m 1 , m 2 , · · · , m n ] ∈ R n × d m and the trainable weight of the channel W Q , W K ,
Revised: [ m 1 , m 2 , · · · , m n ] ∈ R n × d m and the trainable weight of the channel W Q , W K ,

Original: W V ∈ R
Revised: W V ∈ R

Original: d m × d m , we can obtain three different representations Q = MW Q , K = MW K , V = MW V .
Revised: d m × d m , we can obtain three different representations Q = MW Q , K = MW K , V = MW V .

Original: Then, the output messages are˜M =
Revised: Then, the output messages are

Original: Attention( Q , K , V ) =

Original: softmax (cid:18)

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: where d m is the dimension of the messages.
Revised: where d m is the dimension of the messages.

Original: Since the dimensions of the trainable weight are irrelevant to the number of agents, the student can take advantage of the population-invariant property to learn tactics.
Revised: Since the dimensions of the trainable weight are irrelevant to the number of agents, the student can take advantage of the population-invariant property to learn tactics.

Original: Transferable Hierarchical Skills.
Revised: Transferable Hierarchical Skills.

Original: As shown in the dotted box in Fig.
Revised: As shown in the dotted box in Fig.

Original: 1 Right, after receiving the new messages ˜ m j from the channel, each agent takes the high-level action (skill) a h,j = π h,j ( o j , ˜ m j ) to execute the low-level policy a j = π low ( o j , a h,j ) .
Revised: 1 Right, after receiving the new messages ˜ m j from the channel, each agent takes the high-level action (skill) a h,j = π h,j ( o j , ˜ m j ) to execute the low-level policy a j = π low ( o j , a h,j ) .

Original: In this work, we generalize the high-level action (skill) a h,j to a continuous embedding space, so that the skill can be either a latent continuous vector as in DIAYN (Eysenbach et al., 2018), or a categorical distribution for sampling discrete options (Bacon et al., 2017).
Revised: In this work, we generalize the high-level action (skill) a h,j to a continuous embedding space, so that the skill can be either a latent continuous vector as in DIAYN (Eysenbach et al., 2018), or a categorical distribution for sampling discrete options (Bacon et al., 2017).

Original: Implementation.
Revised: Implementation.

Original: We implement the high- and low-level policies in the student with Proximal Policy
Revised: We implement the high- and low-level policies in the student with Proximal Policy

Original: Optimization (PPO) (Schulman et al., 2017).
Revised: Optimization (PPO) (Schulman et al., 2017).

Original: Following the common practice proposed in (Fu et al., 2022), the high-level policy for each agent is learned independently, whereas the low-level policies share parameters, since the most basic action pattern should be the same within different agents.
Revised: Following the common practice proposed in (Fu et al., 2022), the high-level policy for each agent is learned independently, whereas the low-level policies share parameters, since the most basic action pattern should be the same within different agents.

Original: The low-level agent is rewarded by the environment.
Revised: The low-level agent is rewarded by the environment.

Original: The high-level policy takes actions given a fixed interval during training.
Revised: The high-level policy takes actions given a fixed interval during training.

Original: Within this interval, a cumulative low-level reward is used as a high-level reward.
Revised: Within this interval, a cumulative low-level reward is used as a high-level reward.

Original: When the categorical distribution is used to enable a  skill, we would sample an “option” from the categorical distribution and feed the corresponding one-hot embedding to the low-level policy.
Revised: When the categorical distribution is used to enable a discrete skill, we would sample an “option” from the categorical distribution and feed the corresponding one-hot embedding to the low-level policy.

Original:  F URTHER R ELATED W ORK
Revised:  F URTHER R ELATED W ORK

Original: Automatic Curriculum Learning in MARL.
Revised: Automatic Curriculum Learning in MARL.

Original: Curriculum learning is a training strategy inspired by the human learning process, mimicking how humans learn new concepts in an orderly manner, usually based on the difficulty level of the problems (Portelas et al., 2020b).
Revised: Curriculum learning is a training strategy inspired by the human learning process, mimicking how humans learn new concepts in an orderly manner, usually based on the difficulty level of the problems (Portelas et al., 2020b).

Original: The selection of tasks is formulated as a Curriculum Markov Decision Process (CMDP) (Narvekar & Stone, 2018).
Revised: The selection of tasks is formulated as a Curriculum Markov Decision Process (CMDP) (Narvekar & Stone, 2018).

Original: Automatic
Revised: Automatic

Original: Curriculum Learning mechanisms aim to learn a task selection function based on information about past interactions, such as ADR (Akkaya et al., 2019; Mehta et al., 2020), ALP-GMM (Portelas et al., 2020a), SPCL (Jiang et al., 2015), GoalGAN (Florensa et al., 2018), PLR (Jiang et al., 2021b;a),
Revised: Curriculum Learning mechanisms aim to learn a task selection function based on information about past interactions, such as ADR (Akkaya et al., 2019; Mehta et al., 2020), ALP-GMM (Portelas et al., 2020a), SPCL (Jiang et al., 2015), GoalGAN (Florensa et al., 2018), PLR (Jiang et al., 2021b;a),

Original: SPDL (Klink et al., 2020), CURROT (Klink et al., 2022), and graph-curriculum (Svetlik et al., 2017).
Revised: SPDL (Klink et al., 2020), CURROT (Klink et al., 2022), and graph-curriculum (Svetlik et al., 2017).

Original: Inspired by the mechanism of biodiversity in nature, a series of MARL curriculum learning frameworks have recently been proposed with remarkable empirical success.
Revised: Inspired by the mechanism of biodiversity in nature, a series of MARL curriculum learning frameworks have recently been proposed with remarkable empirical success.

Original: These include openended evolution (Banzhaf et al., 2016; Lehman et al., 2008; Standish, 2003), population-based training (Jaderberg et al., 2019;
Revised: These include openended evolution (Banzhaf et al., 2016; Lehman et al., 2008; Standish, 2003), population-based training (Jaderberg et al., 2019;

Original: Liu et al., 2019), meta-learning (Gupta et al., 2021; Portelas et al., 2020c) and training with emergent curriculum (Baker et al., 2019; Leibo et al., 2019; Portelas et al., 2020b).
Revised: Liu et al., 2019), meta-learning (Gupta et al., 2021; Portelas et al., 2020c) and training with emergent curriculum (Baker et al., 2019; Leibo et al., 2019; Portelas et al., 2020b).

Original: In general, these frameworks can be unified under the idea of an automatic curriculum that automatically generates an endless procession of better performing agents by exerting selection pressure among many self-optimizing agents.
Revised: In general, these frameworks can be unified under the idea of an automatic curriculum that automatically generates an endless procession of better performing agents by exerting selection pressure among many self-optimizing agents.

Original: Hierarchical MARL and Communication.
Revised: Hierarchical MARL and Communication.

Original: Hierarchical reinforcement learning (HRL) has been extensively studied to address the sparse reward problem and to facilitate transfer learning.
Revised: Hierarchical reinforcement learning (HRL) has been extensively studied to address the sparse reward problem and to facilitate transfer learning.

Original: Singleagent HRL focuses on learning the temporal decomposition of tasks, either by learning subgoals (Nachum et al., 2018b;a; Sukhbaatar et al., 2018; Nair & Finn, 2019; Wang et al., 2021) or by discovering reusable skills (Daniel et al., 2012; Gregor et al., 2016; Shankar & Gupta, 2020; Sharma et al., 2020).
Revised: Singleagent HRL focuses on learning the temporal decomposition of tasks, either by learning subgoals (Nachum et al., 2018b;a; Sukhbaatar et al., 2018; Nair & Finn, 2019; Wang et al., 2021) or by discovering reusable skills (Daniel et al., 2012; Gregor et al., 2016; Shankar & Gupta, 2020; Sharma et al., 2020).

Original: Recent works about hierarchical MARL have been discussed in the Introduction.
Revised: Recent works about hierarchical MARL have been discussed in the Introduction.

Original: In multi-agent settings, communication has demonstrated success in multi-agent cooperation (Foerster et al., 2016; Das et al., 2019;
Revised: In multi-agent settings, communication has demonstrated success in multi-agent cooperation (Foerster et al., 2016; Das et al., 2019;

Original: Sukhbaatar et al., 2016; Singh et al., 2018; Jiang & Lu, 2018; Kim et al., 2019; Wang et al., 2020a).
Revised: Sukhbaatar et al., 2016; Singh et al., 2018; Jiang & Lu, 2018; Kim et al., 2019; Wang et al., 2020a).

Original: However, existing approaches that extend HRL to multi-agent systems or utilize communication are limited to a fixed number of agents and are hard to transfer with different number of agents.
Revised: However, existing approaches that extend HRL to multi-agent systems or utilize communication are limited to a fixed number of agents and are hard to transfer with different number of agents.

Original: Google Research Football (Kurach et al., 2019).
Revised: Google Research Football (Kurach et al., 2019).

Original: Recent works attempt to tackle multi-agent scenarios in GRF by using a containerized learning framework (Wu et al., 2021), learning from demonstration (Huang et al., 2021), individuality (Jiang & Lu, 2021), and diversity (Li et al., 2021).
Revised: Recent works attempt to tackle multi-agent scenarios in GRF by using a containerized learning framework (Wu et al., 2021), learning from demonstration (Huang et al., 2021), individuality (Jiang & Lu, 2021), and diversity (Li et al., 2021).

Original: However, they mainly focus on single-agent control, or train relatively easy academy tasks in GRF, or use offline expert data to train agents.
Revised: However, they mainly focus on single-agent control, or train relatively easy academy tasks in GRF, or use offline expert data to train agents.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We consider several tasks in two environments, Simple-Spread and Push-Ball in the Multi-agent
Revised: We consider several tasks in two environments, Simple-Spread and Push-Ball in the Multi-agent

Original: Particle-world Environment (MPE) (Lowe et al., 2017), and the challenging 5vs5 task of GRF (Kurach et al., 2019), to further demonstrate the performance of our approach.
Revised: Particle-world Environment (MPE) (Lowe et al., 2017), and the challenging 5vs5 task of GRF (Kurach et al., 2019), to further demonstrate the performance of our approach.

Original: We aim to answer the following three research questions.
Revised: We aim to answer the following three research questions.

Original: Q1 :
Revised: Q1 :

Original: Is curriculum learning needed in the complex large-scale MARL problem?
Revised: Is curriculum learning needed in the complex large-scale MARL problem?

Original: (Sec. 5.2) Q2 :
Revised: (Sec. 5.2) Q2 :

Original: Can our SPC outperform previous curriculumbased MARL methods?
Revised: Can our SPC outperform previous curriculumbased MARL methods?

Original: If so, which components in SPC contributes the most to performance gains?
Revised: If so, which components in SPC contributes the most to performance gains?

Original: (Sec. 5.3) Q3 :
Revised: (Sec. 5.3) Q3 :

Original: Can SPC learn a good curriculum for the student?
Revised: Can SPC learn a good curriculum for the student?

Original: (Sec. 5.4)
Revised: (Sec. 5.4)

Original: 5. E NVIRONMENTS , B ASELINES AND M ETRIC
Revised: 5. E NVIRONMENTS , B ASELINES AND M ETRIC

Original: Environments.
Revised: Environments.

Original: In the GRF 5vs5 scenario, we need to control 4 agents (except the goalkeeper) to compete with the opponent built-in AI.
Revised: In the GRF 5vs5 scenario, we need to control 4 agents (except the goalkeeper) to compete with the opponent built-in AI.

Original: Each agent would observe a compact encoding, which consists of a 115-dimensional vector summarizing many aspects of the game, such as player coordinates, ball possession and direction, active player, and game mode.
Revised: Each agent would observe a compact encoding, which consists of a 115-dimensional vector summarizing many aspects of the game, such as player coordinates, ball possession and direction, active player, and game mode.

Original: The action set available to an individual agent consists of 19 discrete actions such as idle, movement, passing, shooting, dribbling, or sliding.
Revised: The action set available to an individual agent consists of 19 discrete actions such as idle, movement, passing, shooting, dribbling, or sliding.

Original: The GRF provides two types of reward: scoring and checkpoints, to encourage the agent to move the ball forward and have a successful shot.
Revised: The GRF provides two types of reward: scoring and checkpoints, to encourage the agent to move the ball forward and have a successful shot.

Original: In MPE, we investigate Simple-Spread and Push-Ball (see Fig.
Revised: In MPE, we investigate Simple-Spread and Push-Ball (see Fig.

Original: 2a).
Revised: 2a).

Original: In Simple-Spread, there are n agents that need to cover all n landmarks.
Revised: In Simple-Spread, there are n agents that need to cover all n landmarks.

Original: Agents are penalized for collisions and only receive a positive reward when all the landmarks are covered.
Revised: Agents are penalized for collisions and only receive a positive reward when all the landmarks are covered.

Original: In Push-Ball, there are n agents, n balls, and n landmarks.
Revised: In Push-Ball, there are n agents, n balls, and n landmarks.

Original: The agents need to push the balls to cover every landmark.
Revised: The agents need to push the balls to cover every landmark.

Original: A success reward is given after all the landmarks have been covered.
Revised: A success reward is given after all the landmarks have been covered.

Original: Baselines.
Revised: Baselines.

Original: We evaluate the following approaches as baseline in Table 1:
Revised: We evaluate the following approaches as baseline in Table 1:

Original: [Table]
Revised: [Table]

Original: ( Q
Revised: ( Q

Original: [Table]
Revised: [Table]

Original: We compare MARL algorithms to justify curriculum learning in the complex largescale MARL problem.
Revised: We compare MARL algorithms to justify curriculum learning in the complex largescale MARL problem.

Original: Also, we modify
Revised: Also, we modify

Original: VACL by removing the centralized critic for a fair comparison of the MPE.
Revised: VACL by removing the centralized critic for a fair comparison of the MPE.

Original: Due to the difficulty of the GRF, we include a shooting reward to encourage the student to shoot.
Revised: Due to the difficulty of the GRF, we include a shooting reward to encourage the student to shoot.

Original: Metric.
Revised: Metric.

Original: Even if we use the reward to optimize various algorithms, the mean episode reward in such environments cannot show the performance of the agents.
Revised: Even if we use the reward to optimize various algorithms, the mean episode reward in such environments cannot show the performance of the agents.

Original: Therefore, for GRF scenarios, we plot the win rate and the average goal difference, which is the number of goals scored by the
Revised: Therefore, for GRF scenarios, we plot the win rate and the average goal difference, which is the number of goals scored by the

Original: MARL agents minus the number of goals scored by the other team.
Revised: MARL agents minus the number of goals scored by the other team.

Original: The experiments are carried out on 30 nodes, one of which has a 128-core CPU and 4 A100 GPUs.
Revised: The experiments are carried out on 30 nodes, one of which has a 128-core CPU and 4 A100 GPUs.

Original: Each experiment trial is repeated over 5 seeds and runs for 1-2 days.
Revised: Each experiment trial is repeated over 5 seeds and runs for 1-2 days.

Original: 5. T HE N ECESSITY OF C URRICULUM L EARNING
Revised: 5. T HE N ECESSITY OF C URRICULUM L EARNING

Original: First, we describe experiments using MPE.
Revised: First, we describe experiments using MPE.

Original: In contrast to the fully observable setting and the centralized critic in VACL, we consider individual PPO in partially observable environments with default rewards.
Revised: In contrast to the fully observable setting and the centralized critic in VACL, we consider individual PPO in partially observable environments with default rewards.

Original: We randomly pick a starting state, and the episode ends after a fixed number of maximum steps.
Revised: We randomly pick a starting state, and the episode ends after a fixed number of maximum steps.

Original: To be specific, the task space consists of n agents, where n ∈ { 2 , 4 , 8 , 16 } .
Revised: To be specific, the task space consists of n agents, where n ∈ { 2 , 4 , 8 , 16 } .

Original: We set the maximum allowed steps to 25.
Revised: We set the maximum allowed steps to 25.

Original: All evaluations are performed on the target task, where n = 16 .
Revised: All evaluations are performed on the target task, where n = 16 .

Original: IPPO is trained and evaluated directly on the target task.
Revised: IPPO is trained and evaluated directly on the target task.

Original: In Fig.
Revised: In Fig.

Original: 3, we can see that IPPO performs
Revised: 3, we can see that IPPO performs

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: nearly VACL.
Revised: nearly VACL.

Original: SPC achieves a higher coverage rate than the baseline methods, but the improvement is not significant.
Revised: SPC achieves a higher coverage rate than the baseline methods, but the improvement is not significant.

Original: Furthermore, we experimentally investigate the probability variation of different population sizes in Fig.
Revised: Furthermore, we experimentally investigate the probability variation of different population sizes in Fig.

Original: We observe that the curriculum afforded by SPC is approaching the target task.
Revised: We observe that the curriculum afforded by SPC is approaching the target task.

Original: The results illustrate that in a simple environment where the student can directly learn to complete the task, there is no need to apply curriculum learning.
Revised: The results illustrate that in a simple environment where the student can directly learn to complete the task, there is no need to apply curriculum learning.

Original: Then we show the performance comparison with the baselines in GRF.
Revised: Then we show the performance comparison with the baselines in GRF.

Original: We also run CDS (Li et al., 2021) and CMARL (Wu et al., 2021), however, we did not include their performances, since the goal difference reported in CMARL (Wu et al., 2021) is relatively low compared to our method.
Revised: We also run CDS (Li et al., 2021) and CMARL (Wu et al., 2021), however, we did not include their performances, since the goal difference reported in CMARL (Wu et al., 2021) is relatively low compared to our method.

Original: In Fig.
Revised: In Fig.

Original: 5a , we can see that without the curriculum learning scheme, QMix and IPPO cannot perform well in the 5vs5 scenario.
Revised: 5a , we can see that without the curriculum learning scheme, QMix and IPPO cannot perform well in the 5vs5 scenario.

Original: However, IPPO is slightly better than QMix in the scope of MARL algorithms in this scenario.
Revised: However, IPPO is slightly better than QMix in the scope of MARL algorithms in this scenario.

Original: In Fig.
Revised: In Fig.

Original: 5b, we omit the lines of QMix since the mean score is low, affecting the presentation of the figure.
Revised: 5b, we omit the lines of QMix since the mean score is low, affecting the presentation of the figure.

Original: The reason could be that QMix is an off-policy MARL algorithm, which would rely heavily on the replay buffer.
Revised: The reason could be that QMix is an off-policy MARL algorithm, which would rely heavily on the replay buffer.

Original: However, in such sparse reward scenarios, the replay buffer has much less efficient samples for QMix to learn.
Revised: However, in such sparse reward scenarios, the replay buffer has much less efficient samples for QMix to learn.

Original: For example, the replay buffer would contain tons of zero-score samples, leading to a non-promising performance.
Revised: For example, the replay buffer would contain tons of zero-score samples, leading to a non-promising performance.

Original: Meanwhile, IPPO with a shared actor and critic, an on-policy algorithm, would utilize the samples more efficiently.
Revised: Meanwhile, IPPO with a shared actor and critic, an on-policy algorithm, would utilize the samples more efficiently.

Original: Therefore, curriculum learning is a promising solution to the complex large-scale MARL problem.
Revised: Therefore, curriculum learning is a promising solution to the complex large-scale MARL problem.

Original: During our experiments, we found that IPPO or shared parameter PPO can easily achieve good performance in most academic scenarios in GRF.
Revised: During our experiments, we found that IPPO or shared parameter PPO can easily achieve good performance in most academic scenarios in GRF.

Original: However, 5vs5 is an obstacle for agents to handle more complex scenarios.
Revised: However, 5vs5 is an obstacle for agents to handle more complex scenarios.

Original: Due to the limitation of computational resources, we tested SPC in the 11vs11 scenario.
Revised: Due to the limitation of computational resources, we tested SPC in the 11vs11 scenario.

Original: The result can be seen in the Appendix C.
Revised: The result can be seen in the Appendix C.

Original: 5. P ERFORMANCE AND A BLATION S TUDY
Revised: 5. P ERFORMANCE AND A BLATION S TUDY

Original: In the experiments on MPE, In both environments, SPC performs better than VACL.
Revised: In the experiments on MPE, In both environments, SPC performs better than VACL.

Original: Instead of training with continuous relaxation of the categorical distribution of population size in VACL, our bandit teacher achieves a higher success rate at test time, since the population size is a discrete variable in nature.
Revised: Instead of training with continuous relaxation of the categorical distribution of population size in VACL, our bandit teacher achieves a higher success rate at test time, since the population size is a discrete variable in nature.

Original: Also, in Fig.
Revised: Also, in Fig.

Original: 4, we observe that the curriculum provided by SPC is effective in exploring the task space as agents become increasingly competent.
Revised: 4, we observe that the curriculum provided by SPC is effective in exploring the task space as agents become increasingly competent.

Original: In the experiments on GRF, we do not include VACL in our baselines in the GRF, since the implementation in the source code of VACL is heavily based on prior knowledge of specific scenarios, such as the threshold to divide the learning process.
Revised: In the experiments on GRF, we do not include VACL in our baselines in the GRF, since the implementation in the source code of VACL is heavily based on prior knowledge of specific scenarios, such as the threshold to divide the learning process.

Original: We can see that SPC has higher win rate and goal difference than IPPO with uniform task sampling in the 5vs5 football competition.
Revised: We can see that SPC has higher win rate and goal difference than IPPO with uniform task sampling in the 5vs5 football competition.

Original: The experiments on MPE and GRF show that when the teacher is rewarded by the student’s performance,
Revised: The experiments on MPE and GRF show that when the teacher is rewarded by the student’s performance,

Original: [Figure]
Revised: [Figure]

Original: the bandit-based teacher can exploit the student learning stage and give the suitable training tasks to the student.
Revised: the bandit-based teacher can exploit the student learning stage and give the suitable training tasks to the student.

Original: For ablation study, we replace our contextual multi-armed bandit teacher with uniform task sampling and remove the hierarchical part in the student framework.
Revised: For ablation study, we replace our contextual multi-armed bandit teacher with uniform task sampling and remove the hierarchical part in the student framework.

Original: As shown in Figs.
Revised: As shown in Figs.

Original: 5a, 5b, we can clearly see that SPC can achieve a higher win rate and a greater score difference than SPC with uniform and SPC w/o.
Revised: 5a, 5b, we can clearly see that SPC can achieve a higher win rate and a greater score difference than SPC with uniform and SPC w/o.

Original: HRL.
Revised: HRL.

Original: Also, SPC with uniform task sampling outperforms IPPO with uniform task sampling.
Revised: Also, SPC with uniform task sampling outperforms IPPO with uniform task sampling.

Original: The difference between these two methods is only the introduction of HRL.
Revised: The difference between these two methods is only the introduction of HRL.

Original: It shows the contribution of HRL in the 5vs5 football competition.
Revised: It shows the contribution of HRL in the 5vs5 football competition.

Original: When removing HRL and contextual multi-armed bandit, the performance degradation w.r.t.
Revised: When removing HRL and contextual multi-armed bandit, the performance degradation w.r.t.

Original: SPC are similar.
Revised: SPC are similar.

Original: It shows that HRL and the contextual multi-armed bandit seem to contribute equally.
Revised: It shows that HRL and the contextual multi-armed bandit seem to contribute equally.

Original: This can again justify the need for a curriculum learning scheme.
Revised: This can again justify the need for a curriculum learning scheme.

Original: However, we can see that SPC w. uniform has a larger variance in performance than SPC w/o.
Revised: However, we can see that SPC w. uniform has a larger variance in performance than SPC w/o.

Original: HRL.
Revised: HRL.

Original: It means that uniform sampling might introduce more undesired tasks for student training.
Revised: It means that uniform sampling might introduce more undesired tasks for student training.

Original: 5. V ISUALIZATION OF L EARNED C URRICULUM
Revised: 5. V ISUALIZATION OF L EARNED C URRICULUM

Original: We visualize the distribution of task sampling of SPC during training based on a selected trial as shown in Fig.
Revised: We visualize the distribution of task sampling of SPC during training based on a selected trial as shown in Fig.

Original: 6a.
Revised: 6a.

Original: An interesting observation is that the task probability seems nearly uniform.
Revised: An interesting observation is that the task probability seems nearly uniform.

Original: We interpret this into an anti-forgetting mechanism.
Revised: We interpret this into an anti-forgetting mechanism.

Original: We can see that at the beginning of training, the task probability seems to be near-uniform, since the teacher should explore the task space and try to keep track of the student’s learning status.
Revised: We can see that at the beginning of training, the task probability seems to be near-uniform, since the teacher should explore the task space and try to keep track of the student’s learning status.

Original: During training, the probabilities vary over time steps.
Revised: During training, the probabilities vary over time steps.

Original: For example, at about 80-100 million timesteps, we can see a sudden drop in academy empty goal close and academy 3 vs 1 with keeper , since the student almost handles the skills learned in such scenarios.
Revised: For example, at about 80-100 million timesteps, we can see a sudden drop in academy empty goal close and academy 3 vs 1 with keeper , since the student almost handles the skills learned in such scenarios.

Original: However, when training is continued, we can still observe that agents are trained on these tasks more frequently.
Revised: However, when training is continued, we can still observe that agents are trained on these tasks more frequently.

Original: We also visualize the distribution of contexts in Fig.
Revised: We also visualize the distribution of contexts in Fig.

Original: 6b using t-SNE (Van der
Revised: 6b using t-SNE (Van der

Original: Maaten & Hinton, 2008).
Revised: Maaten & Hinton, 2008).

Original: The contexts are collected and stored in a buffer.
Revised: The contexts are collected and stored in a buffer.

Original: We divide the contexts into four classes according to the index.
Revised: We divide the contexts into four classes according to the index.

Original: We can clearly see different parts in terms of different contexts about the final student policy representation.
Revised: We can clearly see different parts in terms of different contexts about the final student policy representation.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Conclusion.
Revised: Conclusion.

Original: We introduce a novel ACL algorithm, Skilled Population Curriculum (SPC).
Revised: We introduce a novel ACL algorithm, Skilled Population Curriculum (SPC).

Original: SPC addresses the scalability and sparse reward issue in current multi-agent system and learns complex behaviors from scratch.
Revised: SPC addresses the scalability and sparse reward issue in current multi-agent system and learns complex behaviors from scratch.

Original: Specifically, to handle the varying number of agents, we incorporate a population-invariant multi-agent communication framework and exploit a hierarchical scheme for each agent to learn skills to deal with sparse rewards.
Revised: Specifically, to handle the varying number of agents, we incorporate a population-invariant multi-agent communication framework and exploit a hierarchical scheme for each agent to learn skills to deal with sparse rewards.

Original: Moreover, to mitigate the non-stationarity, we model the teacher as a contextual bandit, where the context is represented by the student’s policy representation.
Revised: Moreover, to mitigate the non-stationarity, we model the teacher as a contextual bandit, where the context is represented by the student’s policy representation.

Original: Empirical results show that our method achieves state-of-the-art performance on several tasks in the multi-particle environment and the challenging 5vs5 competition in GRF.
Revised: Empirical results show that our method achieves state-of-the-art performance on several tasks in the multi-particle environment and the challenging 5vs5 competition in GRF.

Original: Limitations.
Revised: Limitations.

Original: We acknowledge some limitations of our algorithm.
Revised: We acknowledge some limitations of our algorithm.

Original: First, since the objective of SPC is to solve difficulty tasks, SPC is over-designed when applied to some simple tasks.
Revised: First, since the objective of SPC is to solve difficulty tasks, SPC is over-designed when applied to some simple tasks.

Original: Second, the set of subtasks is still closely related to the environment.
Revised: Second, the set of subtasks is still closely related to the environment.

Original: Lastly, it remains unclear how much the number of agents can affect the dynamics of the environment since we treat it as a black-box optimization
Revised: Lastly, it remains unclear how much the number of agents can affect the dynamics of the environment since we treat it as a black-box optimization.

Original: B P ROOF OF T HEOREM 3. 
Revised: B P ROOF OF T HEOREM 3.

Original: Theorem 3.4.
Revised: Theorem 3.4.

Original: Consider the Lipschitz contextual bandit problem with contexts in [0 , 1] .
Revised: Consider the Lipschitz contextual bandit problem with contexts in [0 , 1] .

Original: The Alg.yields regret E [ R ( T )]
Revised: The Alg.yields regret E [ R ( T )]

Original: = O (cid:0) T 2 / 3 ( LK ln T ) 1 / 3 (cid:1) .
Revised: = O (cid:0) T 2 / 3 ( LK ln T ) 1 / 3 (cid:1) .

Original: Proof.
Revised: Proof.

Original: Let S m be the ϵ -uniform mesh on [0 , 1] , that is, the set of all points in [0 , 1] that are integer multiples of ϵ .
Revised: Let S m be the ϵ -uniform mesh on [0 , 1] , that is, the set of all points in [0 , 1] that are integer multiples of ϵ .

Original: We take ϵ = 1 / ( d − 1) where the integer d is the number of points in S m , which will be adjusted later in the analysis.
Revised: We take ϵ = 1 / ( d − 1) where the integer d is the number of points in S m , which will be adjusted later in the analysis.

Original: We apply Alg. 2 to the context space S m .
Revised: We apply Alg. 2 to the context space S m .

Original: Let f S m ( x ) be a mapping from context x to the closest point in S m :
Revised: Let f S m ( x ) be a mapping from context x to the closest point in S m :

Original: [Equation]
Revised: [Equation]

Original: In each round t , we replace the context x t with f S m ( x t ) and call Exp3 S .
Revised: In each round t , we replace the context x t with f S m ( x t ) and call Exp3 S .

Original: The regret bound will have two components: the regret bound for Exp3 S and (a suitable notion of) the discretization error.
Revised: The regret bound will have two components: the regret bound for Exp3 S and (a suitable notion of) the discretization error.

Original: Formally, let us define the “discretized best response” π ∗ S m : X → Φ : π ∗ S m ( x ) =
Revised: Formally, let us define the “discretized best response” π ∗ S m : X → Φ : π ∗ S m ( x ) =

Original: π ∗ ( f S m ( x )) for each context x ∈ X .
Revised: π ∗ ( f S m ( x )) for each context x ∈ X .

Original: We define the total reward of an algorithm Alg is Reward(Alg) =
Revised: We define the total reward of an algorithm Alg is Reward(Alg) =

Original: (cid:80) Tt =1 r t .
Revised: (cid:80) Tt =1 r t .

Original: Then the regret of Exp3 S and the discretization error are defined as:
Revised: Then the regret of Exp3 S and the discretization error are defined as:

Original: [Equation]
Revised: [Equation]

Original: DE
Revised: DE

Original: [Equation]
Revised: [Equation]

Original: It follows that regret is the sum R ( T )
Revised: It follows that regret is the sum R ( T )

Original: =
Revised: =

Original: R S ( T ) + DE( S ) .
Revised: R S ( T ) + DE( S ) .

Original: We have E [ R S ( T )]
Revised: We have E [ R S ( T )]

Original: = O ( √ TK log K ) from Lemma 3.3, so it remains to upper bound the discretization error and adjust the discretization step ϵ .
Revised: = O ( √ TK log K ) from Lemma 3.3, so it remains to upper bound the discretization error and adjust the discretization step ϵ .

Original: For each round t and the respective context x = x t , r ( π ∗ S ( x ) | f S ( x )) ≥ r ( π ∗ ( x ) | f S ( x )) ≥ r ( π ∗ ( x ) | x ) − ϵL .
Revised: For each round t and the respective context x = x t , r ( π ∗ S ( x ) | f S ( x )) ≥ r ( π ∗ ( x ) | f S ( x )) ≥ r ( π ∗ ( x ) | x ) − ϵL .

Original: The first inequality is determined by the optimality of π ∗ S and the second is determined by Lipschitzness.
Revised: The first inequality is determined by the optimality of π ∗ S and the second is determined by Lipschitzness.

Original: Summing this up over all rounds t , we obtain E [Reward ( π ∗ S )] ≥
Revised: Summing this up over all rounds t , we obtain E [Reward ( π ∗ S )] ≥

Original: Reward [ π ∗ ] − ϵLT .
Revised: Reward [ π ∗ ] − ϵLT .

Original: Thus, the regret is that
Revised: Thus, the regret is that

Original: [Equation]
Revised: [Equation]

Original: For the last inequality, we choose ϵ =
Revised: For the last inequality, we choose ϵ =

Original: ( K log T TL 2 ) 1 / 3 .
Revised: ( K log T TL 2 ) 1 / 3 .

Original: C 11 VS 11 F ULL G AME ON GRF
Revised: C 11 VS 11 F ULL G AME ON GRF

Original: We further conduct experiments on the 11vs11 scenario of GRF.
Revised: We further conduct experiments on the 11vs11 scenario of GRF.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 7, SPC achieves about 50% win rate after training with 200 million timesteps.
Revised: 7, SPC achieves about 50% win rate after training with 200 million timesteps.

Original: [Figure]
Revised: [Figure]

Original: D Q UALITATIVELY ANALYSIS ABOUT LOW - LEVEL SKILLS
Revised: D Q UALITATIVELY ANALYSIS ABOUT LOW - LEVEL SKILLS

Original: We show some statistics, for example, shooting, passing and running, in GRF in terms of different fixed low-level policies (defined by a set of fixed high-level actions).
Revised: We show some statistics, for example, shooting, passing and running, in GRF in terms of different fixed low-level policies (defined by a set of fixed high-level actions).

Original: We evaluate the statistics by only fixing one agent’s high-level action, maintaining other agents with trained SPC and testingtimes.
Revised: We evaluate the statistics by only fixing one agent’s high-level action, maintaining other agents with trained SPC and testingtimes.

Original: We show the results in Table 2.
Revised: We show the results in Table 2.

Original: [Table]
Revised: [Table]

Original: E C ORNER -5 ON GRF
Revised: E C ORNER -5 ON GRF

Original: To further illustrate the effectiveness of the SPC teacher module, we conduct experiments on the corner-5 scenario on GRF, where the target task is to control five of the eleven players to obtain a goal in the GRF academy corner scenario.
Revised: To further illustrate the effectiveness of the SPC teacher module, we conduct experiments on the corner-5 scenario on GRF, where the target task is to control five of the eleven players to obtain a goal in the GRF academy corner scenario.

Original: The experiments are designed to determine whether or not the contextual bandit in SPC outperforms alternative curriculum learning methods to schedule the number of agents in training.
Revised: The experiments are designed to determine whether or not the contextual bandit in SPC outperforms alternative curriculum learning methods to schedule the number of agents in training.

Original: We compare SPC teacher against non-curriculum training (None), uniform task sampling (Uniform), ALP-GMM, and VACL.
Revised: We compare SPC teacher against non-curriculum training (None), uniform task sampling (Uniform), ALP-GMM, and VACL.

Original: The training task space consists of n agents, where n ∈ { 1 , 3 , 5 } .
Revised: The training task space consists of n agents, where n ∈ { 1 , 3 , 5 } .

Original: All teachers have the same base architecture without transformer architecture and hierarchical framework.
Revised: All teachers have the same base architecture without transformer architecture and hierarchical framework.

Original: We also investigate the effect of RNN-based contexts (see
Revised: We also investigate the effect of RNN-based contexts (see

Original: Contextual Bandit and Bandit).
Revised: Contextual Bandit and Bandit).

Original: Fig.
Revised: Fig.

Original: 8 shows the benefit of SPC contextual bandit over other ACL methods after training with one million timesteps.
Revised: 8 shows the benefit of SPC contextual bandit over other ACL methods after training with one million timesteps.

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: F I MPLEMENTATION D ETAILS
Revised: F I MPLEMENTATION D ETAILS

Original: Here we describe the SPC framework.
Revised: Here we describe the SPC framework.

Original: We use the open-sourced Ray RLlib implementation of Proximal Policy Optimization (PPO), which scales out using multiple workers for experience collection.
Revised: We use the open-sourced Ray RLlib implementation of Proximal Policy Optimization (PPO), which scales out using multiple workers for experience collection.

Original: This allows us to use a large amount of rollouts from parallel workers during training to ameliorate high variance and aid exploration.
Revised: This allows us to use a large amount of rollouts from parallel workers during training to ameliorate high variance and aid exploration.

Original: We do multiple rollouts in parallel with distributed workers and use parameter sharing for each agent.
Revised: We do multiple rollouts in parallel with distributed workers and use parameter sharing for each agent.

Original: The trainer broadcasts new weights to the workers after their synchronous sampling.
Revised: The trainer broadcasts new weights to the workers after their synchronous sampling.

Original: We now turn our attention to environment-specific settings.
Revised: We now turn our attention to environment-specific settings.

Original: F. G OOGLE R ESEARCH F OOTBALL
Revised: F. G OOGLE R ESEARCH F OOTBALL

Original: We set five tasks for training the 5vs5 scenario.
Revised: We set five tasks for training the 5vs5 scenario.

Original: They are academy empty goal close , academy pass and shoot with keeper , 3 vs 3 , academy 3 vs 1 with keeper , 5 vs 5 .
Revised: They are academy empty goal close , academy pass and shoot with keeper , 3 vs 3 , academy 3 vs 1 with keeper , 5 vs 5 .

Original: In all scenarios, we do not control our team’s goalkeeper.
Revised: In all scenarios, we do not control our team’s goalkeeper.

Original: In the academy empty goal close , one agent need to move forward and shoot with an empty goal.
Revised: In the academy empty goal close , one agent need to move forward and shoot with an empty goal.

Original: In academy pass and shoot with keeper and 3 vs 3 , two agents are controlled to play against a goalkeeper and 3 players respectively.
Revised: In academy pass and shoot with keeper and 3 vs 3 , two agents are controlled to play against a goalkeeper and 3 players respectively.

Original: In academy 3 vs 1 with keeper , three agents are controlled to play against a center-back and a goalkeeper.
Revised: In academy 3 vs 1 with keeper , three agents are controlled to play against a center-back and a goalkeeper.

Original: In 5 vs 5 , 4 agents are controlled to play against 5 players.
Revised: In 5 vs 5 , 4 agents are controlled to play against 5 players.

Original: Without loss of generality, we initialize all player with fixed positions and roles as center midfielders.
Revised: Without loss of generality, we initialize all player with fixed positions and roles as center midfielders.

Original: We use both MLP and self-attention mechanism for the high-level policy, and use MLP for the low-level policy.
Revised: We use both MLP and self-attention mechanism for the high-level policy, and use MLP for the low-level policy.

Original: For high-level policy, the input is first projected to an embedding using 2 hidden layers with 256 units each and ReLU activation, which is then fed into multi-head self-attention (8 heads, 64 units each).
Revised: For high-level policy, the input is first projected to an embedding using 2 hidden layers with 256 units each and ReLU activation, which is then fed into multi-head self-attention (8 heads, 64 units each).

Original: The output is then projected to the actions and values using another fully connected layer with 256 units.
Revised: The output is then projected to the actions and values using another fully connected layer with 256 units.

Original: For low-level policy, we use MLP with 2 hidden layers withunits each, i.e., the default configuration of policy network in RLlib.
Revised: For low-level policy, we use MLP with 2 hidden layers withunits each, i.e., the default configuration of policy network in RLlib.

Original: F. MPE
Revised: F. MPE

Original: In this environment, agents must cooperate through physical actions to reach a set of landmarks.
Revised: In this environment, agents must cooperate through physical actions to reach a set of landmarks.

Original: Agents observe the relative positions of other agents and landmarks, and are collectively rewarded based on the proximity of any agent to each landmark.
Revised: Agents observe the relative positions of other agents and landmarks, and are collectively rewarded based on the proximity of any agent to each landmark.

Original: In other words, the agents have to ‘cover’ all of the landmarks.
Revised: In other words, the agents have to ‘cover’ all of the landmarks.

Original: Further, the agents occupy significant physical space and are penalized when colliding with each other.
Revised: Further, the agents occupy significant physical space and are penalized when colliding with each other.

Original: The agents need to infer the landmark to cover and move there while avoiding other agents.
Revised: The agents need to infer the landmark to cover and move there while avoiding other agents.

Original: The hyper-parameters of SPC in MPE are shown in Table 4.
Revised: The hyper-parameters of SPC in MPE are shown in Table 4.

Original: In MPE, hyper-parameters such as rollout fragment length, training batch size and SGD minibatch size are adjusted according to horizon of the scenarios so that policy are updated after episodes are done.
Revised: In MPE, hyper-parameters such as rollout fragment length, training batch size and SGD minibatch size are adjusted according to horizon of the scenarios so that policy are updated after episodes are done.

Original: We use the same network as in GRF, but with 128 units for all MLP hidden layers.
Revised: We use the same network as in GRF, but with 128 units for all MLP hidden layers.

Original: Other omitted hyper-parameters follow the default configuration in RLlib PPO implementation.
Revised: Other omitted hyper-parameters follow the default configuration in RLlib PPO implementation.