Original: M ASAI : Multi-agent Summative Assessment Improvement for Unsupervised Environment DesignAnonymous Authors
Revised: M ASAI : Multi-agent Summative Assessment Improvement for Unsupervised Environment Design

Original: Abstract
Revised: Abstract

Original: Reinforcement Learning agents require a distribution of environments for their policy to be trained on.
Revised: Reinforcement Learning agents require a distribution of environments for their policy to be trained on.

Original: The method or process of deﬁning these environments directly impacts robustness and generalization of the learned agent policies.
Revised: The method or process of deﬁning these environments directly impacts robustness and generalization of the learned agent policies.

Original: In single agent reinforcement learning, this problem is often solved by domain randomization, or randomizing the environment and tasks within the scope of the desired operating domain of the agent.
Revised: In single agent reinforcement learning, this problem is often solved by domain randomization, or randomizing the environment and tasks within the scope of the desired operating domain of the agent.

Original: The challenge here is to generate both structured and solvable environments that guide the agent’s learning process.
Revised: The challenge here is to generate both structured and solvable environments that guide the agent’s learning process.

Original: Most recently, works have sought to produce the environments under the Unsupervised Environment Design (UED) formulation.
Revised: Most recently, works have sought to produce the environments under the Unsupervised Environment Design (UED) formulation.

Original: However, these methods lead to a proliferation of adversarial agents to train one agent for a single agent problem in a discretized task domain.
Revised: However, these methods lead to a proliferation of adversarial agents to train one agent for a single agent problem in a discretized task domain.

Original: In this work, we aim to automatically generate environments that are solvable and challenging for the continuous multi-agent setting.
Revised: In this work, we aim to automatically generate environments that are solvable and challenging for the continuous multi-agent setting.

Original: We base our solution on the Teacher-Student relationship with parameter sharing Students where we reimagine the Teacher as an environment generator for UED.
Revised: We base our solution on the Teacher-Student relationship with parameter sharing Students where we re-imagine the Teacher as an environment generator for UED.

Original: Our approach uses one environment generator agent ( Teacher ) for any number of learning agents ( Students ).
Revised: Our approach uses one environment generator agent ( Teacher ) for any number of learning agents ( Students ).

Original: We qualitatively and quantitatively demonstrate that, in terms of multi-agent ( ≥ 8 agents) navigation and steering, Students trained by our approach outperform agents using heuristic search, as well as agents trained by domain randomization.
Revised: We qualitatively and quantitatively demonstrate that, in terms of multi-agent ( ≥agents) navigation and steering, Students trained by our approach outperform agents using heuristic search, as well as agents trained by domain randomization.

Original: Our code is available at LinkWithheld .
Revised: Our code is available at https:

Original: 1. Introduction
Revised: 1. Introduction

Original: Multi-agent navigation is a well-established but difﬁcult problem.
Revised: Multi-agent navigation is a well-established but difﬁcult problem.

Original: Recent approaches had adapted Reinforcement
Revised: Recent approaches had adapted Reinforcement

Original: Learning (RL) to the Multi-agent domain, or Multi-agent
Revised: Learning (RL) to the Multi-agent domain, or Multi-agent

Original: Reinforcement Learning (MARL).
Revised: Reinforcement Learning (MARL).

Original: However, successful navigation learning requires a high degree of generalization to potentially very different environments, which requires designing distribution of tasks and environments that can be used to train and evaluate policies.
Revised: However, successful navigation learning requires a high degree of generalization to potentially very different environments, which requires designing distribution of tasks and environments that can be used to train and evaluate policies.

Original: Nevertheless, manually designing such an effective and appropriate distribution of environments is time-consuming and challenging, as the real-world scenarios are usually complicated and enumerated all of the representative and edge scenes are impractical.
Revised: Nevertheless, manually designing such an effective and appropriate distribution of environments is time-consuming and challenging, as the real-world scenarios are usually complicated and enumerated all of the representative and edge scenes are impractical.

Original: We, therefore, aim to automate this process, under the Unsupervised Environment Design (UED) (Dennis et al., 2021) framework.
Revised: We, therefore, aim to automate this process, under the Unsupervised Environment Design (UED) (Dennis et al., 2021) framework.

Original: UED is the problem of taking the underspeciﬁed environment and a policy, and producing a useful distribution of fully speciﬁed environments in which that policy can be trained, where underspeciﬁed environment means the environment that has free parameters ( e.g. , in navigation, the places for the obstacles, agents, and their goals) which control its feature and behavior (Dennis et al., 2021).
Revised: UED is the problem of taking the underspeciﬁed environment and a policy, and producing a useful distribution of fully speciﬁed environments in which that policy can be trained, where underspeciﬁed environment means the environment that has free parameters ( e.g. , in navigation, the places for the obstacles, agents, and their goals) which control its feature and behavior (Dennis et al., 2021).

Original: After training a policy in the distribution of environments generated by UED, an updated policy is obtained and then use
Revised: After training a policy in the distribution of environments generated by UED, an updated policy is obtained and then use

Original: UED to generate more diverse environments such that the updated policy can continue to be trained.
Revised: UED to generate more diverse environments such that the updated policy can continue to be trained.

Original: Hence, the UED formulation can be used to produce capable policies in increasingly complex and difﬁcult environments based on the abilities of the current policy.
Revised: Hence, the UED formulation can be used to produce capable policies in increasingly complex and difﬁcult environments based on the abilities of the current policy.

Original: (Dennis et al., 2021)
Revised: (Dennis et al., 2021)

Original: In this work, we proposal a novel semi-adversarial training algorithm, M ASAI , which aims to solve the UED problem and improve generalization for MARL problems in novel continuous environments.
Revised: In this work, we proposal a novel semi-adversarial training algorithm, M ASAI , which aims to solve the UED problem and improve generalization for MARL problems in novel continuous environments.

Original: Our algorithm is based on the
Revised: Our algorithm is based on the

Original: Teacher-Student relationship with a novel reward signal for
Revised: Teacher-Student relationship with a novel reward signal for

Original: UED, and has a simple structure: only two policies 1)
Revised: UED, and has a simple structure: only two policies 1)

Original: the
Revised: the

Original: Student ( i.e. , the policy to solve an environment), and 2) the Teacher ( i.e. , the policy to generate free parameters).
Revised: Student ( i.e. , the policy to solve an environment), and 2) the Teacher ( i.e. , the policy to generate free parameters).

Original: The motivation for our algorithm is based on the intuitive
Revised: The motivation for our algorithm is based on the intuitive

Original: Teacher-Student relationship.
Revised: Teacher-Student relationship.

Original: How do Teachers measure whether Students master a concept or not?
Revised: How do Teachers measure whether Students master a concept or not?

Original: In general, Teachers prepare two questions of similar difﬁculty for Students, one as an example and the other as an exercise.
Revised: In general, Teachers prepare two questions of similar difﬁculty for Students, one as an example and the other as an exercise.

Original: The Environment Generator ( i.e. , Teacher ) in every iteration, will produce two sets of environments, one for L EARNING and one for R EVIEW based on the Student ’s performance in the last iteration.
Revised: The Environment Generator ( i.e. , Teacher ) in every iteration, will produce two sets of environments, one for L EARNING and one for R EVIEW based on the Student ’s performance in the last iteration.

Original: The Student will collect experience in the
Revised: The Student will collect experience in the

Original: L EARNING environment and update its parameters.
Revised: L EARNING environment and update its parameters.

Original: After the parameters are updated, the Student will attempt to solve the R EVIEW environments, without updating its parameters.
Revised: After the parameters are updated, the Student will attempt to solve the R EVIEW environments, without updating its parameters.

Original: Further, we argue in RL problems, there are usually two classes of rewards: Task Completion Reward and Agent Reward.
Revised: Further, we argue in RL problems, there are usually two classes of rewards: Task Completion Reward and Agent Reward.

Original: Task Completion Reward contains the rewards when the agent completes certain tasks ( e.g. , in navigation, positive rewards for the agent arrives at its goal), and Agent
Revised: Task Completion Reward contains the rewards when the agent completes certain tasks ( e.g. , in navigation, positive rewards for the agent arrives at its goal), and Agent

Original: Reward includes the cumulative rewards start from initial states until the terminal states ( e.g. , in navigation, penalties for collisions and positive rewards for moving correctly and reaching intermediate goals).
Revised: Reward includes the cumulative rewards start from initial states until the terminal states ( e.g. , in navigation, penalties for collisions and positive rewards for moving correctly and reaching intermediate goals).

Original: The Teacher uses the difference between the reward of the Student on R EVIEW and
Revised: The Teacher uses the difference between the reward of the Student on R EVIEW and

Original: L EARNING environments as its reward signal, to update its parameters, and it aims to maximize the Task Completion
Revised: L EARNING environments as its reward signal, to update its parameters, and it aims to maximize the Task Completion

Original: Rewards while minimize the Agent Rewards.
Revised: Rewards while minimize the Agent Rewards.

Original: M ASAI requires less parameters than the current state-of-the-art by removing the need for an additional antagonist (Dennis et al., 2021).
Revised: M ASAI requires less parameters than the current state-of-the-art by removing the need for an additional antagonist (Dennis et al., 2021).

Original: We qualitatively demonstrate that the Student agents trained by M ASAI learn and exhibit complex and highly valuable emergent behaviours, and the learned policies also generalize well in unseen difﬁcult steering and navigation scenarios in contrast to the agents trained by classic Domain Randomization (DR) methods.
Revised: We qualitatively demonstrate that the Student agents trained by M ASAI learn and exhibit complex and highly valuable emergent behaviours, and the learned policies also generalize well in unseen difﬁcult steering and navigation scenarios in contrast to the agents trained by classic Domain Randomization (DR) methods.

Original: Further, we quantitatively show that under the same conditions, the Student agents trained by
Revised: Further, we quantitatively show that under the same conditions, the Student agents trained by

Original: M ASAI have a lower number of collisions and complete goals faster than the agents trained by DR in the domain of multi-agent navigation and steering.
Revised: M ASAI have a lower number of collisions and complete goals faster than the agents trained by DR in the domain of multi-agent navigation and steering.

Original: 2. Unsupervised Environment Design
Revised: 2. Unsupervised Environment Design

Original: We brieﬂy summarize the recent formalization of the UED problem in this section (Dennis et al., 2021).
Revised: We brieﬂy summarize the recent formalization of the UED problem in this section (Dennis et al., 2021).

Original: The formal deﬁnition of UED is the problem of using an underspeciﬁed environment to produce a distribution over fully speciﬁed environments, which offers the continued or curriculum learning of a particular task-solving policy.
Revised: The formal deﬁnition of UED is the problem of using an underspeciﬁed environment to produce a distribution over fully speciﬁed environments, which offers the continued or curriculum learning of a particular task-solving policy.

Original: We now detail the meaning and relevant distinction of the fully speciﬁed versus underspecﬁed environments.
Revised: We now detail the meaning and relevant distinction of the fully speciﬁed versus underspecﬁed environments.

Original: Fully speciﬁed environments are modeled with a Partially
Revised: Fully speciﬁed environments are modeled with a Partially

Original: Observable Markov Decision Process (POMDP), represented by a tuple (cid:104) A, O, S, T , I , R , γ (cid:105) where A is a collection of actions, O is a collection of observations, S is a set of states, T is the transition function of S × A → ∆( S ) ,
Revised: Observable Markov Decision Process (POMDP), represented by a tuple (cid:104) A, O, S, T , I , R , γ (cid:105) where A is a collection of actions, O is a collection of observations, S is a set of states, T is the transition function of S × A → ∆( S ) ,

Original: I is the observation function of S → O , R is the reward function of S → R , and γ is the discount factor.
Revised: I is the observation function of S → O , R is the reward function of S → R , and γ is the discount factor.

Original: This contrasts the underspeciﬁed environment which can be modeled as an Underspeciﬁed Partially Observable
Revised: This contrasts the underspeciﬁed environment which can be modeled as an Underspeciﬁed Partially Observable

Original: Markov Decision Process (UPOMDP) represented by a tuple (cid:104) A, O, Θ , S M , T M , I M , R M , γ (cid:105) .
Revised: Markov Decision Process (UPOMDP) represented by a tuple (cid:104) A, O, Θ , S M , T M , I M , R M , γ (cid:105) .

Original: The only difference here between a POMDP and a UPOMDP is that a UPOMDP contains a set Θ representing the free parameters of the environments, and it is these free parameters that can be selected at each time step and integrated into the transition function T M as S ×
Revised: The only difference here between a POMDP and a UPOMDP is that a UPOMDP contains a set Θ representing the free parameters of the environments, and it is these free parameters that can be selected at each time step and integrated into the transition function T M as S ×

Original: A × Θ → ∆( S ) .
Revised: A × Θ → ∆( S ) .

Original: The solution to UED proposed in (Dennis et al., 2021) then is to have an environment policy Λ : Π → ∆(Θ T ) where Π is the set of possible policies and Θ T is the set of possible trajectory of environment parameters.
Revised: The solution to UED proposed in (Dennis et al., 2021) then is to have an environment policy Λ : Π → ∆(Θ T ) where Π is the set of possible policies and Θ T is the set of possible trajectory of environment parameters.

Original: In this formulation, UED aims to generate a distribution of environments that caters to the continued learning of a particular agent policy.
Revised: In this formulation, UED aims to generate a distribution of environments that caters to the continued learning of a particular agent policy.

Original: To solve this, the authors proposed PAIRED (Dennis et al., 2021) to approximate the environment policy.
Revised: To solve this, the authors proposed PAIRED (Dennis et al., 2021) to approximate the environment policy.

Original: PAIRED solves the
Revised: PAIRED solves the

Original: UED problem by introducing three agents the antagonist , protagonist , and environment-generating adversary .
Revised: UED problem by introducing three agents the antagonist , protagonist , and environment-generating adversary .

Original: The antagonist is allied with the environment-generating adversary to guide the adversary from constructing unsolvable environments.
Revised: The antagonist is allied with the environment-generating adversary to guide the adversary from constructing unsolvable environments.

Original: The goal of the environment adversary is to build environments in which the antagonist achieves high reward and the protagonist receives a low reward.
Revised: The goal of the environment adversary is to build environments in which the antagonist achieves high reward and the protagonist receives a low reward.

Original: The protagonist is the agent to be trained for solving the environments.
Revised: The protagonist is the agent to be trained for solving the environments.

Original: 3. Multi-agent Summative Assessment Improvement for Unsupervised Environment Design
Revised: 3. Multi-agent Summative Assessment Improvement for Unsupervised Environment Design

Original: We propose a novel semi-adversarial training algorithm,
Revised: We propose a novel semi-adversarial training algorithm,

Original: M ASAI , which is designed to solve the UED problem and improve generalization, inspired by the intuitive TeacherStudent relationship.
Revised: M ASAI , which is designed to solve the UED problem and improve generalization, inspired by the intuitive TeacherStudent relationship.

Original: How do Teachers measure whether
Revised: How do Teachers measure whether

Original: Students master a concept or not?
Revised: Students master a concept or not?

Original: Generally, the Teacher prepares two questions of similar difﬁculty for the Student, and takes one of them as an example to teach ( i.e., Student will hopefully learn the concept from this example), and then use the other as a quiz to test Student’s understanding.
Revised: Generally, the Teacher prepares two questions of similar difﬁculty for the Student, and takes one of them as an example to teach ( i.e., Student will hopefully learn the concept from this example), and then use the other as a quiz to test Student’s understanding.

Original: M ASAI formulates this relationship for multi-agent training.
Revised: M ASAI formulates this relationship for multi-agent training.

Original: M ASAI is summarized in Algorithm 1 and its semantics is shown in Figure 1.
Revised: M ASAI is summarized in Algorithm 1 and its semantics is shown in Figure 1.

Original: There are two types of trainable agent policies in M ASAI ,
Revised: There are two types of trainable agent policies in M ASAI ,

Original: Teacher agent ( i.e., Unsupervised Environment Generator) and Student agent, where the policy is denoted by the symbol π .
Revised: Teacher agent ( i.e., Unsupervised Environment Generator) and Student agent, where the policy is denoted by the symbol π .

Original: Teacher agent aims to produce solvable and valuable free environment parameters (cid:126)θ , and the Student agent aims to learn speciﬁc tasks, for instance, navigation and steering.
Revised: Teacher agent aims to produce solvable and valuable free environment parameters (cid:126)θ , and the Student agent aims to learn speciﬁc tasks, for instance, navigation and steering.

Original: We denote the space of all environment parameters as Θ , and we assume for a Teacher agent policy the produced free environment parameters (cid:126)θ follows the distribution of Θ T .
Revised: We denote the space of all environment parameters as Θ , and we assume for a Teacher agent policy the produced free environment parameters (cid:126)θ follows the distribution of Θ T .

Original: Moreover, we argue that in many RL formulations, it is possible to create two classes of rewards for the agent; the Task
Revised: Moreover, we argue that in many RL formulations, it is possible to create two classes of rewards for the agent; the Task

Original: Completion reward and the Agent reward.
Revised: Completion reward and the Agent reward.

Original: Task Completion
Revised: Task Completion

Original: Algorithm1 M ASAI
Revised: Algorithm

Original: Denote | S | as the number of Student agent shares the same Student policy π S .
Revised: Denote | S | as the number of Student agent shares the same Student policy π S .

Original: Denote R ( τ ) as an operator that sums undiscounted return of | S | Student agents in a trajectory τ .
Revised: Denote R ( τ ) as an operator that sums undiscounted return of | S | Student agents in a trajectory τ .

Original: Randomly initialize the Student π S and the Teacher π T .
Revised: Randomly initialize the Student π S and the Teacher π T .

Original: repeat
Revised: repeat

Original: Use π T to generate L EARNING environment parameters: θ L ∼ Θ T
Revised: Use π T to generate L EARNING environment parameters: θ L ∼ Θ T

Original: Use π T to generate R EVIEW environment parameters: θ R ∼ Θ T Collect the L EARNING trajectory τ
Revised: Use π T to generate R EVIEW environment parameters: θ R ∼ Θ T Collect the L EARNING trajectory τ

Original: L using π S , and compute Agent Reward R Agent ( τ L ) and Task Completion Reward R Task ( τ L ) .
Revised: L using π S , and compute Agent Reward R Agent ( τ L ) and Task Completion Reward R Task ( τ L ) .

Original: Train policy π S with RL update and discounted return from trajectory τ L .
Revised: Train policy π S with RL update and discounted return from trajectory τ L .

Original: Collect the R EVIEW trajectory τ R using π S , and compute Agent Reward R Agent ( τ R ) and Task Completion
Revised: Collect the R EVIEW trajectory τ R using π S , and compute Agent Reward R Agent ( τ R ) and Task Completion

Original: Reward R Task ( τ R ) .
Revised: Reward R Task ( τ R ) .

Original: Compute G AIN Agent = R Agent ( τ R ) − R Agent ( τ L ) .
Revised: Compute G AIN Agent = R Agent ( τ R ) − R Agent ( τ L ) .

Original: Compute G AIN Task =
Revised: Compute G AIN Task =

Original: R Task ( τ R ) − R Task ( τ L ) .
Revised: R Task ( τ R ) − R Task ( τ L ) .

Original: Train policy π T with RL update and minimize
Revised: Train policy π T with RL update and minimize

Original: G AIN Agent while maximize G AIN Task .
Revised: G AIN Agent while maximize G AIN Task .

Original: until convergence
Revised: until convergence

Original: reward is an instant reward, to signal the agent whether it completes the desirable tasks or not.
Revised: reward is an instant reward, to signal the agent whether it completes the desirable tasks or not.

Original: Agent reward contains the cumulative rewards from the initial states until the terminal states, to guide the agent away from low value behaviours ( e.g., in the multi-agent navigation problem, collisions with other agents and static/dynamic obstacles need to avoided), and encourage the agent toward the preferable actions and emergent behaviour policies ( e.g., in the multiagent navigation problem, reaching the intermediate waypoints toward the ﬁnal target should be incentivized, laminar ﬂows and vortices should emerge in conﬂicting ﬂows).
Revised: Agent reward contains the cumulative rewards from the initial states until the terminal states, to guide the agent away from low value behaviours ( e.g., in the multi-agent navigation problem, collisions with other agents and static/dynamic obstacles need to avoided), and encourage the agent toward the preferable actions and emergent behaviour policies ( e.g., in the multiagent navigation problem, reaching the intermediate waypoints toward the ﬁnal target should be incentivized, laminar ﬂows and vortices should emerge in conﬂicting ﬂows).

Original: To train the Teacher agent and the Student agents, we ﬁrst let the Teacher agent construct n L EARNING environments and n R EVIEW environments.
Revised: To train the Teacher agent and the Student agents, we ﬁrst let the Teacher agent construct n L EARNING environments and n R EVIEW environments.

Original: Based on the assumption that the produced free environment parameters (cid:126)θ T follows the distribution of Θ T , the L EARNING and R EVIEW environments share a similar difﬁculty, and if n is large enough, the difﬁculty of the constructed environments would be equivalent.
Revised: Based on the assumption that the produced free environment parameters (cid:126)θ T follows the distribution of Θ T , the L EARNING and R EVIEW environments share a similar difﬁculty, and if n is large enough, the difﬁculty of the constructed environments would be equivalent.

Original: Denote R ( τ ) as an operator that sums undiscounted return of all Student agents in a trajectory τ .
Revised: Denote R ( τ ) as an operator that sums undiscounted return of all Student agents in a trajectory τ .

Original: The Student agents will interact with the L EARNING environments, collect the experience and trajectories τ L , and receive the Task Completion Rewards R Task ( τ L ) and Agent Rewards R Agent ( τ L ) .
Revised: The Student agents will interact with the L EARNING environments, collect the experience and trajectories τ L , and receive the Task Completion Rewards R Task ( τ L ) and Agent Rewards R Agent ( τ L ) .

Original: The weights of the Student agent are then updated with the collected trajectories by, for instance, Proximal Policy Optimization (Schulman et al., 2017) or Twin Delayed
Revised: The weights of the Student agent are then updated with the collected trajectories by, for instance, Proximal Policy Optimization (Schulman et al., 2017) or Twin Delayed

Original: Deep Deterministic policy gradient algorithm (Fujimoto
Revised: Deep Deterministic policy gradient algorithm (Fujimoto

Original: [Figure]
Revised: [Figure]

Original: et al., 2018).
Revised: et al., 2018).

Original: With the updated weights, the Student agent will apply the learned knowledge from L EARNING environments to the R EVIEW environments, and also obtain the Task Completion Rewards R Task ( τ R ) and the Agent
Revised: With the updated weights, the Student agent will apply the learned knowledge from L EARNING environments to the R EVIEW environments, and also obtain the Task Completion Rewards R Task ( τ R ) and the Agent

Original: Rewards R Agent ( τ R ) .
Revised: Rewards R Agent ( τ R ) .

Original: Our main contribution is the M ASAI methodology to train the Teacher agent.
Revised: Our main contribution is the M ASAI methodology to train the Teacher agent.

Original: We introduce the following novel reward signals, the difference between the Task Completion
Revised: We introduce the following novel reward signals, the difference between the Task Completion

Original: Rewards and the Agent Rewards of the Student agents obtained from the R EVIEW and L EARNING environment:
Revised: Rewards and the Agent Rewards of the Student agents obtained from the R EVIEW and L EARNING environment:

Original: [Equation]
Revised: [Equation]

Original: The Student agent always tries to maximize its Task Completion Rewards and Agent rewards, as deﬁned in the RL formulation.
Revised: The Student agent always tries to maximize its Task Completion Rewards and Agent rewards, as deﬁned in the RL formulation.

Original: To generate solvable environments, the Teacher agent also needs to maximize the G AIN Task , as it needs to produce environments that can teach the Student agent to complete more tasks.
Revised: To generate solvable environments, the Teacher agent also needs to maximize the G AIN Task , as it needs to produce environments that can teach the Student agent to complete more tasks.

Original: However, if the Teacher agent where to simply maximize the G AIN Task this would lead to generating trivial environments for the Student agent.
Revised: However, if the Teacher agent where to simply maximize the G AIN Task this would lead to generating trivial environments for the Student agent.

Original: Thus, we argue that it is essential for the Teacher agent to, in addition, minimize G AIN Task .
Revised: Thus, we argue that it is essential for the Teacher agent to, in addition, minimize G AIN Task .

Original: In other words, as long as the Teacher agent is capable of generating solvable environments (by maximize G AIN Task ), it should “set up” the Student agent in the produced environments, to construct more edge cases and hard scenarios at the local and global level, to motivate the Student agents to deviate from its current parameter distribution to further explore better weight structure, ideally leading to a generalization of skills.
Revised: In other words, as long as the Teacher agent is capable of generating solvable environments (by maximize G AIN Task ), it should “set up” the Student agent in the produced environments, to construct more edge cases and hard scenarios at the local and global level, to motivate the Student agents to deviate from its current parameter distribution to further explore better weight structure, ideally leading to a generalization of skills.

Original: The sequence of free parameters (cid:126)θ in each step the Teacher agent produced form the training trajectories, with the addition of G AIN Task ( i.e., maximize ) and − G AIN Agent ( i.e., minimize ) as its reward signal.
Revised: The sequence of free parameters (cid:126)θ in each step the Teacher agent produced form the training trajectories, with the addition of G AIN Task ( i.e., maximize ) and − G AIN Agent ( i.e., minimize ) as its reward signal.

Original: To complete the approach, we train the Student agents and Teacher agent alternatively until convergence or after a certain number of iterations.
Revised: To complete the approach, we train the Student agents and Teacher agent alternatively until convergence or after a certain number of iterations.

Original: 4. Experimental Setup
Revised: 4. Experimental Setup

Original: The proposed M ASAI is a general algorithm to improve the generalization of any policy.
Revised: The proposed M ASAI is a general algorithm to improve the generalization of any policy.

Original: We test M ASAI on the task of navigation and steering due to its popularity in the MARL application and its sensitivity to unseen environments.
Revised: We test M ASAI on the task of navigation and steering due to its popularity in the MARL application and its sensitivity to unseen environments.

Original: 4.1. Teacher
Revised: 4.1. Teacher

Original: In this section, we outline the details of the Teacher agent, including the architecture and state observations in terms of navigation and steering.
Revised: In this section, we outline the details of the Teacher agent, including the architecture and state observations in terms of navigation and steering.

Original: In our experiments, the Teacher agent works in a 16 m × 16 m environment, with a ﬁxed number of agents (16 agents, coloured in red), their goals (16 coloured in green), and static obstacles (28 coloured in black).
Revised: In our experiments, the Teacher agent works in a 16 m × 16 m environment, with a ﬁxed number of agents (16 agents, coloured in red), their goals (16 coloured in green), and static obstacles (28 coloured in black).

Original: In total, there are 60 objects in an environment.
Revised: In total, there are 60 objects in an environment.

Original: See
Revised: See

Original: Figure 2 for example generated environments.
Revised: Figure 2 for example generated environments.

Original: [Figure]
Revised: [Figure]

Original: The observation of the Teacher agent in each frame are an 84 × 84 × 3 image which captures the whole environment ( e.g., Figure 2a), and a scalar k that enumerates the class of the object it should place.
Revised: The observation of the Teacher agent in each frame are an 84 × 84 × 3 image which captures the whole environment ( e.g., Figure 2a), and a scalar k that enumerates the class of the object it should place.

Original: Let k
Revised: Let k

Original: = 1 for static obstacle, k = 2 for goal, and k = 3 for agent.
Revised: = 1 for static obstacle, k = 2 for goal, and k = 3 for agent.

Original: Figure 3 illustrates the observations and actions of the Teacher agent.
Revised: Figure 3 illustrates the observations and actions of the Teacher agent.

Original: In each time step, the Teacher agent takes a visual observation of the environment, and a scalar observation to indicate the class of objects it should place next.
Revised: In each time step, the Teacher agent takes a visual observation of the environment, and a scalar observation to indicate the class of objects it should place next.

Original: The action of the
Revised: The action of the

Original: the
Revised: the

Original: Teacher agent is a two-dimensional coordinate within the environment.
Revised: Teacher agent is a two-dimensional coordinate within the environment.

Original: The Teacher agent at Time Step 0 takes an empty environment and a scalar k = 3 to place the ﬁrst agent.
Revised: The Teacher agent at Time Step 0 takes an empty environment and a scalar k = 3 to place the ﬁrst agent.

Original: At Time Step 1, the Teacher agent takes the visual observation of the environment (currently only one agent) and a scalar k = 2 to determine the place of the goal for this agent.
Revised: At Time Step 1, the Teacher agent takes the visual observation of the environment (currently only one agent) and a scalar k = 2 to determine the place of the goal for this agent.

Original: This process continues after a ﬁxed number of agents and their goals.
Revised: This process continues after a ﬁxed number of agents and their goals.

Original: Finally, the Teacher agent starts to place static obstacles by observing the current environment, with scalar k = 1 as its input (see Time Step 33).
Revised: Finally, the Teacher agent starts to place static obstacles by observing the current environment, with scalar k = 1 as its input (see Time Step 33).

Original: We note the ﬁrst visual observation is always the same at the beginning of the environment construction.
Revised: We note the ﬁrst visual observation is always the same at the beginning of the environment construction.

Original: Thus, the policy gradient method is employed to sample actions from the policy.
Revised: Thus, the policy gradient method is employed to sample actions from the policy.

Original: 4.1.1. A RCHITECTURE
Revised: 4.1.1. A RCHITECTURE

Original: The Teacher agent network architecture consists of a single convolutional layer that connects to an LSTM and then to two fully connected layers which connected to the policy outputs.
Revised: The Teacher agent network architecture consists of a single convolutional layer that connects to an LSTM and then to two fully connected layers which connected to the policy outputs.

Original: The additional scalar observation k is connected directly to the LSTM layer.
Revised: The additional scalar observation k is connected directly to the LSTM layer.

Original: A second network with identical architecture is used to estimate the value function (Dennis et al., 2021).
Revised: A second network with identical architecture is used to estimate the value function (Dennis et al., 2021).

Original: The convolutional layer contains 128 ﬁlters with 3 × 3 kernel, and an LSTM size of 256 cells, two fully connected layers of size 64 each, and a fully connected layer of size 9 to process the object class and connects to the LSTM.
Revised: The convolutional layer contains 128 ﬁlters with 3 × 3 kernel, and an LSTM size of 256 cells, two fully connected layers of size 64 each, and a fully connected layer of size 9 to process the object class and connects to the LSTM.

Original: The policy output is a 2-dimensional location to place the next object in the current environment.
Revised: The policy output is a 2-dimensional location to place the next object in the current environment.

Original: 4.2. Student
Revised: 4.2. Student

Original: In this section, we describe the Student agent for learning local steering and collision avoidance for navigating multiagent environments, or a synthetic crowds agents.
Revised: In this section, we describe the Student agent for learning local steering and collision avoidance for navigating multiagent environments, or a synthetic crowds agents.

Original: The observation of the Student agent in each frame is a 24 dimension vector, consisting of depth detection, relative velocity to the maximum allowed speed, and relative position to the next intermediate waypoint.
Revised: The observation of the Student agent in each frame is a 24 dimension vector, consisting of depth detection, relative velocity to the maximum allowed speed, and relative position to the next intermediate waypoint.

Original: Moreover, we stack the pastframes (temporally widen the observation), which results in a 96-dimensional observation vector for each agent as its policy network input.
Revised: Moreover, we stack the pastframes (temporally widen the observation), which results in a 96-dimensional observation vector for each agent as its policy network input.

Original: The policy output is a 2-dimensional virtual force to be applied to the Student agent rigidbody.
Revised: The policy output is a 2-dimensional virtual force to be applied to the Student agent rigidbody.

Original: The Student agent updates at 12 .
Revised: The Student agent updates at 12 .

Original: 5 Hz.
Revised: 5 Hz.

Original: The reward function for the Student agent is taken from (Haworth et al., 2020b), which is deﬁned as a combination of distance-based rewards.
Revised: The reward function for the Student agent is taken from (Haworth et al., 2020b), which is deﬁned as a combination of distance-based rewards.

Original: Let ∆ =
Revised: Let ∆ =

Original: || pos ( agent i ) − g
Revised: || pos ( agent i ) − g

Original: Hi || , where pos computes the current location of agent i and g
Revised: Hi || , where pos computes the current location of agent i and g

Original: Hi denotes the location of the goal of agent i : 
Revised: Hi denotes the location of the goal of agent i : r goal = (cid:26)if ∆ <exp (cid:0)

Original: [Equation]
Revised: [Equation]

Original: The instantaneous reward signal places a large reward value when reaching goals as soon as possible, while the continuous component guides learning policies toward its goals.
Revised: The instantaneous reward signal places a large reward value when reaching goals as soon as possible, while the continuous component guides learning policies toward its goals.

Original: A second penalty term r collision checks agent-agent or agent-
Revised: A second penalty term r collision checks agent-agent or agent-

Original: Visual Observation

Original: [Figure]

Original: Final Environment

Original: [Figure]
Revised: [Figure]

Original: Visual Observation
Revised: Observation

Original: [Figure]
Revised: [Figure]

Original: Teacher
Revised: Teacher

Original: [Figure]
Revised: [Figure]

Revised: obstacle collisions.

Original: r collision = (cid:26) − 1 if collision occurs otherwise (3)
Revised: r collision = (cid:26) − 1 if collision occurs otherwise (3)

Original: 4.2.1. A RCHITECTURE
Revised: 4.2.1. A RCHITECTURE

Original: The Student agent network architecture consists of a LSTM and then to two fully connected layers which connect to the policy output.
Revised: The Student agent network architecture consists of a LSTM and then to two fully connected layers which connect to the policy output.

Original: A second network with identical architecture is used to estimate the value function (Dennis et al., 2021).
Revised: A second network with identical architecture is used to estimate the value function (Dennis et al., 2021).

Original: The LSTM is of size 256 cells, and two fully connected layers of 64 which connect to either the policy outputs or the value estimate.
Revised: The LSTM is of size 256 cells, and two fully connected layers of 64 which connect to either the policy outputs or the value estimate.

Original: 4.3. Training
Revised: 4.3. Training

Original: The experiments were conducted on a server with 48 Intel Xeon E5-2650 CPUs.
Revised: The experiments were conducted on a server with 48 Intel Xeon E5-2650 CPUs.

Original: To improve reproducibility, we control the random seed for all training and experiments.
Revised: To improve reproducibility, we control the random seed for all training and experiments.

Original: For instance, the initialization of neural networks in each experiment will be the same.
Revised: For instance, the initialization of neural networks in each experiment will be the same.

Original: The inputs and rewards were normalized for both the Student agents and the Teacher agent, and both were trained with Proximal Policy Optimization (PPO) (Schulman et al., 2017) algorithm with a discount factor of 0.99, a learning rate of 0.0001 with Adam optimizer (Kingma & Ba, 2017), and an 8 parallel training environments to collect a batch of episodes, which was used to complete on training update In other words, Teacher agent creates 8 L EARNING and R EVIEW environments each iteration.
Revised: The inputs and rewards were normalized for both the Student agents and the Teacher agent, and both were trained with Proximal Policy Optimization (PPO) (Schulman et al., 2017) algorithm with a discount factor of 0.99, a learning rate of 0.0001 with Adam optimizer (Kingma & Ba, 2017), and an 8 parallel training environments to collect a batch of episodes, which was used to complete on training update In other words, Teacher agent creates 8 L EARNING and R EVIEW environments each iteration.

Original: All Student agents shared the same policy network, i.e. , parameter sharing for multi-agent reinforcement learning.
Revised: All Student agents shared the same policy network, i.e. , parameter sharing for multi-agent reinforcement learning.

Original: We empirically set the maximum allowed steps of the Student agents during training to 250 .
Revised: We empirically set the maximum allowed steps of the Student agents during training to 250 .

Original: 5. Experiments
Revised: 5. Experiments

Original: We qualitatively demonstrate that the Student agent trained by M ASAI generalizes better in unseen environments than by DR and heuristic search, which are supported by the quantitative evaluation.
Revised: We qualitatively demonstrate that the Student agent trained by M ASAI generalizes better in unseen environments than by DR and heuristic search, which are supported by the quantitative evaluation.

Original: We use the DR technique as our primary baseline representing how environments have been generated in past solutions to the multi-agent navigation problem.
Revised: We use the DR technique as our primary baseline representing how environments have been generated in past solutions to the multi-agent navigation problem.

Original: In the navigation and steering task, DR is employed by uniformly and randomly choosing a location to place objects.
Revised: In the navigation and steering task, DR is employed by uniformly and randomly choosing a location to place objects.

Original: In the following experiments, the only difference is the training algorithm, i.e., DR versus M ASAI .
Revised: In the following experiments, the only difference is the training algorithm, i.e., DR versus M ASAI .

Original: 5.1. Qualitative Evaluation
Revised: 5.1. Qualitative Evaluation

Original: Sample environments constructed by DR and using M ASAI are shown in Figure 2.
Revised: Sample environments constructed by DR and using M ASAI are shown in Figure 2.

Original: In a small environment, such as 16 m × 16 m , there is an inﬁnite number of conﬁgurations, but the computational resource is limited.
Revised: In a small environment, such as 16 m × 16 m , there is an inﬁnite number of conﬁgurations, but the computational resource is limited.

Original: It is crucial to reduce the free parameter space and produce environments that can instruct the Student agent to learn (not simply memorize) quickly.
Revised: It is crucial to reduce the free parameter space and produce environments that can instruct the Student agent to learn (not simply memorize) quickly.

Original: M ASAI solves this issue precisely.
Revised: M ASAI solves this issue precisely.

Original: From
Revised: From

Original: Figure 2, we can observe that the DR places objects that are spread around across the environment, due to the uniform distribution sample space.
Revised: Figure 2, we can observe that the DR places objects that are spread around across the environment, due to the uniform distribution sample space.

Original: Moreover, there are cases (Figure 2a 2c) the goals are overlapped with the static obstacles and some agents are stuck between static obstacles (Figure 2b), which result in unsolvable cases.
Revised: Moreover, there are cases (Figure 2a 2c) the goals are overlapped with the static obstacles and some agents are stuck between static obstacles (Figure 2b), which result in unsolvable cases.

Original: While M ASAI tends to cluster the objects and put goals near the static obstacles, and create challenging scenarios, as shown in Figure 2.
Revised: While M ASAI tends to cluster the objects and put goals near the static obstacles, and create challenging scenarios, as shown in Figure 2.

Original: For instance, in Figure 2d and Figure 2e, the central agents are surround by static obstacles, forming challenging training scenario caves .
Revised: For instance, in Figure 2d and Figure 2e, the central agents are surround by static obstacles, forming challenging training scenario caves .

Original: In Figure 2f, M ASAI creates nearly a wall-like conﬁguration in the left to separate agents and their goals.
Revised: In Figure 2f, M ASAI creates nearly a wall-like conﬁguration in the left to separate agents and their goals.

Original: To evaluate the learned Student agent policy, we create several common navigation and steering scenarios.
Revised: To evaluate the learned Student agent policy, we create several common navigation and steering scenarios.

Original: At the smaller scale, these include Egress , Simple wall , Crossing with obstacles , Crossing groups , Surprise , and Rooms egress .
Revised: At the smaller scale, these include Egress , Simple wall , Crossing with obstacles , Crossing groups , Surprise , and Rooms egress .

Original: At the larger scale, we use Bi-directional hallway and Diametric goals scenarios which contain forty-nine dy- namic agents, as shown in Figure 5.
Revised: At the larger scale, we use Bi-directional hallway and Diametric goals scenarios which contain forty-nine dy- namic agents, as shown in Figure 5.

Original: In all evaluations, we use the weights of the Student agent saved at the highest average return (see Figure 7) in both DR and M ASAI to solve the evaluation scenarios.
Revised: In all evaluations, we use the weights of the Student agent saved at the highest average return (see Figure 7) in both DR and M ASAI to solve the evaluation scenarios.

Original: Results for the six smaller scale common steering and navigation scenarios are shown in Figure 4.
Revised: Results for the six smaller scale common steering and navigation scenarios are shown in Figure 4.

Original: In these scenarios we show that M ASAI guides the learning of high value policies for emergent behaviours.
Revised: In these scenarios we show that M ASAI guides the learning of high value policies for emergent behaviours.

Original: In particular, Student agent learns to side step and wait for other Student agents to reduce collisions.
Revised: In particular, Student agent learns to side step and wait for other Student agents to reduce collisions.

Original: This is a complicated behaviour that extends over time that even industry standard steering simulators struggle with or do not produce.
Revised: This is a complicated behaviour that extends over time that even industry standard steering simulators struggle with or do not produce.

Original: However, DR policies never discover this policy.
Revised: However, DR policies never discover this policy.

Original: We also show that polices learned from
Revised: We also show that polices learned from

Original: DR can lead to equilibrium cases where the net force on competing agents reaches equilibrium and the agents never complete the scenario (see Surprise and Crossing with Obstacles in Figure 4).
Revised: DR can lead to equilibrium cases where the net force on competing agents reaches equilibrium and the agents never complete the scenario (see Surprise and Crossing with Obstacles in Figure 4).

Original: Policies learned from M ASAI avoid this problem through side-stepping and stop-and-wait behaviours.
Revised: Policies learned from M ASAI avoid this problem through side-stepping and stop-and-wait behaviours.

Original: These represent real courtesies people give to each other in such low density scenarios.
Revised: These represent real courtesies people give to each other in such low density scenarios.

Original: From Figure 5 and Figure 6, Student agent policy and DR agent policy are able to solve the environments.
Revised: From Figure 5 and Figure 6, Student agent policy and DR agent policy are able to solve the environments.

Original: From the simulation, we notice that the Student agent trained by M ASAI demonstrates important emergent steering skills.
Revised: From the simulation, we notice that the Student agent trained by M ASAI demonstrates important emergent steering skills.

Original: In
Revised: In

Original: Figure 5, initially, the lower group and upper group are aiming to avoid collision by steering far right and far left.
Revised: Figure 5, initially, the lower group and upper group are aiming to avoid collision by steering far right and far left.

Original: Moreover, both M ASAI Student agent and the DR policies cluster in the Diametric Goals scenario and created a large number of collisions, but the Student agent trained by M ASAI reduces the amount of time of clustering much faster than the agent trained by DR through an emergent vortex behaviour.
Revised: Moreover, both M ASAI Student agent and the DR policies cluster in the Diametric Goals scenario and created a large number of collisions, but the Student agent trained by M ASAI reduces the amount of time of clustering much faster than the agent trained by DR through an emergent vortex behaviour.

Original: These are unseen environments in the training environment distribution, showing M ASAI trains more generalized agents.
Revised: These are unseen environments in the training environment distribution, showing M ASAI trains more generalized agents.

Original: 5.2. Quantitative Evaluation
Revised: 5.2. Quantitative Evaluation

Original: To quantitatively estimate the performance of the trained
Revised: To quantitatively estimate the performance of the trained

Original: Student agent policies, we compute the number of collisions and the elapsed time of agents to reach its goal.
Revised: Student agent policies, we compute the number of collisions and the elapsed time of agents to reach its goal.

Original: We also compare the results for agents follows the path planned by
Revised: We also compare the results for agents follows the path planned by

Original: Heuristic Search ( i.e., A* path planning) due to its wide usage in synthetic crowds applications.
Revised: Heuristic Search ( i.e., A* path planning) due to its wide usage in synthetic crowds applications.

Original: Based on Table 1 and Table 2, it is clear the Student agents trained by M ASAI has a superior generalization performance than the agent trained by DR in various metrics regarding the quality of steering in a multi-agent navigation setting.
Revised: Based on Table 1 and Table 2, it is clear the Student agents trained by M ASAI has a superior generalization performance than the agent trained by DR in various metrics regarding the quality of steering in a multi-agent navigation setting.

Original: We note the mean of time used to reach goals by the agents that are driven by heuristic search is the lowest is due to the fact the agents will move straight up to their goals, without any consideration of steering.
Revised: We note the mean of time used to reach goals by the agents that are driven by heuristic search is the lowest is due to the fact the agents will move straight up to their goals, without any consideration of steering.

Original: This will result in the fastest navigation and the least pleasant steering.
Revised: This will result in the fastest navigation and the least pleasant steering.

Original: Initial Scenario
Revised: Initial Scenario

Original: DR
Revised: DR

Original: M ASAI
Revised: M ASAI

Original: E g r e ss
Revised: E g r e ss

Original: S i m p l e
Revised: S i m p l e

Original: W a ll
Revised: W a ll

Original: C r o ss i ng \ w
Revised: C r o ss i ng \ w

Original: O b s .
Revised: O b s .

Original: C r o ss i ng
Revised: C r o ss i ng

Original: G r oup s
Revised: G r oup s

Original: S
Revised: S

Original: u r p
Revised: u r p

Original: r
Revised: r

Original: i s e
Revised: i s e

Original: R oo m s
Revised: R oo m s

Original: E g r e ss
Revised: E g r e ss

Original: D R
Revised: D R

Original: M A S
Revised: M A S

Original: A I
Revised: A I

Original: D R
Revised: D R

Original: M A S
Revised: M A S

Original: A I
Revised: A I

Original: [Table]
Revised: [Table]

Original: of time is second.

Original: [Figure]

Original: ETRICS

Original: HS

Original: DR

Original: M ASAI

Original: A

Original: [Figure]

Original: C OMPLETED

Original: T

Original: [Figure]

Original: C

Original: [Figure]
Revised: [Table]

Original: M EAN C OLLISIONS 13 .10 .9 .

Original: S TD C OLLISIONS 5 .5 .4 .

Original: M EAN T IME TO G OAL 7 .27 .21 .

Original: S TD T IME TO G OAL 0 .7 .3 .
Revised: S TD T IME TO G OAL 0 .7 .3 .

Original: Figure 7 illustrates the average undiscounted return curve during training of the Student agents trained by M ASAI is higher than and trained by DR, which aligns with the quantitative metrics in Table 1 and qualitative evaluation.
Revised: Figure 7 illustrates the average undiscounted return curve during training of the Student agents trained by M ASAI is higher than and trained by DR, which aligns with the quantitative metrics in Table 1 and qualitative evaluation.

Original: [Figure]
Revised: [Figure]

Original: 6. Related Work
Revised: 6. Related Work

Original: Prior works on automating the process of generating environments include DR and Minimax Adversary.
Revised: Prior works on automating the process of generating environments include DR and Minimax Adversary.

Original: DR aims to generate fully speciﬁed environments, without taking the current learning agent policy into consideration (Jakobi,
Revised: DR aims to generate fully speciﬁed environments, without taking the current learning agent policy into consideration (Jakobi,

Original: 1997; Savage, 1951; Graves et al., 2017).
Revised: 1997; Savage, 1951; Graves et al., 2017).

Original: These environment parameters are randomly sampled for each episode, and then a policy is trained over the generated environment.
Revised: These environment parameters are randomly sampled for each episode, and then a policy is trained over the generated environment.

Original: Nevertheless, this does not guarantee policy performance on a carefully designed conﬁguration of free parameters.
Revised: Nevertheless, this does not guarantee policy performance on a carefully designed conﬁguration of free parameters.

Original: Minimax adversary training aims to produce environments by minimizing the reward of the current learning agent policy, i.e., minimax training (Pinto et al., 2016; Morimoto &
Revised: Minimax adversary training aims to produce environments by minimizing the reward of the current learning agent policy, i.e., minimax training (Pinto et al., 2016; Morimoto &

Original: Doya, 2005; Shen et al., 2019).
Revised: Doya, 2005; Shen et al., 2019).

Original: Although these approaches have demonstrated the capability of producing compelling environments, they can fail to generate truly valuable environments.
Revised: Although these approaches have demonstrated the capability of producing compelling environments, they can fail to generate truly valuable environments.

Original: Uniformly random environments will usually fail to generate challenging and sometimes solvable environments, while minimax adversary is incentivized to create environments that are impossible to solve.
Revised: Uniformly random environments will usually fail to generate challenging and sometimes solvable environments, while minimax adversary is incentivized to create environments that are impossible to solve.

Original: In many applications, both of these algorithms fail to produce valuable and solvable structures.
Revised: In many applications, both of these algorithms fail to produce valuable and solvable structures.

Original: The recently introduced PAIRED algorithm is a middle ground of environment generators which maximize regret , producing difﬁcult but solvable tasks (Dennis et al., 2021).
Revised: The recently introduced PAIRED algorithm is a middle ground of environment generators which maximize regret , producing difﬁcult but solvable tasks (Dennis et al., 2021).

Original: We discuss PAIRED in detail in
Revised: We discuss PAIRED in detail in

Original: Section 2.
Revised: Section 2.

Original: Curriculum Learning is a well-studied and widely applied solution to difﬁcult training problems in RL (Bengio et al., 2009; Matiisen et al., 2017; Leibo et al., 2019; Graves et al., 2017; Florensa et al., 2017).
Revised: Curriculum Learning is a well-studied and widely applied solution to difﬁcult training problems in RL (Bengio et al., 2009; Matiisen et al., 2017; Leibo et al., 2019; Graves et al., 2017; Florensa et al., 2017).

Original: In basic curriculum learning, the learning agents is exposed to increasingly difﬁcult tasks to learn or environments to train in.
Revised: In basic curriculum learning, the learning agents is exposed to increasingly difﬁcult tasks to learn or environments to train in.

Original: However, it is increasingly difﬁcult to decide what is ‘difﬁcult’ as well as when, where, and how much a particular element of a curriculum should be used during training.
Revised: However, it is increasingly difﬁcult to decide what is ‘difﬁcult’ as well as when, where, and how much a particular element of a curriculum should be used during training.

Original: Teacher-Student Curriculum
Revised: Teacher-Student Curriculum

Original: Learning (TSCL) is a framework designed for automatic curriculum learning, where the Student tries to learn a complex task and the Teacher automatically chooses sub-tasks from a given set for the Student to train on (Matiisen et al., 2017).
Revised: Learning (TSCL) is a framework designed for automatic curriculum learning, where the Student tries to learn a complex task and the Teacher automatically chooses sub-tasks from a given set for the Student to train on (Matiisen et al., 2017).

Original: The Teacher algorithms rely on the intuition that the Student should practice more of those tasks on which it makes the fastest progress.
Revised: The Teacher algorithms rely on the intuition that the Student should practice more of those tasks on which it makes the fastest progress.

Original: Nevertheless, the Teacher in the proposed algorithm is not parameterized or trainable, it maintains an array of probability for each sub-task, i.e. , only the Student is trainable.
Revised: Nevertheless, the Teacher in the proposed algorithm is not parameterized or trainable, it maintains an array of probability for each sub-task, i.e. , only the Student is trainable.

Original: The Teacher proposes tasks through an epsilon-greedy policy on that probability array.
Revised: The Teacher proposes tasks through an epsilon-greedy policy on that probability array.

Original: Another
Revised: Another

Original: Teacher-Student approach for predicting hospital inpatient admission location separates the Teacher and Student into two neural networks where the Teacher learns to select items from a dataset for the Student to learn on (El-Bouri et al., 2020).
Revised: Teacher-Student approach for predicting hospital inpatient admission location separates the Teacher and Student into two neural networks where the Teacher learns to select items from a dataset for the Student to learn on (El-Bouri et al., 2020).

Original: Our work proposes to address the shortcomings of prior methods when applied to a more difﬁcult problem domain.
Revised: Our work proposes to address the shortcomings of prior methods when applied to a more difﬁcult problem domain.

Original: We apply a novel Teacher-Student approach top the UED problem for continuous multi-agent reinforcement learning.
Revised: We apply a novel Teacher-Student approach top the UED problem for continuous multi-agent reinforcement learning.

Original: M ASAI fully parameterizes both the Teacher agent and Student agent using deep neural networks, where the proposed tasks for Student agents are produced directly without any prior selection algorithms.
Revised: M ASAI fully parameterizes both the Teacher agent and Student agent using deep neural networks, where the proposed tasks for Student agents are produced directly without any prior selection algorithms.

Original: In this work, we propose incentivize the Teacher agent, ( i.e., the Unsupervised
Revised: In this work, we propose incentivize the Teacher agent, ( i.e., the Unsupervised

Original: Environment Generator), to produce free parameters that, when a policy is trained on it, can observe novel reward signals, G AIN , which guides the Teacher agent to construct environments that are not only as difﬁcult as possible but also catered for automatically improving the sophistication of the policy of the Student agents.
Revised: Environment Generator), to produce free parameters that, when a policy is trained on it, can observe novel reward signals, G AIN , which guides the Teacher agent to construct environments that are not only as difﬁcult as possible but also catered for automatically improving the sophistication of the policy of the Student agents.

Original: To test our approach, we look to the local steering and collisions avoidance in multi-agent navigation literature for a difﬁcult problem domain.
Revised: To test our approach, we look to the local steering and collisions avoidance in multi-agent navigation literature for a difﬁcult problem domain.

Original: Navigation and steering is a burgeoning application of
Revised: Navigation and steering is a burgeoning application of

Original: MARL, especially in terms of crowd simulation (Haworth et al., 2020a; Berseth et al., 2020; Lee et al., 2018; Lowe et al., 2017; Lan et al., 2020; Sun et al., 2019; Bisagno et al., 2019).
Revised: MARL, especially in terms of crowd simulation (Haworth et al., 2020a; Berseth et al., 2020; Lee et al., 2018; Lowe et al., 2017; Lan et al., 2020; Sun et al., 2019; Bisagno et al., 2019).

Original: The problem is continuous, multi-agent, and often semi-chaotic.
Revised: The problem is continuous, multi-agent, and often semi-chaotic.

Original: Additionally, earlier work in comparative crowds analysis proposed a carefully crafted rule-based algorithm to generate challenging environments to evaluate steering algorithms (Kapadia et al., 2011).
Revised: Additionally, earlier work in comparative crowds analysis proposed a carefully crafted rule-based algorithm to generate challenging environments to evaluate steering algorithms (Kapadia et al., 2011).

Original: However, the space of rule based procedural content (environment) generation is beyond the scope of this paper.
Revised: However, the space of rule based procedural content (environment) generation is beyond the scope of this paper.

Original: The proposed
Revised: The proposed

Original: M ASAI approach is not limited to navigation and steering and ideally can be applied to many other difﬁcult continuous
Revised: M ASAI approach is not limited to navigation and steering and ideally can be applied to many other difﬁcult continuous

Original: MARL problems.
Revised: MARL problems.

Original: 7. Discussion
Revised: 7. Discussion

Original: In this work, we proposed an algorithm for solving the unsupervised environment design problem in the continuous multi-agent reinforcement learning domain.
Revised: In this work, we proposed an algorithm for solving the unsupervised environment design problem in the continuous multi-agent reinforcement learning domain.

Original: Our method estimates the environment generator policy by carefully formulating a training procedure informed by the TeacherStudent approach and guided by a novel reward signal designed to produce compelling, challenging, and solvable environments in an automatic curriculum for training agents in difﬁcult problem domains.
Revised: Our method estimates the environment generator policy by carefully formulating a training procedure informed by the TeacherStudent approach and guided by a novel reward signal designed to produce compelling, challenging, and solvable environments in an automatic curriculum for training agents in difﬁcult problem domains.

Original: Our method produces such environments and that the resultant training regime leads to highly valuable policies.
Revised: Our method produces such environments and that the resultant training regime leads to highly valuable policies.

Original: We showed that the approach solves the UED problem for training multi-agent navigation and the the trained agents learn valuable emergent polices which generalize in entirely unseen environments.
Revised: We showed that the approach solves the UED problem for training multi-agent navigation and the the trained agents learn valuable emergent polices which generalize in entirely unseen environments.

Original: Limitations of the method are mainly related to the underlying methodology for training in the MARL domain.
Revised: Limitations of the method are mainly related to the underlying methodology for training in the MARL domain.

Original: We utilize parameter sharing which has high utility and mitigates issues like non-stationarity.
Revised: We utilize parameter sharing which has high utility and mitigates issues like non-stationarity.

Original: However, there exists many approaches to multi-agent training which may or may not be adapted into this framework.
Revised: However, there exists many approaches to multi-agent training which may or may not be adapted into this framework.

Original: Additionally, it remains to be seen how successful such a method would be in a real world setting with highly uncertain or noisy inputs.
Revised: Additionally, it remains to be seen how successful such a method would be in a real world setting with highly uncertain or noisy inputs.

Original: For future work, we plan to propose extensions to the
Revised: For future work, we plan to propose extensions to the

Original: PAIRED algorithm for the multi-agent continuous domain.
Revised: PAIRED algorithm for the multi-agent continuous domain.

Original: Moreover, we also want to test M ASAI on other tasks, such as computer vision and natural language processing.
Revised: Moreover, we also want to test M ASAI on other tasks, such as computer vision and natural language processing.

Revised: Acknowledgements

Revised: This work was supported by the Jamie Cassels Undergraduate Research Award 2020-21 at the University of Victoria.