Original: M
Revised: M

Original: ULTI
Revised: ULTI

Original: -A GENT H IERARCHICAL R EINFORCEMENT L EARNING FOR H UMANOID N AVIGATION
Revised: -A GENT H IERARCHICAL R EINFORCEMENT L EARNING FOR H UMANOID N AVIGATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Multi-agent reinforcement learning is a particularly challenging problem.
Revised: Multi-agent reinforcement learning is a particularly challenging problem.

Original: Current methods have made progress on cooperative and competitive environments with particle-based agents.
Revised: Current methods have made progress on cooperative and competitive environments with particle-based agents using communication and centralized training.

Original: Not enough progress has been made on solutions that could operate in the real world with interaction, dynamics, and humanoid navigation strategies.
Revised: Little progress has been made on solutions that could operate in the real world with interaction, dynamics, and humanoid navigation strategies.

Original: In this work, we make a signiﬁcant step in multi-agent models on simulated humanoid navigation by combining Multi-Agent Reinforcement Learning with Hierarchical Reinforcement Learning.
Revised: In this work, we make a signiﬁcant step in multi-agent models on simulated humanoid navigation by combining Multi-Agent Reinforcement Learning with Hierarchical Reinforcement Learning.

Revised: Speciﬁcally, we develop a partial parameter sharing approach wherein the lower level of the hierarchy is shared enabling learning using decentralized methods.

Revised: This drastically reduces the overall parameter space in the multi-agent problem and introduces structure in the optimization problem.

Original: We build on top offoundational prior  work in learning  low-level physical controllers for locomotion and add a layer to learn decentralized  policies for multi-agent goal-directed collision avoidance systems.
Revised: We build on top of prior foundational work in learning goal-conditioned policies to learn low-level physical controllers for balance and walking. These lower-level controllers are task-agnostic and can be shared by higher-level policies. Overtop of these goal conditioned policies, we can train decentralized heterogeneous policies for multi-agent goal-directed collision avoidance.

Original: Surprisingly, our results show that with this combination of methods common RL techniques can be used for ﬁnding heterogeneous policies.
Revised: Surprisingly, our results show that with this combination of methods, RL techniques can be used for ﬁnding strong policies.

Original: A video of our results on a multi-agent pursuit environment can be seen here 
Revised: A video of our results on a multi-agent pursuit environment can be seen here 1 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multi-Agent Reinforcement Learning (MARL) is a very difﬁcult problem space.

Original: Reinforcement learning methods rely heavily upon stationary environments wherein a learned policy can be improved by maximizing reward based on observations.

Original: With MARL, where agents learn in the shared environment, leads to agents making learning updates using data based on the previous dynamics of the environment.

Original: This is further complicated when the agents are not simple in their actions or representations.

Original: For example, physically enabled humanoid agents must learn balance control, navigation, and goal completion strategies.

Original: Yet if one goal of AI is to have agents that can operate in the real world they will need to reason over physical interaction amoung other agents.

Original: Deep Reinforcement Learning (DRL) has been successful at solving complex planning tasks.
Revised: Deep Reinforcement Learning (DRL) has been successful in solving complex planning tasks.

Original: Given a sizeable computational budget DRL has displayed superhuman performance on many games (Mnih et al., 2015; Silver et al., 2017).
Revised: Given a sizeable computational budget DRL has displayed superhuman performance on many games (Mnih et al., 2015; Silver et al., 2017).

Original: However, less progress has been made on the MARL problem space, possibly due to the non-stationary optimization of multiple changing policies which is not easily overcome by collecting more data (OpenAI, 2018).
Revised: However, less progress has been made on the Multi-Agent Reinforcement Learning (MARL) problem space, possibly due to the non-stationary optimization of multiple changing policies, which is not easily overcome by collecting more data (OpenAI, 2018).

Original: If the goal is to create agents that can operate in a dynamic multi-agent world, more stable methods with novel forms of communication are needed.
Revised: If the goal is to create agents that can operate in a dynamic multi-agent world, more stable methods with novel forms of communication are needed.

Original: The trend to make progress on MARL has been to simplify the optimization problem.
Revised: The trend to make progress on MARL has been to simplify the optimization problem.

Original: For example, converting the multi-agent problem into a single agent centralized model results in large gains in performance but can increase the number of network parameters  (Lowe et al., 2017).
Revised: For example, converting the multi-agent problem into a single agent centralized model results in large gains in performance but can increase the number of network parameters signiﬁcantly and imposes a constraint on the generalization to the number of agents (Lowe et al., 2017).

Original: By using recurrent policies, signiﬁcant compute, and constraints on the amount the policy is allowed to change, it is possible to beat the best humans at the multi-agent game of Dota (OpenAI, 2018).
Revised: By using recurrent policies, signiﬁcant computational power, and constraints on the amount the policy is allowed to change between updates, it is possible to beat the best humans at the multi-agent game of Dota (OpenAI, 2018).

Original: While these methods have shown promiss they  have not yet displayed success in complex multiagent environments.
Revised: While these methods have shown promise, they require signiﬁcant amounts of compute and have not yet displayed success in complex and dynamic multi-agent environments.

Original: It has been possible to produce competitive behaviour using asymmetric selfplay but this work is limited to few agents (Bansal et al., 2017).
Revised: While recent work has been successful in producing competitive behaviour using asymmetric self-play, this work is limited to two agents (Bansal et al., 2017).

Original: The driving question then is, how can we simplify the MARL optimization problem while maintaining the heterogeneous nature of multi-agent problems?

Original: In this work, we propose thedeep integration of MARL with Hierarchical Reinforcement Learning (HRL) to produce heterogeneous humanoid agents that can both navigate and interact in dynamic simulation.
Revised: In this work, we propose the integration of MARL with Hierarchical Reinforcement Learning (HRL) to produce heterogeneous humanoid agents that can both navigate and interact in dynamic simulation.

Original: Speciﬁcally, we propose a method to reduce the difﬁculty in the MARL policy optimization problem.
Revised: Speciﬁcally, we propose a method to reduce the complexity in the MARL policy optimization problem using a type of parameter sharing.

Original: While previous methods have focused on re-framing the problem as a type of single-agent RL problem, our method preserves the important features of heterogeneous agent behaviour without adding more network parameters (Lowe et al., 2017).
Revised: While previous methods have focused on re-framing the problem as a type of single-agent RL problem, we show that this does not scale. Our method preserves the essential features of heterogeneous agent behaviour without adding more network parameters (Lowe et al., 2017).

Original: This use of HRL  has many advantages over current methods.
Revised: This use of HRL with a goal conditioned lower layer (Kaelbling, 1993), has signiﬁcant advantages over current methods.

Original: We use shared parameter methodsbut only for task-agnostic portions of the policy.
Revised: We use shared parameter methods for task-agnostic portions of the policy similar to multi-task learning (Caruana, 1997; Argyriou et al., 2007;

Revised: Lu et al., 2017) where each high-level agent is a different task.

Original: Given the shared sub-policy, the optimization is simpliﬁed and allows us to learn complex multi-agent policies with signiﬁcantly less data.
Revised: Given the shared sub-policy, the optimization is simpliﬁed and allows learning complex multi-agent policies with signiﬁcantly less data.

Original: This method represents, to the best of our knowledge, the ﬁrst method for model-free multi-agent physical character control for locomotion and navigation.
Revised: This method represents, to the best of our knowledge, the ﬁrst method for heterogeneous multi-agent physical character control for locomotion, navigation, and behaviour.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Simulated robot and physical character control is a rich area of research with many solutions and approaches.
Revised: Simulated robot and physical character control is a rich area of research with many solutions and approaches.

Original: Neural models focused on training neural networks by receiving joint or body sensor feedback as input and producing appropriate joint angles as output (Geng et al., 2006; Kun &
Revised: Neural models focused on training neural networks by receiving joint or body sensor feedback as input and producing appropriate joint angles as output (Geng et al., 2006; Kun &

Original: Miller III, 1996; Miller III, 1994).
Revised: Miller III, 1996; Miller III, 1994).

Original: A biped character’s movement controller set can also be manually composed using simple control strategies and feedback learning (Yin et al., 2007; Faloutsos et al., 2001).
Revised: A biped character’s movement controller set can also be manually composed using simple control strategies and feedback learning (Yin et al., 2007; Faloutsos et al., 2001).

Original: HRL has been proposed as a solution to handling many of the issues with current RL techniques that have trouble with long horizons and week signal.
Revised: HRL has been proposed as a solution to handling many of the issues with current RL techniques that have trouble with long horizons and weak signal.

Original: Many frameworks have been proposed for HRL but none seem to be the obvious choice for any particular problem (Sutton et al., 1999; Dayan & Hinton, 1993; Dietterich, 1999).
Revised: Many frameworks have been proposed for HRL, but none seem to be the obvious choice for any particular problem (Sutton et al., 1999; Dayan & Hinton, 1993; Dietterich, 1999).

Original: One difﬁculty in HRL design is ﬁnding a reasonable communication representation to condition the lower level on.
Revised: One difﬁculty in HRL design is ﬁnding a reasonable communication representation to condition the lower level.

Original: Some methods pretrain the lower level on a random distribution (Heess et al., 2016; Peng et al., 2017) and other learn a more constructive latent encoding (Nair et al., 2018; Eysenbach et al., 2019; Gupta et al., 2018).
Revised: Some methods pretrain the lower level on a random distribution (Heess et al., 2016; Peng et al., 2017; Merel et al., 2018) and others learn a more constructive latent encoding (Nair et al., 2018; Eysenbach et al., 2019; Gupta et al., 2018).

Original: There is also the present challenge of learning multiple level of the hierarchy concurrently (Vezhnevets et al., 2017; Nachum et al., 2018; Levy et al., 2017).
Revised: There is also the present challenge of learning multiple levels of the hierarchy concurrently (Vezhnevets et al., 2017; Nachum et al., 2018; Levy et al., 2017a).

Original: We use a goal based approach that uses  footstep space representation that is randomly sampled to learn a task-agnostic lower level of control.
Revised: Here a goal-based approach that uses a footstep space representation that is pretrained to learn a task-agnostic lower level of control.

Revised: This provides the high-level policies with a single shared model that is conditioned to produce diverse behaviour based on a latent goal input.

Original: 2. M ULTI -A GENT D EEP R EINFORCEMENT L EARNING
Revised: 2. M ULTI -A GENT D EEP R EINFORCEMENT L EARNING

Original: There are many types of multi-agent learning problems including cooperative, competitive, and with or without communication (Bu et al., 2008; Tan, 1993; Panait & Luke, 2005).
Revised: There are many types of multi-agent learning problems, including cooperative-competitive and withwithout communication (Bu et al., 2008; Tan, 1993; Panait & Luke, 2005).

Original: Recent work converts the MARL problem to a single agent setting by using a single Q-function across all agents (Lowe et al., 2017).
Revised: Recent work converts the MARL problem to a single agent setting by using a single Q-function across all agents (Lowe et al., 2017).

Original: Additional work focuses on the problem of learning communication methods between agents (Foerster et al., 2016).
Revised: Additional work focuses on the problem of learning communication methods between agents (Foerster et al., 2016).

Original: While progress is being made, MARL is notoriously difﬁcult due to the non-stationary optimization issue, even in the cooperative case (Claus & Boutilier, 1998).
Revised: While progress is being made, MARL is notoriously tricky due to the non-stationary optimization issue, even in the cooperative case (Claus & Boutilier, 1998).

Original: In this work, we apply a partial parameter sharing method assuming all agents are partly homogeneous at a tast-agnostic level and optimize similar goals (Gupta et al., 2017).
Revised: In this work, we apply a partial parameter sharing method assuming all agents share task-agnostic locomotion and optimize similar goals (Gupta et al., 2017).

Original: There exists few environments speciﬁcally created for MARL evaluation (Zheng et al., 2018; Suarez et al., 2019).
Revised: There exists few environments speciﬁcally created for MARL evaluation (Zheng et al., 2018; Suarez et al., 2019).

Original: The focus of these environments is often a type of strategy learning and cooperation (Tian et al., 2017; Vinyals et al., 2017).
Revised: The focus of these environments is often a type of strategy learning and cooperation (Tian et al., 2017; Vinyals et al., 2017).

Original: MARL is a growing area of research and as such will need increasingly complex environments to evaluate algorithms.
Revised: MARL is a growing area of research and, as such, will need increasingly complex environments to evaluate algorithms.

Original: In this work, we are interested in the overlapping problems of control and perception.
Revised: In this work, we are interested in the overlapping problems of control and perception.

Original: To this end, we have created the ﬁrst simulation of its type that affords multiple physics-based control tasks with variable numbers of agents.
Revised: To this end, we have created the ﬁrst simulation of its type that affords multiple physics-based control tasks with variable numbers of agents.

Original: Recent work has begun to combine MARL and HRL but is limited to simple environments, uses additional methods to stabilize the optimization, and includes communication (Tang et al., 2018; Han et al., 2019).
Revised: Recent work has begun to combine MARL and HRL but is limited to simple environments, uses additional methods to stabilize the optimization, and includes communication (Tang et al., 2018; Han et al., 2019).

Revised: This work focuses on a decentralized approach for interacting models that generalize across scenarios, agent numbers, and tasks.

Original: Our work represents a new paradigm in learning within the multi-agent navigation domain–beyond simple navigation control strategies.
Revised: Our work represents a new paradigm in learning within the multi-agent navigation domain that goes beyond simple navigation control strategies.

Original: We show compelling AI that learns navigation and gameplay strategies with fully articulated physical characters.
Revised: We show compelling AI that learns navigation and gameplay strategies with fully articulated physical characters.

Original: This is achieved through a novel learning strategy that produces high value policies for a complicated control problem.
Revised: This is achieved through a novel learning strategy that produces high-value policies for a complicated control problem.

Original:  P OLICY R EPRESENTATION AND L EARNING
Revised:  P OLICY R EPRESENTATION AND L EARNING

Original: In this section, we outline key details of the general Reinforcement Learning (RL) framework.
Revised: In this section, we outline key details of the general Reinforcement Learning (RL) framework.

Original: RL is formulated on the Markov Dynamic Process (MDP) framework: at every time step t , the world (including the agent) exists in a state s t ∈ S , wherein the agent is able to perform actions a t ∈ A , sampled from a policy π ( s t , a t ) which results in a new state s t +1 ∈ S according to the transition probability function T ( s t +1 | a t , s t ) .
Revised: RL is formulated on the Markov Dynamic Process (MDP) framework: at every time step t , the world (including the agent) exists in a state s t ∈ S , wherein the agent is able to perform actions a t ∈ A , sampled from a policy π ( s t , a t ) which results in a new state s t +1 ∈ S according to the transition probability function T ( s t +1 | a t , s t ) .

Original: Performing action a t in state s t produces a reward r t from the environment; the expected future discounted reward from executing a policy π is:
Revised: Performing action a t in state s t produces a reward r t from the environment; the expected future discounted reward from executing a policy π is:

Original: [Equation]
Revised: [Equation]

Original: where T is the maximum time horizon, and γ is the discount factor, indicating the planning horizon length.
Revised: where T is the maximum time horizon, and γ is the discount factor, indicating the planning horizon length.

Original: The agent’s goal is to optimize its policy, π ( ·| θ π ) , by maximizing J ( π ) .
Revised: The agent’s goal is to optimize its policy, π ( ·| θ π ) , by maximizing J ( π ) .

Original: The policies in the work are trained using the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017).
Revised: The policies in the work are trained using the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017).

Original: The value function is trained using TD( λ ).
Revised: The value function is trained using TD( λ ).

Original: 3. H IERARCHICAL R EINFORCEMENT L EARNING
Revised: 3. H IERARCHICAL R EINFORCEMENT L EARNING

Original: In HRL the original MDP is split intotwo different MDPs that should be easier to solve seperately.
Revised: In HRL the original MDP is separated into different MDPs that should be easier to solve separately.

Original: In practice, this is accomplished by learning two different policies.
Revised: In practice, this is accomplished by learning two different policies in two different layers.

Original: The lower level policy is trained ﬁrst and is often conditioned on a latent variable or goal g .
Revised: The lower level policy is trained ﬁrst and is often conditioned on a latent variable or goal g .

Original: The lower level policy π lo ( a | s, g ) is constructed in a way to give it temporally correlated behaviour depending on the g .
Revised: The lower level policy π lo ( a | s, g ) is constructed in a way to give it temporally correlated behaviour depending on the g .

Original: After the lower level policy is trained it is used to help solve the original MDP using a seperate policy
Revised: After the lower level policy is trained, it is used to help solve the original MDP using a separate policy

Original: π hi ( g | s ) .
Revised: π hi ( g | s ) .

Original: This policy learns to provide goals to the lower policy to maxmize rewards.
Revised: This policy learns to provide goals to the lower policy to maximize rewards.

Original:  M ULTI -A GENT R EINFORCEMENT L EARNING
Revised:  M ULTI -A GENT R EINFORCEMENT L EARNING

Original: The extension to the MDP framework for MARL is a partially observable Markov game (Littman, 1994).
Revised: The extension to the MDP framework for MARL is a partially observable Markov game (Littman, 1994).

Original: A Markov game has a collection of N agents, each with its own set of actions A 0 , . . .
Revised: A Markov game has a collection of N agents, each with its own set of actions A 0 , . . .

Original: , A N and  O 0 , . . . , O Nthat are partial observations of the full state space S .
Revised: , A N and partial observations O 0 , . . . , O N of the full state space S .

Original: Each agent i has its own policy π ( a | o i , θ i ) that models the probability of selecting an action.
Revised: Each agent i has its own policy π ( a | o i , θ i ) that models the probability of selecting an action.

Original: The environment transition function is a function of the full state and each agent’s actions T ( S (cid:48) | S, A 0 , . . . ,
Revised: The environment transition function is a function of the full state and each agent’s actions T ( S (cid:48) | S, A 0 , . . . ,

Original: A N ) .
Revised: A N ) .

Original: Each agent i receives a reward r i for taking a particular action  given a partial observation
Revised: Each agent i receives a reward r i for taking a particular action a i given a partial observation

Original: o i and its objective is to maximize this reward over time (cid:80)
Revised: o i and its objective is to maximize this reward over time (cid:80)

Original: Tt =

Revised: Tt =

Original: γ t r ti , where γ is the discount factor and T is the time horizon.
Revised: γ t r ti , where γ is the discount factor and T is the time horizon.

Original: The policy gradient can be computed for each agent as
Revised: The policy gradient can be computed for each agent as

Original: [Equation]
Revised: [Equation]

Original: where d θ = (cid:82) S (cid:80) Tt =

Revised: where d θ = (cid:82) S (cid:80) Tt =

Original: γ t p 0 ( o 0 )( o 0 → o | t, π 0 ) do 0 is the discounted state distribution, p 0 ( o ) represents the initial state distribution, and p 0 ( o 0 )( o 0 →
Revised: γ t p 0 ( o 0 )( o 0 → o | t, π 0 ) do 0 is the discounted state distribution, p 0 ( o ) represents the initial state distribution, and p 0 ( o 0 )( o 0 →

Original: o | t, π 0 ) models the likelihood of reaching state s by starting at state o 0 and following the policy π ( a, o | θ π ) for T steps (Silver et al., 2014).
Revised: o | t, π 0 ) models the likelihood of reaching state s by starting at state o 0 and following the policy π ( a, o | θ π ) for T steps (Silver et al., 2014).

Original: Here
Revised: Here

Original: A π ( o, a ) represents the advantage function estimator GAE( λ )in (Schulman et al., 2016).
Revised: A π ( o, a ) represents the advantage function estimator GAE( λ ) (Schulman et al., 2016).

Original: The challenge in MARL is that each agent learns separately and this often results in a non-stationary learning problem.
Revised: The challenge in MARL is that each agent learns separately, and this often results in a non-stationary learning problem.

Original: Each agent is learning how to estimate the dynamics and advantage of an environment that is affected by other evolving agents.
Revised: Each agent is learning how to estimate the dynamics and advantage of an environment that is affected by other actively learning agents.

Original: Data collected and used for learning becomes inaccurate after any other agent’s training update.
Revised: Data collected and used for learning becomes inaccurate after any learning update; thus, the problem is non-stationary.

Original: We propose a method for reducing the inaccuracies between updates by using HRL.
Revised: MADDPG reformulates the above problem into a fully observable problem with a centralized Q-function using the equation: [Equation] Here Q πi ( o , a ) is a centralized action-valued function that gives the Q value for agent i . Where o = { o 0 t , . .

Revised: , o N t } is the observations of all agents and a = { a 0 t , . .

Revised: . , a N t }

Revised: all agent actions at time t and θ = { θ 0 , . .

Revised: , θ N } is the collection of policy parameters for each agent.

Revised: This framework will allow the method to train the policies together, but it treats the agents as disjoint distributions to optimize.

Revised: Instead, we propose a decentralized method that does not need to learn a complex Qfunction and reduces the optimization challenges by using HRL.

Original: 4. M ULTI -A GENT H IERARCHICAL R EINFORCEMENT L EARNING
Revised: 4. T ASK - AGNOSTIC L OCOMOTION C ONTROLLER (LC)

Revised: The LC, the lower-level policy in our design, is designed to learn a robust and diverse policy π ( a t | o t , g t , θ lot ) based on a latent goal g t varaible.

Revised: The footstep goals g L = { ˆ p 0 , ˆ p 1 , ˆ θ root } in Figure 1, consist of the agent root relative distances of the next two footsteps on the ground plane and the desired facing direction at the ﬁrst step’s end.

Revised: This goal description is motivated by work that shows people may plan steering decisions two foot placements ahead (Zaytsev et al., 2015).

Revised: The LC learns to place its feet as accurately as it can to match these step goals using the reward r

Revised: Lg = exp( − 0 .

Revised: 2 || s gchar − g L || 2 ) .

Revised: The better the LC learns this behaviour, the more responsive the controller will be to provided goals.

Revised: 4. M ULTI -A GENT H IERARCHICAL R EINFORCEMENT L EARNING

Original: We construct a multi-agent learning structure that takes advantage of hierarchical design, or MultiAgent Hierarchical Reinforcement Learning (MAHRL).
Revised: We construct a multi-agent learning structure that takes advantage of hierarchical design or MultiAgent Hierarchical Reinforcement Learning (MAHRL).

Original: Each agent has its own higher level policy (Multi-Agent Navigation Controller (NC))
Revised: Each agent has its own higher level policy (Multi-Agent Navigation Controller (NC))

Original: π ( g | o, θ hii ) and a shared  lower level policy (Locomotion Controller (LC)) π ( a | o, g, θ lo ) .
Revised: π ( g | o, θ hii ) and a shared task agnositic lower level policy (LC) π ( a | o, g, θ lo ) .

Original: This method allows us to introduce more structure into the difﬁclut multi-agent optimization problem.
Revised: This method allows us to introduce more structure into the difﬁclut multi-agent optimization problem.

Revised: This change alters the underlying MDP, such that the policy is queried for a new action every k timesteps.

Revised: This also changes the MDP method by reducing the dimensoinality of the action space to speciﬁying goals g while using the low-level policy to produce more temporally consistent behaviour in the original action space.

Revised: The low-level policy is used in a deterministic manner to futher reduce variance introduced into the problem.

Original: We will use the notation π ( a i
Revised: We will use the notation π ( a i

Original: | o
Revised: | o

Original: i , θ hii , θ lo ) to denote the policy induced by the pair of policies.
Revised: i , θ hii , θ lo ) to denote the policy induced by the pair of policies.

Original: While current research shows that it is challenging but possible to train a two-level hierarchy concurrently (Nachum et al., 2018; ? ), we instead pretrain the lower level policy and leave it ﬁxed and shared across all agents.
Revised: While current research shows that it is challenging but possible to train a two-level hierarchy concurrently (Nachum et al., 2018; Levy et al., 2017b), we instead pretrain the lower level policy and leave it ﬁxed and shared across all agents.

Original: This reduces the MARL problem from learning the details of locomotion via joint torques for each agent to learning goal-based footstep plans for each agent.
Revised: This reduces the MARL problem from learning the details of locomotion via joint torques for each agent to learning goal-based footstep plans for each agent.

Original: The use of HRL is key to the method.
Revised: The use of HRL is key to the method.

Original: When the challenge in MARL is dealing with what can be large changes in the distribution of states visited by the agent the use of a temporally correlated[Figure] structure  signiﬁcantly reduces the non-stationarity.
Revised: When the challenge in MARL is dealing with what can be large changes in the distribution of states visited by the agent, the use of a temporally correlated structure given by the goal-conditioned LC signiﬁcantly reduces the non-stationarity.

Original: Not only is each agent sharinga portion of its network parameters with each other agent,  this portion has  been carefully constructed to provide structured exploration for the task–greatly reducing the number of network parameters that need to be learned and transforming the action space.
Revised: Not only is each agent sharing its network parameters with each other agent, but this portion has also been carefully constructed to provide structured exploration for the task, thus greatly reducing the number of network parameters that need to be learned. This is in contrast to centralized methods that take a step away from the goal of solving the heterogeneous problem in a scalable way.

Original: We extend the analysis of this work by combining the PPO algorithm in a MARL partial parameter sharing setting.
Revised: We extend the analysis of this work by combining the PPO algorithm in a MARL partial parameter sharing setting.

Original: Details of the algorithm can be found in the supplementary material.
Revised: Details of the algorithm can be found in the supplementary material.

Original: This modiﬁcation controls the way d θ i
Revised: The use of the LC controls the way d θ i

Original: ( o i ) can change for each agent making it easier for each agent to estimate other agents potential changes in behaviour.
Revised: ( o i ) can change for each agent making it easier for each agent to estimate other agents potential changes in behaviour because the LC is already trained to produce a speciﬁc type of useful behaviour that is a subset of the full space.

Revised: We also create a version of MADDPG that uses the HRL structure and show that it does not perform as well.

Original:  L EARNING H IERARCHICAL M ULTI -A GENT N AVIGATION C ONTROL
Revised:  L EARNING H IERARCHICAL M ULTI -A GENT N AVIGATION C ONTROL

Original: To solve the hierarchical learning problem, we train in a bottom-up fashion sharing the LC policy among heterogenous NC policies.
Revised: To solve the hierarchical learning problem, we train in a bottom-up fashion sharing the LC policy among heterogenous NC policies.

Original: To do this, the levels of the hierarchy communicate through shared actions and state in a feedback loop that is built to reﬂect biological biomechanical locomotion system in humans.
Revised: The levels of the hierarchy communicate through shared actions and state in a feedback loop that is meant to reﬂect human locomotion.

Original: The NC’s objective is to provide footstep placement goals a H =
Revised: The NC’s objective is to provide footstep placement goals a H =

Original: g L for the LC as seen in the right hand side of Figure 1.
Revised: g L for the LC as seen in the right hand side of Figure 1.

Original: These footstep goals are produced as two-step plans.
Revised: These footstep goals are produced as two-step plans.

Original: Each step is parameterized with its root relative placement, angle on the ground, centre of mass heading vector, and a time signature.
Revised: Each step is parameterized with its root-relative placement, angle on the ground, centre of mass heading vector, and a time signature.

Original: The NC is queried for a new footstep action every 0.5 s .
Revised: The NCs are queried for a new footstep action every 0.5 s .

Original: The NC decides what action to take based on the egocentric velocity ﬁeld E in front of the agent, its pose s char and the NC goal g H ,  C=
Revised: The NCs decides what action to take based on the egocentric velocity ﬁeld E in front of the agent, its pose s char and the NC goal g H , which together form the NC state C

Original:  { E, s char , g H } , seen in the left box of Figure 1.
Revised: = { E, s char , g H } , seen in the left box of Figure 1.

Original: 5. S TATE S PACE
Revised: 5. S TATE S PACE

Original: In several studies, it has been shown that optic ﬂow, an inherently egocentric phenomenon of mapping velocities to regions of vision, is key to sighted locomotion (Bruggeman et al., 2007; Warren Jr et al., 2001).
Revised: In several studies, it has been shown that optic ﬂow, an inherently egocentric phenomenon of mapping velocities to regions of vision, is key to sighted locomotion (Bruggeman et al., 2007; Warren Jr [Figure]

Original:  Additionally, the  ﬁeldof vision directly impacts walking ability and locomotion stability (Jansen et al., 2011).
Revised: et al., 2001). Additionally, the visual ﬁeld directly impacts walking ability and locomotion stability (Jansen et al., 2011).

Original: Taken together, vision’s role in locomotion forms an egocentric velocity ﬁeld where perceived distance and movement play different roles in locomotion control (Turano et al., 2005).
Revised: Taken together, vision’s role in locomotion forms an egocentric velocity ﬁeld where perceived distance and movement play different roles in locomotion control (Turano et al., 2005).

Original: This evidence has been used previously to deﬁne multi-agent navigation models, both by constructing a discretized egocentric ﬁeld (Kapadia et al., 2009) and by learning the discretization of an egocentric ﬁeld (Long et al., 2017).
Revised: This evidence has been used previously to deﬁne multi-agent navigation models, both by constructing a discretized egocentric ﬁeld (Kapadia et al., 2009) and by learning the discretization of an egocentric ﬁeld (Long et al., 2017).

Original: The NC uses as input an egocentric velocity ﬁeld relative to the agent’s location and rotation.
Revised: The NC uses as input an egocentric velocity ﬁeld relative to the agent’s location and rotation.

Original: This egocentric velocity ﬁeld E is 32 × 32  over a space of 5x5 m , starting 0 .
Revised: This egocentric velocity ﬁeld E is 32 × 32 samples over a space of 5x5 m , starting 0 .

Original: 5 m behind the agent and extending 4 .
Revised: 5 m behind the agent and extending 4 .

Original: 5 m in front, shown in  Figure 1(a).
Revised: 5 m in front, shown in the left hand side of Figure 1.

Original: The velocity ﬁeld consist of two image channels in the x and y direction of a square area directly in front of the agent, where each sample is calculated as the velocity relative to the agent (Bruggeman et al., 2007; Warren Jr et al., 2001).
Revised: The velocity ﬁeld consists of two image channels in the x and y directions of a square area directly in front of the agent, where each sample is calculated as the velocity relative to the agent (Bruggeman et al., 2007; Warren Jr et al., 2001).

Original: The current pose of the agent is included next, followed by the NC goal.
Revised: The current pose of the agent is included next, followed by the NC goal.

Original: The NC goal g H} consists of two values, the agent relative direction and distance to the current spatial goal location. Both the egocentric velocity ﬁeld sampling and the goal location state features can be seen in the left box of Figure 1. 5. R EWARD Navigation combines the desire to move towards goals, while avoiding collisions with other objects, in an energy efﬁcient way.
Revised: The NC goal g H

Original: The methods purpose is to implicitly learn low effort local turning behaviour while avoiding collisions without the need to explicitly describe its operation.

Original: To elicit this behaviour, a combination of reward signals are used.

Original: Primarily, a dense reward is used to encourage the agent to walk in the direction of its current goal.

Original: [Equation]

Original: where v com is the agent’s velocity and u Ttar is a normalized vector in the direction of the goal.

Original: [-For this work, a desired speed of v com = 1 . 
-]
Revised: [-For this work, a desired speed of v com = 1 . 
-]

Original: m/s is used.

Original: A directional reward is not enough to encourage the agent to proactively reach its goal.

Original: To reinforce the importance of goal reaching behaviour, a large reward r goal for reaching the goal is added–being within √ 2 m .

Original: In this work, the value r goal = 20 is used and comes from horizon =

Original: 1 / (1 − γ ) and the maximum reward the agent can otherwise receive is max ( r d ) = 1 , a reward that will be more important than travelling to the goal should be at least horizon ∗ max ( r d ) .

Original: This reward greatly increases the goal reaching behaviour.

Original: However, it can have the unintended effect of making the agent seek its goal aggressively, by trampling other agents.

Original: To reduce this behaviour, a repulsive cost was added when the agent is within √ 3 m other agents

Original: [Equation]

Original: where A is the set of agents in the simulation, and dist ( a, a (cid:48) ) computes the Euclidean distance between the Centre of Mass (COM) of agents a and a (cid:48) .

Original: We empirically found r s = 2 .

Original: [-5 and l = 3 . 
-]
Revised: [-5 and l = 3 . 
-]

Original: work well for deﬁning a distance dependent penalty in this case.

Original: This is intended to roughly approximate the power law of pedestrian interactions (Karamouzas et al., 2014).

Original: A similar repulsive cost is applied between the agent a andobstacles OB in the scene r Hb ( a ) = (cid:80) ob ∈ OB − 2 .
Revised: consists of two values, the agent relative direction and the distance to the current spatial goal location

Original: 5 for each obstacle within 1 m of a .

Original: The high reward for reaching the goal makes the agent very singleminded.

Original: The obstacle penalty is introduced to prevent the agents from using obstacles as affordances to regain balance.

Original: Many RL simulation environments use a ﬂag to indicate the episode or simulation end.

Original: This indicates that either the agent has reach its time limit or the agent has entered into an unrecoverable area of the state space, such as a fall.

Original: With multiple agents, the likelihood of a fall is high.

Original: It is not clear what to do when there are multiple agents being simulated.

Original: Terminating early, when one agent has fallen, is sub-optimal for other agents doing well, and waiting for every agent to fall wastes compute resources while most agents are collecting unhelpful data.

Original: We chose to reset the simulation when more than half the agents have fallen.

Original: However, fallen agents continue to act and need to be heavily penalized, so they receive a ﬁxed reward of − 5 –lower than any other reward.

Original: The NC’s ﬁnal reward function is a combination of the task rewards and behavioural costs:

Original: [Equation]

Original: 5. S IMULATION E NVIRONMENT & T RAINING
Revised: 5. S IMULATION E NVIRONMENT & T RAINING

Original: We construct a collection of physics-based simulation tasks to evaluate the method on.
Revised: We construct a collection of physics-based simulation tasks to evaluate the method.

Original: At initialization, each agent is randomly rotated, and the initial velocities of the agent’s links are determined by a selected motion capture clip using ﬁnite differences and rotated to align with the agent’s reference frame.
Revised: At initialization, each agent is randomly rotated, and the initial velocities of the agent’s links are determined by a selected motion capture clip using ﬁnite differences and rotated to align with the agent’s reference frame.

Original: Goal locations are randomly generated in locations that are at least 1 m away from any obstacle.
Revised: Goal locations are randomly generated in locations that are at least 1 m away from any obstacle.

Original: Each agent is randomly placed in the scene such that it does not intersect with any other agent or obstacle.
Revised: Each agent is randomly placed in the scene such that it does not intersect with any other agent or obstacle.

Original: The number and density of agents in the simulation vary depending on the task.
Revised: The number and density of agents in the simulation vary depending on the task.

Original: For training, we found that starting with 3 agents in the environment is a good trade-off between computation cost and the generalization ability of the resulting learned policy.
Revised: For training, we found that starting with 3 agents in the environment is a good trade-off between computation cost and the generalization ability of the resulting learned policy.

Original: Environment speciﬁcs are given in Table 1.
Revised: Environment speciﬁcs are given in Table 1.

Original: We consider the simulation and training environment to be, to the best of our knowledge, another novel contribution.
Revised: We consider the simulation and training environment to be, to the best of our knowledge, another novel contribution.

Original: While some simulators exist that support physics-based simulation for robots (Brockman et al., 2016; Tassa et al., 2018), few support more than one agent, with at most 2 . Other libraries focus on supporting different kinds of MARL conﬁgurations for particle-based [Table] right
Revised: While some simulators exist that support physics-based simulation for robots (Brockman et al., 2016; Tassa et al., 2018), few support more than one agent, with at most 2 which interact beyond physical simulation and actually solve their tasks with respect to each other.

Original: [Table] around [Figure] agents (Lowe et al., 2017).
Revised: Other libraries focus on supporting different kinds of MARL conﬁgurations for particle-based agents (Lowe et al., 2017).

Original: Our proposed approach represents the ﬁrst physics-based simulation of its kind that supports MARL.
Revised: Our proposed approach represents the ﬁrst physics-based simulation of its kind that supports MARL.

Original: The NC uses convolutional layers followed by dense layers.
Revised: The NC uses convolutional layers followed by dense layers.

Original: The particular network used is as follows: 16 convolutional ﬁlters of size 6 × 6 and stride 2 × 2 , 16 convolutional ﬁlters of size 3 × 3 and stride 1 × 1 , the structure is ﬂattened and the character and goal features s char , g H are concatenated, a dense layer of 256 units and a dense layer of 128 units are used at the end.The network uses
Revised: The particular network used is as follows: 16 convolutional ﬁlters of size 6 × 6 and stride 2 × 2 , 16 convolutional ﬁlters of size 3 ×and stride 1 × 1 , the structure is ﬂattened and the character and goal features s char , g H are concatenated, a dense layer of 256 units and a dense layer of 128 units are used at the end.

Original:  Rectiﬁed Linear Unit (ReLU) activations throughout except for after the last layer which uses a tanh activation that outputs values between [ − 1 , 1] .
Revised: The network uses Rectiﬁed Linear Unit (ReLU) activations throughout except for after the last layer which uses a tanh activation that outputs values between [ − 1 , 1] .

Original: All network inputs are standardized with respect to all states observed so far.
Revised: All network inputs are standardized with respect Name [Table] size

Revised: [Table]

Revised: location

Original: The rewards are scaled by a running variance.
Revised: Pursuit [ [Table] [Table] random random

Revised: Soccer

Revised: [Table]

Revised: [Table]

Revised: towards centre + N (

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ) none

Revised: Mall [3 , 5] [

Revised: [Table]

Revised: 10] 10 ×

Revised: [Table]

Revised: random random

Revised: Bottleneck [3 , 5] 10 × 20 m right + N (0 , 0 . 15) around

Original:  That is, the variance is computed from a running computation during training that is updated after every data collection step.
Revised: [Figure] to all states observed so far. The rewards are scaled by a running variance. That is, the variance is computed from a running computation during training that is updated after every data collection step.

Original: The batch size used for PPO is 256  with a smaller batch size of 64 for the value function.
Revised: The batch size used for PPO is 256 , with a smaller batch size of 64 for the value function.

Original: The policy learning-rate and value function learning-rate are 0 . 000 and 0 .
Revised: The policy learning-rate and value function learning-rate are 0 . 0001 and 0 .

Original: 001  respectively.
Revised: 001 , respectively.

Original: The value function is updated four times for every policy update.
Revised: The value function is updated four times for every policy update.

Original: The NC also uses the Adam stochastic gradient optimization method (Kingma & Ba, 2014) to train the Artiﬁcial Neural Network (ANN) parameters.
Revised: The NC also uses the Adam stochastic gradient optimization method (Kingma & Ba, 2014) to train the Artiﬁcial Neural Network (ANN) parameters.

Original:  R ESULTS
Revised:  R ESULTS

Original: In this section, we demonstrate the efﬁcacy of the proposed method.
Revised: In this section, we demonstrate the efﬁcacy of MAHRL.

Revised: We introduce a new set of challenging multi-agent simulations and show that the previous method can not solve these tasks.

Revised: However, using MAHRL many of these tasks can be solved with a modest amount of data and computational power.

Original: We separate our evaluation into two sections.
Revised: We separate our evaluation into four sections.

Original: First, we examine the performance of the model in terms of training and learned policies.
Revised: We examine the performance of MAHRL in terms of training, quantitative metrics, and learned policies.

Original: Second, we examine the performanceof our arpproach in terms of computation cost and generalizability with respect to the number of agents in the environment. 
Revised: Then we examine the performance in terms of computation cost and generalizability over the number of agents in the environment. [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Original: 6. L EARNING
Revised: 6. L EARNING

Original: We evaluate MAHRL in complex multi-agent environments by ﬁrst examining the performance of the reward function over training episodes.
Revised: We evaluate MAHRL in complex multi-agent environments by ﬁrst examining the performance of the reward function over training episodes.

Original: We show that MAHRL performs much better than the basic PPO algorithm without any hierarchical structure in the 5-agent mall in Figure 2(a).
Revised: We show that MAHRL performs much better than the basic PPO algorithm without any hierarchical structure in the 5-agent Mall in Figure 2a.

Original: After a lengthy training session the basic PPO is not able to even produce a standing behaviour.
Revised: After a lengthy training session, the basic PPO is not able to even produce a standing behaviour.

Original: We ﬁnd that
Revised: We ﬁnd that

Original: MAHRL performs well and is able to learn navigation behaviour after a short amount of training.
Revised: MAHRL performs well and is able to learn navigation behaviour after a short amount of training.

Original: We ﬁnd that the hierarchical design provides a signiﬁcant improvement in this case.
Revised: We ﬁnd that the hierarchical design provides a signiﬁcant improvement in this case.

Original: We also compare  MADDPG  and ﬁnd that it does better than PPO but strugles to produce good  behaviour.
Revised: We also compare to MADDPG in Figure 2a,3a and ﬁnd that, while it does better than PPO as it also uses the HRL structure, it struggles to produce good coordinated behaviour.

Original: We beleive this is related to the large Q-network that needs to be created for MADDPG.
Revised: We believe this is related to the large Q-network that needs to be learned for MADDPG.

Original: In Figure 2(b) we show the learning curves training MAHRL with a full humanoid model.

Original: Even with this increased control complexity out method is able to learn succesful navigation strategies shown in Figure 5.
Revised: Qualitatively, throughout training, even with the increased control complexity, our method is able to learn successful navigation strategies shown in Figure 8.

Original: Each agent in the scenario quickly develops strong navigation behaviours that become more conservative over time as agents value avoiding collisions.
Revised: Each agent in the scenario quickly develops strong navigation behaviours that become more conservative over time as agents value avoiding collisions.

Original: This can result in agents taking longer paths in the environment.
Revised: This can result in agents taking longer paths in the environment.

Revised: MAHRL, is also applied to a Pursuit environment.

Revised: In this environment, there is one agent (agent 0) with the same navigation goal as in previous environments.

Revised: As well, two additional agents (pursuer 0,1) that have the goal of reaching agent 0.

Revised: This is accomplished by setting a high-level goal g H for the pursuers to the location of agent 0.

Original: [Figure]
Revised: In Figure 3a, we show a comparative analysis of the learning curves of MAHRL, MADDPG, and PPO. We note that qualitatively, the three agents all begin to increase their average reward via their navigation objective, as learning progresses the pursuing agents outperform agent 0, shown in Figure 3b and as they get better agent 0 has an increasingly difﬁcult time reaching its own navigation targets. An example of this behaviour is shown in Figure 7. Last MAHRL is applied to a 2-on-2 soccer simulation, Soccer shown in Figure 4b.

Revised: In this simulation, there are two teams of 2 bipedal agents.

Revised: The goal of the members of each team is to kick the ball into the other team’s goal area (shown in green).

Revised: The task is particularly challenging to learn as each agent now has to learn with one cooperative agent and two adversarial agents.

Revised: We ﬁnd that MAHRL works well at learning policies for agents to locate and kick the ball into the opposing team’s goal.

Revised: The learning curves for two different methods that use MAHRL are shown in Figure 4a, on based on PPO and another based on TD3.

Revised: [Figure]

Revised: ) ego ﬁeld x (b) ego ﬁeld y

Original: By extracting the gradients on the input features for the value function in the learned model we can examine some artifacts of what is learned.
Revised: By extracting the gradients on the input features for the value function in the learned model, we can examine some artifacts of what is learned.

Original: Recall, that for input we include in the state of the NC a simple model of an egocentric perceptual ﬁeld–a square region in front of the agent.
Revised: Recall that for input, we include in the state of the NC a simple model of an egocentric perceptual ﬁeld– a rectangular region in front of the agent.

Original: We show that our models learns two important aspects of navigation with respect to this ﬁeld.
Revised: We show that our models learns two important aspects of navigation with respect to this ﬁeld.

Original: The magnitude of the velocity ﬁeld gradients in our learned model reveal that the learning process has developed an egocentric velocity ﬁeld attenuated with distance.
Revised: The magnitude of the velocity ﬁeld gradients in our learned model reveals that the learning process has developed an egocentric velocity ﬁeld attenuated with distance.

Original: This can be seen in Figure 3a.
Revised: This can be seen in Figure 5a.

Original: Interestingly,  this ﬁeld is biased toward the rightward direction (down in the ﬁgures).
Revised: Interestingly, [Figure] this ﬁeld is biased toward the rightward direction (down in the ﬁgures).

Original: This bias supports reciprocal collision avoidance in counter ﬂows.
Revised: This bias supports reciprocal collision avoidance in counter ﬂows.

Original: Tofurther evaluate MAHRL, it is applied to a pursuit environment. In this environment there is one agent (agent 0) with the same navigation goal as in previous environments.
Revised: To evaluate learned navigation strategies quantitatively, we capture the mean number of collision events overall agents for each episode in several instantiations of the Mall environment. We perform this study over the state-of-the-art MADDPG method, MAHRL with and without Heterogeneous agents, and PPO.

Original: As well, 2 additional agents (pursuer 0,1) that have the goal of reaching agent 0.
Revised: The results are shown in Figure 6a. For each model, we perform 155 policy rollouts over several random seeds. The basic PPO algorithm does not appear to learn anything. A KruskalWallis rank-sum test and post-hoc Conover’s test with both FDR and Holm corrections for ties show the MAHRL methods signiﬁcantly outperform others (p ¡ 0.01).

Original: This is accomplished by setting a highlevel goal g H for the pursuers to the location of agent 0. In Figure 2(c), the three agents all begin to increase their average reward via their navigation objective, as learning progresses the [-pursuing agents outperform agent 
-]
Revised: To evaluate the pursuit and evade strategies quantitatively, we capture the mean number of collision events overall agents for each episode in several instantiations of the [-pursuing agents outperform agent 
-]game. We perform this study over the MADDPG method, MAHRL, and PPO. The results can be seen in Figure 6.

Original:  and as they get better agent 0 has an increasingly difﬁcult time reaching its own navigation targets.
Revised: For each model, we run 155 simulations over several random seeds. A Kruskal-Wallis rank-sum test and post-hoc

Original: An example of this behaviour is shown in Figure 4.
Revised: Conover’s test with FDR and Holm corrections show the MAHRL and MADDPG methods signiﬁcantly outperform PPO ( p < 0 . 01) . However, MAHRL is not signiﬁcantly different from MADDPG ( p = 0 . 45) , but the distribution is skewed lower than MADDPG with more zero collision samples. 6. Q UALITATIVE R ESULTS

Original: To evaluate the NC qualitatively we show that agents learn to navigate arbitrary environments while avoiding collisions with both obstacles and other agents.
Revised: To evaluate the NC policies qualitatively, we show that agents learn to navigate arbitrary scenarios while avoiding collisions with both obstacles and other agents.

Original: First, we show a rasterized version of an example episode from the pursuit environment in Figure 4 where agents learn to corner and tackle.
Revised: First, we show a rasterized version of an example episode from the Pursuit environment in Figure 7 where agents learn to corner and tackle.

Original: Then, we show that agents can successfully and continuously navigate complicated environments of [Figure] [Figure]
Revised: Then, we show that agents can successfully and continuously navigate complicated environments of forced interactions, as seen in Figure 9a (Kapadia et al., 2011).

Original: T i m e i n M i nu t e s
Revised: Finally, robust clogging and trampling are shown in both low and high-density bottleneck egress scenarios, respectively Figure 9b.

Original: [Figure]
Revised: [Figure]

Original: forced interactions as seen in Figure 6a (Kapadia et al., 2011). Finally, robust clogging and trampling is shown in both low and high density bottleneck egress scenarios respectively Figure 6b.
Revised: [Figure]

Original: 6. C OMPUTATION AND G ENERALIZATION
Revised: 6. C OMPUTATION AND G ENERALIZATION

Original: The deep integration of physical character control and distributed multi-agent navigation comes with a cost directly dependent on the number of active agents.
Revised: The deep integration of physical character control and distributed multi-agent navigation comes with a cost directly dependent on the number of active agents.

Original: In this section, we show two results in the same experiment, the computational costs of increasing the number of agents and the model generalization to more difﬁcult scenarios.
Revised: In this section, we show two results in the same experiment, the computational costs of increasing the number of agents and the model generalization to more difﬁcult scenarios.

Original: For two scenarios, mall and bottleneck, the number of agents is increased, and we record the average reward and computation time from the simulation.
Revised: For two scenarios, mall and bottleneck, the number of agents is increased and we record the average reward and computation time from the simulation.

Original: The agent-computation curve in Figure 7(a) indicates a linear trend in computational cost.
Revised: The agent-computation curve in Figure 10(a) indicates a linear trend in computational cost.

Original: While at agents counts in the 20 s the simulation is not real-time, the most computational expensive part is not the learning system but the physics simulation.
Revised: While at agents counts in the 20 s the simulation is not real-time, the most computationally expensive part is not the learning system but the physics simulation.

Original: Computationally efﬁcient articulated body simulation is an active area of research (Erez et al., 2015).
Revised: Computationally efﬁcient articulated body simulation is an active area of research (Erez et al., 2015).

Original: For accurate and stable simulation we use a physics time-step of 0 .
Revised: For accurate and stable simulation, we use a physics time-step of 0 .

Original: 0005 s .
Revised: 0005 s .

Original: To evaluate the learned policies ability to generalize with group size, we vary the number of agents for select tasks after a homogenious policy has been trained.
Revised: To evaluate the learned policy’s ability to generalize with group size, we vary the number of agents for select tasks after a homogeneous policy has been trained.

Original: The longer the policy can maintain a high average reward the better the generalization.
Revised: The longer the policy can maintain a high average reward, the better the generalization.

Original: In addition, the average reward for two different types of policy training styles are compared.
Revised: In addition, the average reward for two different types of policy training styles is compared.

Original: The ﬁrst method trains on a single task at a time, the second method uses multi-task learning in hopes that a more generalizable task-independent structure is aquired.
Revised: The ﬁrst method trains on a single task at a time; the second method uses multi-task learning in hopes that a more generalizable task-independent structure is acquired.

Original: The multi-task method, often preferring to optimize easier tasks, does not appear to learn more robust policies compared to the scenario-space based method (Kapadia et al., 2011).
Revised: The multi-task method, often preferring to optimize easier tasks, does not appear to learn more robust policies compared to the scenario-space based method (Kapadia et al., 2011).

Original: All generalizaiton results can be seen in Figure 7(b).
Revised: All generalization results can be seen in Figure 10(b).

Original: However, generalization remains a known and open issue of DRL methods (Zhang et al., 2018)
Revised: However, generalization remains a known and open issue of DRL methods (Zhang et al., 2018)

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we present a novel method where, for the ﬁrst time, multi-agent navigation and physical character control are integrated.
Revised: In this paper, we present a novel method where, for the ﬁrst time, multi-agent navigation and physical character control are integrated.

Original: Multiple heterogenious interacting agents can experience physical interactions, handle physical perturbations, and produce robust physical phenomenon like falling and slipping.
Revised: Multiple heterogeneous interacting agents can experience physical interactions, handle physical perturbations, and produce robust physical phenomena like falling and slipping.

Original: To achieve this, we developed an integrated model of MARL and HRL with partial parameter sharing .
Revised: To achieve this, we developed an integrated model of MARL and HRL with partial parameter sharing .

Original: The evaluation of this approach shows how valuable it is for addressing the non-stationary learning problem of MARL in complex multi-agent scenarios.
Revised: The evaluation of this approach shows how valuable it is for addressing the non-stationary learning problem of MARL in complex multi-agent scenarios.

Original: In particular, the heterogeneous pursuer scenario learns complex routing and tackling behaviours akin to state-of-the-art competitive self-play approaches.
Revised: In particular, the heterogeneous Pursuit and Soccer scenarios learn complex routing and tackling behaviours akin to state-of-the-art competitive self-play approaches.

Original: While our method produces promising results, the work is limited by the ﬁxed LC partial parameter sharing.
Revised: While our method produces promising results, the work is limited by the ﬁxed LC partial parameter sharing.

Original: There is room for research in the area of training the LC and NC concurrently.
Revised: There is room for research in the area of training the LC and NC concurrently.

Original: Still, theproposed method beneﬁts greatly from this stability.
Revised: For the

Original: For the NC, we introduced a set of reward functions to encourage human-like behaviour while navigating with other agents.
Revised: NC, we introduced a set of reward functions to encourage human-like behaviour while navigating with other agents.

Original: These rewards are motivated from the literature  but balancing them is its own challenge.
Revised: The literature motivates these rewards, but balancing them is its own challenge.

Original: In the future, it may be beneﬁcial to use additional data-driven imitation terms to encourage human-like paths.
Revised: In the future, it may be beneﬁcial to use additional data-driven imitation terms to encourage humanlike paths.

Original: Finally, great effort was made to create an agent that is robust to the number of agents in the simulation.
Revised: Finally, considerable effort was made to create combined locomotion, navigation, and behaviour controller that is robust to the number of agents in the simulation.

Original: However, robust generalization remains an open problem.
Revised: However, robust generalization remains an open problem.

Original:  S UPPLEMENTARY M ATERIAL 8. L OCOMOTION C ONTROLLER 
Revised:  S UPPLEMENTARY M ATERIAL 8. L OCOMOTION C ONTROLLER 8. LC D ETAILS 8.2. R EWARD

Original: The Low-Level Controller (LLC) is designed to solve the problem of coordinating joint torques over time to mimic selected motion capture data and facilitate robust balance and locomotion.

Original: The LLC is queried at a frequency of 30 Hz to produce joint target angles for per-joint PD controllers.

Original: The details of this approach are outlined in the following sections.

Original: 8. R EWARD

Original: We found that a combination of imitation and end effector rewards leads to a robust and reactive motion.
Revised: We found that a combination of imitation and end effector rewards leads to a robust and reactive motion.

Original: The goal is to reward joint angle targets which result in mimicking source motions.
Revised: The goal is to reward joint angle targets which result in mimicking source motions.

Original: The combination of these competing rewards leads to a complex balancing problem of many objectives– trading off imitation behaviour for footstep matching.
Revised: The combination of these competing rewards leads to a complex balancing problem of many objectives– trading off imitation behaviour for footstep matching.

Original: 8.2. I MITATION
Revised: 8.2. I MITATION

Original: The imitation method is predicated on time indexed 0 .
Revised: The imitation method is predicated on time indexed 0 .

Original: 5 s long reference motions
Revised: 5 s long reference motions

Original: q i ( t ) created from motion capture data and augmented with footstep parameterized labels.
Revised: q i ( t ) created from motion capture data and augmented with footstep parameterized labels.

Original: The difference between the agent and the desired pose to imitate is computed as the weighted difference between the time indexed pose and the agent r Lp
Revised: The difference between the agent and the desired pose to imitate is computed as the weighted difference between the time indexed pose and the agent r Lp

Original: = exp( − 0 .
Revised: = exp( − 0 .

Original: 5 (cid:80)
Revised: 5 (cid:80)

Original: i || s ichar
Revised: i || s i

Original:  − q i  t ) || 2 ) .
Revised: char − q i ( t ) || 2 ) .

Original: The motion database contains 7 s of segmented motion that coarsely covers the space of possible footstep positions and the motions necessary to realize those footsteps.
Revised: The motion database contains 7 s of segmented motion that coarsely covers the space of possible footstep positions and the motions necessary to realize those footsteps.

Original: The accuracy and diversity of the LLC’s motion can be increased or made more robust with additional motion if desired, including stylized, disordered, running, and more.
Revised: The accuracy and diversity of the Low-Level Controller (LLC)’s motion can be increased or made more robust with additional motion if desired, including stylized, disordered, running, and more.

Original: 8.2. F OOTSTEP GOAL

Original: The footstep goals g L = { ˆ p 0 , ˆ p 1 , ˆ θ root } in Figure 1(c), consist of the agent root relative distances of the next two footsteps on the ground plane and the desired facing direction at the ﬁrst step’s end.

Original: This goal description is motivated by work that shows people may plan steering decisions two foot placements ahead Zaytsev et al.

Original: The LLC is designed to learn to place its feet as accurately as it can to match these step goals using the reward

Original: r Lg = exp( − 0 .

Original: 2 || s gchar − g L || 2 ) .

Original: The better the LLC learns this behaviour, the stronger the communication channel between it and the High-Level Controller (HLC) will be.

Original: However, if the agent matches these footstep goals too well, the smoothness of the motion will degrade.
Revised: However, if the agent matches these footstep goals too well, the smoothness of the motion will degrade.

Original: A balance is struck between learning robust footstep placement strategies and perfect mimicry–the reference motions and foot placements guide naturalness and should not be over-ﬁtted.
Revised: A balance is struck between learning robust footstep placement strategies and perfect mimicry–the reference motions and foot placements guide naturalness and should not be over-ﬁtted.

Original: 8.2. E ND EFFECTOR
Revised: 8.2. E ND EFFECTOR

Original: An additional reward is given to encourage the policy to match the end-effector positions in the imitation motion r Le = exp( − 0 .
Revised: An additional reward is given to encourage the policy to match the end-effector positions in the imitation motion r Le = exp( − 0 .

Original: 15 (cid:80) e || s echar − q e
Revised: 15 (cid:80) e || s echar − q e

Original: ( t )
Revised: ( t )

Original: || 2 ) .
Revised: || 2 ) .

Original: Reward is also given for how well the agent matches the imitated motion’s COM r Lcom = exp − 0 .
Revised: Reward is also given for how well the agent matches the imitated motion’s Centre of Mass (COM) r Lcom = exp( − 0 .

Original: 2 (cid:80) ||
Revised: 2 (cid:80) ||

Original: s comchar − q com ( t ) || 2 )
Revised: s comchar − q com ( t ) || 2 )

Original: 8.2. T ORQUE PENALTY
Revised: 8.2. T ORQUE PENALTY

Original: To smooth the actions generated from the control policy, a reward for maximizing the above rewards with minimal torque cost is used r Lτ = exp( − 0 .
Revised: To smooth the actions generated from the control policy, a reward for maximizing the above rewards with minimal torque cost is used r Lτ = exp( − 0 .

Original: 2 (cid:80) i τ ( s ichar )) .
Revised: 2 (cid:80) i τ ( s ichar )) .

Original: These torques are normalized by speciﬁed torque limits that keep the agent from displaying unrealistic strength.
Revised: These torques are normalized by speciﬁed torque limits that keep the agent from displaying unrealistic strength.

Original: 8. T RAINING & S IMULATION E NVIRONMENT
Revised: 8. T RAINING & S IMULATION E NVIRONMENT

Original: The LLC state includes the proprioceptive-like joint information.
Revised: The LLC state includes the proprioceptive-like joint information.

Original: In particular, the components of s char consist of the COM relative distances and velocities of each links as well as the rotation and angular velocities of each link as shown in Figure 1(d).
Revised: In particular, the components of s char consist of the COM relative distances and velocities of each links as well as the rotation and angular velocities of each link as shown in Figure 1(d).

Original: The trained policy actions are in the form of angle-axis targets for per-joint Proportional Derivative (PD) controllers.
Revised: The trained policy actions are in the form of angle-axis targets for per-joint Proportional Derivative (PD) controllers.

Original: In this way, the action is used to set the desired position of each joint of the agent.
Revised: In this way, the action is used to set the desired position of each joint of the agent.

Original: This can be seen in Figure 1(e).
Revised: This can be seen in Figure 1(e).

Original: The method is also able to learn policies for agents of different types.
Revised: The method is also able to learn policies for agents of different types.

Original: In Figure ?? we additionally show the method using a full humanoid LLC character.
Revised: In Figure 8 we additionally show the method using a full humanoid LLC character.

Original: The agent is simulated with the physics-based rigid body simulation environment Bullet Bullet (2015).
Revised: The agent is simulated with the physics-based rigid body simulation environment Bullet Bullet (2015).

Original: The policy is trained in an on-policy fashion.
Revised: The policy is trained in an on-policy fashion.

Original: At the beginning of each episode a goal g L is sampled from the simulation and updated every 0 .
Revised: At the beginning of each episode a goal g L is sampled from the simulation and updated every 0 .

Original: 5 s .
Revised: 5 s .

Original: Episodes end when either a time limit T is reached or the agent falls.
Revised: Episodes end when either a time limit T is reached or the agent falls.

Original: Between training rounds 4096 transition tuples are collected by simulating episodes in parallel.
Revised: Between training rounds 4096 transition tuples are collected by simulating episodes in parallel.

Original: After enough tuples have been collected 16 minibatch updates are performed with a batch size of 256 .
Revised: After enough tuples have been collected 16 minibatch updates are performed with a batch size of 256 .

Original: The value function is updated 64 with a minibatch size of 64 .
Revised: The value function is updated 64 with a minibatch size of 64 .

Original: For the policy and value function.
Revised: For the policy and value function.

Original: The policy and value function are both modeled using a ANN with 2 hidden layers of size 512 and 256 .
Revised: The policy and value function are both modeled using a ANN with 2 hidden layers of size 512 and 256 .

Original: The network uses ReLU activations through except for after the last layer where a tanh activation that outputs values between [ − 1 , 1] is used.
Revised: The network uses ReLU activations through except for after the last layer where a tanh activation that outputs values between [ − 1 , 1] is used.

Original: The Adam stochastic gradient optimization method is used to train the ANN parameters Kingma & Ba (2014).
Revised: The Adam stochastic gradient optimization method is used to train the ANN parameters Kingma & Ba (2014).

Original: The learning rates for the policy and value function are 0 .
Revised: The learning rates for the policy and value function are 0 .

Original: 000 and 0 .
Revised: 0001 and 0 .

Original: 001 respectively.
Revised: 001 respectively.

Original: Details related to the learning algorithm can be found in the supplementary material.
Revised: Details related to the learning algorithm can be found in the supplementary material.

Original: Many previous methods have created robust controllers via imitation.
Revised: Many previous methods have created robust controllers via imitation.

Original: These methods were not intended to be used in crowds or dynamic environments with potentially random disturbances from other interacting agents in the simulation.
Revised: These methods were not intended to be used in crowds or dynamic environments with potentially random disturbances from other interacting agents in the simulation.

Original: Given that we intend to use this agent in a physical multiagent simulation where the agent may bump into other agents or obstacles when trained in a crowded setting, we apply additional methods to simulate pushes that may be encountered.
Revised: Given that we intend to use this agent in a physical multiagent simulation where the agent may bump into other agents or obstacles when trained in a crowded setting, we apply additional methods to simulate pushes that may be encountered.

Original: That is, random pushes between 50 − 150 N are applied every 3 − 5 s for a duration of 0 .
Revised: That is, random pushes between 50 − 150 N are applied every 3 − 5 s for a duration of 0 .

Original: 1 − 0 .
Revised: 1 − 0 .

Original: 3 s to increase the robustness of the controller.
Revised: 3 s to increase the robustness of the controller.

Original: Similarly, the motions in the motion database start from the same facing direction.
Revised: Similarly, the motions in the motion database start from the same facing direction.

Original: To make the agent more robust to distributed crowd scenarios, the agent’s initial facing direction is randomized during training.
Revised: To make the agent more robust to distributed crowd scenarios, the agent’s initial facing direction is randomized during training.

Original: 8. L EARNING D ETAILS
Revised: 8.3. NC R EWARD

Revised: Navigation combines the desire to move towards goals, while avoiding collisions with other objects, in an energy efﬁcient way.

Revised: The methods purpose is to implicitly learn low effort local turning behaviour while avoiding collisions without the need to explicitly describe its operation.

Revised: To elicit this behaviour, a combination of reward signals are used.

Revised: Primarily, a dense reward is used to encourage the agent to walk in the direction of its current goal.

Revised: [Equation]

Revised: where v com is the agent’s velocity and u Ttar is a normalized vector in the direction of the goal.

Original: {+For this work, a desired speed of v com = 1 . 
+}
Revised: {+For this work, a desired speed of v com = 1 . 
+}

Revised: m/s is used.

Revised: A directional reward is not enough to encourage the agent to proactively reach its goal.

Revised: To reinforce the importance of goal reaching behaviour, a large reward r goal for reaching the goal is added–being within √ 2 m .

Revised: In this work, the value r goal = 20 is used and comes from horizon =

Revised: 1 / (1 − γ ) and the maximum reward the agent can otherwise receive is max ( r d ) = 1 , a reward that will be more important than travelling to the goal should be at least horizon ∗ max ( r d ) .

Revised: This reward greatly increases the goal reaching behaviour.

Revised: However, it can have the unintended effect of making the agent seek its goal aggressively, by trampling other agents.

Revised: To reduce this behaviour, a repulsive cost was added when the agent is within √ 3 m other agents

Revised: [Equation]

Revised: where A is the set of agents in the simulation, and dist ( a, a (cid:48) ) computes the Euclidean distance between the COM of agents a and a (cid:48) .

Revised: We empirically found r s = 2 .

Original: {+5 and l = 3 . 
+}
Revised: {+5 and l = 3 . 
+}

Revised: work well for deﬁning a distance dependent penalty in this case.

Revised: This is intended to roughly approximate the power law of pedestrian interactions (Karamouzas et al., 2014).

Revised: A similar repulsive cost is applied between the agent a and obstacles OB in the scene

Revised: r

Revised: Hb ( a ) =

Revised: (cid:80) ob ∈ OB − 2 .

Revised: 5 for each obstacle within 1 m of a .

Revised: The high reward for reaching the goal makes the agent very single-minded.

Revised: The obstacle penalty is introduced to prevent the agents from using obstacles as affordances to regain balance.

Revised: Many RL simulation environments use a ﬂag to indicate the episode or simulation end.

Revised: This indicates that either the agent has reach its time limit or the agent has entered into an unrecoverable area of the state space, such as a fall.

Revised: With multiple agents, the likelihood of a fall is high.

Revised: It is not clear what to do when there are multiple agents being simulated.

Revised: Terminating early, when one agent has fallen, is sub-optimal for other agents doing well, and waiting for every agent to fall wastes compute resources while most agents are collecting unhelpful data.

Original: The modiﬁcation to this algorithm for the MARL method is to change line 7 − 8 to execute an action for each agent in the simulation in parallel.
Revised: We chose to reset the simulation when more than half the agents have fallen. However, fallen agents continue to act and need to be heavily penalized, so they receive a ﬁxed reward of − 5 which represents the largest penalty in our system. The NC’s ﬁnal reward function is a combination of the task rewards and behavioural costs: [Equation]

Original: The Inputs are standardized wrt to all states seen so far.
Revised: if fallen

Original: The rewards are divided by the variance.
Revised: [Equation] Hd + r Ha ( a ) + r Hb ( a ) otherwise.

Original: The variance is computed from a running computation during training that is updated after every data collection step. The advantage is batch normed.
Revised: 8. L EARNING D ETAILS

Original: 1: Randomly initialize model parameters θ π and θ v 2: while not done do 3: for i ∈ { 0 , . . .
Revised: 1: Randomly initialize model parameters θ π and θ v 2: while not done do 3: for i ∈ { 0 , . . .

Original: N } do 4:
Revised: N } do 4:

Original: τ i ← {} 5: for t ∈ { 0 , . . .
Revised: τ i ← {} 5: for t ∈ { 0 , . . .

Original: , T } do 6: a t ← π ( ·| s t , θ π ) 7: s t
Revised: , T } do 6: a t ← π ( ·| s t , θ π ) 7: s t

Original: +1 , r t ← execute a t in environment 8: τ i,t ← <
Revised: +1 , r t ← execute a t in environment 8: τ i,t ← <

Original: s t , a t , r t > 9: s t ← s t + 10: end for 11: end for 12:
Revised: s t , a t , r t > 9: s t ← s t + 10: end for 11: end for 12:

Original: Update value function V π ( · ) parameters θ v using { τ 0 , . .
Revised: Update value function V π ( · ) parameters θ v using { τ 0 , . .

Original: . , τ N } 13:
Revised: . , τ N } 13:

Original: Update policy parameters θ π using { τ 0 , . .
Revised: Update policy parameters θ π using { τ 0 , . .

Original: . , τ N } 14: end while
Revised: . , τ N } 14: end while

Original: [Figure]
Revised: [Figure]

Revised: The modiﬁcation to this algorithm for the MARL method is to change line 7 − 8 to execute an action for each agent in the simulation in parallel.

Revised: The Inputs are standardized wrt to all states seen so far.

Revised: The rewards are divided by the variance.

Revised: The variance is computed from a running computation during training that is updated after every data collection step.

Original:  Hyper Parameter Exploration Parameter exploration is a key process in acheiving the best results for DRL methods.
Revised: The advantage is batch normed. Hyper Parameter Exploration Parameter exploration is a key process in acheiving the best results for DRL methods.

Original: In this work we explored different network architectures, policy variances, annealing the variance, learning rates and activation types.
Revised: In this work we explored different network architectures, policy variances, annealing the variance, learning rates and activation types.

Original: The results with the best performance are reported in the paper.
Revised: The results with the best performance are reported in the paper.

Original: 8. R ESULTS D ETAILS 8.5. LC H YPERPARAMETER E XPLORATION
Revised: 8. R ESULTS D ETAILS 8.5. LC H YPERPARAMETER E XPLORATION

Original: The LC hyperparameter exploration learning curves in Figure 8 illustrate the process of ﬁnding optimal training parameters.
Revised: The LC hyperparameter exploration learning curves in Figure 11 illustrate the process of ﬁnding optimal training parameters.

Original: It was found that annealing the policy variance over time did not increase the learning efﬁciency.
Revised: It was found that annealing the policy variance over time did not increase the learning efﬁciency.

Original: As well, the best settings for the policy variance and kl threshold are 0 .
Revised: As well, the best settings for the policy variance and kl threshold are 0 .

Original: 2 and 0 .
Revised: 2 and 0 .

Original: 25 respectively
Revised: 25 respectively.

Original: . 8.5. M ULTI - AGENT N AVIGATION U NIT T ASKS
Revised: [Figure] 8.5. M ULTI - AGENT N AVIGATION U NIT T ASKS

Original: In this section, we show several multi-agent navigation tasks and their qualitative performance using our method.
Revised: In this section, we show several multi-agent navigation tasks and their qualitative performance using our method.

Original: These are rendered as birds-eye views of scenarios with each agents trajectory over time rendered.
Revised: These are rendered as birds-eye views of scenarios with each agents trajectory over time rendered.

Original: Each agents goal is rendered as a red point within a red circle and the shortest linear path to that from the agent’s current position is rendered as a red line.
Revised: Each agents goal is rendered as a red point within a red circle and the shortest linear path to that from the agent’s current position is rendered as a red line.

Original: We show that agent’s learn to navigate towards arbitrary goals repeatedly hitting its mark with each new goal in Figure 9.
Revised: We show that agent’s learn to navigate towards arbitrary goals repeatedly hitting its mark with each new goal in Figure 12.

Original: Predictive reciprocal collision avoidance is important in multi-agent navigation and lends naturalness to[Figure] qualitative results.
Revised: Predictive reciprocal collision avoidance is important in multi-agent navigation and lends naturalness to qualitative results.

Original: We show our method learns high value reipcrocal collision avoidane strategies, and we overlay the sample points for the velocity state space to illustrate how this is acheived in
Revised: We show our method learns high value reipcrocal collision avoidane strategies, and we overlay the sample points for the velocity state space to illustrate how this is acheived in

Original: Figure 10.
Revised: Figure 13.

Original: We show that this generalizes to the introduction of obstacles in both Figure 11 & 12. [Figure]
Revised: We show that this generalizes to the introduction of obstacles in both Figure 14 & 15.

Original: 8. A DDITIONAL D ISCUSSION 8.6. LC A GILITY
Revised: 8. A DDITIONAL D ISCUSSION 8.6. LC A GILITY

Original: While often in the crowd simulation literature it is common to have agents that can turn on point oror produce holonomic motions, this is unrealistic for articulated agents.
Revised: While often in the crowd simulation literature it is common to have agents that can turn on point or produce holonomic motions, this is unrealistic for articulated agents.

Original: Interestingly, the agent learned stopping and in-place turning behaviour which was not contained in the imitation data the LC was trained on, indicating the system can generate behaviours beyond its design.
Revised: Interestingly, the agent learned stopping and in-place turning behaviour which was not contained in the imitation data the LC was trained on, indicating the system can generate behaviours beyond its design.

Original: However, the agents in this work do not make unrealistic sharp turns.
Revised: However, the agents in this work do not make unrealistic sharp turns.

Original: This is in part related to the NC being able to avoid falls, however there are many other factors:
Revised: This is in part related to the NC being able to avoid falls, however there are many other factors:

Original: The LC motion capture data, g L selection during
Revised: The LC motion capture data, g L selection during

Original: LC training and that the LC was not trained with other agents.
Revised: LC training and that the LC was not trained with other agents.

Original: Progress in any of these areas can improve the responsiveness of the LC and is left for future work.
Revised: Progress in any of these areas can improve the responsiveness of the LC and is left for future work.

Original:  While the LC is goal driven, this goal is only based on foot placement, a fully interactive agent can have many other types of short term goals, including where to put one’s hands to manipulate items in the environment, like doors.
Revised: [Figure] While the LC is goal driven, this goal is only based on foot placement, a fully interactive agent can have many other types of short term goals, including where to put one’s hands to manipulate items in the environment, like doors.