Original: E MERGENT C OMPLEXITY VIA M ULTI -A GENT C OMPETITION
Revised: E MERGENT C OMPLEXITY VIA M ULTI -A GENT C OMPETITION

Original: Trapit Bansal ∗
Revised: Trapit Bansal ∗

Original: UMass Amherst
Revised: UMass Amherst

Original: Jakub Pachocki
Revised: Jakub Pachocki

Original: OpenAI
Revised: OpenAI

Original: Szymon Sidor
Revised: Szymon Sidor

Original: OpenAI
Revised: OpenAI

Original: Ilya Sutskever
Revised: Ilya Sutskever

Original: OpenAI
Revised: OpenAI

Original: Igor Mordatch
Revised: Igor Mordatch

Original: OpenAI
Revised: OpenAI

Original: A BSTRACT
Revised: A BSTRACT

Original: Reinforcement learning algorithms can train agents that solve problems in complex, interesting environments.
Revised: Reinforcement learning algorithms can train agents that solve problems in complex, interesting environments.

Original: Normally, the complexity of the trained agent is closely related to the complexity of the environment.
Revised: Normally, the complexity of the trained agent is closely related to the complexity of the environment.

Original: This suggests that a highly capable agent requires a complex environment for training.
Revised: This suggests that a highly capable agent requires a complex environment for training.

Original: In this paper, we point out that a competitive multi-agent environment trained with self-play can produce behaviors that are far more complex than the environment itself.
Revised: In this paper, we point out that a competitive multi-agent environment trained with self-play can produce behaviors that are far more complex than the environment itself.

Original: We also point out that such environments come with a natural curriculum, because for any skill level, an environment full of agents of this level will have the right level of difﬁculty.
Revised: We also point out that such environments come with a natural curriculum, because for any skill level, an environment full of agents of this level will have the right level of difﬁculty.

Original: This work introduces several competitive multi-agent environments where agents compete in a 3D world with simulated physics.
Revised: This work introduces several competitive multi-agent environments where agents compete in a 3D world with simulated physics.

Original: The trained agents learn a wide variety of complex and interesting skills, even though the environment themselves are relatively simple.
Revised: The trained agents learn a wide variety of complex and interesting skills, even though the environment themselves are relatively simple.

Original: The skills include behaviors such as running, blocking, ducking, tackling, fooling opponents, kicking, and defending using both arms and legs.
Revised: The skills include behaviors such as running, blocking, ducking, tackling, fooling opponents, kicking, and defending using both arms and legs.

Original: A highlight of the learned behaviors can be found here: https://goo.gl/eR7fbX.
Revised: A highlight of the learned behaviors can be found here: https://goo.gl/eR7fbX.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement Learning (RL) is exciting because good reinforcement learning algorithms exist (Mnih et al., 2015;
Revised: Reinforcement Learning (RL) is exciting because good reinforcement learning algorithms exist (Mnih et al., 2015;

Original: Silver et al., 2016; Schulman et al., 2015a;
Revised: Silver et al., 2016; Schulman et al., 2015a;

Original: Mnih et al., 2016; Schulman et al., 2015b; Lillicrap et al., 2015; Schulman et al., 2017), allowing us to train agents that accomplish a great variety of interesting tasks.
Revised: Mnih et al., 2016; Schulman et al., 2015b; Lillicrap et al., 2015; Schulman et al., 2017), allowing us to train agents that accomplish a great variety of interesting tasks.

Original: We can train an agent to play Atari games from pixels (Mnih et al., 2015) or get humanoids to walk (Schulman et al., 2017).
Revised: We can train an agent to play Atari games from pixels (Mnih et al., 2015) or get humanoids to walk (Schulman et al., 2017).

Original: RL is exciting partly because it is easy to envision an RL algorithm producing a broadly competent agent when trained on an appropriate curriculum of environments.
Revised: RL is exciting partly because it is easy to envision an RL algorithm producing a broadly competent agent when trained on an appropriate curriculum of environments.

Original: In general, training an agent to perform a highly complex task requires a highly complex environment, and these can be difﬁcult to create.
Revised: In general, training an agent to perform a highly complex task requires a highly complex environment, and these can be difﬁcult to create.

Original: However, there exists a class of environments where the behavior produced by the agents can be far more complex than the environments; this is the class of the competitive multi-agent environments trained with self-play.
Revised: However, there exists a class of environments where the behavior produced by the agents can be far more complex than the environments; this is the class of the competitive multi-agent environments trained with self-play.

Original: Such environments have two very attractive properties: (1) Even very simple competitive multi-agent environments can produce extremely complex behaviors.
Revised: Such environments have two very attractive properties: (1) Even very simple competitive multi-agent environments can produce extremely complex behaviors.

Original: For example, the game of Go has very simple rules, but the strategies needed to win are extremely complex.
Revised: For example, the game of Go has very simple rules, but the strategies needed to win are extremely complex.

Original: This is because the complexity of these environments is produced by the competing agents that act in it.
Revised: This is because the complexity of these environments is produced by the competing agents that act in it.

Original: Thus, as the other agents become more competent, the environment effectively becomes more complex.
Revised: Thus, as the other agents become more competent, the environment effectively becomes more complex.

Original: (2) When trained with self-play, the competitive multi-agent environment provides the agents with a perfect curriculum.
Revised: (2) When trained with self-play, the competitive multi-agent environment provides the agents with a perfect curriculum.

Original: This happens because no matter how weak or strong an agent is, an environment populated with other agents of comparable strength provides the right challenge to the agent, facilitating maximally rapid learning and avoiding getting stuck.
Revised: This happens because no matter how weak or strong an agent is, an environment populated with other agents of comparable strength provides the right challenge to the agent, facilitating maximally rapid learning and avoiding getting stuck.

Original: Self-play in competitive multi-agent environments is not a new idea – it has already been explored in TD-gammon (Tesauro, 1995) and reﬁned in AlphaGo (Silver et al., 2016) and Dota 2 (OpenAI).
Revised: Self-play in competitive multi-agent environments is not a new idea – it has already been explored in TD-gammon (Tesauro, 1995) and reﬁned in AlphaGo (Silver et al., 2016) and Dota 2 (OpenAI).

Original: In both cases, the resulting behavior was far more complex than the environment itself, and the self-play approach provided the agents with a perfectly tuned curriculum for each task.
Revised: In both cases, the resulting behavior was far more complex than the environment itself, and the self-play approach provided the agents with a perfectly tuned curriculum for each task.

Original: In this paper, we investigate whether the idea of competitive multi-agent environments can yield fruit in other domains: speciﬁcally, in the domain of continuous control, where balance, dexterity, and manipulation are the key skills.
Revised: In this paper, we investigate whether the idea of competitive multi-agent environments can yield fruit in other domains: speciﬁcally, in the domain of continuous control, where balance, dexterity, and manipulation are the key skills.

Original: In more detail, we introduce several multi-agent tasks with competing goals in a 3D world with simulated physics, using the MuJoCo framework (Todorov et al., 2012), where the agents would need to learn highly developed motor skills in order to succeed in the competitive environment.
Revised: In more detail, we introduce several multi-agent tasks with competing goals in a 3D world with simulated physics, using the MuJoCo framework (Todorov et al., 2012), where the agents would need to learn highly developed motor skills in order to succeed in the competitive environment.

Original: We train the agents using a distributed implementation of a recent policy gradient algorithm, Proximal Policy
Revised: We train the agents using a distributed implementation of a recent policy gradient algorithm, Proximal Policy

Original: Optimization (Schulman et al., 2017).
Revised: Optimization (Schulman et al., 2017).

Original: By adding a simple exploration curriculum to aid exploration in the environment we ﬁnd that agents learn a high level of dexterity in order to achieve their goals, in particular we ﬁnd numerous emergent skills for which it may be difﬁcult to engineer a reward.
Revised: By adding a simple exploration curriculum to aid exploration in the environment we ﬁnd that agents learn a high level of dexterity in order to achieve their goals, in particular we ﬁnd numerous emergent skills for which it may be difﬁcult to engineer a reward.

Original: Speciﬁcally, the agents learned a wide variety of skills and behaviors that include running, blocking, ducking, tackling, fooling opponents, kicking, and defending using arms and legs.
Revised: Speciﬁcally, the agents learned a wide variety of skills and behaviors that include running, blocking, ducking, tackling, fooling opponents, kicking, and defending using arms and legs.

Original: Highlight of the learned behaviors on the various tasks can be found here: https://goo.gl/eR7fbX
Revised: Highlight of the learned behaviors on the various tasks can be found here: https://goo.gl/eR7fbX

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this section, we review some background on policy gradient methods, Proximal Policy Optimization and related work in the multi-agent reinforcement learning domain.
Revised: In this section, we review some background on policy gradient methods, Proximal Policy Optimization and related work in the multi-agent reinforcement learning domain.

Original: Notation: We consider multi-agent Markov games (Littman, 1994).
Revised: Notation: We consider multi-agent Markov games (Littman, 1994).

Original: A Markov game for N agents is a partially observable Markov decision process (MDP) deﬁned by: a set of states S describing the state of the world and the possible joint conﬁguration of all the agents, a set of observations
Revised: A Markov game for N agents is a partially observable Markov decision process (MDP) deﬁned by: a set of states S describing the state of the world and the possible joint conﬁguration of all the agents, a set of observations

Original: O 1 , . . .
Revised: O 1 , . . .

Original: , O N of each agent, a set of actions of each agent
Revised: , O N of each agent, a set of actions of each agent

Original: A 1 , . . . ,
Revised: A 1 , . . . ,

Original: A N , a transition function T :
Revised: A N , a transition function T :

Original: S ×
Revised: S ×

Original: A 1 · · · A N → S determining distribution over next states, and a reward for each agent i which is a function of the state and the agent’s action
Revised: A 1 · · · A N → S determining distribution over next states, and a reward for each agent i which is a function of the state and the agent’s action

Original: r
Revised: r

Original: i : S ×
Revised: i : S ×

Original: A i → R .
Revised: A i → R .

Original: Agents choose their actions according to a stochastic policy π θ i : O i ×
Revised: Agents choose their actions according to a stochastic policy π θ i : O i ×

Original: A i
Revised: A i

Original: →
Revised: →

Original: [0 , 1] , where θ i are the parameters of the policy.
Revised: [0 , 1] , where θ i are the parameters of the policy.

Original: For continuous control problems considered here, π θ is Gaussian where the mean and variance are deep neural networks with parameter θ .
Revised: For continuous control problems considered here, π θ is Gaussian where the mean and variance are deep neural networks with parameter θ .

Original: Each agent i aims to maximize its own total expected return
Revised: Each agent i aims to maximize its own total expected return

Original: R i =
Revised: R i =

Original: (cid:80) Tt =

Revised: (cid:80) Tt =

Original: γ t r it , where γ is a discount factor and T is the time horizon
Revised: γ t r it , where γ is a discount factor and T is the time horizon

Original: Policy Gradient: Policy gradient methods work by directly computing an estimate of the gradient of policy parameters in order to maximize the expected return using stochastic gradient descent.
Revised: Policy Gradient: Policy gradient methods work by directly computing an estimate of the gradient of policy parameters in order to maximize the expected return using stochastic gradient descent.

Original: These methods are behind much of the recent success in using deep neural networks for control (Schulman et al., 2015b; Heess et al., 2017;
Revised: These methods are behind much of the recent success in using deep neural networks for control (Schulman et al., 2015b; Heess et al., 2017;

Original: Lillicrap et al., 2015; Silver et al., 2016).
Revised: Lillicrap et al., 2015; Silver et al., 2016).

Original: Such methods are also attractive because they don’t require an explicit model of the world.
Revised: Such methods are also attractive because they don’t require an explicit model of the world.

Original: There are several different expressions for the policy gradient estimator which have the form g .. =
Revised: There are several different expressions for the policy gradient estimator which have the form g .. =

Original: E [ A t ∇ θ log π θ ] .
Revised: E [ A t ∇ θ log π θ ] .

Original: Different choices of A t lead to different algorithms, for example taking the sample return of a trajectory A t = (cid:80) t r t leads to the REINFORCE algorithm (Williams, 1992).
Revised: Different choices of A t lead to different algorithms, for example taking the sample return of a trajectory A t = (cid:80) t r t leads to the REINFORCE algorithm (Williams, 1992).

Original: However, such algorithms suffer from high variance in the gradient estimates and it’s typical to use a baseline, such as a value function baseline, to ameliorate the high variance.
Revised: However, such algorithms suffer from high variance in the gradient estimates and it’s typical to use a baseline, such as a value function baseline, to ameliorate the high variance.

Original: Generalized advantage estimation (Schulman et al., 2015b) takes this approach of using a learned value function to reduce variance at the cost of some bias and using an exponentially weighted estimator of the advantage function.
Revised: Generalized advantage estimation (Schulman et al., 2015b) takes this approach of using a learned value function to reduce variance at the cost of some bias and using an exponentially weighted estimator of the advantage function.

Original: Proximal Policy Optimization (PPO):
Revised: Proximal Policy Optimization (PPO):

Original: Achieving good results with policy gradient algorithms requires carefully tuning the step-size (Schulman et al., 2015a).
Revised: Achieving good results with policy gradient algorithms requires carefully tuning the step-size (Schulman et al., 2015a).

Original: Moreover, most policy gradient methods perform one gradient update per sampled trajectory and have high sample complexity.
Revised: Moreover, most policy gradient methods perform one gradient update per sampled trajectory and have high sample complexity.

Original: Recently, Schulman et al.
Revised: Recently, Schulman et al.

Original: (2017) proposed the PPO algorithm which addresses both these problems.
Revised: (2017) proposed the PPO algorithm which addresses both these problems.

Original: This uses a surrogate objective which is maximized while penalizing large changes to the policy.
Revised: This uses a surrogate objective which is maximized while penalizing large changes to the policy.

Original: Let l t ( θ ) =
Revised: Let l t ( θ ) =

Original: π θ ( a t | s t ) π θold ( a t | s t ) denote the likelihood ratio.
Revised: π θ ( a t | s t ) π θold ( a t | s t ) denote the likelihood ratio.

Original: Then PPO optimizes the objective:
Revised: Then PPO optimizes the objective:

Original: [Equation]
Revised: [Equation]

Original: t , clip ( l t ( θ ) , 1 − (cid:15), 1 + (cid:15) )
Revised: t , clip ( l t ( θ ) , 1 − (cid:15), 1 + (cid:15) )

Original: ˆ A t ) (cid:105) , where ˆ A t is the generalized advantage estimate and clip ( l t ( θ ) , 1 − (cid:15), 1 + (cid:15) )
Revised: ˆ A t ) (cid:105) , where ˆ A t is the generalized advantage estimate and clip ( l t ( θ ) , 1 − (cid:15), 1 + (cid:15) )

Original: clips l t ( θ ) in the interval [1 − (cid:15), 1 + (cid:15) ] .
Revised: clips l t ( θ ) in the interval [1 − (cid:15), 1 + (cid:15) ] .

Original: The algorithm alternates between sampling multiple trajectories from the policy and performing several epochs of SGD on the sampled dataset to optimize this surrogate objective.
Revised: The algorithm alternates between sampling multiple trajectories from the policy and performing several epochs of SGD on the sampled dataset to optimize this surrogate objective.

Original: Since the state value function is also simultaneously approximated, the error for the value function approximation is also added to the surrogate objective to compute the complete objective function (Schulman et al., 2017).
Revised: Since the state value function is also simultaneously approximated, the error for the value function approximation is also added to the surrogate objective to compute the complete objective function (Schulman et al., 2017).

Original: Related Work: Tan (1993) explored the multi-agent setting with independently learning agents using Q-learning, in particular exploring advantages of cooperative agents over independent agents in a 2D grid world.
Revised: Related Work: Tan (1993) explored the multi-agent setting with independently learning agents using Q-learning, in particular exploring advantages of cooperative agents over independent agents in a 2D grid world.

Original: This was further explored by Matignon et al.
Revised: This was further explored by Matignon et al.

Original: (2012) again in the cooperative setting.
Revised: (2012) again in the cooperative setting.

Original: A lot of the work on multi-agent RL is focused on cooperative settings, see Busoniu et al.
Revised: A lot of the work on multi-agent RL is focused on cooperative settings, see Busoniu et al.

Original: (2008) for a review of multi-agent RL and Panait & Luke (2005) for a review focused on cooperative settings.
Revised: (2008) for a review of multi-agent RL and Panait & Luke (2005) for a review focused on cooperative settings.

Original: Stanley & Miikkulainen (2004) trained agents in a competitive 2D world, using evolutionary strategies to evolve both weights and structure of policies with competition as a ﬁtness measure.
Revised: Stanley & Miikkulainen (2004) trained agents in a competitive 2D world, using evolutionary strategies to evolve both weights and structure of policies with competition as a ﬁtness measure.

Original: Tampuu et al.
Revised: Tampuu et al.

Original: (2017) studied the application of deep Q-learning to train Pong agents with competitive and collaborative rewarding schemes.
Revised: (2017) studied the application of deep Q-learning to train Pong agents with competitive and collaborative rewarding schemes.

Original: He et al.
Revised: He et al.

Original: (2016) used deep Q-learning to model competitive games where only one agent is learning and the Q network implicitly models the opponent.
Revised: (2016) used deep Q-learning to model competitive games where only one agent is learning and the Q network implicitly models the opponent.

Original: Silver et al.
Revised: Silver et al.

Original: (2016) used self-play with deep reinforcement learning techniques to master the game of Go.
Revised: (2016) used self-play with deep reinforcement learning techniques to master the game of Go.

Original: Sukhbaatar et al.
Revised: Sukhbaatar et al.

Original: (2017) introduced a self-play method for generating an automatic training curriculum in single-agent environments.
Revised: (2017) introduced a self-play method for generating an automatic training curriculum in single-agent environments.

Original: From a game-theoretic perspective, Heinrich &
Revised: From a game-theoretic perspective, Heinrich &

Original: Silver (2016) studied ﬁctitious self-play for achieving approximate Nash equilibrium in zero-sum games like Poker.
Revised: Silver (2016) studied ﬁctitious self-play for achieving approximate Nash equilibrium in zero-sum games like Poker.

Original: Recently, Foerster et al.
Revised: Recently, Foerster et al.

Original: (2017a) introduced an algorithm which explicitly accounts for the fact that the opponent is also learning and showed that it can achieve cooperation in iterated prisoner’s dilemma, however the algorithm requires access to the opponent’s parameters.
Revised: (2017a) introduced an algorithm which explicitly accounts for the fact that the opponent is also learning and showed that it can achieve cooperation in iterated prisoner’s dilemma, however the algorithm requires access to the opponent’s parameters.

Original: Recently, Lowe et al.
Revised: Recently, Lowe et al.

Original: (2017) and Foerster et al.
Revised: (2017) and Foerster et al.

Original: (2017b)
Revised: (2017b)

Original: proposed methods for centralized learning in multi-agent domains, where the idea is to use an actor-critic method with a central critic which can observe the joint state and actions of all agents in order to reduce variance, evaluating on 2D games and StarCraft.
Revised: proposed methods for centralized learning in multi-agent domains, where the idea is to use an actor-critic method with a central critic which can observe the joint state and actions of all agents in order to reduce variance, evaluating on 2D games and StarCraft.

Original: In this work, we do not rely on centralized training and address the variance problem by using very large batchsize through a distributed implementation of the PPO algorithm.
Revised: In this work, we do not rely on centralized training and address the variance problem by using very large batchsize through a distributed implementation of the PPO algorithm.

Original: Moreover, we study fully competitive settings in a 3D world with simulated physics whereas prior applications have focused on toy 2D worlds or game-theoretic problems.
Revised: Moreover, we study fully competitive settings in a 3D world with simulated physics whereas prior applications have focused on toy 2D worlds or game-theoretic problems.

Original: Recent work on learning dexterous locomotion skills in 3D environments by adding complexity in the agent’s environment (Heess et al., 2017) is also related.
Revised: Recent work on learning dexterous locomotion skills in 3D environments by adding complexity in the agent’s environment (Heess et al., 2017) is also related.

Original: However, whereas Heess et al.
Revised: However, whereas Heess et al.

Original: (2017) learn complex behaviours by engineering complexity into the environment design and by engineering dense reward functions for these environments, the resultant complexity in our work is due to the presence of other learning agents in a simple environment.
Revised: (2017) learn complex behaviours by engineering complexity into the environment design and by engineering dense reward functions for these environments, the resultant complexity in our work is due to the presence of other learning agents in a simple environment.

Original: Our work is also related to early work in the graphics community (Sims, 1994) on evolving creature morphology in varying environments using genetic algorithm and work in animation (Wampler et al., 2010) for adversarial games.
Revised: Our work is also related to early work in the graphics community (Sims, 1994) on evolving creature morphology in varying environments using genetic algorithm and work in animation (Wampler et al., 2010) for adversarial games.

Original: The competitive multi-agent learning framework is also related to generative adversarial networks (Goodfellow et al., 2014) and work on learning robust grasping policies through an adversary (Pinto et al., 2017).
Revised: The competitive multi-agent learning framework is also related to generative adversarial networks (Goodfellow et al., 2014) and work on learning robust grasping policies through an adversary (Pinto et al., 2017).

Original:  C OMPETITIVE E NVIRONMENTS
Revised:  C OMPETITIVE E NVIRONMENTS

Original: We introduce four competitive environments and experiment with two types of agents.
Revised: We introduce four competitive environments and experiment with two types of agents.

Original: In this we focus on two agent worlds, that is 1-vs-1 games, though these environments can be extended to include multiple agents for a mixed competitive and co-operative setup.
Revised: In this we focus on two agent worlds, that is 1-vs-1 games, though these environments can be extended to include multiple agents for a mixed competitive and co-operative setup.

Original: We will now describe the four environments and the competitive rewards in each environment.
Revised: We will now describe the four environments and the competitive rewards in each environment.

Original: Figure 1 shows a rendering of the environments.
Revised: Figure 1 shows a rendering of the environments.

Original: We consider two three-dimensional agent bodies: ant and humanoid .
Revised: We consider two three-dimensional agent bodies: ant and humanoid .

Original: The ant is a quadrupedal body with 12 DoF and 8 actuated joints.
Revised: The ant is a quadrupedal body with 12 DoF and 8 actuated joints.

Original: Humanoid has 23 DoF and 17 actuated joints.
Revised: Humanoid has 23 DoF and 17 actuated joints.

Original: Run to Goal: The agents start by facing each other in a 3D world and they each have goals on the opposite side of the word (see Fig.1a).
Revised: Run to Goal: The agents start by facing each other in a 3D world and they each have goals on the opposite side of the word (see Fig.1a).

Original: The agent that reaches its goal ﬁrst wins.
Revised: The agent that reaches its goal ﬁrst wins.

Original: Reaching the goal before the opponent gives a reward of +1000 to the agent and -1000 to the opponent.
Revised: Reaching the goal before the opponent gives a reward of +1000 to the agent and -1000 to the opponent.

Original: If no agent reaches its goal then they both get -1000.
Revised: If no agent reaches its goal then they both get -1000.

Original: You Shall Not Pass: This is the same world as the previous task, but one agent (the blocker) now has the objective of blocking the other agent from reaching it’s goal while not falling down.
Revised: You Shall Not Pass: This is the same world as the previous task, but one agent (the blocker) now has the objective of blocking the other agent from reaching it’s goal while not falling down.

Original: If the blocker is successful in preventing the opponent from reaching the goal and is standing at the end of episode then it gets +1000 reward, if it is not standing then it gets 0 reward, and the opponent gets -1000 reward.
Revised: If the blocker is successful in preventing the opponent from reaching the goal and is standing at the end of episode then it gets +1000 reward, if it is not standing then it gets 0 reward, and the opponent gets -1000 reward.

Original: If the opponent is successful in reaching it’s goal
Revised: If the opponent is successful in reaching it’s goal

Original: then it gets +1000 reward and the blocker gets -1000 reward.
Revised: then it gets +1000 reward and the blocker gets -1000 reward.

Original: Sumo: The agents compete on a round arena (see Fig.1c) and the goal of each agent is to either knock the other agent to the ground or to push them out of the ring.
Revised: Sumo: The agents compete on a round arena (see Fig.1c) and the goal of each agent is to either knock the other agent to the ground or to push them out of the ring.

Original: The winner gets +1000 and the other agent gets -1000.
Revised: The winner gets +1000 and the other agent gets -1000.

Original: If there is a draw then both agents get -1000.
Revised: If there is a draw then both agents get -1000.

Original: Kick and Defend: This a standard penalty shootout (see Fig.1d).
Revised: Kick and Defend: This a standard penalty shootout (see Fig.1d).

Original: One agent has to kick a ball through the goal, which has a ﬁxed width of 6 units, while the other agent defends.
Revised: One agent has to kick a ball through the goal, which has a ﬁxed width of 6 units, while the other agent defends.

Original: Successful kick or defend gives the agent +1000 reward and the opponent -1000 reward.
Revised: Successful kick or defend gives the agent +1000 reward and the opponent -1000 reward.

Original: The defender cannot go beyond the goal-keeping area which is a distance 3 units from the goal, doing so terminates the game with a penalty of -1000 for the defender.
Revised: The defender cannot go beyond the goal-keeping area which is a distance 3 units from the goal, doing so terminates the game with a penalty of -1000 for the defender.

Original: We give two additional rewards for defender: if defender is successful and it made contact with the ball then it gets additional +500 reward, and if the defender is successful and still standing at the end of the game then it gets another additional reward of +500.
Revised: We give two additional rewards for defender: if defender is successful and it made contact with the ball then it gets additional +500 reward, and if the defender is successful and still standing at the end of the game then it gets another additional reward of +500.

Original: We found the latter two rewards to yield more realistic looking defending behaviors.
Revised: We found the latter two rewards to yield more realistic looking defending behaviors.

Original:  T RAINING C OMPETITIVE A GENTS
Revised:  T RAINING C OMPETITIVE A GENTS

Original: In this section we describe the multi-agent training framework.
Revised: In this section we describe the multi-agent training framework.

Original: We use a policy gradient algorithm,
Revised: We use a policy gradient algorithm,

Original: Proximal Policy Optimization (PPO) (Schulman et al., 2017), described previously.
Revised: Proximal Policy Optimization (PPO) (Schulman et al., 2017), described previously.

Original: We adopt a decentralized training approach and use a distributed implementation of PPO for very large scale multi-agent training.
Revised: We adopt a decentralized training approach and use a distributed implementation of PPO for very large scale multi-agent training.

Original: This allows us to use really large batch-sizes during training ameliorating the variance problem to some extent while also aiding in exploration.
Revised: This allows us to use really large batch-sizes during training ameliorating the variance problem to some extent while also aiding in exploration.

Original: Our distributed PPO implementation is similar to the implementation of Heess et al.
Revised: Our distributed PPO implementation is similar to the implementation of Heess et al.

Original: (2017), where instead of the KL penalty we used the clipped objective as proposed in PPO (Schulman et al., 2017).
Revised: (2017), where instead of the KL penalty we used the clipped objective as proposed in PPO (Schulman et al., 2017).

Original: We do multiple rollouts in parallel for each agent and have separate optimizers for each agent.
Revised: We do multiple rollouts in parallel for each agent and have separate optimizers for each agent.

Original: We collect a large amount of rollouts from the parallel workers and for each agent optimize the objective with the collected batch on 4 GPUs.
Revised: We collect a large amount of rollouts from the parallel workers and for each agent optimize the objective with the collected batch on 4 GPUs.

Original: The approach is same as synchronous actor critic of Mnih et al.
Revised: The approach is same as synchronous actor critic of Mnih et al.

Original: Instead of estimating a truncated generalized advantage estimate (GAE) from a small number of steps per rollout, as in Schulman et al.
Revised: Instead of estimating a truncated generalized advantage estimate (GAE) from a small number of steps per rollout, as in Schulman et al.

Original: ; Heess et al.
Revised: ; Heess et al.

Original: , we estimate GAE from the full rollouts.
Revised: , we estimate GAE from the full rollouts.

Original: This is important as the competition reward is a sparse reward given at the termination of the episode.
Revised: This is important as the competition reward is a sparse reward given at the termination of the episode.

Original: There are further challenges in applying distributed PPO to train multiple competitive agents.
Revised: There are further challenges in applying distributed PPO to train multiple competitive agents.

Original: One is the problem of exploration with sparse reward and second is the choice of opponent during training which effects the stability of training.
Revised: One is the problem of exploration with sparse reward and second is the choice of opponent during training which effects the stability of training.

Original: We now turn our attention to these issues.
Revised: We now turn our attention to these issues.

Original: 4. E XPLORATION C URRICULUM
Revised: 4. E XPLORATION C URRICULUM

Original: The success of agents in the competitive games requires the agents to occasionally solve the task (i.e. win the competition) by random actions.
Revised: The success of agents in the competitive games requires the agents to occasionally solve the task (i.e. win the competition) by random actions.

Original: The probability of this happening in most games is minuscule as they require as a prerequisite some fundamental motor skills like the ability to walk.
Revised: The probability of this happening in most games is minuscule as they require as a prerequisite some fundamental motor skills like the ability to walk.

Original: For example, the only way a kicker in the kick-and-defend task would achieve any positive reward is if it moves towards the ball and causes sufﬁcient displacement to it so as to make it go past the goal boundaries which is also obstructed by a defender.
Revised: For example, the only way a kicker in the kick-and-defend task would achieve any positive reward is if it moves towards the ball and causes sufﬁcient displacement to it so as to make it go past the goal boundaries which is also obstructed by a defender.

Original: This is a problem of training from sparse reward which is an active area of current research (Andrychowicz et al., 2017).
Revised: This is a problem of training from sparse reward which is an active area of current research (Andrychowicz et al., 2017).

Original: To overcome this problem, we can use simple dense rewards at each step to allow the agents to learn basic motor skills initially.
Revised: To overcome this problem, we can use simple dense rewards at each step to allow the agents to learn basic motor skills initially.

Original: Such rewards have been previously researched for tasks like walking forward and standing up, see for e.g. Schulman et al.
Revised: Such rewards have been previously researched for tasks like walking forward and standing up, see for e.g. Schulman et al.

Original: (2015b)
Revised: (2015b)

Original: and
Revised: and

Original: Duan et al.
Revised: Duan et al.

Original: However, engineering such dense rewards for the competitive tasks is not straight forward.
Revised: However, engineering such dense rewards for the competitive tasks is not straight forward.

Original: Moreover, such engineered rewards defeat the purpose of the competitive multi-agent training as we would like the agents to beneﬁt from the natural curriculum arising from the multi-agent training.
Revised: Moreover, such engineered rewards defeat the purpose of the competitive multi-agent training as we would like the agents to beneﬁt from the natural curriculum arising from the multi-agent training.

Original: To overcome this chicken-and-egg problem, we instead propose to use a simple curriculum for training.
Revised: To overcome this chicken-and-egg problem, we instead propose to use a simple curriculum for training.

Original: We use a dense reward at every step in the beginning phase of the training to allow agents to learn basic motor skills, like walking forward or being able to stand, which would increase the probability of random actions from the agent yielding a positive reward.
Revised: We use a dense reward at every step in the beginning phase of the training to allow agents to learn basic motor skills, like walking forward or being able to stand, which would increase the probability of random actions from the agent yielding a positive reward.

Original: We refer to this reward as the exploration reward .
Revised: We refer to this reward as the exploration reward .

Original: The exploration reward is gradually annealed to zero, in favor of the competition reward, to allow the agents to train for the majority of the training using the sparse competition reward.
Revised: The exploration reward is gradually annealed to zero, in favor of the competition reward, to allow the agents to train for the majority of the training using the sparse competition reward.

Original: This is achieved using a linear annealing factor α .
Revised: This is achieved using a linear annealing factor α .

Original: So, at time-step t , if the exploration reward is s t , the
Revised: So, at time-step t , if the exploration reward is s t , the

Original: [Figure]
Revised: [Figure]

Original: competition reward is R and T is the termination time-step, then the reward is:
Revised: competition reward is R and T is the termination time-step, then the reward is:

Original: [Equation]
Revised: [Equation]

Original: This ameliorates the problem of exploration with the sparse reward, which is particularly tough in a 3D world with simulated physics and complex agents like humanoid, while still beneﬁting from training for the sparse competition reward for the majority of the training.
Revised: This ameliorates the problem of exploration with the sparse reward, which is particularly tough in a 3D world with simulated physics and complex agents like humanoid, while still beneﬁting from training for the sparse competition reward for the majority of the training.

Original: During a typical training run, the agents would train on the dense reward for only about 10-15% of the training epochs.
Revised: During a typical training run, the agents would train on the dense reward for only about 10-15% of the training epochs.

Original: The dense rewards used are described in the Appendix A and are generally composed of the following terms: distance to goal, velocity in x-direction, control cost, impact cost, standing reward.
Revised: The dense rewards used are described in the Appendix A and are generally composed of the following terms: distance to goal, velocity in x-direction, control cost, impact cost, standing reward.

Original: These rewards are adopted from existing work and we did not tune weights on the various reward terms.
Revised: These rewards are adopted from existing work and we did not tune weights on the various reward terms.

Original: In particular, it is important to note that there is no dense reward term for many of the complex emergent behaviors and we also show in the experiment section how the learned behaviors are affected if we do not anneal the dense reward to beneﬁt from optimizing the sparse competition reward.
Revised: In particular, it is important to note that there is no dense reward term for many of the complex emergent behaviors and we also show in the experiment section how the learned behaviors are affected if we do not anneal the dense reward to beneﬁt from optimizing the sparse competition reward.

Original: 4. O PPONENT S AMPLING
Revised: 4. O PPONENT S AMPLING

Original: In the competitive multi-agent framework, all agents are simultaneously training in opponent pairs.
Revised: In the competitive multi-agent framework, all agents are simultaneously training in opponent pairs.

Original: Thus, the skill of opponents encountered during training could have signiﬁcant impact on the learning of the agents.
Revised: Thus, the skill of opponents encountered during training could have signiﬁcant impact on the learning of the agents.

Original: We found that training agents against the most recent opponent leads to imbalance in training where one agent becomes more skilled than the other agent early in training and the other agent is unable to recover.
Revised: We found that training agents against the most recent opponent leads to imbalance in training where one agent becomes more skilled than the other agent early in training and the other agent is unable to recover.

Original: Fig.
Revised: Fig.

Original: 2a shows the rewards during training with this naive approach (for the “run to goal” task with ant).
Revised: 2a shows the rewards during training with this naive approach (for the “run to goal” task with ant).

Original: Instead, we found that training against random old versions of the opponent to work much better.
Revised: Instead, we found that training against random old versions of the opponent to work much better.

Original: Thus, during training, for each rollout for an agent we sample old parameters for the opponent.
Revised: Thus, during training, for each rollout for an agent we sample old parameters for the opponent.

Original: Fig.
Revised: Fig.

Original: 2b shows the rewards for agents trained using this strategy.
Revised: 2b shows the rewards for agents trained using this strategy.

Original: This leads to more stable training and more robust policies.
Revised: This leads to more stable training and more robust policies.

Original: We further analyze the effect of this opponent sampling in the experiments section.
Revised: We further analyze the effect of this opponent sampling in the experiments section.

Original: Note that for self-play this means that the policy at any time should be able to defeat random older versions of itself, thus ensuring continual learning.
Revised: Note that for self-play this means that the policy at any time should be able to defeat random older versions of itself, thus ensuring continual learning.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We train agents for the four competitive tasks using the training methods described previously.
Revised: We train agents for the four competitive tasks using the training methods described previously.

Original: Our aim is to show that competitive multi-agent training provides a natural curriculum during learning which allows agents to learn complex behaviors.
Revised: Our aim is to show that competitive multi-agent training provides a natural curriculum during learning which allows agents to learn complex behaviors.

Original: We provide additional training details and analyze various aspects of the competitive multi-agent training in this section.
Revised: We provide additional training details and analyze various aspects of the competitive multi-agent training in this section.

Original: A highlight of the learned behaviors can be seen in the videos.
Revised: A highlight of the learned behaviors can be seen in the videos.

Revised: Code for the environments as well as learned policy parameters for agents on all the environments are available: https://github.com/openai/multiagent-competition.

Original: 5. E XPERIMENTAL D ETAILS
Revised: 5. E XPERIMENTAL D ETAILS

Original: Policies and Value Functions: We compare both MLP and LSTM for the policies and the value functions.
Revised: Policies and Value Functions: We compare both MLP and LSTM for the policies and the value functions.

Original: MLP had 2 hidden layers with 128 units each.
Revised: MLP had 2 hidden layers with 128 units each.

Original: For LSTM networks, the input was ﬁrst projected to a 128 dimensional embedding using a fully connected layer with ReLU activation which is then fed into a single-layer LSTM with 128 hidden state dimension and the output is projected to the action dimension using another fully connected layer.
Revised: For LSTM networks, the input was ﬁrst projected to a 128 dimensional embedding using a fully connected layer with ReLU activation which is then fed into a single-layer LSTM with 128 hidden state dimension and the output is projected to the action dimension using another fully connected layer.

Original: We used Gaussian policies with mean given by the output of the networks and a diagonal covariance matrix whose entries are also treated as trianable parameters.
Revised: We used Gaussian policies with mean given by the output of the networks and a diagonal covariance matrix whose entries are also treated as trianable parameters.

Original: The policy outputs are clipped to lie within the control range.
Revised: The policy outputs are clipped to lie within the control range.

Original: We used
Revised: We used

Original: MLP policy and value functions for the run-to-goal and you-shall-not-pass environments, and LSTM policy and value function for sumo and kick-and-defend.
Revised: MLP policy and value functions for the run-to-goal and you-shall-not-pass environments, and LSTM policy and value function for sumo and kick-and-defend.

Original: This is because earlier experiments did not yield good results with MLP policy on these tasks.
Revised: This is because earlier experiments did not yield good results with MLP policy on these tasks.

Original: For LSTM policy we used truncated BPTT with a truncation of 10 timesteps.
Revised: For LSTM policy we used truncated BPTT with a truncation of 10 timesteps.

Original: The policy and the value functions have separate parameters.
Revised: The policy and the value functions have separate parameters.

Original: For the asymmetric games, you-shall-not-pass and kick-and-defend, we use separate policies for the two agents in a game.
Revised: For the asymmetric games, you-shall-not-pass and kick-and-defend, we use separate policies for the two agents in a game.

Original: Observations: For the Ant body we use all the joint angles of the agent, its velocity of all its joints, the contact forces acting on the body and the relative position and all the joint angles for the opponent.
Revised: Observations: For the Ant body we use all the joint angles of the agent, its velocity of all its joints, the contact forces acting on the body and the relative position and all the joint angles for the opponent.

Original: For the Humanoid body, in addition to the above we also give the centre-of-mass based inertia tensor, velocity vector and the actuator forces for the body.
Revised: For the Humanoid body, in addition to the above we also give the centre-of-mass based inertia tensor, velocity vector and the actuator forces for the body.

Original: In addition to these, there are other environment speciﬁc observations.
Revised: In addition to these, there are other environment speciﬁc observations.

Original: For the Sumo environment, we give the torso’s orientation vector as the input, the radial distance from the edge of the ring of all the agents and the time remaining in the game.
Revised: For the Sumo environment, we give the torso’s orientation vector as the input, the radial distance from the edge of the ring of all the agents and the time remaining in the game.

Original: For kick-and-defend, we give the relative position of the ball from the agent, the relative distance of the ball from goal and the relative position of the ball from the two goal posts.
Revised: For kick-and-defend, we give the relative position of the ball from the agent, the relative distance of the ball from goal and the relative position of the ball from the two goal posts.

Original: Note that none of the agents observe the complete global state of the multi-agent world and only observe relevant sub-parts of the state vector to keep observations as close to real-world scenarios as possible.
Revised: Note that none of the agents observe the complete global state of the multi-agent world and only observe relevant sub-parts of the state vector to keep observations as close to real-world scenarios as possible.

Original: Algorithm Parameters: We use Adam (Kingma & Ba, 2014) with learning rate 0 .
Revised: Algorithm Parameters: We use Adam (Kingma & Ba, 2014) with learning rate 0 .

Original: The clipping parameter in PPO (cid:15) = 0 . 2 , discounting factor γ = 0 .
Revised: The clipping parameter in PPO (cid:15) = 0 . 2 , discounting factor γ = 0 .

Original: 995 and generalized advantage estimate parameter λ = 0 .
Revised: 995 and generalized advantage estimate parameter λ = 0 .

Original: Each iteration, we collect 409600 samples from the parallel rollouts and perform multiple epochs of PPO training in mini-batches consisting of 5120 samples.
Revised: Each iteration, we collect 409600 samples from the parallel rollouts and perform multiple epochs of PPO training in mini-batches consisting of 5120 samples.

Original: For MLP policies we did 6 epochs of SGD per iteration and for LSTM policies we did 3 epochs.
Revised: For MLP policies we did 6 epochs of SGD per iteration and for LSTM policies we did 3 epochs.

Original: We don’t use any entropy bonus.
Revised: We don’t use any entropy bonus.

Original: We found l 2 regularization of the policy and value network parameters to be useful.
Revised: We found l 2 regularization of the policy and value network parameters to be useful.

Original: The co-efﬁcient α t in eq.
Revised: The co-efﬁcient α t in eq.

Original: 1 for the exploration reward is annealed to 0 in 500 iterations for all the environments except for kick-and-defend in which it is annealed in 1000 iterations.
Revised: 1 for the exploration reward is annealed to 0 in 500 iterations for all the environments except for kick-and-defend in which it is annealed in 1000 iterations.

Original: 5. L EARNED B EHAVIORS
Revised: 5. L EARNED B EHAVIORS

Original: We observe numerous interesting learned behaviors demonstrated by the agents as a result of the complexity arising out of the competitive multi-agent training.
Revised: We observe numerous interesting learned behaviors demonstrated by the agents as a result of the complexity arising out of the competitive multi-agent training.

Original: Different random seeds often lead to somewhat different behaviors.
Revised: Different random seeds often lead to somewhat different behaviors.

Original: Refer to the videos for highlights of the learned policies on all the tasks.
Revised: Refer to the videos for highlights of the learned policies on all the tasks.

Original: On Run-to-Goal, we observe the quadruped Ants demonstrate behaviors like blocking, standing robustly, using legs to topple the opponent and running towards the goal.
Revised: On Run-to-Goal, we observe the quadruped Ants demonstrate behaviors like blocking, standing robustly, using legs to topple the opponent and running towards the goal.

Original: Humanoids try to avoid each other and run towards their goal really fast, occasionally they will bump into each other with force and try to recover from the impact.
Revised: Humanoids try to avoid each other and run towards their goal really fast, occasionally they will bump into each other with force and try to recover from the impact.

Original: On You-Shall-Not-Pass, we observe the blocking humanoid learn to block by raising its hand while the other humanoid eventually learned to duck in order to cross.
Revised: On You-Shall-Not-Pass, we observe the blocking humanoid learn to block by raising its hand while the other humanoid eventually learned to duck in order to cross.

Original: On Sumo, we observe multiple different strategies used by the Ant and
Revised: On Sumo, we observe multiple different strategies used by the Ant and

Original: Humanoid.
Revised: Humanoid.

Original: Humanoids, for example, demonstrate a stable ﬁghting stance and learned to knock the opponent using their heads.
Revised: Humanoids, for example, demonstrate a stable ﬁghting stance and learned to knock the opponent using their heads.

Original: In a different run, we observe that one agent learned to charge towards the opponent whereas the opponent tried to fool it by stepping out of the opponents way at the edge of the ring.
Revised: In a different run, we observe that one agent learned to charge towards the opponent whereas the opponent tried to fool it by stepping out of the opponents way at the edge of the ring.

Original: On kick-and-defend, we observe that the kicker learned a good kicking policy where it can go towards random ball positions, uses its feet to kick the ball high and tries to avoid the defender.
Revised: On kick-and-defend, we observe that the kicker learned a good kicking policy where it can go towards random ball positions, uses its feet to kick the ball high and tries to avoid the defender.

Original: We also see a fooling behavior in the kicker’s motions where it moves left and right quickly once close to the ball to fool the defender.
Revised: We also see a fooling behavior in the kicker’s motions where it moves left and right quickly once close to the ball to fool the defender.

Original: The defender learned to defend by moving in response to the motion of the kicker and using its hands and legs to obstruct the ball.
Revised: The defender learned to defend by moving in response to the motion of the kicker and using its hands and legs to obstruct the ball.

Original: These movement strategies are not just useful in competition, for example the skills learned in the
Revised: These movement strategies are not just useful in competition, for example the skills learned in the

Original: Sumo can transfer to other situations even without other agents.
Revised: Sumo can transfer to other situations even without other agents.

Original: In one case, we took the agent trained on the multi-agent Sumo task and faced it with the task of standing while being perturbed by wind forces.
Revised: In one case, we took the agent trained on the multi-agent Sumo task and faced it with the task of standing while being perturbed by wind forces.

Original: The agent receives the zero vector for parts of the opponent observation.
Revised: The agent receives the zero vector for parts of the opponent observation.

Original: We found that the agent managed to stay upright despite never seeing the windy environment or observing wind forces.
Revised: We found that the agent managed to stay upright despite never seeing the windy environment or observing wind forces.

Original: Please see Appendix B.1 for details of the experiment and quantitative results.
Revised: Please see Appendix B.1 for details of the experiment and quantitative results.

Original: Refer to the video for a demonstration.
Revised: Refer to the video for a demonstration.

Original: 5. E FFECT OF E XPLORATION C URRICULUM
Revised: 5. E FFECT OF E XPLORATION C URRICULUM

Original: In section 4.1 we introduced an exploration curriculum to help agents explore in a 3D world.
Revised: In section 4.1 we introduced an exploration curriculum to help agents explore in a 3D world.

Original: One question that arises is the extent to which the outcome of learning is affected by this exploration reward and to explore the beneﬁt of this exploration reward.
Revised: One question that arises is the extent to which the outcome of learning is affected by this exploration reward and to explore the beneﬁt of this exploration reward.

Original: As already argued, we found theexploration reward to be crucial for learning as otherwise the agents are unable to explore the sparse
Revised: As already argued, we found the

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  competition reward.
Revised: exploration reward to be crucial for learning as otherwise the agents are unable to explore the sparse competition reward.

Original: However, the learned behaviors are mostly a result of the natural curriculum arising out of the multi-agent competition and not due to the dense exploration reward.
Revised: However, the learned behaviors are mostly a result of the natural curriculum arising out of the multi-agent competition and not due to the dense exploration reward.

Original: To see this, ﬁrst note that we do not give any reward for many of the complex learned behaviours described previously.
Revised: To see this, ﬁrst note that we do not give any reward for many of the complex learned behaviours described previously.

Original: We further test this by not annealing the exploration reward and always having a dense reward which is a sum of the exploration reward and the competition reward.
Revised: We further test this by not annealing the exploration reward and always having a dense reward which is a sum of the exploration reward and the competition reward.

Original: We take these agents trained without curriculum and pit them against agents trained with exploration curriculum.
Revised: We take these agents trained without curriculum and pit them against agents trained with exploration curriculum.

Original: We plot the average win-rates over 800 games at various intervals during training in Fig.
Revised: We plot the average win-rates over 800 games at various intervals during training in Fig.

Original: 3, for the sumo and kick-and-defend environments.
Revised: 3, for the sumo and kick-and-defend environments.

Original: For kick-and-defend there are two plots, one where kicker is trained with curriculum while keeper without it and vice versa.
Revised: For kick-and-defend there are two plots, one where kicker is trained with curriculum while keeper without it and vice versa.

Original: Observe that the agents trained with curriculum beat the non-curriculum agents by a margin.
Revised: Observe that the agents trained with curriculum beat the non-curriculum agents by a margin.

Original: We found agents trained without curriculum exhibit either non-optimal behaviors for the competition or end up optimizing for a particular component of the dense reward.
Revised: We found agents trained without curriculum exhibit either non-optimal behaviors for the competition or end up optimizing for a particular component of the dense reward.

Original: For example, for Sumo, the agents just learn to stand and move towards center of the arena, and for kick-and-defend, the defender optimizes for being able to stand up but doesn’t learn to defend while the kicker learns a non-optimal strategy of carrying the ball with itself to the goal (rather than kicking) – a policy which is easily defeated by a defender trained with curriculum.
Revised: For example, for Sumo, the agents just learn to stand and move towards center of the arena, and for kick-and-defend, the defender optimizes for being able to stand up but doesn’t learn to defend while the kicker learns a non-optimal strategy of carrying the ball with itself to the goal (rather than kicking) – a policy which is easily defeated by a defender trained with curriculum.

Original: Moreover, training without curriculum also takes more samples to learn.
Revised: Moreover, training without curriculum also takes more samples to learn.

Original: We also show these behaviours qualitatively in the videos.
Revised: We also show these behaviours qualitatively in the videos.

Original: These results echo some recent ﬁndings (albeit in the single agent case), like Andrychowicz et al.
Revised: These results echo some recent ﬁndings (albeit in the single agent case), like Andrychowicz et al.

Original: (2017) who found that optimizing for the sparse reward yields better return than optimizing for hand crafted dense rewards.
Revised: (2017) who found that optimizing for the sparse reward yields better return than optimizing for hand crafted dense rewards.

Original: For the competitive multi-agent case, these results shed further light on the importance of the natural curriculum.
Revised: For the competitive multi-agent case, these results shed further light on the importance of the natural curriculum.

Original: 5. E FFECT OF O PPONENT S AMPLING
Revised: 5. E FFECT OF O PPONENT S AMPLING

Original: In section 4.2, we introduced the past opponent sampling method for training competitive agents simultaneously.
Revised: In section 4.2, we introduced the past opponent sampling method for training competitive agents simultaneously.

Original: This choice of opponent could be important as it affects the natural curriculum for the agents.
Revised: This choice of opponent could be important as it affects the natural curriculum for the agents.

Original: We test different opponent sampling strategies by considering a threshold on the oldest opponent for each agent.
Revised: We test different opponent sampling strategies by considering a threshold on the oldest opponent for each agent.

Original: That is, instead of uniform random over the entire history, we can consider sampling opponent from Uniform( δv, v ) where v is the iteration number for the latest available parameters of the opponent and δ ∈ [0 , 1] is a threshold.
Revised: That is, instead of uniform random over the entire history, we can consider sampling opponent from Uniform( δv, v ) where v is the iteration number for the latest available parameters of the opponent and δ ∈ [0 , 1] is a threshold.

Original: Thus, δ = 1 . 0 corresponds to the latest available opponent and δ = 0 .
Revised: Thus, δ = 1 . 0 corresponds to the latest available opponent and δ = 0 .

Original: 0 corresponds to uniform sampling over the entire history.
Revised: 0 corresponds to uniform sampling over the entire history.

Original: We train agents on the Sumo task via self-play, using a δ ∈ { 1 . 0 , 0 . 8 , 0 . 5 , 0 . 0 } and pit the four agents against each other to understand which sampling strategy leads to more robust policies.
Revised: We train agents on the Sumo task via self-play, using a δ ∈ { 1 . 0 , 0 . 8 , 0 . 5 , 0 . 0 } and pit the four agents against each other to understand which sampling strategy leads to more robust policies.

Original: Since the agents have different skills and strengths at various points during training, we compute a Monte Carlo estimate of the expected win-rate for two agents that have seen the same number of samples taken at a random point during training.
Revised: Since the agents have different skills and strengths at various points during training, we compute a Monte Carlo estimate of the expected win-rate for two agents that have seen the same number of samples taken at a random point during training.

Original: This is done by taking average of the win-rates of 30 agents at intervals of 100 iterations after a burn-in of 3000 iterations, where each win-rate is computed from an average over 800 episodes.
Revised: This is done by taking average of the win-rates of 30 agents at intervals of 100 iterations after a burn-in of 3000 iterations, where each win-rate is computed from an average over 800 episodes.

Original: Table 1a reports the results for Humanoid and Table 1b reports the results for
Revised: Table 1a reports the results for Humanoid and Table 1b reports the results for

Original: Ant.
Revised: Ant.

Original: First note that training against the latest opponent leads to worst performance, as argued earlier.
Revised: First note that training against the latest opponent leads to worst performance, as argued earlier.

Original: Surprisingly, we found that uniform random ( δ = 0 . 0 ) over the entire history to have the highest win-rate for Ant and δ = 0 .
Revised: Surprisingly, we found that uniform random ( δ = 0 . 0 ) over the entire history to have the highest win-rate for Ant and δ = 0 .

Original: 5 to have the highest win-rate for Humanoid.
Revised: 5 to have the highest win-rate for Humanoid.

Original: This could be because
Revised: This could be because

Original: Ant with random policy on a small arena is still a good opponent while a Humanoid with random policy is unable to stand and thus always looses in a few steps.
Revised: Ant with random policy on a small arena is still a good opponent while a Humanoid with random policy is unable to stand and thus always looses in a few steps.

Original: The differences in these win-rates for different sampling strategies show that the choice of the opponent during sampling is important and care must be taken while designing training algorithms for such competitive environments.
Revised: The differences in these win-rates for different sampling strategies show that the choice of the opponent during sampling is important and care must be taken while designing training algorithms for such competitive environments.

Original: 5. L EARNING R OBUST P OLICIES
Revised: 5. L EARNING R OBUST P OLICIES

Original: Over-ﬁtting to a particular dataset is often a problem in supervised learning.
Revised: Over-ﬁtting to a particular dataset is often a problem in supervised learning.

Original: Similar problems can arise in reinforcement learning setups when there is no or little variation in the environment.
Revised: Similar problems can arise in reinforcement learning setups when there is no or little variation in the environment.

Original: We discovered two such problems in our competitive multi-agent training framework and we analyze and propose solutions to address these issues.
Revised: We discovered two such problems in our competitive multi-agent training framework and we analyze and propose solutions to address these issues.

Original: 5.5. R ANDOMIZATION IN W ORLD
Revised: 5.5. R ANDOMIZATION IN W ORLD

Original: In order to learn robust policies which generalize better we can introduce randomness in the environment, for example the arena radius for the sumo environment can be randomized, the ball position for the kick-and-defend environment can be randomized, agent start positions can be randomized.
Revised: In order to learn robust policies which generalize better we can introduce randomness in the environment, for example the arena radius for the sumo environment can be randomized, the ball position for the kick-and-defend environment can be randomized, agent start positions can be randomized.

Original: However, we found that while randomization is crucial to learn policies which generalize better, it might hinder learning early on as there might be too many things for the agents to explore.
Revised: However, we found that while randomization is crucial to learn policies which generalize better, it might hinder learning early on as there might be too many things for the agents to explore.

Original: Indeed, we observe that in kick-and-defend the agents are unable to learn to kick with a lot of randomization in both the ball and agent positions, whereas when trained with no randomization the learned policies are overﬁt to the particular position of the ball (see Fig. 4).
Revised: Indeed, we observe that in kick-and-defend the agents are unable to learn to kick with a lot of randomization in both the ball and agent positions, whereas when trained with no randomization the learned policies are overﬁt to the particular position of the ball (see Fig. 4).

Original: Thus, in order to learn policies that generalize well, we introduce a simple curriculum in the randomization where we start with a small amount of randomization which is easier to solve and then gradually increase the randomization during training.
Revised: Thus, in order to learn policies that generalize well, we introduce a simple curriculum in the randomization where we start with a small amount of randomization which is easier to solve and then gradually increase the randomization during training.

Original: We found this curriculum to work well for all the environments.
Revised: We found this curriculum to work well for all the environments.

Original: 5.5. C OMPETING AGAINST ENSEMBLE OF POLICIES
Revised: 5.5. C OMPETING AGAINST ENSEMBLE OF POLICIES

Original: Another related problem that we observed is over-ﬁtting to the behavior of the opponent when trained for very long.
Revised: Another related problem that we observed is over-ﬁtting to the behavior of the opponent when trained for very long.

Original: This results in policies which are good against particular types of opponents but do not generalize to other opponents (say opponents trained with a different random seed).
Revised: This results in policies which are good against particular types of opponents but do not generalize to other opponents (say opponents trained with a different random seed).

Original: This overﬁtting can also be observed in win-rates against opponent during training, where one would see oscillations as agents try to adapt to their particular opponent and changes in their strategies.
Revised: This overﬁtting can also be observed in win-rates against opponent during training, where one would see oscillations as agents try to adapt to their particular opponent and changes in their strategies.

Original: To overcome this we propose learning multiple policies simultaneously.
Revised: To overcome this we propose learning multiple policies simultaneously.

Original: Thus, there is a pool of policies and in each rollout for a particular policy one of the other policies is selected at random as the opponent (in symmetric games, the same policy can also be an opponent).
Revised: Thus, there is a pool of policies and in each rollout for a particular policy one of the other policies is selected at random as the opponent (in symmetric games, the same policy can also be an opponent).

Original: This is similar to multi-task learning (Caruana, 1998) where the same network is used to model multiple related tasks which allows sharing of statistical strength among tasks and reduces overﬁtting.
Revised: This is similar to multi-task learning (Caruana, 1998) where the same network is used to model multiple related tasks which allows sharing of statistical strength among tasks and reduces overﬁtting.

Original: In this case, the pool of all policies as opponents – current and throughout the history of training – creates a natural distribution over related tasks for multi-task learning.
Revised: In this case, the pool of all policies as opponents – current and throughout the history of training – creates a natural distribution over related tasks for multi-task learning.

Original: We found random policy initialization to provide enough diversity between agent policies, however techniques that explicitly encourage diversity (Liu & Wang, 2016) can potentially be incorporated in the future.
Revised: We found random policy initialization to provide enough diversity between agent policies, however techniques that explicitly encourage diversity (Liu & Wang, 2016) can potentially be incorporated in the future.

Original: In order to test the robustness of training policies in an ensemble, we experiment on the Sumo environment with Ant and Humanoid bodies.
Revised: In order to test the robustness of training policies in an ensemble, we experiment on the Sumo environment with Ant and Humanoid bodies.

Original: We train a pool of three policies in an ensemble and take the policy with the highest average training reward in the last 500 iterations as the best ensemble policy.
Revised: We train a pool of three policies in an ensemble and take the policy with the highest average training reward in the last 500 iterations as the best ensemble policy.

Original: We also train three independent policies via self-play, that is just a single policy is trained in a run, and again take the policy with the highest average training reward in last 500 iterations as the best self-play policy.
Revised: We also train three independent policies via self-play, that is just a single policy is trained in a run, and again take the policy with the highest average training reward in last 500 iterations as the best self-play policy.

Original: Then we pit the best ensemble policy against the best self-play policy and record average win-rates over 800 games.
Revised: Then we pit the best ensemble policy against the best self-play policy and record average win-rates over 800 games.

Original: Fig.
Revised: Fig.

Original: 5 shows the win-rates over training iterations (after 1000 iterations of training).
Revised: 5 shows the win-rates over training iterations (after 1000 iterations of training).

Original: We ﬁnd that training in ensemble performs signiﬁcantly better for the humanoid body, whereas for ant the performance is similar to training a single policy.
Revised: We ﬁnd that training in ensemble performs signiﬁcantly better for the humanoid body, whereas for ant the performance is similar to training a single policy.

Original: Again we suspect this is because there is not enough variability in the behavior of ant across different runs.
Revised: Again we suspect this is because there is not enough variability in the behavior of ant across different runs.

Original: While training single policies might occasionally get stuck in a local minima and learn suboptimal behaviors, we found that when training in an ensemble to be more robust to such minima.
Revised: While training single policies might occasionally get stuck in a local minima and learn suboptimal behaviors, we found that when training in an ensemble to be more robust to such minima.

Original: Qualitatively, we see more robust behavior of the humanoid trained in ensemble (see video).
Revised: Qualitatively, we see more robust behavior of the humanoid trained in ensemble (see video).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have presented several new competitive multi-agent 3D physically simulated environments.
Revised: We have presented several new competitive multi-agent 3D physically simulated environments.

Original: We demonstrate the development of highly complex skills in simple environments with simple rewards.
Revised: We demonstrate the development of highly complex skills in simple environments with simple rewards.

Original: In future work it would be interesting to conduct larger scale experiments in more complex environments that encourage agents to both compete and cooperate with each other.
Revised: In future work, it would be interesting to conduct larger scale experiments in more complex environments that encourage agents to both compete and cooperate with each other.

Original: Incorporation of additional skills, such as reasoning about other agents, potentially via techniques from Foerster et al.
Revised: Incorporation of additional skills, such as reasoning about other agents, potentially via techniques from Foerster et al.

Original: (2017a), may also be important in our setting
Revised: (2017a), may also be important in our setting

Original: A E XPLORATION R EWARDS
Revised: A E XPLORATION R EWARDS

Original: We deﬁne the dense exploration rewards used for the tasks in this section.
Revised: We deﬁne the dense exploration rewards used for the tasks in this section.

Original: Our exploration reward terms are based on adapting the rewards deﬁned previously for the training humanoids and quadrupeds to walk (Duan et al., 2016; Schulman et al., 2015b).
Revised: Our exploration reward terms are based on adapting the rewards deﬁned previously for the training humanoids and quadrupeds to walk (Duan et al., 2016; Schulman et al., 2015b).

Original: We ﬁrst review this locomotion reward and then deﬁne the task-speciﬁc dense rewards.
Revised: We ﬁrst review this locomotion reward and then deﬁne the task-speciﬁc dense rewards.

Original: These rewards take the form r t ( s, a )
Revised: These rewards take the form r t ( s, a )

Original: = v fwd + c t ( s, a ) + C alive where v fwd is the velocity in the forward direction, a bonus for standing C alive and costs for impact and action c t ( s, a ) .
Revised: = v fwd + c t ( s, a ) + C alive where v fwd is the velocity in the forward direction, a bonus for standing C alive and costs for impact and action c t ( s, a ) .

Original: We considered the following locomotion reward deﬁned for Humanoid-v1 environment in OpenAI Gym package:
Revised: We considered the following locomotion reward deﬁned for Humanoid-v1 environment in OpenAI Gym package:

Original: [Equation]
Revised: [Equation]

Original: where F impact is the contact force vector clipped to values between 1 and 1, and C alive is a bonus for the center of the body being at a certain height, deﬁned as C alive =
Revised: where F impact is the contact force vector clipped to values between 1 and 1, and C alive is a bonus for the center of the body being at a certain height, deﬁned as C alive =

Original: +5 if 2 . 0 ≥ z body ≥ 1 . 0 , else 0 .
Revised: +5 if 2 . 0 ≥ z body ≥ 1 . 0 , else 0 .

Original: Similarly, the following is the reward for quadruped locomotion:
Revised: Similarly, the following is the reward for quadruped locomotion:

Original: [Equation]
Revised: [Equation]

Original: where C alive =
Revised: where C alive =

Original: +

Revised: +

Original: if 1 . 0 ≥ z body ≥ 0 .
Revised: if 1 . 0 ≥ z body ≥ 0 .

Original: 2 , else 0 .
Revised: 2 , else 0 .

Original: In the following, superscript h refers to humanoid agents and superscript q refers to quadruped.
Revised: In the following, superscript h refers to humanoid agents and superscript q refers to quadruped.

Original: We redeﬁne C alive to be +5 if z body ≥ 1 . 0 , else − 5 for humanoid, and C alive
Revised: We redeﬁne C alive to be +5 if z body ≥ 1 . 0 , else − 5 for humanoid, and C alive

Original: = +

Revised: = +

Original: if z body ≥ 0 .
Revised: if z body ≥ 0 .

Original: 28 , else − 1 for quadruped.
Revised: 28 , else − 1 for quadruped.

Original: Run to Goal For humanoids, reward is r h ( s, a ) − | x − g | where x − g is the l 1 distance of the agent from the goal g along the x -axis.
Revised: Run to Goal For humanoids, reward is r h ( s, a ) − | x − g | where x − g is the l 1 distance of the agent from the goal g along the x -axis.

Original: For ant, reward is similar r q ( s, a ) − | x − g | .
Revised: For ant, reward is similar r q ( s, a ) − | x − g | .

Original: You Shall not Pass For the agent whose goal is to reach the other side, the reward is same as for run-to-goal.
Revised: You Shall not Pass For the agent whose goal is to reach the other side, the reward is same as for run-to-goal.

Original: For the blocking agent, the reward for humanoid is c h ( s, a ) + C alive + | x (cid:48) − g | where | x (cid:48) − g | is the distance of opponent to the goal.
Revised: For the blocking agent, the reward for humanoid is c h ( s, a ) + C alive + | x (cid:48) − g | where | x (cid:48) − g | is the distance of opponent to the goal.

Original: Sumo For humanoids, reward is c h ( s, a ) + C alive − ( x 2 + y 2 ) 0 .
Revised: Sumo For humanoids, reward is c h ( s, a ) + C alive − ( x 2 + y 2 ) 0 .

Original: 5 , where the last term is distance from the center of the ring.
Revised: 5 , where the last term is distance from the center of the ring.

Original: Similarly for ant: c q ( s, a ) + C alive − ( x 2 + y 2 ) 0 .
Revised: Similarly for ant: c q ( s, a ) + C alive − ( x 2 + y 2 ) 0 .

Original: Kick and Defend: For kicker, reward is r h ( s, a ) −|| x − b ||−| b x − g | , where b is the ( x, y ) position of the ball, b x is the x -coordinate of the ball and g is the x -coordinate of the goal-post.
Revised: Kick and Defend: For kicker, reward is r h ( s, a ) −|| x − b ||−| b x − g | , where b is the ( x, y ) position of the ball, b x is the x -coordinate of the ball and g is the x -coordinate of the goal-post.

Original: For defender, reward is c h ( s, a ) + C alive + | b x − g | where for C alive we only gave positive reward if the defender was in front of the goal area.
Revised: For defender, reward is c h ( s, a ) + C alive + | b x − g | where for C alive we only gave positive reward if the defender was in front of the goal area.

Original: [Table]
Revised: [Table]

Original: B A DDITIONAL R ESULTS B. T RANSFER RESULTS
Revised: B A DDITIONAL R ESULTS B. T RANSFER RESULTS

Original: We took the agent trained on the multi-agent Sumo task and faced it with the task of standing while being perturbed by wind forces.
Revised: We took the agent trained on the multi-agent Sumo task and faced it with the task of standing while being perturbed by wind forces.

Original: The agent receives a zero vector for parts of the observation space which correspond to the opponent.
Revised: The agent receives a zero vector for parts of the observation space which correspond to the opponent.

Original: We calculate the number of steps before the agent falls down (i.e. when z body ≤ 0 . 5 ) or the agent is pushed out of the arena and report the average steps over 200 episodes.
Revised: We calculate the number of steps before the agent falls down (i.e. when z body ≤ 0 . 5 ) or the agent is pushed out of the arena and report the average steps over 200 episodes.

Original: Episodes last a maximum of 500 time steps.
Revised: Episodes last a maximum of 500 time steps.

Original: In half the episodes the wind force is applied in a radially outwards direction and in the remaining half it is applied in the radially inwards direction.
Revised: In half the episodes the wind force is applied in a radially outwards direction and in the remaining half it is applied in the radially inwards direction.

Original: We allow 50 steps for the agent to stabilize and apply the force at intervals of 50 steps where in between the intervals the force magnitude is decayed at a constant rate:
Revised: We allow 50 steps for the agent to stabilize and apply the force at intervals of 50 steps where in between the intervals the force magnitude is decayed at a constant rate:

Original: [Equation]
Revised: [Equation]

Original: otherwise where F ∈ { 200 , 300 , 400 , 500 , 600 } .
Revised: otherwise where F ∈ { 200 , 300 , 400 , 500 , 600 } .

Original: We compare with a humanoid agent trained in a single agent environment for the task of walking.
Revised: We compare with a humanoid agent trained in a single agent environment for the task of walking.

Original: We used same LSTM policy architecture as used for the Sumo agent and trained the humanoid in the publicly available OpenAI
Revised: We used same LSTM policy architecture as used for the Sumo agent and trained the humanoid in the publicly available OpenAI

Original: Gym Humanoid-v1 environment using PPO.
Revised: Gym Humanoid-v1 environment using PPO.

Original: We then apply force on this agent using the same method as above where the direction of the force is in the direction the agent is walking in half the episodes and opposite to it in the remaining half.
Revised: We then apply force on this agent using the same method as above where the direction of the force is in the direction the agent is walking in half the episodes and opposite to it in the remaining half.

Original: We record average number of steps to fall using the same condition as for the Sumo agent.
Revised: We record average number of steps to fall using the same condition as for the Sumo agent.

Original: Table 2 shows the average number of steps over 200 episodes along with the standard deviation.
Revised: Table 2 shows the average number of steps over 200 episodes along with the standard deviation.

Original: We see that the humanoid trained in Sumo is more robust to adversarial forces and able to withstand large magnitude of force for many steps.
Revised: We see that the humanoid trained in Sumo is more robust to adversarial forces and able to withstand large magnitude of force for many steps.