Original: N EURAL MMO: A MASSIVELY MULTIPLAYER GAME ENVIRONMENT FOR INTELLIGENT AGENTS
Revised: N EURAL MMO: A MASSIVELY MULTIPLAYER GAME ENVIRONMENT FOR INTELLIGENT AGENTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present an artiﬁcial intelligence research platform inspired by the human game genre of MMORPGs (Massively Multiplayer Online Role-Playing Games, a.k.a.
Revised: We present an artiﬁcial intelligence research platform inspired by the human game genre of MMORPGs (Massively Multiplayer Online Role-Playing Games, a.k.a.

Original: MMOs).
Revised: MMOs).

Original: We demonstrate how this platform can be used to study behavior and learning in large populations of neural agents.
Revised: We demonstrate how this platform can be used to study behavior and learning in large populations of neural agents.

Original: Unlike currently popular game environments, our platform supports persistent environments, with variable number of agents, and open-ended task descriptions.
Revised: Unlike currently popular game environments, our platform supports persistent environments, with variable number of agents, and open-ended task descriptions.

Original: The emergence of complex life on
Revised: The emergence of complex life on

Original: Earth is often attributed to the arms race that ensued from a huge number of organisms all competing for ﬁnite resources.
Revised: Earth is often attributed to the arms race that ensued from a huge number of organisms all competing for ﬁnite resources.

Original: Our platform aims to simulate this setting in microcosm: we conduct a series of experiments to test how large-scale multiagent competition can incentivize the development of skillful behavior.
Revised: Our platform aims to simulate this setting in microcosm: we conduct a series of experiments to test how large-scale multiagent competition can incentivize the development of skillful behavior.

Original: We ﬁnd that population size magniﬁes the complexity of the behaviors that emerge and results in agents that out-compete agents trained in smaller populations.
Revised: We ﬁnd that population size magniﬁes the complexity of the behaviors that emerge and results in agents that out-compete agents trained in smaller populations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Life on Earth can be viewed as a massive multiagent competition.
Revised: Life on Earth can be viewed as a massive multiagent competition.

Original: The cheetah evolves an aerodynamic proﬁle in order to catch the gazelle, the gazelle develops springy legs to run even faster: species have evolved ever new capabilities in order to outcompete their adversaries.
Revised: The cheetah evolves an aerodynamic proﬁle in order to catch the gazelle, the gazelle develops springy legs to run even faster: species have evolved ever new capabilities in order to outcompete their adversaries.

Original: The success of biological evolution has inspired many attempts to emulate it in silico, ranging from genetic algorithms that bear only loose resemblance to natural processes, to full-blown simulations of “artiﬁcial life”.
Revised: The success of biological evolution has inspired many attempts to emulate it in silico, ranging from genetic algorithms that bear only loose resemblance to natural processes, to full-blown simulations of “artiﬁcial life”.

Original: A recurring question has been: at what level of abstraction should we simulate the competitive game of life?
Revised: A recurring question has been: at what level of abstraction should we simulate the competitive game of life?

Original: In recent years, the ﬁeld of deep reinforcement learning (RL) has embraced a related approach: train algorithms by having them compete in simulated games (Silver et al., 2016; susan zhang, 2018; Jaderberg et al., 2018).
Revised: In recent years, the ﬁeld of deep reinforcement learning (RL) has embraced a related approach: train algorithms by having them compete in simulated games (Silver et al., 2016; OpenAI; Jaderberg et al., 2018).

Original: Such games are immediately interpretable and provide easy metrics derived from the game’s “score” and win conditions.
Revised: Such games are immediately interpretable and provide easy metrics derived from the game’s “score” and win conditions.

Original: However, popular game benchmarks are currently still limited: they typically deﬁne a narrow, episodic task, with a small ﬁxed number of players.
Revised: However, popular game benchmarks are currently still limited: they typically deﬁne a narrow, episodic task, with a small ﬁxed number of players.

Original: In contrast, life on Earth involves a persistent environment, an unbounded number of players, and a seeming “open-endedness,” where ever new and more complex species emerge over time, with no end in sight (Stanley et al., 2017).
Revised: In contrast, life on Earth involves a persistent environment, an unbounded number of players, and a seeming “open-endedness”, where ever new and more complex species emerge over time, with no end in sight (Stanley et al., 2017).

Original: Our aim is to develop a simulation platform (see Figure 1) that captures important properties of life on Earth, while also borrowing from the interpretability and abstractions of human-designed games.
Revised: Our aim is to develop a simulation platform (see Figure 5) that captures important properties of life on Earth, while also borrowing from the interpretability and abstractions of human-designed games.

Original: To this end, we turn to the game genre of Massively Multiplayer Online Role-Playing
Revised: To this end, we turn to the game genre of Massively Multiplayer Online Role-Playing

Original: Games (MMORPGs, or MMOs for short).
Revised: Games (MMORPGs, or MMOs for short).

Original: These games involve a large, variable number of players competing to survive and prosper in persistent and far-ﬂung environments.
Revised: These games involve a large, variable number of players competing to survive and prosper in persistent and far-ﬂung environments.

Original: Our platform simulates a “Neural MMO” – an MMO in which each agent is a neural net that learns to survive using RL.
Revised: Our platform simulates a “Neural MMO” – an MMO in which each agent is a neural net that learns to survive using RL.

Original: We demonstrate the capabilities of this platform through a series of experiments that investigate emergent complexity as a function of the number of agents and species that compete in the simulation.
Revised: We demonstrate the capabilities of this platform through a series of experiments that investigate emergent complexity as a function of the number of agents and species that compete in the simulation.

Original: We ﬁnd that large populations act as competitive pressure that encourages exploration of the environment and the development of skillful behavior.
Revised: We ﬁnd that large populations act as competitive pressure that encourages exploration of the environment and the development of skillful behavior.

Original: In addition, we ﬁnd that when agents are organized into species (share policy parameters), each species naturally diverges from the others to occupy its own behavioral niche.
Revised: In addition, we ﬁnd that when agents are organized into species (share policy parameters), each species naturally diverges from the others to occupy its own behavioral niche.

Original: Upon publication, we will opensource the platform in full.
Revised: Upon publication, we will opensource the platform in full.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: Multiagent Reinforcement Learning Multiagent reinforcement learning has received increased attention in recent years (Lowe et al., 2017; Mordatch & Abbeel, 2017; Bansal et al., 2017; Lanctot et al., 2017;
Revised: Multiagent Reinforcement Learning Multiagent reinforcement learning has received increased attention in recent years (Lowe et al., 2017; Mordatch & Abbeel, 2017; Bansal et al., 2017; Lanctot et al., 2017;

Original: Yang et al., 2018a; Zheng et al., 2017; susan zhang, 2018; Jaderberg et al., 2018).
Revised: Yang et al., 2018a; Zheng et al., 2017; OpenAI; Jaderberg et al., 2018).

Original: Unlike single agent environments often well modeled by classic Markov Decision Processes (MDPs), multiagent interaction often introduces nonstationarity in the transition dynamics of the environment due to the continual learning and co-adaptation of other agents.
Revised: Unlike single agent environments often well modeled by classic Markov Decision Processes (MDPs), multiagent interaction often introduces nonstationarity in the transition dynamics of the environment due to the continual learning and co-adaptation of other agents.

Original: While previous work has attempted to analyze emergent complexity in interactions of groups of 2agents (Bansal et al., 2017; susan zhang, 2018; Jaderberg et al., 2018), we focus on large populations of agents in a complex world.
Revised: While previous work has attempted to analyze emergent complexity in interactions of groups of 2agents (Bansal et al., 2017; OpenAI; Jaderberg et al., 2018), we focus on large populations of agents in a complex world.

Original: Zheng et al.
Revised: Zheng et al.

Original: (2017) also analyze behavior of many agents, but the task setting is considerably simpler, and agents are directly rewarded for foraging or ﬁghting.
Revised: (2017) also analyze behavior of many agents, but the task setting is considerably simpler, and agents are directly rewarded for foraging or ﬁghting.

Original: In contrast, our work focuses on complexity in a diverse environment where agents are only rewarded for survival; doing so requires maintaining their health through food and water while navigating partially obstructed terrain and with various forms of combat.
Revised: In contrast, our work focuses on complexity in a diverse environment where agents are only rewarded for survival; doing so requires maintaining their health through food and water while navigating partially obstructed terrain and with various forms of combat.

Original: Yang et al.
Revised: Yang et al.

Original: (2018b) also simulate populations scaling to millions of learning agents. However, they study only predator-prey population dynamics in this setting, ﬁnding similarities to predator-prey dynamics in nature.
Revised: (2018b) also simulate populations scaling to millions of learning agents, but they focus on predator-prey population dynamics in this setting.

Original: Artiﬁcial Life “Artiﬁcial life” projects aim to simulate the complexity of biological life (Langton, 1997), often framed as a multiagent competition, and have demonstrated that this setting can lead to the development of skilled behaviors (Yaeger, 1994) and capable morphologies (Sims, 1994).
Revised: Artiﬁcial Life “Artiﬁcial life” projects aim to simulate the complexity of biological life (Langton, 1997), often framed as a multiagent competition. This setting can lead to the development of skilled behaviors (Yaeger, 1994) and capable morphologies (Sims, 1994).

Original: The setting we consider, in which an agent coadapts alongside others is sometimes called coevolution (Ficici & Pollack, 1998).
Revised: We consider the setting of coevolution (Ficici & Pollack, 1998) in which agents coadapt alongside others.

Original: Our platform can simulate a kind of artiﬁcial life, but at a relatively high level of abstraction, importing design elements from games.
Revised: Our platform can simulate a kind of artiﬁcial life, but at a relatively high level of abstraction.

Revised: While some basic features of our environment (movement, food, water) are similar to those in (Hern´andezOrallo et al., 2011; Strannegrd et al., 2018), our environment is grounded in the established game genre of MMOs.

Original: Unlike most past work in this area, our platform is builtto around deep reinforcement learning, where each agent is a neural net trained with distributed policy gradients, and includes tools for visualizing properties speciﬁc to this setting.
Revised: Unlike most past work in this area, our platform is built around deep reinforcement learning, where each agent is a neural net trained with distributed policy gradients, and includes tools for visualizing properties speciﬁc to this setting.

Original: Game Platforms for Intelligent Agents
Revised: Game Platforms for Intelligent Agents

Original: The Arcade Learning Environment (ALE) (Bellemare et al., 2013) and Gym Retro (Nichol et al., 2018) provide 1000+ limited scope arcade games most often used to test individual research ideas or generality across many games.
Revised: The Arcade Learning Environment (ALE) (Bellemare et al., 2013) and Gym Retro (Nichol et al., 2018) provide 1000+ limited scope arcade games most often used to test individual research ideas or generality across many games.

Original: Strictly better performance at a large random subset of games is a reasonable metric of quality, and strikingly divergent performance can motivate further research into a particular mode of learning.
Revised: Strictly better performance at a large random subset of games is a reasonable metric of quality, and strikingly divergent performance can motivate further research into a particular mode of learning.

Original: For example, Montezumas Revenge in ALE is difﬁcult even with reasonable exploration because of the associative memory required to collect and use keys.
Revised: For example, Montezumas Revenge in ALE is difﬁcult even with reasonable exploration because of the associative memory required to collect and use keys.

Original: However, recent results have brought into question the overall complexity each individual environment (Cuccu et al., 2018) and strong performance in such tasks (at least in those not requiring precise reﬂexes) is not particularly difﬁcult for humans.
Revised: However, recent results have brought into question the overall complexity each individual environment (Cuccu et al., 2018) and strong performance in such tasks (at least in those not requiring precise reﬂexes) is not particularly difﬁcult for humans.

Original: More recent work has demonstrated success on multiplayer games including Go (Silver et al., 2016), the Multiplayer Online Battle Arena (MOBA) DOTA2 (susan zhang, 2018), and Quake 

Revised: More recent work has demonstrated success on multiplayer games including Go (Silver et al., 2016), the Multiplayer Online Battle Arena (MOBA) DOTA2 (OpenAI), and Quake

Original: Capture the Flag (Jaderberg et al., 2018).
Revised: Capture the Flag (Jaderberg et al., 2018).

Original: Each of these projects has advanced our understanding of a class of algorithms.
Revised: Each of these projects has advanced our understanding of a class of algorithms.

Original: However, these games were limited to 2-12 players, are round based on the order of an hour, lack persistence, and lack the game mechanics supporting large persistent populations – there is still a large gap in environment complexity compared to the real world.
Revised: However, these games were limited to 2-12 players, are round based on the order of an hour, lack persistence, and lack the game mechanics supporting large persistent populations – there is still a large gap in environment complexity compared to the real world.

Original: RPGs MMORPGs are the multiplayer analogs to the narrower game genre of RPGs (Role-Playing Games).
Revised: MMORPGs The game genre we focus on in this paper is MMORPGs, which are role-playing games (RPGs) in which many human players take part.

Original: The latter includes game franchises such as Pokemon, Final Fantasy,The Witcher, and Fallout. They involve reasoning on the order of several to several hundred hours–longer time horizons than MOBAs–but they lack the complexity induced by multiagent interaction and competition–this is our motivation for creating an MMO rather than an RPG.
Revised: RPGs, such as Pokemon and Final Fantasy, involve reasoning over up to hundreds of hours of persistent gameplay – a much longer time horizon than in MOBAs.

Original: Like the real world, RPGs confront the player with problems that have many valid solutions, and choices that have long term consequences.
Revised: Like the real world, RPGs confront the player with problems that have many valid solutions, and choices that have long term consequences.

Original: For example, whereas the arcade game Montezumas Revenge has a single core mechanic (ﬁnd keys, unlock doors), the game Prey allows players to proceed by remembering the code to an unguarded side door, acquiring weapons to ﬁght through the main entrance, or bypassing the encounter altogether by searching out a hidden side passage.
Revised: MMOs are the (massively) multiplayer analogs to RPGs.

Original: This makes well-designed RPGs compelling candidates for a variety of research ideas in isolation, but they are also well suited to simultaneously testing multiple research ideas at scale.
Revised: They are typically run across several servers , each of which contains a copy of the environment and supports hundreds to millions of concurrent players . Good MMOs involve a curriculum of challenges that require increasingly clever usage of the game systems.

Original: Given multiple paths to proceed, each requiring different learned skills (e.g., exploration, memory, knowledge of game mechanics), the ability to reason over them all in conjunction allows players to choose the quickest, safest, or least costly path.
Revised: Early game content is accessible to new players, but skills required for late game content are inaccessible (and often incomprehensible) to those not intimately familiar with the game. Players have to acquire resources and “level up” in order to reach more advanced stages of the game.

Original: As well designed MMOs with single player are effectively RPGs, this means that Neural MMOs have the potential to provide the beneﬁts of a rich single agent environment as well.
Revised: Such a curriculum is present in many game genres, but only MMOs contextualize it within social and economic structures approaching the scale of the real world.

Original:  N EURAL MMO
Revised:  N EURAL MMO

Original: We present a persistent and massively multiagent environment that adopts many core MMO properties.
Revised: We present a persistent and massively multiagent environment that deﬁnes foraging and combat systems over procedurally generated maps (see Appendix).

Original: Our platform aims to mirror the MMO lifecycle: developers typically produce a small base game, with persistent play and consistent content updates slowly shaping this restricted core into a sprawling macrocosm.
Revised: The purpose of the platform is to discover game mechanics that support complex behavior and agent populations that can learn to make use of them. We follow the iterative development cycle of human MMOs: developers create balanced mechanics while players maximize their skill in utilizing them . The initial conﬁgurations of our systems are the results of several iterations of balancing, but are by no means ﬁxed: every numeric parameter presented is editable within a simple conﬁguration ﬁle. [Table]

Original: Base mechanics are built upon over arich set of challenges that produce increasingly engaging play later in the game.
Revised: A map consists of a set of discrete tiles, which are positions an agent can occupy.

Original: Skills needed for early content are easy for new players to acquire, but late game content is usually unapproachable (and often incomprehensible) to those who have not progressed signiﬁcantly through the game.
Revised: On each step of the simulations, agents may move one tile North/South/East/West. Agents competes for food tiles while periodically reﬁlling their water supply from inﬁnite water tiles.

Original: This property acts as a natural curriculum from toy to complex that makes MMO environments amenable to scale up without saturating the task or sacriﬁcing efﬁciency at small scale.
Revised: They may attack each other using any of three attack options, each with different damage values and trade offs. The environment assumes only that agents receive local game state and output a decision.

Original: This makes Neural MMOs a suitable platform for studying multiagent interaction at small scale, but as content develops, we believe the platform will grow to suit an increasing set of key problems in reinforcement learning. This initial release includes procedural map generation as show in in Figure 2, the base environment, and two content modules: foraging and basic combat.
Revised: The environment is agnostic to the source of that decision, be it a neural network or a hardcoded algorithm. We have tested up to 100M agent trajectories (lifetimes) on 100 cores in 1 week.

Original: The code base already contains additional module code for trade, gathering, crafting, item systems, equipment, communication, and trade to name a few.
Revised: The code base already contains additional module code for trade, gathering, crafting, item systems, equipment, communication, and trade to name a few.

Original: We are actively balancing and integrating these into the neural API.
Revised: We are actively balancing and integrating these into the neural API.

Original:  3. B ASE ENVIRONMENT
Revised:  A RCHITECTURE AND T RAINING

Original: Many MMOs use an internal tile grid representation and smooth animations during rendering. We adopt this model, with agents trained on local game state. Agents observe  a square crop of tiles as well as some visible properties of all agents in the corresponding tiles.
Revised: Input Agents observe local game state–all tiles within a ﬁxed L 1 distance of their current position, including tile terrain types and the visible properties of occupying agents. This is an efﬁcient equivalent representation of what a human sees on the screen without requiring rendering.

Original: Terrain tiles may be passable or impassible. We also include lava tiles, which kill entities upon contact. Agents may move one square in any of the four cardinal directions each environment timestep or “tick”. We add one new agent into the game on each tick within a region of designated spawn tiles, with an optional population cap.
Revised: Output Agents output action choices for the next time step (“game tick”).

Original: This is only the most bare bones set of game mechanics required to run anything, somewhat equal to a tech demo of performance and functionality, but without playable content. 3. F ORAGE Foraging adds health, food, and water quantities to each entity. A snapshot is shown in Figure 3.
Revised: For the experiments below, actions consist of one movement and one attack.

Original: Food and water decrease constantly; entities begin taking damage when they run out of either, but begin slowly healing if they are well fed and well hydrated. Water is collected at a ﬁxed rate by standing adjacent to water tiles. Food is collected at a ﬁxed rate by standing on forest tiles; however, these have a limited supply of food that renews slowly over time. This imposes a carrying capacity: survival is trivial with a single agent, but requires intelligent exploration in the presence of competition attempting to do the same.
Revised: As described in Framework, movement options are: North, South, East, West, Pass (no movement).

Original: Combat enables agents to attack each other.
Revised: Attack options are: Melee, Range,

Original: A snapshot is shown in Figure 3. It adds melee, ranged, and magic-based combat. Each combat style has a different maximum range of effect and damage value. This combat formulation was empirically chosen to incentivize behavior dependent on the states of other agents and to facilitate consistent and interpretable interactions.
Revised: Mage.

Original: The Neural MMO setting requires support for  a large, variable number of agents that run concurrently. As a complex game environment, it also requires a front end visualizer in order to observe agent policies and relevant statistics.
Revised: This is purely ﬂavor; for those not familiar with MMOs, each attack option simply applies a preset amount of damage at a preset effective distance.

Original: We provide two APIs for our environment.
Revised: The environment will attempt to execute both actions.

Original: One is a minimally modiﬁed multiagent extension of the Gym VecEnv API (Brockman et al., 2016), which distributes environment computation of observations and centralizes training and inference.
Revised: Invalid actions, such as moving into a stone wall, are ignored.

Original: While this standardization is convenient, it incurs additional communications overhead in our setting that we bypass with a fast native API.
Revised: Our policy architecture is detailed in the Appendix.

Original: This interface is simpler to use and pins the environment  and agents on it to the same CPU core. Full trajectories run locally on the same core as the environment.
Revised: We provide a simple preprocessor that embeds and ﬂattens this stimulus into a single ﬁxed length environment vector and a list of entity embeddings.

Original: Interprocess communication is only required infrequently to synchronize gradients across all environments on a master core. We currently do the backwards pass on the same CPU cores for convenience, but plan on separating this into an asynchronous experience queue for GPU workers. We also provide a renderer well suited to test time visualizations. This comes packaged with research tools for visualizing useful metrics, including value map ”ghosting” as described in Figure 3, exploration maps, dependence of other agents upon other agents, and attack maps for combat (see
Revised: We apply a linear layer to the preprocessed embeddings followed by three output heads for movement, attacks.

Original: Experiments).  A RCHITECTURE AND T RAINING To train agents, we use a policy gradient method Williams (1992) with a  value function baseline. Our policy architecture consists of an environment embedding network, a movement head, and an attack head, as shown in Figure 4.
Revised: There is also a standard value head which is trained to predict the discounted expected lifetime of the agent.

Original: The environment embedding network is a fully-connected network mapping from input stimulus to a 16 dimensional hidden vector (see below). Each head is  a small fully connected network, with hidden dimensions throughout ranging from 16 to 32, for a total of 50k to 100k parameters. Environment embedding is not shared across heads. For foraging experiments, the attack network is still present for convenience, but the chosen actions are ignored.
Revised: Each head is also a linear layer.

Original: New abilities and components can be added to model by adding additional heads.
Revised: New types of action choices can be included by adding additional heads.

Original: We ﬁnd that, under some settings, training can be unstable. To mitigate numerical collapse, we add an entropy bonus of 0.01 during training. We have conﬁrmed that this entropy bonus is not needed to learn exploration, and is included purely for the sake of numerical stability. We optimize with default Adam parameters.
Revised: We train with simple policy gradients plus a value baseline.

Original: Agents receive only a stream of reward 1.
Revised: Agents receive only a stream of reward 1.

Original: We postprocess trajectories using a discount factor of 0.99. The environment network creates a stimulus vector by embedding the local region of the world visible to the agent. It embeds each tiles material index and concatenates with the number of occupying entities and ﬂattens. It then concatenates the resultant vector with the (unlearned) features of the current entity.
Revised: Rewards are postprocessed by a discounting factor, producing returns equal to a discounted estimate of the agent’s time until death.

Original: These include visible properties such as health, food, water, and position.

Original: In order to handle the variable number of visible entities, a linear layer is applied to each followed by 1D max pooling.

Original: Attention Bahdanau et al.

Original: (2014) may appear the more conventional approach, but recently susan zhang (2018) demonstrated that simpler and more efﬁcient max pooling sufﬁces.

Original: The pooled entity vector is then concatenated with the tile and self embeddings to produce a single vector and followed by a ﬁnal linear layer as shown in Figure 4.

Original: The value network uses the learned environment embedding to predict the discounted expected lifetime of the agent, equal to summing over a discounted trajectory of 1’s.

Original: We found it possible to obtain good performance without discounting, but training was less stable.
Revised: We found it possible to obtain good performance without discounting, but training was less stable.

Original: The movement head and attack head select a movement direction (North, South, East, West, Pass) and attack style (Melee, Range, Mage), respectively.

Original: Our ﬁnal set of experiments prescribes targeting to the agent with lowest health.

Original: This does not preclude agents from learning intelligent straﬁng and attack patterns.

Original: We have experimented with learned entity targeting using attention over candidate entity embeddings and cell targeting based off a ﬂat list of relative cell locations.

Original: We ﬁnd that these are increasingly difﬁcult to learn due to required sample complexity: while we were able to learn semi-reasonable policies with learned entity targeting, we were not able to reproduce these results with cell targeting.

Original: We believe these to be feasible tasks at relatively low compute scale, but perhaps not in our setting with simplest possible architecture and training conﬁguration.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We present an initial series of experiments using our platform to explorelarge population multiagent interaction.
Revised: We present an initial series of experiments using our platform to explore multiagent interactions in large populations.

Original: We ﬁnd that agent competence scales with population size.
Revised: We ﬁnd that agent competence scales with population size.

Original: In particular, increasing the maximum number of concurrent players (#ent) magniﬁes exploration and increasing the maximum number of populations with unshared weights (#pop) magniﬁes niche formation.
Revised: In particular, increasing the maximum number of concurrent players ( N ent ) magniﬁes exploration and increasing the maximum number of populations with unshared weights ( N pop ) magniﬁes niche formation.

Revised: Agents are sampled uniformly from a number of “populations”–identical architectures with unshared weights.

Revised: This is for efﬁciency–see technical details.

Original: In all experiments, we ﬁx an upper cap on maximum number of concurrent agents per experiment and per simulation of the environment.
Revised: Technical details We run each experiment using 100 worlds. We deﬁne a constant C over the set of worlds W . For each world w ∈ W , we uniformly sample a c ∈ (1 , 2 , ...C ) .

Revised: We deﬁne ”spawn cap” such that if world w has a spawn cap c , the number of agents in w cannot exceed c .

Original: One agent is spawned per game tick, provided that doing so would exceed the spawn cap.
Revised: In each world w , one agent is spawned per game tick provided that doing so would exceed the spawn cap c of w .

Original: Agents are sampled uniformly from a number of “populations”.

Original: Ideally, we would ﬁx #ent = #pop, as is the case in standard MMOs (humans are independent networks with unshared weights).
Revised: Ideally, we would ﬁx N ent = N pop , as is the case in standard MMOs (humans are independent networks with unshared weights).

Original: However, this incurs sample complexity proportional to number of populations.
Revised: However, this incurs sample complexity proportional to number of populations.

Original: We therefore share parameters across groups of up to 16 agents for efﬁciency.
Revised: We therefore share parameters across groups of up to 16 agents for efﬁciency.

Original: 6. S ERVER M ERGE : ELO FACILITATES GAME MECHANICS AND INTERPRETABILITY
Revised: 5. S ERVER M ERGE : ELO FACILITATES GAME MECHANICS AND INTERPRETABILITY

Original: We  evaluate the effects on foraging performance of training with larger and more populations.
Revised: We perform four experiments to evaluate the effects on foraging performance of training with larger and more populations.

Original: At train time, we vary the spawn cap  andnumber of populations in (16, 32, 64, 128) and (1, 2, 4, 8), respectively.
Revised: For each experiment, we ﬁx N pop and a spawn cap 1 . These are paired from (1, 2, 4, 8) and (16, 32, 64, 128).

Original: We trainon 100 prebuilt but randomly generated maps for a ﬁxed number of trajectories per population 2.
Revised: We train for a ﬁxed number of trajectories per population 9. Evaluating the inﬂuence of these variables is nontrivial.

Original: Each world is randomly assigned a maximum number of entities between 16 and  the spawn cap. Mean lifetime is not comparable among experiments, as much of the task difﬁculty is induced by proximity to environment carrying capacity.
Revised: The task difﬁculty is highly dependent on the size and competence of populations in the environment: mean agent lifetime is not comparable among experiments. Furthermore, there is no analog procedure for evaluating relative player competence among MMO servers. However, MMO servers sometimes undergo merges whereby the player bases from multiple servers are placed within a single server.

Original: We propose tournament style evaluation in order to directly compare policies learned in different experiment settings.
Revised: As such, we propose tournament style evaluation in order to directly compare policies learned in different experiment settings.

Original: Tournaments are formed by simply concatenating the player bases of each experiment–this is identical to an MMO server merge.
Revised: Tournaments are formed by simply concatenating the player bases of each experiment.

Original: Results are shown in Figure 5: we vary the maximum number of agents at test time and ﬁnd that agents trained in larger settings consistently outperform agents trained in smaller settings.
Revised: Results are shown in Figure 3: we vary the maximum number of agents at test time and ﬁnd that agents trained in larger settings consistently outperform agents trained in smaller settings.

Original: Multiagent competition is a curriculum magniﬁer , not a curriculum.

Original: We do not expect that, given any rudimentary environment, multiagent competition will produce complex and interesting behavior– the task must contain something to magnify.

Original: We have demonstrated that, given a reasonable foraging task, multiagent competition leads to foraging behavior that scales with #ent and #pop.

Original: We now introduce the combat module as an additional learnable mode of variation on top of foraging and observe more interesting policies as a result, with agent actions  strongly coupled to the states of other agents.
Revised: When we introduce the combat module as an additional learnable mode of variation on top of foraging, we observe more interesting policies as a result. Agent actions become strongly coupled with the states of other agents.

Original: As a sanity, we also conﬁrm that all of the populations trained with combat handily outperform all of the foraging runs with random combat policies.
Revised: As a sanity, we also conﬁrm that all of the populations trained with combat handily outperform all of the populations trained with only foraging.

Original: While tournaments demonstrate the curriculum magnifying effect of multiagent competition, it is not clear a priori exactly where the effect comes from.

Original: To better understand this result, we decouple this variability into two modes: maximum number of concurrent players (#ent) and maximum number of populations with unshared weights (#pop).
Revised: To better understand theses results, we decouple our analysis into two modes of variability: maximum number of concurrent players ( N ent ) and maximum number of populations with unshared weights ( N pop ).

Original: This allows us to examine the effects of each factor independently.
Revised: This allows us to examine the effects of each factor independently.

Original: We ﬁnd that environment randomization also encourages exploration; wetherefore perform these experimentsprimarily as qualitative evaluations on a ﬁxed map.
Revised: In order to isolate the effects of environment randomization, which also encourages exploration, we perform these experiments on a ﬁxed map.

Original: This disentangles correlated factors and produces more immediately obvious results; however, results persist to some capacity in the environment randomized setting.
Revised: Isolating the effects of these variables produces more immediately obvious results, discussed below.

Original: Weaddress this brieﬂy in the Discussion section.
Revised: We brieﬂy examine the randomized setting in Discussion.

Original: 6. #E NTITY : M ULTIAGENT C OMPETITION M AGNIFIES E XPLORATION
Revised: 5. N ent : M ULTIAGENT C OMPETITION M AGNIFIES E XPLORATION

Original: In the natural world, competition between animals can incentivize them to spread out in order to avoid conﬂict.
Revised: In the natural world, competition between animals can incentivize them to spread out in order to avoid conﬂict.

Original: From Figure 6, we observe that overall exploration scales with number of concurrent agents, with no other variable factors (the map used is shown in Figure 7).
Revised: We observe that overall exploration scales with number of concurrent agents, with no other variable factors (see Figure 4; the map used is shown in Figure 5).

Original: Agents learn to explore only because the presence of other agents provides a natural and beneﬁcial curriculum for doing so
Revised: Agents learn to explore only because the presence of other agents provides a natural and beneﬁcial curriculum for doing so.

Original: . 6. #P OPULATION : M ULTIAGENT C OMPETITION M AGNIFIES N ICHE F ORMATION
Revised: 5. N pop : M ULTIAGENT C OMPETITION M AGNIFIES N ICHE F ORMATION

Original: We ﬁnd that, given a sufﬁciently large and resource rich environment, different populations of agents tend to separate to avoid competing with other populations.
Revised: We ﬁnd that, given a sufﬁciently large and resource rich environment, different populations of agents tend to separate to avoid competing with other populations.

Original: Both MMOs and the real world often reward masters of a single craft more than jacks of all trades.
Revised: Both MMOs and the real world often reward masters of a single craft more than jacks of all trades.

Original: From Figure 7, specialization to particular regions of the map scales with number of populations.
Revised: From Figure 5, specialization to particular regions of the map scales with number of populations.

Original: This suggests that the presence of other populations force agents to discover a single advantageous skill or trick.
Revised: This suggests that the presence of other populations force agents to discover a single advantageous skill or trick.

Original: That is, increasing the number of populations results in diversiﬁcation to separable regions of the map.
Revised: That is, increasing the number of populations results in diversiﬁcation to separable regions of the map.

Original: As entities cannot out-compete other agents of their population with shared weights, they tend to seek areas of the map that contain enough resources to sustain their population.
Revised: As entities cannot out-compete other agents of their population with shared weights, they tend to seek areas of the map that contain enough resources to sustain their population.

Original: Regions that are difﬁcult to get to or otherwise unoccupied are especially desirable; this is revealed by observing value maps over time.
Revised: Regions that are difﬁcult to get to or otherwise unoccupied are especially desirable; this is revealed by observing value maps over time.

Original:  D ISCUSSION 7. E NVIRONMENT R ANDOMIZED E XPLORATION
Revised:  D ISCUSSION 6. M ULTIAGENT COMPETITION IS A CURRICULUM MAGNIFIER

Revised: Jungle climates produce more biodiversity than deserts.

Revised: Deserts produce more biodiversity than the tallest mountain peaks.

Revised: To current knowledge, Earth is the only planet to produce life at all: the initial conditions for formation of intelligent life are of paramount importance.

Revised: The same holds true in simulation: human MMOs mirror this phenomenon.

Revised: Some games produce more complex and engaging play than others–those most successful garner large and dedicated playerbases that come to understand the game systems better than the developers.

Revised: Feedback helps drive development and expansion over a period of years, but the amount of effort required to create that initial seed is large.

Revised: It is unreasonable to expect pure multiagent competition to produce diverse and interesting behavior if the environment does not support it.

Original:  The trend of increasing exploration with increasing entity number is clear when training on  a single map as seen in Figure 6, 7.
Revised: This is because multiagent competition is a curriculum magniﬁer , not a curriculum . The multiagent setting is interesting because learning is responsive to competitive and collaborative pressures of other learning agents–but the environment must support and facillitate such pressures in order for multiagent interaction to drive complexity. There is room for debate among reasonable individuals as to theoretical minimum complexity seed environment required to produce complexity on par with that of the real world. However, this is not our objective, and we do not have a reasonable estimate of this quantity.

Original: It is perhaps more subtle once environment randomization is incorporated.
Revised: We have chosen to model our environment after MMOs, even though they may be more complicated than the minimum required environment class, because they are known to support the types of interactions we are interested in while maintaining engineering and implementation feasibility. This is not true of any other class environments we are aware of: exact physical simulations are computationally infeasible, and previously studied genres of human games lack crucial elements of complexity (see Background). While some may see our efforts as cherrypicking environment design, we believe this is precisely the objective: developers cherrypick game design decisions to support complexity commensurate with engaging play at the level of general human intelligence. The player base uses these design decisions to create strategies far beyond the imagination of the developers.

Revised: 6. E NVIRONMENT R ANDOMIZED E XPLORATION

Revised: The trend of increasing exploration with increasing entity number is clear when training on a single map as seen in Figure 4, 5, but it is more subtle with environment randomization.

Original: From Figure 8, all population sizes explore adequately.
Revised: From Figure 6, all population sizes explore adequately.

Original: We believe that this is because “exploration” as deﬁned by map coverage is not as difﬁcult a problem as developing robust policies.
Revised: We believe that this is because “exploration” as deﬁned by map coverage is not as difﬁcult a problem as developing robust policies.

Original: As demonstrated by the Tournament experiments,the exploration learned by smaller populations is more brittle and does not generalize to scenarios with more competitive pressure–even against a similar number of agents.
Revised: As demonstrated by the Tournament experiments, smaller populations learn brittle policies that do not generalize to scenarios with more competitive pressure–even against a similar number of agents.

Original: 7. A GENT -A GENT D EPENDENCIES
Revised: 6. A GENT -A GENT D EPENDENCIES

Original: We visualize agent-agent dependencies in Figure 9 using “ghosting” similar to that used to produce global value maps.
Revised: We visualize agent-agent dependencies in Figure 7.

Original: We ﬁx an agent at the center of a hypothetical map crop.
Revised: We ﬁx an agent at the center of a hypothetical map crop.

Original: For each position visible to that agent, we fake another agent and compute the value function estimate of the resultant pair.
Revised: For each position visible to that agent, we fake another agent and compute the value function estimate of the resultant pair.

Original: We ﬁnd that agents learn policies dependent on those of other agents in both the foraging and combat environments.
Revised: We ﬁnd that agents learn policies dependent on those of other agents in both the foraging and combat environments.

Original: 7. A DDITIONAL I NSIGHTS
Revised: 6. A DDITIONAL I NSIGHTS

Original: We brieﬂy detail several miscellaneous investigations and subtle points of interest in Figure 10.
Revised: We brieﬂy detail several miscellaneous investigations and subtle points of interest in Figure 8.

Original: First, we visualize learned attack patterns of agents.
Revised: First, we visualize learned attack patterns of agents.

Original: Each time an agent attacks, we splat the attack type to the screen, with red, green, and blue corresponding to melee, range, and mage, respectively.
Revised: Each time an agent attacks, we splat the attack type to the screen.

Original: There are a few valid strategies as per the environment.
Revised: There are a few valid strategies as per the environment.

Original: Melee is intentionally overpowered: learning to utilize it at close range serves as a sanity check.
Revised: Melee is intentionally overpowered: learning to utilize it at close range serves as a sanity check.

Original: This also incentivizes agents to keep their distance, as the ﬁrst to strike wins, and priority is determined arbitarily.
Revised: This also cautions agents to keep their distance, as the ﬁrst to strike wins.

Original: From Figure 10 and observation of the policies, we ﬁnd that this behavior is learned when targeting is automated.
Revised: From Figure 8 and observation of the policies, we ﬁnd that this behavior is learned when targeting is automated.

Revised: Our ﬁnal set of experiments prescribes targeting to the agent with lowest health.

Original: This is more problematic when targeting is learned jointly with attack style selection. We note that this formulation requires an attentional mechanism to handle variable number of visible targets; this is not yet numerically stable.
Revised: Jointly learning attack style selection and targeting requires an attentional mechanism to handle variable number of visible targets. We have experimented with this, but results are not yet numerically stable.

Original: We believe ﬁxing this is likely to resolve the targeting issues. However, policies learned using automatic targeting are still compelling: agents learn to strafe at the edge of their attack radius, attacking opportunistically.
Revised: Solving this is likely to increase policy complexity, but policies learned using automatic targeting are still compelling: agents learn to strafe at the edge of their attack radius, attacking opportunistically.

Original: Second, a note on tournaments.
Revised: Second, a note on tournaments.

Original: We equate number of trajectories trained upon as a fairest possible metric of training progress.
Revised: We equate number of trajectories trained upon as a fairest possible metric of training progress.

Original: However, one possible source of bias is number of gradient steps (more steps on smaller batch size vs fewer steps on larger batch size), which is not currently comparable due to implementation speciﬁcs. We are actively correcting this. Note that we cannot train for equal ﬂop count, as sample complexity is proportional to number of populations, which would result in a biased comparison. However, even in this scenario, we ﬁnd the trend persists when evaluating at maximum spawn cap 128.
Revised: We experimented with normalizing batch size but found that larger batch size always leads to more stable performance.

Original: As training progresses, the trend pushes to lower spawn caps as well, but with more mixed results.
Revised: Batch size is split among species: larger populations outperform smaller populations even though the latter are easier to train.

Original: Finally, a quick note on niche formation.
Revised: Finally, a quick note on niche formation.

Original: Obtaining clean visuals is dependent on having an environment where interaction with other agents is unfavorable.
Revised: Obtaining clean visuals is dependent on having an environment where interaction with other agents is unfavorable.

Original: When this is not the case, niche formation may still occur in another space (e.g., diverse attack policies).
Revised: When this is not the case, niche formation may still occur in another space (e.g., diverse attack policies).

Original: This is the importance of niche formation–we expect it to stack well with populationbased training (Jaderberg et al., 2017) and other such methods that require sample diversity
Revised: This is the importance of niche formation–we expect it to stack well with population-based training (Jaderberg et al., 2017) and other such methods that require sample diversity. In SANTA FE INSTITUTE STUDIES A PPENDIX F RAMEWORK

Revised: This section contains full environment details that are useful but not essential to understanding the base experiments.

Revised: All parameters in the following subsystems are conﬁgurable; we provide only sane defaults obtained via balancing.

Revised: 7. E NVIRONMENT T ILES

Revised: A grid of tiles 2 represents game state.

Revised: Agents may move one tile North/South/East/West each game tick.

Revised: They can also choose to Pass (i.e. not make a movement action).

Revised: Each tile object contains:

Revised: • Terrain type ∈ { grass, forest, stone, water, lava } .

Revised: Lava tiles kill agents upon contact.

Revised: Stone and water tiles are impassible.

Revised: • Occupying agents: a reference mapping of agents standing on the tile.

Revised: 7. F ORAGING

Revised: The foraging system implements gathering based survival by introducing:

Revised: • Food: initialized to 32, decremented 1 per tick, incremented 1 by occupying forest tiles, which contain 1 food each.

Revised: Once the resources of a particular tile are consumed, they regenerate probablistically over time.

Revised: • Water: decremented once per tick, incremented by standing adjacent to a water tile.

Revised: • Health: decremented once per tick if either food or water is zero.

Revised: Incremented if both food and water are above a threshold.

Revised: A snapshot is shown in Figure 2.

Revised: These deﬁnitions of resources impose a carrying capacity.

Revised: This incurs an arms race of exploration strategies in populations of agents above the carrying capacity: survival is trivial with a single agent, but requires intelligent exploration in the presence of competition attempting to do the same.

Revised: 7. C OMBAT

Revised: The combat system implements three different attack ”styles”:

Revised: • Melee: Inﬂicts 10 damage at 1 range

Revised: • Ranged: Inﬂicts 2 damage at 1-2 range • Mage: Inﬂicts 1 damage at 3 range and freezes the target in place, preventing movement (but not attacks) for two ticks

Revised: A snapshot is shown in Figure 2.

Revised: These deﬁnitions of combat impose trade offs in each style.

Revised: Melee combat is high risk high return.

Revised: Ranged combat produces less risky but more prolonged conﬂicts.

Revised: Mage combat does little damage, but allows agents to retreat or cut off opponents’ escape by freezing them in place.

Revised: The best strategy is not obvious, again imposing an arms race to discover the best strategy in the presence of many agents attempting to do the same.

Revised: Technical details:

Revised: • Attack range is deﬁned by l1 distance: ”1 range” is a 3X3 grid centered on the attacker.

Revised: Range 2 is a 5X5 grid, etc.

Revised: • Incentive: Animals hunt each other out of hunger or to protect/seize resources.

Revised: Agents receive food/water resources equal to the damage they inﬂict on other agents.

Revised: • Spawn Killing Agents are immune during their ﬁrst 15 game ticks alive.

Revised: This prevents an exploit known as ”spawn killing” whereby players are repeatedly attacked immediately upon entering the game.

Revised: Human games often contain similar mechanism to prevent this strategy, as it results in uninteresting play.

Revised: 7. API

Revised: We provide two APIs for this:

Revised: Gym Wrapper

Revised: We provide a minimal extension of the Gym VecEnv API (Brockman et al., 2016) that adds support for variable numbers of agents per world and at any given time.

Revised: This API distributes environment computation of observations and centralizes training and inference.

Revised: While this standardization is convenient, MMOs differ signiﬁcantly from arcade games, which are easier to standardize under a single wrapper.

Revised: The Neural MMO setting requires support for a large, variable number of agents that run concurrently, with aggregation across many randomly generated environments.

Revised: As such, the Gym framework incurs additional communications overhead in our setting that we bypass with a fast native API.

Revised: Native This interface is simpler to use and pins the environment and agents on it to the same CPU core.

Revised: Full trajectories run locally on the same core as the environment.

Revised: Interprocess communication is only required infrequently to synchronize gradients across all environments on a master core.

Revised: We currently do the backwards pass on the same CPU cores for convenience, but plan on separating this into an asynchronous experience queue for GPU workers.

Revised: 7. R ENDERER

Revised: We also provide a front end visualizer packed with research tools for analyzing observe agent policies and relevant statistics.

Revised: We leave detailed documentation of this to the source release, but it includes:

Revised: • 2D game renderer: shown in Figure• 3D game renderer in beta • Value map ”ghosting” (See Figure 2) • Exploration maps • Interagent dependence • Attack maps (See Experiments)

Revised: Preprocessor:

Revised: • Embed indicies corresponding to each tile into a 7D vector.

Revised: Also concatenates with the number of occupying entities.

Revised: • Project visible attributes of nearby entities to 32D • Flatten the tile embeddings • Max pool over entity embeddings to handle variable number of observations

Revised: • Concatenate the two •

Revised: This produces an embedding, which is computed separately for each head.

Revised: Technical details

Revised: • For foraging experiments, the attack network is still present for convenience, but the chosen actions are ignored.

Revised: • It then concatenates the resultant vector with the (unlearned) features of the current entity.

Revised: These include visible properties such as health, food, water, and position.

Revised: • In order to handle the variable number of visible entities, a linear layer is applied to each followed by 1D max pooling.

Revised: Attention Bahdanau et al.

Revised: (2014) may appear the more conventional approach, but recently OpenAI demonstrated that simpler and more efﬁcient max pooling sufﬁces.

Revised: • The pooled entity vector is then concatenated with the tile and self embeddings to produce a single vector and followed by a ﬁnal linear layer as shown in Figure 10.