Original: M ETA A RCADE : A C ONFIGURABLE E NVIRONMENT S UITE FOR D EEP R EINFORCEMENT L EARNING AND M ETA -L EARNING
Revised: M ETA A RCADE : A C ONFIGURABLE E NVIRONMENT S UITE FOR D EEP R EINFORCEMENT L EARNING AND M ETA -L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Most approaches to deep reinforcement learning (DRL) attempt to solve a single task at a time.
Revised: Most approaches to deep reinforcement learning (DRL) attempt to solve a single task at a time.

Original: As a result, most existing research benchmarks consist of individual games or suites of games that have common interfaces but little overlap in their perceptual features, objectives, or reward structures.
Revised: As a result, most existing research benchmarks consist of individual games or suites of games that have common interfaces but little overlap in their perceptual features, objectives or reward structures.

Original: To facilitate research into knowledge transfer among trained agents (e.g. via multi-task and meta-learning), more environment suites that provide conﬁgurable tasks with enough commonality to be studied collectively are needed.
Revised: To facilitate research into knowledge transfer among trained agents (e g. via multi-task and meta-learning), more environment suites that provide conﬁgurable tasks with enough commonality to be studied collectively are needed.

Original: In this paper we present Meta Arcade, a tool to easily deﬁne and conﬁgure custom 2D arcade games that share common visuals, state spaces, action spaces, game components, and scoring mechanisms.
Revised: In this paper we present Meta Arcade, a tool to easily deﬁne and conﬁgure custom 2D arcade games that share common visuals, state spaces, action spaces, game components and scoring mechanisms.

Original: Meta Arcade differs from prior environments in that both task commonality and conﬁgurability are prioritized: entire sets of games can be constructed from common elements, and these elements are adjustable through exposed parameters.
Revised: Meta Arcade differs from prior environments in that both task commonality and conﬁgurability are prioritized: entire sets of games can be constructed from common elements and these elements are adjustable through exposed parameters.

Original: We include a suite of 24 predeﬁned games that collectively illustrate the possibilities of this framework and discuss how these games can be conﬁgured for research applications.
Revised: We include a suite of 24 predeﬁned games that collectively illustrate the possibilities of this framework and discuss how these games can be conﬁgured for research applications.

Original: We provide several experiments that illustrate how Meta Arcade could be used, including single-task benchmarks of predeﬁned games, sample curriculum-based approaches that change game parameters over a set schedule, and an exploration of transfer learning between games.
Revised: We provide several experiments that illustrate how Meta Arcade could be used, including single-task benchmarks of predeﬁned games, sample curriculum-based approaches that change game parameters over a set schedule, and an exploration of transfer learning between games

Revised: A CKNOWLEDGEMENTS

Revised: Primary development of Meta Arcade was funded by the DARPA Lifelong Learning Machines (L2M) Program.

Revised: Additionally, this work relates to Department of Navy award N00014-20-1issued by the Ofﬁce of Naval Research.

Revised: The United States Government has a royalty-free license throughout the world in all copyrightable material contained herein.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Early deep reinforcement learning (DRL) (Mnih et al., 2013; Schulman et al., 2015; Lillicrap et al., 2016; Silver et al., 2016; Schulman et al., 2017) algorithms focused on performance on very narrowly-deﬁned tasks.
Revised: Early deep reinforcement learning (DRL) (Mnih et al., 2013; Schulman et al., 2015; Lillicrap et al., 2016; Silver et al., 2016; Schulman et al., 2017) algorithms focused on performance on very narrowly-deﬁned tasks.

Original: Accordingly, existing benchmark environments for DRL tend to be narrow in scope.
Revised: Accordingly, existing benchmark environments for DRL tend to be narrow in scope.

Original: For example, DQN Mnih et al.
Revised: For example, DQN Mnih et al.

Original: (2013), one of the ﬁrst works to combine deep neural networks and reinforcement learning, played individual Atari 2600 games using the Atari Learning Environment (ALE) (Bellemare et al., 2013).
Revised: (2013), one of the ﬁrst works to combine deep neural networks and reinforcement learning, played individual Atari 2600 games using the Atari Learning Environment (ALE) (Bellemare et al., 2013).

Original: These gamesare excellent benchmarks for narrow single-task learning, but have little in common with each other, making them unsuitable for multitask learning or meta-learning (as noted in Yu et al.
Revised: These games have little in common with each other, making them unsuitable for multitask learning or meta-learning (as noted in Yu et al.

Original: Moving away from such narrow training domains remains a very difﬁcult challenge, requiring environments that allow a graceful transition to multiple tasks and expose precise control over the underlying distributions.
Revised: Moving away from such narrow training domains remains a very difﬁcult challenge, requiring environments that allow a graceful transition to multiple tasks and expose precise control over the underlying distributions.

Original: In this work we present Meta Arcade, an attempt to address this need.
Revised: In this work we present Meta Arcade, an attempt to address this need.

Original: Meta Arcade is a suite of lightweight environments designed for multi-task learning and meta-learning, with an emphasis on conﬁgurability.
Revised: Meta Arcade is a suite of lightweight environments designed for multi-task learning and meta-learning, with an emphasis on conﬁgurability.

Original: Game elements, colors, dynamics, and objectives are all parameterized with support for distribution sampling, such that a single game can be varied slightly or altered to the extent that it constitutes a new task.
Revised: Game elements, colors, dynamics, and objectives are all parameterized with support for distribution sampling, such that a single game can be varied slightly or altered to the extent that it constitutes a new task.

Original: Common arcade game objectives like collecting blocks, bouncing balls, and avoiding obstacles are re-used between games.
Revised: Common arcade game objectives like collecting blocks, bouncing balls, and avoiding obstacles are re-used between games.

Original: If multiple tasks are deﬁned, these common objectives will result in clear overlap between the games, making them viable for multitask approaches.
Revised: If multiple tasks are deﬁned, these common objectives will result in clear overlap between the games, making them viable for multitask approaches.

Original: The Meta Arcade framework and suite provides two major assets to researchers.
Revised: The Meta Arcade framework and suite provides two major assets to researchers.

Original: First, an environment for RL that is fully parameterized so that it can be tuned to the needs of individual studies, including examinations of task variation or domain shift as a direct focus.
Revised: First, an environment for RL that is fully parameterized so that it can be tuned to the needs of individual studies, including examinations of task variation or domain shift as a direct focus.

Original: Second,the capability is provided for an entire space of arcade-inspired tasks that have common parameters, enabling research that requires sets of distinct but related tasks.
Revised: Second, an entire space of arcade-inspired tasks that have common parameters, enabling research that requires sets of distinct but related tasks.

Original: We have deﬁned 24 unique games that can be constructed using Meta Arcade, and describe how others may be built.
Revised: We have deﬁned 24 unique games that can be constructed using Meta Arcade, and describe how others may be built.

Original: Meta Arcade will be available on github (release in progress) with accompanying documentation and examples.
Revised: Meta Arcade is available on github (https://github.com/darpal2m/meta-arcade) with accompanying documentation and examples.

Original: Additionally, we contribute a series of experiments that include acquiring single-task expertise, curriculum-based learning, and multitask learning.
Revised: Additionally, we contribute a series of experiments that include acquiring single-task expertise, curriculum-based learning, and multitask learning.

Original: This manuscript details the capabilities of Meta Arcade and the results of our experiments.
Revised: This manuscript details the capabilities of Meta Arcade and the results of our experiments.

Original: In
Revised: In

Original: Section 2 we discuss other DRL environments and how they support the current research landscape.
Revised: Section 2 we discuss other DRL environments and how they support the current research landscape.

Original: In Section 3 we describe Meta Arcade in detail and how it can be used.
Revised: In Section 3 we describe Meta Arcade in detail and how it can be used.

Original: Benchmarks and other experimental results are presented in Section 4, and a discussion of future work is presented in
Revised: Benchmarks and other experimental results are presented in Section 4, and a discussion of future work is presented in

Original: Section 5.
Revised: Section 5.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There are a variety of environments that currently support DRL research, both for single tasks and, to a lesser extent, for multiple tasks.
Revised: There are a variety of environments that currently support DRL research, both for single tasks and, to a lesser extent, for multiple tasks.

Original: In addition to the discrete setting of ALE, DDPG (Lillicrap et al., 2016) introduced several 2D continuous control tasks and used MuJoCo (Todorov et al., 2012) as a platform for building such environments.
Revised: In addition to the discrete setting of ALE, DDPG (Lillicrap et al., 2016) introduced several 2D continuous control tasks and used MuJoCo (Todorov et al., 2012) as a platform for building such environments.

Original: Later MuJoCo was used to build 3D environments, as introduced alongside generalized advantage estimation (Schulman et al., 2018).
Revised: Later MuJoCo was used to build 3D environments, as introduced alongside generalized advantage estimation (Schulman et al., 2018).

Original: ALE, MuJoCo environments, classic control environments, and others are maintained in the OpenAI Gym (Brockman et al., 2016).
Revised: ALE, MuJoCo environments, classic control environments, and others are maintained in the OpenAI Gym (Brockman et al., 2016).

Original: While these benchmark environmentsare critical to testing and evaluating single-task experts, they tend to present narrow tasks, which lead to policies that are hyper-ﬁt to a speciﬁc problem (Zhang et al., 2018; Parisotto et al., 2016).
Revised: These benchmark environments tend to present narrow tasks, which lead to policies that are hyperﬁt to a speciﬁc problem (Zhang et al., 2018; Parisotto et al., 2016).

Original: This concern can be addressed algorithmically, but may be more naturally addressed by broadening the scope of the environments.
Revised: This concern can be addressed algorithmically, but may be more naturally addressed by broadening the scope of the environments.

Original: One common avenue towards robustness is to envision a task not as a static problem but rather as a distribution of task parameters, from visual distortions to changing dynamics (Zhao et al., 2020).
Revised: One common avenue towards robustness is to envision a task not as a static problem but rather as a distribution of task parameters, from visual distortions to changing dynamics (Zhao et al., 2020).

Original: This general idea, termed domain randomization, was ﬁrst presented for supervised learning in Tobin et al.
Revised: This general idea, termed domain randomization, was ﬁrst presented for supervised learning in Tobin et al.

Original: and laid the groundwork for policy-based success in the robotic manipulation of a Rubik’s Cube (OpenAI et al., 2019).
Revised: and laid the groundwork for policy-based success in the robotic manipulation of a Rubik’s Cube (OpenAI et al., 2019).

Original: Notably, Procgen (Cobbe et al., 2019), which includes the game Coin Run (Cobbe et al., 2018), deﬁnes tasks that are built procedurally and have key visuals randomized.
Revised: Notably, Procgen (Cobbe et al., 2019), which includes the game Coin Run (Cobbe et al., 2018), deﬁnes tasks that are built procedurally and have key visuals randomized.

Original: If a target environment is difﬁcult to learn but can be modiﬁed to create more easily-learned variants, it can be useful to supply a learning curriculum (Bengio et al., 2009; MacAlpine & Stone, 2018;
Revised: If a target environment is difﬁcult to learn but can be modiﬁed to create more easily-learned variants, it can be useful to supply a learning curriculum (Bengio et al., 2009; MacAlpine & Stone, 2018;

Original: Narvekar et al., 2020; Luo et al., 2020). Several environments provide options towards customization or even tend towards being a framework for building scenarios.
Revised: Narvekar et al., 2020; Luo et al., 2020), which may require some control over the underlying environment.

Original: ViZDoom (Kempka et al., 2016) and DeepMind Lab (Beattie et al., 2016) both provide tools for creating custom challenges that maintain the fundamental mechanisms of the games.
Revised: ViZDoom (Kempka et al., 2016) and DeepMind Lab (Beattie et al., 2016) both provide tools for creating custom challenges.

Original: However, it is difﬁcult to build out new tasks for these platforms, and ultimately all scenarios look more like variations of the original game than distinct tasks.
Revised: However, it is difﬁcult to build out new tasks for these platforms, and ultimately all scenarios look more like variations of the original game than distinct tasks.

Original: Explicit simulation frameworks like Unity (Juliani et al., 2018) and MuJoCo provide the means to create entirely new environments or modify existing examples, but require full development efforts.
Revised: Explicit simulation frameworks like Unity (Juliani et al., 2018) and MuJoCo provide the means to create entirely new environments or modify existing examples, but require full development efforts.

Original: Multi-task learning examines training competent agents on several tasks simultaneously or sequentially, rather than solely on instances of a single task.
Revised: Multi-task learning examines training competent agents on several tasks simultaneously or sequentially, rather than solely on instances of a single task.

Original: Ideally, the different tasks have features or skills that overlap such that there is an advantage to learning them together Parisotto et al.
Revised: Ideally, the different tasks have features or skills that overlap such that there is an advantage to learning them together Parisotto et al.

Original: ; Yu et al.
Revised: ; Yu et al.

Original: While a suite of environments is needed for multi-task RL research, ensuring that tasks are sufﬁciently similar or have sufﬁciently overlapping skills to enable transfer is difﬁcult.
Revised: While a suite of environments is needed for multi-task RL research, ensuring that tasks are sufﬁciently similar or have sufﬁciently overlapping skills to enable transfer is difﬁcult.

Original: Works such as Meta-World (Yu et al., 2019) and the Sonic benchmark (Nichol et al., 2018b) have been designed with this requirement in mind.
Revised: Works such as Meta-World (Yu et al., 2019) and the Sonic benchmark (Nichol et al., 2018b) have been designed with this requirement in mind.

Original: Suites which are based around a common game, such as the StarCraft II Learning Environment (Vinyals et al., 2017) minigames, tend to inherently include this task overlap.
Revised: Suites which are based around a common game, such as the StarCraft II Learning Environment (Vinyals et al., 2017) minigames, tend to inherently include this task overlap.

Original: This overlap is also useful if the tasks will be learned in sequence, as in lifelong learning and its variations.
Revised: This overlap is also useful if the tasks will be learned in sequence, as in lifelong learning and its variations.

Original: Alchemy (Wang et al., 2021) takes these ideas further by re-sampling the underlying game structure each episode, such that every episode is a different draw from a common task distribution.
Revised: Alchemy (Wang et al., 2021) takes these ideas further by re-sampling the underlying game structure each episode, such that every episode is a different draw from a common task distribution.

Original: Finally, another approach to producing more generalizable agents is meta-learning, or “learning-tolearn” (Schmidhuber, 1987; Hospedales et al., 2020), which enables adaptation to new tasks in a structured way.
Revised: Finally, another approach to producing more generalizable agents is meta-learning, or “learning-tolearn” (Schmidhuber, 1987; Hospedales et al., 2020), which enables adaptation to new tasks in a structured way.

Original: The meta-learning objective may be to determine what knowledge is transferable between tasks, to learn how to transfer that knowledge, or to learn how to do so efﬁciently, with the goal of quickly becoming proﬁcient at a new task.
Revised: The meta-learning objective may be to determine what knowledge is transferable between tasks, to learn how to transfer that knowledge, or to learn how to do so efﬁciently, with the goal of quickly becoming proﬁcient at a new task.

Original: Current research questions in this area include how to apply common meta-learning techniques (Finn et al., 2017;
Revised: Current research questions in this area include how to apply common meta-learning techniques (Finn et al., 2017;

Original: Nichol et al., 2018a; Rakelly et al., 2019) to problems with discrete action spaces and to problems with sparse rewards.
Revised: Nichol et al., 2018a; Rakelly et al., 2019) to problems with discrete action spaces and to problems with sparse rewards.

Original: MetaWorld (Yu et al., 2019) and Alchemy (mentioned above) are two recent environments that were speciﬁcally designed to support meta-learning research.
Revised: MetaWorld (Yu et al., 2019) and Alchemy (mentioned above) are two recent environments that were speciﬁcally designed to support meta-learning research.

Original: The goal of Meta Arcade is to enable further exploration of multitask and meta-learning by providing game environments that are useful for these burgeoning research areas.
Revised: The goal of Meta Arcade is to enable further exploration of multitask and meta-learning by providing game environments that are useful for these burgeoning research areas.

Original: Meta Arcade has the variety, speed, and ease of use that accompanies arcade games as a research benchmark, while providing the rich conﬁgurability and shared task structure needed for multitask learning, meta-learning, and related topics.
Revised: Meta Arcade has the variety, speed, and ease of use that accompanies arcade games as a research benchmark, while providing the rich conﬁgurability and shared task structure needed for multitask learning, meta-learning, and related topics.

Original:  M ETA A RCADE 3. E NVIRONMENT S UITE
Revised:  M ETA A RCADE 3. E NVIRONMENT S UITE

Original: The Meta Arcade environment suite seeks to facilitate many of the above research directions through a highly ﬂexible set of tools for training and evaluatingsophisticated DRL algorithms.
Revised: The Meta Arcade environment suite seeks to facilitate many of the above research directions through a highly ﬂexible set of tools for training and evaluating DRL algorithms.

Original: A primary focus was to create a suite of arcade games with many areas of overlap, such that the collection has a diversity of tasks similar to ALE while having enough commonality among tasks to be suitable for multi-task learning and meta-learning.
Revised: A primary focus was to create a suite of arcade games with many areas of overlap, such that the collection has a diversity of tasks similar to ALE while having enough commonality among tasks to be suitable for multi-task learning and meta-learning.

Original: Each game is parameterized and can be easily modiﬁed or extended, as is discussed in the next section.
Revised: Each game is parameterized and can be easily modiﬁed or extended.

Original: Meta Arcade includes 24 predeﬁned games that have the following characteristics:
Revised: Meta Arcade includes 24 predeﬁned games that have the following characteristics:

Original: All games present a 84x84x3 pixel-based state space, and all games have identical action spaces (6 discrete actions).
Revised: All games present a 84x84x3 pixel-based state space, and all games have identical action spaces (6 discrete actions).

Original: Some games will not use all 6 actions, but will instead map unused actions to no-ops internally.
Revised: Some games will not use all 6 actions, but will instead map unused actions to no-ops internally.

Original: The actions correspond to [No-op, Up, Down, Left, Right, Shoot] and  have identical effects in all games where they are used.
Revised: A similar entity is controlled in all games, and the actions have identical effects in all games where they are used.

Original: Continuous actions are also supported as an option.
Revised: Continuous actions are also supported as an option.

Original: A similar entity is controlled by the player in all games.

Original: An overview of the game display is shown in Figure 1.
Revised: An overview of the game display is shown in Figure 1.

Original: Meta Arcade was built with Pygame 1 , a platform-agnostic Python API for video game development, making Meta Arcade immediately accessible to developers already using Python deeplearning frameworks.
Revised: Meta Arcade was built with Pygame 1 , a platform-agnostic Python API for video game development, making Meta Arcade immediately accessible to developers already using Python deep-learning frameworks.

Original: Meta Arcade can optionally be run headlessly.
Revised: Meta Arcade can optionally be run headlessly.

Original: Games are constructed from a common set of game mechanics, visuals, and scoring systems.
Revised: Games are constructed from a common set of game mechanics, visuals, and scoring systems.

Original: Across games, entities will have similar interactions with other objects and incur similar rewards (if appli- cable).

Original: Games have a variety of overlapping skills such that a policy to solve one game is potentially useful for others.
Revised: Games have a variety of overlapping skills such that a policy to solve one game is potentially useful for others.

Original: For example, multiple games involve a paddle and a ball while others involve navigation in four directions.

Original: A sampling of games with some highlighted commonalities can be seen in Figure 1.
Revised: A sampling of games with some highlighted commonalities can be seen in Figure 1.

Revised: Across games, entities will have similar interactions with other objects and incur similar rewards.

Original: The maximum score for all games is 100 points and the minimum score is either 0 or -100, depending on whether it is possible to lose the game (as opposed to the game timing out, which happens after several thousand frames).
Revised: The maximum score for all games is 100 points and the minimum score is either 0 or -100, depending on whether it is possible to lose the game (as opposed to the game timing out, which happens after several thousand frames).

Original: Rewards are event-driven and sparse.
Revised: Rewards are event-driven and sparse.

Original: Common scoring mechanisms are built around this scale and are summarized in the table below.
Revised: Common scoring mechanisms are built around this scale and are summarized in the table below.

Original: More than one scoring mechanism can be present in a game.

Original: [Table]

Original: +

Original: [Table]

Original: Player hits collectable block + Block Value

Original: [Table]

Original: Ball

Original: [Table]

Original: side +100 (win)

Original: [Table]

Original: [Table]

Original: Ball hits collectable block + Block Value

Original: [Table]

Original: + Block Value

Original: Bullet hits opponent +100 (win)

Original: Bullet hits player -

Original: [Table] [Table] falls passed player -100 (loss) There are no lives in Meta Arcade, as past approaches to solving arcade games often manually split episodes among lives anyway (Mnih et al., 2013; 2015).
Revised: There are no lives in Meta Arcade, as past approaches to solving arcade games often manually split episodes among lives anyway (Mnih et al., 2013; 2015).

Original: Block-collecting games divide 100 points among some target number of blocks.
Revised: Block-collecting games divide 100 points among some target number of blocks.

Original: This leads to some initially unintuitive results for single games in which a loss is heavily weighted against progress.
Revised: This leads to some initially unintuitive results for single games in which a loss is heavily weighted against progress.

Original: For example, collecting 19 out ofblocks in breakout results in a score of -5 (+5 points x 19 and -100 for missing the ball).
Revised: For example, collecting 19 out ofblocks in breakout results in a score of -5 (+5 points x 19 and -100 for missing the ball).

Original: This does not prevent learning of the games but does require care when examining results.
Revised: This does not prevent learning of the games but does require care when examining results.

Original: We believe this is a reasonable tradeoff in exchange for a common scoring system among all games, which may aid in the learning of value functions with similar magnitudes.
Revised: We believe this is a reasonable tradeoff in exchange for a common scoring system among all games, which may aid in the learning of value functions with similar magnitudes.

Original:  The predeﬁned games in Meta Arcade present a range of difﬁculties; some are quickly solvable with off-the-shelf DRL algorithms while others are challenging for even customized approaches.
Revised: [Table] The predeﬁned games in Meta Arcade present a range of difﬁculties.

Original: This spectrum is presented intentionally to provide easy games for rapid testing and challenge games to motivate future research.
Revised: This spectrum is presented intentionally to provide easy games for rapid testing and challenge games to motivate future research.

Original: Not all games are necessarily solvable in isolation, and may require curricula, transfer learning, or other advanced methods to solve.
Revised: Not all games are necessarily solvable in isolation [Table] and may require curricula, transfer learning, or other advanced methods to solve.

Original: 3. C APABILITIES FOR M ODERN DRL R ESEARCH
Revised: 3. C APABILITIES FOR M ODERN DRL R ESEARCH

Original: While many environments are presented to the research community as static problems, there is often a close collaboration in the creation of new algorithms and new environments.
Revised: While many environments are presented to the research community as static problems, there is often a close collaboration in the creation of new algorithms and new environments.

Original: Environments may need to be specially built to explore a research idea, or a novel environment may spark a new research direction.
Revised: Environments may need to be specially built to explore a research idea, or a novel environment may spark a new research direction.

Original: With this in mind, Meta Arcade games are editable and are designed to be tweaked as needed for a given research avenue.
Revised: With this in mind, Meta Arcade games are editable and are designed to be tweaked as needed for a given research avenue.

Original: While many predeﬁned games are supplied, Meta Arcade can be thought of as a game creation tool as much as a suite of speciﬁc challenges.
Revised: While many predeﬁned games are supplied, Meta Arcade can be thought of as a game creation tool as much as a suite of speciﬁc challenges.

Original: New games can easily be created from scratch or by editing an existing game.
Revised: New games can easily be created from scratch or by editing an existing game.

Original: Each game is entirely deﬁned by a JSON ﬁle (see Figure 5 in the appendix) which lists the relevant game mechanics and their attributes.
Revised: Each game is entirely deﬁned by a JSON ﬁle (see Figure 5 in the appendix) which lists the relevant game mechanics and their attributes.

Original: The sizes, speeds, colors, and behaviors of any game can be directly changed through these ﬁles, or modiﬁed using environment tools from python scripts.
Revised: The sizes, speeds, colors, and behaviors of any game can be directly changed through these ﬁles, or modiﬁed using environment tools from python scripts.

Original: Furthermore, any value in the game deﬁnitions can be replaced with a distribution to be sampled from rather than a static value.
Revised: Furthermore, any value in the game deﬁnitions can be replaced with a distribution to be sampled from rather than a static value.

Original: Gaussian distributions, uniform distributions, and special color distributions are supported.
Revised: Gaussian distributions, uniform distributions, and special color distributions are supported.

Original: This enables exploration of task distributions, domain randomization, and domain shifts.
Revised: This enables exploration of task distributions, domain randomization, and domain shifts.

Original: Some examples of parameter changes can be found in Figure 2.
Revised: Some examples of parameter changes can be found in Figure 2.

Revised: Options are also provided for image-post processing, including hue, value, and saturation shifts, color inversion, and image rotation.

Original: Additionally, Meta Arcade includes some simple tools for building training curricula based on multiple tasks, or many copies of a task with shifted parameters.
Revised: Additionally, Meta Arcade includes some simple tools for building training curricula based on multiple tasks, or many copies of a task with shifted parameters.

Original: Games may be grouped together in a pool of tasks to be sampled, repeated for a speciﬁc number of episodes or steps, or interpolated between in the conﬁguration parameter space.
Revised: Games may be grouped together in a pool of tasks to be sampled, repeated for a speciﬁc number of episodes or steps, or interpolated between in the conﬁguration parameter space.

Original: Together with access to game parameters, this enables the construction of tasks which may change over time, may be interwoven with other tasks, or may be combined in other manners.
Revised: Together with access to game parameters, this enables the construction of tasks which may change over time, may be interwoven with other tasks, or may be combined in other manners.

Original: This type of functionality is essential to explore areas like continual learning, transfer learning, and curriculum learning.
Revised: This type of functionality is essential to explore areas like continual learning, transfer learning, and curriculum learning.

Original: Some examples of the environment’s behavior under simple curricula can be seen in Figure 3.
Revised: Some examples of the environment’s behavior under simple curricula can be seen in Figure 3.

Original: The most powerful and nuanced feature of the curriculum tools is the ability to linearly interpolate between game parameters, including those deﬁned by distributions.
Revised: The most powerful and nuanced feature of the curriculum tools is the ability to linearly interpolate between game parameters, including those deﬁned by distributions.

Original: Over a set number of episodes, two games deﬁned by identically keyed JSON ﬁelds can be interpolated between at each reset call.
Revised: Over a set number of episodes, two games deﬁned by identically keyed JSON ﬁelds can be interpolated between at each reset call.

Original: If a distribution is speciﬁed, the parameters of that distribution are interpolated.
Revised: If a distribution is speciﬁed, the parameters of that distribution are interpolated.

Original: For example, the standard deviation of an object’s size could be changed over time, or the number of blocks could be sampled from a uniform distribution that has changing bounds.
Revised: For example, the standard deviation of an object’s size could be changed over time, or the number of blocks could be sampled from a uniform distribution that has changing bounds.

Original: This is incredibly powerful for slowly changing the difﬁculty of a game or controlling domain randomization as a function of time.
Revised: This is incredibly powerful for slowly changing the difﬁculty of a game or controlling domain randomization as a function of time.

Original: Finally, some image post-processing is included in the JSON parameter ﬁles and may be changed as a fundamental aspect of the game.

Original: The hue, saturation, and value of the image can be adjusted.

Original: There is also an option for color inversion and image rotation.

Original: These tools are useful to quickly produce multiple variants of a game for studying a distribution of visual representations on a common task.

Original: For example, rather than changing the color of several game elements, a simple hue shift can affect the entire image.

Original: As with all other parameters, these image alterations can be replaced with distributions and interpolated between.

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: We present the results of several experiments to demonstrate the range of features available in Meta
Revised: We present the results of several experiments to demonstrate the range of features available in Meta

Original: Arcade and their applications to DRL research.
Revised: Arcade and their applications to DRL research.

Original: These experiments can be broken into three sections: (1) Single-task benchmarking of all predeﬁned games using PPO (Schulman et al., 2017)
Revised: These experiments can be broken into three sections: (1) Single-task benchmarking of all predeﬁned games using PPO (Schulman et al., 2017)

Original: (This also serves as a summary of all available games), (2) Selected curricula-based approaches for challenging games and the exploration of domain distributions, and (3) Transfer learning experiments on subsets of the games explore utilizing their common features.
Revised: (This also serves as a summary of all available games), (2) Selected curricula-based approaches for challenging games and the exploration of domain distributions, and (3) Transfer learning experiments on subsets of the games.

Original: We have chosen to discuss the results after each subsection since the results of each experiment often inﬂuenced the design of the next.

Original: 4. PPO B ENCHMARKING
Revised: 4. PPO B ENCHMARKING

Original: To demonstrate the solvability of predeﬁned games and provide a point of reference for other researchers, we used Proximal Policy Optimization (PPO)
Revised: To demonstrate the solvability of predeﬁned games and provide a point of reference for other researchers, we used Proximal Policy Optimization (PPO)

Original: (Schulman et al., 2017) to try to learn each game separately.
Revised: (Schulman et al., 2017) to try to learn each game separately.

Original: This is useful not only for understanding the difﬁculty of a given game but also proved useful for designing the games themselves and even debugging.
Revised: This is useful not only for understanding the difﬁculty of a given game but also proved useful for designing the games themselves and even debugging.

Original: In an effort to provide a range of reasonable difﬁculty, several games were tweaked in an iterative process with PPO benchmarking.
Revised: In an effort to provide a range of reasonable difﬁculty, several games were tweaked in an iterative process with PPO benchmarking.

Original: In some respects, this experiment could be considered part of the environment design process.
Revised: In some respects, this experiment could be considered part of the environment design process.

Revised: A full table of the composition of each predeﬁned game can be found in the appendix (Table 6).

Original: We trained PPO once on each game for 10 million frames, using 8 workers distributed with the
Revised: We trained PPO once on each game for 10 million frames, using 8 workers distributed with the

Original: Message Passing Interface (MPI) (Forum, 1994; Tesser, 2016).
Revised: Message Passing Interface (MPI) (Forum, 1994; Tesser, 2016).

Original: Gradients are computed in each worker and averaged in the main process.
Revised: Gradients are computed in each worker and averaged in the main process.

Original: A standard network architecture for Atari was used (Mnih et al., 2013).
Revised: A standard network architecture for Atari was used (Mnih et al., 2013).

Original: Speciﬁc hyperparameters can be found in the appendix, Section A.1.
Revised: Speciﬁc hyperparameters can be found in the appendix, Section A.1.

Original: The results from benchmarking all 24 games can be clearly split into those that were successful and unsuccessful, with ”unsuccessful” meaning that trained agent failed to consistently reach a positive score.
Revised: The results from benchmarking all 24 games can be clearly split into those that were successful and unsuccessful, with ”unsuccessful” meaning that trained agent failed to consistently reach a positive score.

Original: 18 of the 24 were successfully learned, often with nearly maximal performance.
Revised: 18 of the 24 were successfully learned, often with nearly maximal performance.

Original: The training results of the successful games are summarized in Table 2.
Revised: The training results of the successful games are summarized in Table 2.

Original: The only games which were not clearly successful in this category are Juggling, Keep Ups, and Pong Breakout, but performed well enough to be considered successful learning.
Revised: The only games which were not clearly successful in this category are Juggling, Keep Ups, and Pong Breakout. However, performance was high enough to be considered successful learning.

Original: In contrast, the six unsuccessfully learned games can be viewed inthe Appendix in Table 3.
Revised: The six unsuccessfully learned games can be viewed in Table 3.

Original: These games tended to converge to average scores around -100 or 0, corresponding to complete failure or avoiding play entirely.
Revised: These games tended to converge to average scores around -100 or 0, corresponding to complete failure or avoiding play entirely.

Original: These games appear to be challenging because it is very hard to discover rewards naturally or without incurring some penalty, which ultimately discourages learning the needed behaviors for success.
Revised: These games appear to be challenging because it is very hard to discover rewards naturally or without incurring some penalty, which ultimately discourages learning the needed behaviors for success.

Original: 4. C URRICULA
Revised: 4. C URRICULA

Original: In an effort to resolve the learning of some of the challenging problems seen in benchmarking and demonstrate some advanced features of the Meta Arcade environment suite, we designed learning curricula for Dungeon, Freeway, Invasion, Lava Maze, and Tunneler.
Revised: In an effort to resolve the learning of some of the challenging problems seen in benchmarking and demonstrate some advanced features of the Meta Arcade environment suite, we designed learning curricula for Dungeon, Freeway, Invasion, Lava Maze, and Tunneler.

Original: These curricula span the original 10 million training frames but introduce the games’ full complexities over time.
Revised: These curricula span the original 10 million training frames but introduce the games’ full complexities over time.

Original: Haunted
Revised: Haunted

Original: Hallway is omitted because it was unclear how to create a curriculum that did not change the game’s design.
Revised: Hallway is omitted because it was unclear how to create a curriculum that did not change the game’s design.

Original: The detailed results of the curriculum experiments can be found in Table 5 in the Appendix.
Revised: The detailed results of the curriculum experiments can be found in Table 5 in the Appendix.

Original: The general form of each curriculum was to start with what we believe to be an easier set of parameters that would allow early success (in other words, parameters that allow reward discovery with random actions).
Revised: The general form of each curriculum was to start with what we believe to be an easier set of parameters that would allow early success (in other words, parameters that allow reward discovery with random actions).

Original: Over the ﬁrst 4 million frames, this parameter distribution is gradually expanded and interpolated to include increasingly difﬁcult parameters during training, but without necessarily removing the easier versions from the set of possibilities.
Revised: Over the ﬁrst 4 million frames, this parameter distribution is gradually expanded and interpolated to include increasingly difﬁcult parameters during training, but without necessarily removing the easier versions from the set of possibilities.

Original: From 4 million to 8 million frames, the easy parameters are slowly excluded to focusin on the speciﬁc parameters that describe the full game.
Revised: From 4 million to 8 million frames, the easy parameters are slowly excluded to focus on the speciﬁc parameters that describe the full game.

Original: Some difﬁculty may still be introduced during this period to completely match the full game.
Revised: Some difﬁculty may still be introduced during this period to completely match the full game.

Original: From 8 million steps onward, the original game is played with no alteration.
Revised: From 8 million steps onward, the original game is played with no alteration.

Original: The last entries in Table 5 illustrate how the games may change during training according to the high-level schedule described above.
Revised: The last entries in Tableillustrate how the games may change during training according to the high-level schedule described above.

Original: The curricula approach proved very successful in training most of the challenging games.
Revised: The curricula approach proved very successful in training most of the challenging games.

Original: All the games show ﬁnal performances that reﬂect a high level of skill, with average scores far above their
Revised: All the games show ﬁnal performances that reﬂect a high level of skill, with average scores far above their

Original: PPO benchmark counterparts.
Revised: PPO benchmark counterparts.

Original: The key trend during training seems to be that the policy quickly 
Revised: The key trend during training seems to be that the policy quickly reaches strong play on the easy game parameters, and then tries to maintain high performance as the

Original: [Table]
Revised: [Table]

Original: reaches strong play on the easy game paramteres, and then tries to maintain high performance as the game parameters become more difﬁcult. Note that the Invasion curriculum leads to a slow ramp-up 
Revised: game parameters become more difﬁcult. Note that the Invasion curriculum leads to a slow ramp-up in score since the early games have fewer blocks than are needed to end the game (a score of 100 is

Original: [Table]
Revised: [Table]

Original: in score since the early games have fewer blocks than are needed to end the game (a score of 100 is not actually possible until further into the curriculum).
Revised: not actually possible until further into the curriculum).

Original: Several games also show a performance dip that coincides with a switch to the ﬁnal game, which may indicate that the transition to the full game should have been drawn out over more frames for stability.
Revised: Several games also show a performance dip that coincides with a switch to the ﬁnal game, which may indicate that the transition to the full game should have been drawn out over more frames for stability.

Original: 4. T RANSFER L EARNING FOR M ULTIPLE G AMES
Revised: 4. T RANSFER L EARNING FOR M ULTIPLE G AMES

Original: While the curriculum experiments explored variations within a single task, our ﬁnal set of experiments serve as exploratory examples of transfer learning between different tasks in Meta Arcade.
Revised: While the curriculum experiments explored variations within a single task, our ﬁnal set of experiments serve as exploratory examples of transfer learning between different tasks in Meta Arcade.

Original: We focus on predeﬁned games with a common color scheme to ease transfer.
Revised: We focus on predeﬁned games with a common color scheme to ease transfer.

Original: Future work could examine transfer between color schemes or domain randomization over colors – both are easily implemented given the conﬁgurability of the environment.
Revised: Future work could examine transfer between color schemes or domain randomization over colors – both are easily implemented given the conﬁgurability of the environment.

Original: A full table of the characteristics of predeﬁned games, as well as their objectives, can be found in the appendix (Table 6).

Original: Each of these games could also be varied in appearance and dynamics, and it does not necessarily include characteristics of potential new games deﬁned to study transfer.

Original: We ﬁrst select three predeﬁned games which include skills that could readily transfer to other games.
Revised: We ﬁrst select three predeﬁned games which include skills that could readily transfer to other games.

Original: The three games are learned together with a single PPO policy (multitask learning) for 50 million steps.
Revised: The three games are learned together with a single PPO policy (multitask learning) for 50 million steps.

Original: We use a model checkpoint from 30 million steps as a starting point for additional games that may have common skills or features.
Revised: We use a model checkpoint from 30 million steps as a starting point for additional games that may have common skills or features.

Original: The decision to use the 30 million step checkpoint was inspired by Parisotto et al.
Revised: The decision to use the 30 million step checkpoint was inspired by Parisotto et al.

Original: (2016), which notes that training for a long time may lead to overﬁt policies that that do not transfer well between tasks.
Revised: (2016), which notes that training for a long time may lead to overﬁt policies that that do not transfer well between tasks.

Original: Unlike Parisotto et al.
Revised: Unlike Parisotto et al.

Original: (2016), we did not re-initialize the ﬁnal layer or change the model size, as this experiment is exploratory.
Revised: (2016), we did not re-initialize the ﬁnal layer or change the model size, as this experiment is exploratory.

Original: When attempting transfer, the new games were trained for 1 million steps, as in Nichol et al.
Revised: When attempting transfer, the new games were trained for 1 million steps, as in Nichol et al.

Original: (2018b).
Revised: (2018b).

Original: The ﬁrst experiment examined transfer between paddle-based games.
Revised: The ﬁrst experiment examined transfer between paddle-based games.

Original: The three multitask games were Breakout, Duel, and Pong, which demonstrate a variety of game objectives that may include a paddle for the player.
Revised: The three multitask games were Breakout, Duel, and Pong, which demonstrate a variety of game objectives that may include a paddle for the player.

Original: These three tasks were trained jointly to create a single policy that performed well on all three games.
Revised: These three tasks were trained jointly to create a single policy that performed well on all three games.

Original: Transfer from this policy was attempted to Battle Pong, Erosion, Pong
Revised: Transfer from this policy was attempted to Battle Pong, Erosion, Pong

Original: Breakout, and Shootout independently.
Revised: Breakout, and Shootout independently.

Original: These results can be seen in Figure 4 (Top).
Revised: These results can be seen in Figure 4 (Top).

Original: The second experiment examined recombination of more diverse skills, including two-dimensional movement.
Revised: The second experiment examined recombination of more diverse skills, including two-dimensional movement.

Original: The three multitask games were Collect Five (demonstrating block collection), Hedge
Revised: The three multitask games were Collect Five (demonstrating block collection), Hedge

Original: Maze (demonstrating the traversal objective), and Erosion (demonstrating shooting).
Revised: Maze (demonstrating the traversal objective), and Erosion (demonstrating shooting).

Original: Transfer was attempted to Avalanche, Dungeon (without hazards), Seek Destroy, and Tunneler.
Revised: Transfer was attempted to Avalanche, Dungeon (without hazards), Seek Destroy, and Tunneler.

Original: These results can be seen in Figure 4 (Bottom).
Revised: These results can be seen in Figure 4 (Bottom).

Original: The paddle-based experiments showed clear transfer among the games using a bouncing ball.
Revised: The paddle-based experiments showed clear transfer among the games using a bouncing ball.

Original: This mechanism appears in both Breakout and Pong, which were closely tied during multitask training, indicating that this common skill was possibly learned in a generic way to satisfy both games.
Revised: This mechanism appears in both Breakout and Pong, which were closely tied during multitask training, indicating that this common skill was possibly learned in a generic way to satisfy both games.

Original: When transferring to Battle Pong and Pong Breakout, there was a signiﬁcant beneﬁt over starting from scratch.
Revised: When transferring to Battle Pong and Pong Breakout, there was a signiﬁcant beneﬁt over starting from scratch.

Original: There was little beneﬁt when transferring to Erosion or Shootout, which are shooting-basedgames.
Revised: There was little beneﬁt when transferring to Erosion or Shootout, which are shooting-based

Original: However, these games are learned incredibly quickly and perhaps a more difﬁcult game was needed to visualize the importance of transfer.

Original: The second experiment using a more diverse range of skills had less clear results.
Revised: The second experiment using a more diverse range of skills had less clear results.

Original: There was no observed beneﬁt to transfer aside from the game Tunneler, which was trained for 3 million steps instead of 1 million to ﬂush out this result.
Revised: There was no observed beneﬁt to transfer aside from the game Tunneler, which was trained for 3 million steps instead of 1 million to ﬂush out this result.

Original: This was surprising, as Tunneler is one of the more challenging predeﬁned games in the suite, appearing unsolvable by PPO in baselining.
Revised: This was surprising, as Tunneler is one of the more challenging predeﬁned games in the suite, appearing unsolvable by PPO in baselining.

Original: Presumably, the combination of Hedge Maze (reach the other side) and Erosion (destroy blocks) allowed the transfer learner to more readily experience the goal, which lead to successful training.
Revised: Presumably, the combination of Hedge Maze (reach the other side) and Erosion (destroy blocks) allowed the transfer learner to more readily experience the goal, which lead to successful training.

Original: How to best choose what games to include in a meta-learning curriculum is an area of research worthy of further exploration with Meta Arcade.
Revised: How to best choose what games to include in a meta-learning curriculum is an area of research worthy of further exploration with Meta Arcade.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: In this paper we presented Meta Arcade, a tool to create 2D arcade games that are built from common mechanisms, such that skills or learned features from one game may be applicable to others.
Revised: In this paper we presented Meta Arcade, a tool to create 2D arcade games that are built from common mechanisms, such that skills or learned features from one game may be applicable to others.

Original: The games are highly customizable by changing a common set of parameters, enabling a vast array of variations for any one game.
Revised: The games are highly customizable by changing a common set of parameters, enabling a vast array of variations for any one game.

Original: We provide a set of single-task learner benchmarks on the games using PPO, and then show how taking advantage of Meta Arcade’s features and curriculum learning enable successful learning on tasks where single-task learning failed.
Revised: We provide a set of single-task learner benchmarks on the games using PPO, and then show how taking advantage of Meta Arcade’s features and curriculum learning enable successful learning on tasks where single-task learning failed.

Original: Future work formalizing this into pedagogy would be beneﬁcial for problems where such conﬁgurability is possible.
Revised: Future work formalizing this into pedagogy would be beneﬁcial for problems where such conﬁgurability is possible.

Original: We also summarized initial results demonstrating successful policy transfer from sets of Meta Arcade games to other games in the suite.
Revised: We also summarized initial results demonstrating successful policy transfer from sets of Meta Arcade games to other games in the suite.

Original: Future work could use Meta Arcade to characterize such transfer between tasks, explore when transfer is possible, or look to continual learning for sequences of tasks.
Revised: Future work could use Meta Arcade to characterize such transfer between tasks, explore when transfer is possible, or look to continual learning for sequences of tasks.

Original: While we focused on multitask learning, the application of meta-learning algorithms to Meta Arcade transfer experiments would also be of considerable interest. 
Revised: While we focused on multitask learning, the application of meta-learning algorithms to Meta Arcade transfer experiments would also be of considerable interest. D ISCLAIMER

Revised: Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reﬂect the views of the Ofﬁce of Naval Research.

Revised: The views, opinions, and/or ﬁndings expressed are those of the author(s) and should not be interpreted as representing the ofﬁcial views or policies of the Department of Defense or the U.S. Government.

Original: A A PPENDIX A. PPO I MPLEMENTATION AND H YPERPARAMETERS
Revised: A A PPENDIX A. PPO I MPLEMENTATION AND H YPERPARAMETERS

Original: In the benchmarking and curriculum experiments, the following hyperparameters were used for
Revised: In the benchmarking and curriculum experiments, the following hyperparameters were used for

Original: PPO:
Revised: PPO:

Original: [Table]
Revised: [Table]

Original: Parameters such as the clipping parameter and entropy coefﬁcient were held ﬁxed during training, and not annealed towards zero as in some implementations.
Revised: Parameters such as the clipping parameter and entropy coefﬁcient were held ﬁxed during training, and not annealed towards zero as in some implementations.

Original: A. C URRICULUM R ESULTS
Revised: A. C URRICULUM R ESULTS

Original: In Table 5, we show the learning curves and explanatory diagrams of the curricula described in
Revised: In Table 5, we show the learning curves and explanatory diagrams of the curricula described in

Original: Section 4.2.
Revised: Section 4.2.

Original: A. T ASK B OUNDARY D ISCUSSION
Revised: A. T ASK B OUNDARY D ISCUSSION

Original: When the parameters of a task are randomized or modiﬁable, as in Meta Arcade, there is a gray area between varying the parameters of the game and introducing a separate task.
Revised: When the parameters of a task are randomized or modiﬁable, as in Meta Arcade, there is a gray area between varying the parameters of the game and introducing a separate task.

Original: The conﬁgurablility of the environment suite makes it easier to investigate this boundary.
Revised: The conﬁgurablility of the environment suite makes it easier to investigate this boundary.

Original: In our experiments, this was especially relevant to curriculum learning: while the task slowly changed over time, the start and end conﬁgurations often looked quite different from each other.
Revised: In our experiments, this was especially relevant to curriculum learning: while the task slowly changed over time, the start and end conﬁgurations often looked quite different from each other.

Original: Most curricula for games with the objective to cross the screen had a common starting point – the possibility of a blank play area with no obstacles.
Revised: Most curricula for games with the objective to cross the screen had a common starting point – the possibility of a blank play area with no obstacles.

Original: If this were deﬁned as a separate task then these curricula could be considered multitask problems with at least one task in common.
Revised: If this were deﬁned as a separate task then these curricula could be considered multitask problems with at least one task in common.

Original: Similarly, the curriculum for Invasion started with blocks that were collectable instead of harmful, in order to encourage positioning the paddle beneath the falling blocks.
Revised: Similarly, the curriculum for Invasion started with blocks that were collectable instead of harmful, in order to encourage positioning the paddle beneath the falling blocks.

Original: This mechanism is also found in the predeﬁned game Avalanche.
Revised: This mechanism is also found in the predeﬁned game Avalanche.

Original: In general, it is useful to think of the predeﬁned games as data points in the space of possible tasks, and it is up to future researchers to deﬁne metrics for similarity and uniqueness.
Revised: In general, it is useful to think of the predeﬁned games as data points in the space of possible tasks, and it is up to future researchers to deﬁne metrics for similarity and uniqueness.

Original: A. P REDEFINED G AMES C HARACTERISTICS 
Revised: A. P REDEFINED G AMES C HARACTERISTICS For

Original: For reference, we provide a list of predeﬁned games and which environment components they include (Table 6).
Revised: reference, we provide a list of predeﬁned games and which environment components they include (Table 6).

Original: This is especially useful for determining if any two (or more) games may be candidates for transfer learning.
Revised: This is especially useful for determining if any two (or more) games may be candidates for transfer learning.

Original: [Table]
Revised: [Table]

Original: T a b l e 6 : S u mm a r y
Revised: T a b l e 6 : S u mm a r y

Original: o
Revised: o

Original: f
Revised: f

Original: s k ill s a nd e l e m e n t s c o mm on t o t h e s e t
Revised: s k ill s a nd e l e m e n t s c o mm on t o t h e s e t

Original: o f p r e d e ﬁ n e d g a m e s .
Revised: o f p r e d e ﬁ n e d g a m e s .

Original: [Table] [Figure]
Revised: [Table] [Figure]