Original: D OJO : A  L ARGE S CALEB ENCHMARK FOR M ULTIT ASK R EINFORCEMENT L EARNINGAnonymous authors
Revised: D OJO : A B ENCHMARK FOR L ARGE S CALE M ULTIT ASK R EINFORCEMENT L EARNING

Original: Paper under double-blind review
Revised: e11809917@student.tuwien.ac.at

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce Dojo , a reinforcement learning environment intended as a benchmark for evaluating RL agents’ capabilities in the areas of multi-task learning, generalization, transfer learning, and curriculum learning.
Revised: We introduce Dojo , a reinforcement learning environment intended as a benchmark for evaluating RL agents’ capabilities in the areas of multi-task learning, generalization, transfer learning, and curriculum learning.

Original: In this work, we motivate our benchmark, compare it to existing methods, and empirically demonstrate its suitability for the purpose of studying cross-task generalization.
Revised: In this work, we motivate our benchmark, compare it to existing methods and empirically demonstrate its suitability for the purpose of studying cross-task generalization.

Original: We establish a multi-task baseline across the whole benchmark as a reference for future research and discuss the achieved results and encountered issues.
Revised: We establish a multi-task baseline across the whole benchmark as a reference for future research and discuss the achieved results and encountered issues.

Original: Finally, we provide experimental protocols and evaluation procedures to ensure that results are comparable across experiments.
Revised: Finally, we provide experimental protocols and evaluation procedures to ensure that results are comparable across experiments.

Original: We also supply tools allowing researchers to easily understand their agents’ performance across a wide variety of metrics.
Revised: We also supply tools allowing researchers to easily understand their agents’ performance across a wide variety of metrics

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent research in reinforcement learning has yielded an abundance of learning algorithms able to learn to perform complex sequential decision-making tasks without human supervision (Mnih et al., 2015; Silver et al., 2016; Hessel et al., 2018).
Revised: Recent research in reinforcement learning has yielded an abundance of learning algorithms able to learn to perform complex sequential decision-making tasks without human supervision (Mnih et al., 2015; Silver et al., 2016; Hessel et al., 2018).

Original: While these results are remarkable, in most cases, each trained agent can only perform a single, specific task and new agent instances need to be trained from scratch for every subsequent task (Hessel et al., 2019; Vithayathil Varghese &
Revised: While these results are remarkable, in most cases, each trained agent can only perform a single, specific task and new agent instances need to be trained from scratch for every subsequent task (Hessel et al., 2019; Vithayathil Varghese &

Original: Mahmoud, 2020).
Revised: Mahmoud, 2020).

Original: Additionally, prior work has shown that learned policies are often brittle and can break down when environment features are even slightly perturbed (Raileanu et al., 2020).
Revised: Additionally, prior work has shown that learned policies are often brittle and can break down when environment features are even slightly perturbed (Raileanu et al., 2020).

Original: This can also be observed within research on generalization where policies learned through RL have been shown to generalize poorly to previously unseen tasks (Cobbe et al., 2019; 2020).
Revised: This can also be observed within research on generalization where policies learned through RL have been shown to generalize poorly to previously unseen tasks (Cobbe et al., 2019; 2020).

Original: The same issues may also hinder the adoption of RL for practical applications, where robustness and the ability to adapt within a rich and dynamically changing world are essential.
Revised: The same issues may also hinder the adoption of RL for practical applications, where robustness and the ability to adapt within a rich and dynamically changing world are essential.

Original: For these challenges, the field of multi-task reinforcement learning (MTRL) has emerged as a potential solution.
Revised: For these challenges, the field of multi-task reinforcement learning (MTRL) has emerged as a potential solution.

Original: MTRL is concerned with developing single RL agents that can perform a whole spectrum of tasks without needing to be retrained.
Revised: MTRL is concerned with developing single RL agents that can perform a whole spectrum of tasks without needing to be retrained.

Original: Apart from practical advantages, these methods aim to yield policies that are substantially more robust, generalize better to other tasks within the training distribution, and transfer to entirely different tasks (Team et al., 2021).
Revised: Apart from practical advantages, these methods aim to yield policies that are substantially more robust, generalize better to other tasks within the training distribution, and transfer to entirely different tasks (Team et al., 2021).

Original: Additionally, multitask learning agents can exhibit better data efficiency since experience can be shared across tasks (Espeholt et al., 2018).
Revised: Additionally, multitask learning agents can exhibit better data efficiency since experience can be shared across tasks (Espeholt et al., 2018).

Original: Standardized benchmarks such as ImageNet (Deng et al., 2009) and the Arcade Learning Environment (ALE)
Revised: Standardized benchmarks such as ImageNet (Deng et al., 2009) and the Arcade Learning Environment (ALE)

Original: (Bellemare et al., 2013; Machado et al., 2018) have been a cornerstone of research in many areas of machine learning.
Revised: (Bellemare et al., 2013; Machado et al., 2018) have been a cornerstone of research in many areas of machine learning.

Original: They allow researchers to reliably compare entirely dissimilar approaches to important problems and serve as a measure of research progress.
Revised: They allow researchers to reliably compare entirely dissimilar approaches to important problems and serve as a measure of research progress.

Original: While many such benchmarks exist for general RL, we believe that most of these are not well suited for research in multi-task RL.
Revised: While many such benchmarks exist for general RL, we believe that most of these are not well suited for research in multi-task RL.

Original: If that is indeed the case, the continued reliance on them could greatly impede further research progress.
Revised: If that is indeed the case, the continued reliance on them could greatly impede further research progress.

Original: Thus, in this paper, we present Dojo , a new benchmark aiming to fill this gap.
Revised: Thus, in this paper, we present Dojo , a new benchmark aiming to fill this gap.

Original: Dojo consists of close to 2000 different video game tasks drawn from several existing libraries.
Revised: Dojo consists of close to 2000 different video game tasks drawn from several existing libraries.

Original: In the implementation, special attention was paid to performance, configurability, and ease of use.
Revised: In the implementation, special attention was paid to performance, configurability, and ease of use.

Original: In particular, Dojo closely adheres to the gym environment API (Brockman et al., 2016) and can, for the most part, be used as a drop-in replacement for existing benchmarks while also supporting more specific use cases.
Revised: In particular, Dojo closely adheres to the gym environment API (Brockman et al., 2016) and can, for the most part, be used as a drop-in replacement for existing benchmarks while also supporting more specific use cases.

Original: In this work, we particularly focus on applications in multi-task learning, but we believe that the benchmark will also be a useful tool in related areas such as curriculum learning, transfer learning, and other open-ended learning settings as well.
Revised: In this work, we particularly focus on applications in multi-task learning, but we believe that the benchmark will also be a useful tool in related areas such as curriculum learning, transfer learning, as well as other open-ended learning settings.

Original: In Section 2 we give an overview of the current issues present in multi-task RL and discuss how recent methods have attempted to address them.
Revised: In Section 2 we give an overview of the current issues present in multi-task RL and discuss how recent methods have attempted to address them.

Original: We then list available benchmarks and discuss how they are used to (but frequently fail to) evaluate approaches to these issues.
Revised: We then list available benchmarks and discuss how they are used to (but frequently fail to) evaluate approaches to these issues.

Original: In Section 3, we describe the specifics of our own benchmark, compare it to others, and detail how it avoids the listed pitfalls.
Revised: In Section 3, we describe the specifics of our own benchmark, compare it to others, and detail how it avoids the listed pitfalls.

Original: In Section 4, we present experiments performed to support the design of the benchmark itself, better understand the effect of knowledge transfer within Dojo, and establish a preliminary multi-task baseline.
Revised: In Section 4, we present experiments performed to support the design of the benchmark itself, better understand the effect of knowledge transfer within Dojo, and establish a preliminary multi-task baseline.

Original:  R ELATED W ORK 2. M ULTI - TASK REINFORCEMENT LEARNING
Revised:  R ELATED W ORK 2. M ULTI - TASK REINFORCEMENT LEARNING

Original: Multi-task RL is uniquely affected by a number of issues not present in general single-task RL.
Revised: Multi-task RL is uniquely affected by a number of issues not present in general single-task RL.

Original: This section provides a brief overview of these and discusses existing solutions from the literature.
Revised: This section provides a brief overview of these and discusses existing solutions from the literature

Original: Task interference.
Revised: Task interference.

Original: When learning multiple tasks jointly, tasks may interfere with each other for a variety of reasons.
Revised: When learning multiple tasks jointly, tasks may interfere with each other for a variety of reasons.

Original: Negative knowledge transfer refers to the case where a behavior learned in one task negatively affects the agent’s learning progress in another – for example by impeding exploration of the environment (Vithayathil Varghese & Mahmoud, 2020).
Revised: Negative knowledge transfer refers to the case where a behavior learned in one task negatively affects the agent’s learning progress in another – for example by impeding exploration of the environment (Vithayathil Varghese & Mahmoud, 2020).

Original: Catastrophic forgetting can also destabilize learning, particularly in the continual learning case (Fedus et al., 2020).
Revised: Catastrophic forgetting can also destabilize learning, particularly in the continual learning case (Fedus et al., 2020).

Original: Finally, differences in the return magnitudes of different tasks which in turn cause differences in the error magnitudes can cause an imbalance of learning progress across tasks (Hessel et al., 2019;
Revised: Finally, differences in the return magnitudes of different tasks which in turn cause differences in the error magnitudes can cause an imbalance of learning progress across tasks (Hessel et al., 2019;

Original: Schaul et al., 2021).
Revised: Schaul et al., 2021).

Original: Since the return distribution is generally unknown and non-stationary, naive normalization methods are insufficient.
Revised: Since the return distribution is generally unknown and non-stationary, naive normalization methods are insufficient.

Original: PopArt normalization (van Hasselt et al., 2016), originally introduced for the single-task setting, normalizes targets online but preserves the original unnormalized value function by modifying the last layer of the value network.
Revised: PopArt normalization (van Hasselt et al., 2016), originally introduced for the single-task setting, normalizes targets online but preserves the original unnormalized value function by modifying the last layer of the value network.

Original: Return-based scaling (Schaul et al., 2021) is an algorithm-agnostic method that instead rescales the error based on a normalization factor derived from statistics of the environment.
Revised: Return-based scaling (Schaul et al., 2021) is an algorithm-agnostic method that instead rescales the error based on a normalization factor derived from statistics of the environment.

Original: PopArt-IMPALA (Hessel et al., 2019) combines the IMPALA algorithm from (Espeholt et al., 2018) with PopArt normalization and applies it to the multi-task setting.
Revised: PopArt-IMPALA (Hessel et al., 2019) combines the IMPALA algorithm from (Espeholt et al., 2018) with PopArt normalization and applies it to the multi-task setting.

Original: Training speed.
Revised: Training speed.

Original: Although large computational costs and long training times have also been a trait of much of recent work in general RL (Badia et al., 2020; Kapturowski et al., 2019; Schrittwieser et al., 2020; Hessel et al., 2018), these issues affect multi-task RL to an even greater extent.
Revised: Although large computational costs and long training times have also been a trait of much of recent work in general RL (Badia et al., 2020; Kapturowski et al., 2019; Schrittwieser et al., 2020; Hessel et al., 2018), these issues affect multi-task RL to an even greater extent.

Original: Evaluation of MTRL methods necessitates experiments with benchmarks consisting of larger numbers of task, since MTRL-specific issues may only be observable there.
Revised: Evaluation of MTRL methods necessitates experiments with benchmarks consisting of larger numbers of tasks, since MTRL-specific issues may only be observable there.

Original: Thus, training speed has been a focus area of some recent work in MTRL.
Revised: Thus, training speed has been a focus area of some recent work in MTRL.

Original: In particular, IMPALA (Espeholt et al., 2018), an actor-critic agent focusing on scalable, training throughput, and computational efficiency, has been used as a research platform and baseline in some recent work (Hessel et al., 2019; Luo et al., 2020).
Revised: In particular, IMPALA (Espeholt et al., 2018), an actor-critic agent focusing on scalability, training throughput, and computational efficiency, has been used as a research platform and baseline in some recent work (Hessel et al., 2019; Luo et al., 2020).

Original: 2. B ENCHMARKS IN REINFORCEMENT LEARNING
Revised: 2. B ENCHMARKS IN REINFORCEMENT LEARNING

Original: In this section we provide an overview of popular existing benchmarks in RL and discuss their suitability for the purpose of MTRL research.
Revised: In this section, we provide an overview of popular existing benchmarks in RL and discuss their suitability for the purpose of MTRL research.

Original: We focus on visual, game-based environments, since these are most closely related to our benchmark.
Revised: We focus on visual, game-based environments since these are most closely related to our benchmark.

Original: For a more comprehensive list of RL environments commonly used to study generalization (many of which are also relevant for multi-task learning) see
Revised: For a more comprehensive list of RL environments commonly used to study generalization (many of which are also relevant for multi-task learning) see

Original: Kirk et al.
Revised: Kirk et al.

Original: Arcade Learning Environment (ALE).
Revised: Arcade Learning Environment (ALE).

Original: Since its introduction in 2013, the ALE (Bellemare et al., 2013; Machado et al., 2018) has been one of the primary benchmarks used in RL research.
Revised: Since its introduction in 2013, the ALE (Bellemare et al., 2013; Machado et al., 2018) has been one of the primary benchmarks used in RL research.

Original: It consists of 57 Atari games, each of which exposes a discrete action space with up to 18 actions and uses RGB images as observations.
Revised: It consists of 57 Atari games, each of which exposes a discrete action space with up to 18 actions and uses RGB images as observations.

Original: While the diversity of games in the ALE has been regarded as a benefit for evaluating general RL methods, the minimal task overlap and stark differences between games make it less suitable for MTRL research.
Revised: While the diversity of games in the ALE has been regarded as a benefit for evaluating general RL methods, the minimal task overlap and stark differences between games make it less suitable for MTRL research.

Original: Procgen.
Revised: Procgen.

Original: This suite of 16 games specifically developed for RL research, replicates some of the games in the ALE, but adds procedural level generation and improved configurability and perfor- mance (Cobbe et al., 2020).
Revised: This suite of 16 games specifically developed for RL research, replicates some of the games in the ALE, but adds procedural level generation and improved configurability and perfor- mance (Cobbe et al., 2020).

Original: Like its predecessor CoinRun (Cobbe et al., 2019), Procgen has been used primarily for research in generalization (Hilton et al., 2020; Raileanu et al., 2020;
Revised: Like its predecessor CoinRun (Cobbe et al., 2019), Procgen has been used primarily for research in generalization (Hilton et al., 2020; Raileanu et al., 2020;

Original: Laskin et al., 2020).
Revised: Laskin et al., 2020).

Original: Meta Arcade.
Revised: Meta Arcade.

Original: While similar to the ALE, Meta Arcade enables researchers to modify many aspects of each provided game, define new games, and allows for a degree of procedural generation within games.
Revised: While similar to the ALE, Meta Arcade enables researchers to modify many aspects of each provided game, define new games, and allows for a degree of procedural generation within games.

Original: The benchmark is intended for research in meta and multi-task learning and includespredefined games (Staley et al., 2021).
Revised: The benchmark is intended for research in meta and multi-task learning and includespredefined games (Staley et al., 2021).

Original: gym-retro.
Revised: gym-retro.

Original: This library serves as an interface between classic video games and RL algorithms by exposing various emulators as RL environments conforming to the gym-API (Nichol et al., 2018).
Revised: This library serves as an interface between classic video games and RL algorithms by exposing various emulators as RL environments conforming to the gym-API (Nichol et al., 2018).

Original: The gym-retro package ships with around 1000 game integrations included, but due to per-game differences in action, observation, and reward spaces can be cumbersome for use in multi-task learning.
Revised: The gym-retro package ships with around 1000 game integrations included, but due to per-game differences in action, observation, and reward spaces can be cumbersome for use in multi-task learning.

Original: DeepMind Lab.
Revised: DeepMind Lab.

Original: DeepMind Lab (Beattie et al., 2016) is a library providing access to a collection of 3d game-based tasks that has been used in similar settings as the ALE.
Revised: DeepMind Lab (Beattie et al., 2016) is a library providing access to a collection of 3d game-based tasks that has been used in similar settings as the ALE.

Original: It includes 10 predefined environments along with other contributed environments such as the 30 DMLab-30 tasks.
Revised: It includes 10 predefined environments along with other contributed environments such as the 30 DMLab-30 tasks.

Original: MineRL.
Revised: MineRL.

Original: MineRL is an open-ended environment based on the game of Minecraft and includes more than 20 predefined tasks.
Revised: MineRL is an open-ended environment based on the game of Minecraft and includes more than 20 predefined tasks.

Original: It has been employed in competitions on sample efficient RL with human demonstrations (Shah et al., 2021;
Revised: It has been employed in competitions on sample efficient RL with human demonstrations (Shah et al., 2021;

Original: Guss et al.).
Revised: Guss et al.).

Original: We find that many of the existing benchmarks suffer from several issues that make them unsuitable for research in MTRL.
Revised: We find that many of the existing benchmarks suffer from several issues that make them unsuitable for research in MTRL.

Original: Ideally, MTRL benchmarks should contain a large number of tasks including both clusters of similar and dissimilar tasks.
Revised: Ideally, MTRL benchmarks should contain a large number of tasks including both clusters of similar and dissimilar tasks.

Original: This would allow researchers to evaluate how well their algorithms can generalize across related tasks, avoid interference between unrelated tasks, and scale to large numbers of tasks.
Revised: This would allow researchers to evaluate how well their algorithms can generalize across related tasks, avoid interference between unrelated tasks, and scale to large numbers of tasks.

Original: In contrast, most existing benchmarks include only few, largely unrelated tasks with little skill overlap (such as the ALE or the different Procgen games).
Revised: In contrast, most existing benchmarks include only few, largely unrelated tasks with little skill overlap (such as the ALE or the different Procgen games).

Original: There,
Revised: There,

Original: MTRL approaches have little benefit over conventional single-task methods, while suffering from additional complexity.
Revised: MTRL approaches have little benefit over conventional single-task methods, while suffering from additional complexity.

Original: Furthermore, the limited number of tasks makes it difficult to investigate how
Revised: Furthermore, the limited number of tasks makes it difficult to investigate how

Original: MTRL methods scale with the number of tasks.
Revised: MTRL methods scale with the number of tasks.

Original: Environments that use procedural generation to increase diversity (such as CoinRun, Procgen, and Meta Arcade) necessarily only randomize certain parts of each game.
Revised: Environments that use procedural generation to increase diversity (such as CoinRun, Procgen, and Meta Arcade) necessarily only randomize certain parts of each game.

Original: While appropriate for research in generalization, this scenario differs from the typical multi-task learning setup which generally includes more significant differences between tasks.
Revised: While appropriate for research in generalization, this scenario differs from the typical multi-task learning setting which generally includes more significant differences between tasks.

Original: Finally, some benchmarks are affected by performance or usability issues.
Revised: Finally, some benchmarks are affected by performance or usability issues.

Original: In particular,
Revised: In particular,

Original: MineRL suffers from the relatively high computational cost of the underlying game and gym-retro requires substantial task-specific preprocessing as mentioned above.
Revised: MineRL suffers from the relatively high computational cost of the underlying game and gym-retro requires substantial task-specific preprocessing as mentioned above.

Original:  D OJO BENCHMARK SPECIFICATION
Revised:  D OJO BENCHMARK SPECIFICATION

Original: The main goal of Dojo is to provide a highly challenging environment that is only efficiently solvable by generalizing and transferring knowledge across games.
Revised: The main goal of Dojo is to provide a highly challenging environment that is only efficiently solvable by generalizing and transferring knowledge across games.

Original: To this end, the benchmark includes many similar tasks to provide ample opportunities for positive knowledge transfer as well as many dissimilar tasks to evaluate how well agents can avoid negative knowledge transfer and task interference problems.
Revised: To this end, the benchmark includes many similar tasks to provide ample opportunities for positive knowledge transfer as well as many dissimilar tasks to evaluate how well agents can avoid negative knowledge transfer and task interference problems.

Original: Finally, the benchmark is easy to use with existing algorithms and includes all necessary tools for preprocessing and evaluation.
Revised: Finally, the benchmark is easy to use with existing algorithms and includes all necessary tools for preprocessing and evaluation.

Original: 3. T ASKS
Revised: 3. T ASKS

Original: The benchmark consists of around 1100 video games from which almost 2000 different RL tasks can be derived.
Revised: The benchmark consists of around 1100 video games from which almost 2000 different RL tasks can be derived.

Original: These games include 104 games drawn from Atari (via the ALE), 925 games from the
Revised: These games include 104 games drawn from Atari (via the ALE), 925 games from the

Original: SNES, NES, GameBoy, Genesis and SMS game consoles (via the gym-retro library), 16 games from the Procgen library, and 23 games that are adaptations of those included in the Meta Arcade library.
Revised: SNES, NES, GameBoy, Genesis and SMS game consoles (via the gym-retro library), 16 games from the Procgen library, and 23 games that are adaptations of those included in the Meta Arcade library.

Original: For many of these games, several variants or options are available: all procgen and MetaArcade games support customizing their graphical appearance and many gym-retro games provide several different levels and difficulty options.
Revised: For many of these games, several variants or options are available: all procgen and MetaArcade games support customizing their graphical appearance and many gym-retro games provide several different levels and difficulty options.

Original: By exposing each game variant as a separate task to be solved, the variety within the benchmark is increased further and brings the total number of tasks close to 2000.
Revised: By exposing each game variant as a separate task to be solved, the variety within the benchmark is increased further and brings the total number of tasks close to 2000.

Original: A small selection of games from the benchmark can be seen in Figure 1.
Revised: A small selection of games from the benchmark can be seen in Figure 1.

Original: In an effort to standardize the use of Dojo we defined several subsets of tasks intended for different kinds of experiments.
Revised: In an effort to standardize the use of Dojo we defined several subsets of tasks intended for different kinds of experiments.

Original: These are supplied in the dojo.tasks module.
Revised: These are supplied in the dojo.tasks module.

Original: In addition to the full set of tasks ( dojo full ), we provide a slightly reduced version ( dojo core ), which excludes 350 hard-exploration tasks with exceptionally sparse rewards.
Revised: In addition to the full set of tasks ( dojo full ), we provide a slightly reduced version ( dojo core ), which excludes 350 hard-exploration tasks with exceptionally sparse rewards.

Original: We expect work focusing on MTRL, generalization, and transfer learning to primarily use the core set while work on exploration in multitask settings to use the full benchmark.
Revised: We expect work focusing on MTRL, generalization, and transfer learning to primarily use the core set while work on exploration in multitask settings to use the full benchmark.

Original: Additionally, a number of clusters of semantically related tasks  are provided as easier test-beds for multi-task learning.
Revised: Additionally, a number of clusters of semantically related tasks (platforming, fighting, and air combat games) are provided as easier test-beds for multi-task learning.

Original: 3. A CTIONS AND OBSERVATIONS
Revised: 3. A CTIONS AND OBSERVATIONS

Original: To enable simple use of the environment with existing algorithms, action and observation spaces should be shared across all tasks.
Revised: To enable simple use of the environment with existing algorithms, action and observation spaces should be shared across all tasks.

Original: This is non-trivial, since many of the games differ in the resolution of observations or the kinds of available actions.
Revised: This is non-trivial, since many of the games differ in the resolution of observations or the kinds of available actions.

Original: In particular, games intended for human players were originally developed for different game consoles with different physical controllers.
Revised: In particular, games intended for human players were originally developed for different game consoles with different physical controllers.

Original: Dojo therefore uses a unified action space with 15 discrete actions which are mapped to task-specific action schemes in a sensible way.
Revised: Dojo therefore uses a unified action space with 15 discrete actions which are mapped to task-specific action schemes in a sensible way.

Original: This mapping was chosen such that the semantics of each action are roughly the same across tasks.
Revised: This mapping was chosen such that the semantics of each action are roughly the same across tasks.

Original: Actions that are invalid for a certain task are automatically mapped to the “NOOP” action.
Revised: Actions that are invalid for a certain task are automatically mapped to the “NOOP” action.

Original: The exact mapping is provided in Appendix C. For more advanced use, an action mask is provided that can be used with algorithm-specific implementations of invalid action masking such as for instance discussed in Huang & Onta ˜ n ´ on (2020).
Revised: The exact mapping is provided in Appendix C. For more advanced use, an action mask is provided that can be used with algorithm-specific implementations of invalid action masking such as for instance discussed in Huang & Onta ˜ n ´ on (2020).

Original: The included
Revised: The included

Original: Rainbow-DQN example agent implements this by discarding invalid actions both during environment interaction and within the DQN update step.
Revised: Rainbow-DQN example agent implements this by discarding invalid actions both during environment interaction and within the DQN update step.

Original: The observations for all tasks consist of the screen content of the respective game, but can differ both in resolution and used color space.
Revised: The observations for all tasks consist of the screen content of the respective game, but can differ both in resolution and used color space.

Original: Therefore, all observations are converted to grayscale and scaled to a common resolution of 72 × 72 pixels using area interpolation from OpenCV.
Revised: Therefore, all observations are converted to grayscale and scaled to a common resolution of 72 × 72 pixels using area interpolation from OpenCV.

Original: These hyperparameters were found to be good choices in experiments with a subset of games (see Section 4.3), but can be scaled up depending on the available hardware.
Revised: These hyperparameters were found to be good choices in experiments with a subset of games (see Section 4.3), but can be scaled up depending on the available hardware.

Original: 3. R EWARDS
Revised: 3. R EWARDS

Original: Next we turn our attention to the reward function.
Revised: Next we turn our attention to the reward function.

Original: Dojo reuses the task-specific reward functions provided by the underlying libraries but optionally applies arough normalization step and reward clipping.
Revised: Dojo reuses the task-specific reward functions provided by the underlying libraries but optionally applies a normalization step and reward clipping.

Original: This normalization is necessary since the unnormalized episodic returns range across ten orders of magnitude as can be seen in Figure 2.
Revised: This normalization is necessary since the unnormalized episodic returns range across ten orders of magnitude as can be seen in Figure 2.

Original: Still, use of more advanced normalization schemes such as PopArt is recommended since the applied normalization (dividing by the mean absolute return under the uniform random policy) does not account for the shifting return distribution over the course of training.
Revised: Still, use of more advanced normalization schemes such as PopArt is recommended since the applied normalization (dividing by the mean absolute return under the uniform random policy) does not account for the shifting return distribution over the course of training.

Original: 3. O THER CONSIDERATIONS
Revised: 3. O THER CONSIDERATIONS

Original: Task scheduling.
Revised: Task scheduling.

Original: In existing work in MTRL, all tasks to be learned are commonly run in parallel (Espeholt et al., 2018; Hessel et al., 2019).
Revised: In existing work in MTRL, all tasks to be learned are commonly run in parallel (Espeholt et al., 2018; Hessel et al., 2019).

Original: This approach is computationally efficient and also minimizes the issue of catastrophic forgetting compared to executing tasks sequentially.
Revised: This approach is computationally efficient and also minimizes the issue of catastrophic forgetting compared to executing tasks sequentially.

Original: However, due to the large number of tasks and the fact that gym-retro is not thread-safe (meaning each task needs to be run in a separate process), the high resulting memory requirement make this approach infeasible for Dojo.
Revised: However, due to the large number of tasks and the fact that gym-retro is not thread-safe (meaning each task needs to be run in a separate process), the high resulting memory requirement make this approach infeasible for Dojo.

Original: Instead, a pool of worker processes is maintained, each of which executes a single task that is assigned to it by a central task scheduler.
Revised: Instead, a pool of worker processes is maintained, each of which executes a single task that is assigned to it by a central task scheduler.

Original: Upon episode termination, the scheduler selects a new task to assign to the worker.
Revised: Upon episode termination, the scheduler selects a new task to assign to the worker.

Original: The default scheduling policy chooses tasks based on the total number of environment interactions allocated to each task so far.
Revised: The default scheduling policy chooses tasks based on the total number of environment interactions allocated to each task so far.

Original: To reduce the scheduling overhead, tasks are only replaced after having been run for a minimum of 500 frames in the current session.
Revised: To reduce the scheduling overhead, tasks are only replaced after having been run for a minimum of 500 frames in the current session.

Original: By changing the order and frequency with which tasks are run, the task scheduler can also induce a specific curriculum on the learning process.
Revised: By changing the order and frequency with which tasks are run, the task scheduler can also induce a specific curriculum on the learning process.

Original: Thus, Dojo may also be a useful tool in the area of curriculum learning research.
Revised: Thus, Dojo may also be a useful tool in the area of curriculum learning research.

Original: Seeding.
Revised: Seeding.

Original: To ensure that results are reproducible, Dojo implements a seeding system that ensures that runs that are initialized with the same seed are exactly equal.
Revised: To ensure that results are reproducible, Dojo implements a seeding system that ensures that runs that are initialized with the same seed are exactly equal.

Original: In particular, the task scheduling and game initializations are deterministically derived from the given seed.
Revised: In particular, the task scheduling and game initializations are deterministically derived from the given seed.

Original: Statistics & logging.
Revised: Statistics & logging.

Original: Dojo includes a logging module that continuously records statistics related to the performance of the agent as well as data on internals of the benchmark itself.
Revised: Dojo includes a logging module that continuously records statistics related to the performance of the agent as well as data on internals of the benchmark itself.

Original: Data includes per-episode statistics (return, clipped return, episode length, and the episode action distribution) and global statistics (distribution of actions over time, system resource utilization and performance statistics, and statistics related to the task scheduling).
Revised: Data includes per-episode statistics (return, clipped return, episode length, and the episode action distribution) and global statistics (distribution of actions over time, system resource utilization and performance statistics, and statistics related to the task scheduling).

Original: Additionally, 30-second video snapshots of all currently active tasks are periodically recorded.
Revised: Additionally, 30-second video snapshots of all currently active tasks are periodically recorded.

Original: Implementation.
Revised: Implementation.

Original: Dojo is implemented in Python and makes use of numpy, OpenCV, and the underlying libraries of the individual games.
Revised: Dojo is implemented in Python and makes use of numpy, OpenCV, and the underlying libraries of the individual games.

Original: The environment parallelization is done via Ray, which exhibits a similar performance as other implementations of vectorized environments, but allows for simpler dynamic scheduling of tasks.
Revised: The environment parallelization is done via Ray, which exhibits a similar performance as other implementations of vectorized environments, but allows for simpler dynamic scheduling of tasks.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We perform a number of experiments to (1) determine whether knowledge transfer between tasks in the benchmark genuinely accelerates learning, (2) create a baseline multi-task agent, and (3) find a good choice of preprocessing methods.
Revised: We perform a number of experiments to (1) determine whether knowledge transfer between tasks in the benchmark genuinely accelerates learning, (2) create a baseline multi-task agent, and (3) find a good choice of preprocessing methods.

Original: 4. K NOWLEDGE T RANSFER
Revised: 4. K NOWLEDGE T RANSFER

Original: To evaluate the impact of positive knowledge transfer, we perform the following experiment.
Revised: To evaluate the impact of positive knowledge transfer, we perform the following experiment.

Original: We manually determine clusters of semantically related tasks including a cluster of “Pacman”-like games, a cluster of boxing games, and a cluster of air combat games (see Appendix B for the full list).
Revised: We manually determine clusters of semantically related tasks including a cluster of “Pacman”-like games, a cluster of boxing games, and a cluster of air combat games (see Appendix B for the full list).

Original: While games within a cluster differ substantially in their visual presentation and precise environment dynamics, they all share some underlying structure and require similar skills.
Revised: While games within a cluster differ substantially in their visual presentation and precise environment dynamics, they all share some underlying structure and require similar skills.

Original: Thus, we expect an agent to perform better at a task when trained jointly with tasks from the same cluster than with unrelated other tasks.
Revised: Thus, we expect an agent to perform better at a task when trained jointly with tasks from the same cluster than with unrelated other tasks.

Original: To confirm this, we train a Rainbow-DQN (Hessel et al., 2018) agent as implemented in Schmidt & Schmied (2021) on tasks all sampled either from the same cluster or from the union of all clusters.
Revised: To confirm this, we train a Rainbow-DQN (Hessel et al., 2018) agent as implemented in Schmidt & Schmied (2021) on tasks all sampled either from the same cluster or from the union of all clusters.

Original: The agent used the IMPALA-CNN architecture from Espeholt et al.
Revised: The agent used the IMPALA-CNN architecture from Espeholt et al.

Original: (2018) with twice the number of channels and was trained for 20 million frames in each run.
Revised: (2018) with twice the number of channels and was trained for 20 million frames in each run.

Original: The results presented in Figure 3 show a strong performance improvement in 73% of tasks when trained jointly with tasks drawn from the same cluster.
Revised: The results presented in Figure 3 show a strong performance improvement in 73% of tasks when trained jointly with tasks drawn from the same cluster.

Original: For reference, the mean and median difference in evaluation scores across tasks were 72% and 43% respectively.
Revised: For reference, the mean and median difference in evaluation scores across tasks were 72% and 43% respectively.

Original: However, since the maximal achievable scores depend highly on the specific task, the previous metric (percentage of tasks in which the performance improved) is more meaningful.
Revised: However, since the maximal achievable scores depend highly on the specific task, the previous metric (percentage of tasks in which the performance improved) is more meaningful.

Original: For additional context, the performance difference between the same-cluster trained agent and the uniform random policy baseline, is also provided.
Revised: For additional context, the performance difference between the same-cluster trained agent and the uniform random policy baseline, is also provided.

Original: Overall, we can conclude that many tasks within Dojo do indeed share some underlying structure making multi-task learning likely a fruitful approach to solving the benchmark.
Revised: Overall, we can conclude that many tasks within Dojo do indeed share some underlying structure making multi-task learning likely a fruitful approach to solving the benchmark.

Original: (a) Per-task performance difference when train- ing on a task together with tasks from the same semantic cluster instead of with semanti- cally unrelated tasks.
Revised: (a) Per-task performance difference when train- ing on a task together with tasks from the same semantic cluster instead of with semanti- cally unrelated tasks.

Original: For example, the agent performed around 1300% better in the rightmost task ( retro:ArtOfFighting-Snes ), when trained jointly with other fighting games, compared to when trained with arbitrary other tasks.
Revised: For example, the agent performed around 1300% better in the rightmost task ( retro:ArtOfFighting-Snes ), when trained jointly with other fighting games, compared to when trained with arbitrary other tasks.

Original: (b) Per-task performance increase of the agent trained on tasks from the same cluster jointly compared to the uniformrandom policy baseline. While many of the tasks do
Revised: (b) Per-task performance increase of the agent trained on tasks from the same cluster jointly compared to the uniform

Original: [Table]
Revised: [Table]

Original: 4. M ULTI - TASK LEARNING
Revised: 4. M ULTI - TASK LEARNING

Original: Since we were able to confirm the effectiveness of knowledge transfer within small clusters of tasks, we now turn our attention to the full benchmark.
Revised: Since we were able to confirm the effectiveness of knowledge transfer within small clusters of tasks, we now turn our attention to the full benchmark.

Original: To facilitate future research with Dojo, we want to establish a baseline for all tasks and verify that Dojo indeed presents a difficult challenge to current methods.
Revised: To facilitate future research with Dojo, we want to establish a baseline for all tasks and verify that Dojo indeed presents a difficult challenge to current methods.

Original: To this end, we train a Rainbow-DQN agent with the IMPALA architecture on the full suite of tasks.
Revised: To this end, we train a Rainbow-DQN agent with the IMPALA architecture on the full suite of tasks.

Original: Due to compute limitations the agent was trained for only 250 million frames, corresponding to a relatively low 2.5 hours of game play time per task.
Revised: Due to compute limitations the agent was trained for only 250 million frames, corresponding to a relatively low 2.5 hours of game play time per task.

Original: While the mean clipped improvement across tasks (clipped at ± 200% for each task) was 46%, Figure 4 clearly shows that the agent actually performed worse in a substantial share of tasks and in several tasks the performance fully collapsed to zero.
Revised: While the mean clipped improvement across tasks (clipped at ± 200% for each task) was 46%, Figure 4 clearly shows that the agent actually performed worse in a substantial share of tasks and in several tasks the performance fully collapsed to zero.

Original: We hypothesize that this is due to the effects of negative knowledge transfer combined with the short training duration.
Revised: We hypothesize that this is due to the effects of negative knowledge transfer combined with the short training duration.

Original: We also identified a further problem whereby the issue of differences in the return scales of different tasks (as discussed in Section 2.1) may be magnified when using prioritized experience replay (PER) (Schaul et al., 2016).
Revised: We also identified a further problem whereby the issue of differences in the return scales of different tasks (as discussed in Section 2.1) may be magnified when using prioritized experience replay (PER) (Schaul et al., 2016).

Original: This is because PER prioritizes samples from the replay buffer based on the TD-error, poten- tially leading to a disproportionate oversampling of tasks with large return scales in addition to the imbalance within the update steps themselves.
Revised: This is because PER prioritizes samples from the replay buffer based on the TD-error, poten- tially leading to a disproportionate oversampling of tasks with large return scales in addition to the imbalance within the update steps themselves.

Original: 4. P REPROCESSING SETTINGS
Revised: 4. P REPROCESSING SETTINGS

Original: Using the raw high-resolution full-color images as observations preserves the most information, but results in a large memory and compute cost, particularly when using experience replay.
Revised: Using the raw high-resolution full-color images as observations preserves the most information, but results in a large memory and compute cost, particularly when using experience replay.

Original: Thus, we perform a number of experiments to determine sensible default hyperparameters that maximize resource-efficiency while not impeding learning progress due to too much loss of information.
Revised: Thus, we perform a number of experiments to determine sensible default hyperparameters that maximize resource-efficiency while not impeding learning progress due to too much loss of information.

Original: We train a Rainbow-DQN agent individually on 12 randomly chosen tasks for 3.2M frames each, using the parameters listed in Table 1.
Revised: We train a Rainbow-DQN agent individually on 12 randomly chosen tasks for 3.2M frames each, using the parameters listed in Table 1.

Original: [Table]
Revised: [Table]

Original: Among color and grayscale observations there was no consistently better option, indicating that color information is largely irrelevant for solving these tasks.
Revised: Among color and grayscale observations there was no consistently better option, indicating that color information is largely irrelevant for solving these tasks.

Original: Overall, we found grayscale observations with a resolution of 72 × 72 pixels to strike a good balance between performance and resourceefficiency.
Revised: Overall, we found grayscale observations with a resolution of 72 × 72 pixels to strike a good balance between performance and resourceefficiency.

Original: With this setting, an efficiently implemented DQN agent would require approximately 5GB of memory for its replay buffer.
Revised: With this setting, an efficiently implemented DQN agent would require approximately 5GB of memory for its replay buffer.

Original: The full results can be found in Appendix A.
Revised: The full results can be found in Appendix A.

Original:  C ONCLUSION AND FUTURE WORK
Revised:  C ONCLUSION AND FUTURE WORK

Original: In this work, we provided an overview of current methods and open problems in multi-task reinforcement learning, discussed a number of shortcomings affecting existing benchmarks in this setting, and introduced a new benchmark intended to support future research in this area.
Revised: In this work, we provided an overview of current methods and open problems in multi-task reinforcement learning, discussed a number of shortcomings affecting existing benchmarks in this setting, and introduced a new benchmark intended to support future research in this area.

Original: We performed an array of experiments that aided in the design of the benchmark, helped to better understand the role of knowledge transfer in small task clusters, and yielded a preliminary baseline for the benchmark as a whole.
Revised: We performed an array of experiments that aided in the design of the benchmark, helped to better understand the role of knowledge transfer in small task clusters, and yielded a preliminary baseline for the benchmark as a whole.

Original: Within thenear future we intend to implement several additional improvements to the benchmark.
Revised: In the future we intend to implement several additional improvements to the benchmark.

Original: These include enhancements to the invalid action masking to allow for more fine-grained masking for Atari games, implementation of other useful task-schedulers and the option to use custom schedulers for better usability in curriculum learning research.
Revised: These include enhancements to the invalid action masking to allow for more fine-grained masking for Atari games, implementation of other useful task-schedulers and the option to use custom schedulers for better usability in curriculum learning research.

Original: Using these tools, we then plan to perform a more in-depth analysis of existing multi-task learning methods, particularly in regard to how they can be used to improve data-efficiency and speed up exploration
Revised: Using these tools, we then plan to perform a more in-depth analysis of existing multi-task learning methods, particularly in regard to how they can be used to improve data-efficiency and speed up exploration

Original:  P REPROCESSING 
Revised: A P REPROCESSING EXPERIMENTS

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: B T ASK CLUSTERS
Revised: B T ASK CLUSTERS

Original: [Table]
Revised: [Table]

Original: C A CTION MAPPING
Revised: C A CTION MAPPING

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]