Original: D IAGNOSING AND EXPLOITING THE COMPUTATIONAL DEMANDS OF VIDEOS GAMES FOR DEEP REINFORCEMENT LEARNING
Revised: D IAGNOSING AND EXPLOITING THE COMPUTATIONAL DEMANDS OF VIDEOS GAMES FOR DEEP REINFORCEMENT LEARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Humans learn by interacting with their environments and perceiving the outcomes of their actions.
Revised: Humans learn by interacting with their environments and perceiving the outcomes of their actions.

Original: A landmark in artificial intelligence has been the development of deep reinforcement learning (dRL) algorithms capable of doing the same in video games, on par with or better than humans.
Revised: A landmark in artificial intelligence has been the development of deep reinforcement learning (dRL) algorithms capable of doing the same in video games, on par with or better than humans.

Original: However, it remains unclear whether the successes of dRL models reflect advances in visual representation learning, the effectiveness of reinforcement learning algorithms at discovering better policies, or both.
Revised: However, it remains unclear whether the successes of dRL models reflect advances in visual representation learning, the effectiveness of reinforcement learning algorithms at discovering better policies, or both.

Original: To address this question, we introduce the Learning Challenge Diagnosticator (LCD), a tool that separately measures the perceptual and reinforcement learning demands of a task.
Revised: To address this question, we introduce the Learning Challenge Diagnosticator (LCD), a tool that separately measures the perceptual and reinforcement learning demands of a task.

Original: We use LCD to discover a novel taxonomy of challenges in the Procgen benchmark, and demonstrate that these predictions are both highly reliable and can instruct algorithmic development.
Revised: We use LCD to discover a novel taxonomy of challenges in the Procgen benchmark, and demonstrate that these predictions are both highly reliable and can instruct algorithmic development.

Original: More broadly, the LCD reveals multiple failure cases that can occur when optimizing dRL algorithms over entire video game benchmarks like Procgen , and provides a pathway towards more efficient progress.
Revised: More broadly, the LCD reveals multiple failure cases that can occur when optimizing dRL algorithms over entire video game benchmarks like Procgen , and provides a pathway towards more efficient progress.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Gibson famously argued that “The function of vision is not to solve the inverse problem and reconstruct a veridical description of the physical world.
Revised: Gibson famously argued that “The function of vision is not to solve the inverse problem and reconstruct a veridical description of the physical world.

Original: [... It] is to keep perceivers in contact with behaviorally relevant properties of the world they inhabit” (reviewed in Warren 2021).
Revised: [... It] is to keep perceivers in contact with behaviorally relevant properties of the world they inhabit” (reviewed in Warren 2021).

Original: The field of deep reinforcement learning (dRL) has followed Gibson’s tenet since the seminal introduction of deep Q-networks (DQN) (Mnih et al., 2015).
Revised: The field of deep reinforcement learning (dRL) has followed Gibson’s tenet since the seminal introduction of deep Q-networks (DQN) (Mnih et al., 2015).

Original: DQNs rely on reward feedback to train their policies and perceptual systems at the same time to learn to play games from a tabula rasa .
Revised: DQNs rely on reward feedback to train their policies and perceptual systems at the same time to learn to play games from a tabula rasa .

Original: This end-to-end approach of training on individual environments and tasks has supported steady progress in the field of dRL, and newer reinforcement learning algorithms have yielded agents that achieve human or super-human performance in a variety of challenges – from Chess to Go and from Atari games to
Revised: This end-to-end approach of training on individual environments and tasks has supported steady progress in the field of dRL, and newer reinforcement learning algorithms have yielded agents that achieve human or super-human performance in a variety of challenges – from Chess to Go and from Atari games to

Original: Starcraft (Mnih et al., 2015; Silver et al., 2017; 2018; Vinyals et al., 2019).
Revised: Starcraft (Mnih et al., 2015; Silver et al., 2017; 2018; Vinyals et al., 2019).

Original: But Gibson also argued that the ecological niche of animals allows them to exploit task-agnostic mechanisms to simplify the perceptual or behavioral demands of important tasks, like how humans rely on optic flow for navigation (Warren, 2021).
Revised: But Gibson also argued that the ecological niche of animals allows them to exploit task-agnostic mechanisms to simplify the perceptual or behavioral demands of important tasks, like how humans rely on optic flow for navigation (Warren, 2021).

Original: In the decades since Gibson’s writings, it has been found that humans can efficiently find or learn bespoke perceptual features that aid performance on a single task (Li et al., 2004; Scott et al., 2007; Roelfsema et al., 2010; Emberson, 2017), or they can exploit previously learned generalist representations and task abstractions that are useful across multiple tasks and environments (Wiesel & Hubel, 1963; Watanabe et al., 2001; Emberson, 2017; Lehnert et al., 2020;
Revised: In the decades since Gibson’s writings, it has been found that humans can efficiently find or learn bespoke perceptual features that aid performance on a single task (Li et al., 2004; Scott et al., 2007; Roelfsema et al., 2010; Emberson, 2017), or they can exploit previously learned generalist representations and task abstractions that are useful across multiple tasks and environments (Wiesel & Hubel, 1963; Watanabe et al., 2001; Emberson, 2017; Lehnert et al., 2020;

Original: O’Reilly, 2001).
Revised: O’Reilly, 2001).

Original: While there have been attempts at building similarly flexible dRL agents through meta-reinforcement learning (Frans et al., 2018; Xu et al., 2018; 2020; Houthooft et al., 2018; Gupta et al., 2018; Chelu et al., 2020; Pong et al., 2021), these approaches ignore the complexities of perceptual learning and carry large computational burdens that limit them to simplistic scenarios.
Revised: While there have been attempts at building similarly flexible dRL agents through meta-reinforcement learning (Frans et al., 2018; Xu et al., 2018; 2020; Houthooft et al., 2018; Gupta et al., 2018; Chelu et al., 2020; Pong et al., 2021), these approaches ignore the complexities of perceptual learning and carry large computational burdens that limit them to simplistic scenarios.

Original: There is a pressing need for approaches to training dRL agents that can meet the computational demands of a wide variety of environments and tasks.
Revised: There is a pressing need for approaches to training dRL agents that can meet the computational demands of a wide variety of environments and tasks.

Original: One way to build generalist agents is to first reliably diagnose where the computational challenges of a given environment and task lie and adjust the agent to those demands.
Revised: One way to build generalist agents is to first reliably diagnose where the computational challenges of a given environment and task lie and adjust the agent to those demands.

Original: Is the perceptual challenge onerous?
Revised: Is the perceptual challenge onerous?

Original: Is the reward signal for credit assignment especially sparse?
Revised: Is the reward signal for credit assignment especially sparse?

Original: Even partial answers to these questions are instructive for improving an agent, for instance, by pre-determining the extent to which
Revised: Even partial answers to these questions are instructive for improving an agent, for instance, by pre-determining the extent to which

Original: [Figure]
Revised: [Figure]

Original: M o s t s t r u c t u r e d
Revised: M o s t s t r u c t u r e d

Original: L ea s t s t r u c t u r e d
Revised: L ea s t s t r u c t u r e d

Original: C o m p u t e d i ﬀ e r e n ce i n t o t a l r e w a r d
Revised: C o m p u t e d i ﬀ e r e n ce i n t o t a l r e w a r d

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: V i s u a l r e p r e s e n t a t i o n s
Revised: V i s u a l r e p r e s e n t a t i o n s

Original: [Figure]
Revised: [Figure]

Original: C o m p u t e d i ﬀ e r e n ce i n t o t a l r e w a r
Revised: C o m p u t e d i ﬀ e r e n ce i n t o t a l r e w a r

Original: [Figure]
Revised: [Figure]

Original: (b) (c) (d)
Revised: (b) (c) (d)

Original: [Figure]
Revised: [Figure]

Original: learning or it relies on feedback from the world to tune its policy and perception versus drawing from previously learned representations and task abstractions.
Revised: learning or it relies on feedback from the world to tune its policy and perception versus drawing from previously learned representations and task abstractions.

Original: The introduction of diverse video game challenges for dRL, such as the Procgen Benchmark (Cobbe et al., 2019), can serve as a starting point for this investigation.
Revised: The introduction of diverse video game challenges for dRL, such as the Procgen Benchmark (Cobbe et al., 2020), can serve as a starting point for this investigation.

Original: For example, take the game “Plunder” from Procgen (Figure 1a).
Revised: For example, take the game “Plunder” from Procgen (Figure 1a).

Original: Plunder has simple gameplay rules but poses a visual challenge: an agent is asked to shoot all objects that look like a provided cue.
Revised: Plunder has simple gameplay rules but poses a visual challenge: an agent is asked to shoot all objects that look like a provided cue.

Original: The difficulty here lies in assessing whether each object in the environment is the same or different than the cue; a visual routine that is difficult to learn for neural networks (Vaishnav et al., 2022; Kim et al., 2018).
Revised: The difficulty here lies in assessing whether each object in the environment is the same or different than the cue; a visual routine that is difficult to learn for neural networks (Vaishnav et al., 2022; Kim et al., 2018).

Original: For this reason, an agent that can draw from prior experience in learning relevant perceptual routines may perform better than one with a perceptual system tuned for this specific task from scratch.
Revised: For this reason, an agent that can draw from prior experience in learning relevant perceptual routines may perform better than one with a perceptual system tuned for this specific task from scratch.

Original: In contrast, the objects and environments of a game like “Leaper”
Revised: In contrast, the objects and environments of a game like “Leaper”

Original: (Figure 1a) can be identified by color.
Revised: (Figure 1a) can be identified by color.

Original: The game asks agents to use those objects to get from one side of the screen to the other, a task that can likely be learned from reward feedback.
Revised: The game asks agents to use those objects to get from one side of the screen to the other, a task that can likely be learned from reward feedback.

Original: Contributions.
Revised: Contributions.

Original: The ability to systematically, autonomously, and reliably identify an appropriate strategy for learning a task in a given environment would enable the design of generalist dRL agents that can flexibly adapt to challenges as humans do.
Revised: The ability to systematically, autonomously, and reliably identify an appropriate strategy for learning a task in a given environment would enable the design of generalist dRL agents that can flexibly adapt to challenges as humans do.

Original: To make progress towards this goal, we introduce the Learning Challenge Diagnosticator (LCD), a novel tool that identifies the specific computational challenges of an environment and task.
Revised: To make progress towards this goal, we introduce the Learning Challenge Diagnosticator (LCD), a novel tool that identifies the specific computational challenges of an environment and task.

Original: When applied to video games, like those in Procgen , the
Revised: When applied to video games, like those in Procgen , the

Original: LCD measures the difficulty of encoding task-relevant perceptual properties versus the difficulty of optimal policy discovery (i.e., determining which actions lead to rewarding outcomes).
Revised: LCD measures the difficulty of encoding task-relevant perceptual properties versus the difficulty of optimal policy discovery (i.e., determining which actions lead to rewarding outcomes).

Original: • We develop a modified, parameterizable version of Procgen , to systematically manipulate the perceptual and reinforcement learning challenges presented to agents in each individual game.
Revised: • We develop a modified, parameterizable version of Procgen , to systematically manipulate the perceptual and reinforcement learning challenges presented to agents in each individual game.

Original: We provide our version of Procgen and all experimental code at https://anonymous.4open.
Revised: We provide our version of Procgen and all experimental code at https://anonymous.4open.

Original: science/r/lcd-procgen/ .
Revised: science/r/lcd-procgen/ .

Original: • The LCD reveals a novel taxonomy of computational challenges in the games of our modified Procgen .
Revised: • The LCD reveals a novel taxonomy of computational challenges in the games of our modified Procgen .

Original: Some are more visually complex, some are more challenging from the point of view of reinforcement learning, and others strain agents across both of these axes.
Revised: Some are more visually complex, some are more challenging from the point of view of reinforcement learning, and others strain agents across both of these axes.

Original: This taxonomy is preserved across different dRL algorithms and the heterogeneity of the challenges associated with the Procgen games suggests that adopting a single “one size fits all” approach to learning, as is standard in dRL, is suboptimal.
Revised: This taxonomy is preserved across different dRL algorithms and the heterogeneity of the challenges associated with the Procgen games suggests that adopting a single “one size fits all” approach to learning, as is standard in dRL, is suboptimal.

Original: • To address the visual challenges of Procgen games, we adopt a self-supervised visual front-end, which learns perceptual groups from motion cues in games without reward feedback.
Revised: • To address the visual challenges of Procgen games, we adopt a self-supervised visual front-end, which learns perceptual groups from motion cues in games without reward feedback.

Original: To test if reinforcement learning challenges can, in principle, be alleviated, we develop a proof-of-concept approach with reward shaping (Skinner, 1938).
Revised: To test if reinforcement learning challenges can, in principle, be alleviated, we develop a proof-of-concept approach with reward shaping (Skinner, 1938).

Original: • We exploit the computational taxonomy of Procgen revealed by the LCD to shape the design of agents for each game.
Revised: • We exploit the computational taxonomy of Procgen revealed by the LCD to shape the design of agents for each game.

Original: These agents learn more efficiently and perform significantly better than one-size-fits-all dRL agents, suggesting the potential for “adaptive” agents.
Revised: These agents learn more efficiently and perform significantly better than one-size-fits-all dRL agents, suggesting the potential for “adaptive” agents.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Representation learning in dRL
Revised: Representation learning in dRL

Original: Perhaps the main contribution of DQNs was to demonstrate that it is possible to jointly learn visual representations and policies through reward maximization from a tabula rasa to achieve high scores on Atari games (Mnih et al., 2015).
Revised: Perhaps the main contribution of DQNs was to demonstrate that it is possible to jointly learn visual representations and policies through reward maximization from a tabula rasa to achieve high scores on Atari games (Mnih et al., 2015).

Original: Despite this extraordinary achievement, the extrinsic rewards available to algorithms like DQNs have been found to yield poor sample complexity, limit model scale, and cap maximum performance (Jaderberg et al., 2016;
Revised: Despite this extraordinary achievement, the extrinsic rewards available to algorithms like DQNs have been found to yield poor sample complexity, limit model scale, and cap maximum performance (Jaderberg et al., 2016;

Original: Laskin et al., 2020).
Revised: Laskin et al., 2020).

Original: For this reason, there has been a growing number of accounts showing that standard reward learning can be augmented with auxiliary learning objectives and/or pre-training via self-supervision to take steps towards ameliorating limitations of reward-based learning (Jaderberg et al., 2016;
Revised: For this reason, there has been a growing number of accounts showing that standard reward learning can be augmented with auxiliary learning objectives and/or pre-training via selfsupervision to take steps towards ameliorating limitations of reward-based learning (Jaderberg et al., 2016;

Original: Shelhamer et al., 2017; Stooke et al., 2021; Laskin et al., 2020;
Revised: Shelhamer et al., 2017; Stooke et al., 2021; Laskin et al., 2020;

Original: Dittadi et al., 2021; Zhang et al., 2020).
Revised: Dittadi et al., 2021; Zhang et al., 2020; Radosavovic et al. ; Xiao et al.).

Original: Relatedly, it has been shown that initializing agents with structured visual representations can lead to faster learning and better performance in reinforcement learning (Tassa et al., 2018; Davidson & Lake, 2020;
Revised: Relatedly, it has been shown that initializing agents with structured visual representations can lead to faster learning and better performance in reinforcement learning (Tassa et al., 2018; Davidson & Lake, 2020;

Original: Laskin et al., 2020; Stooke et al., 2021).
Revised: Laskin et al., 2020; Stooke et al., 2021).

Original: Measuring visual challenges in artificial intelligence An essential contribution that the cognitive sciences have made to artificial intelligence over the past decade has been the introduction of computational challenges that are easy for humans to solve but strain the capabilities of neural networks.
Revised: Measuring visual challenges in artificial intelligence An essential contribution that the cognitive sciences have made to artificial intelligence over the past decade has been the introduction of computational challenges that are easy for humans to solve but strain the capabilities of neural networks.

Original: These challenges have identified deficiencies in the solutions deep neural networks (DNNs) tend to learn for contour tracing (Linsley et al., 2018b;
Revised: These challenges have identified deficiencies in the solutions deep neural networks (DNNs) tend to learn for contour tracing (Linsley et al., 2018b;

Original: Kim et al., 2020; Tay et al., 2021), segmentation (Kim et al., 2020), object tracking (Linsley et al., 2021), object recognition (Geirhos et al., 2021; 2018), and visual reasoning problems (Fleuret et al., 2011; Kim et al., 2018; Vaishnav et al., 2022).
Revised: Kim et al., 2020; Tay et al., 2021), segmentation (Kim et al., 2020), object tracking (Linsley et al., 2021), object recognition (Geirhos et al., 2021; 2018), and visual reasoning problems (Fleuret et al., 2011; Kim et al., 2018; Vaishnav et al., 2022).

Original: The essential lesson from these studies is that measuring model performance on data sampled from the same distribution that training data came from is not sufficient for establishing challenges and measuring progress on them (Funke et al., 2021).
Revised: The essential lesson from these studies is that measuring model performance on data sampled from the same distribution that training data came from is not sufficient for establishing challenges and measuring progress on them (Funke et al., 2021).

Original: Instead, it is critical to test models on out-ofdistribution samples to find and evaluate challenges (Geirhos et al., 2021; 2018; Linsley et al., 2021; 2020).
Revised: Instead, it is critical to test models on out-ofdistribution samples to find and evaluate challenges (Geirhos et al., 2021; 2018; Linsley et al., 2021; 2020).

Original: We adopt this strategy to measure dRL agent performance in this work.
Revised: We adopt this strategy to measure dRL agent performance in this work.

Original: [Figure]
Revised: [Figure]

Original: Biologically inspired mechanisms for improving neural networks There are multiple examples of the DNN challenges being resolved by the introduction of biologically inspired mechanisms.
Revised: Biologically inspired mechanisms for improving neural networks There are multiple examples of the DNN challenges being resolved by the introduction of biologically inspired mechanisms.

Original: Mechanisms for attention have made significant contributions by enabling DNNs to limit noise and clutter, amplify task-relevant features, and subsequently achieve better performance and sample efficiency during learning (Linsley et al., 2019).
Revised: Mechanisms for attention have made significant contributions by enabling DNNs to limit noise and clutter, amplify task-relevant features, and subsequently achieve better performance and sample efficiency during learning (Linsley et al., 2019).

Original: Others have found that horizontal connections, inspired by those in the early visual cortex of primates, can resolve DNN limitations in solving contour tracing tasks (Linsley et al., 2018a).
Revised: Others have found that horizontal connections, inspired by those in the early visual cortex of primates, can resolve DNN limitations in solving contour tracing tasks (Linsley et al., 2018a).

Original: There is also evidence that forcing dRL agents to acquire compositional abstractions, inspired by those described in the cognitive sciences, helps them learn policies that generalize to novel environments (Lehnert et al., 2020).
Revised: There is also evidence that forcing dRL agents to acquire compositional abstractions, inspired by those described in the cognitive sciences, helps them learn policies that generalize to novel environments (Lehnert et al., 2020).

Original: The mechanisms we propose to solve the computational challenges revealed by the LCD in Procgen are similarly inspired by solutions to those problems adopted by primates and humans.
Revised: The mechanisms we propose to solve the computational challenges revealed by the LCD in Procgen are similarly inspired by solutions to those problems adopted by primates and humans.

Original: Metrics for measuring progress in dRL Progress in dRL has been classically measured by the aggregate rewards and sample efficiency of models in games (Taylor & Stone, 2009; Agarwal et al., 2021; Kirk et al., 2021).
Revised: Metrics for measuring progress in dRL Progress in dRL has been classically measured by the aggregate rewards and sample efficiency of models in games (Taylor & Stone, 2009; Agarwal et al., 2021; Kirk et al., 2021).

Original: The Procgen benchmark took a major step forward beyond what is standard in the field by measuring agent performance on game parameterizations that fell outside of the distribution used for training.
Revised: The Procgen benchmark took a major step forward beyond what is standard in the field by measuring agent performance on game parameterizations that fell outside of the distribution used for training.

Original: While current metrics can adjudicate between different algorithms based on performance in a game, they do not offer insights into the specific computational challenges that agents face.
Revised: While current metrics can adjudicate between different algorithms based on performance in a game, they do not offer insights into the specific computational challenges that agents face.

Original: Our LCD therefore represents a major conceptual leap beyond current approaches to measuring progress in dRL, by quantifying the specific perceptual and reinforcement learning challenges found in a given game, and enabling dRL researchers to more precisely resolve the deficiencies of agents.
Revised: Our LCD therefore represents a major conceptual leap beyond current approaches to measuring progress in dRL, by quantifying the specific perceptual and reinforcement learning challenges found in a given game, and enabling dRL researchers to more precisely resolve the deficiencies of agents.

Original:  T HE L EARNING C HALLENGE D IAGNOSTICATOR
Revised:  T HE L EARNING C HALLENGE D IAGNOSTICATOR

Original: The ability to identify and measure the computational challenges a dRL agent faces is an important step toward designing better algorithms for reinforcement learning.
Revised: The ability to identify and measure the computational challenges a dRL agent faces is an important step toward designing better algorithms for reinforcement learning.

Original: However, to the best of our knowledge, this problem has not yet been addressed in the field.
Revised: However, to the best of our knowledge, this problem has not yet been addressed in the field.

Original: The LCD achieves this goal through a three-step procedure, which separately identifies the perceptual challenge and  reinforcement learning challenge offered by any specific game: ( i ) We begin by modifying a game to put its perceptual representations and reinforcement learning problem under experimental control.
Revised: The LCD achieves this goal through a three-step procedure, which reveals the computational challenges facing an agent. By revealing the computational demands of every game, the LCD enables more efficient algorithm development and insights into how these demands may interact – or not – during learning (see Section 4). Here, we focus on identifying the perceptual versus reinforcement learning challenges offered by any specific game, yielding the following steps: ( i ) We begin by modifying a game to put its perceptual representations and reinforcement learning problem under experimental control.

Original: ( ii ) Next, a set of agents learn to play different versions of a game.
Revised: ( ii ) Next, a set of agents learn to play different versions of a game.

Original: In each version, either the perceptual challenge is perturbed while the reinforcement learning challenge is held constant, or vice versa.
Revised: In each version, either the perceptual challenge is perturbed while the reinforcement learning challenge is held constant, or vice versa.

Original: These perturbations are implemented by sampling either a different perceptual representation of the game or reward scheme.
Revised: These perturbations are implemented by sampling either a different perceptual representation of the game or reward scheme.

Original: ( iii ) Finally, we measure the integrated reward trajectories of agents in response to each type of perturbation.
Revised: ( iii ) Finally, we measure the integrated reward trajectories of agents in response to each type of perturbation.

Original: By separately averaging the scores of agents facing perceptual perturbations and reward perturbations, we compute scores describing the perceptual challenge ( ϕ ) and the reinforcement learning challenge ( ψ ) of each game (Fig.
Revised: By separately averaging the scores of agents facing perceptual perturbations and reward perturbations, we compute scores describing the perceptual challenge ( ϕ ) and the reinforcement learning challenge ( ψ ) of each game (Fig.

Original: 1b and c).
Revised: 1b and c).

Original: Procgen benchmark.
Revised: Procgen benchmark.

Original: We applied the LCD to Procgen (Cobbe et al., 2019), a challenging dRL benchmark consisting of 16 games, each with distinct graphics and gameplay.
Revised: We applied the LCD to Procgen (Cobbe et al., 2020), a challenging dRL benchmark consisting of 16 games, each with distinct graphics and gameplay.

Original: Procgen is opensource and implemented in Python and C++, which made it possible to customize a variety of game parameters for implementing the LCD.
Revised: Procgen is opensource and implemented in Python and C++, which made it possible to customize a variety of game parameters for implementing the LCD.

Original: The parameters we manipulated were game assets (sprites and backgrounds) and reward functions.
Revised: The parameters we manipulated were game assets (sprites and backgrounds) and reward functions.

Original: Procgen generates game levels procedurally, enabling precise control over the levels used for training versus testing and as a result, proper tests of generalization.
Revised: Procgen generates game levels procedurally, enabling precise control over the levels used for training versus testing and as a result, proper tests of generalization.

Original: We followed the training and testing protocol outlined in (Cobbe et al., 2019) in each of our experiments.
Revised: We followed the training and testing protocol outlined in (Cobbe et al., 2020) in each of our experiments.

Original: In brief, all agents were trained for 200M steps across 500 training levels and evaluated on Procgen’s held-out games to measure generalization.
Revised: In brief, all agents were trained for 200M steps across 500 training levels and evaluated on Procgen’s held-out games to measure generalization.

Original: dRL algorithms.
Revised: dRL algorithms.

Original: We apply the LCD to agents trained with either the standard and popular proximal policy optimization (Schulman et al., 2017)
Revised: We apply the LCD to agents trained with either the standard and popular proximal policy optimization (Schulman et al., 2017)

Original: (PPO) or phasic policy gradient (PPG) (Cobbe et al., 2021), a more recent algorithm that attained state-of-the-art performance in Procgen .
Revised: (PPO) or phasic policy gradient (PPG) (Cobbe et al., 2021), a more recent algorithm that attained state-of-the-art performance in Procgen .

Original: PPO hyperparameters were selected to match those used in (Cobbe et al., 2019) for Procgen ’s hard mode.
Revised: PPO hyperparameters were selected to match those used in (Cobbe et al., 2020) for Procgen ’s hard mode.

Original: Similarly, for PPG we used the hyperparameters from (Cobbe et al., 2021).
Revised: Similarly, for PPG we used the hyperparameters from (Cobbe et al., 2021).

Original: The walltime for a single training run of one of these agents averaged 24 hours per game on a standard 16-core Intel Xeon
Revised: The walltime for a single training run of one of these agents averaged 24 hours per game on a standard 16-core Intel Xeon

Original: Gold 6242 CPU with a 24G Titan RTX GPU.
Revised: Gold 6242 CPU with a 24G Titan RTX GPU.

Original: We ran ∼ 450 experiments across 32 GPUs, which took over 10 K GPU hours of computing time.
Revised: We ran ∼ 450 experiments across 32 GPUs, which took over 10 K GPU hours of computing time.

Original: Agent performance at test time was computed using the interquartile mean (IQM) of the final returns (Agarwal et al., 2021).
Revised: Agent performance at test time was computed using the interquartile mean (IQM) of the final returns (Agarwal et al., 2021).

Original: LCD: Perceptual challenge.
Revised: LCD: Perceptual challenge.

Original: To perturb the perceptual complexity in a game and measure its perceptual challenge, we varied the visual input provided to an agent when learning to play the game.
Revised: To perturb the perceptual complexity in a game and measure its perceptual challenge, we varied the visual input provided to an agent when learning to play the game.

Revised: Motivated by decades of cognitive science work on the visual representations that humans rely on for efficiently learning to interact with their environments (Ullman, 1984; Roelfsema et al., 2000;

Original: We tested the impact of learning on three different types of visual inputs, which varied in the amount of structure in the environment they conveyed to agents: the original frame, figure-ground segmentations of each frame, or semantic segmentations of each frame (Fig.
Revised: Roelfsema, 2006b), we tested how well agents could learn to play games when given three different types of visual inputs with varying structure: the original frame, figure-ground segmentations of each frame, or semantic segmentations of each frame (Fig.

Original: 1b,c).
Revised: 1b,c).

Original: Figure-ground segmentations labeled background and object pixels differently, whereas semantic segmentations labeled each element of the frame differently, with consistent labels across frames (see Appendix. A.6).
Revised: Figure-ground segmentations labeled background and object pixels differently, whereas semantic segmentations labeled each element of the frame differently, with consistent labels across frames (see Appendix. A.6).

Original: We measured the perceptual challenge of each game by comparing the performance of three agents trained to solve it, where each agent learned a policy over one of the three different types of visual inputs (Appendix Fig. 6).
Revised: We measured the perceptual challenge of each game by comparing the performance of three agents trained to solve it, where each agent learned a policy over one of the three different types of visual inputs (Appendix Fig. 6).

Original: In this condition, rewards and the reinforcement learning challenge in games were kept as normal.
Revised: In this condition, rewards and the reinforcement learning challenge in games were kept as normal.

Original: We began by computing performance for each agent as the area under the cumulative reward curves (AUC) achieved by agents during generalization (Appendix
Revised: We began by computing performance for each agent as the area under the cumulative reward curves (AUC) achieved by agents during generalization (Appendix

Original: Fig.
Revised: Fig.

Original: 8), which implicitly captured both maximum performance and the sample efficiency needed to get there.
Revised: 8), which implicitly captured both maximum performance and the sample efficiency needed to get there.

Original: Next, we computed the relative change in AUC from the original frame to figure-ground and semantic AUCs, and took the average of these ratios as our final score ϕ (Fig. 1b).
Revised: Next, we computed the relative change in AUC from the original frame to figure-ground and semantic AUCs, and took the average of these ratios as our final score ϕ (Fig. 1b).

Original: The perceptual challenge score ϕ of each game was then normalized to the range [0 , 1] according to the min and max ϕ across all games (Appendix A.2).
Revised: The perceptual challenge score ϕ of each game was then normalized to the range [0 , 1] according to the min and max ϕ across all games (Appendix A.2).

Original: A game with a ϕ approaching 1 indicates that it presents a greater perceptual challenge for reward-based learning, whereas a small ϕ means that reward-based discovery of task-relevant features is sufficient to perform well on that task.
Revised: A game with a ϕ approaching 1 indicates that it presents a greater perceptual challenge for reward-based learning, whereas a small ϕ means that reward-based discovery of task-relevant features is sufficient to perform well on that task.

Original: LCD:
Revised: LCD:

Original: Reinforcement learning challenge.
Revised: Reinforcement learning challenge.

Original: To perturb reinforcement learning in a game, we manipulated the sparsity of rewards available to an agent.
Revised: To perturb reinforcement learning in a game, we manipulated the sparsity of rewards available to an agent.

Original: We did this by decreasing the probability p of receiving a reward at every rewarding event, while also scaling reward magnitudes by 1 /p to preserve the expected magnitude of rewards, and by extension the optimal policy, across all perturbations (Appendix A.3).

Original: We used reward perturbations of p ∈ { 1 , 0 . 75 , 0 . 25 } (Appendix Fig. 7). All agents were given semantic segmentations as inputs, which minimized the perceptual challenge of games and controlled for potential perceptual confounds, such as how the perceptual
Revised: By manipulating reward sparsity, we were able to degrade “learning signal” available to an agent while also preserving the optimal policy,

Original: [Figure]
Revised: [Figure]

Original: S c o r e
Revised: c o r e

Original: S c
Revised: S c

Original: o r e
Revised: o r e

Original: [Figure]
Revised: [Figure]

Original: n n o r m a li z e d s c o r e
Revised: n n o r m a li z e d s c o r e

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: across all perturbations (Appendix A.3).

Revised: We did this by decreasing the probability of receiving a reward at every rewarding event, while also scaling reward magnitudes by 1 /p to preserve the expected magnitude of rewards.

Revised: We used reward perturbations of p ∈ { 1 , 0 . 75 , 0 .

Revised: 25 } (Appendix

Revised: Fig. 7).

Revised: All agents were given semantic segmentations as inputs, which minimized the perceptual challenge of games and controlled for potential perceptual confounds, such as how the perceptual complexity of the pixel inputs of a given game may non-linearly interact with this reinforcement learning perturbation.

Original: We measured the reinforcement learning challenge of each game by comparing the performance of agents trained to solve each of the reward perturbations (Appendix Fig. 9).
Revised: We measured the reinforcement learning challenge of each game by comparing the performance of agents trained to solve each of the reward perturbations (Appendix Fig. 9).

Original: First, we again computed
Revised: First, we again computed

Original: AUCs of the generalization reward curves for each agent, as above but here for different values of p .
Revised: AUCs of the generalization reward curves for each agent, as above, but here for different values of p .

Original: Next, we computed the absolute AUC change from an agent trained on one p to the next and averaged across all changes, yielding the score ψ (Appendix A.4), which was normalized to [0 , 1] in the same way as ϕ .
Revised: Next, we computed the absolute AUC change from an agent trained on one p to the next and averaged across all changes, yielding the score ψ (Appendix A.4), which was normalized to [0 , 1] in the same way as ϕ .

Original: A ψ close to 1 means that it presents a significant reinforcement learning challenge.
Revised: A ψ close to 1 means that it presents a significant reinforcement learning challenge.

Original: Conversely, a game with a small ψ indicates that reward sparsity is not the primary challenge.
Revised: Conversely, a game with a small ψ indicates that reward sparsity is not the primary challenge.

Original: Diagnosing the computational challenges of the Procgen benchmark.
Revised: Diagnosing the computational challenges of the Procgen benchmark.

Original: We began by applying the LCD to PPO agents trained to solve each game in Procgen .
Revised: We began by applying the LCD to PPO agents trained to solve each game in Procgen .

Original: Doing so revealed a clear taxonomy of the computational challenges of each game (Fig.
Revised: Doing so revealed a clear taxonomy of the computational challenges of each game (Fig.

Original: 1d): Fruitbot, Jumper, Miner, Heist, Chaser, and Bossfight are, in relative terms, perceptually easy and have easy reinforcement learning.
Revised: 1d): Fruitbot, Jumper, Miner, Heist, Chaser, and Bossfight are, in relative terms, perceptually easy and have easy reinforcement learning.

Original: Caveflyer, Dodgeball, Plunder, Coinrun, Ninja, Climber, and Starpilot are perceptually hard and have easy reinforcement learning.
Revised: Caveflyer, Dodgeball, Plunder, Coinrun, Ninja, Climber, and Starpilot are perceptually hard and have easy reinforcement learning.

Original: Bigfish and Leaper are perceptually easy and have hard reinforcement learning, and Maze is challenging for perception and reinforcement learning.
Revised: Bigfish and Leaper are perceptually easy and have hard reinforcement learning, and Maze is challenging for perception and reinforcement learning.

Original: The challenges of
Revised: The challenges of

Original: Maze, in particular, are likely driven by the need to trace and plan paths through a maze; routineswhich are known to be challenging for neural networks and draw upon feedback mechanisms in humans (Linsley et al., 2018a; Roelfsema et al., 2000; Roelfsema, 2006; Ullman, 1984).
Revised: Maze, in particular, are likely driven by the need to trace and plan paths through a maze; routines

Original: Maze
Revised: Maze

Original: Leaper
Revised: Leaper

Original: Chaser
Revised: Chaser

Original: Bossﬁght Heist
Revised: Bossﬁght Heist

Original: Miner
Revised: Miner

Original: Jumper
Revised: Jumper

Original: Climber
Revised: Climber

Original: Starpilot
Revised: Starpilot

Original: Ninja
Revised: Ninja

Original: Coinrun
Revised: Coinrun

Original: Dodgeball
Revised: Dodgeball

Original: Fruitbot
Revised: Fruitbot

Original: Caveﬂyer Plunder
Revised: Caveﬂyer Plunder

Original: Bigﬁsh
Revised: Bigﬁsh

Original: Dodgeball
Revised: Dodgeball

Original: Leaper
Revised: Leaper

Original: Chaser
Revised: Chaser

Original: Heist
Revised: Heist

Original: Fruitbot
Revised: Fruitbot

Original: Bossﬁght
Revised: Bossﬁght

Original: Bigﬁsh
Revised: Bigﬁsh

Original: MazeCoinrun
Revised: MazeCoinrun

Original: Starpilot
Revised: Starpilot

Original: Climber
Revised: Climber

Original: Miner
Revised: Miner

Original: Jumper
Revised: Jumper

Original: Plunder
Revised: Plunder

Original: Ninja
Revised: Ninja

Original: Caveﬂyer ϕ PPO ϕ PP
Revised: Caveﬂyer ϕ PPO ϕ PP

Original: G    ψ PPO ψ PP
Revised: G    ψ PPO ψ PP

Original: G   ρ = 0.63, p < 0.
Revised: G   ρ = 0.63, p < 0.

Original: ρ = 0.45, p < 0.
Revised: ρ = 0.45, p < 0.

Revised: which are known to be challenging for neural networks and draw upon feedback mechanisms in humans (Linsley et al., 2018a; Roelfsema et al., 2000; Roelfsema, 2006a; Ullman, 1984).

Original:  V ALIDATING LCD TAXONOMIES BY BUILDING BETTER AGENTS
Revised:  V ALIDATING LCD TAXONOMIES BY BUILDING BETTER AGENTS

Original: Our general approach for validating the computational taxonomy of the LCD is to demonstrate that the gains of targeted solutions to the challenges coincide with the LCD’s predictions.
Revised: Our general approach for validating the computational taxonomy of the LCD is to demonstrate that the gains of targeted solutions to the challenges coincide with the LCD’s predictions.

Original: We develop separate solutions for perceptual challenges and reinforcement learning challenges and describe those methods and results here.
Revised: We develop separate solutions for perceptual challenges and reinforcement learning challenges and describe those methods and results here.

Original: Developing solutions to perceptual challenges.
Revised: Developing solutions to perceptual challenges.

Original: We began by constructing a new perceptual front-end that we believed could address the perceptual challenges of Procgen games.
Revised: We began by constructing a new perceptual front-end that we believed could address the perceptual challenges of Procgen games.

Original: Inspired by the reliance of humans and other animals on optic flow to effectively and efficiently navigate through their environments (Warren et al., 2001; Warren, 2021), we turned to motion perception as the base inductive bias for our agents.
Revised: Inspired by the reliance of humans and other animals on optic flow to effectively and efficiently navigate through their environments (Warren et al., 2001; Warren, 2021), we turned to motion perception as the base inductive bias for our agents.

Original: Specifically, we rely on the motion of objects in the world to extract perceptual groups that could instruct appropriate behavior.
Revised: Specifically, we rely on the motion of objects in the world to extract perceptual groups that could instruct appropriate behavior.

Original: It has been found that training DNNs to predict the optic flow between successive frames of video can induce the ability to segment object-like superpixels from complex scenes (Liu et al., 2021).
Revised: It has been found that training DNNs to predict the optic flow between successive frames of video can induce the ability to segment object-like superpixels from complex scenes (Liu et al., 2021).

Original: Here, we build off this prior work to develop a self-supervised pre-training strategy for solving perceptual challenges in Procgen .
Revised: Here, we build off this prior work to develop a self-supervised pre-training strategy for solving perceptual challenges in Procgen .

Original: Our architecture consists of three separate DNNs: ( i ) a ResNet18 (He et al., 2015) feature pyramid network (FPN) (Lin et al., 2016) for predicting the optic flow ( Flow encoder ) between two frames from successive timepoints of a sequence, frame t and frame t + 1 , ( ii ) a biologically inspired recurrent neural network (Linsley et al., 2018a)
Revised: Our architecture consists of three separate DNNs: ( i ) a ResNet18 (He et al., 2015) feature pyramid network (FPN) (Lin et al., 2016) for predicting the optic flow ( Flow encoder ) between two frames from successive timepoints of a sequence, frame t and frame t + 1 , ( ii ) a biologically inspired recurrent neural network (Linsley et al., 2018a)

Original: ( Object encoder ) for encoding the contents of frame t , and ( iii ) a fully convolutional (Long et al., 2015) decoder that learns how to use the two encoders’ outputs to match frame t to frame t
Revised: ( Object encoder ) for encoding the contents of frame t , and ( iii ) a fully convolutional (Long et al., 2015) decoder that learns how to use the two encoders’ outputs to match frame t to frame t

Original: + 1 via a differentiable warping module.
Revised: + 1 via a differentiable warping module.

Original: This model was pre-trained to learn visual representations in a purely self-supervised manner, by minimizing the mean squared error between two successive frames sampled from the seven games of Procgen which had consistent motion (Chaser, Leaper, Dodgeball, Climber, Bossfight, Starpilot, and Bigfish).
Revised: This model was pre-trained to learn visual representations in a purely self-supervised manner, by minimizing the mean squared error between two successive frames sampled from the seven games of Procgen which had consistent motion (Chaser, Leaper, Dodgeball, Climber, Bossfight, Starpilot, and Bigfish).

Original: We sampled frames for training from 14,000 videos (2,000 per game) in batches offor 50 k iterations using the Adam optimizer (Kingma & Ba, 2014) and a learning rate of 1 e − 4 .
Revised: We sampled frames for training from 14,000 videos (2,000 per game) in batches offor 50 k iterations using the Adam optimizer (Kingma & Ba, 2014) and a learning rate of 1 e − 4 .

Original: As an alternative to our approach for self-supervision, we also evaluated the effectiveness of state-ofthe-art features for one-shot recognition from a clip-ViT-B-32 model pre-trained on 400M natural images and captions (Radford et al., 2021).
Revised: As an alternative to our approach for self-supervision, we also evaluated the effectiveness of state-ofthe-art features for one-shot recognition from a clip-ViT-B-32 model pre-trained on 400M natural images and captions (Radford et al., 2021).

Original: Perceptual challenge evaluations.
Revised: Perceptual challenge evaluations.

Original: We evaluated the LCD predictions of the perceptual challenges ϕ in each game by testing if preprocessing frames with our self-supervised front-end improved agent performance more for games with higher values of ϕ ( i.e. , the front-end was fixed and not trained with reward).
Revised: We evaluated the LCD predictions of the perceptual challenges ϕ in each game by testing if preprocessing frames with our self-supervised front-end improved agent performance more for games with higher values of ϕ ( i.e. , the front-end was fixed and not trained with reward).

Original: Because our self-supervised front-end relied on the game motion to extract perceptual groups, we focused our analysis on the same seven games with dynamic elements that it was pretrained on.
Revised: Because our self-supervised front-end relied on the game motion to extract perceptual groups, we focused our analysis on the same seven games with dynamic elements that it was pretrained on.

Original: As predicted by the LCD, our front-end improved performance significantly more for perceptually challenging games (hard ϕ ) than perceptually simple games (easy ϕ ; Fig. 3b).
Revised: As predicted by the LCD, our front-end improved performance significantly more for perceptually challenging games (hard ϕ ) than perceptually simple games (easy ϕ ; Fig. 3b).

Original: Overall, the benefit of this front-end correlated significantly with the ϕ predicted for each game (Spearman’s ρ = 0 .
Revised: Overall, the benefit of this front-end correlated significantly with the ϕ predicted for each game (Spearman’s ρ = 0 .

Original: 79 between ϕ and the AUC of performance curves, p < . 05 ).
Revised: 79 between ϕ and the AUC of performance curves, p < . 05 ).

Original: To understand the extent that our results merely demonstrate the value of pre-trained visual representations, we repeated our analysis while using a pre-trained CLIP as the perceptual front-end for agents.
Revised: To understand the extent that our results merely demonstrate the value of pre-trained visual representations, we repeated our analysis while using a pre-trained CLIP as the perceptual front-end for agents.

Original: But while CLIP embeddings yield state-of-the-art performance in one-shot classification (Radford et al., 2021), they did not help agents learn more effective policies, faster (Appendix Fig. 12).
Revised: But while CLIP embeddings yield state-of-the-art performance in one-shot classification (Radford et al., 2021), they did not help agents learn more effective policies, faster (Appendix Fig. 12).

Original: In other words, LCD’s perceptual challenges are more readily addressed by algorithmic solutions that induce object-like representations.
Revised: In other words, LCD’s perceptual challenges are more readily addressed by algorithmic solutions that induce object-like representations.

Original: Guiding algorithmic development with the LCD.
Revised: Guiding algorithmic development with the LCD.

Original: Given the success of our self-supervised perceptual front-end, an obvious approach towards building better agents is to simply use this front-end on every game.
Revised: Given the success of our selfsupervised perceptual front-end, an obvious approach towards building better agents is to simply use this front-end on every game.

Original: However, the impact of our front-end on performance was significantly blunted when measured on all games (the difference between green bars in Hard ϕ versus All Games, Fig. 3b).
Revised: However, the impact of our front-end on performance was significantly blunted when measured on all games (the difference between green bars in Hard ϕ versus All Games, Fig. 3b).

Original: This finding points to a larger problem in algorithmic development for RL: without a reliable approach for diagnosing the specific computational challenges in a game, the effectiveness of reasonable solutions can be blunted or even appear to hurt agents, as our front end nominally does here.
Revised: This finding points to a larger problem in algorithmic development for RL: without a reliable approach for diagnosing the specific computational challenges in a game, the effectiveness of reasonable solutions can be blunted or even appear to hurt agents, as our front end nominally does here.

Original: Reward shaping for resolving reinforcement learning challenges.
Revised: Reward shaping for resolving reinforcement learning challenges.

Original: The reinforcement learning challenges revealed by the LCD are due to the sparsity of rewards available to agents.
Revised: The reinforcement learning challenges revealed by the LCD are due to the sparsity of rewards available to agents.

Original: As a partial solution to this problem, we investigated the impact of “reward shaping” (Skinner, 1938) on performance.
Revised: As a partial solution to this problem, we investigated the impact of “reward shaping” (Skinner, 1938) on performance.

Original: Reward shaping is a technique from animal training where supplemental rewards are provided to make the learning problem easier Wiewiora (2010).
Revised: Reward shaping is a technique from animal training where supplemental rewards are provided to make the learning problem easier Wiewiora (2010).

Original: We adopted this for three games amenable to reward shaping: Heist, Leaper, and Maze; the latter two LCD identified as having significant reinforcement learning challenges.
Revised: We adopted this for three games amenable to reward shaping: Heist, Leaper, and Maze; the latter two LCD identified as having significant reinforcement learning challenges.

Original: In Heist, agents must collect three keys and unlock their matching doors before they can reach the rewarding gem.
Revised: In Heist, agents must collect three keys and unlock their matching doors before they can reach the rewarding gem.

Original: We provided agents with additional rewards when they collected a key or opened a door.
Revised: We provided agents with additional rewards when they collected a key or opened a door.

Original: In Leaper, agents must safely get across several lanes of roads followed by several rows of rivers.
Revised: In Leaper, agents must safely get across several lanes of roads followed by several rows of rivers.

Original: Lanes have moving cars agents must avoid, while rivers have moving logs agents must stay on.
Revised: Lanes have moving cars agents must avoid, while rivers have moving logs agents must stay on.

Original: We gave agents additional rewards whenever they reached a new lane or river.
Revised: We gave agents additional rewards whenever they reached a new lane or river.

Original: In Maze, agents must wander through a complex maze to get to a reward of cheese; however, the topological complexity of the mazes can deem rewards difficult to discover.
Revised: In Maze, agents must wander through a complex maze to get to a reward of cheese; however, the topological complexity of the mazes can deem rewards difficult to discover.

Original: To address this problem, we encourage exploration of novel locations through intermediate rewards Bellemare et al.
Revised: To address this problem, we encourage exploration of novel locations through intermediate rewards Bellemare et al.

Original: Ecoffet et al.
Revised: Ecoffet et al.

Original: In all of these cases, agents did not get these auxiliary rewards during test time.
Revised: In all of these cases, agents did not get these auxiliary rewards during test time.

Original: Reinforcement learning challenge evaluations.
Revised: Reinforcement learning challenge evaluations.

Original: Agents trained with reward shaping on Heist ( t = 1 .
Revised: Agents trained with reward shaping on Heist ( t = 1 .

Original: 69 , p < . 05 )
Revised: 69 , p < . 05 )

Original: , Leaper ( t = 41 .
Revised: , Leaper ( t = 41 .

Original: 03 , p < .
Revised: 03 , p < .

Original: 001 ), and Maze ( t = 2 .
Revised: 001 ), and Maze ( t = 2 .

Original: 69 , p < .
Revised: 69 , p < .

Original: 01 ) performed significantly better than those trained on the normal versions of each game (all tests are one-tailed t -tests;
Revised: 01 ) performed signifi- cantly better than those trained on the normal versions of each game (all tests are one-tailed t -tests;

Original: Appendix Fig. 13).
Revised: Appendix Fig. 13).

Original: Moreover, the improvements in performance from reward shaping significantly correlated with the ψ values of each game tested here (Spearman’s ρ = 0 . 92 , p < . 001 ).
Revised: Moreover, the improvements in performance from reward shaping significantly correlated with the ψ values of each game tested here (Spearman’s ρ = 0 . 92 , p < . 001 ).

Original: The LCD-derived taxonomy translates across dRL algorithms.
Revised: The LCD-derived taxonomy translates across dRL algorithms.

Original: Our findings thus far indicate that an LCD calibrated with PPO agents makes reliable predictions about the challenges of individual games in Procgen .
Revised: Our findings thus far indicate that an LCD calibrated with PPO agents makes reliable predictions about the challenges of individual games in Procgen .

Original: But despite the popularity of PPO, it has been surpassed by more recent dRL algorithms, like PPG, which achieved state-of-the-art performance on Procgen .
Revised: But despite the popularity of PPO, it has been surpassed by more recent dRL algorithms, like PPG, which achieved state-of-the-art performance on Procgen .

Original: To what extent does the LCD taxonomy derived from PPO translate to more effective dRL algorithms?
Revised: To what extent does the LCD taxonomy derived from PPO translate to more effective dRL algorithms?

Original: We tested this question by applying the LCD to agents trained with PPG (Appendix Figs.
Revised: We tested this question by applying the LCD to agents trained with PPG (Appendix Figs.

Original: and found that its predictions were significantly correlated with those from an LCD calibrated on PPO agents (Fig. 4).
Revised: and found that its predictions were significantly correlated with those from an LCD calibrated on PPO agents (Fig. 4).

Original: This means that our PPO-derived taxonomy can be relied on prospectively for algorithmic development on Procgen .
Revised: This means that our PPO-derived taxonomy can be relied on prospectively for algorithmic development on Procgen .

Original: The whole is only sometimes greater than the sum of the parts in dRL.
Revised: The whole is only sometimes greater than the sum of the parts in dRL.

Original: Training perceptionfor-action, end-to-end, is the standard approach to dRL ever since the introduction of DQNs.
Revised: Training perceptionfor-action, end-to-end, is the standard approach to dRL ever since the introduction of DQNs.

Original: Here, we tested whether agents trained on Procgen experienced a superlinear benefit of co-training perception and action or not.
Revised: Here, we tested whether agents trained on Procgen experienced a superlinear benefit of co-training perception and action or not.

Original: We did this by computing two performance differentials.
Revised: We did this by computing two performance differentials.

Original: In both cases, the baseline was a PPO agent trained on pixels.
Revised: In both cases, the baseline was a PPO agent trained on pixels.

Original: ( i ) the performance of a PPG agent trained on semantic segmentation inputs minus baseline, and ( ii ) the performance of a PPG agent trained on pixels minus baseline plus the performance of a PPO agent trained on semantic segmentation inputs minus baseline (Fig. 5).
Revised: ( i ) the performance of a PPG agent trained on semantic segmentation inputs minus baseline, and ( ii ) the performance of a PPG agent trained on pixels minus baseline plus the performance of a PPO agent trained on semantic segmentation inputs minus baseline (Fig. 5).

Original: Co-training perception for action helped performance in 6 of 16 games: Leaper, Maze, Chaser,
Revised: Co-training perception for action helped performance in 6 of 16 games: Leaper, Maze, Chaser,

Original: Jumper, Starpilot, and Bigfish.
Revised: Jumper, Starpilot, and Bigfish.

Original: Co-training did not help in 10 of 16 games: Heist, Dodgeball,
Revised: Co-training did not help in 10 of 16 games: Heist, Dodgeball,

Original: Fruitbot, Miner, Coinrun, Bossfight, Plunder, Ninja, Caveflyer, and Climber.
Revised: Fruitbot, Miner, Coinrun, Bossfight, Plunder, Ninja, Caveflyer, and Climber.

Original: This finding poses an intriguing prospect.
Revised: This finding poses an intriguing prospect.

Original: It suggests the existence of different “minibenchmarks” within Procgen that researchers should tap into, pending on whether they wish to quantify “im provements” to an agent’s policy learning al gorithm, its representation learning system, or synergies between the two.
Revised: It suggests the existence of different “minibenchmarks” within Procgen that researchers should tap into, depending on whether they wish to quantify “improvements” to an agent’s policy learning algorithm, its representation learning system, or synergies between the two.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Ever since the introduction of the DQN, the field of dRL has consistently succeeded in reaching and exceeding human performance on defined tasks: beating grand masters at the game of Go, or exceeding the average performance of humans on the Atari benchmark.
Revised: Ever since the introduction of the DQN, the field of dRL has consistently succeeded in reaching and exceeding human performance on defined tasks: beating grand masters at the game of Go, or exceeding the average performance of humans on the Atari benchmark.

Original: However, even in these successes, it was clear that certain games presented different challenges than others.
Revised: However, even in these successes, it was clear that certain games presented different challenges than others.

Original: Take “Montezuma’s Revenge”, which the original DQN could not solve.
Revised: Take “Montezuma’s Revenge”, which the original DQN could not solve.

Original: Subsequent analysis of Montezuma’s
Revised: Subsequent analysis of Montezuma’s

Original: Revenge led to the conclusion that the failures of the DQN were because of its long time horizon and sparse rewards; challenges which, these days, have positioned it as a critical test for dRL algorithms (Ecoffet et al., 2019; Roderick et al., 2018; Salimans & Chen, 2018).
Revised: Revenge led to the conclusion that the failures of the DQN were because of its long time horizon and sparse rewards; challenges which, these days, have positioned it as a critical test for dRL algorithms (Ecoffet et al., 2019; Roderick et al., 2018; Salimans & Chen, 2018).

Original: We believe that the field of dRL can benefit from a systematic approach to diagnosing such computational challenges in any benchmark.
Revised: We believe that the field of dRL can benefit from a systematic approach to diagnosing such computational challenges in any benchmark.

Original: Our LCD tool is a major step towards this goal.
Revised: Our LCD tool is a major step towards this goal.

Original: By applying our LCD to the Procgen benchmark, we discover a novel taxonomy of those games, organized according to the relative perceptual and reinforcement learning challenges of each.
Revised: By applying our LCD to the Procgen benchmark, we discover a novel taxonomy of those games, organized according to the relative perceptual and reinforcement learning challenges of each.

Original: This taxonomy delivers two prescriptions for algorithmic development in dRL.
Revised: This taxonomy delivers two prescriptions for algorithmic development in dRL.

Original: First, because the computational challenges of games in benchmarks like Procgen are not i.i.d., there will be low signal for adjudicating between algorithms designed to improve a problem that is not over-represented in the benchmark.
Revised: First, because the computational challenges of games in benchmarks like Procgen are not i.i.d., there will be low signal for adjudicating between algorithms designed to improve a problem that is not over-represented in the benchmark.

Original: Second, although humans are capable of learning perception-for-action, and dRL agents generally learn in this way, not all games in benchmarks like Procgen benefit from doing so.
Revised: Second, although humans are capable of learning perception-for-action, and dRL agents generally learn in this way, not all games in benchmarks like Procgen benefit from doing so.

Original: In the short-term, these problems can be addressed by using the LCD to select games in a benchmark that are aligned with the algorithmic design goals of researchers.
Revised: In the short-term, these problems can be addressed by using the LCD to select games in a benchmark that are aligned with the algorithmic design goals of researchers.

Original: In the long-term, there is a need in the field for new benchmarks that can tap into a wider range of computational challenges with less bias than is found in Procgen .
Revised: In the long-term, there is a need in the field for new benchmarks that can tap into a wider range of computational challenges with less bias than is found in Procgen .

Original: We believe that our LCD is an important tool for advancing the pace of progress in dRL, and we release our experimental code to support this goal.
Revised: We believe that our LCD is an important tool for advancing the pace of progress in dRL, and we release our experimental code to support this goal.

Original:  E THICS STATEMENT
Revised:  E THICS STATEMENT

Original: Our work is a contribution to deep reinforcement learning and so inherits the concerns already inherent in dRL applications.
Revised: Our work is a contribution to deep reinforcement learning and so inherits the concerns already inherent in dRL applications.

Original: Currently, our work has been applied to harmless toy videogame tasks, but improper application of dRL to real world applications can have negative externalities on society, especially when algorithms make poor decisions.
Revised: Currently, our work has been applied to harmless toy videogame tasks, but improper application of dRL to real world applications can have negative externalities on society, especially when algorithms make poor decisions.

Original: Depending on the domain of application, these can include discriminatory decisions against marginalized groups or serious injury and fatality if control of vehicles or critical systems are involved.
Revised: Depending on the domain of application, these can include discriminatory decisions against marginalized groups or serious injury and fatality if control of vehicles or critical systems are involved.

Original: Advancing agents’ ability to make better decisions would help mitigate this issue and eventually broaden the domains in which dRL can help, and our contribution of identifying limitations in current dRL agents will help in that regard.
Revised: Advancing agents’ ability to make better decisions would help mitigate this issue and eventually broaden the domains in which dRL can help, and our contribution of identifying limitations in current dRL agents will help in that regard.

Original:  R EPRODUCIBILITY STATEMENT
Revised:  R EPRODUCIBILITY STATEMENT

Original: We have taken several steps to ensure the reproducibility of our work.
Revised: We have taken several steps to ensure the reproducibility of our work.

Original: We have made all code for our modified Procgen environment and the self-supervised model for perceptual grouping available at https://anonymous.4open.science/r/lcd-procgen/ .
Revised: We have made all code for our modified Procgen environment and the self-supervised model for perceptual grouping available at https://anonymous.4open.science/r/lcd-procgen/ .

Original: For PPO, we have used the code available at https://github.com/openai/train-procgen along with the hyperparameters found in Cobbe et al.
Revised: For PPO, we have used the code available at https://github.com/openai/train-procgen along with the hyperparameters found in Cobbe et al.

Original: For PPG, we have used the code at https: //github.com/openai/phasic-policy-gradient and the hyperparameters in Cobbe et al.
Revised: For PPG, we have used the code at https: //github.com/openai/phasic-policy-gradient and the hyperparameters in Cobbe et al.

Original: Finally, we have reported the computing resources used and the number of experiments run under “dRL algorithms” of Section 3.
Revised: Finally, we have reported the computing resources used and the number of experiments run under “dRL algorithms” of Section 3.

Original: A A PPENDIX A. N ETWORK ARCHITECTURE
Revised: A A PPENDIX A. N ETWORK ARCHITECTURE

Original: Following Cobbe et al.
Revised: Following Cobbe et al.

Original: (2019), we used the convolutional actor-critic IMPALA architecture described in Espeholt et al.
Revised: (2020), we used the convolutional actor-critic IMPALA architecture described in Espeholt et al.

Original: This is a deeper network than what has been used previously, consisting of 15 convolutional layers, 16 residual blocks and 1.6M parameters.
Revised: This is a deeper network than what has been used previously, consisting of 15 convolutional layers, 16 residual blocks and 1.6M parameters.

Original: We refer the reader to (Espeholt et al., 2018) for details.
Revised: We refer the reader to (Espeholt et al., 2018) for details.

Original: For raw pixel, semantic segmentation and figure-ground segmentation, the number of input channels was fixed to three.
Revised: For raw pixel, semantic segmentation and figure-ground segmentation, the number of input channels was fixed to three.

Original: This controls the model architecture to be identical regardless of input.
Revised: This controls the model architecture to be identical regardless of input.

Original: For semantic or figure-ground masks, we replicated the masked input three times and fed that into each of the three convolutional input channels.
Revised: For semantic or figure-ground masks, we replicated the masked input three times and fed that into each of the three convolutional input channels.

Original: Our self-supervised visual model outputs 16 channel perceptual masks (at the same spatial resolution of the inputs) which are concatenated with the RGB values and fed into the actor-critic network for policy learning.
Revised: Our self-supervised visual model outputs 16 channel perceptual masks (at the same spatial resolution of the inputs) which are concatenated with the RGB values and fed into the actor-critic network for policy learning.

Original: A. 
Revised: A. R EWARD NORMALIZATION

Original: R EWARD NORMALIZATION

Original: Given that each task involves varying reward magnitudes, episode rewards r were normalized to facilitate comparison across tasks.
Revised: Given that each task involves varying reward magnitudes, episode rewards r were normalized to facilitate comparison across tasks.

Original: Normalized rewards r ′ were computed asr
Revised: Normalized rewards r ′ were computed as

Original: [Equation]
Revised: [Equation]

Original: R min R max − R min , where
Revised: , where

Original: R max is the theoretical maximum score attainable in an episode while R min is the average score a random agent would attain.
Revised: R max is the theoretical maximum score attainable in an episode while R min is the average score a random agent would attain.

Original: A. S TOCHASTIC REWARD FEEDBACK
Revised: A. S TOCHASTIC REWARD FEEDBACK

Original: Here, we prove that the expected return under policy π does not change under our manipulation of reward probability.
Revised: Here, we prove that the expected return under policy π does not change under our manipulation of reward probability.

Original: We proceed in two parts.
Revised: We proceed in two parts.

Original: First, we show that the expected reward received from a rewarding event remains unchanged.
Revised: First, we show that the expected reward received from a rewarding event remains unchanged.

Original: Then we show that the action-value function Q π ( s, a ) for policy π remains unchanged.
Revised: Then we show that the action-value function Q π ( s, a ) for policy π remains unchanged.

Original: By implication, this means the optimal policy does not change either.
Revised: By implication, this means the optimal policy does not change either.

Original: Let R ( s, a ) denote the original reward function at state s and action a .
Revised: Let R ( s, a ) denote the original reward function at state s and action a .

Original: In our manipulation, rewards are generated stochastically with probability p ( s, a ) , but reward magnitudes are rescaled by [Equation]
Revised: In our manipulation, rewards are generated stochastically with probability p ( s, a ) , but reward magnitudes are rescaled by 1 p ( s,a ) .

Original: Definition A.1 (Modified reward function) .
Revised: Definition A.1 (Modified reward function) .

Original: The modified reward function ˜ R ( s, a ) is given by
Revised: The modified reward function ˜ R ( s, a ) is given by

Original: [Equation]
Revised: [Equation]

Original: where η ( s, a ) ∼ Bern [ p ( s, a )] .
Revised: where η ( s, a ) ∼ Bern [ p ( s, a )] .

Original: Lemma A.1.
Revised: Lemma A.1.

Original: The expected value of ˜ R ( s, a ) is R ( s, a ) .
Revised: The expected value of ˜ R ( s, a ) is R ( s, a ) .

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: since E η [ η ( s, a )]
Revised: since E η [ η ( s, a )]

Original: = p ( s, a ) .
Revised: = p ( s, a ) .

Original: Using this lemma, we now prove our main result.
Revised: Using this lemma, we now prove our main result.

Original: Let Q π ( s, a ) denote the action-value function induced by the original reward function R ( s, a ) while following policy π , and let ˜ Q π ( s, a ) denote the action-value function induced by ˜ R ( s, a ) while following the same policy.
Revised: Let Q π ( s, a ) denote the action-value function induced by the original reward function R ( s, a ) while following policy π , and let ˜ Q π ( s, a ) denote the action-value function induced by ˜ R ( s, a ) while following the same policy.

Original: Theorem A.2.
Revised: Theorem A.2.

Original: The action-value function ˜
Revised: The action-value function ˜

Original: Q π ( s, a ) is given by Q π ( s, a ) .
Revised: Q π ( s, a ) is given by Q π ( s, a ) .

Original: Proof.
Revised: Proof.

Original: As is common practice in RL, we shall discount future rewards by γ ∈ [0 , 1] on each future time step.
Revised: As is common practice in RL, we shall discount future rewards by γ ∈ [0 , 1] on each future time step.

Original: Before beginning, we shall introduce some notation.
Revised: Before beginning, we shall introduce some notation.

Original: We shall denote the current time step by t .
Revised: We shall denote the current time step by t .

Original: We shall denote the state and action at any time τ by s τ and a τ .
Revised: We shall denote the state and action at any time τ by s τ and a τ .

Original: Finally, by an abuse of notation, we shall use E π [ · ] to denote expectations over state-action trajectories generated by both the policy π and the state-transition probability p ( s ′ | s, a ) , where s ′ is the successor state.
Revised: Finally, by an abuse of notation, we shall use E π [ · ] to denote expectations over state-action trajectories generated by both the policy π and the state-transition probability p ( s ′ | s, a ) , where s ′ is the successor state.

Original: We now proceed to the proof.
Revised: We now proceed to the proof.

Original: [Equation]
Revised: [Equation]

Original: Note that in our experimental manipulations, we used a single reward probability p that was independent of s and a .
Revised: Note that in our experimental manipulations, we used a single reward probability p that was independent of s and a .

Original: A. P ERFORMANCE CHANGE WITH INCREASING CREDIT ASSIGNMENT DIFFICULTY
Revised: A. P ERFORMANCE CHANGE WITH INCREASING CREDIT ASSIGNMENT DIFFICULTY

Original: To quantify the average change with increasing credit assignment difficulty, we first computed the reward curve AUC AUC ( p ) for p = 0 .
Revised: To quantify the average change with increasing credit assignment difficulty, we first computed the reward curve AUC AUC ( p ) for p = 0 .

Original: We next computed the absolute change in AUC between successive values of p ; that is, | AUC ( p = 1 . 0) − AUC ( p = 0 .
Revised: We next computed the absolute change in AUC between successive values of p ; that is, | AUC ( p = 1 . 0) − AUC ( p = 0 .

Original: | , | AUC ( p = 0 . 75) − AUC ( p = 0 . 5)
Revised: | , | AUC ( p = 0 . 75) − AUC ( p = 0 . 5)

Original: | , and | AUC ( p = 0 . 5) − AUC ( p = 0 .
Revised: | , and | AUC ( p = 0 . 5) − AUC ( p = 0 .

Original: 25) | .
Revised: 25) | .

Original: Finally, we averaged over these measures to get our final measure of average change.
Revised: Finally, we averaged over these measures to get our final measure of average change.

Original: A. S UPPLEMENTARY F IGURES
Revised: A. S UPPLEMENTARY F IGURES

Original: While performing evaluations the reward mechanisms were set to the original configuration thus effectively only testing the of these on learning learned from natural language supervision.
Revised: While performing evaluations the reward mechanisms were set to the original configuration thus effectively only testing the of these on learning learned from natural language supervision.

Original: We test the extent to which generalist visual representations support policy learning.
Revised: We test the extent to which generalist visual representations support policy learning.

Original: We find that while CLIP Radford et al.
Revised: We find that while CLIP Radford et al.

Original: (2021), a transformer-based architecture, is state-of-the-art on zero-shot image classification, its representations do not support policy learning adeptly.
Revised: (2021), a transformer-based architecture, is state-of-the-art on zero-shot image classification, its representations do not support policy learning adeptly.

Original: We train agents on both the “easy” and “hard” versions of Procgen .
Revised: We train agents on both the “easy” and “hard” versions of Procgen .

Original: While our agent was able to learn certain tasks in the “easy” mode, they predominantly struggled in the “hard” mode.
Revised: While our agent was able to learn certain tasks in the “easy” mode, they predominantly struggled in the “hard” mode.

Original: X-axis denotes the number of “interactions” an agent performs with its environment during training and Y-axis denotes rewards.
Revised: X-axis denotes the number of “interactions” an agent performs with its environment during training and Y-axis denotes rewards.

Original: We trained these agents for a total of 200M steps.
Revised: We trained these agents for a total of 200M steps.

Original: A. S PECIFICS OF OUR PERCEPTUAL PARAMETERIZATIONS OF THE Procgen BENCHMARK
Revised: A. S PECIFICS OF OUR PERCEPTUAL PARAMETERIZATIONS OF THE Procgen BENCHMARK

Original: [Table]
Revised: [Table]

Original: There are two entity IDs: (1) the player, and (2) all other fish.
Revised: There are two entity IDs: (1) the player, and (2) all other fish.

Original: [Table]
Revised: [Table]

Original: There are seven entity IDs: (1) the player, (2) player projectiles, (3) the enemy, (4) enemy projectiles, (5) the enemy’s shield, (6) meteors, and (7) explosions.
Revised: There are seven entity IDs: (1) the player, (2) player projectiles, (3) the enemy, (4) enemy projectiles, (5) the enemy’s shield, (6) meteors, and (7) explosions.

Original: [Table]
Revised: [Table]

Original: There are eight entity IDs: (1) the player, (2) enemy ships, (3) player projectiles, (4) targets, (5) the goal landing pad, (6) meteors, (7) explosions, and (8) the surrounding terrain.
Revised: There are eight entity IDs: (1) the player, (2) enemy ships, (3) player projectiles, (4) targets, (5) the goal landing pad, (6) meteors, (7) explosions, and (8) the surrounding terrain.

Original: [Table]
Revised: [Table]

Original: There are seven entity IDs: (1) the player, (2) enemies, (3) enemies in their weakened state, (4) enemy eggs, (5) small collectible squares, (6) large collectible stars, and (7) the walls.
Revised: There are seven entity IDs: (1) the player, (2) enemies, (3) enemies in their weakened state, (4) enemy eggs, (5) small collectible squares, (6) large collectible stars, and (7) the walls.

Original: [Table]
Revised: [Table]

Original: There are four entity IDs: (1) the player, (2) enemies, (3) the star rewards, and (4) the surrounding walls, ground, and platforms.
Revised: There are four entity IDs: (1) the player, (2) enemies, (3) the star rewards, and (4) the surrounding walls, ground, and platforms.

Original: [Table]
Revised: [Table]

Original: There are eight entity IDs: (1) the player, (2) enemies, (3) enemy barriers, (4) saw obstacles, (5) lava obstacles, (6) the goal star, (7) crates, and (8) the surrounding walls and ground.
Revised: There are eight entity IDs: (1) the player, (2) enemies, (3) enemy barriers, (4) saw obstacles, (5) lava obstacles, (6) the goal star, (7) crates, and (8) the surrounding walls and ground.

Original: [Table]
Revised: [Table]

Original: There are seven entity IDs: (1) the player, (2) player projectiles, (3) the enemies, (4) enemy projectiles, (5) the locked exit, (6) the unlocked exit, and (7) the walls.
Revised: There are seven entity IDs: (1) the player, (2) player projectiles, (3) the enemies, (4) enemy projectiles, (5) the locked exit, (6) the unlocked exit, and (7) the walls.

Original: fruitbot
Revised: fruitbot

Original: There are eight entity IDs: (1) the player, (2) player projectiles, (3) rewarding objects, (4) penalizing objects, (5) locks, (6) locked doors, (7) the goal, and (8) barriers and the terrain below and above the beginning and end of each episode, respectively.
Revised: There are eight entity IDs: (1) the player, (2) player projectiles, (3) rewarding objects, (4) penalizing objects, (5) locks, (6) locked doors, (7) the goal, and (8) barriers and the terrain below and above the beginning and end of each episode, respectively.

Original: [Table]
Revised: [Table]

Original: There are six entity IDs: (1) the player, (2-4) up to three unique keys and their corresponding locks, (5) the goal crystal, and (6) the walls
Revised: There are six entity IDs: (1) the player, (2-4) up to three unique keys and their corresponding locks, (5) the goal crystal, and (6) the walls

Original: [Table]
Revised: [Table]

Original: There are six entity IDs: (1) the player, (2) spikes, (3) the goal, (4) the needle of the compass and the bar indicating the distance to the goal, (5) the circle of the compass, and (6) the surrounding terrain
Revised: There are six entity IDs: (1) the player, (2) spikes, (3) the goal, (4) the needle of the compass and the bar indicating the distance to the goal, (5) the circle of the compass, and (6) the surrounding terrain

Original: [Table]
Revised: [Table]

Original: There are six entity IDs: (1) the player, (2) cars, (3) the road, (4) log platforms, (5) the water, and (6) the finish line.
Revised: There are six entity IDs: (1) the player, (2) cars, (3) the road, (4) log platforms, (5) the water, and (6) the finish line.

Original: [Table]
Revised: [Table]

Original: There are three entity IDs: (1) the player, (2) the goal cheese, and (3) the walls.
Revised: There are three entity IDs: (1) the player, (2) the goal cheese, and (3) the walls.

Original: [Table]
Revised: [Table]

Original: There are five entity IDs: (1) the player, (2) boulders, (3) crystals, (4) the exit, and (5) dirt.
Revised: There are five entity IDs: (1) the player, (2) boulders, (3) crystals, (4) the exit, and (5) dirt.

Original: [Table]
Revised: [Table]

Original: There are seven entity IDs: (1) the player, (2) player projectiles, (3) the goal mushroom, (4) bombs, (5) explosions, (6) the bar indicating jump charge, and (7) the surrounding walls, ground, and platforms.
Revised: There are seven entity IDs: (1) the player, (2) player projectiles, (3) the goal mushroom, (4) bombs, (5) explosions, (6) the bar indicating jump charge, and (7) the surrounding walls, ground, and platforms.

Original: [Table]
Revised: [Table]

Original: There are eight entity IDs: (1) the player and ally ships, (2) player projectiles, (3) enemy ships, including the cue ship, (4) the circle behind the cue ship, (5) barriers, (6) explosions, (7) the time-countdown status bar, and (8) the points-accrued status bar.
Revised: There are eight entity IDs: (1) the player and ally ships, (2) player projectiles, (3) enemy ships, including the cue ship, (4) the circle behind the cue ship, (5) barriers, (6) explosions, (7) the time-countdown status bar, and (8) the points-accrued status bar.

Original: [Table]
Revised: [Table]

Original: There are eleven entity IDs: (1) the player, (2) player projectiles, (3) slow enemies flyers, (4) fast enemies flyers, (5) enemy flyer projectiles, (6) enemy turrets, (7) enemy turret projectiles, (8) meteors, (9) clouds, (10) the finish line, and (11) explosions.
Revised: There are eleven entity IDs: (1) the player, (2) player projectiles, (3) slow enemies flyers, (4) fast enemies flyers, (5) enemy flyer projectiles, (6) enemy turrets, (7) enemy turret projectiles, (8) meteors, (9) clouds, (10) the finish line, and (11) explosions.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: and
Revised: and

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ships
Revised: ships

Original: [Table] [Table]
Revised: [Table] [Table]

Original: points
Revised: points

Original: [Table]
Revised: [Table]

Original: status
Revised: status

Original: [Table]
Revised: [Table]

Original: other
Revised: other

Revised: [Table]

Revised: A. I MPROVEMENTS IN GENERALIZATION PERFORMANCE ACROSS EASY AND HARD ϕ TASKS

Original: [Table]
Revised: [Table]